Jeff On Games

Edit: This post was superseded by this one. Basically, I realized that this approach for C++ is way to hard and not worth the effort. So, if you're interested in testing generated code, especially in C++, I'd go read the other article. However, if you're curious as to how to build, load and test a C .dll, this would be the article for you.

So, as you may have noticed, I’m enamored with the concept of automated testing. In my opinion, it’s really the only way you can actually be sure your code is working as intended. You can’t rely on you code reading skills, and even if could, things that “makes sense” don’t always work. Now, I know unit testing doesn’t actually prove your code works (though there are ways to do that as well), but it does let you know that it works in at least the cases you’ve tested (in perpetuity if you run the tests after every build).

Recently, I ran into an interesting problem concerning unit testing . For our metrics suite, Orbus is creating a code generator, and supplying templates for various languages. The problem was how do you write a unit test (or functional test in this case) for a code generator? In my mind there are only a few options:

1. Take a test case and write out the code you want the generator to generate. Compare the generated code to the known value with a diff utility.
2. Run the generator once, and take this as the known value. Compare the generated code to the known value, again with a diff utility.
3. In a functional test, run the generator, compile the resulting code, then run it with various inputs, testing its outputs (much like a standard unit test).

In my mind, the first and second options are an exercise in futility. In reality, you don’t care about the text in the functions, you care about the fact that the generator produced functions that take certain parameters, and produce certain output. By trying to compare the text, you don’t actually get one of the main benefits of automated testing: discovering whether changes you made internal to the code (via refactoring or by adding additional functionality) have broken it. In this case, you want to be able to change the code generated inside the functions whenever you want, so long as the signatures remain the same and so long as the produce they proper output to external libraries. Basically, you want to treat your generated code as a black box.

The problem with the third option is that it’s really hard to do for some languages. Scripting languages, it’s pretty easy right? Just load up your trusty interpreter and feed it a few scripts that utilize your library. .NET languages have it pretty easy as well, since most of them have CodeDom objects that allow you to compile assemblies on the fly and load them. .NET’s reflection also allows you to check the number and types of parameters you’re loading, so no problems there. In reality we really only have a problem with native, compiled languages. For me, this was specifically a problem with native C++. Thankfully, I’ve actually “solved” the problem (at least for a C interface) in C# unit testing. Here are the steps our functional tests go through to test our generated code:

1. Generate the code to a temporary directory and build it into a .dll. The .dll I built exposes all of the API functions that someone could use from the generated code. This isn’t hard to do using preprocessor macros and the like, and we need to do it anyway so that potential clients can use our API in a .dll if they so choose. To aid in the building process, the generator also builds a .vcproj file (it’s just XML) for all of the generated files and then uses devenv /build to build it.
2. Load the library and do any initialization. Using LoadLibrary, you can get a handle to a loaded .dll pretty easily. Initialization is where things get hairy. In our API, we need to supply a connection object to a (static) initialization function. This object is really what generates the “output” of our library, so I wanted the connection object to be accessible to our C# unit tests. To do this, I created two mock objects in C++/CLI, one managed connection object and one unmanaged connection object that contained an auto_gcroot structure to the managed connection. I then pass the native class to the initialization routine, like so:
   

   typedef bool (*InitFunc)(Connection*);
   Initfunc proc = (InitFunc) GetProcAddress(module,”InitFunction”);
   MockNativeConnection* pconnection = new MockNativeConnection(managedConnection);
   proc(pconnection);
   

   Now all calls into my libraries will use that connection object (it’s a singleton) so it’s a pretty easy way for me to check that everything is running nicely.

3. Use P/Invoke to call out to your system. Now, you could write typedefs out for all the functions you’ve generated and call them in C++/CLI using the same “GetProcAddress” from above, but since I want to do the actual tests in C#, it makes more sense to use the DllImport attribute from C# to import the methods you’re looking for. Furthermore, P/Invoke will do automatic marshaling of most of your types, and will automatically do name matching for you. In general, it's just an easier interface to use over GetProcAddress. Here’s an example:
   

   [DllImport("MyDll.dll", CharSet = CharSet.Ansi)]
   static extern int LogEvent(string asDataPoint1, int aiDataPoint2);
   
   [Test]
   public void TestLogEvent()
   {
   	LogEvent ("My Test", 1);
   
   	Assert.AreEqual("LogEvent", _connection.ProcedureName);
   	Assert.AreEqual("My Test",  _connection.Parameter[0]);
   	Assert.AreEqual(1, _conneciton.Parameter[1]);
   }
   

   This tests that my event generated method (LogEvent) told the connection object to execute the “LogEvent” procedure with the two parameters I supplied.

Now this may seem like a lot of work, but it’s worth it if you fell that you’re either going to be changing your generators a lot (or if you want the peace of mind of having these tests around) or if (like me) you’re going to be generating for a lot of different languages and platforms that can all compile to .dlls. With this library, I should be able to test any language that compiles to a native .dll no problem (so long as it exposes a C-like interface and doesn’t use any structures….)

Next, I’m going to work on grabbing C++ objects from the .dll instead of just a straight C interface and testing those. This stands to be a much more daunting task, since, unlike P/Invoke, I can’t rely on auto marshaling, and I can’t actually code the structures if I want them to be reusable. I’ll write up what I find.