GCS Professional Programmers Panel is Back

Last year and the year before, we did a panel at GDC's Game Career Seminar for students (and others) to ask questions of professional programmers. We're doing it again this year, and we want your questions!

If you have questions, you please tweet them with the hash tag #gcsppp and we'll select the best to be part of the panel. In addition, I'll be watching that hash tag during the panel and pulling our favorite questions from the audience to ask the panelists.

Our panelists this year are:

  • Mike Acton - Engine Director at Insomniac Games
  • Brett Douville - Formerly Lead Programmer at Bethesda Game Studios, now independent
  • Anna Kipnis - Senior Gameplay Programmer at DoubleFine
  • Sven Bergström - Developer for Celsius game studios and underscorediscovery

If you have questions for them, be sure to get them in early, and we hope to see you at the panel!

GDC 2013 – Programmers Panel

GDC is over, and it was fairly awesome. I'll have a quick "thoughts on GDC" post later, but I wanted to get the information for the GCS panel I participated in up as soon as possible.

GDC did not record this panel this year, but thankfully, we have a fairly good unofficial recording of the panel thanks to Jeremiah Ingham. (sssshhh, don't tell GDC)

You can also just download the file, if you'd like.

The only real resources we recommended this year were at the end, where I asked each of the panelists recommended a book, and here they are:

If you have other questions of the panelists, please feel free to ping us on twitter. We're all pretty approachable (especially if we can respond in 140 characters or less).

Transparent Persistence

I'm in the process of refactoring some code for Go Home Dinosaurs and I've run into an interesting problem.

Because GHD was originally designed to be a microtransaction based game, and because we wanted to have (essentially) cloud saves, it's using web services to keep track of player progress, coin totals, purchases, inventory, etc. Now, we want portions of this to be pushed into local save data. Because the schedule on GHD was so tight, there's a ton of code that just assumes that the server will be there, and it also (correctly) assumes that we want to update player information incrementally on the server. It's therefore filled with calls like this:

    GameDataServer.SendServerMessage("updateField_request", data);

// in some other portion of code
void HandleMessage(Message msg)
{
    switch(msg.Type)
    {
        case "updateField_response":
            // actually update the field with the new value
            field = msg.Data.NewValue;
            break;
    }
}

The problem is that this doesn't really fit with an offline mode. In order to combat this, one of my colleges wrote a "fake response" method, which, if there's no server to talk to, queues a fake "success" message for every call. While this was certainly the fastest, most straight forward way to accomplish an "offline" mode quickly, I'm not a huge fan. It's too coupled to the thought that there's a server between us and the persistence store. It also means that the server's responses and the "fake" responses need to be kept in sync, and I'm never a fan of "if you change this, make sure to change this" code. It's way too likely to break.

But how to combat this? I'm not sure I have an answer for this, given the current design. All I can do is outline what might work, and what I want.

Generally what I want is transparent persistence. I don't what to have to understand *how* the platform wants to persist things. I want to edit my data and then say "persist" and have the platform figure out the best way to perform this.

The first step here, I think, is to take a lesson from MMOs, and ignore trying to cache unconfirmed changes. Most MMOs make the change locally and send a message to the server, requesting that the change be made. For persistence, successful calls are generally ignored, because we know that what the server says and what the client says are now in sync (or are in sync from when we sent the message). It's only in the case of an error that things have to be handled. Generally, the correct response is then to rollback whatever transaction was made on the client, and hope the player doesn't notice (or present an error message). Since the server can (and should) return it's version of the variable in the error response, it means that you can remove a lot of special case caching code that can easily get mucked up (or start performing double duty, which is the case with at least one of our caching variables). It also means that platforms that always succeed in changing data don't have to send fake "success" messages. Only the (hopefully rare) errors in server calls need to be handled.

Second step is to start keeping track of "dirty" fields, and updating them as needed (whenever persist gets called). However, our current server code (which I didn't write, and some of which I'm trying to keep intact for a bit) assumes (again, correctly) updates for every single transaction. It's hard to say "the player has x coins." Instead, it asks to told how many coins the player gained. You can't send the entire inventory, you have to send "I'm adding / removing the following item from the inventory." Or "I'm purchasing the following item and it should be added to the inventory." These are common client – server interaction things, even in MMOs because it means that the server can actually confirm that the logic you're performing matches what it expects, but it does make doing transparent persistence very difficult.

The only way I can think to do this even remotely well is to push each update to the player to a platform intermediary. Platforms that persist the data locally ignore incremental updates and just save all the data once a full "persist" call is made. Platforms that persist the data remotely do the opposite, pushing each update to the server, rolling back errors, and ignoring requests for full persistence, or sending full persistence requests and to confirm that the local and remote versions are still in sync.

Resources from the Professional Programmer’s Panel

I got an email recently asking about the resources we went over in our GDC 2012 panel this year. I figured it would be useful to have these in one place, so here we go!

Game development sites mentioned

Blogs:

Books:

  • Code by Charles Petzold

Mike also recommended (several times) that you get on Twitter (http://www.twitter.com), which is a piece of advice I whole heartedly agree with. Then you can follow all of the panelists.

When we were talking about game jams, we mentioned some longer game jams, including the Experimental Gameplay Project, Ludem Dare, and the competitions held on TIGSource.

If you have other resources for beginning game programmers, let me know and I'll add them to the list.

Finding D3D9 Leaks

I ran into this problem working in a DirectX 9 application recently, and found it useful enough that I thought I'd share. We were running into an issue where returning from an alt-tab was not recreating the device properly. This is usually caused by a created D3DPOOL_DEFAULT asset not being released and recreated before attempting to recover the lost device. As a result, you get this message:

Direct3D9: (ERROR) :All user created D3DPOOL_DEFAULT surfaces must be freed before ResetEx can succeed. ResetEx Fails.
Direct3D9: (ERROR) :ResetEx failed and ResetEx/TestCooperativeLevel/Release are the only legal APIs to be called subsequently

One additional thing that was throwing me was this error message:

Direct3D9: :Window has been subclassed; cannot restore!

This can probably safely be ignored, and may be a symptom of the back buffer leaking specifically, but generally this whole error set is a sign of leaking D3D objects.

If this is something innocuous, like a single texture, you can sometimes figure it out by simply enabling the DirectX debug libraries, running and exiting your application, which will give you the AllocId of the leaking asset, which you can then break on. However, a large number of leaks indicate that something lower level may be leaking, and potentially a leaking reference to something, like the back buffer (which was our case). How to find it?

Here's what I ended up having to do.

First we did what we need to do for every bug, reduce it to the fewest steps possible.

Next, run PIX and take a full capture, until you've gotten to the point of the leak, then close. If you have a lot of steps to reproduce, you'll want to make sure you've got a lot of hard drive space, and some time on your hands, because it can take a while.

Once you have your full PIX run you can look at this column:

And set it to Never. This will show you every D3D object that's leaking, regardless of how many AllocIds it has (interesting tidbit, a single D3D object can allocate multiple AllocIds). If you move to the last frame (and wait for PIX to catch up, which, again, can take a while) you can even see how many references exist during shutdown. You could even walk each frame and see when the reference counts of your leaking assets go up.

In our case, like I said, we were leaking a reference to the back buffer. Most obvious was to look for calls to IDirect3DDevice9::GetBackBuffer, but none of those were leaking. Less obvious was to look for cases of IDirect3DDevice9::GetDepthStencilSurface and IDirect3DDevice9::GetRenterTarget(0). In our case, a call to GetRenderTarget was leaking as a result of a short circuit in a logic check, resulting in the back buffer reference to dangle in a very specific logic case.

Granted, this leaking does not pose a lost device recovery problem with DirectX 10 or DirectX 11, but since many engines are still DX9 based, I thought I would share.

Debugging Embedded Mono

It's been announced that Go Home Dinosaurs will be released using Google's Native Client technology later this year, but what isn't known (as much) is we're using Mono / C# as our "higher level" logic and scripting system. Generally, I'm a huge fan of C# as a language, and the performance benefits just seal the deal for me. The one issue we have right now, though, is debugging embedded Mono, especially in conjunction with the game running in Visual Studio. There aren't very many instructions on how to do this online, so I thought I'd share our experience.

There are two very weird issues when debugging embedded Mono. First, it currently works exactly opposite of how most debuggers work. In most soft debuggers, the application starts the server and the debugger connects to it. In Mono soft debugging, the debugger creates the server and the client connects to it. Second, the Mono soft debugger uses signals / exceptions to work halt threads. This means that you can't debug both with Visual Studio and Mono Develop, because Visual Studio will pull the signals before they get to the application. This also prevents

However, there are changes in the works from the Unity team which fix both of these issues. However, they are not currently part of a stable mono release (or in the Elijah's version for Native Client). Once they are, I may revisit this whole article, but that's where we are right now.

So, with all of that in mind, how do you get this working right now?

Set Up MonoDevelop

First up, grab MonoDevelop.

Second up, you will want to add an environment variable to your system called MONODEVELOP_SDB_TEST and set it to 1. This gives you access to the following menu item when you have a solution loaded:

Which brings up this dialog:

Now, for some reason, I've never been able to get MonoDevelop to launch the application directly. It always fails attempting to redirect standard out, and then immediately crashes (I could look at the code to figure out why this is, but I haven't had a chance to). We'll come back to this dialog in a minute.

Prepare Your Code

There are a few things you need to add to your code to get debugging to work. The first few lines can occur at initialization, regardless of whether you're debugging or not:

mono_debug_init(MONO_DEBUG_FORMAT_MONO);
mono_debug_domain_create(mMonoDomain);

These should be done before you load any assemblies.

In addition, if you're running Mono from a background thread (like we are), and not running it on your foreground thread, after initialization you need to detach the foreground thread. This is because when Mono attempts to halt all threads when it hits a breakpoint, it can only do so when you're executing Mono code. If your main thread never executes any Mono code, it never halts, which means debugging doesn't work. This is good practice regardless, because if the GC needs to halt all of your threads, it will also fail if a non-Mono executing thread is still attached.

Last thing you'll need to do in code is set up the debugger. Here's the code:

     bool isDebugging = AresApplication::Instance()->IsDebuggingMono();
     const char* options[] = {
          "--debugger-agent=transport=dt_socket,address=127.0.0.1:10000"
     };
     mono_jit_parse_options(1, (char**)options);

This needs to be combined with any other options you may have (I only have one other, which is –soft-breakpoints), and should only be done if you want to start debugging. If the Mono client can't connect to the debugging server when it starts up, it will call exit, which is no good.

Lastly, if you're using the Microsoft C# compiler, instead of the Mono one, you'll need to generate mdbs from all of your pdbs. There's thankfully a utility for this. We actually now perform this step as a post build event, which keeps the mdbs up to date with latest code.

Put it all together

Alright, so let's get this party started!

Open MonoDevelop. Set some breakpoints, then open this dialog again.

Don't change anything, and click "Listen".

Now start your application outside of Visual Studio with whatever command line arguments you chose to signal to your app you want to debug in Mono.

Everything should now connect and you should be debugging Mono in your app!

Let me know if you have any issues!

GGJ 2012: Eat Sheep (and Die)

This weekend I participated in Global Game Jam for the third time (after taking a hiatus in 2011 to go speak at conference overseas). This year, I was at the UVA site and took part in a 3 person team with my brother in law and an awesome UVA student. The result was an interesting two player competitive game called Eat Sheep (and Die).

This jam was way less stressful than any other jam I've participated in. We essentially had something fun to play by middle of day 2, and just polished things throughout the end of day 2 and day 3. As a result, this is probably the most complete jam game I've ever worked on. Very little went wrong. It was great.

So what when right?

  1. AngelXNA – Power of Angel combined with the power of XNA! This was the third time I've used Angel in a jam and every time it's improved. We were able to get levels in games quickly, spawn actors, and do all sorts of things quickly and easily. It actually worked really well. Again.
  2. Abandoning Design – The original point of the game was to have a cyclic back and forth of helping and hurting the other player. On your turn, you would have your goal, and a directive to help or hurt your opponent. On the next turn, you would be given a bonus based on how well or poorly your opponent did. At some point, we realized there was never any reason to help your opponent. It just wasn't worth it. So we abandoned the helping, turn based component, and went with real time versus. This makes it questionable whether we obeyed the theme, but whatever. The game was much better as a result.

What went wrong?

  1. AngelXNA? – So, the one thing I don't like about Angel and XNA is that it does have a huge barrier to entry for people playing the game after the Jam. I know I don't bother playing a GGJ game that requires a download unless it comes VERY highly recommended. I will, however, look at pretty much every HTML5 or Flash game. That said, Angel is awesome, so maybe I have to figure out a way to fix this. Native Client maybe?

That's it. Not too many lessons this time out because everything went so well. It's a weird feeling. Really looking forward to next year!

Updates to AngelXNA

One of the tips I gave in my recent talk is to have a tool for GGJ. My tool of choice is AngelXNA, because it combines the power of XNA with the power of Angel. When I was demoing it to the other attendees, I realized I hadn't added a few demo screens that are really necessary.

To that end, I've added two demo screens showcasing two features that are in AngelXNA that aren't in Angel. The first is the Editor. While I had a problem with the Editor in the last GGJ, Fire Hose used it a lot for prototyping and I think we've worked out most of the kinks. It's not perfect, but it does make creating levels in Angel super easy. There are shortcuts for cloning, scaling, and rotating actors, which makes the level editing process super quick.

The second feature has several aspects, but comes down to the ability to use sprite sheets and the ability to create named, data driven animations. There are two parts to this. The first part is the Tiling system. By adding a .tileinfo file parallel to any given image file, it will automatically start rendering Actors that use that image as tiled. Here's an example .tileinfo:

IsAnim = true
FrameSize = Vector2(160, 200)
AnimDirection = TiledAnimDirection.Horizontal
StartFrame = Vector2(0, 0)
EndFrame = Vector2(0, 2)

It specifies that this file contains an animation, that each frame is 160x200, that the animation frames run left to right, starts at 0,0, and ends at 0,2. You can also use a non-anim and specify a specific tile to render.

Once you have these .tileinfos specified, you can specify animation files. Here's the example anim file:

BeginAnimation("Idle", TiledAnimRenderPath.Create())
    AnimDelay = 0.03
    File = "images/devo/devo_idle_strip"
    AnimType = AnimationType.Loop
EndAnimation()

BeginAnimation("Run", TiledAnimRenderPath.Create())
    AnimDelay = 0.03
    File = "images/devo/devo_run_strip"
    AnimType = AnimationType.Loop
EndAnimation()

BeginAnimation("Fall", TiledAnimRenderPath.Create())
    AnimDelay = 0.03
    File = "images/devo/devo_fall_strip"
    AnimType = AnimationType.OneShot
EndAnimation()

This allows you to call just Actor.PlayAnimation("Idle") or Actor.PlayAnimation("Run").

This is still a bit more complicated than I'd like, but does work really well. The only thing I'd like to improve is the specification of the render path. AngelXNA actually generally selects the correct render path automatically, but the data driving system, specifically the animation system, isn't detecting its render path. I can probably fix this, which I'll hopefully do after GGJ.

On Singletons

Inspired by a Twitter discussion with Liam Devine about my previous blog post, I've decided to take a bit of time to talk about my opinion on Singletons. In my previous post, I used a lot of them, though generally, I'm not a huge fan. In fact, I've been known to dedicate non-trivial amounts of time in talks to the subject of why singletons suck. Singletons are just globals under another name, and they have almost all of the same drawbacks as globals, and don't provide too many benefits. Each singleton is a potential bottleneck, a potential threading nightmare, and almost always creates giant side effects throughout the code base every time they're used. In addition, if you're doing unit testing, use of singletons / globals requires significant set-up overhead, which can make tests more complicated.

So what gives? Why am I using them here?

Generally, there's nothing intrinsically wrong with a Singletons or globals. It's their overuse that becomes a problem. As mentioned above, each singleton needs to be managed carefully because it can become a potential bottleneck across your entire code. However, when you're working with something that does need to be global, a Singleton is sometimes a better option, since you can insist certain conditions be met before an instance can be returned.

A good example is a Fixed String Table. If you're unfamiliar with this, here's the idea: every string in your code is translated to a hashed number. In some cases, this translation is done manually, but for the sake of code readability and simplicity, I do the translation in code any time a FixedString object is created. In order to make sure that same string always returns the same hash, and to make sure we can translate the hash to a string, we put the string into a table. This table could be a static member of FixedString, but if you want to, say, tell the table where to allocate its memory from, or how much memory it's allowed to use, you need to initialize said table. Again, you could use a static method for this, but this is hiding what's actually happening. Furthermore, if your table class is any amount of complex, or has functions for accessing usage data, you're essentially going to be pushing that all into static methods of FixedString. It's a pain, when you are, in essence, creating a global interface anyway, since FixedString itself will probably be in global scope. At least by moving things into a singleton you potentially have the option of creating multiple tables in the future, should you so choose.

But what about all those Factory singletons from my initial post?

First, a point of clarification: in the previous post, I made it look like every factory was a singleton. I did this for simplicity, and I apologize. It actually isn't true. My factories aren't singletons. In reality, the code reads like this:

Application::Instance()->GetFactory()->RegisterComponent(…);

Application is one of the other few singletons I have in my engine. Now, I'll admit that this isn't much better than having the factories themselves be singletons, since they're retrievable from a singleton anyway. If you look at the code, it looks like my component code is now coupled to both the factory AND the application.

But in reality that's not true. My factories aren't coded or tested as singletons. If I wanted to create another factory I could, and all my unit tests will still work. And the components themselves aren't coupled to the factories or the application at all. The only thing that couples factories, the components, and the application is the auto-registration system, which serves as a bridge between all three systems. Each system is still usable stand alone.

This just leaves whether or not the Application as a singleton / manager is a good idea. The answer is, no. No it isn't. But since it is one of few, it sticks out like a sore thumb when you type it, and in code review. Every time you see or type Application::Instance() should be like warning lights going off in your brain that you're potentially doing something dangerous. BUT, since I'm mostly doing this during initialization, and at no other point, I'm okay with it.