On Singletons

Inspired by a Twitter discussion with Liam Devine about my previous blog post, I've decided to take a bit of time to talk about my opinion on Singletons. In my previous post, I used a lot of them, though generally, I'm not a huge fan. In fact, I've been known to dedicate non-trivial amounts of time in talks to the subject of why singletons suck. Singletons are just globals under another name, and they have almost all of the same drawbacks as globals, and don't provide too many benefits. Each singleton is a potential bottleneck, a potential threading nightmare, and almost always creates giant side effects throughout the code base every time they're used. In addition, if you're doing unit testing, use of singletons / globals requires significant set-up overhead, which can make tests more complicated.

So what gives? Why am I using them here?

Generally, there's nothing intrinsically wrong with a Singletons or globals. It's their overuse that becomes a problem. As mentioned above, each singleton needs to be managed carefully because it can become a potential bottleneck across your entire code. However, when you're working with something that does need to be global, a Singleton is sometimes a better option, since you can insist certain conditions be met before an instance can be returned.

A good example is a Fixed String Table. If you're unfamiliar with this, here's the idea: every string in your code is translated to a hashed number. In some cases, this translation is done manually, but for the sake of code readability and simplicity, I do the translation in code any time a FixedString object is created. In order to make sure that same string always returns the same hash, and to make sure we can translate the hash to a string, we put the string into a table. This table could be a static member of FixedString, but if you want to, say, tell the table where to allocate its memory from, or how much memory it's allowed to use, you need to initialize said table. Again, you could use a static method for this, but this is hiding what's actually happening. Furthermore, if your table class is any amount of complex, or has functions for accessing usage data, you're essentially going to be pushing that all into static methods of FixedString. It's a pain, when you are, in essence, creating a global interface anyway, since FixedString itself will probably be in global scope. At least by moving things into a singleton you potentially have the option of creating multiple tables in the future, should you so choose.

But what about all those Factory singletons from my initial post?

First, a point of clarification: in the previous post, I made it look like every factory was a singleton. I did this for simplicity, and I apologize. It actually isn't true. My factories aren't singletons. In reality, the code reads like this:

Application::Instance()->GetFactory()->RegisterComponent(…);

Application is one of the other few singletons I have in my engine. Now, I'll admit that this isn't much better than having the factories themselves be singletons, since they're retrievable from a singleton anyway. If you look at the code, it looks like my component code is now coupled to both the factory AND the application.

But in reality that's not true. My factories aren't coded or tested as singletons. If I wanted to create another factory I could, and all my unit tests will still work. And the components themselves aren't coupled to the factories or the application at all. The only thing that couples factories, the components, and the application is the auto-registration system, which serves as a bridge between all three systems. Each system is still usable stand alone.

This just leaves whether or not the Application as a singleton / manager is a good idea. The answer is, no. No it isn't. But since it is one of few, it sticks out like a sore thumb when you type it, and in code review. Every time you see or type Application::Instance() should be like warning lights going off in your brain that you're potentially doing something dangerous. BUT, since I'm mostly doing this during initialization, and at no other point, I'm okay with it.

Getting Around Indeterminate Initialization

So, as I'm sure many seasoned veterans of know, the initialization order of static variables is undefined by the C++ standard. While this generally isn't a problem, it can quickly become a problem if you're doing something like overriding the memory manager, as is the case in most games. Any structure that wants to call out to a custom allocator, must wait for the allocator to be initialized. One way around this is to have the allocator as a self-instantiating singleton, but what if you're doing something more complicated?

An example is a self-registration system. This is a fairly common occurrence: you have a factory class that takes care of instantiating other classes based on something data-driven. In my scenario, this is a component system, but it really could be anything. Each component needs to register itself with the factory class so the factory can proceed to instantiate them. You have two options here: one is to have a giant init method that looks something like this:

void InitFactory()
{
     Factory.RegisterComponent(ComponentA);
     Factory.RegisterComponent(ComponentB);
     Factory.RegisterComponent(ComponentC);
     Factory.RegisterComponent(ComponentD);
     Factory.RegisterComponent(ComponentE);
     // and so on...
}

The problem with this is that it requires you #include every component header, and remember to register every component manually. Not my idea of fun.

The second is to use an auto registration system. Basically, you create an auto-registration template class:

template <class Component>
class ComponentAutoRegister
{
public:
	ComponentAutoRegister();
};

template <class Component>
ComponentAutoRegister<Component>::ComponentAutoRegister()
{
     Factory::Instance()->RegisterComponent(...);
}

And then add that to any component class:

class MyComponent
{
    static ComponentAutoRegister<MyComponent> __AutoRegistration;
};

When the static auto-registration template class is constructed, it will automatically register your class with the factory.

The problem is this: If your factory class has any other dependencies, it will need to initialize those first. If this usually happens post-main, or as part of any other variable being statically initialized, you can't use this auto-registration system. Damn.

Here's my scenario: I have a table of fixed strings that needs to be initialized with a table page size and an allocator before it can be used. My component factory uses fixed strings as keys for each component. Instead of attempting to make a long chain of singletons which would only end up coupling all the systems together (something I wanted to avoid) I slightly edited my auto-registration system to auto-register a callback instead, which is then called post initialization:

template <class Component>
class ComponentAutoRegister
{
public:
	ComponentAutoRegister();

	static void PerformAutoRegister();
};

template <class Component>
void ComponentAutoRegister<Component>::PerformAutoRegister()
{
	Factory::Instance()->RegisterComponent(...);
}

template <class Component>
ComponentAutoRegister<Component>::ComponentAutoRegister()
{
	Application::AddPostInitHook(&PerformAutoRegister);
}

The callbacks are held in an opaque linked list, managed by a zero-overhead pool pulled out of system memory. It could actually be held in an array as well, but being able to free the memory once post-initialization is complete makes me happy.

And this could easily be extended to support multiple layers of auto-registered post-initialization, so long as the layers are kept free of dependencies. For now, I only need the one layer, so I'll keep it simple.

This may seem hacky, but it does work pretty well. What are your thoughts?

Thoughts on Crunch

Crunch has been coming up in the news again, mostly because of the Team Bondai issues, and generally, I like to avoid talking about issues like this on my blog, but I figured I'd share some of my thoughts and see what people think. As a disclaimer, I have never gone through what anyone would call a "death march," which is what Team Bondai had to deal with. I've had to do crunch, but never on a huge magnitude, which I'll get to. I'm also going to take what some would say is a "controversial" stance on crunch. Here's the not controversial part: crunch is always bad, never required, and always avoidable, and a team should do everything it can to avoid it. But it's not always that simple.

Types of Crunch

The thing is, there are lots of types of crunch. And they're all different, and affect the team in different ways, and on different levels. In my mind, there are three major types:

  1. Self-imposed crunch: This is the crunch that some people in the game industry do on their own accord. They stay late every night because they want to. They want to get that one last thing in, they want to polish the hell out of whatever, or get that one last boss battle in. This is a personal choice, and there's really not much you can do about it. However, it should be clear to other team members that this type of behavior is not condoned or required. Don't reward them for this behavior. Otherwise you get into the second type of crunch.
  2. Team de-facto crunch: This is where the team "decides on its own" that it's going to crunch, when the culture necessitates it, or when management does nothing to stop it. When too many people start self-imposing crunch, other people at the team will start to look down on those that "aren't working as hard," and peer pressure will cause them to stay later, or work longer. This is particularly true in companies with lots of younger workers who are not only particularly vulnerable to this type or peer pressure (they really want to impress their superiors) and also believe that this is just the way the game industry is. If this bubbles up to management, there becomes this situation where long hours are expected, and people that don't do them are considered less valuable employees. Don't let this happen. As a manager, it is your job to make sure self-imposed crunch is seen as an anomaly, and to do your best to keep people from doing it.
  3. Mandatory Crunch: Just what it sounds like. Management, for some reason, requires longer hours. This is the most common form of crunch we hear about, and the worst. To producers / managers out there, just because you do the same hours does not make mandatory crunch okay. Actually, nothing really makes mandatory crunch okay. There are no excuses here. Mandatory crunch is always a failure somewhere down the pipe.

The first two types of crunch are cultural and occur at all studios, but can be mitigated once you recognize the warning signs.

In addition there are three levels of crunch:

  1. Mitigated crunch: The increase in hours is not huge, maybe 10 extra hours of work during the week, the length is known and can be planned for.
  2. Long crunch: The increase in hours is high. Working 60 to 80 hours during a week with no break. The length can still be known, but may be a long way off.
  3. Death march: The increase in hours is high, and there is no end in sight, or the end is repeatedly pushed back.

A Mandatory Death March is the worst type of crunch, and when we talk about "crunch" we're usually talking about mandatory long crunch or mandatory death march. There are no excuses for them, but we should take a look at the causes to figure out if they're avoidable.

Causes of Mandatory Crunch

Everyone looks at crunch as a failure of management. I agree, but there are a lot more factors to it than that. Here, in my mind are the general causes of crunch.

  1. Company Policy: The Epic model. There are no important deadlines coming up, there's nothing that's actually requiring crunch, but the company has decided to require everyone always work 60 to 80 hour weeks. Not okay. There is no excuse for this other than management being completely inept. It affects work quality and general quality of life. Don't do it.
  2. Bad Information: This is what happened to Team Bondai. They were constantly told (and believed) that the game was almost finished. In such cases, many team members, believing the end is in sight, will do self-imposed crunch, which eventually will lead to de-facto crunch. In addition, this is where a death march starts, and tends to never end.
  3. Bad Planning: This is the most common cause of crunch, and management gets rightly blamed for it all the time. But there are two types of bad planning:
    1. Bad Low Level Planning: Even the best management team can make a mistake, because they can only make decisions based on what they're told by the people below them. If what they're told is wrong, their planning will be wrong. So, if you're at the low level, don't fudge your numbers. Tell management the truth so they can make the right decisions.
    2. Bad High Level Planning: Even with all the right information, managers, especially bad managers, will plan incorrectly.
  4. Shifting Requirements: Even with good planning and good information, games are a highly creative and highly iterative product, and when something's not working or isn't fun, it has to be changed. Unlike other industries where you can work "to spec", the "spec" of a game is frequently fluid, which makes planning for the long term almost impossible. This, unlike the other causes, isn't anyone's fault, per-se, since a team can come to the mutual decision that something needs to be reworked, but it's something that frequently gets ignored while doing planning
  5. Deadline: When all is said and done, this is the real cause of crunch. When a deadline is involved any amount of error in planning or requirements hits a head when the deadline looms. Some studios are in a position where the deadline can be pushed, but not all, especially when dealing with a multi-game publisher that has to allocate resources at various times (QA, marketing, operations, etc) to make sure the game launches without a hitch, so they can move on to the next one. In these situations, moving the deadline is impossible.

The Myths: Crunch is a Given and Crunch is Always Avoidable

Both of the above statements are wrong. If the causes of crunch are listed above, avoiding crunch is as simple as avoiding those mistakes. Don't make crunch company policy, always give your employees all the information, plan well at all levels, have a solid design document, and don't have a hard deadline. Simple. But, if any of those slip, you have to make a hard decision: cut features, cut polish, or crunch.

If your company has a lot of experience or a lot of leeway, you can always avoid crunch. However, if you're a start-up working for a publisher, on a hard launch deadline, under contract, and you want to make the best game possible, chances are someone, and it's not necessarily management, is going to make a mistake, and that mistake is going to cause some amount of crunch.

Was it avoidable? Yes, probably at some point it was, and yes someone made a mistake somewhere. Crunch is always a failure somewhere down the line. But at some point in the project, it will become obvious that some amount of crunch will be needed. The next step is to mitigate it.

Mitigating Crunch

Crunch does not always mean Quality of Life or Quality of Work has to suffer, it's just that in most circumstances it does, because of the way it's implemented. I have gone through two sets of crunch, both mandatory, one long, one mitigated, and both sucked. But there are a few things that I found helped me, and I think would help others, get through any period of crunch

  1. Provide All the Information: Inform everyone how much more you expect them to work, what needs to be done, and where the deadline is. If ANY of this information changes, make sure the team knows. I have been in and heard about crunch situations where information about deadlines or cut features was withheld, and this only makes people angry
  2. Allow people to choose their extra hours: People are different and have different home lives. Some people work better in the morning, others at night, others over the weekend. Do not blanket ask everyone to stay late. Instead, let them choose when to work and they will choose times that will not only make them most productive, but will interfere with their lives the least.
  3. Allow people to work from home: Set up your infrastructure to let people work from home. Some people can be just as productive at home as they can be in the office, and this removes the stress of a long commute and improves QoL since they can be around for their loved ones.
  4. Allow people during longer periods of crunch to take some time off: This is huge for me, and I've never been anywhere that does this. If your crunch is going to last a long time, or looks like it will need to be lengthened, give people a day off. Say "Okay, everyone can take a day off free sometime in the next two weeks." Giving them a time frame allows them to plan for things like short trips, or time with their family. They'll come back rested, happier and more productive.
  5. Make sure they have and know the reward: In a start-up, the reward is in the stock, but in larger companies, make sure people know they will be compensated of extra hours, and then compensate them.

Hopefully this makes sense. I don't like crunch either, and it is avoidable for companies that have really experienced teams and a certain amount of give, but some companies do not and will not have the luxury to avoid crunch entirely. Obviously, great companies will always avoid crunch, usually because they've learned from their mistakes. But realize that a good company may crunch, but will always find a way to mitigate it when and if it happens, and will learn from their mistakes on their way to becoming a great company.

Cardboard Jam: Best Jam Ever

This weekend Boston Game Jams ran Cardboard Jam, a game jam where everyone made board or card games instead of making digital games. I have to say, I think I had way more fun at this game jam than at almost any other digital jam I've been a part of for a few reasons.

First, there's no tech choices or learning curve. At game digital game Jams, the first thing you have to do (once you have a game you want to create) is decide what technology you want to work in. This can be tough when you have multiple people who all come with different tech backgrounds and make them try to work together. Either you end up choosing teams based on tech that people know, or a few people end up working in tech they are unfamiliar with. This can make many digital game jams more about overcoming technical challenges, rather than overcoming design challenges in the game.

Second, other than the theme of the materials, there was no theme for this Jam. Though originally I thought this was a detriment, I am beginning to think that the lack of theme contributed to making this Jam awesome. In every game jam I've participated in, the themes have been aesthetic. They're themes like "immigration," "extinction," or "deception," and although they offer a good constraint on the Jam (and place everyone on equal footing for implementing an idea), they force most Jammers to think in terms of aesthetic first, mechanic last. This is why most Jammers can't "finish," because they're actually pushing for an aesthetic, rather than mechanical goal.

Last, jamming in board games meant that iteration cycles were really, really fast, which meant that you could focus on and tune the game and the mechanics quickly, which resulted in better games across the board. The game I worked on took about 10 minutes to play, and after initial discussion, we did nothing but play it, with short discussions in between. We probably play tested the game 50 times, each time with slightly tweaked rules to attempt to address problems we were seeing in the previous play through. Iteration times like this not only let you learn more about the game you're creating, but more about game design in general, since you see the results of you actions quickly. This speed of iteration and learning would have been impossible in a digital game jam, since implementing rule changes requires too many cycles. In digital game jams, if you get two to three full playtests in before time is up you're lucky, and that's including for teams that use prebuilt engines like Unity.

So in closing, if you get a chance to participate in a board game jam, you absolutely should. There's less stress, more learning, and way more collaboration. That, and you end up with an actually complete board game at the end, which is super awesome.

The Post Where I Update My Blogroll

I was at Northeastern last night talking with programming students about getting into the game industry.  As with most students, they were very curious about what, specifically, they should learn in order to make it easier for them to get into the game industry, and where they should learn it.  I gave some standard advice (which I'll probably write up a blog post about at some point, maybe?) but I realized that one way to find really good places to learn is through the blogrolls and recommendations of talented developers.

And then I realized I haven't updated my blogroll, probably in a year at least.  So, I've updated it this morning, fixing links to places like t-machine, removing some long dead blogs, and making a few additions.  Some recent additions:

  • Games From Within: Noel is now an iPhone developer, but his development methodologies are really what I look at in his blog.  He updates fairly frequently and his articles are almost always good.
  • Bitsquid: I'm not sure really what Bitsquid is working on, but their programming posts are fantastic.
  • Insomniac R&D: Awesome posts from Insomniac devs
  • #AltDevBlogADay: There is so much awesome in this (basically aggregate) feed that I often I can't keep up with it.
  • Brett's Footnotes: Brett is a lead programmer at Bethesda Game Studios, but wasn't there when I was.  He's awesome and his posts are always interesting.

I will do my best to keep my blogroll up to date from now on, though I can't make any promises.

GDC Career Seminar Talk: 20 Lessons Learned

GDC is over, and my talk "From College To Industry: 20 Lessons Learned for Getting the Most out of Your Early Career" went fairly well I think.  The room was packed, which was no small feat considering how big it was, but I think that was mainly because those attending the Career Seminar had nowhere else to go during that time.  I could be wrong, though.  The title could have been that good.

Regardless, the slides for the talk are now posted and available for anyone.  I've been asked to give this talk a few more times, but I think I'm going to revise it a bit.  I really want to emphasize the two points about knowing business and making sure all the founders of a company are on the same page.  Of all the points in my talk, these two feel like the most important, but also the most belabored by other speakers, and also the most ignored.  They may be lessons that you have to learn the hard way, but want to do everything I can to really hammer those points home.  Hopefully I can accomplish that with revisions.

If you attended the talk, I'd love to hear your opinions and feedback.

Data Driven Programming Talk

I've posted my talk on getting the most out of data driven programming that I gave at Game Forum Germany a few weeks ago on my website. The talk recommends reflection as solution to a lot of the potential pitfalls of needing a single editable location for fields and properties and, interestingly, fellow Toolsmith Geoff Evans has an article in Game Developer this month about that same topic, and specifically how they solved it in Helium. So if you're thinking about using reflection to do automatic UI generation and serialization, definitely give that article a read.

I've been told that there will be video of the talk at some point, and I'll let you know when that's available.

Upcoming Talks

Just to keep people in the know, I'm giving two talks in the coming months. The first will be at Games Forum Germany (site is in German, obviously, thought the talk will be in English) on either January 27th or 28th, I don't really know which. The talk is titled "Data Driven is Half the Battle" and will be talking about why just setting up data driven code isn't enough, and how (both from a high level and a low level) you can create a system for giving others on your team the ability to edit your data quickly and easily.

The second talk will be at GDC, as part of the Game Career Seminar, which takes place on the Friday of the conference and is focused at students. That talk is called "From College to Industry: 20 Lessons for Getting the Most out of your Early Career" and is about, well, getting the most out of your early career. It is basically a lessons learned talk about my 10 years attending GDC, and my 6 years in the game industry. While that isn't a really long time in the game industry (anymore), I feel that some of the lessons I've learned over that time will only really be apropos to students for a short time, and thus it is most useful for me to give the talk now, when it will be most useful to them. I will be practicing this talk twice in front of student groups around the Boston area, once at WPI and once at a local SIGGRAPH meeting, to make sure that the talks really hit all the right notes and lessons and are as useful as possible to the GDC student population.

Both talk's slides will be posted on line after the talks, and the GFG talk will probably be cross posted to the Toolsmiths, since it is a tools related talk.

5 Games You Should Play

I've been thinking recently that I should make a list of my favorite games of all time, especially as I find it harder and harder to find games that I like. For now, though, I want to mention several games that don't have the wide spread popularity I think they should, even in the development community. I'm always surprised when I'm talking to game developers and bring up specific games that I feel are "cannon" that they haven't played. So here's a list of 5 specific titles I think you should play.

  1. Strife: I mention Strife in conversation a lot, and I'm always surprised how few people have even heard of it. Strife is a plot based, almost open world RPG like first person shooter, based on the Doom 2 engine, and was actually the last game to use the Doom 2 engine. In my opinion, the game was largely ignored mostly because it game out the same year as Quake, and the graphics couldn't compete with the more modern 3D engine. That said, the story is reasonable, the dialog is good to great at times, there are interesting choices to be made (though, unfortunately, mostly in dialog) and the level design is pretty fantastic. If you play Strife, you'll realize why I feel horribly let down by 90% of FPS games. If you can find a copy of the game, you can either play it in DOSBOX, or use the zdoom engine, which will allow the game to run natively in windows.
  2. Star Control 2: Now available in a renamed open source format, there is NO excuse for anyone to not play this game, save lack of time, but even then I'd say you should find the time. Another open world game, Star Control 2 basically gives you a galaxy to explore, and very little direction, at least at the beginning. However, it's very good about chaining elements together, or giving you pointers if you get stuck. The story and dialog are some of the best I've ever encountered in a game. The choices, again mostly though dialog, have actual impact, and you feel repercussions for most things that you do, eventually. Just play Star Control with a pad and paper, because people will tell you coordinates for things, names of stars, and information, which no in game system keeps track of, one of its few failings.
  3. Beyond Good and Evil: A platformer RPG with an interesting story, an interesting mechanic involving photography, great action / escape sequences, and amazing music, this game got passed up by the general buying public partially because the title has nothing to do with the game and partially because there was little to no marking for it. Thankfully, an HD version is being released in 2011, and you should pick it up when it comes out. When you play it, pay close attention to the puzzle design, and how the puzzles build on each other from stage to stage, as well as the variety of encounters throughout the game. In addition, note the various music, camera, and speed tricks that are used to provide the game with a really amazing atmosphere.
  4. Quest for Glory 4: If you know me, you know I can't talk about games without bringing the conversation around eventually to Quest for Glory 4. This is adventure gaming done right, with light RPG elements that keep the game especially interesting. The 4th game in the series I feel is especially interesting because it deals with a lot of really dark and mature subject matter, and although it's standard lock and key puzzles (you don't really have too many choices about how to solve puzzles), very frequently in the game you'll find yourself wondering if you actually did what was "right". The story aside, the game also has fantastic puzzle design, and does not contain any of Sierra's traditional "unsolvable" puzzles.
  5. Intelligent Qube: This game is almost unknown, and actually extremely rare. It's a real time puzzle game for the PS1 where you're a guy on a big box trying to get rid of other boxes before you get crushed by them. The game is exceedingly hard without practice, but the style of it is amazing, specifically because of the music and sound. It was so insanely epic for what was, essentially, a pretty standard puzzler. If you can find a copy (or emulate a copy) I highly recommend you do.

Simplicity in AngelXNA

I know I've promised tutorials on AngelXNA, and I actually have the first one written, but I've been distracted by crunch on my current project and don't feel comfortable just posting the first portion without more to follow. In addition, the first tutorial feels way too long, so I'm hoping to add some minor changes to Angel to make getting started even easier, specifically by setting up GameManager and default screens in ClientGame, rather than forcing you to make them yourself.

But enough about that, this post is about rendering.

Angel is really nice in that it makes a lot of things really simple and really transparent. With a few exceptions, everything does what you expect. There are very few pre-conditions for any given call in Angel, so you don't have to worry about whether you've set up X or Y before calling things, or worry that calls will fail because objects weren't in the right state. That just doesn't exist In Angel. With that mind, I'm looking for a way to add a piece of complexity and functionality to Angel without sacrificing the external simplicity.

In AngelXNA, all actors render as a rectangle by default (we didn't port over circles from Angel C++), so by just putting an actor in the scene, you get something rendering. If you want to make it a sprite, you call Actor.SetSprite or (simpler) Actor.Sprite = "ContentFile". Animations follow a similar pattern, loading a sequence of files using Actor.LoadSpriteFrames (or, again, you can use Actor.Sprite = "ContentFile_001" and it will automatically load the sequence). Then you can play the animation sequence using Actor.PlaySpriteAnimation.

The problem is, now, I want to add two additional render paths, one for rendering static and animated sprite sheets, and one that allows you to specify and load multiple named animations (so you could say Acor.PlaySpriteanimation("Jump")). This is easily done by either inheriting from Renderable or Actor, but inheriting from Renderable loses all of the functionality in Actor, and inheriting from Actor means I can't use these render paths in other sub classes of Actor (namely PhysicsActor). In addition, if these render paths were reusable *outside* of actor, it might make alleviate some problems we've had adding things as Actors that aren't just to get their rendering properties.

So my question is, how do I make this simple? One option is to just add more methods and properties. Actor.SetSpriteSheet(), Actor.SpriteSheet, Actor.LoadSpriteSheetAnim() etc. This does have the benefit of being consistent, but doesn't make those render paths reusable. I could create an IRenderPath interface and have one for each render path, using SetRenderPath to assign a specific render path to an Actor and remap the current functions to creating those render paths. In addition, it means the SpriteSheetRenderPath could make use of a reusable SpriteSheet for both tilled maps as well as character animations. This, however, I feel makes Angel overly complicated. Maybe some combination of the two makes more sense?

I feel like I've spent way too much time thinking about this, and would love to get it implemented, hopefully soon.