Jeff On Games

Crunch has been coming up in the news again, mostly because of the Team Bondai issues, and generally, I like to avoid talking about issues like this on my blog, but I figured I'd share some of my thoughts and see what people think. As a disclaimer, I have never gone through what anyone would call a "death march," which is what Team Bondai had to deal with. I've had to do crunch, but never on a huge magnitude, which I'll get to. I'm also going to take what some would say is a "controversial" stance on crunch. Here's the not controversial part: crunch is always bad, never required, and always avoidable, and a team should do everything it can to avoid it. But it's not always that simple.

Types of Crunch

The thing is, there are lots of types of crunch. And they're all different, and affect the team in different ways, and on different levels. In my mind, there are three major types:

1. Self-imposed crunch: This is the crunch that some people in the game industry do on their own accord. They stay late every night because they want to. They want to get that one last thing in, they want to polish the hell out of whatever, or get that one last boss battle in. This is a personal choice, and there's really not much you can do about it. However, it should be clear to other team members that this type of behavior is not condoned or required. Don't reward them for this behavior. Otherwise you get into the second type of crunch.
2. Team de-facto crunch: This is where the team "decides on its own" that it's going to crunch, when the culture necessitates it, or when management does nothing to stop it. When too many people start self-imposing crunch, other people at the team will start to look down on those that "aren't working as hard," and peer pressure will cause them to stay later, or work longer. This is particularly true in companies with lots of younger workers who are not only particularly vulnerable to this type or peer pressure (they really want to impress their superiors) and also believe that this is just the way the game industry is. If this bubbles up to management, there becomes this situation where long hours are expected, and people that don't do them are considered less valuable employees. Don't let this happen. As a manager, it is your job to make sure self-imposed crunch is seen as an anomaly, and to do your best to keep people from doing it.
3. Mandatory Crunch: Just what it sounds like. Management, for some reason, requires longer hours. This is the most common form of crunch we hear about, and the worst. To producers / managers out there, just because you do the same hours does not make mandatory crunch okay. Actually, nothing really makes mandatory crunch okay. There are no excuses here. Mandatory crunch is always a failure somewhere down the pipe.

The first two types of crunch are cultural and occur at all studios, but can be mitigated once you recognize the warning signs.

In addition there are three levels of crunch:

1. Mitigated crunch: The increase in hours is not huge, maybe 10 extra hours of work during the week, the length is known and can be planned for.
2. Long crunch: The increase in hours is high. Working 60 to 80 hours during a week with no break. The length can still be known, but may be a long way off.
   
3. Death march: The increase in hours is high, and there is no end in sight, or the end is repeatedly pushed back.
   

A Mandatory Death March is the worst type of crunch, and when we talk about "crunch" we're usually talking about mandatory long crunch or mandatory death march. There are no excuses for them, but we should take a look at the causes to figure out if they're avoidable.

Causes of Mandatory Crunch

Everyone looks at crunch as a failure of management. I agree, but there are a lot more factors to it than that. Here, in my mind are the general causes of crunch.

1. Company Policy: The Epic model. There are no important deadlines coming up, there's nothing that's actually requiring crunch, but the company has decided to require everyone always work 60 to 80 hour weeks. Not okay. There is no excuse for this other than management being completely inept. It affects work quality and general quality of life. Don't do it.
   
2. Bad Information: This is what happened to Team Bondai. They were constantly told (and believed) that the game was almost finished. In such cases, many team members, believing the end is in sight, will do self-imposed crunch, which eventually will lead to de-facto crunch. In addition, this is where a death march starts, and tends to never end.
   
3. Bad Planning: This is the most common cause of crunch, and management gets rightly blamed for it all the time. But there are two types of bad planning:
   
   1. Bad Low Level Planning: Even the best management team can make a mistake, because they can only make decisions based on what they're told by the people below them. If what they're told is wrong, their planning will be wrong. So, if you're at the low level, don't fudge your numbers. Tell management the truth so they can make the right decisions.
      
   2. Bad High Level Planning: Even with all the right information, managers, especially bad managers, will plan incorrectly.
      
4. Shifting Requirements: Even with good planning and good information, games are a highly creative and highly iterative product, and when something's not working or isn't fun, it has to be changed. Unlike other industries where you can work "to spec", the "spec" of a game is frequently fluid, which makes planning for the long term almost impossible. This, unlike the other causes, isn't anyone's fault, per-se, since a team can come to the mutual decision that something needs to be reworked, but it's something that frequently gets ignored while doing planning
   
5. Deadline: When all is said and done, this is the real cause of crunch. When a deadline is involved any amount of error in planning or requirements hits a head when the deadline looms. Some studios are in a position where the deadline can be pushed, but not all, especially when dealing with a multi-game publisher that has to allocate resources at various times (QA, marketing, operations, etc) to make sure the game launches without a hitch, so they can move on to the next one. In these situations, moving the deadline is impossible.
   

The Myths: Crunch is a Given and Crunch is Always Avoidable

Both of the above statements are wrong. If the causes of crunch are listed above, avoiding crunch is as simple as avoiding those mistakes. Don't make crunch company policy, always give your employees all the information, plan well at all levels, have a solid design document, and don't have a hard deadline. Simple. But, if any of those slip, you have to make a hard decision: cut features, cut polish, or crunch.

If your company has a lot of experience or a lot of leeway, you can always avoid crunch. However, if you're a start-up working for a publisher, on a hard launch deadline, under contract, and you want to make the best game possible, chances are someone, and it's not necessarily management, is going to make a mistake, and that mistake is going to cause some amount of crunch.

Was it avoidable? Yes, probably at some point it was, and yes someone made a mistake somewhere. Crunch is always a failure somewhere down the line. But at some point in the project, it will become obvious that some amount of crunch will be needed. The next step is to mitigate it.

Mitigating Crunch

Crunch does not always mean Quality of Life or Quality of Work has to suffer, it's just that in most circumstances it does, because of the way it's implemented. I have gone through two sets of crunch, both mandatory, one long, one mitigated, and both sucked. But there are a few things that I found helped me, and I think would help others, get through any period of crunch

1. Provide All the Information: Inform everyone how much more you expect them to work, what needs to be done, and where the deadline is. If ANY of this information changes, make sure the team knows. I have been in and heard about crunch situations where information about deadlines or cut features was withheld, and this only makes people angry
2. Allow people to choose their extra hours: People are different and have different home lives. Some people work better in the morning, others at night, others over the weekend. Do not blanket ask everyone to stay late. Instead, let them choose when to work and they will choose times that will not only make them most productive, but will interfere with their lives the least.
3. Allow people to work from home: Set up your infrastructure to let people work from home. Some people can be just as productive at home as they can be in the office, and this removes the stress of a long commute and improves QoL since they can be around for their loved ones.
4. Allow people during longer periods of crunch to take some time off: This is huge for me, and I've never been anywhere that does this. If your crunch is going to last a long time, or looks like it will need to be lengthened, give people a day off. Say "Okay, everyone can take a day off free sometime in the next two weeks." Giving them a time frame allows them to plan for things like short trips, or time with their family. They'll come back rested, happier and more productive.
5. Make sure they have and know the reward: In a start-up, the reward is in the stock, but in larger companies, make sure people know they will be compensated of extra hours, and then compensate them.

Hopefully this makes sense. I don't like crunch either, and it is avoidable for companies that have really experienced teams and a certain amount of give, but some companies do not and will not have the luxury to avoid crunch entirely. Obviously, great companies will always avoid crunch, usually because they've learned from their mistakes. But realize that a good company may crunch, but will always find a way to mitigate it when and if it happens, and will learn from their mistakes on their way to becoming a great company.

I was at Northeastern last night talking with programming students about getting into the game industry.  As with most students, they were very curious about what, specifically, they should learn in order to make it easier for them to get into the game industry, and where they should learn it.  I gave some standard advice (which I'll probably write up a blog post about at some point, maybe?) but I realized that one way to find really good places to learn is through the blogrolls and recommendations of talented developers.

And then I realized I haven't updated my blogroll, probably in a year at least.  So, I've updated it this morning, fixing links to places like t-machine, removing some long dead blogs, and making a few additions.  Some recent additions:

• Games From Within: Noel is now an iPhone developer, but his development methodologies are really what I look at in his blog.  He updates fairly frequently and his articles are almost always good.
• Bitsquid: I'm not sure really what Bitsquid is working on, but their programming posts are fantastic.
• Insomniac R&D: Awesome posts from Insomniac devs
• #AltDevBlogADay: There is so much awesome in this (basically aggregate) feed that I often I can't keep up with it.
• Brett's Footnotes: Brett is a lead programmer at Bethesda Game Studios, but wasn't there when I was.  He's awesome and his posts are always interesting.

I will do my best to keep my blogroll up to date from now on, though I can't make any promises.

I've posted my talk on getting the most out of data driven programming that I gave at Game Forum Germany a few weeks ago on my website. The talk recommends reflection as solution to a lot of the potential pitfalls of needing a single editable location for fields and properties and, interestingly, fellow Toolsmith Geoff Evans has an article in Game Developer this month about that same topic, and specifically how they solved it in Helium. So if you're thinking about using reflection to do automatic UI generation and serialization, definitely give that article a read.

I've been told that there will be video of the talk at some point, and I'll let you know when that's available.

I've been talking on twitter about working on some stuff involving 2D soft shadows. I was going to write a long blog post about how I got 2D soft shadows working in XNA using the GameDev article in combination with Catalin's implementation and Christian's implementation in D. However, trying to write the post, I wasn't sure if I could make it any clearer, as I'm not sure if I understand it completely myself.

So, instead, I'm just going to post the XNA port I made almost directly from Christian's D implementation without much comment. I will say that, instead of using the images that both Catalin and Christian are using, I've move most of that into shaders. This isn't necessary, but it's in preparation for some other stuff I want to do later with this project.

So if you're interested, my XNA implementation is here. Enjoy!

It's been a while since I posted, but a lot's been happening. If you've visited the site, you probably noticed a theme change, but more importantly, you may have noticed I am no longer at Orbus Gameworks.

There are many reasons I decided to leave Orbus, and maybe some day I'll write a long blog post on the subject, but the short story is that I decided it was time for me to get back into creating games full time. Although Orbus focused on game middleware, the technology we were working on wasn't really game development, and I felt like I was getting more and more behind in the skills and systems that are essential to being a programmer in the game industry.

So I have moved on to another start-up, but this time a game studio: Fire Hose Games.

Working at Fire Hose has been an adjustment, though not a bad one. Because of the size of the team and the schedule, I've been forced to think and program very differently from what I'm used to. It's pushing me out of my comfort zone, which is always a good thing, if only to see different ways things can be accomplished, even if they don't jive with the way you work as a programmer.

Generally, I'm really looking forward to my work with Fire Hose, I'm excited about our product, and I'm looking forward to having shipped another game.

I've been spending a bunch of time reading about continuations and microthreading recently, just trying to wrap my head around them and find areas where they might make sense, specifically in C# using the ISO CLI systems. Of course, the example that most people give is AI, like so the one provided by this article:

 
IEnumerable Patrol ()
{
     while (alive){
          if (CanSeeTarget ()) {
               yield return Attack ();
          } else if (InReloadStation){
               Signal signal = AnimateReload ();
               yield return signal;
          } else {
               MoveTowardsNextWayPoint ();
               yield return TimeSpan.FromSeconds (1);
          };
     }
     yield break;
}
 

Now, implementation details aside (I'm not a fan of the authors implementation of the scheduler), you can then have a scheduler for all of your microthreads that will run each method until it terminates. Run one (or more) of these schedulers in a background thread and everything's awesome right?

Well, I'm not so sure. I have questions about this myself. For example, I would think that this pattern is more useful in cases where you have a lot of things that need to be done in sequence, but potentially in other threads or for multiple frames. For example the "AnimateReload" state from the example. Well, what if that state looked more like this?

 
       if (NeedsReload)
       {
            Signal coverFound = BeginFindCover();
            yield return coverFound;
            if (CoverPoint != null)
            {
                 Signal atCover = MoveToCover(CoverPoint);
                 yield return atCover;
            }
            Signal reloadAnimation = AnimateReload();
            yield return reloadAnimation;
       }
 

This is a more common use case in my mind. Have a procedure that takes many frames of execution and has portions that can be delegated to background tasks (such as path finding, which might take a few frames) and have them written linearly so that it's easier to read.

Here's the problem. Because we've now put the "Alive" check farther and farther away from the actual call, we've ended up in a situation where, at any time, the AI can die, and thus its microthread will need to die with it. To combat this, we'd have to check if the AI is actually alive after every yield, creating this code:

 
     if (NeedsReload)
     {
          Signal coverFound = BeginFindCover();
          yield return coverFound;
          if (!Alive)
               yield break;
          if (CoverPoint != null)
          {
              Signal atCover = MoveToCover(CoverPoint);
              yield return atCover;
              if (!Alive)
                   yield break;
          }
 
          Signal reloadAnimation = AnimateReload();
          yield return reloadAnimation;
      }
 

To say nothing of if the state of the AI was forcibly changed for whatever reason (say, for example, if a Team AI Manager has told him to start running away, or regroup, etc). The only way I can think to combat this is through a special AIYield function which would handle most of that, but even then, I find it hard to think of a way to tell the system it needs to break from the current update thread and start a new one. Reschedule?

What do others think? Am I over thinking this? Am I thinking about this the wrong way? Am I using the wrong tool?

So Xbox Live Community Games went live a few days ago, though I was away at MIGS when it launched, so I missed it. However, already there's a lot of stuff up there and a lot of it looks fairly cool (although, certainly, there's a ton of stuff that's just terrible). However, there's one thing that I want to say to people that are thinking about making community games: really THINK about your trial time. Since you have a set amount of time to play with (unlike non-community games which can kind of decide on their own), you need to use that time wisely. Remember: people are not going to buy your game if your trial isn't good, and so far, I've yet to download a game with a good trial. Some tips:

• Your tutorial should take less time than the trial allowance. Let me actually play your game and I might be more interested in buying it. If you have more rules than you can express in a limited amount of time, think about pushing more advanced tips to farther into the game.
• If your beginning isn't that interesting, don't start me at the beginning of the game. Get me to the most interesting part quickly. Of course, that may be difficult, so, instead, just make your beginning really interesting.
• Honestly, just try to sell me on *something* that makes your game unique.

I'm still excited about the possibilities of Community Arcade, but I'm really not as interested in weeding though all of the crap.

So I picked up a copy of Mirror’s Edge the other day, and I’ll have a full post about how much I enjoyed that game some time in the future (probably after I get back from MIGS), but right now I just want to talk about something that really bothers me about modern applications:Auto Updaters.

Honestly, I have no problem with auto updaters as a concept. If my application needs updating, by all means, update it when it needs updating.My problem is with the auto updaters that run as a background process, constantly, without my approval (usually), and really without any need. Here’s the list of auto updaters currently running on my system as background processes:

• Google Update x2 at 2.6 megs each
• The HP scheduler x2 at 956k each
• The Install Shield Updater x2 at 1.1 meg each
• Antivirus update at 1.5 meg

In addition, if I hadn’t disabled it, I could have the Apple auto update service running as well.Who knows how much memory that would take up (and for each user I’m sure).

In my opinion, there are only two systems that are allowed to constantly check for updates: your operating system, and stay resident protection programs like virus scanners and anti-spyware. All other programs should only EVER query for an update when you run them. All lot of applications I have do this, though even they’re not perfect. Here’s how I see a good auto update system working.

• Never have a stay resident program for updating. If that requires that you compile the same system into multiple programs to check for updates, so be it.
• Never have an auto updater suggest that I download new software. I’m looking at you Apple. I don’t want Safari… got it?
• Check for updates only when the application boots.
• Inform me when an upgrade is available, and ask for on of these three options: “Always Upgrade Automatically”, “Upgrade Now”, “Never Upgrade”. If I select “Always Upgrade Automatically”, all upgrades should be silent. Unless you change something visually, I shouldn’t even know an upgrade occurred.
• If you absolutely must have a stay resident program checking for updates, it MUST understand multiple users. Thankfully, Avast! actually did this right.

In actuality, the best solution here is to have a central system that checks for programs that need to be updated, and where to get information about that. This is where package Linux package management is actually far and away better than most everything else out there. That said, although it’s ridiculously simple to use, I don’t see it being ridiculously simple to develop for, especially if you’re developing an application outside of the bounds of the central repositories. At least there are no upgrade daemons in Linux (that I know of anyway…).

Following in Darius's footsteps, I've added my blog to the Facebook Blog Network, and you can click here to join. In addition, my blog should now have a very nice Blog Network widget on the side that will allow you to join my blog network, so, that's great!

In addition, I'm always happy to have new friends on Facebook, so please feel free to add me if you haven't. That said, please mention in your invite that you're a blog reader. I do occasionally get friend invites from people I do not know, and I do not generally accept them. If you have already sent me an invite and I have not responded, this is probably the reason. Just resend and let me know how you found me.

So I had a conversation last night with my good friend Steve about my decision to start using Mercurial. Talking to him, I realized I hadn't really posted much on what I think about distributed version control systems (DVCS), so the switch to Mercurial may have taken many people off guard. So, I wanted to spend a post (maybe two) talking about what I see as the advantages of DVCS over a traditional "central server" (VCS) mentality.

The Leap of Faith

Like many, about the idea of version control without a central server scares me. I don't trust my own machine, or my developer's machines, to be in any way fault tolerant. My server, on the other hand, has a RAID 5 controller in it, and is backed up off-site weekly (though if I were checking in more I'd probably back up nightly). Having that central server keep track of my changes feels safer, so I shied away from DVCS, opting instead for centralized version control with Subversion with user branches (more on that here). Even with tools like svnmerge though, Subversion utterly fails at merge tracking, especially from multiple branches bidirectionally. It was just never scaled to do that. Since then, I've watched several videos on the design of distributed version control (two on Git (here and here), one on Mercurial) and I made the leap of faith to attempt to see what DVCS is like.

The Centralized Model, Distributed

Now, distributed version control is amazing for open source projects, but what about working in a company where team communication and process is key? Where coordination is a requirement? Well, the nice thing about distributed version control is that it can, if you need it to, support a centralized model. Even in open source projects, you have what you call an "upstream" server, which is what everyone consults for the latest and greatest "official" changes to a product. You could, if you wanted to, push every commit you made to the upstream server, and pull every so often to make sure you have the latest version. In that case it would be exactly like using centralized version control (except it takes a more hard drive space, as you're holding a repository on your hard drive, not just the source code). Sure, you have to train people to commit, then push, but once they understand that, there's no issue. In addition, your centralized version of your tree can have pre/post-commit hooks just like any other version control system, allowing you to create check-in gauntlets should the need arise. That said, I'm sure other source control providers make it easier (the new Source Safe and Perforce I believe have GUIs for create check-in gauntlets, but I could be wrong). Still, with an easily extended system (like Mercurial), these additions wouldn't be hard.

Encouraging Experimentation and Checkpoints

So in this case, why use a DVCS if it's just going to work like centralized version control. Well, even in this situation, you now have more options available to you than you did before. First is the ability to checkpoint whenever you want, without affecting other developers. The key reason most people give for using version control is the Chinese proverb "The palest ink is better than the best memory." So what not increase your memory by encouraging your developers to commit as often as possible? And if they can do this even when their build is broken or when they're not completely finished with something, why can't we allow them to do this? In a centralized model, this is almost impossible (though packing apparently solves this partially). Then, when a developer is done, their commit can have all of the changes they just made, with a history of what they did. Of course, it doesn't have to (rebasing is always an option) but having it there can help you see potential problems or flaws in thought process.

In addition, distributed version control can encourage experimentation in branches. The problem I have had with branches in the past is that they are a pain in the ass to manage on a centralized server. Even Subversion, which makes branching easy, doesn't manage merging well at all, especially bi-directional merging. I've even had problems with Perforce merging in the past. Regardless of that, though, working off of branches and creating branches is never easy. Not as easy as it is in distributed version control environments anyway. In Mercurial, branches are just clones of the current repository. You can work in them, share them with others, and delete them without the central server being involved. This may sound bad, but it does encourage developer experimentation and sharing. If I can branch simply off of whatever my current state is, experiment a bit, show that change to other developers and get feedback, then merge back into my main development branch easily, that opens up a lot of opportunities for small team experimental work.

Here's an example of where I could have used this in the past. One of my former companies did not do and did not encourage unit testing, and I could not convince my coworkers (or the management) to let me try it in some of our newer libraries. I decided, though, to do unit testing on any new modules I wrote anyway. I made a copy of the library into a directory that had a unit test environment set up, worked on the library (writing new tests, making them pass, etc), then copied my changes back over when I was done, and committed them to the central server. The unit test folder, though, was completely unversioned. If I'd had a DVCS, I could have cloned the library, done my work, then pulled changes back, leaving the unit tests versioned in their own folder. In addition, any other developer that was also working on those libraries could have easily pulled the unit tests from me, thus spreading a new practice at the company. This may be an edge case, but it is an example where simple branching and merging would have encouraged experimentation between developers.

Shelving, Packing, Branching

This gets into the second option you have with distributed version control. DVCS makes it easy for developers to share uncommitted changes with each other. Other source control providers give you the option through "shelving" or "packing", but I can't see it being as easy as it is with distributed version control. I can pull changes from anyone, on any branch, into a cloned version of my own repository to test things out without talking to a central server. Those changes don't need to be based off of the same trunk and can easily be merged by whichever developer at whatever point, then pushed to the upstream server, all while retaining who made the original changes. This has to be experimented with to really see the full benefit, and honestly I haven't done it enough, but I can see where it would be useful. Unlimitted simple branching with the ability to push and pull changes from any developer coupled with a strong revision history just sounds nice to me.

Fault Tolerance

The last thing I want to talk about is when things go wrong. We try to avoid it, but every so often, our central server goes down. Sure, we have backups off site, maybe we have a passive failover server, but if we don't, our central server going down is very problematic. I have had this happened to me, and at fairly large companies that can afford lots of nice servers. If this happens in a traditional VCS, development (basically) stops for the day. Not so much in distributed environments. Since I can commit to my local repository all I want and share with everyone else without the benefit of a central server, a dead upstream doesn't affect me at all. This, in my mind, is pretty awesome.

More…

I will say that I'm using a DVCS in a very small team environment, but looking at it, I believe it could scale very easily. I also think there are lots of interesting ways to use DVCS, especially in agile environments where small teams may need to communicate changes without affecting central efforts. I really want to go into these, but this is already too long, so maybe I'll talk about them tomorrow. Until then, let me know what you think.