GDC 2013 – Programmers Panel

  By Jeff | April 2, 2013 - 5:19 pm | , , ,
2 Comments

GDC is over, and it was fairly awesome. I'll have a quick "thoughts on GDC" post later, but I wanted to get the information for the GCS panel I participated in up as soon as possible.

GDC did not record this panel this year, but thankfully, we have a fairly good unofficial recording of the panel thanks to Jeremiah Ingham. (sssshhh, don't tell GDC)

You can also just download the file, if you'd like.

The only real resources we recommended this year were at the end, where I asked each of the panelists recommended a book, and here they are:

If you have other questions of the panelists, please feel free to ping us on twitter. We're all pretty approachable (especially if we can respond in 140 characters or less).

Using Waf to build an iOS Universal Framework

  By Jeff | December 10, 2012 - 11:34 am |
1 Comment

I've used a lot of build systems, and so far my favorite (when I'm "forced" to use something outside of Visual Studio) is probably waf.  Besides being based on a proper scripting language, waf doesn't feel like it's doing a whole lot of "wicked voodoo magic" behind the scenes (like scons sometimes does).  It's also fairly easy to extend, and we used it to make and deploy all of our Nacl executables.

Today, I figured out a way to make waf do some interesting things concerning iOS and creating libraries / frameworks.

iOS Universal Frameworks require that you create libraries for 2 versions of the simulator (i386 and x86_64) as well as 2 versions of ARM (armv6 and armv7) if you want to run on all of those platforms.  Each of these libraries is then linked together using a tool called lipo.  Waf can easily build one library, but building multiple, especially as part of one build step, can be tricky.  What you do is create a separate environment for each target, then link them together in a single build step. Here's how it goes.

First, waf doesn't understand how to handle .m / .mm files and pass them on to clang properly, so we want to teach it how to do that. The simplest way to do that is like so:

@TaskGen.extension('.mm')
def mm_hook(self, node):
    "Bind the c++ file extensions to the creation of a :py:class:`waflib.Tools.cxx.cxx` instance"
    return self.create_compiled_task('mm', node)

class mm(Task.Task):
    "Compile MM files into object files"
    run_str = '${CXX} ${ARCH_ST:ARCH} ${MMFLAGS} ${FRAMEWORKPATH_ST:FRAMEWORKPATH} ${CPPPATH_ST:INCPATHS} ${DEFINES_ST:DEFINES} ${CXX_SRC_F}${SRC} ${CXX_TGT_F}${TGT}'
    vars    = ['CXXDEPS'] # unused variable to depend on, just in case
    ext_in  = ['.h'] # set the build order easily by using ext_out=['.h']

This will use whatever the CXX compiler is set to (which we'll set to clang) using MMFLAGS over CXXFLAGS or CCFLAGS. Everything else will stay the same. This can be added anywhere in the file, but I tend to add it under configure, before build.

Next, we set up configure to create an ios base environment, off of which we're going to base other targets.  Because I use this script to build for multiple environments on multiple operating systems, I check to make sure that this is being performed on Mac OS before creating the environment.

IOS_MIN_VER = '6.0'
IOS_SDK_LOC = '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneSimulator6.0.sdk'
IOS_SDK_SIM_LOC = '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator6.0.sdk'

def configure(conf):
    # ... Any other configure steps
    
    # Make sure we're running on a Mac
    if sys.platform.startswith('darwin'):
        conf.setenv('ios')
        # Set the C++ plugin here to use clang over gcc  g++
        conf.env.CXX = 'clang'
        conf.env.CC = 'clang'
        conf.env.LIPO = 'lipo'
        conf.load('gxx')
        conf.load('gcc')
        # set any CFLAGS, CXXFLAGS, MMFLAGS and DEFINES for ios in general

Next, we create an environment for each target (debug or release) and each platform (i386, x86_64, armv6 and armv7). If you're not targeting an x86_64 simulator or an armv6 device, you don't need to build these, but since post first build this is fairly quick, I just build all of them.

        for arch, tgt in itertools.product(['i386', 'x86_64', 'armv6', 'armv7'], ['dbg','rel']):
            conf.setenv('ios')
            newEnv = conf.env.derive()
            newEnv.detach()
            conf.setenv('ios_' + arch + '_' + tgt, newEnv)
            #... set any target specific flags 
            # These flags set up the sysroot for ios based on simulator or device
            if arch.startswith('arm'):
                addflags = [ '-mios-version-min=' + IOS_MIN_VER, '-isysroot' + IOS_SDK_LOC ]
            else:
                addflags = [ '-mios-version-min=' + IOS_MIN_VER, '-isysroot' + IOS_SDK_LOC ]
            conf.env.CCFLAGS += addflags
            conf.env.CXXFLAGS += addflags
            conf.env.MMFLAGS += addflags    
            conf.env.ARCH = arch

Now we have to create commands that use each of these environments. This is exactly the same as using multiple variants / platforms, if you've seen that example. The other thing we're going to do is make a LIPO command for each target, so we don't have to re-specify how to LIPO our libs together (this is useful if you've got multiple like I do).

def init(ctx):
    # any other architectures / platforms

    for arch in ['i386', 'x86_64', 'armv6', 'armv7']:
        for tgt in ['dbg', 'rel', 'retail']:
            for inheritingClass in (BuildContext, CleanContext, InstallContext, UninstallContext):
                name = inheritingClass.__name__.replace('Context','').lower()
                class tempClass(inheritingClass):
                    # this is used in the next step to actually call the step
                    cmd = name + '_ios_' + arch + '_' + tgt
                    # This must match your environment name, and sets the output directory
                    variant = 'ios_' + arch + '_' + tgt  
                    # These are for testing in your build script, if you need them
                    architecture = arch
                    target = tgt
                    platform = 'ios'

      for tgt in ['dbg', 'rel', 'retail']:
          class tempClass(BuildContext):
              cmd = 'lipo_ios_' + tgt
              fun = 'lipo_ios'
              variant = 'ios'
              target = tgt
              platform = 'ios'

Last, we create a build_ios command to build all the variants for a target (I default to release), and a command called lipo_ios to LIPO each one together. Here's how it goes.

def _lipo_lib(ctx, lib_name):
    ctx(rule='lipo -create ${SRC} -output ${TGT}',
        shell = True,
        target = 'lib%s-%s.a' % (lib_name, ctx.target),
        source = ['build/ios_%s_%s/lib%s.a' % (x, ctx.target, lib_name) for x in IOS_ARCHS]
    )

def build_ios(ctx):
    for x in ['i386', 'x86_64', 'armv6', 'armv7']:
        waflib.Options.commands.append('build_ios_' + x + '_rel')

    waflib.Options.commands.append('lipo_ios_rel')

def lipo_ios(ctx):
    """ This creates universal libraries in ios_target made from the architecture 
        builds of each lib. """
    _lipo_lib(ctx, 'A')
    _lipo_lib(ctx, 'B')
    _lipo_lib(ctx, 'C')

So this is not at all straight forward, and took a lot of trial and error, so hopefully this helps some people.

The upshot here is that waf's dependency detection in terms of libraries is way better than XCode's, so you can use multiple dependent libraries together and it will only ever rebuild exactly what's necessary. However, the preprocessor that determines include dependency only supports C / C++, not Objective-C's #import directive. So I would avoid doing this if you're working a lot with Objective-C dependencies, at least until waf gets real objective-C support.

Also, you could use these same principles to create build_ios_simulator and build_ios_device targets instead, without too much effort, but I like having all targets build into the universal libs, and going from there.

More Speaking Engagements: UVA, VCU and AltDev

  By Jeff | October 22, 2012 - 9:29 pm |
Leave a comment

I've got a bunch more speaking engagements lined up!

First up, the weekend of November 10th I'll be speaking at the Alt Dev Student Summit, with a talk titled OOP Is the Root of All Evil, of course riffing a bit off of the phrase "pre-mature optimization is the root of all evil." The idea behind this talk is that pure OOP, especially OOP that doesn't think about how data is going to be used or the performance tradeoffs incurred, is pre-mature pessimization, and that frequently the argument that OOP makes more understandable, more reusable, or more maintainable code is frequently an illusion. I'll also be looking at how certain OOP patterns actually promote bad coding practices, but are used because they're considered OK from a pattern perspective.

Earlier, on October 30th, I'll be speaking at UVA to the Student Game Developers club, doing a dry run of my talk to the AltDev Student Summit.

Lastly, on November 6th, I'll be speaking at VCU to their chapter of SIGGRAPH. This will be the same talk I gave at JMU earlier this year called Getting Started Making Games. I am still planning on recording a shortened version of this talk for posting online, and I still need to post the slides. This will hopefully happen shortly.

More on Open Offices

  By Jeff | October 19, 2012 - 11:11 am |
4 Comments

After my talk with Mike on Leadership, we got into a discussion about open vs. closed office spaces. Mike is in favor of the open office while I (obviously) am not. Mike's assertion (which I actually agree with) is that you can create an interuptionless culture in an open office space, and still get the benefits of an open space that removes the psychological barriers to communication. In addition, Mike asserts that, if a person knows they're going to have to have lots of uninterrupted time, they can always work from home (not necessarily true, but I'll give him the benefit here).

These are two just fundamentally different approaches: one that favors open communication, and creative collaboration, versus one that favors individual productivity, potentially at the expense of team productivity. However I believe that closed offices are better for two reasons.

First, I believe flow and interruptionless work are important to personal productivity and job satisfaction. This doesn't mean that I believe that everyone should always go uninterrupted all the time, but that a culture that minimizes these by default is important.

Second, because I believe it is easier to foster an open culture in a closed office than to foster a closed culture in an open office. What this means is that I think that you can work very hard to foster a culture of few interruptions and respecting people's time and space in an open office, but that when it comes down to it, the open office itself works against this. In contrast, while a closed office makes open culture harder, but does not necessarily work against an open culture, especially if said office provides open spaces as well.

With all of this said, Mike also pointed out one very key point:

Most likely, neither Mike nor I are ever going to be in a position to actually be able to plan out an office.

Speaking at JMU: Getting Started Making Games

  By Jeff | September 25, 2012 - 3:42 pm |
2 Comments

Next Wednesday (October 3rd) I'm giving a talk at James Madison University to their ACM. Although they've labeled it a "Tech Talk," my actual talk is going to be much more general, titled Getting Started Making Games. It's inspired by this talk and website, but moves beyond what tools you can use and resources are available to give some real practical advice on how to make your first games, and what constitutes a reasonable first game. Then I'll tackle where you can go from there, and discuss techniques for failing early, and avoiding productivity black holes.

According to the ACM people, the talk is open to non-students so if you're in the area, drop by HHS 1302 at about 7:30 and say hi!

Open Environments in a Closed Office

  By Jeff | September 19, 2012 - 3:50 pm | , , ,
Leave a comment

I've been railing against open offices and office meetings (especially micro meetings) on my blog for the last few weeks, and I have one more thing to add to this series. I'd like to talk about creativity in a "meetingless" / "interruptionless" office. Generally, people that believe in open offices believe in the creativity of the micro-meeting, and they believe in the collaborative power of spontaneous large group meetings (hence the open office). There is some truth to the fact that a spontaneous large group meeting, or hallway discussions, can lead to some very interesting and creative decisions. But why? And how can you create a creative environment within a constraint of few to no meetings?

First, let me say that an interruptionless office does not mean that there will be no hallway discussions, discussions over coffee, lunch, donuts, etc. In fact, I've found that even in semi-closed offices, these happen more often, because people feel more at liberty to just walk around. There's no chance of them disturbing anyone directly, so when they feel the need to be creative, they'll wander the halls, go to the cafeteria, or sit in a lounge, and just talk.

Second, as I talked about in the "Meetingless" post, a Meetingless office doesn't have no meetings, it just creates a culture of most group meetings being optional, minimizes interruptions caused by meetings, creates times that are guaranteed to have no meetings, and creates alternatives to split-second, micro meetings and micro decisions that I feel are an overall team detriment.

In general, there are three things that I feel open offices achieve automatically, that a closed office needs to work on:

  1. Removing psychological barriers to communication. As I've stated before, I admit that walls create psychological barriers to communication. To remove these you need to create an atmosphere that people are approachable in and outside of their office. Team building (the next point) helps with this, but managers, and leaders have to take extra steps to make sure that they are approachable. A few suggestions are to implement skip levels, hold frequent one-on-ones (which you should be doing anyway) and to meet outside your office. Coffee shops and established open areas in the office are great for this.
  2. Foster team building.  This doesn't mean everyone has to do trust falls, but it does mean you need to make efforts to bring the team together. This can be anything from catering lunch (which is what Fog Creek does), hosting in office Happy Hours, office Board Game Night (a personal favorite), or other office parties. Just make sure people mingle.
  3. Create open spaces for work and discussion.  This may sound like it goes against everything I've said, but I do still believe in having portions of the office be open spaces. Sometimes, you want to get out of your office, sit on a couch, and relax. Maybe go with a few other team mates to look over a problem. Maybe discuss something in a smaller open environment. Make these comfortable spaces and give employees the means to work in them (laptops or otherwise portable workstations, whiteboards, lots of power and wireless) and people will.  Even more so if they know they won't disturb others by doing so.

Generally, I feel like, cost aside, closed offices create happier, more productive employees, and that the benefits of an open office can be simulated more readily than those of a closed office.

The Interruption-less Office

  By Jeff | September 10, 2012 - 4:07 pm | , , ,
Leave a comment

In a previous post, I talked a bit about why I'm not a fan of open offices, and one of the main reasons is because they are too prone to various types of interruptions that actually keep creative people from being "in the zone." In addition, the types of "meetings" that happen in open offices can actually create a culture where split second decisions and decisions between small groups are the "norm." While this is fine in smaller studios, it can become an issue as teams get larger.

First, let's look at the common types of meetings that are encouraged in an open office, or one that encourages meetings, and see how we can remove most of them.

  • Quick Questions – You know these meetings because they're always prefaced with the person interrupting you saying, "Hey, quick question…" Frequently these are anything but quick, and almost always it is not imperative that they be answered immediately. However, in an open office, the person will frequently ask you directly, which means you need to give an immediate response. These meeting can be handled in private chat rooms where the question can be either answered by someone else or answered at a more opportune moment.
  • Class A Bugs – These are bugs are actively blocking, and need to be fixed ASAP (and I'm talking actually ASAP, in that people are actively losing work). These things happen. My recommendations for this are, first, make sure that all bugs are documented and tracked. Frequently with class A bugs like this, people just report it to the person they "think" is responsible verbally. If they're wrong, you can get two people working on the same bug at the same time, which can be hugely problematic. Once the bug is documented and tracked then you can interrupt the responsible coder (or producer responsible for assigning it).
  • Micro Meetings – Micro meetings are when (usually) two people decide to discuss an issue quickly to try to come up with a solution. These actually have two forms:
    • Idea bouncing – One person has an idea they want to flesh out, or run past someone, and just wants to talk it out. Frequently, this doesn't actually require the other person, just something to talk to. (I know a designer that keeps a stuffed monkey on his desk for exactly this purpose). I'm actually okay with these types of meetings provided they do not actually interrupt someone's work
    • This Isn't Working – These meetings come about because someone feels something "isn't working" and needs to be reworked. In my experience, these meetings have several drawbacks, besides the interruptions they frequently cause. First is that they rarely involve all of the people that need to be part of the decision making process, which means that  certain people are left out of the loop and surprised by changes they didn't know were going to happen. Second, these meetings frequently cause or require snap decisions without a real good amount of thought, which can be problematic. Lastly, decisions made at these meetings can contradict other decisions that were already made and documented. Now not only do you have a change that isn't documented based on a decision process that isn't documented, but it contradicts documentation that does exist. Again, these are probably better handled in group chat rooms, so that all people invested can see the discussion, the decision, and update the documentation accordingly.
  • Micro Meeting Turned Team Creative Session – This is really what people that want open offices cite as a benefit of open offices. Start with a "This Isn't Working" micro-meeting which becomes a full fledge, whole room creative discussion about how various systems and designs should work, interrupting everyone. The results of these meetings can be great creatively, but because they're always impromptu and records are scant, decisions made can affect entire teams without their knowledge or input. These meetings have the same all the same drawbacks as micro meetings, except with the added benefit of you keep several members of your team from working (including those that aren't in the micro meeting) for several hours. These should be avoided at all costs
  • Actual Meetings – Then there are actual meetings, which I'm not against, but only in certain forms, which I'll talk about in a bit.

In general, you should be always be striving to move to an "interruption-less" and "meeting-less" office. But, there are a few types of meetings that I'm okay with, with one caveat: you should strive to have days with no meetings at all, and as many of them in a row as possible. In addition, if you can schedule meetings to overlap with other times that people are taking breaks (lunch, for example) this means you're not interrupting actual work flow. This gives people lots of time to get into the groove not only on a single day, but multiple consecutive days.

The type of meetings I'm generally okay with:

  • Review Meetings – We have to review our work. Artists need art review, coders need code review, designers need design review and the whole game needs to be reviewed as a whole. It's important, and frequently it's important to do this with groups rather than one on one (depending on the type of review). If you can get these to be outside of meetings with tools (like Review Board) that's great.
  • Creative Meetings – Large group creative meetings can be great, but they have their own place and time, and their own rules. This is the subject of another post.
  • Planning Meetings – Generally I'm okay with planning meetings, provided they're shorter. This is for sure a good candidate for a lunch meeting.

Any other types of meetings I'm missing? Do any of these meetings work better in an open office? Am I right about most of these meetings being avoidable?

On Air Hangout with Mike Acton

  By Jeff | September 7, 2012 - 5:05 pm | ,
3 Comments

In case you haven't heard, this coming Monday, September 10th, I'm doing an On Air Hangout with Mike Acton, talking to him about his thoughts on leadership. If you can't tell from my recent blog posts, leadership, and how to be a good leader, are new interests of mine. So, I'm super excited to talk to Mike about his role not only as a lead at Insomniac, but also about his role as an industry leader, and get his thoughts on how to create better leaders in the game industry.

On Air Hangouts are cool because they are recorded and posted to You Tube immediately after they're done, and as soon as that happens, I'll post the link to the recorded hangout here.

Update: If you missed it, check out the results of the hangout on You Tube, and let me know what you think!

Thoughts on Open Offices

  By Jeff | September 4, 2012 - 1:36 pm | ,
4 Comments

There's been a lot of talk on twitter lately (some of it pushed by me) about open offices, interruptions at work, and generally how to avoid being multitasked to the point of lost productivity (frequently the cause of being interrupted and asked to fix too many things at once).

From an individual standpoint, there are things you can do to make sure you stay on task, but that's the subject of another post. For now, I want to focus on the external factors of interruption, and the negative impacts there. This is probably the first of several posts on the factors that lead to interruptions at the office.

First, let me talk about open offices. Many of us, especially at game companies, work in open offices. No one really knows why, except that it is less expensive to build cubicles and set up tables than it is to build actual offices. The given reason for using open offices is that they foster team collaboration, information sharing, and that they work as a way to bring the team together. By having an open office, people can (essentially) eavesdrop on each other's conversations, and offer their input to creative or technical decisions, should they want to. But the downside of an open office is that everyone can hear your conversations, whether they want to or not.

"Open offices aren't a problem" you say. "If you don't want to be disturbed, just put on your headphones. Have a standing rule that you can't be disturbed if you have headphones on, and your music will drown out the office noise." But what if you're like me and don't like the feel of headphones (they push my glasses into the side of my head). In addition, I can't think clearly on hard problems with music playing. I can only listen to music when I have a clear direction, or when I'm doing a less thought intensive task (I'm sure many people are like me in that respect). If you have the rule of "you can't be disturbed" and you're actively encouraging people to escape from office chatter, how are you say that the open office actually encourages creativity and collaboration? Instead it forces interruption unless people explicitly block it out.

Interestingly, I've not been able to find any studies on open offices fostering collaboration or productivity, and neither could the authors of Peopleware, though they could find numerous studies showing how interruptions and open offices hurt individual productivity. Though not backed by science, listen to Jason Fried's TED talk about open offices, and why "Work doesn't happen at work." Anecdotally, I'm sure you've experienced exactly what he's talking about. Fog Creek takes this to the extreme. Everyone has an office and can't be interrupted when the door is closed. It also has a no meeting culture, which I may also write a blog post on. People communicate through private chat. Bugs and support requests must go through their bug tracker.

The problem is, I think an open floor plan does encourage communication in some ways. Not through eavesdropping, but by removing psychological the barriers between you and other people. I think people are more likely to come talk to you if you're sitting at a desk on an open floor than if you're sitting in an office. Additionally, being able to just poke your head up and ask a question feels less intrusive than having to walk into someone's office. That said, this is kind of the point. If you do a Google search on the cost of even minor interruptions, you'll find that they can be extremely damaging, to productivity, to stress levels, and to quality of work.

For me, 90% of these "micro meetings," questions, etc, are better handled somewhere that would keep a record, a place that questions asked could be asked as easily, and more efficiently, and be easily ignored if someone is "in the zone" or doesn't want to be disturbed. Fog Creek recommends HipChat (which is what Fire Hose uses), but 37 Signals own Campfire also gets good reviews. These both have the added benefit of, if you have remote workers, it Is less likely that they will be excluded from the decision making process if your team is good about using chat for minor / micro discussions over open office eavesdropping.

What do other people think? Am I missing a benefit of open office plans (other than cost)? Do people feel that the interruptions of an open office aren't as bad as I make them out to be?

Transparent Persistence

  By Jeff | August 27, 2012 - 10:23 am |
Leave a comment

I'm in the process of refactoring some code for Go Home Dinosaurs and I've run into an interesting problem.

Because GHD was originally designed to be a microtransaction based game, and because we wanted to have (essentially) cloud saves, it's using web services to keep track of player progress, coin totals, purchases, inventory, etc. Now, we want portions of this to be pushed into local save data. Because the schedule on GHD was so tight, there's a ton of code that just assumes that the server will be there, and it also (correctly) assumes that we want to update player information incrementally on the server. It's therefore filled with calls like this:

    GameDataServer.SendServerMessage("updateField_request", data);

// in some other portion of code
void HandleMessage(Message msg)
{
    switch(msg.Type)
    {
        case "updateField_response":
            // actually update the field with the new value
            field = msg.Data.NewValue;
            break;
    }
}

The problem is that this doesn't really fit with an offline mode. In order to combat this, one of my colleges wrote a "fake response" method, which, if there's no server to talk to, queues a fake "success" message for every call. While this was certainly the fastest, most straight forward way to accomplish an "offline" mode quickly, I'm not a huge fan. It's too coupled to the thought that there's a server between us and the persistence store. It also means that the server's responses and the "fake" responses need to be kept in sync, and I'm never a fan of "if you change this, make sure to change this" code. It's way too likely to break.

But how to combat this? I'm not sure I have an answer for this, given the current design. All I can do is outline what might work, and what I want.

Generally what I want is transparent persistence. I don't what to have to understand *how* the platform wants to persist things. I want to edit my data and then say "persist" and have the platform figure out the best way to perform this.

The first step here, I think, is to take a lesson from MMOs, and ignore trying to cache unconfirmed changes. Most MMOs make the change locally and send a message to the server, requesting that the change be made. For persistence, successful calls are generally ignored, because we know that what the server says and what the client says are now in sync (or are in sync from when we sent the message). It's only in the case of an error that things have to be handled. Generally, the correct response is then to rollback whatever transaction was made on the client, and hope the player doesn't notice (or present an error message). Since the server can (and should) return it's version of the variable in the error response, it means that you can remove a lot of special case caching code that can easily get mucked up (or start performing double duty, which is the case with at least one of our caching variables). It also means that platforms that always succeed in changing data don't have to send fake "success" messages. Only the (hopefully rare) errors in server calls need to be handled.

Second step is to start keeping track of "dirty" fields, and updating them as needed (whenever persist gets called). However, our current server code (which I didn't write, and some of which I'm trying to keep intact for a bit) assumes (again, correctly) updates for every single transaction. It's hard to say "the player has x coins." Instead, it asks to told how many coins the player gained. You can't send the entire inventory, you have to send "I'm adding / removing the following item from the inventory." Or "I'm purchasing the following item and it should be added to the inventory." These are common client – server interaction things, even in MMOs because it means that the server can actually confirm that the logic you're performing matches what it expects, but it does make doing transparent persistence very difficult.

The only way I can think to do this even remotely well is to push each update to the player to a platform intermediary. Platforms that persist the data locally ignore incremental updates and just save all the data once a full "persist" call is made. Platforms that persist the data remotely do the opposite, pushing each update to the server, rolling back errors, and ignoring requests for full persistence, or sending full persistence requests and to confirm that the local and remote versions are still in sync.