Sorry for misunderstanding where you’re coming from. I’ve found that the types of people that read my blog are pretty varied, so I couldn’t take for granted you were an MMO developer.

You are absolutely right about using the database as the fixed definition. I think it’s great for most circumstances so long as you’re looking to specify what your data looks like and have that dictate your code. So long as your code generator can understand (say) 90% of what can happen in a database (constraints, nulls, FKs, and lookup tables) than this is the way to go. If you’re working from a fixed set of features (or want to supply features that can’t easilly be specified in a DB schema) this become tricky.

For us, we’re not as interested in trying to create a data definition and go from there. What we’re looking at a system where we have a very strict set of rules for what your data looks like and what the stored procedures look like. We have specific sets of features that aren’t nested to the way the data looks, but the way the data behaves, and we’d rather keep that as detached from the data as possible. Additionally, we’re working cross platform, and it’s easier to go from a definition to multiple databases than read different types of db schemas and figure out the proper code to generate (though there are some great ORM libraries that do it). In general, I can’t see how Orbus can go from database to code for our particular tool. It just doesn’t strike me as an option. That said, I may be missing something.

I will say that there are actually other solutions I’ve thought up for this problem that bypass the diff solution or that can supplement one. The first actually works well if you have a tool that you’re using to edit your data definitions. The idea is to supply “diff hints” as part of the upgraded schema, adding hints like “Rename”, “MovedFrom” etc that will show up in the diff, but who’s only meaning is to supply information about how the schema has changed. When you’re editing the schema directly (which is the case with our tool currently), this isn’t really an option, but we’re working from a much more constrained set of features so I may actually be able to get renames and moves to work pretty well (I’m writing up most of the unit tests for it now)

The second solution is to provide an alter system yourself, where you do what might be considered a migration script. You save each iteration and the final resulting schema. Unfortunately that gets really unruly really quickly.

I’m an MMO developer, and currently we use a home-rolled tool to generate source code and SQL schema from a common definition for our game. The more I’ve used it, the more I’m convinced that not only is this the wrong thing to do, but also that there’s no reason why it should be done this way and not just be done from the database. After all, if that is the authoritative state of the data, it should form the authoritative format of the data too. It also just gives you yet another thing that can potentially get out of sync.

I think in our case we ended up with this tool due to a distrust of relational databases and a belief that writing a row should be as transparent and as SQL-free as possible, for the benefit of those that don’t know databases. Making the internal tool the authority on both source code and SQL schema allows this, but severely hinders you in other ways, such as schema migration as you know. Rearranging the work flow to work ‘natively’ in SQL just seems to be the more reasonable thing to do.

One idea, given the constraint we work under, is to have the generation tool demand that there is only ever 1 column change per version. It can compare the new XML with the current DB state and look for the changed, added, or deleted column. If a column in the DB has been replaced by one in the XML, the tool can safely assume it’s renamed, and can preserve the data. To completely replace a column would require 2 commits - one to remove the column, then one to add it. A diff can never really tell the difference between a replacement and a renaming without some external context, so a system like this bypasses that problem. It’s a bit more hassle in the replacement case, but in my experience that’s actually a more rare requirement.

I have actually done things that way in the past, generating my business objects from the data instead of the other way around. Unfortunately, I’m stuck working the other way simply because of the type of clients I’m working with (game developers) and what we want to actually want to do. Because of that, I have to have a generated database, generated “business logic” and generated stored procedures. It’s unfortunately unavoidable given the task.

But, yes, if I were working on a web site or business site, I would go (and have gone) the other direction, Unfortunately, I’m still forced to generate my database, so the problem still exists.