Finding D3D9 Leaks

I ran into this problem working in a DirectX 9 application recently, and found it useful enough that I thought I'd share. We were running into an issue where returning from an alt-tab was not recreating the device properly. This is usually caused by a created D3DPOOL_DEFAULT asset not being released and recreated before attempting to recover the lost device. As a result, you get this message:

Direct3D9: (ERROR) :All user created D3DPOOL_DEFAULT surfaces must be freed before ResetEx can succeed. ResetEx Fails.
Direct3D9: (ERROR) :ResetEx failed and ResetEx/TestCooperativeLevel/Release are the only legal APIs to be called subsequently

One additional thing that was throwing me was this error message:

Direct3D9: :Window has been subclassed; cannot restore!

This can probably safely be ignored, and may be a symptom of the back buffer leaking specifically, but generally this whole error set is a sign of leaking D3D objects.

If this is something innocuous, like a single texture, you can sometimes figure it out by simply enabling the DirectX debug libraries, running and exiting your application, which will give you the AllocId of the leaking asset, which you can then break on. However, a large number of leaks indicate that something lower level may be leaking, and potentially a leaking reference to something, like the back buffer (which was our case). How to find it?

Here's what I ended up having to do.

First we did what we need to do for every bug, reduce it to the fewest steps possible.

Next, run PIX and take a full capture, until you've gotten to the point of the leak, then close. If you have a lot of steps to reproduce, you'll want to make sure you've got a lot of hard drive space, and some time on your hands, because it can take a while.

Once you have your full PIX run you can look at this column:

And set it to Never. This will show you every D3D object that's leaking, regardless of how many AllocIds it has (interesting tidbit, a single D3D object can allocate multiple AllocIds). If you move to the last frame (and wait for PIX to catch up, which, again, can take a while) you can even see how many references exist during shutdown. You could even walk each frame and see when the reference counts of your leaking assets go up.

In our case, like I said, we were leaking a reference to the back buffer. Most obvious was to look for calls to IDirect3DDevice9::GetBackBuffer, but none of those were leaking. Less obvious was to look for cases of IDirect3DDevice9::GetDepthStencilSurface and IDirect3DDevice9::GetRenterTarget(0). In our case, a call to GetRenderTarget was leaking as a result of a short circuit in a logic check, resulting in the back buffer reference to dangle in a very specific logic case.

Granted, this leaking does not pose a lost device recovery problem with DirectX 10 or DirectX 11, but since many engines are still DX9 based, I thought I would share.