Twitter  Facebook  Google+  YouTube  E-Mail  RSS
The One Man MMO Project
The story of a lone developer's quest to build an online world :: MMO programming, design, and industry commentary
Invest In Your Debug Build
By Robert Basler on 2013-10-15 01:32:35
Homepage: email:one at onemanmmo dot com

I always hated working on games that didn't have a working debug build. On some the framerate was too low. Some didn't even build. There was always one real reason for it: somebody in charge didn't think it was worth having a running debug build. But the cost of a broken debug build is a huge increase in bugfixing time. Debugging a non-debug build results in misleading values in your debugging windows, bad callstacks, and heavy reliance on static analysis of trace output to figure out what has gone wrong. In short: it has a huge cost in developer time.

I work almost exclusively in a debug build. I don't have the time to do anything else. I recently greatly increased my draw distance and added a whole bunch of new props to my scene. This put a much heavier load on the CPU for both loading and rendering. For the first 30 seconds after connect to the server the debug build limped along at 2 FPS and was intermittently jerky as new terrain streamed in during gameplay. Last week I was forced to spend a couple days optimizing my debug build.

There were a bunch of problems I identified:

  • With my recent increase in draw distance, after connection I had 72 terrain blocks loading, generating meshes, and loading texture and normal data.
  • I didn't like the way static prop models for terrain blocks were still loading long after the terrain block was rendering. I wanted all the props and the terrain block to appear simultaneously.
  • Texture loading was done on the main thread and each of the 24 new props had either 6MB or 15MB of texture data. Some of those textures took 6,000ms to load!
  • Prop model loading was done asynchronously, but a bunch of work to set up those models in the scenegraph was done on the main thread, sometimes hundreds of time per terrain block.

The first big improvement was to move texture loading to run asynchronously in the threadpool. It had been a TODO for a while, an a 6,000ms frame spike can't be ignored, so there wasn't much question what to tackle first. That gave me the full CPU power of all available threads to load and convert those PNG's to my internal format. (I have a cool thread pool system where I can create micro-tasks and hand them to the threadpool to run. It's pretty slick -- I should blog about that some time.) At some point I may need to go the final step and convert the PNG's to my internal format at build time so they can be loaded directly into RAM ready to go.

With the big texture loads running asynchronously, most of the time the main loop was averaging 15FPS during heavy loading which is good enough for debugging. I was however, still seeing single frames which would spike as long as 12,000ms.

The first problem I tackled was when a lot of props were attached to a single terrain block. Putting all those props into the scenegraph took some time. This was a reasonably simple task of rearranging a bunch of code and making sure there were no thread safety issues. I moved it right after the model load in the threadpool. Only once the props were all setup would the terrain block and set of props be added to the scenegraph for rendering.

The total frametime was still spiking. When all those props and the terrain block hit the renderer simultaneously for the first time, there was a large amount of time being spent by the culling system, and then by the renderer itself.

The culling system has to measure the size of all the new meshes to determine if they can be culled. To do that it does a projection on all of the vertices of each model on the CPU to build a bounding box. Some models have a lot of vertices, and that takes a little while to do. I tried a bunch of things on this but the final solution was to modify the vertex list itself to do the projection and measurement with the least possible number of CPU cycles. I managed to double the speed of the culler but that still wasn't nearly enough. I had a little shot of clever then, and I modified the game to set up the culling data while it was doing the other asynchronous scenegraph setup.

With the culling spike obliterated, last up was a 5,000ms draw call which occurred when I did the first render of all those new vertices and textures. I tried a new tool this week: NVidia NSight. It indicated that the bulk of the frame time was calls to glTexImage2D (it has this cool timeline view where you can zoom into a frame to get down to individual rendering calls and see how long they take. Unfortunately I couldn't think of any way to fix this issue except to give up on my desire to have all the props appear simultaneously with the terrain block. Truth be told, I could have done a bunch of fancy stuff to send all that data to the GPU ahead of when I wanted to actually render it, but instead, I limited the renderer to sending one new material list to the GPU at a time. That spreads the shader compilation, vertex transfer and texture loads of new models out over a few frames. The prop pop-in problem is really only very noticeable right after connection. Once most of the props are loaded, they appear at the same time as new terrain blocks. I might just hide this problem with a load screen.

I made two last changes. I fixed a bug in my prop layout system, reducing the number of props per block to more managable levels and I reduced twenty-four 3MB diffuse textures to a single 3MB texture for artistic reasons. In this case, the performance improvement was just a perk.

The debug build is again running smoothly at 15fps during heavy loading with no frame time spikes and around 25FPS the rest of the time. If you think all this work was a waste, don't forget that I can now debug again at full speed. Or just consider that a faster debug build means a higher framerate on lower-end hardware which gives a bigger potential market for the game.

Back to work on new stuff...

New Comment

Cookie Warning

We were unable to retrieve our session cookie from your web browser. If pressing F5 once to reload this page does not get rid of this message, please read this to learn more.

You will not be able to post until you resolve this problem.

Comment (You can use HTML, but please double-check web link URLs and HTML tags!)
Your Name
Homepage (optional, don't include http://)
Email (optional, but automatically spam protected so please do)
What color is a lemon? (What's this?)

  Admin Log In

[The Imperial Realm :: Miranda] [Blog] [Gallery] [About]
Terms Of Use & Privacy Policy