Twitter  Facebook  YouTube  E-Mail  RSS
The One Man MMO Project
The story of a lone developer's quest to build an online world :: MMO programming, design, and industry commentary
Crash Reporter
By Robert Basler on 2014-07-10 22:51:58
Homepage: email:one at onemanmmo dot com

Something I think is really important in shipping a product is getting the number of bugs down to zero. The fewer bugs you have, the less people will need to rely on support, and the more time I can spend adding features. I used this theory on a previous product with considerable success.

So before I send the game out for people to play I thought it would be good to have web-based crash reporting in place.

Google Breakpad

I spent two afternoons looking at Google Breakpad which is the clear leader of the canned solutions. Most of that time was spent trying to figure out how to build it. The documentation for Breakpad is abysmal. It's more of a puzzle than documentation. While I was eventually able to build the client library, when I considered the trouble of figuring out its server-side and the amount of code I already had that it would duplicate, I decided to build my own. I really can't figure out why Breakpad doesn't include a Visual Studio solution for the server-side tools (which I'm still interested in because Breakpad's server-side has a cross-platform dump analyzer which may be useful should my crash volume get too large to tackle by hand.)

Breakpad did prove hugely useful as a code example.

The Many Sides of Windows Crash Recording

There are no less than six different hooks you need in a Windows program to catch most faults. I'm not ruling out the possibility of more, but these are what I have so far.
    mPreviousFilter = SetUnhandledExceptionFilter( LairUnhandledExceptionFilter );
mPreviousInvalidParameterHandler = _set_invalid_parameter_handler( InvalidParameterHandler );
mPreviousPurecallHandler = _set_purecall_handler( PurecallHandler );
_CrtSetReportMode( _CRT_ASSERT, 0 );
mPreviousTerminateHandler = set_terminate( TerminateHandler );

SetUnhandledExceptionFilter sets a top-level hook for Windows' Structured Exceptions. This gets most of the good crashes.

_set_invalid_parameter_handler hooks into CRT's function parameter checking.

_set_purecall_handler is where the program ends up if you call a pure function.

set_terminate is a per-thread hook to the CRT terminate call which is called when lots of things go wrong. For now I'm only setting this on the main thread. Might have to look at setting that per-thread some day.

I also added a try/catch block around the whole game and around every thread to catch unhandled C++ exceptions.
// Whole game goes here
catch (...)
Lair::Thread::UnhandledException( L"unknown type", "main" );

_CrtSetReportMode turns off CRT assert message boxes so they pass to my handler.

Here are the signatures for the handlers you'll need:
    LONG WINAPI MyUnhandledExceptionFilter( struct _EXCEPTION_POINTERS *ExceptionInfo )
void InvalidParameterHandler( const wchar_t * expression, const wchar_t * function,
const wchar_t * file, unsigned int line, uintptr_t pReserved )
void PurecallHandler(void)
void TerminateHandler()

One important thing to note is that almost all of these facilities do not work when running under a debugger, so I use IsDebuggerPresent() to turn the crash reporter off entirely when debugging.

Windows Dump Files

Writing a dump file on Windows is pretty straightforward. MSDN provides a nice example of using MiniDumpWriteDump, although it is missing a call to CloseHandle( hDumpFile ) at the end. One other change I made is to load dbghelp.dll at runtime. Dump files are written to the Windows TEMP directory so that they will be automatically cleaned up by Disk Cleanup if the LairCrashReporter fails to delete them.

One thing that is not straightforward about creating a dump file is the Exception Information. Normally during exception handling you get a pointer to an EXCEPTION_POINTERS structure you can pass to MiniDumpWriteDump. For asserts and all the CRT problems you don't get an EXCEPTION_POINTERS structure. I originally tried going without, but a dump without a call stack is not very useful. Luckily I found this handy article which has a nice code example that will build an EXCEPTION_POINTERS structure from your current thread state. And unlike some other really scary example code I found, it is backed by being direct from CRT. Look for "Retrieving Exception Information".

Out of Process Crash Dump

The dump documentation strongly suggests doing a dump from another process which is not currently crashing. This makes a lot of sense, so I wrote a LairCrashReporter program which Miranda runs on startup. LairCrashReporter opens a named pipe which Miranda connects to (while everything is still running well.) When Miranda crashes it does the absolute minimum to send all the information for the dump over the pipe to LairCrashReporter which does the dump write. Interestingly enough, while EXCEPTION_POINTERS contains pointers to addresses in the crashing process, MiniDumpWriteDump doesn't care. That's nice.

Log to Server

One the dump file is written, LairCrashReporter uses HTTP POST to send the crash dump along with any additional information about the crash (file, line, error message) so it can be logged in a database. Luckily I already had everything to do the HTTP transmit. Then I get rid of the dump file.


I added a couple popups to LairCrashReporter to notify the user when it succeeds (or fails) to log a dump.

The Future

I still have to write the server-side scripting to log the crashes. I found this program which looks like it will extract a callstack from a dump file and output it as text. That will be handy to help eliminate duplicates if volumes go up.

By Robert Basler on 2019-11-08 14:42:11
Homepage: email:one at onemanmmo dot com
Addendum: Some virus programs will block transmission of the dump file unless you lightly encrypt the dump data when sending it from the client.

New Comment

Cookie Warning

We were unable to retrieve our cookie from your web browser. If pressing F5 once to reload this page does not get rid of this message, please read this to learn more.

You will not be able to post until you resolve this problem.

Comment (You can use HTML, but please double-check web link URLs and HTML tags!)
Your Name
Homepage (optional, don't include http://)
Email (optional, but automatically spam protected so please do)
What is 5 multiplied by 1? (What's this?)

  Admin Log In

[Home] [Blog] [Video] [Shop] [Press Kit] [About]
Terms Of Use & Privacy Policy