I got 50 clients logged in this afternoon - squeeeee! So maybe that doesn't sound like much for an MMO, but its a pretty huge step forward. The goal of this initial load testing was to get 50 game clients into the game at once with 5000 units moving around and fighting. At the start of last week it would start crashing with around 3 clients in the game. Over the last week and a half I've fixed 24 bugs, some small, some rather challenging. I still have two more I'm working on and eight I've seen only once.

As I suspected, the code for simulating movement of the tanks isn't efficient enough yet. With 20 players in the game - 2000 tanks all driving around, the server started running too slowly to keep up with the simulation. So I turned off unit movement temporarily and after another day of bugfixing I was able to get all 50 clients and 5000 units in-game. The next step will be to do some work on the unit movement code and see if I can get those 5000 units all moving around and fighting.

It's great having a tool to find all my bugs, it'd just be nice if it wouldn't find them all at once! Every day the last week has gone like this: I start ten-fifteen instances of Fauxranda, wait a minute or two, eventually a subset of them would crash. I'd fire up the debugger on each dead instance, check each one and I'd end up with 2 or 3 new bugs. Sometimes something on the server side would crash. I'd then fix the bugs, some quickly, some over long hours, then repeat.

I've been thinking a lot the last couple of days that a completely deterministic game server/client with support for input recording and playback (like this) sure would make some of these bugs a lot easier to fix. Every time I ran into a new bug, I could just restart the program, play back the input that caused it to die, and figure out the bug easy-peasy. A recording system like that is a little tricky for the multithreaded aspects of the code, but might be attainable on the server without too much effort.

A really useful tool during all of this load testing is Windows 7 (and later) Resource Monitor. Access it using Ctrl-Alt-Delete, select Task Manager, Performance tab. I can put check marks next to all my server processes and monitor their CPU, network, memory and disk performance in real time.

What Resource Monitor has told me is all good news. The server's network usage is perfect! It uses at most 1K/second for each connected client even under load, usually less. Memory footprint is better than expected, and CPU usage is good for everything except the unit movement simulation.

On the weekend I watched this rather scary presentation on how cheaters attack DayZ (I've also been following along with the Gaffer on Games series on Building a Game Network Protocol which touches on the topic.) Miranda's networking has never been designed to thwart deliberate attacks, so I spent one evening hardening the server against malformed packets and other malicious behaviour. There's still more to do, but it's a start.

I'll be back to load testing tomorrow - working to get those 5000 units all moving. Really I just want to log in with the regular client with all those tanks in game to see what it looks like.