Twitter  YouTube  E-Mail  RSS
The One Man MMO Project
The story of a lone developer's quest to build an online world :: MMO programming, design, and industry commentary
International Cascade of Fail
By Robert Basler on 2011-12-14 15:39:13
Homepage: email:one at onemanmmo dot com

I like to do little features in the evening, separate from the bigger stuff I work on during the day. Sunday evening I thought a Terms of Service (TOS) screen would be a nice easy feature.

Little did I suspect I would encounter an international cascade of fail trying to get the TOS to actually work.


The quick and easy approach to adding the TOS screen would be to store the TOS on the client and point the in-game browser at it. But then you have to worry about updating it or having someone mess with it with Notepad. Another option would be to put a document on the webserver, but people have been talking a lot about invalidating website click-through agreements by messing with their javascript or running a proxy.

An easy solution just wasn't going to make the lawyers happy. We need secure storage and transit for the TOS document to the player. Sigh.

So I coded a TOS step into the login sequence, added a packet to send the full TOS text to the client, modified the login server to be able to load the TOS from the database, created TosTool to load the TOS from a file and store it in a database table, added support for multiple TOS documents and versioning, added a page handler for the TOS form to the client, and I was ready to give it a go. Most of that was done Sunday before I went to bed.

Monday morning, I needed a sample document for debugging so I grabbed a copy of the TOS from Turbine's Lord of the Rings online. A lawyer will write the real TOS for my game sometime later. I pasted the text into the HTML page design for the TOS screen and did a couple of little edits. Then I tried to load the file into the database. TosTool ran for ages, and ages, and ages. Hmmm.

Fail #1: The class that reads text files, which I haven't touched in a year, has a bug where if the last line of the text file it is reading doesn't have a trailing carriage return, it loops back to the beginning of the file and reads the whole thing again, over and over and over. Fixed bug #1.

Fail #2: TosTool was now able to read the file and write the TOS text into the database. Looking at the results in the mysql client, there were long sequences of junk characters sprinkled throughout the document. Ewwww... Taking a closer look at the input file, it is UTF-8 encoded and includes the trademark symbol and directional quotations -- lots of non-ascii characters in otherwise English text. Internally my game uses UCS-2, so once data is into the game, there isn't much danger of it messing up - at least for EFIGS languages. The game was correctly identifying the input file as UTF-8, but it turns out the code that converts the UTF-8 to UCS-2 (which I wrote ages ago and probably never used) is all sorts of broken. Fixed bug #2.

Fail #3: Ran the fixed UTF-8 code, and it is still all sorts of broken. Looking at the character comparisons I notice that visual studio is displaying the char constants as negative. Ack! So I learned that Visual Studio defaults to signed char -- a default I've never found particularly helpful. Added the /J switch to the project. That fixed the comparisons, but Visual Studio still stubbornly displays the character constants as negative in the debug view. Boo.

Fail #4: Ok, so the TOS text is loading nicely into TosTool and writing neatly to the database. Looking at the database in the mysql client, it looks good now. The MySQL client prints out a few junk characters, so it obviously doesn't support UTF-8, but at least the strings of junk are the right length now.

Fail #5: Next up was to load the TOS from the database into the service that manages logins. I run that. ASSERT. In a function called AsciiToUnicode. Huh? The data is supposed to be UTF-8! It turns out that the database classes have support for UTF-8 while writing to the database, but the reading routines don't, they think everything coming from the database is ASCII. I've never used anything but ASCII in any of the data I've written before so I never noticed this oversight.

Fail #6: Looking closely at the loaded TOS again, there were still some anomolies. MySQL has certain characters it doesn't like in queries. These characters are where SQL injection attacks come from. Checking into it, strings were being escaped while going to the database, but not unescaped coming back. Again, I haven't put a lot of fancy text into the game's database, so I never noticed.

So at this point, I'm thinking I need to do some more work on my unit tests. But, no time for that now, I've got a broken game to fix. Soldiering on...

Fail #7: With the TOS now downloading to the client, I notice it takes foreverrrrrrrrrr to do it. Looking at the debug trace, the data is coming down fine, one K, every second. Hey! Transmission rate limiting works! Kinda sucks when transferring a 21K TOS document though. So I modified the server to increase the transmission rate limit while sending the TOS so it takes just a couple of seconds.

Success. It is Wednesday afternoon and I have a working TOS screen. And a whole whack of good bugs fixed. Never underestimate the amount of work required for an easy little feature.

By omeg on 2012-02-28 06:43:30
Homepage: email:
Ah, proper internationalization is "fun" in C/C++. I worked for a few years as an i18n engineer, really opens up eyes on stuff you can run into. And makes higher-level/managed languages tempting. ;)
Are you going to have in-game text localized? How are you handling complex names like "Three-handed giant sword of awesomeness" which can have variable word order etc in other languages?
By Robert Basler on 2012-02-28 10:11:03
Homepage: email:one at onemanmmo dot com
Yes, in-game text is localized. I spent a lot of time working with the internationalization people at EA on the library I maintained, so I've been there. It was localized in 13 languages when I left.

I started by making all internal strings UCS2 which gets me most of the way there for supporting other languages.

All the strings are stored in a spreadsheet with one column per language and a tool turns that into a table with a fast hash lookup by ascii tag so a string lookup looks like:

String AppFolder = Lair::LocaleDb::GetString( "AppFolder" );

To deal with variable word ordering, I used some template function goodness to implement a system like C#'s String.Format.

String server = ( Format::Format( L"Server: {0}/{1}") % serverName % systemVersion ).GetString();

So all formattable classes have a FormatString method which pretty-prints them. Indices let me specify the order of items inserted into formatting strings. So a formatting string where the parameters are out of order would look something like this:

"{3:4cc} {0} bytes to send this update {1} maxSendRate {2}"

What follows the : is a formatting code. 4cc tells it to print as a four character code. I have formatting commands for numbers, dates, times etc. Formatting is based on the user's locale, so there is a great table of number, currency and other formats for each supported locale.

Any class that implements the FormatString method can also be passed to any string formatting function as a parameter which is super-duper convenient.

As for names, at this point at least, my items don't have dynamically constructed names. Each component has a name, and a bunch of stats and that's what you see.
By omeg on 2012-02-28 16:33:33
Homepage: email:
Very nice! No pesky dynamic names simplify things a lot.
On a slightly related note - how do you store client's assets/resources? Some existing archive format or your own?
By Robert Basler on 2012-02-28 17:02:38
Homepage: email:one at onemanmmo dot com
I built an asset management system a while back, I haven't blogged about it since it predates the blog. I need to make it support asynchronous loading, so maybe I'll talk about it soon.

In any case, assets are stored a couple different ways. I have what I call "asset containers." I have three types of asset containers.

I have an asset container which can create a giant blob with an index to access the files inside which remains on disk. Most data goes in these. I haven't really thought too much about how to group the data files to make patching reasonable but that'll be an issue.

I have a second asset container that can load the asset blob file into memory in one go and refer to it there from then on. That's handy for fonts and stuff that is needed throughout the game's runtime.

The third asset container I use during development and it just creates an index to files stored in the regular windows directory.

I have 500,000+ asset files at the moment which take about 4GB, so the directory container really saves on build time.

New Comment

Cookie Warning

We were unable to retrieve our cookie from your web browser. If pressing F5 once to reload this page does not get rid of this message, please read this to learn more.

You will not be able to post until you resolve this problem.

Comment (You can use HTML, but please double-check web link URLs and HTML tags!)
Your Name
Homepage (optional, don't include http://)
Email (optional, but automatically spam protected so please do)
What color is a lemon? (What's this?)

  Admin Log In

[Home] [Blog] [Video] [Shop] [Press Kit] [About]
Terms Of Use & Privacy Policy