QuantumG's Blog

The never ending effort to recreate a game from 1993 continues. There's now nine developers in the project and none of us do much. Every now and then I get bored and fix something.. go to check it in and discover that, shock, someone else has checked something in too! Then I have to update and resolve conflicts. Take the good with the bad I guess. Anyway, in release 0.2 you will be happy to discover that we have done some work to get sound faithfully reproduced. For a while, we were focusing on music and trying to get the SDL midi synthesiser to play back this ancient XMIDI format. Believe it or not, this was actually acheived. Slight problem though, Syndicate used custom instruments. This was not uncommon at the time. The result was that the music just didn't sound good.. but it was good enough to ignore for a while. Having declared that I wasn't going to touch sound.. I got bored one day and changed my mind. I told myself that I really ment I wasn't going ...

I was recently asked what I would do differently if I was writing a decompiler from scratch today. Having worked on decompilers for the past 9 years myself and worked with others who have worked on decompilers for longer still, I like to think we now know enough about it to recognise some of the mistakes we've made and identify some areas of improvement. The primary difficulty of decompilation is that most analysis requires interprocedural information. Most everyone recognises this need early on so there's an assumption that you need to keep the intermediate representation for the entire program in memory at the same time. For many years we've had a hell of a lot of memory to play with (4 gig will be standard on desktop PCs next year) and with 64 bit architectures and virtual memory you can swap to disk for as much memory as you need. So it would appear that structuring a decompiler like a compiler, with intermediate disk formats for each compilation module is unnecessa...

What's the most annoying thing about trying to write a game engine? Ohh, I'm spoiled for choice, ok, I think the most annoying thing is getting together some half decent looking art so you don't go insane testing your engine with stick figures. Well, I recently came up with an interesting solution.. why not use some existing art from one of your most favourite games? You may note, I've done this before . This time, I picked a game that is a little more recent, but not too recent, GTA3. No, not Vice City, not San Andreas, but those two are the same engine, just updated some. As I've mentioned before, I have a liking of the OGRE graphics engine . So the first order of business was to get the models out of GTA3 and into OGRE's mesh format. The format used by GTA3 is the Renderman DFF file format . The individual dff files are concatenated together in the models\gta3.img file, which is indexed by the models\gta3.dir file. There's plenty of tools around t...

Recently, Greg K.H. made an offer to write free linux kernel drivers to any company that will provide him with docs to their hardware. Some BSD folks had a problem with Greg making this offer to people who would only give up their docs if the developer signed an NDA. Theo de Raadt chimed in declaring that "it is a fucking farce" , which is his usual way. As far as they are concerned, an open source driver without docs is not open at all. That may be so, but I still think having source code.. any source code.. which actually works, is better than having no docs or source code. It would be nice if companies would hand over docs without an NDA.. that is certainly preferable for everyone.. but some of them just won't, and getting them to hand over docs under an NDA with the ok for someone to write a free driver (be it GPL or BSD licensed) - that's something and it shouldn't be ignored. Now, if companies start doing this and drivers start showing up as GPL, but wi...

In the last few years we've seen the rise of the so called distributed revision control systems. These are, essentially, tool support for maintaining your own personal version of an open source project. Let's take, as an example, the WINE project . They use Git , the same version control system as the Linux kernel. Suppose I have a favourite application that I would like to run using WINE. There might be some things that my app requires which WINE doesn't yet support. I don't mind implementing these things, but I really couldn't be bothered going through the hassle of satisfying every whim of the WINE developers (how they like their patches submitted, how they like their changelog written, etc). What happens when WINE releases a new version which I want to upgrade to? Do I keep an entire source code tree around just to run my favorite app? What if I have more than one favorite app? Git lets me update my source code tree and merge any updates that conflict wi...

People that provide rpm files, but not tar.gz/tgz files, should be shot. Typically this isn't so much of a problem, as I can use alien on my debian based distro to convert an rpm into a deb, but I'm in the road right now and don't have access to a machine with that. So, after struggling to get the rpm utilities installed on a naked linux box, where I didn't have root, I've decided that enough is enough. I figure what I need is a web site where I can submit the rpm and it will create a tar.gz for me (ala ps2pdf.com ). And I figure that other people need this too. So I've sucked up the cost and done it. Here it is: www.rpm2tgz.com . Of course, I've put a lot of Google ads on the site. Hopefully this will give me enough income after a year to cover the renewal of the registration and the hosting. Otherwise I might just let it lapse.

Argh. It's 2006, and I still don't have a good decompiler. All is not lost. Thankfully, there are still interesting things to decompile that are both small and contain lots of stuff that makes decompilation easy (e.g., symbols, relocations). So, let's do it manually using some trustworthy old fashioned tools: a disassembler, a text editor and some string processing tools. Let's choose a target. I'm going to go with a linux kernel module because they are small, contain symbols and relocations and because there exist GPL ones that I won't get in trouble for reverse engineering publicly. Just choosing something at random from /lib/modules on my Ubuntu linux box I come across new_wlan_acl.ko from the madwifi-ng drivers. Right, now we need a disassembly. No problem. Just do objdump -d --adjust-vma=0x8000000 new_wlan_acl.ko > out.dis . That almost gives me the object as it would look mapped into the linux kernel. Slight problem though, none of the reloca...

I, like many people, own a robotic vacuum cleaner. It's crap. With all the advanced robotic technology it has on board, it's still lacking in vacuum cleaner technology . Although it is bagless, it's not the good kind of bagless. There's no double vortex mechanism here. In fact, there's very little suction on it at all. That's kind of important for a vacuum cleaner I think. Besides which, the robotic technology isn't all that "advanced" anyway. It still gets stuck in corners or on that same part of the rug. So when I feel the carpet needs a bit of a spruce, I pull out the trusty Vax and put my back into it. Where's my maidbot? It's been 40 years since the The Jetsons and I am still waiting for my flying car, err, I mean, maidbot. Now, of course, I realise that it would be a bit expensive to get together a crack team of Japanese scientists just to make me a maidbot (and yes, they have to be Japanese scientists) and that no amount o...

First of all, let's get the rant out of the way. Cygwin is a big pile of junk. It's like everyone who touches Cygwin gets the clue beaten out of them before they're allowed to hack on it. Here's a tip guys, if you're going to try to do something really hard (like build a POSIX compatible software layer on top of an operating system that holds many parts of the POSIX standard as the anti-thesis of its design) you have to put a lot of effort into it . Ok, done. I recently had this problem: Fatal server error: could not open default font 'fixed' for which I googled, lots and lots and lots and found no adequate solution. I found the Cygwin/X Frequently Asked Questions and it had two suggests as to what the problem could be: You don't actually have the xorg-x11-fnts package installed (duh, thanks guys, yeah, that wasn't the first thing I checked). The mount point for /usr/X11R6/lib/X11/fonts was invalid at the time that Cygwin's setup.exe...

I'm a Ubuntu user but I also run a number of Windows machines for work purposes. I was actually shocked recently to discover that CUPS supports my Brother HL-5140 laser printer without me having to download any binary blobs from the Brother website. So that's good. What's not so great is that gnome-cups-ui has no concept of workgroups or domains for connecting via smb to a windows printer. When you go to "add new printer" or when you view the connection properties after adding the new printer, the fields you are presented with are: Host Printer Username Password After googling for a bit, I discovered that people have figured out that if you put the workgroup name in the Host field and then enter username:password@host/Printer into the Printer field and nothing into either the username or password fields, you can print! Curious about this obscurity, I did an apt-get source gnome-cups-ui and hunted around for a bit. Eventually, I found some code that looks basicall...

If you pop over to the NVIDIA web site and download the 3d card drivers for Linux, you'll note that there is a /usr/src/nv directory. In that directory is source code to the "thin layer" to the Linux kernel which NVIDIA links their binary blob. This, of course, makes no legal difference - NVIDIA are still extending the Linux kernel and therefore it is unlawful to distribute the Linux kernel along with the NVIDIA drivers, but NVIDIA doesn't do that, so it's not a problem - for them. Anyway, that's a side issue. I was thinking, recently, about the Linux kernel's "tainted" flag. Essentially, whenever you install a kernel module that isn't under some accepted open source license, the kernel sets a flag so that developers know there is a chance that any bugs you report might have been caused by code they can't fix. In general, this is not such a big deal as kernel modules are typically small and easy to isolate. The NVIDIA graphics driv...

Sometimes copyright law is just stoopid. Sometimes the rules just don't apply. Have you heard the story of Steve McDonald and White Stripes? Here's nice flash animation of what happened, or you can keep reading.. Steve McDonald is a veteran member of the band Redd Kross. He likes the White Stripes, but he thought they would sound better if they had a bass guitarist. So he appointed himself. He had the equipment, and the skill, so he made up some bass tracks and added them to his favourite White Stripes songs. He then posted those songs on the Redd Kross website - without permission. Of course, this could land him in hot water, but luckily he bumped into Jack White who gave his assurances that he wasn't going to sue. Before that happened I'm guessing Steve McDonald just didn't give a damn.. after all, he's a rocker, man. I'm not a rocker, but I am a rebel, ask anyone. I once posted the full c99 standard to my web site so people wouldn't have ...

Hehe, after struggling to place the head, torso and legs of pedestrians together to make the animations work in FreeSynd for the last week or so I happened to find this information about HELE-0.ANI, HFRA-0.ANI and HSTA-0.ANI. Turns out these three files contain just about everything you need to know to draw objects in the game. There are 1970 animations which are made up of frames taken from a pool of 8949, each of which are made up of elements taken from a pool of 10486, each of which index into the available 1180 game sprites. The amount of work required to manually reproduce these animations (and that I had intended on doing) is phenomonal. A conservative estimate is that this information has knocked a year off the development time. In just one day (today) I have figured out the exact placement of units on the map, fixed the drawing of animations and cleaned up a lot of very ugly code. The result is that agents can now walk around, with and without all the different weapons ...

Breaking my own rule that improvements should be belayed until such time as the original game has been reimplemented, I've added waypoints and path finding to agent movement in FreeSynd . I found this documentation of the A* path finding algorithm to be most enlightening.. although, frankly, the sketch is more than adequate to implement a useful algorithm for agent movement. It always annoyed me when I told my agents to go to point X and they got stuck inside buildings or took some inordinately long path. Now you can hold down the ctrl key and enter waypoints manually, or you can let the path finder do its job (this is the default). Pressing the ctrl key without clicking the mouse will display the selected agents' current path(s) as a bright yellow line. At the moment the planning algorithm does not take elevated terrain into account, so trying to send agents onto bridges or up stairs simply won't work just yet. There's still a hell of a lot to do before the game ...

Back when Mike and I started the Boomerang decompiler project, we had the choice to start from scratch or to use our work on the UQBT project as a base. There was pros and cons to both approaches, but in the end we decided that UQBT had quite a lot of value in it, so we went with that. Unfortunately, at the time Mike and I started working on a decompiler based on UQBT, the copyright was owned by a number of parties. We had to talk to lawyers from the University of Queensland, Sun Microsystems and the individual people who had worked on the codebase over the years. Initially the lawyers were of a single opinion: no way. The individual contributors, however, were of mixed opinions: what for? and which license? As it turned out, answering the second question also answered the first question and started to put the lawyers minds at ease. The goal of the Boomerang decompiler project was decided, way back in 2001, to be a guiding force in bringing about a market for decompilation techn...

I've recently been working on some game related projects. First, there's the RPG engine I've started which I call Stallion . I've received just about no interested about this from anyone so I havn't been paying it much attention. Now and then I get the desire to hack on an RPG and none of the available codebases for graphical RPGs interest me as much as the ones for textual RPGs. So why not hack on a textual RPG? Because it's just so dry without players to consider, and with players to consider you've just gotta be so dedicated. Then there's my other project. I've always been a fan of Bullfrog's Syndicate. My fan page gets a lot of hits (probably because I give out copies of the game on it). If you've never played Syndicate you probably won't understand the attraction. You'll also think we're all crazy for fooling around with DosBox or old hardware to get the game to actually run and never consider doing the like yourse...

Havn't been working on Boomerang much lately. I made a few changes to make the exports of DLL files appear as entrypoints in the GUI. Seems to work, will check it in soon. I also made some recent changes that handle gotos inside switches better. This means that magic like Duff's Device now decompile sensibly.

A small number of library procedures are "called" but never actually return. Eventually I'd like to have a way to specify these procedures with anotations in the signature files, but for the moment they are hard coded in the frontend. That's acceptable for the moment as there is only five: _exit, exit, ExitProcess, abort and _assert . Thing is, what happens when you have a branch over one of these calls, as you often do. Typically you end up with phi statements that refer to the call or the statements before it because there's no way to show that these definitions are killed by the call but not defined by it. We could add this to the SSA form but a simpler solution is available. Whenever the decompiler determines that the destination of a call is one of these no return procedures then we simply drop the out edge of the containing basic block. Without an out edge the definitions cannot flow beyond the call. Using dataflow based type analysis and some of m...

Continuing with my ongoing test program extract_kyra.exe from the scummvm tools I've been looking at the very first call in main . It would appear that this exe was compiled with stack checking runtime code enabled. That very first call is to a short little procedure that takes a single parameter in the eax register; the number of bytes to subtract from esp . Here's a disassembly of the procedure: push ecx mov ecx, esp add ecx, 8 loc_405336: cmp eax, 1000h jb short loc_40534D sub ecx, 1000h or dword ptr [ecx], 0 sub eax, 1000h jmp short loc_405336 loc_40534D: sub ecx, eax or dword ptr [ecx], 0 mov eax, esp mov esp, ecx mov ecx, [eax] mov eax, [eax+4] jmp eax It not only s...

I've been promising to do this for a while, so here goes. I have downloaded a small open source utility called extract_kyra which is part of the scummvm tools . It doesn't really matter what the utility does. In fact, I prefer not to know as it gives me an unfair advantage compared to, say, decompiling malware. What's important is that this tool is under the GPL, so it is permisible for me to decompile it. I will be writing a number of these posts that describe all the issues I've run into and the steps I've completed to produce the decompilation. I promise that I will not go and download the source code for this program until such time as I believe my decompilation is "complete". The first problem with decompiling this program is that Boomerang has failed to find main() . I can tell this because the load phase of the workflow does not list main as a found entrypoint, it only lists start . This is because this exe was compiled with MinGW and it...