Scripts

1What a script is

A script file starts life as text, a .q: SCRIPT … ENDSCRIPT blocks, plus data such as tables of tricks, menus and cheats. Neversoft's compiler turns that into a .qb: a stream of one-byte tokens with their operands. Names are never stored as text. Every GotParam or Skater becomes a CRC32 of its lowercased name, which is why the engine can compare names with one integer compare. A THUG1 .qb ends with a table that maps those checksums back to names, for debugging. That table is the only reason a .qb can be decompiled with its names.

The game keeps all of its scripts, 232 of them, in one archive: Data/pre/qb.pre. A .pre holds files like a zip, LZSS compressed. THUG2 uses the same format under the name .prx. American Wasteland changed everything: its .qb is a list of typed definitions, script bodies are compressed on their own, the names live in separate .dbg files, and the archives are a .pak plus a .pab.

The port runs the retail qb.pre, not the .q files in its own source tree. They don't quite match. So a fix starts from what the game actually runs.

2The round trip

Four commands, in both directions. Each stage shows its time for the whole game, on an Apple M4:

1. qb.pre232 scripts, LZSS, 1.3 MB
2. 01unpackqb unpackLZSS decoded, folders kept21 ms
3. 02decompile*.qb → *.qnames from the checksum table75 ms
4. 03edit*.qplain text, any editoryou
5. 04compile-target thug1same bytes as retail50 ms
6. 05packqb packorder and compression as retail182 ms

1. qb.pak + .pab268 scripts, 2.4 MB
2. 01unpack+ .pakinfoentry order, types, flags16 ms
3. 02decompile-names dbg.pak139,692 names from .dbg files113 ms
4. 03edit*.qsame language, three pragmasyou
5. 04compile-target thawsections, LZSS, script CRC168 ms
6. 05pack.pak + .pabidentical archive8 ms

The port itself doesn't repack qb.pre. It ships patch files that the engine loads on top of the retail scripts. Its build still compiles those with the C++ compiler; qb's output for them is the same.

3Unpack: a ring buffer full of spaces

A .pre is a header (size, version 0xabcd0003, file count) and then the files, each with its size, compressed size, name and the name's checksum. The compression is LZSS, Haruhiko Okumura's from 1989: a 4 KB ring buffer that starts out full of spaces, eight flag bits per byte, and a match is a 12 bit offset plus a 4 bit length. A file of mostly { } and tabs compresses to about a third of its size.

Packing it back so it matches retail took two rules nobody wrote down. The order isn't alphabetical: gamemenu.qb comes before game_progress_menu.qb. That's Windows' directory order, uppercased (M sorts before _), with files before folders. And not every file is compressed: three small scripts were stored as is even though LZSS would have saved them a few bytes. Comparing all 182 of THUG's .pre archives showed the cut-off: compressed only if that saves at least 15%. Files compressed to exactly 85.0% are compressed.

The padding between files in retail archives isn't zeros. It's whatever was in Neversoft's memory in 2003. That part doesn't round trip.

4Decompile: tokens back to text

Decompiling walks the tokens, prints each one, and indents by IF, BEGIN and brackets. Names come from the checksum table at the end. Two things need more than that. RANDOM blocks are stored as a weight table plus jump offsets, so the decompiler follows the jumps to find where each RANDOMCASE starts and where it ends. And a name that happens to be a keyword, like a parameter called OR, is written as %"OR", so it doesn't compile back as a keyword. An unknown checksum is written as is, %0x1a2b3c4d%, and still compiles back to the same four bytes.

5Compile: a lexer with Flex's manners

The original compiler, QScript, used a Flex lexer. qb's lexer is hand-written but follows Flex's rules: at every position, the longest match wins, and the earlier rule wins a tie. That's how RANDOM_RANGE stays one keyword and IFX stays a name. It also copies Flex's oddest result: /* a **/ b */ is one comment, because the regular expression Neversoft wrote lets **/ continue it.

THUG2 added short jumps. An IF carries the 2 byte distance to its ELSE or ENDIF, so the game can skip a whole block without reading it. The compiler leaves a hole and fills it in when it gets there:

The C++ compiler had a bug that only shows up outside English. Its \x escape parsed the hex digits into a local variable, then added the variable to itself instead of to the string. "r\xE9f\xE9rez", French for refer, compiled to "rrez". Nine scripts in the THUG source tree have strings like that. qb compiles them the way retail has them.

6American Wasteland: a different animal

THAW (2005) kept the language and changed the file. A .qb is now a list of typed sections: integers, floats, strings, vectors, structs, arrays and scripts, each tagged with the checksum of its source file's path on the build machine (f:\projects\aspyr\thaw\skate7\data\…: Aspyr made the PC version, and THAW was the series' seventh game). Each script's bytecode is compressed separately and stored with a CRC. Working out what that CRC covered took a few guesses. A script that does nothing has the CRC 0xffffffff, the CRC of nothing, so line ends had to be skipped. After that, all 5,215 scripts in qb.pak matched:

The scripts also gained one token, 0x4A: a struct of constants, stored in binary form right inside the bytecode: ready to use, not tokens to read. The compiler uses it for a constant { } on one line after a command, after RETURN, or as a script's default parameters. In qb.pak that rule matches 6,148 structs out of 6,209. The other 61 were left as tokens for reasons the bytecode doesn't keep, so the decompiler marks them /*tokens*/, and the compiler respects the mark:

The archives were the easy part, once the padding gave itself away. Every file starts on a 32 byte boundary, the gaps are filled with 0xCD, and every archive ends with the same 32 bytes, AB AB AB AB, twelve CDs, sixteen ABs. Those are the Visual C++ debug heap's fill patterns, written to disk by the tool that built the archives. They're the same in every archive, so they rebuild exactly.

7Things found in the scripts

A decompiled game reads like the team's notes. A few favourites:

The loop police

The first file in qb.pre is aaaaloopchecker.qb, named so it sorts first. The engine watches for scripts that loop forever, and this file tells it which functions count as waiting, plus three scripts it should leave alone:

Developer builds

startup.qb still knows the game was once run off a devkit and not a disc. Off the disc, it hands out the Neversoft skaters, starts the viewer, and opens the script debugger. One developer had a startup script of their own:

Two years later, American Wasteland's msongy.qb does the same thing. The constant struct after Change is one of those 0x4A tokens: the whole { All_Levels_Unlocked = 1 } is 24 bytes of binary struct inside the script.

One in six

The pro skating in the skate shop congratulates you sometimes, not always. QScript can't say "one in six", so the script says it with six equally weighted cases, five of them empty:

A rail you couldn't leave

Dropping off a rail (GrindRelease) only works on the PS2 in retail, gated by a script, not the engine. On PC the rail just held on. The port's fix is a patch that sets it on every platform, starting from the decompiled retail script:

Scripts inside data

American Wasteland's create-a-skater tables carry tiny scripts as values, 218 of them, each with the source line numbers it was compiled from. //#line keeps those numbers, so the file compiles back exactly:

8First real job

Options › About has twinkling sparkles. Three of them were square, black ones, because their texture, se_dt_glitter01, is made for particles: it has no alpha at all (936 of its 1,024 pixels are opaque black), which doesn't matter when it's blended additively, since black adds nothing. Drawn as a plain sprite, it's a black square. The fix is three stars fewer, compiled with qb, which produced the same 7,700 bytes as the C++ compiler:

9Benchmarks

An Apple M4, macOS 27, Go 1.26.1 against QScript built with Apple clang 21 at -O3. Each figure is a median, on one thread. In process, the same files are loaded into memory and converted in a loop: Go's benchmarks for qb, a small harness linked against QScript's libraries for the C++ side.

The first Go version compiled at 8 MB/s, ten times slower than C++: its lexer tried some 70 regular expressions at every position. Matching names and keywords by hand got it to 1.4× slower. Then a profile put 80% of the time in madvise, the runtime handing memory back, because the token list was allocated far too big. With that fixed and the remaining rules written by hand, it's twice as fast as the C++ compiler.

A Go program takes longer to start than a C++ one, so one run per file loses. One run for the whole folder wins by 6 to 10 times.

10Checked against retail

"Byte for byte" is the only test that doesn't need the game running. It's also the strictest.

Source

qb is open source, with its README, the tests and the benchmarks: github.com/epiqm/qb.