The Console Doom hacking project / Console specs

[Kaiser]

For a few years I've been on a mission to allow users to edit and manipulate console ports of Doom. Among these ports are the Jaguar, GameBoy Advance, Nintendo64 and Playstation versions.

As of now I have completely hacked all of these ports, and I am currently developing some tools to allow users to create their own maps for these console ports.

So how different are these versions? And how can they be edited? Below are some specs on how this particular port of Doom works and how you can edit the data.

--Jaguar Doom--

This is probably the most easiest of all console versions of Doom to edit with. The IWAD starts at offset 0x40000 in the rom. The pointer that tells the engine where to load the IWAD in the offset is found at 0x368f. Changing the values at this offset will effect where the IWAD is loaded at in the memory.

Jaguar Doom is completely identical to PC doom, except with the lack of music and crushers, although being on a console, Jaguar Doom is pretty darn fast.

Differences

The only major difference in the game is that most of the wad's lumps are compressed to save rom space, and the sprites and textures are loaded/stored differently.

Sprites are probably the most complexed and its data are hardcoded into the game. The data thats hardcoded contains a table that specifies how many frames that this sprite has, what lump order its in, and if its mirrored or not.

You can check out this table in sprinfo.c in the Jaguar Doom source code.

Jaguar Doom contains no patches, everything is a single texture. The textures are somewhat similar to the RAW format and its data is drawn horizontally and not vertically.

The IWAD has no major change, except all the lump positions/offsets are always in multiples of 4. This is the same for all console ports. Any lump size that are not in multiples of 4 will get extra bytes added until (size % 4 == 0). But the lump directory in the wad will still read that lump by the original size. Confused? Good.

To indicate a compressed lump, the first character of the lump name is added by 0x80. From there when that lump is loaded, the game will see that its a compressed lump and will decompress that lump. (Information about the compression algorithm is found at the end of this thread.)

Here's a screenshot of a custom map in Jaguar Doom


--Gameboy Advance Doom--

Another very easy port to edit.. sorta. GBA Doom is exactly the same as Jaguardoom, including the sprite tables, data format, everything. However, all the hud graphics and sounds/music are hard coded and is outside the wad. All the jaguardoom hud grapics etc are still stored in the IWAD.. hmm.

The IWAD starts at offset 0x942BC in the rom, and the actual pointer that tells the game where to load the IWAD at is located at 0x4E38. Again changing this value will effect where the IWAD is loaded in the memory.

All the hud graphic data is found right after the IWAD. The pointer that tells the game where to locate the main status HUD graphic is at offset 0x17090. Changing this value, of course, will effect where the graphic is loaded in the memory. I don't have the offsets/locations of the other hud graphics...

All sounds in GBA Doom are stored as signed chars and the offset of where the data begins is at 0x92A5C.
The pointer for the sounds begins at 0x1E514.

Like JaguarDoom, GBA doom uses the same confusing sprite tables and that is hard coded into the game but can be modified though...

Differences

The only major difference in GBA Doom is that it uses a special set of segs for the maps. The format is very different than the standard lump format. The structure is set out like so:

typedef struct
{
short v1;
short v2;
int xDistance;
int yDistance;
short side1, side2;
int v2_y;
int v1_y;
int v2_x;
int v1_x;
short sector1, sector2;
int lineID;
} gbaSegs_t;

All integer values are short swapped.

Unfortunately, these datas are hardcoded as well! Crap. Fortunately, you can easily modify where they are loaded in memory, so you can have them point to a custom data in your wad if you wanted to.

The beginning of the special seg data starts at 0x5DA4D8 and the table that contains these pointers for the data starts at 0x6C76D0. This pointer here is what you will need to change if you want to use a custom map using this special seg data. I have released a tool to allow users to make custom maps called GBADMAKE however its very hard to use and I am currently working on a more improved tool.. though I just don't feel like working on it...
Though interested, I'll be more than happy to assist creating custom gbadoom maps.

Here's a screenshot of Kaiser_2.wad converted to GBA Doom



--PlayStation Doom--

This is where things gets difficult. PSX Doom is among the best looking Doom console ports around and is probably the most coolest.

PSX Doom stores the main IWAD in the /abin CDROM directory. It contains all the graphic/hud data as well as additional lumps and other stuff.

All the maps are stored in the /MAPDIRXXX directory with the IWAD header included. All lumps are compressed.

Differences

There's A LOT of changes. First off, PSX Doom uses 16 bit colors instead of the standard 8 bit color mode. It still uses a palette and the sprites no long users the Jaguar/GBA table, but a different kind of table that is stored in the SPRITES1 lump in the main psxdoom IWAD. I have yet to take a look at this lump.

PSX Doom is based on a heavily modified Jaguar Doom engine, and most of the data are cached. The sounds, textures and sprites are stored in separate files that each map uses for caching. MAPSPRXX.IMG for example contains all the sprite lumps (in compressed format) that the map will use. There are 24 bit data headers before each lump that specifies the information (like lump size, pointer, offset) but I have yet to study this further. Same for MAPTEXXX.IMG. The sounds are stored as .LCD files and they can be extracted using PSOUND which you can find on the Internet.

Music in PSXDoom are ambient and spooky and is probably the best feature in PSXDoom. The music for each map are stored as .LCD files in the Music directory. Each music file contains about 2 - 3 sound samples, which those are manipulated by the .LCD file to create the music (Map02's music contains only 3 sounds! Hard to image that only these three sounds creates such a nice ambient music).

All the texture names in PSXDoom have been changed and there are also several unused textures and sprites found in the IWAD as well.

The fire sky you see in some of the maps are generated automatically. A good example of this can be found in the allergo source files (exflame.c).

One of the major features in PSXDoom is colored lighting and sound ambience. PSXDoom's sectors uses a different format than that of the PC Doom's sectors. The format is structured like so:

typedef struct {
short ceiling, floor;
char ceilTex[8];
char flrTex[8];
byte light;
byte colorID;
short type, tag;
byte flag;
byte misc;
} psxDoomSector_t;

colorID points to a specific color table in LIGHTS.lmp in the main IWAD. This sets the color value for the sector. Flag appears to effect the sound ambiance. Value 1 is used for outdoors, and 256 appears to give the sound a echo-ish effect. Not sure what misc does and what the other flags are.

The LIGHTS lump is a simple RGB table that contains 256 difference RGB values. I disassembled the LIGHTS lump and uploaded the table as a HTML file which you can check out here
http://home.houston.rr.com/svkaiser/Doom/PSXDOOM_LIGHTS.htm

Here are several values for other things:
Things:
64 Chain and Hook

Thing Flags:
32 Set Translucent??
64 Nightmare Creature
128 Specture

LineDef Flags:
0x400 Transparent1
0xC00 Transparent2

LineDef Types:
142 Play ClubDoom Music :-)
200 Scroll Left
201 Scroll Right

Sector Specials:
200 Slow Pulse
201 Unknown
202 Rapid Flicker
204 Fast Flicker

Last, in addition to the Map data lumps such as THINGS, LINEDEFS, BLOCKMAP etc, there is an additional map lump called LEAFS. More info about LEAFS is found at the end of this thread.

I've released a WIP tool called PSXMAKE which you can get here:
http://home.houston.rr.com/svkaiser/Doom/PSXMAKE.zip

Note that there are no documentation on the tool and its still being worked on but does the basic operations like converting and compressing/decompressing.

The major challenge in importing custom maps is that you have to actually create an .BIN or .ISO of your PSXDoom CD image and edit that file from there. I am also working on a tool to import custom maps into a .ISO/.BIN file, but this may take a while. If there is a free/open source .ISO/.BIN editor that allows you to import custom files please let me know.

Below are some screenshots of PsxDoom hacking




--Doom 64--

The most hardest and most difficult console port of Doom it edit with! Despite of Doom64 being a console port of Doom, it is a completely different game.

Doom64 still uses the original Doom engine, a lot of its aspects are still the same. Most of Doom2's bugs are still present in Doom64 like double closing door sounds and wall running.

The main Doom64 IWAD can be found at 0x63D10. I have yet to find the pointer that points where the IWAD should be loaded at. If anyone knows please pm me.

There's not much to say on the rom offsets, its not that of Jaguar Doom or GBA Doom, and the pointers are impossible to find.

Offset 0x5b080 contains a table of strings and music data. Changing the values will effect what music is loaded for that map and what map string will be displayed on the automap.

Offset 0x5ba84 contains the texture animation table.. nuff said.

Other information can be found here:
Thing/Frame/State info http://home.houston.rr.com/svkaiser/d64FrameInfo.txt
IWAD Info http://home.houston.rr.com/svkaiser/D64lumplist.txt

All lumps are compressed using the same algorithm that Jaguar/Psx uses. However, there is another algorithm that I have yet to figure out and this is used on the Textures, Maps and Demo lumps. It doesn't appear to be Huffman nor Deflate but it has the same ratio of that of a standard zip file.

The maps are stored as single lumps, but when decompressed, they are actual IWAD files.

'3d' objects that you see in Doom64 like bridges etc, are nothing more of a gimmick. If you lower a sky ceiling, anything behind that sector can be seen though. And that rendering trick was used to give certain areas a '3d' apperence. I have uploaded a few maps that I have converted to the PC/Zdoom format if anyone is interested. (Link below).

Another cool (but pointless) feature is that if a wall/linedef is missing a texture, it gets replaced with a texture that says 'I Suck At Making Maps' :P

Sprites and textures all contains its own palette within the lump. Converting the palettes is a pain in the arse though. Sorry none of the palettes are uploaded.. (only got 5mb of space on my ftp site >_<)

Okay.. almost though here.....

The major change is the map format. Everything except Subsectors, Segs, Nodes, Reject and Blockmap has been updated/modified.

Since I am getting lazy at typing here's the text file of the changed data structures here
http://home.houston.rr.com/svkaiser/d64Specs.txt

Note about the triggers: if you have a Linedef set as a Macro line, then you must have a macro created for it, otherwise Doom64 with crash with unknown Macro ID message.

Now about Macros, Macros contains a sequence of linedef triggers. Thats it. The first header in a macro lump contains the number of linedef types used and the number of macros. Then each macro contains a number of sequences and each sequence contains a delay, tag, and then linedef type. All sequences per macro are executed at once, so you need to specify a delay if you want certain sequences to trigger right after the one before it. Doom64 is kinda picky when it comes to macros, if you have 6 linedefs set as a Macro line, then you must have 6 macros created otherwise it crashes with that error message.

I have included Map01 (Staging Area)'s macro as a text file, which you can find here:

http://home.houston.rr.com/svkaiser/Doom/Macros_Map01.txt


Each map also has a LIGHTS lump which is basically the same thing as PSX Doom. I have included a lights chart table for the converted maps which you can find below.

Here's a link of some of the maps that I ripped from the save states.
http://home.houston.rr.com/svkaiser/Doom/d64MapsConverted.zip

I converted the maps to Zdoom/Hexen format for convinence purposes. Most of the additional linedef data is set to the zdoom line arguments so you can see what trigger types are used etc.

I am currently working on a set of tools to allow users to make doom64 maps as well as working on a modified version of Slade to support Doom64 Mapping as well. Dunno when that will be done.

Oh yeah, here's a custom test map I made for Doom64.



Things should be easy once I get the changes added to Slade.

That wraps up the Doom64 specs/info...


Information about LEAFS

PSXDoom and Doom64 both uses a Map data lump called LEAFS, which is followed after BLOCKMAP.

In these two console ports, the map's subsectors are rendered as full convex polygons, very much like GL ports, and to do this, all the edges of a subsector must be calculated.

LEAFS is basically GL_SEGS, GL_VERT, and GL_SSECT all rolled into one.

Each 'leaf' begins with the number of vertexes used, followed by each vertex and seg index. The number of leafs must be the same for the number of subsectors. There must not be any single segs within a subsector.

Doom64 however, uses 'normal' segs and subsectors. So there are actually single segs in several subjectors. Leafs however have included additional vertexes to help create a three sided (or more) subsector, so LEAFS does all the work. I don't know how the fuck you do this, instead compile a map with GL_BSP, then use PSXMAKE to generate the Leafs from the GL lumps instead.

Vertexes in Psxdoom and Doom64 are stored as 16.16 fixed values. Very much the same as GL Vertexes. Except of having GL_VERTS, Psx/N64 versions has both Vertexes/GL_VERTS as one lump.

The encoding/decoding you see in Jaguar/PSX/N64 versions uses a form of LZSS compression.
Info about this is found here:
http://michael.dipperstein.com/lzss/

Special thanks to Mike for identifying the compression method btw.

Basically the encoding works as it collects a series of previously seen bytes and then it calculates how many there are in a length and at what position where the duplicate set is found at. Pretty neat. I have also released a standalone version of my Console Doom lump compression tool which you can grab here
http://home.houston.rr.com/svkaiser/hosted/LMPZSS.zip

Beware, as it is really slow when encoding. Not useful though and its just for demonstration purposes.

Anyways that wraps up the specs for the console doom ports and as you can see, the ability to make custom maps for these ports are becoming an reality.

I am also accepting feedback, or help from those who are interested in contributing to the project and helping out. I would really like to know what algorithm they used to compress the maps/textures/demos in Doom64, but that may likely never happen.

I released a tool to import GBA Doom a while back but was utterly useless and I will be working on another tool that will be easier to use.

I pretty much got all the loose strings together, now I just need to get everything into one nice knot... or something.

PsxDoom hacking is almost done and Doom64 hacking is in the works (just need to get to work on that Slade modification). Anyways that sums up everything.

Any questions?

[Dr. Zin]

Whoa!

[Amaster]

Here is kaiser_2.wad running on a DS:
http://jasonm.youfailit.net/kaiser_gba.jpg

I would be very interested in your conversion tools so I can have pwads on the go.

[DaniJ]

Excellent work thus far Kaiser, I'm glad to hear you continued on with the project after you last spoke to me concerning this.

If we can decipher the compression algorithm used for the maps I'm very confident we'll be able to read most of the data direct from the ROM in the updated Doom64TC (currently in progress) running under Doomsday with relatively minor changes (for example, the linedef macro system can already be emulated exactly using XG's chain sequences so an algorithm to extract the macros and recreate them as XG line types at runtime is trivial).

Thanks for thing/frame/state tables (how complete/accurate are they?), I'll begin updating Doom64TC to use the same IDs.

However, being able to run custom maps in the original Doom64 via an emulator kinda makes that a little redundant but it would still be pretty cool to have.

[Kaiser]

the thing/frame/state table is 100% accurate as the data was read straight out of the rom.

Also, the unknowns you see in the thing table information are refering to the palette index and translucency.

For example, the Demon has two palettes, PALSARG0 and PALSARG1, the second palette is for the Specture demon.

In the thing definition for the Specture, Unknown1 is set as 1, which looks up PALSARG1.

The second Unknown2 value is self explainatory :P
There are certain indexes in the palette that indicates translucent pixels. The unknown2 value just sets the strength of the tranlucency.

[Kaiser]

Amaster said:

Here is kaiser_2.wad running on a DS:
http://jasonm.youfailit.net/kaiser_gba.jpg

I would be very interested in your conversion tools so I can have pwads on the go.

Nice ^_^

I still have the tools on my hard drive, but I lost the documentation a long time ago when I first posted them on DW. Though if still interested, I can pm you a simple step by step process in what to do. But you may need to do some converting on the pwad as the texture names in gbadoom are different.

[Linguica]

Out of curiosity, have you looked at GBA Doom 2? From what I can remember it was basically a remake of Doom 2 using a completely different engine built especially for GBA FPS games.

[Kaiser]

I have, and of course it is nothing like that of gbadoom 1. No iwad files, anything.

The only thing I manage to find are sound tables, which contains the name of the sound sfx and what 'pointer' they are assigned to. If I can remember correctly its kinda set out like Zdoom/Hexen's SNDINFO.. like some sfx had directories like "/sfx/monsters/impsee" etc.

I also managed to find RAW formated textures and the palette, but no sprites or maps.

I do plan on taking a closer look at it once I get the other console ports taken care of.

[Quasar]

SNES DOOM was also running an entirely new custom engine. There's nothing in the ROM that bears much resemblence to the original game code or data.

[Antidote]

Hmm, I don't remember the Imps in PSXDOOM being translucent. I have the game right here, I'm playing it right now. They are not translucent.

Also in the PSXDOOM version there was a new enemy "Nightmare Spectre" or was it Demon? I can't remember, because I don't have to booklet anymore. This baddie was green and translucent. In PSXDOOM, the Archvile is absent. There were new levels, Club Doom, Doom Mansion, some other with a Cyberdemon and a ton of Barons... too lazy to check the name. I don't remember any other differences other than the ones you and I both mentioned. I played PSXDOOM for far too long.


I'm sorry, but I know nothing of the engine, IWAD, lines/things/side or whatever, I'm not a modder.

[Kaiser]

Exactly, thats why I posted the screenshot, because I hacked the property flag for the imps to be transparent. There is also a flag that sets the 'Nightmare' mode for an enemy, so its not just the Demon that can be the Nightmare monster. And yes, the archvile is replaced by a chain and hook sprite.

[Antidote]

No, it has nothing to do with the difficulty setting... the Nightmare Spectre is a new enemy that is tougher and stronger (takes more hits from a shotgun) than the Spectre/Demon.

Really? The Archvile was changed to a hook/chain sprite? Haha that's very strange, I wonder why they would do that?

And for the Imp thing: Oh.

[Manc]

No see there's a "Nightmare mode" flag for any creature, just like there's a translucent flag for any creature.

[Antidote]

I'm am not talking about flags... I'm talking about a specific monster... after some rumaging around I found the booklet.

Here's a photo of the booklet, to show I didn't make it up.


Here's a photo of the "Nightmare Spectre" listed in the booklet as a monster/enemy.

[Kaiser]

And I am saying that the "Nightmare Demon" is just the regular Demon with a 'special property' that enables a Nightmare mode which gives that object more health and a greenish tone. This can be done for any enemy though hacking.

[Antidote]

Oh oh oh, sorry.

I wasn't reading it right, thought you were talking about the difficulty.

[LexiMax]

This is some seriously fascinating stuff. Excillent work Kaiser.

[Tormentor667]

AlexMax said:

This is some seriously fascinating stuff. Excillent work Kaiser.

This is what I though when I read this. Great work Kaiser, this is really amazing what you are doing... though, I absolutely don't understand most of it :) I am not into coding that much ;)

[machine_devil]

Damn... great stuff! ^^

I remember when in '98, I really wanted to make my own Doom levels and play them on PSX Doom.

[Doom_user]

If Doom 64 maps can now be opened in an editor, is it possible to figure out how the secret message in "BREAKDOWN" is accessed?

[Kaiser]

There is no secret message. We've been fooled.

[Koko Ricky]

This information has been quite interesting, although I'm not sure what use it'll have, but then again I don't mod.

Edit: I remember reading about this so-called secret message for awhile, who started that rumor anyway?

[Quasar]

With enough specific information, it could be possible for PC DOOM ports to directly support console DOOMs. Example: Put PSX DOOM into your CD-ROM drive and play it with a PC source port. I'm thinking about the possibility of such things for Eternity currently, especially for the simpler versions that don't require a lot of new features (Full DOOM 64 support is not likely, for example).

[Kaiser]

Jaguar Doom and Gbadoom can easily be run/emulated with a source port. Gbadoom may require a little bit more work because of the hardcoded hud graphics but still easy nothertheless.
Psxdoom may require slight tweaking and workarounds.


Speaking of which, with the source code of jaguardoom being released, I was thinking about porting Doom2 over to the Jaguar console port. Only problem is using the development kits for the jaguar system as all of its tools are runned in dos..
Though DOS-Box should do the trick..

[Doom_user]

Thing Flags:
32 Set Translucent??
64 Nightmare Creature
128 Specture

In PSX Doom, is the normal Spectre actually a different enemy, or is it just a Demon that's been given the "Set Translucent??" flag?

I take it the "Nightmare Creature" flag increases a monster's hitpoints and turns it green, but how much exactly are the hitpoints increased and does it do anything else, like increase the monster's speed or the amount of damage it inflicts?

What does the "Specture" flag do?

[Kaiser]

Specture makes it transparent.. but it won't work unless flag 32 is set.. which this must be enabled to get the other flags to work.

The Specture and Nightmare Demon is basically a regular Demon with the special flag set. So you could give a regular imp these characteristics as well.

The Nightmare flag just increases its hitpoints, but not sure by how much.

[machine_devil]

I seem to believe all that occurs with the hitpoints of the Nightmare flag is that it gets multiplied by two...

I've tried that flag on a Zombieman and I could just tell a slight difference in his hitpoints (HP from 20 to 40?).

The Imp, there was a bit of a change with his hitpoints (HP from 60 to 120?).

And with Nightmare Spectre(or Nightmare-flagged Demon), I imagine the hitpoints just go from 150 to 300.

[ShaneAmp]

Good to hear progess is still being made with this! Being able to play Custom Maps on the PSX is something ive always hoped would be possible, N64 to, but that may only be possible in an emulator.

My EPSX wont run a Doom.iso properly, and i cant burn a cd right now. Do all the textures names get converted to, or is it even possible to run a converted map, using this program yet? Sorry to ask, i would try myself only if i could, im interested to know!

[Kaiser]

While its possible to run a psxdoom map in a pc port, you need to rename all of the textures.

Also, psxdoom has different light level behaviors as a lot of light levels are below 96, while some are at normal levels, making everything look extremly dark.

The Psxmake tool will convert a psxdoom map, but does not rename the textures or fix the light values.

[ShaneAmp]

Yeah i did that with the last tool you released, extracted the first few maps and textures and put them in the wad. The lighting was all wrong, so i just set every sector to the same light level. But what i meant was running a PC map on the PSX, if a map is replaced with an ISO editor. Im guessing it would be the same situation, you would have to rename the textures to the PSX names first, and fix the lighting.