How did people figure out the WAD/level format for Doom?

[OpenRift]

It's sort of a thing we take for granted now with the wealth of editors and utilities at our disposal. 

 

When Doom came out in December of '93, there wasn't any form of documentation on the WAD format or Doom's level format (am I wrong?). But somehow despite having next to nothing to work with, Raphaël Quinet and Brendon Wyber managed to put together Doom's first level editor, DEU in just over a month (released January 26th, 1994). Was this some highly dedicated reverse engineering magic? Or were they perhaps given some documentation from id? Was there some now obscure BBS post from an id employee that gave them some leads on how the formats worked?

 

It's also rather interesting that the earliest documented WAD for Doom, Origwad, wasn't even made in a level editor, but was made by hand and finished on March 7 of '94. Perhaps DEU 2.00 wasn't usable enough at that point? It's hard to really gage that from a modern perspective, given how much clunkier it is compared to the editors we have today.

Edited September 18, 2023 by OpenRift

[Kappes Buur]

When opening a pwad with Slade3, the lumps in the mapnamespace will tell you the map format, for example

 

 

 

[Midway64]

I think OpenRift meant how did people figure it out rather than asking help on how to identify the level format.

[Kappes Buur]

The only reference I have is Joe Pantuso's book DOOM Game Editol, as on page 13

 

Spoiler

 

maybe that gives some insight

Edited September 19, 2023 by Kappes Buur

[Redneckerz]

On 9/18/2023 at 11:01 PM, OpenRift said:

It's sort of a thing we take for granted now with the wealth of editors and utilities at our disposal. 

 

When Doom came out in December of '93, there wasn't any form of documentation on the WAD format or Doom's level format (am I wrong?). But somehow despite having next to nothing to work with, Raphaël Quinet and Brendon Wyber managed to put together Doom's first level editor, DEU in just over a month (released January 26th, 1994). Was this some highly dedicated reverse engineering magic? Or were they perhaps given some documentation from id? Was there some now obscure BBS post from an id employee that gave them some leads on how the formats worked?

 

It's also rather interesting that the earliest documented WAD for Doom, Origwad, wasn't even made in a level editor, but was made by hand and finished on March 7 of '94. Perhaps DEU 2.00 wasn't usable enough at that point? It's hard to really gage that from a modern perspective, given how much clunkier it is compared to the editors we have today.

Recent research highlights that DEU5BETA (March 17, 1994) was the first with a node builder. ORIGWAD and CROSS.WAD were done by hand, Alistair Brown himself stating he crafted a custom program for it. We also have hints that the first custom WAD was dated as early as February, by David Simpson. Sadly, Simpson never shared that WAD, thinking it wasn't all that useful.

 

The second node routine likely is NODE_GEN.zip by Jason Hoffoss (March 30, 1994). It is a node generator first and for most.

 

I recommend reading deathzor's posts about these, and some of mines aswell:

Deathzor:

 

RED:

 

 

+ Part 3 describing HFDE, IDE, MKD2EDIT, WAD Master, WADMSTR, Laura Beyer's Level Editor (LBLC), SuperWADED, viDOOM and The Doom Editor -

 

[slowfade]

You guys should write a long, in-depth article/essay about all that history. Would be great.

[Gez]

On mardi 19 septembre 2023 at 1:35 AM, Kappes Buur said:

The only reference I have is Joe Pantuso's book DOOM Game Editor, as on page 13

Yeah, about what I expected. Open file in hex editor, try to figure it out. The .WAD format used by Doom is very simple, there's no compression or encryption, so it's a rather straightforward job to create a wad viewer. Once that's done, the next step is figuring out how each lump works. Again, there's no compression or encryption for any of them (at least in the original PC release of Doom), so you can separate a lump from the rest of the wad and look at it in a hex editor to figure out how it works.

 

Some lump types are very straightforward. Flats, for example, are just pixel dumps. The only things you need to figure out is the dimensions (64x64 in this case) and the fill order (if you get it wrong you'll get a mirrored or rotated graphic, so it's very quickly rectified). Something like PNAMES is also very simple, because of the plain strings in it which correspond to lump names, and how regular it is. TEXTURE1/2 is only a little bit harder, because each texture can have a variable number of patches, but that can be figured out pretty quickly too.

 

If we assume a regular structure, we'll have x size for each item in the list, and possibly y size for header or footer data (e.g. number of items in the list). So for example, counting things in-game didn't work, but another approach you can do is this: If we assume a regular structure, we'll have x size for each item in the list, and possibly y size for header or footer data (e.g. number of items in the list). So you take a look at each THINGS lump's size for each map, and get a list:

E1M1: 1430

E1M2: 2620

E1M3: 3800

E1M4: 2540

E1M5: 2930

E1M6: 4630

E1M7: 3580

E1M8: 1260

E1M9: 2370

The first obvious thing here is that each time, you have a multiple of 10. But the number for tens is sometimes odd, sometimes even, so it cannot be, say, a multiple something greater than 10. Now if there were header or footer data, that could change things up, but at this point it's tempting to assume there isn't and see how far you go with ten bytes per thing. Spoiler alert, you'll go quite far because it's the good answer.

So you start looking at what kind of values you get for those ten bytes. If you make a table of ten columns and fill it with the first ten or so things from each map's THINGS lump, you'll notice some patterns.

20 04 E0 F1 5A 00 01 00 07 00 
F0 03 F0 F1 5A 00 02 00 07 00 
50 04 F0 F1 5A 00 03 00 07 00 
C0 03 F0 F1 5A 00 04 00 07 00 
20 01 E0 F3 5A 00 30 00 07 00 
20 01 E0 F2 5A 00 30 00 07 00 
10 02 10 F3 5A 00 EC 07 07 00 
10 02 B0 F3 5A 00 EC 07 07 00
70 0D 70 F2 87 00 B9 0B 0F 00

So the last two columns tend to be 07 00. And before that, we have a remarkable 1, 2, 3, 4 series going on, but then we have other values after, it's not a regular count. This can be observed for every level. It's pretty easy from there to figure out that we have five different values that are two bytes each. Most things tend to have 07 00 as their last value, but there are some exceptions. The third value tends to be always one of eight possible values, with no exceptions: 00 00, 2D 00, 5A 00, 87 00, B4 00, E1 00, 0E 01, and 3B 01. Convert from little-endian hex to decimal and you get 0, 45, 90, 135, 180, 225, 270, 315. Those sure look like angles to me! And so on.

Edited September 27, 2023 by Gez