Boom / MBF demo header format

[Never_Again]

For vanilla demo specs we have UDS. For Boom and MBF ones there don't appear to be any publicly available docs*, and not everybody can read source code. Even those who can probably don't have it all neatly organized in a table in their heads, I imagine. So here's the first step towards rectifying this situation. Ideally, this should end up as a nice detailed Wiki article. Or maybe two or more, as it would be nice to have an in-depth comparison of technical differences between various complevels.

Here are the results of my amateur research based on empirical testing and supplemented with dipping into Prboom-plus' codebase (the main source being g_game.c). I'm confident that entryway, Quasar, RjY and kb1 would spot any accuracies and fill the gaps the quickest, but contributions from anyone who has delved into the depths of demo (as well as net, savegame, etc.) compatibility are welcome.


General observations

Boom demo header is 109 bytes long and incorporates all of the fields from the post-v1.2 format. It appears to actually use only bytes 0x00 through 0x1A (except for byte 0x11, which appears to be unused by Boom and its derivatives and is explicitly set to zero by *G_WriteOptions), as well as bytes 0x4D-0x50 (players present). The rest is presumably space for future expansion and consists of zeroes.

MBF demo header is in the same format, just replaces the Boom signature with its own. It takes advantage of the unused bytes in the Boom header (starting with 0x1B) to add new fields. Complevels 8 and higher use this format, and Prboom-plus continued to add new fields within this framework.

Nevertheless, the two demo formats are not mutually interchangeable, at least as far as demo playback is concerned.
First, MBF-format demos appear to be using different format for the data section (I suspect it's longtics by default), as merely changing the Boom signature of a -complevel 9 demo to that of MBF and vice versa leads to an almost instant crash with an unknown savegame error.
Second, some unused zero-value bytes in the Boom header have the value of "1" in the MBF header, even when that actually reproduces Boom behavior, e.g. the above-mentioned 0x1B byte (monster_infighting).

I organized the data portion of my findings into three lists, posted separately below for your examination.

* - yes, I'm aware of Mbflmpc. It's old and of very limited functionality - doesn't support Boom demos, drag-n-drop, CMD and doesn't read any of the newer fields.

edit: typo

[Never_Again]

Boom v2.02 demo header

0x00		Version = 0xCA (202)
0x01-0x06	Boom signature (0x1D B O O M 0xE6)
0x07		Compatibility (0 or 1)
0x08		Skill level
0x09		Episode #
0x0A		Level #
0x0B		Play mode: 0 = Single/coop, 1 = DM, 2 = AltDeath
0x0C		Console player: 0 = 1st, 1 = 2nd, etc.
0x0D		monsters_remember = 1
0x0E		Variable friction = 1
0x0F		weapon_recoil = 0
0x10		Allow pushers = 1
0x11		unused? (default = 0)
0x12		player_bobbing = 1
0x13		-respawn
0x14		-fast
0x15		-nomonsters
0x16		Demo insurance (0 or 1)
0x17-0x1A	Random seed
[...]
0x4D		player 0 is in game
0x4E		player 1 is in game
0x4F		player 2 is in game
0x50		player 3 is in game

[Never_Again]

MBF demo header

0x00		Version
0x01-0x06	MBF signature (0x1D M B F 0xE6 0x00)
0x07		Compatibility
0x08		Skill level
0x09		Episode #
0x0A		Level #
0x0B		Play mode: 0 = Single/coop, 1 = DM, 2 = AltDeath
0x0C		Console player: 0 = 1st, 1 = 2nd, etc.
0x0D		monsters_remember
0x0E		Variable friction
0x0F		weapon_recoil
0x10		Allow pushers
0x11		unused? (default = 0)
0x12		player_bobbing
0x13		-respawn
0x14		-fast
0x15		-nomonsters
0x16		Demo insurance
0x17-0x1A	Random seed
0x1B		monster_infighting
0x1C		player_helpers (# of dogs)
0x1D		Classic BFG
0x1E		Beta emulation
0x1F-0x20	friend_distance (default = 128 / 0x00 0x80)
0x21		monster_backing
0x22		monster_avoid_hazards
0x23		monster_friction
0x24		help_friends
0x25		dog_jumping
0x26		Climb Steep Stairs (AKA monkeys)
0x27		comp_telefrag
0x28		comp_dropoff (0 = live monsters can be knocked off from high places)
0x29		comp_vile (ghosts)
0x2A		comp_pain (21 Lost Souls limit)
0x2B		comp_skull (Lost Souls stuck behind walls)
0x2C		comp_blazing (fast doors make double closing sound)
0x2D		comp_doorlight (0 = tagged doors trigger special lighting)
0x2E		comp_model (vanilla linedef trigger model)
0x2F		comp_god (0 = god mode is absolute)
0x30		comp_falloff (0 = torque)
0x31		comp_floors (0 = use Boom's floor motion behavior)
0x32		comp_skymap (0 = sky is affected by invul)
0x33		comp_pursuit (monsters don't give up pursuit of targets)
0x34		comp_doorstuck (monsters get stuck on doortracks)
0x35		comp_staylift (0 = monsters don't randomly walk off moving lifts)
0x36		comp_zombie (dead players can exit levels)
0x37		comp_stairs (1 = build stairs like in vanilla)
0x38		comp_infcheat (0 = cheats have no time limit)
0x39		comp_zerotags (0 = special linedefs don't trigger zero-tagged sectors)
0x3A		comp_moveblock (1 = allow keygrabs and Manc shots go thru walls)
0x3B		comp_respawn - objects not on map at game start respawn at (0,0))
0x3C		comp_sound (1 = vanilla quirks in sound code)
0x3D		comp_666 (1 = emulate pre-Ultimate BossDeath behavior)
0x3E		comp_soul (1 = Lost Souls bounce off flat surfaces)
0x3F		comp_maskedanim (1 = middle textures on 2-sided lines do not animate)
0x40		comp_ouchface (1 = ouch face like in vanilla)
0x41		comp_maxhealth (1 = MaxHealth in DEH only applies to health potions)
0x42		comp_translucency (1 = disables Boom translucency)
[...]
0x4D		player 0 is in game
0x4E		player 1 is in game
0x4F		player 2 is in game
0x50		player 3 is in game

edit: added default to friend_distance
edit2: shortened longest line
edit3: corrected comp_falloff's offset, added missing comp_floors, confirmed comp_respawn

[Never_Again]

Bonus content:

Compatibility options in Prboom-plus,
the corresponding bytes in the MBF demo header
and their aliases in the source code

Options -> Setup -> DOOM Compatibility

01 - Any monster can telefrag on MAP30 			- 0x27	comp_telefrag
02 - Some objects never hang over tall ledges		- 0x28	comp_dropoff
03 - Objects don't fall under their own weight		- 0x30	comp_falloff
04 - Monsters randomly walk off of moving lifts		- 0x35	comp_staylift
05 - Monsters get stuck on doortracks			- 0x34	comp_doorstuck
06 - Monsters don't give up pursuit of targets		- 0x33	comp_pursuit
07 - Arch-Vile resurrects invincible ghosts		- 0x29	comp_vile
08 - Pain Elementals limited to 21 lost souls		- 0x2A	comp_pain
09 - Lost souls get stuck behind walls			- 0x2B	comp_skull
10 - Blazing doors make double closing sounds		- 0x2C	comp_blazing
11 - Tagged doors don't trigger special lighting	- 0x2D	comp_doorlight
12 - God mode isn't absolute				- 0x2F	comp_god
13 - Powerup cheats are not infinite duration		- 0x38	comp_infcheat
14 - Dead players can exit levels			- 0x36	comp_zombie
15 - Sky is unaffected by invulnerability		- 0x32	comp_skymap
16 - Use exactly Doom's stairbuilding method		- 0x37	comp_stairs
17 - Use exactly Doom's floor motion behavior		- 0x31	comp_floors
18 - Use exactly Doom's movement clipping code		- 0x3A	comp_moveblock
19 - Use exactly Doom's linedef trigger model		- 0x2E	comp_model
20 - Linedef effects work with sector tag = 0		- 0x39	comp_zerotags
21 - Emulate pre-Ultimate BossDeath behaviour		- 0x3D	comp_666
22 - Lost souls don't bounce off flat surfaces		- 0x3E	comp_soul
23 - 2S middle textures do not animate			- 0x3F	comp_maskedanim
24 - Retain quirks in Doom's sound code			- 0x3C	comp_sound
25 - Use Doom's buggy "Ouch" face code			- 0x40	comp_ouchface
26 - Max Health in DEH applies only to potions		- 0x41	comp_maxhealth
27 - No predefined translucency for some things		- 0x42	comp_translucency

Options -> Setup -> Weapons

28 - Weapon recoil					- 0x0F	weapon_recoil
29 - View bobbing					- 0x12	player_bobbing

Options -> Setup -> Enemies

30 - Monster Infighting When Provoked			- 0x1B	monster_infighting
31 - Remember Previous Enemy				- 0x0D	monsters_remember
32 - Monster Backing Out				- 0x21	monster_backing
33 - Climb Steep Stairs					- 0x26	monkeys
34 - Intelligently Avoid Hazards			- 0x22	monster_avoid_hazards
35 - Affected by Friction				- 0x23	monster_friction
36 - Rescue Dying Friends				- 0x24	help_friends
37 - Number Of Single-Player Helper Dogs		- 0x1C	player_helpers
38 - Distance Friends Stay Away				- 0x20	friend_distance
39 - Allow dogs to jump down				- 0x25	dog_jumping

corrected comp_falloff' offset

[dew]

Thanks, this is great stuff.

[kb1]

Yes, this is good stuff. Are you making something like a universal demo viewer, or just trying to document the various demo formats? I believe PrBoom-Plus has added a demo footer as well, which you might want to look into. I think it holds information about which wads are loaded, which is all kinds of helpful when trying to play back demos made with complex wad combinations.

[Never_Again]

kb1 said:

Are you making something like a universal demo viewer, or just trying to document the various demo formats?

My interest in the subject by sparked by this 2011 post by vdgg and the thread he started a few months later:

vanilla vs. boom compatibility feature list (complevel 2 and 9)

That was posted here in Source Ports but except for a short comment by Quasar elicited no response from programmers. For shame.


What I'd also like to see is an updated version of LMPC that would keep its CMD-line and LMP->TXT->LMP functionality and add these:
· a nice GUI
· support for Boom and MBF formats
· ability to parse and edit demo footer
· direct editing of all header fields
· one-click conversion between vanilla formats

Opulent did some work on Boom-LMPC not so long ago. It's too early in development to speak of any big improvement; so far it doesn't choke on the demo footer and reads a few basic fields from the header, fewer than either LMPC or Mbflmpc. Doesn't read correctly the data section of MBF-format demos even with the "-l" switch, only of Boom v2.02 demos. edit: actually it does, although not in a straightforward manner.

I am not a programmer; entryway is too lazy busy for trifles like these; RjY doesn't do Windoze. That leaves Quasar and you, possibly fraggle, axdoom1 and Linguica as the men for the job. ;)
The main point is: if a guy who can barely write "Hello World!" without consulting a book every few seconds has figured this much on his own, surely any of you with years of DOOM programming behind you could fill in the blanks and write a new demo util in a day. All the info is there, either in this thread or in Prboom-plus' sources.

kb1 said:

I believe PrBoom-Plus has added a demo footer as well, which you might want to look into. I think it holds information about which wads are loaded, which is all kinds of helpful when trying to play back demos made with complex wad combinations.

Tell me about it. ;)

Demo footer is simplicity itself: a WAD file tacked onto the end of a demo. The basic entries are:
· VERSION
· PORTNAME
· CMDLINE

Optional:
· MLOOK (can really bloat the file)
· COMMENT*

The CMDLINE may include the following parameters:
· -iwad
· -file
· -deh
· -solo-net (coop with one player only)
· -emulate (for older Prboom(-plus) versions emulation)
· -spechit
· -set overrun_spechit_emulate
· -set overrun_intercept_emulate
· -set overrun_reject_emulate
· -complevel (for vanilla-format demos)
· -nomonsters (for -complevel 0 demos)
· -respawn (for -complevel 0 demos)
· -fast (for -complevel 0 demos**)

Technical details are in r_demo.c.
For anyone with any experience with programming WAD file utils writing a demo footer parser/editor should be a cakewalk.


* - COMMENT can be created and edited with SLADE3, Prboom-plus seems to ignore it
** - which is wrong, BTW: v1.2 didn't have -fast, it was not introduced until v1.4

[myk]

Never_Again said:
What I'd also like to see is an updated version of LMPC that would keep its CMD-line and LMP->TXT->LMP functionality and add these:
· a nice GUI
· support for Boom and MBF formats
· ability to parse and edit demo footer
· direct editing of all header fields
· one-click conversion between vanilla formats

Also -longtics (v1.11, aka v1.91) support! It would simply read the raw input data differently, with the usual vanilla header format.

[Never_Again]

Of course. MBF support implies "-longtics", I think. Currently no demo tool can read longtics demos properly.

edit: after going through Boom-LMPC's readme I got it to parse a v2.5.1.4 Prboom-plus "-complevel -1" demo* correctly. You need to use "-l 1" as well "-d x".

A further complication: turns out that the size of the MBF header of Prboom-plus demos is not constant. Mot just between versions, either.
- Opulent sites 153 bytes for v2.5.1.3.
- By comparing the offset-corrected output of Boom-LMPC to the hex data of the demo linked above I arrived at 114 bytes for v2.5.1.4.
- Creating a test demo with "-complevel 17" and same Prb+ version produced two results:
· no gameplay data (exit as soon as the screen starts to melt) - 109 bytes (Boom v2.02 standard)
· playing for a few seconds - 159
· played another WAD for a few secs - got 164. Ran same demo through Boom-LMPC and "-f 164", got garbage. "-f 165" and suddenly everything is alright.
WTF?

Additionally, there doesn't seem to be a way to turn off longtics for a "-complevel 17" demo. Certainly "-shorttics" has no effect.

* - d2twid-fda-map04.lmp from here


edit2: to whoever moved this from Source Ports do Speed Demos - not a good idea. All the programmers read that sub-forum, few ever post to the demo one. Besides, this thread is about programming, not speedrunning.

[ClumsyDoomer]

Wow. LMPC doesn't work for me since I've bought a new computer and installed Windows 7 x64 OS (Boom-lmpc works fine on it, but it can't compile demo files). I'll be really happy when someone will finally release an improved version of LMPC.

[RjY]

Wow indeed, this must have taken some time to compile! Nice job.

Never_Again said:

MBF demo header[/b]

[...]
0x2F		comp_god (0 = god mode is absolute)
0x30		? (default = 0)
0x31		comp_falloff (0 = torque)
[...]

I am convinced this must be a mistake. There can be no gaps in the comp_options array. Internally they are numbered sequentially (defined by an enum in doomstat.h) - comp_god is 8 and comp_falloff is 9. Their values are stored in an array indexed by that enum, and written out in a loop, one after the other. comp_falloff is at 0x30, comp_floors at 0x31, etc.

... Wait, hold up, I finally see what's happened. comp_floors is missing. The rest of the offsets seem correct. comp_maskedanim is certainly at 0x3f. comp_skymap is at 0x32. You just need to move comp_falloff back to 0x30 and insert comp_floors at 0x31.

That's a relief, I've spent half the evening worrying "how the hell can the compatibility array have a hole in it??". Nothing else appears incorrect at first glance.

On LMPC (slightly off topic, hence behind the cut)

Spoiler

I've never used LMPC. I don't seem to need to look at individual demo tics very often. I usually end up just using a hex dumper or similar. For example, a demo which starts off with some strafe50:

% od -An -td1 -vw4 -j13 test.lmp
    0    0    0    0
   50  -50    0    0
   50  -50    0    0
   50  -50    0    0
[...]

Skipping a Boom/MBF demo header is -j109 instead of -j13 (conveniently, 109 is an easy demo-related number to remember), longtics requires -w5 instead of 4. You can grep/awk for patterns, etc.

[Never_Again]

Done, thank you, RjY. That confirms 0x3B to be comp_respawn, incidentally. Funny how I had comp_floors in the third list with the correct number already.

What do you think about 0x11?

[Opulent]

The evolved source ports(prboom, mbf, prboom-plus, eternity, etc...) make this task cumbersome. LMPC had the luxury of a single fixed demofile structure. The various "standard" ports are each a seperate game in themselves.... then there is the compatibility modes and emulation of other ports and emulation of emulation. You are now talking about at least 25 different demofile formats. This is one of the reasons why Boom-LMPC is clunky... the input data is flexible instead of rigid.

I'd have to look over the source again, but I'm thinking someone smarter than me would be needed to interpret the demofile header flag settings to determine the correct last byte of the demofile header.
In hindsight, Chi Hoang or TeamTNT should have established a new demo format with an end-of-header 0x80 byte in the early days of creating a source-port executable with different logic behaviors than the real game. Although this wouldn't be version109 compatible, it would have only been necessary in non-realgame-emulated demos(and thus the engine could handle them).

As it is, there are some loops of logic in the code to figure out the compatibility mode and "game" version of the demo. The size of the header is static for Boom, DOOM, or MBF, but fluctuates for prboom modes. (eg. If a demo was recorded by prboom-plus-2.5.1.3, the header could be 13 bytes, or 153, or 140, etc... and it might have longtics enabled or not and it may or may not have a footer.)
I agree that it isn't that complex; as it only needs to determine the end-of-header byte to process very nearly all of the demofile data accurately.
Prboom-plus determines the options from the version byte, then determines the possible options from there. I have yet to make sense of the serial memcpy parts as controlled by the prboom compatibility flag.
Hmm... that gives me an idea of how to solve this...

[Vermil]

Opulent said:

In hindsight, Chi Hoang or TeamTNT should have established a new demo format with an end-of-header 0x80 byte in the early days of creating a source-port executable with different logic behaviors than the real game.

The only new demo format in Doom port history that I am aware of, was created by Doomsday in its early days, but it was ignored by basically everyone, perhaps because it couldn't play back Vanilla Doom demo's.

It’s kind of a shame that it didn’t gain traction for future demo usage though, as it was a lot better integrated than any other port has integrated demo’s; to my knowledge Dday is the only port to support in engine demo recording and playback to this day (via console commands). All other ports still require the use of the command line.

Perhaps due to how little it was used, Risen3D removed all demo functionality and Dday has since disabled it due to its ongoing rewrite.

[DaniJ]

Vermil said:

The only new demo format in Doom port history that I am aware of, was created by Doomsday in its early days, but it was ignored by basically everyone, perhaps because it couldn't play back Vanilla Doom demo's.

The Doomsday demo format is radically different to the vanilla and Boom demo formats. Doomsday's is built on top of a serialized stream of networked game deltas. Supporting the Doomsday demo format in other ports would be a considerable amount of work and cannot be folded into support for vanilla/Boom format. So as most ports still use vanilla's model of syncing networked games it makes sense to continue using the vanilla(-extended) demo format too.

Risen 3D has no multiplayer support so they likely removed demos along with it (uses the same code for interpreting game deltas).

[RjY]

Never_Again said:

What do you think about 0x11?

You are correct to have labelled it as unused, I think. It is ignored on read and set identically to zero on write in boom202, mbf, and pr+. I assume it's some vestigial remnant of a long-gone Boom feature.

In fact I will take a wild guess that it was once the setting named "over_under". It seems Boom temporarily had code for full 3d thing clipping, that is to say the opposite of infinitely tall actors. The feature is mentioned in the changelog at the end of g_game.c and in log_rsp.txt, as having been buggy, experimental, targeted at Phase II, and eventually removed.

[kb1]

I've been wanting to build something like this for a long time. Unfortunately, it is low on the priority list. I'm still struggling to get my sync debug code published and documented (which will shed some light onto demo formats, in an unrelated way).

[Opulent]

Hey, this actually does part of one of the things you asked for!

http://www.doomworld.com/vb/doom-speed-demos/72105-boom-lmpc-2-0/

[Quasar]

Well the deal for me is that basically we already document this, in the source code. I'm pretty sure every write done to the demo header by EE has a comment next to it stating what it does. You probably don't even need to understand C at a high level in order to grasp it, so, that's why I've not felt particularly motivated to go copypaste that stuff here so far.

[Maes]

Has anyone been able to determine whether "v1.11" (or "Doom Classic for Doom 3: BFG Edition") actually has a functional demo recording/playback engine?

The last time I glimpsed at its SC, it appears to use longtics internally (by using a short for storing angleturn), and can also play back traditional v1.9 demos, however its ability to actually record new demos is dubious.

For that matter, I don't know if anyone has been able to get it to use PWADs (without IWAD merging), let alone record demos with it.