Specifications for source port demo formats?

[Xymph]

Are specifications of demo formats in source ports available somewhere?

 

The wiki (and the original LMP specs on which it's based) describe only the vanilla games. My next bot-script project is to generate tables about built-in demos like these but I soon encountered demos from ports I don't know the format of.

 

Most frequently is (Pr)Boom:

byte 0 = version 202

byte 1 = 29 (meaning what?)

bytes 2-5 = "Boom"

byte 6 = 230 (meaning?)

byte 7 = 0 (always?)

bytes 8-10 = skill, episode, map (I think)

and then? The same mode & player flags as in vanilla?

 

Also quite often used is MBF:
byte 0 = version 203 or 214

byte 1 = 29 (meaning?)

bytes 2-4 = "MBF"

byte 5 = 230 (meaning?)

byte 6-7 = 0 (always?)

bytes 8-10 = skill, episode, map (I think)

and then?

 

Tics appear to take up 4 bytes as in vanilla. Correct?

 

Rare occurrences for PWAD built-in demos are Doom Legacy/ZDoom (in Herian 2 and Phobia - The Age) and Eternity (only Doom: The Lost Episode; version 255). I guess I'll backburner/skip those.

 

So, any insights into the version 202/203/214 formats?

Edited February 25, 2021 by Xymph
Fix MBF version

[Daerik]

 

[GoneAway]

Hello. If you speak ruby, then this repo may be really helpful for you: https://github.com/kraflab/lmp

 

I think there are scattered posts with information on the forums, but iirc I did run into some misinformation while figuring things out.

 

This should walk you through the demo header and its variations between versions (it's not documentation but code, but hopefully it's relatively clear - feel free to ask questions):

 

https://github.com/kraflab/lmp/blob/4f7417fba5fe5ccb9d5c872e6ad35819169fc229/lib/lmp/vanilla/demo.rb#L37

https://github.com/kraflab/lmp/blob/4f7417fba5fe5ccb9d5c872e6ad35819169fc229/lib/lmp/doom/demo.rb#L14

https://github.com/kraflab/lmp/blob/master/lib/lmp/boom/demo.rb#L20-L52

https://github.com/kraflab/lmp/blob/4f7417fba5fe5ccb9d5c872e6ad35819169fc229/lib/lmp/mbf/demo.rb#L25-L71

 

Format detection:

https://github.com/kraflab/lmp/blob/4f7417fba5fe5ccb9d5c872e6ad35819169fc229/lib/lmp/engine.rb#L10-L18

 

Tic size is 4 bytes originally, but demos using longtics have 5 bytes (turn value is 2 bytes instead of 1).

[Xymph]

Thanks for the quick follow-ups, I guess that header format post would probably have popped up if the thought to search-before-posting would have crossed my mind. But today it didn't. %-)

Since I only need to determine demo lengths (and skill/slot) I can skip all other header bytes, and the above provides ample info to accomplish that. My ruby understanding is minimal, but code is code so it's still helpful. And I can use Boom-LMPC to verify my results.

 

Btw, analog to the header format post, perhaps an admin better move this topic into Source Ports too?

[Keyboard_Doomer]

This post from the above-linked thread has some incorrect info:

- MBF header by itself doesn't imply longtics. In fact longtics are not available for the original MBF compatibility (-complevel 11) at all. Only -complevel -1 (and 17) implies longtics. That means demos with 214 as byte 0.
- Starting with Boom the header size has been exactly 109 bytes. This is still the case for -complevel -1. There's no variable length of the header.

 

Demos that make use of UMAPINFO have 27 extra bytes before the standard header - that means the header size is 40 (27+13) bytes for vanilla complevels and 136 (27+109) bytes for Boom and beyond.

 

I imagine the versions in the byte 0 might be of interest so here they are:
0-4 - Doom <=1.2/Heretic/Hexen, byte 0 was skill at this point, version was not written to the demo. Note that since there was no range checking a wrong value for -skill could result in other values as byte 0 - those would be most prominently other values under 100 and values in the range 240-255.
104 - Doom 1.4
106 - Doom 1.666 (-complevel 1)
109 - Doom (2) 1.9/Ultimate Doom/Final Doom (-complevel 2/3/4)
        - also possibly PrBoom+ emulating Doom 1.2 (-complevel 0), DosDoom 0.47 (-complevel 5), early TASDoom (-complevel 6), Boom emulating vanilla (-complevel 7) or theoretically other ancient source port with non-perfect vanilla compatibility
110 - TASDoom (-complevel 6)
111 - same as 109 + -longtics
200 - Boom 2.00 (NOT -complevel 7, which is actually Boom's emulated vanilla compatibility)
201 - Boom 2.01 (-complevel 8)
202 - Boom 2.02 (-complevel 9)
203 - depends on the signature:
        -> "Boom" - LxDoom
        -> "MBF" - MBF (-complevel 11)
210-213 - old PrBoom complevels (-complevel 13-16)
214 - current PrBoom+ compatibility (-complevel -1 / -complevel 17), always uses longtics
255 - depends on the signature that starts from the next byte:
        -> "ETERN" - Eternity,
        -> "PR+UM" - PrBoom+ with UMAPINFO, check byte 27 (i.e. 28th byte) for the "true" compatibility
        -> neither - most likely Doom <=1.2/Heretic/Hexen demo that was accidentally recorded with "-skill 0"

 

...and there's more for demos that are not compatible with PrBoom+

 

I have spent some time investigating this sort of stuff, including for demos not compatibile with PrBoom+, so feel free to ping me if you get stuck on something and I'll see if I will be able to help.
 

Edited February 25, 2021 by Keyboard_Doomer

[Xymph]

13 hours ago, Keyboard_Doomer said:

I imagine the versions in the byte 0 might be of interest so here they are:

In the PWADs covered on the wiki, I encountered one more: 107 (v1.7) in Earth and NJ Doom 2.

 

I've implemented 202/203/214 in my script but encountered what appears to be a general inaccuracy in the Boom-LMPC: for versions 203 it logs tic data numbered 1 through XXXX, but in the header overview presents "Length of demo:" with "(XXXX-1 gametics)". E.g. Claustrophobia 1024 DEMO1 there are 3017 tics logged (my script confirms the same number) but Length states 3016. Same with several other demos.

 

With v202 it appears Boom-LMPC assumes a header larger than 109 bytes, as the tic log starts later in the file and the count is lower. E.g. Vile Flesh DEMO1 is reported as 3730 long, but I think it's 3757 (27 more); and for DEMO3 5666 (tool) vs. 5684 (script) or 18 more.

Oddly, DEMO2 here uses v106, which the tool shows having 1851/1850 tics, and my script reports 1850 -- so here the tool appears to log 1 tic too many.

 

With v214 and the -l flag, Boom-LMPC still counts tics instead of longtics. E.g. Disjunction DEMO1 5297 (logged) or 5296 (Length), but the tic log shows impossible values for the first three bytes. My script reports 4268 longtics.

 

Are these known issues with that tool, or am I missing / misinterpreting something?

Without verification by another tool I am less certain my own scripting is correct, but counting offsets in hex dumps confirms all of my scripting results. So I'd like to understand what's going on, or know that I can safely (but unfortunately) disregard Boom-LMPC. Its source code has never been released?

[Xymph]

I added 255/Eternity support too, because after a look in its source code it was quite easy (the header would be 98 bytes, if I'm not mistaken).

But I'm puzzled with the results for DEMO4 in the aforementioned Doom: The Lost Episode: 8813 regular tics with 1 player should result in 4:11.80 (251.80 seconds) playing time, but after installing Eternity and watching with stopwatch in hand, it runs for about 2:06, or exactly half that time. Btw, the player dies after about 42 seconds already, then the death view just idles on.

Could anyone explain the factor two difference in running time?

Edited February 25, 2021 by Xymph

[Quasar]

6 hours ago, Xymph said:

I added 255/Eternity support too, because after a look in its source code it was quite easy (the header would be 98 bytes, if I'm not mistaken).

But I'm puzzled with the results for DEMO4 in the aforementioned Doom: The Lost Episode: 8813 regular tics with 1 player should result in 4:11.80 (251.80 seconds) playing time, but after installing Eternity and watching with stopwatch in hand, it runs for about 2:06, or exactly half that time. Btw, the player dies after about 42 seconds already, then the death view just idles on.

Could anyone explain the factor two difference in running time?

Sounds like it is desync'd to me.

[Edward850]

If it runs at exactly half the time, could it have been recorded with longtics but with no compatibility header?

Edited February 26, 2021 by Edward850

[dew]

47 minutes ago, Quasar said:

Sounds like it is desync'd to me.

The desync is obvious, perhaps just playing an ancient EE demo on a more modern executable. But longtics shouldn't double the demo duration, that's why I pinged you guys heh. Could it be, I dunno, mouselook information?

[Keyboard_Doomer]

Didn't check thoroughly but your observations about Boom-LMPC 2.0 seem to be correct. The reason it seems to have these issues is simply that, as he says in the Doomworld thread for the release, Opulent put only some limited time into the development.

 

I assume for the Eternity demo you go with 4 bytes per tic. However, the demo was recorded with Eternity 3.35 which used 8 bytes per tic. Note that current version of Eternity stores 14 bytes per tic. Reference:

https://github.com/team-eternity/eternity/blob/6612b6fdd99a1f2848772732c7a5db64218f436f/source/g_game.cpp#L1592

Edited February 26, 2021 by Keyboard_Doomer

[Xymph]

8 hours ago, Quasar said:

Sounds like it is desync'd to me.

No, I originally tried it in EE v4.latest, but a run in v3.35.90 that the demo was recorded in, also ends after 2:06.

 

7 hours ago, Keyboard_Doomer said:

I assume for the Eternity demo you go with 4 bytes per tic. However, the demo was recorded with Eternity 3.35 which used 8 bytes per tic. Note that current version of Eternity stores 14 bytes per tic. Reference:

Ah, that's it, I didn't fully understand all the version checks at first but now they make sense. Table added. Thanks.

 

Re. Boom-LMPC, then I'll go with what makes sense in my own script and with verification in hex dumps. I also added the tics column to the wiki tables for reference.

[Xymph]

A month of (on and off) researching and coding has led to the LmpStats library, which also represents my current knowledge-in-code about demo formats. No doubt there is plenty more to learn...

Edited March 27, 2021 by Xymph

[Xymph]

On 2/25/2021 at 1:37 AM, Keyboard_Doomer said:

203 - depends on the signature:
        -> "Boom" - LxDoom
        -> "MBF" - MBF (-complevel 11)

According to DoomStruct, byte 204 for MBF can be inserted into this list. And during LmpStats development I found that CDoom used bytes 205-207.

[Keyboard_Doomer]

That would be the 2.04 version released a few years ago. Unfortunately, not supported in PrBoom+. I made a reply in the release thread now because somehow I had missed it originally D: