Special Lump names: WTF?

[Maes]

While trying to add some "flats and sprites" in PWADs support to Mocha Doom, I stumbled upon Michael "Optimus" Kargas Doomonstration games, not quite the typical PWAD but one that is "almost" vanilla compatible (and thus would be interesting to get to work), and with limit-stretching effects to match, which quite unexpectedly (?) had extra sprite + flat lumps.

OK, so exactly what's the spec for those?

Vanilla has a couple of F_START and F_END markers which definitively mark the end of the flat storage, but there can apparently be nested Fx_START and Fx_END markers inside, where x=1 and 2.

However, I noticed that dmnsgame.wad had a "FF_START" and "F_END" markers to match. So what exactly is the standard here? Are FF_START and F_END considered standard in Boom/ZDoom? Are there any other schemes?

Also, when scanning for multiple such lumps, how are limit-removing source ports supposed to work?

E.g. I should coalesce them so that their picnums are contiguous? E.g. let's say I have 100 flats in the IWAD, with picnums 0-99 and 100 more in a PWAD, that don't clash with my original ones (I know that vanilla would override the internal ones completely in this case). I should merge them in a 0-199 fashion?

In case of name clashes/overriding, I should give priority/redirect to PWAD lumps?

Fun fact: due to the more strict way that Mocha Doom reads lumps, I can't perform Doom's cheap trick of starting to read actual data from the marker lump's position without some extra hacking: the reader will use the stated lumpinfo length property when asked to read a specific structure from disk and will allocate...well, zero bytes for zero-length lumps, so I have to find the first non-marker, non-zero lump after that (the opposite process is applied to end markers, where I have to locate the last nonzero lump before the marker).

[Graf Zahl]

Maes said:

Vanilla has a couple of F_START and F_END markers which definitively mark the end of the flat storage, but there can apparently be nested Fx_START and Fx_END markers inside, where x=1 and 2.

However, I noticed that dmnsgame.wad had a "FF_START" and "F_END" markers to match. So what exactly is the standard here? Are FF_START and F_END considered standard in Boom/ZDoom? Are there any other schemes?

F1_START etc. are ignored. They have no meaning at all. In fact, if you replaced them with valid flats you can use them as textures in the game.

FF_START etc. were introduced by Deutex and Boom treated them as equivalents for F_START etc. so it could load WADs made for merging by Deutex. So your support of these depends on the compatibility you are after.

Also, when scanning for multiple such lumps, how are limit-removing source ports supposed to work?

Depends on your data organization. If you keep Doom's index based management you need to coalesce these resources.

ZDoom doesn't do this though because it never uses lumps directly as textures. It creates a texture object for each such lump and adds that to the texture manager - which does not require strict sorting of textures by type.

In case of name clashes/overriding, I should give priority/redirect to PWAD lumps?

As always: Who comes last, wins.

[Gez]

Maes said:

Vanilla has a couple of F_START and F_END markers which definitively mark the end of the flat storage, but there can apparently be nested Fx_START and Fx_END markers inside, where x=1 and 2.

However, I noticed that dmnsgame.wad had a "FF_START" and "F_END" markers to match. So what exactly is the standard here? Are FF_START and F_END considered standard in Boom/ZDoom? Are there any other schemes?

The trick here is that Doom will see the lumps this way:

1. All the lumps from doom(2).wad (playpal, colormap, endoom, etc. up to F_END)
2. All the lumps from the pwads

It considers them as a single sequential blob, rather than as separate files. When it looks for lumps, it starts from the end. And now that's where it gets clever: all the lumps that are between the (last) F_END and the (last) F_START are flat. Since the PWAD contains an F_END but no F_START, that means that the flat range goes from the IWAD to the PWAD. And that's how you can add new flats easily.

The FF_START marker is just for tools so that they can actually get a real start marker in the pwad. Doom will not treat it as a special lump.

If only Doom didn't check for validity of sprite names, it would have been possible to use the same trick to add new sprites as well.

Final note: although you'll find P_START and P_END markers, the engine does not actually use them. So you can add patches easily without having to fuss around with markers. Of course, you have to monkey with PNAMES instead.

[Graf Zahl]

Gez said:

Of course, you have to monkey with PNAMES instead.

... and all that mess to save a few kilobytes of lump data to be loaded on game startup that never remains in memory...

[Ladna]

PrBoom+ has this little function:

static int IsMarker(const char *marker, const char *name)
{
  return !strncasecmp(name, marker, 8) ||
    // doubled first character test for single-character prefixes only
    // FF_* is valid alias for F_*, but HI_* should not allow HHI_*
    (marker[1] == '_' && *name == *marker && !strncasecmp(name+1, marker, 7));
}

So you can kind of tell how things should go from that.

[Maes]

Gez said:

When it looks for lumps, it starts from the end.

There's a small duality to take into account here: the built-in searching function. Whether I'm using a hashtable or the old backwards search, in case of name clashes between PWADs and IWAD both search methods will give priority to the former, so the only way to regain access to the "original" lump (including the original F_START and F_END) would be to do an exhaustive index-based search rather than using W_GetNumForName().

Gez said:

And now that's where it gets clever: all the lumps that are between the (last) F_END and the (last) F_START are flat. Since the PWAD contains an F_END but no F_START, that means that the flat range goes from the IWAD to the PWAD. And that's how you can add new flats easily.

Now, wait a minute. I know that in the case of pre-merged PWADs that totally replace ALL flats used by Doom with their own, including both F_START and F_END markers, this approach works OK.

But If I have only the original F_START and only the PWAD's F_END, then Doom will try to load EVERY lump in between those two as a flat, and that will include any non-floor patch lump as well (it will ignore the original F_END). Right?

E.g. if in the IWAD F_START and F_END are lumpindexes 1000 and 1100 accordingly, then there are other 800 non-related lumps and I add a PWAD that has JUST 100 flats between FF_START and F_END, then by your sayings I will have a range of "flats" going from lumpindexes 1000 to 2000. Obviously F_END is not supposed to be used without a matching F_START, unless the flats are the very last lumps in the IWAD.

[4mer]

That's just the way it is

[fraggle]

Maes said:

Now, wait a minute. I know that in the case of pre-merged PWADs that totally replace ALL flats used by Doom with their own, including both F_START and F_END markers, this approach works OK.

But If I have only the original F_START and only the PWAD's F_END, then Doom will try to load EVERY lump in between those two as a flat, and that will include any non-floor patch lump as well (it will ignore the original F_END). Right?

E.g. if in the IWAD F_START and F_END are lumpindexes 1000 and 1100 accordingly, then there are other 800 non-related lumps and I add a PWAD that has JUST 100 flats between FF_START and F_END, then by your sayings I will have a range of "flats" going from lumpindexes 1000 to 2000. Obviously F_END is not supposed to be used without a matching F_START, unless the flats are the very last lumps in the IWAD.

Yeah, this is the classic clever hack that is used to add custom flats in PWADs. That's why it's FF_START..F_END instead of FF_START..FF_END. Lumps are only allowed to be used as flats if they're between the F_START..F_END tags - fortunately Doom doesn't care if you include other lumps in there that aren't really flats at all.

Unfortunately the same trick doesn't work for sprites because the startup code does some preprocessing, so any lumps that aren't proper sprites cause the game to fall over.

[kb1]

Maes said:

But If I have only the original F_START and only the PWAD's F_END, then Doom will try to load EVERY lump in between those two as a flat, and that will include any non-floor patch lump as well (it will ignore the original F_END). Right?

Yep. But this doesn't really hurt anything (for flats), unless you try to render one of those lumps as a flat. As mentioned above, however, sprites are a different story, as the engine will proceed to check all the lumps, and crash when it hits one of the "inbetween" lumps.
I strongly recommend taking a look at (and implementing) Boom's W_CoalesceMarkedResource, which really takes the headache out of worrying about load order and multiple blocks. It reorders the lump list and makes access to the lumps appear contiguous - especially useful for sprites.

[Maes]

kb1 said:

Yep. But this doesn't really hurt anything (for flats), unless you try to render one of those lumps as a flat.

*smacks forehead* Right... even though even bogus lumps will be assigned sequential picnums by the engine, the actual matching of flat names (that appear in sectors) to internal picnums will (hopefully) skip over any bogus ones, even though that probably means wasting some entries in the translation table (and any other auxiliary structures) for bogus lumps.

It should be possible to do a "clean parse" and by scanning all F_ and FF_ lumps sequentially (without W_GetNumForName) and determine the proper course of action (merge/append/override) as well as allocating just the right number of picnums.

I will take a look at Boom's implementation at this point, though a re-implementation is not out of the question (especially since Java has much nicer built-in tools for doing such operations).

[kb1]

Maes said:

I will take a look at Boom's implementation at this point, though a re-implementation is not out of the question (especially since Java has much nicer built-in tools for doing such operations).

Yeah, it does exactly what you're looking for, I think. And, it's pretty nice - you can add new namespaces easily (?_START/?_END) without too much hassle. (I was looking to do my own implementation too, but dropped the Boom code in instead, and haven't looked back.)

Boom really does deserve some credit in that regard - many of the Boom enhancements really serve to generalize what was once shaky, get-it-out-the-door code. In a lot of cases, whole functions can be swapped out without negative side-effects.

[Quasar]

kb1 said:

Boom really does deserve some credit in that regard - many of the Boom enhancements really serve to generalize what was once shaky, get-it-out-the-door code. In a lot of cases, whole functions can be swapped out without negative side-effects.

Only to a degree, though, unfortunately. For every wad hash and limit removal there's a d_deh.c or a "Killoughized" function that nobody can understand except the guy who wrote it >_>

Some of the implementation details were short-sighted as well, even if they seemed awesome at the time. Gen specs and sector bits especially fit into this category.

[kb1]

Yeah, it's a mixed bag, that's for sure. Still, a valiant attempt, and still the closest thing we have to a "base" version to modify from, unfortunately.