Introducing ZokumBSP

[dew]

Zokum, aka Kim Roar Foldoy Hauge and a member of the Norwegian AV mafia, has been working on a vanilla-optimized fork of Zennode called ZokumBSP recently. His main goal is to allow vanilla maps to grow bigger by optimizing and shrinking the blockmap size. His work is starting to bear fruits, the numbers shown in his impressively detailed AV case study are nothing short of amazing. I've discussed this with essel already and without unnecessary fanfares, we should switch to this nodebuilder for E2/E3.

Motivation:
After consulting proper optimization flags with zokum I've ran some tests on some of our most critical cases, Speedtraps and Unstable Journey.
Speedtraps: 65508B (zennode), 53960B (zokum o=2), 53842B (zokum o=3)
Journey: 65440B (zennode), 55336B (zokum o=2), 55246B (zokum o=3)

This leaves quite a lot of extra room for our most glutonous boys and girls to add care & love to their megamaps. Tarnsman is already giggling with sadistic glee when he thinks about the stuff he will do to e3map30.

To use the nodebuilder from DoomBuilder, just copy zokumbsp.exe to .\Compilers\Nodebuilders and create a config file, let's say zokumbsp.cfg, then select it in Game Configurations.

compilers
{
	zokumbsp
	{
		interface = "NodesCompiler";
		program = "zokumbsp.exe";
	}
}

nodebuilders
{
	zokumbsp_dev
	{
		title = "zokumBSP - Dev Optimized";
		compiler = "zokumbsp";
		parameters = "%FI -bso=2 %FI -n2 -nu -r- -t";
	}
  
        zokumbsp_release
	{
		title = "zokumBSP - Release Optimized";
		compiler = "zokumbsp";
		parameters = "%FI -bso=3 %FI -n2 -nu -r- -t";
	}

	zokumbsp_fast
	{
		title = "zokumBSP - Fast (no reject)";
		compiler = "zokumbsp";
		parameters = "-n3 -nq -rz %FI -o %FI";
	}
}

A word of warning: optimization level 3 is extremely time-expensive. Speedtraps or Journey take over 4 minutes to build on an i7 processor, so save that for the final release if you really really really need those extra 100 bytes, heh. Even level 2 takes me about 15 seconds, so consider knocking it to just -bso=0 for development purposes.

Zokum also offers manual optimization using linedef tag 999, which will set the nodebuilder to ignore these lines and reduce blockmap size. This is very interesting for e.g. ceiling detail, but also rather experimental and it requires caution, not to mention potential Zdoom incompatabilities, so please ignore this feature for now. :)

---
edit: This thread was moved from the BtSX section, so pardon some of the inhouse phrasing. The issues encountered seem quite important on the bigger scale, so public access for port devs seemed like a good idea.

[Mechadon]

Damn, too bad we didn't have something like this when I was actually working on Unstable Journey :P. This is super awesome though! I dunno if I'll be making any major additions or changes to Unstable Journey anytime soon (unless you guys have any cool ideas), but this will no doubt be useful in the future.

[esselfortium]

Good post :) This will hopefully be a very useful tool going forwards!

I would recommend adding -n2 -nu to those configs, as it's what BTSX currently uses with ZenNode to reduce seg splits.

[dew]

Updated the optimization flags. If anything's messed up or improvable, just edit it directly and mention it down here.

[esselfortium]

Mechadon said:

Damn, too bad we didn't have something like this when I was actually working on Unstable Journey :P. This is super awesome though! I dunno if I'll be making any major additions or changes to Unstable Journey anytime soon (unless you guys have any cool ideas), but this will no doubt be useful in the future.

Much like how Traversd nearly gave dew and I a heart attack with his map12 joke in the e1 builds thread, I think the testers would all greatly prefer to avoid seeing any more major changes to Unstable Journey :)

[Mechadon]

Haha, yea I imagine so! Don't worry, I don't think I'll be doing anything to that map apart from bug fixes at this point ;)

[Vorpal]

Fair warning, while I'm excited for zokumbsp's achievement I'm not sure the whirld is ready for it. I chucked some of my vanilla maps through it, great results, ~15% savings on blockmap as advertised. However every non-vanilla port I tried had a total breakdown:

Do we need to harass the source port community, or did I just do something dumb? Or does zokum need to fix something? Nodebuilding and code is all voodoo to me so I'm just flailing here

[esselfortium]

That's weird... I've passed it along to zokum on IRC.

[esselfortium]

@Vorpal: Which settings are you using to build that map?

[Vorpal]

heuristic blockmap offset configuration and default nodes/reject:

-bo=3 -n -r

[dew]

Alright, can confirm the weirdness on the e1map07 build I made with zokumbps. My flags were:

-bso=2 -n2 -nu -r- -t

I only noticed in Eternity after revenants started jumping off one ledge, heh. Then I noticed door edges behaving strangely, but it was nowhere near the utter brokenness of your video. Geezy is broken as well, but pr+ and choco work properly.

Unrelated: choco enforces lump count limit now. While I applaud fraggle's precision, this literally only serves to screw with our development. :)
Unrelated 2: Eternity doesn't emulate the all-knowing revenant in map07, heh.Bleh, was running it without the -vanilla param.

[esselfortium]

Hrmmmm. Well, that's really disappointing. While I hope to see this fixed in ZDoom and EE so that projects in the future can make use of it, this makes me very hesitant to use it for BTSX unless it can somehow be worked around on Zokum's end, to behave properly on current port versions. Considering the ZDoom-derived ports that are affected (Zandro, etc.) I imagine it'll probably be a while before a fix is actually widely distributed enough to avoid causing problems for players.

[Linguica]

What's causing it to screw up? Is this a linguortal situation where an "optimization" breaks vanilla behavior?

[dew]

Geezy messes up no matter what optimization flags I use. E.g., none at all. Interestingly enough, different spots (less of them?) were broken with o=0. Faithful ports work fine.

I'll just rebuild Metal Mothers with zennode, E1 doesn't need to/shouldn't experiment anyway. We'll see what zokum figures out, it's not like E2 is getting released a week after E1, heh.

[fraggle]

dew said:

Unrelated: choco enforces lump count limit now. While I applaud fraggle's precision, this literally only serves to screw with our development. :)

Maybe you can get chocorenderlimits updated to disable the limit or replace it with a warning message.

Failing that, all the old versions are still available for download.

[dew]

Yeah, this is something CRL could give a warning about without aborting the game. For my case I just used the public E1 release along the test map, that one's still under the lump limit.

[Tarnsman]

Vorpal said:

Fair warning, while I'm excited for zokumbsp's achievement I'm not sure the whirld is ready for it. I chucked some of my vanilla maps through it, great results, ~15% savings on blockmap as advertised. However every non-vanilla port I tried had a total breakdown:

This seems like a feature not a bug.

[Quasar]

Can we get more specific than "messes up" with what's going on in advanced ports? That doesn't really give me any ideas on what the problem is.

[Linguica]

this bit from MBF.txt is worrisome:

* Blockmap
+ Blockmap generation algorithm rewritten (Note: this is not TeamTNT's)
+ Improved dynamic blockmap list maintenance (more robust and faster)

A Lee-ism broke linguortals, so I would be 0% surprised if it did the same to advanced blockmap tricks as well.

[dew]

Linguica said:

this bit from MBF.txt is worrisome:
A Lee-ism broke linguortals, so I would be 0% surprised if it did the same to advanced blockmap tricks as well.

Verified for working: vanilla, choco, prboom-plus cl2, eternity -vanilla
Verified for broken: prboom-plus cl9, eternity in default, geezy & queezy, zdaemon


Well, looks like you were right. Eternity performs flawlessly with the -vanilla param and blows up when using the default, which I assume uses Boom optimizations.
The skeleton that ambushes me in the second run is not from the ledge batch, it's the all-knowing skeleton you can hear waking up in the next room at start. That's also faithful to vanilla!

For test purposes: the map built with zennode and with zokumbsp. Run it with some public version of e1 (e.g. the latest), those are guaranteed to run on choco without hitting the lump limit.

[printz]

I just added linguortal support in Eternity, using the same logic as in PrBoom+. Do maps built in ZokumBSP work with the very latest DRDteam devbuilds of EE now?

EDIT: Just tried the "zok" wad with and without the linguortal fix. I didn't readily notice anything wrong. What am I supposed to see as a bug?

EDIT: OK, I see some half-blocking walls... Regardless how I try it, with linguortals or not, the walls still are problematic. But using -vanilla still fixes it...

[Linguica]

The blockmap compression stuff has nothing to do with linguortals, beyond the fact that both exploit the file format in a way that was unintended by the developer but nevertheless a "legal" construct.

[printz]

I just checked: P_CreateBlockMap (the function designated to create blockmap from scratch if the saved one is bad) is not called when loading that map. So the Boomism/MBFism may be elsewhere, probably in a more critical spot. Until it's fixed, this seems like something really bad overall.

[printz]

Double posting because it's "news": as we know, each linedef list per map block begins with a 0 in the file (see wiki). In theory that 0 is a starter with no actual meaning; in practice vanilla Doom reads it anyway and assumes it's linedef #0 from the map, thus each map block has a useless reference to linedef 0 (which is not going to affect gameplay, because that linedef usually is far away from the map block). A Boomism introduced in ports, not necessarily by Lee Killough, is the following:

   // original was reading delimiting 0 as linedef 0 -- phares
   offset = *(blockmap + offset);
   list = blockmaplump + offset;

   // killough 1/31/98: for compatibility we need to use the old method.
   // Most demos go out of sync, and maybe other problems happen, if we
   // don't consider linedef 0. For safety this should be qualified.

   // killough 2/22/98: demo_compatibility check
   // skip 0 starting delimiter -- phares
   if(!demo_compatibility)
      list++;     

So what can we do? Delete this fix in the name of vanilla tricks compatibility?

It's possible there are other differences, too, I didn't check every test case with this.

[Quasar]

printz said:

Double posting because it's "news": as we know, each linedef list per map block begins with a 0 in the file (see wiki). In theory that 0 is a starter with no actual meaning; in practice vanilla Doom reads it anyway and assumes it's linedef #0 from the map, thus each map block has a useless reference to linedef 0 (which is not going to affect gameplay, because that linedef usually is far away from the map block). A Boomism introduced in ports, not necessarily by Lee Killough, is the following:

So what can we do? Delete this fix in the name of vanilla tricks compatibility?

It's possible there are other differences, too, I didn't check every test case with this.

I'd suggest this behavior needs to be maintained by the new builder, as it's actually part of the format. Failing that, only skip the "0 delimiter" if it's ACTUALLY zero? Seems logical. Will need yet more demo comp checks however.

[scifista42]

Quasar said:

only skip the "0 delimiter" if it's ACTUALLY zero? Seems logical.

What if the delimiter is 0, but the map involves a trick that depends on linedef 0 being checked in that block?

[printz]

scifista42 said:

What if the delimiter is 0, but the map involves a trick that depends on linedef 0 being checked in that block?

I suppose this is a bilateral discussion. Zokum may update his tool to work for most ports.

[Graf Zahl]

You'll never get all engines to update to such a change. If those 0-delimiters are not kept as they were this node builder will cause endless chaos for maps using it.

Also don't forget that some users also stick with older versions of certain ports for various reasons.

Either this works correctly or it will be considered broken - and 'correctly' here means that it shouldn't needlessly break older engine versions.

[Linguica]

If it makes a map that works correctly in doom.exe/doom2.exe then by definition it works correctly as a Doom node builder

[Graf Zahl]

That's a rather narrowminded and ignorant view, considering that the number of people playing with other engines is magnitudes higher, including older versions - for example, I have no idea how many still use GZDoom 1.8.6, which was the last version that ran on OpenGL 1.5. That engine version won't receive an update, I could neither add a check to handle this kind of blockmap nor an override to ignore it. Correct or not, there's many users out there using one or the other old version of any engine that's out there and this needs to be taken under consideration as well.

If you don't want this node builder to end up a failure it shouldn't require specially fixed versions of any current source port. These ports exist and the change that causes the problems now has been in the wild for longer than Doom2.exe has been the only game in town, it's not that it's only one specific version of one specific port that has problems, but virtually ALL of them.