Conflicts with DEHACKED replacing the strings in ZDoom

[Graf Zahl]

I just saw this report on the ZDoom forum:

 

https://forum.zdoom.org/viewtopic.php?f=2&t=56837

 

In short, because DEHACKED is treated as a retroactive alteration of the string table, the lump in the Freedoom IWADs will override any native string table extension in any PWAD. I am not sure how other advanced ports deal with it, but I'd guess it's the same there - if they have a native string definition system it may not be able to deal with a DEHACKED lump in an IWAD.

 

So two questions:

 

1. Which other ports are affected by this problem?

2. What can be done about it? It's not trivial to fix on the engine side.

 

[Rachael]

Being a bit of an outsider with regards to Freedoom development, I've always found myself wondering if Freedoom really needed those strings. I get the whole effort to gear towards uniqueness and stuff, but I think the DEHACKED thing should be restricted to using actual Dehacked on the vanilla executable, and let the source ports import the strings themselves.

 

This should be done, obviously, after they've been finalized because it would be very difficult to change them later on after everyone's brought in their own copies. But I still am a bit iffy on whether the string changes are even needed.

Edited June 10, 2017 by Rachael

[MrFlibble]

A while ago I reported the same issue for MBF 2.04, and Gerwin added a command line parameter to ignore embedded DEH/BEX lumps. However this only partially fixes the problem as the port will not load PWAD-specific DEH/BEX lumps if they're not supplied as separate files. I think I might have also posted about the same issue in the PrBoom+ thread but not sure. UPD: yep, here it is.

 

As far as I can tell, the only true workaround is to have dedicated executables with Freed∞m-specific strings already in.

[Voros]

From what I know, Freedoom isn't targeted at the vanilla executable, but rather Chocolate Doom. The DEHACKED lump in Freedoom is a really just a BEX file, and is supported by Chocolate Doom (using "*allow-extended-strings*" comment to allow Chocolate to parse the file). For ZDoom based ports, they can simply use the finalised strings and make it a part of their port, but not all ports are ZDoom based.

 

One of reasons for not using actual Dehacked (IIRC), was because the lump would contain the original strings from Doom, which can cause legal issues (INAL). That and it's plain ugly.

 

IMO, this bug should be dismissed for now. By the time Freedoom does reach v1.0, its upto the developers to include Freedoom's strings, background flats, PAR times, etc in their ports. By then, the DEHACKED might be removed, if Chocolate Doom also includes them in the exe.

 

Of course, that's just my opinion.

[Graf Zahl]

3 hours ago, Voros said:

IMO, this bug should be dismissed for now. By the time Freedoom does reach v1.0, its upto the developers to include Freedoom's strings, background flats, PAR times, etc in their ports. By then, the DEHACKED might be removed, if Chocolate Doom also includes them in the exe.

 

Of course, that's just my opinion.

That's really not how this should be done. Even putting port specific definition lumps into the IWAD would be better (e.g. LANGUAGE for ZDoom and whatever else is needed for Eternity or 3DGE.)

[Rachael]

I think both Eternity and 3DGE can use the same set of DDF files for that purpose. But I don't think such functionality is available in a lot of other ports - what would you do, for example, for PrBoom+, Chocolate Doom, Retro/Crispy/etc/whatever other fork of Chocolate there is?

[Gez]

IMHO it's not different from when you play a wad with the "wrong" IWAD, such as TNT.wad instead of doom2.wad. The level name strings will not be what's expected. Basically, it's fine.

[Graf Zahl]

If not for the very simple issue that Freedoom is supposed to be usable as a stand-in for the commercial IWADs...

 

[Gez]

And? Beside the cosmetic issue of a few strings not being what was expected, it works fine, doesn't it? A demo would remain in sync regardless, wouldn't it?

 

Other cosmetic things not being what was expected include textures, sprites, sounds.

[chungy]

It is non-trivial, but I believe the correct thing to do is respect a user-loaded (or PWAD-loaded) DeHackEd file based on the Doom strings and treat them with priority over the Freedoom lump. It is what Chocolate Doom does presently (I believe PrBoom-Plus too).

 

On 6/10/2017 at 0:31 AM, Rachael said:

Being a bit of an outsider with regards to Freedoom development, I've always found myself wondering if Freedoom really needed those strings. I get the whole effort to gear towards uniqueness and stuff, but I think the DEHACKED thing should be restricted to using actual Dehacked on the vanilla executable, and let the source ports import the strings themselves.

 

This should be done, obviously, after they've been finalized because it would be very difficult to change them later on after everyone's brought in their own copies. But I still am a bit iffy on whether the string changes are even needed.

The point of including the DEHACKED is to basically remove a possible trademark-infringing case. Bethesda lawyers might look at Freedoom and see BFG9000s and Cacodemons and such and believe our project to be infringing. The technicality of it being inside the engine source code may not have any bearing against such a case. Fraggle even received an email from John Carmack once requesting that all Doom names be removed.
 

On 6/10/2017 at 5:42 AM, Voros said:

IMO, this bug should be dismissed for now. By the time Freedoom does reach v1.0, its upto the developers to include Freedoom's strings, background flats, PAR times, etc in their ports. By then, the DEHACKED might be removed, if Chocolate Doom also includes them in the exe.

No, we will never expect source ports to include Freedoom strings in the ports. It's a maintainability nightmare and just stupid on many levels. I would rather remove the special strings than ever ask such a thing.

[Graf Zahl]

This is completely missing the issue, though.

 

ZDoom allows changing strings with its own native definition method. Dehacked will always override them because it cannot be parsed intermittently as Dehacked contains lots of other data besides strings.

 

So what happens is:

 

PWAD defines strings per LANGUAGE.

IWAD defines strings per DEHACKED.

 

IWAD wins because DEHACKED is parsed later.

I guess the same will happen with any other port having a similar dual setup.

 

[chungy]

Thank you, I misunderstood the problem.

 

Is there a good reason for DEHACKED getting priority? Can it be reversed or does this create compatibility issues?

[Graf Zahl]

It's the native lump which defines the string labels and their original content. This gets parsed as one of the first things because other parts of the engine need the list of labels. Dehacked only gets parsed long afterward when everything else it may modify is also fully initialized. Another problem here is that ZDoom is language aware which imposes additional restrictions on how Dehacked string replacements are being handled vs. natively defined ones.

 

Also let's not forget that this most likely not only applies to ZDoom but to any port which externalizes its string table to a definition lump that is user-modifiable directly.

 

[Rachael]

I think the best solution to ZDoom is to include a directive that gets ignored (a pseudo-"comment" like how QuickBasic did it) by other ports, and forces the use of the language lump, instead. When ZDoom detects such a meta-command, it will cease processing on that DEHACKED file and let the mod's own LANGUAGE lumps take over.

 

(For reference, here's how QuickBasic did it: https://en.wikibooks.org/wiki/QBasic/Appendix#INCLUDE_.28QUICKbasic_Only.29

 

By default, any line that starts with an apostrophe is a comment. That works in backwards with GWBASIC and prior versions of Basic. However, if a $ dollar sign is appended to the apostrophe, it creates a meta command that only QuickBasic understands.)

Edited July 17, 2017 by Rachael

[chungy]

I hadn't thought of that. There is already a special comment string for Chocolate Doom to load/parse the file automatically. It would be great to tell ZDoom-based ports the proper idea of how to treat it too in a similar fashion.

[Gez]

There's already a way to force ZDoom to ignore any DEHACKED processing, it's to include a lump named DEHSUPP.

[Graf Zahl]

Yes, but keep in mind that this may cause problems with ZDaemon.

 

[chungy]

ZDaemon still exists?

[Graf Zahl]

Well, it exists. We just ignore it... :D

 

[chungy]

I'll continue to ignore ZDaemon if you don't mind. :)

 

I've been thinking about a response with regard to DEHSUPP, and while it might work around the issue, I don't think it actually solves anything. If a port simply does not support DeHackEd, or does not support the DEHACKED lump, the engine falls back to the Doom strings and life moves on. However, to explicitly disable loading of the DEHACKED lump goes against the intention of including it in the first place, even if ZDoom is the only port that respects a DEHSUPP lump.

 

I think the real problem is: Why is a lump in an IWAD overriding anything set in a PWAD? That seems to be backwards from the entire idea of "Internal WAD" and "Patch WAD". Nothing in the IWAD should ever take precedence over a PWAD, the entire point of PWADs is to modify the game.

 

Perhaps ZDoom simply didn't take into account that an IWAD could have a DEHACKED lump (it is quite unusual, but both Freedoom and HacX do it), but that really ought to be fixed in my opinion.

[Rachael]

I think that's much easier said than done, with how DEHACKED actually works. ZDoom has to process all of its internal stuff first in order for DEHACKED to be even functional - since all the actors are external now, not having them loaded beforehand makes them impossible to modify. Naturally, the LANGUAGE lump is loaded around the same time too - which is, before the IWAD is even checked for, since it kind of makes sense to have all the pre-IWAD script processing done in order to properly present the IWAD selection box as well as give basic diagnostics about the system it is being run on (in the user's own language, if possible...).

 

To me, it seems like DEHACKED more or less suits its original purpose in ZDoom - it's simply a crude hacking tool, its operations are added in at the last minute for game processing after the resources are already loaded - which creates the problem we presently see now.

 

This is why I suggested the //$commented meta-command killswitch - a special code that only GZDoom recognizes that shuts down DEHACKED processing and forces it to use the LANGUAGE lump, instead. It might be a bit unpleasant, but it's effective.

[Quasar]

This is not a problem for EE as far as I'm aware. The native default strings are stored in the EXE. DeHackEd applies after EDF (which defines almost everything else that DeHackEd can edit) at startup, and DeHackEd can even be processed at runtime.

[Graf Zahl]

 

20 hours ago, Rachael said:

I think that's much easier said than done, with how DEHACKED actually works. ZDoom has to process all of its internal stuff first in order for DEHACKED to be even functional - since all the actors are external now, not having them loaded beforehand makes them impossible to modify. Naturally, the LANGUAGE lump is loaded around the same time too - which is, before the IWAD is even checked for, since it kind of makes sense to have all the pre-IWAD script processing done in order to properly present the IWAD selection box as well as give basic diagnostics about the system it is being run on (in the user's own language, if possible...).

Actually, no. Language is loaded after the IWAD. It has to because there's $ifgame directives in there that wouldn't work otherwise.

 

20 hours ago, chungy said:

I think the real problem is: Why is a lump in an IWAD overriding anything set in a PWAD? That seems to be backwards from the entire idea of "Internal WAD" and "Patch WAD". Nothing in the IWAD should ever take precedence over a PWAD, the entire point of PWADs is to modify the game

It's because Dehacked is not a native definition method. In addition it mixes very different things that go into very different parts of the engine which makes it impossible to parse intermittently with LANGUAGE lumps.

 

All this becomes even more complicated by supporting multiple languages. When the language changes all LANGUAGE lumps will be reparsed and the Dehacked strings added on top again.

[Voros]

How about including some code that recognises Freedoom's "# *allow-extended-strings*" comment and thus forces it it load up first alongside the IWAD?

[Graf Zahl]

It cannot be loaded up first. The entire system is not designed like that. Dehacked string replacements are an entirely different layer over LANGUAGE defined strings.

 

The best you can expect is an engine flag set in IWADINFO that forces it to ignore the DEHACKED lump inside.

Sorry, but this was designed 10+ years ago when custom IWADs weren't a thing and DEHACKED lumps inside IWADs even less. There's just no simple way to retroactively allow PWAD LANGUAGE lumps override IWAD DEHACKED lumps.

 

[Jon]

Fundamentally this seems like a (hard) ZDoom bug and should be fixed in ZDoom.

[Graf Zahl]

The only way to "fix" this would be to render LANGUAGE inoperable. Not an option.

 

[Voros]

Where exactly in the code is the execution of loading LANGUAGE and DeHacked lumps at?

[Gez]

Meh I don't even consider that a real bug anyway. Just a consequence of Freedoom wanting to be Doom II without being Doom II.

[Graf Zahl]

51 minutes ago, Voros said:

Where exactly in the code is the execution of loading LANGUAGE and DeHacked lumps at?

LANGUAGE in stringtable.cpp, DEHACKED in d_dehacked.cpp.