DEHACKED replacement proposal (with PoC code)

[kgsws]

Well, technically this is not a replacement as this can coexist with DEHACKED.

 

Anyway. I am attempting to propose DEHACKED replacement that would allow greater flexibility without adding extra complexity to the code.

I have already created proof-of-concept code for crispy-doom (if you want to compile this you have to use cmake).

Attached zip file contains a demo WAD that can be run with modified crispy doom.

 

Here's the video if you can't compile crispy doom yourself:

 

 

Now about new specification - everything is up for discussion, even name DOOMHACK can be changed.

• new lump, called DOOMHACK
  • it is cumulative, you can have multiple DOOMHACK lumps even across multiple WADs without any collisions
  • it contains tree-like structure encoded using CBOR
  • allows creation of new ammo things and weapons
  • allows modification of existing ammo things and weapons
  • allows addition of new sounds
  • allows addition of new sprites
  • it contains 5 main sections
    • sounds
      • array of strings
      • each string is sound lump name
      • DS prefix is included and currently required by the code
      • duplicate sounds are not added
    • ammos
      • object containing one or more objects
      • each key is new ammo type name
      • allows access to ammo properties, including backpack maximum
      • up to 16 unique ammo types (4 are built-in)
    • weapons
      • object containing one or more objects
      • each key is new weapon name
      • current implementation allows up to 32 unique weapons (9 are built-in)
      • weapons can be put to slots 1 to 8
    • things
      • object containing one or more objects
      • each key is new thing type name
    • states
      • array of objects
      • contains all new states provided by this lump
      • sprite is encoded as uint32_t
      • stop, or S_NULL for next state is used if no Next is specified
        • setting Next to zero will trigger relative indexing and use first state in this lump!
      • allows using custom code pointers, including new ones with custom arguments
  • states are indexed relatively to this lump
    • negative index specifies jump to original Doom state (from info.h)
    • this applies to anything that requires state number
    • this allows multiple DOOMHACK lumps
  • ammo / weapon / thing names allows overriding of original stuff, like DEHACKED
    • names are taken from ZDoom
    • ammo / weapon / thing names are not connected in any way
      • weapon pickup can have different name than weapon itself
    • example: if you use name Archvile in things, the original MT_VILE will be modified instead of creation of new thing type
  • things with SPECIAL flag can now use new state called Touch
    • if present, thing will use this state instead of default (sprite-based) pickup handling
    • toucher mobj (player) will be set as special->target
  • weapon and ammo things are not connected to weapon and ammo types
    • use new Touch state with A_PickupAmmo or A_PickupWeapon code pointer
  • integer values are directly copied, there is no difference between
    • int
    • fixed_t
    • angle_t
    • it's up to generator tool how to handle this
  • a new set of code pointers should be mandatory, for cross-compatibility

 

Now some examples. Since CBOR is binary format, everything will be represented in pseudo-JSON instead.
 

Spoiler

 

Main object:

{
	"sounds":
	[
		"dsmagic",
		"dsenergy"
	],
	"ammos":
	{
		"Magic": { ... },
		"Energy": { ... }
	},
	"states":
	[
		{
			{"Sprite": 1482178888, "Frame": 0, "Tics": 8, "Next": 1},
			{"Sprite": 1482178888, "Frame": 1, "Tics": 8}
		}
	],
	"things":
	{
		"magicPickup": { ... },
		"energyPickup": { ... }
	},
	"weapons":
	{
		"magicWeapon": { ... }
	}
}

 

State properties:

{
	"Sprite": 1482178888,	// "HAXX"
	"Frame": 2,	// supports 32768 for BRIGHT
	"Tics": 8,
	"Next": 33,	// relative index in this array
	"Action": "A_Projectile",
	"Args":	// arguments are accessed directly by name
	{
		"Type": "Rocket",
		"Offset": 524288,	// 8.0
		"Angle": 8388608	// 0x00800000
	}
}

 

Thing properties (not all of them):

"CustomThing":
{
	"Height": 2097152,	// 32.0
	"Radius": 1048576,	// 16.0
	"Flags":	// array of strings; all flags are cleared before processing this array; array contains only flags for activation
	[
		"solid"
	],
	"StateSpawn": 23,	// relative index into states array
	"StateDeath": -457,	// S_TROO_DIE1 in info.h
	"SoundDeath": "dsbgdth1"
}

 

Weapon properties:

"CustomWeapon":
{
	"Slot": 8,
	"AmmoType": "Cell",
	"AmmoUse": 1,
	"StateReady": 12,
	"StateDeselect": 13,
	"StateSelect": 14,
	"StateFire": 15,
	"StateFlash": 20,
	"SelectionOrder": 290
}

 

Ammo properties:

"CustomAmmo":
{
	"ClipSize": 33,
	"Max": 333,
	"BackpackClips": 3, // (optional) amount of clips in backpack
	"BackpackMax": 400 // (optional) new maximum with backpack
}

 

 

 

Some notes to my example code:

• inventory (weapon and ammo) was fit into existing space in player structure
  • this limits weapons to 32 and ammo to 16
• sprite names are not dynamic, artificial limit is 511 (but over 200 already exist)
• some modification were already done by crispy doom team
  • most notably, handling PSPR code pointers - this is also required for my implementation to work properly
• this is only a demo
  • i did not fix some crispy features, like weapon recoil
  • many doom code pointers are not exported, yet

 

What we need to think about:

• proposed format
• property names
  • some names are ZDoom based
  • some are just random, some have specific idea behind
    • State prefix for state-based properties
    • Sound prefix for sound-based properties
• capitalization
  • my code is case-sensitive
  • capitalization is not entirely consistent
• scripting
  • from my demo you can see that my intention is to replace voodoo-doll scripting
  • how much turing-complete does this have to be?
• default values for new stuff
  • i have added a minimal amount of default values
  • most properties start with zero
• generalising some built-in behavior
  • boss explosion immunity
  • Archvile notarget and instant retaliation
  • lost soul is monster (+ISMONSTER flag does not exist)
  • hardcoded player animations
    • i have addressed some, but not all of them
  • and probably more
  • this would require extending FLAGs field to 64 bits
    • it's easy to extend mobjinfo_t
    • mobj_t is an issue though (save game)
      • spawnpoint does not seem to use type anywhere
      • this would give us 16 bits

 

The real strength of this lump is in tools that can create it.

I have included python script that can generate this lump from pseudo-DECORATE text files.

Sources are included too.

 

And now again. Nothing is set. Everything is only a proposal. We don't even need to call this DOOMHACK.

Also, my code is freely available to any source port without any limitations.

 

So, any interest or is this a dead end?

doomhack_test.zip

[Edward850]

The rather key point of dehacked at all is it runs on vanilla Doom, as in the DOS executable, allowing it to have unlimited range in support. Your proposal lacks any kind of demonstration of any of this running on that.

Edited July 7, 2023 by Edward850

[Individualised]

7 minutes ago, Edward850 said:

The rather key point of dehacked at all is it runs on vanilla Doom, as in the DOS executable. Your proposal lacks any kind of demonstration of any of this running on that.

Presumably this would be supported by ACE Engine which is 'technically' vanilla but I don't think we're counting that. As such this isn't a replacement for DeHackEd, there's no way you could use this with vanilla Doom. I think it's a cool idea on it's own, but I'd be more interested in an updated version of DeHackEd that adds support for codepointers that the original DeHackEd was never able to edit, even if there was no reason it couldn't.

[kgsws]

28 minutes ago, Edward850 said:

The rather key point of dehacked at all is it runs on vanilla Doom, as in the DOS executable, allowing it to have unlimited range in support. Your proposal lacks any kind of demonstration of any of this running on that.

That's not quite true anymore. Yes, original DEHACKED was used to modify DOS EXE. But nowadays DEHACKED is also used without even considering original DOS EXE.

Besides, i can make it work in DOS EXE. I actually intend to do that in some or other manner.

 

17 minutes ago, Individualised said:

codepointers that the original DeHackEd was never able to edit, even if there was no reason it couldn't.

Well, there is a reason. Code pointers are addresses to code memory. However, code memory has random (different) address on each PC.

To address this issue, there's a list of relocations. This list basically specifies where in the memory those addresses have to be fixed to point at correct code.

This list has a set size and nobody bothered to address this issue. Yes, it is technically possible but it quickly goes beyond the scope of the original, simple, DEHACKED EXE patching.

[Individualised]

7 minutes ago, kgsws said:

That's not quite true anymore. Yes, original DEHACKED was used to modify DOS EXE. But nowadays DEHACKED is also used without even considering original DOS EXE.

Besides, i can make it work in DOS EXE. I actually intend to do that in some or other manner.

DeHackEd and it's limitations are considered as part of vanilla Doom by most of the Doom mapping community. This is why it's supported by even source ports that are only supposed to have those features and nothing more such as Chocolate Doom. This new specification wouldn't be considered vanilla therefore it wouldn't be a replacement for DeHackEd. It's due to historical reasons rather than technical ones.

7 minutes ago, kgsws said:

Well, there is a reason. Code pointers are addresses to code memory. However, code memory has random (different) address on each PC.

To address this issue, there's a list of relocations. This list basically specifies where in the memory those addresses have to be fixed to point at correct code.

This list has a set size and nobody bothered to address this issue. Yes, it is technically possible but it quickly goes beyond the scope of the original, simple, DEHACKED EXE patching.

I'm aware of all of this. I think you misunderstood what I was saying.

Edited July 7, 2023 by Individualised

[Bauul]

The demo video certainly looks very impressive.

 

From a lay-person's point of view, can you outline what this offers that something like Decohack+DSDHacked does not, and what the benefit is of implementing this in a source port over something like full-blown DECORATE?

[Kappes Buur]

indeed, most impressive

[Doomy__Doom]

Feels like there are 2 things here mashed together:

a) content definition method that transpiles to a dehacked patch (or put another way - a decohack parser directly in a source port)

b) dehacked extensions

 

Regarding a), that would be cool to have in any port, although cost-to-benefit is probably not nice, considering how little effort it takes to copy decohack's output into slade.

 

Regarding b), an important thing about dehacked is that new standards build on predecessors - bex is "vanilla and ...", mbf is "bex and ...", mbf21 is "mbf and ...". Dehextra and DSDHacked are a bit on the side, but still. So any proposal for more dehacked features should build on top of latest standards ("mbf21/dsdhacked and more weapon/ammo slots") so as to not reinvent the wheel around already solved issues (e.g. mbf21 generalized boss explosion immunity, dsdhacked handles arbitrary custom sound definitions).

 

If I'm completely misunderstanding OP and a single standard is the point (insert xkcd here), then the question transforms to "why not existing Zdoom or Edge stuff" I guess.

[kgsws]

1 hour ago, Individualised said:

DeHackEd and it's limitations are considered as part of vanilla Doom by most of the Doom mapping community.

Oh, i have not realized that. Well, that really makes this kinda pointless.

 

1 hour ago, Individualised said:

I'm aware of all of this. I think you misunderstood what I was saying.

Yeah, i guess i did.

 

1 hour ago, Bauul said:

From a lay-person's point of view, can you outline what this offers that something like Decohack+DSDHacked does not

It allows you to add new weapons and ammo types. It adds new code pointers with more flexibility, including sector movement.

It also allows cumulative modding, so you can like create mod for other mod.

 

1 hour ago, Bauul said:

what the benefit is of implementing this in a source port over something like full-blown DECORATE?

It's much less complex and intrusive to support this than DECORATE.

Obviously, DECORATE already provides flexibility and features i aim to add using this.

 

4 minutes ago, Doomy__Doom said:

an important thing about dehacked is that new standards build on predecessors - bex is "vanilla and ...", mbf is "bex and ...", mbf21 is "mbf and ...". Dehextra and DSDHacked are a bit on the side, but still. So any proposal for more dehacked features should build on top of latest standards ("mbf21/dsdhacked and more weapon/ammo slots") so as to not reinvent the wheel around already solved issues (e.g. mbf21 generalized boss explosion immunity, dsdhacked handles arbitrary custom sound definitions).

Yes. I am actually attempting to propose a fresh start. Without old limitations but also something easily extendable in the future. And importantly, easy to implement.

(though i hope my notion of "easy" is not too skewed)

And yes, i am ready for this to be completely rejected.

 

8 minutes ago, Doomy__Doom said:

If I'm completely misunderstanding OP and a single standard is the point (insert xkcd here), then the question transforms to "why not existing Zdoom or Edge stuff" I guess.

It would be cool if every source port supported DECORATE. Then i would not be proposing this.

[OpenRift]

1 hour ago, kgsws said:

 Besides, i can make it work in DOS EXE. I actually intend to do that in some or other manner.

I think making that a priority would be wisest way of going about it, that way this new method of modding doesn't remain just an odd offshoot of Crispy Doom and instead could prompt the more advanced source ports to implement it as a feature.

[BUYXRAYS]

Damn. This looks nice.

Although, there is one issue.

If any source port wants to add this, they have to fork your modified version of crispy doom and merge your code with theirs.

Apart from that, this is really good

[Gregor]

Not exactly on topic but since we're talking about DeHackEd, there's one thing i never fully grasped. How far is GZDoom DeHackEd compatible anyways? A lot of mods which include DeHackEd lumps contain a separate DECORATE patch to make certain custom changes work in GZDoom. Like in KDiKDiZD, some of the bosses attacks had to be translated to ZScript to get them to work correctly in GZDoom, but most of the other changes work just fine. So some things obviously don't work as is and have to be translated to DECORATE/ZScript. This seems to just be a problem with vanilla DeHackEd. But what things break and why?

[TheNoob_Gamer]

47 minutes ago, Gregor said:

How far is GZDoom DeHackEd compatible anyways?

GZDoom, in the name of compatibility and modder flexibility, generally doesn't support very hacky modding/mapping implementations (especially when said methods rely on engine bugs).

A quick peek inside KDiKDiZD's Dehacked tells me that certain enemies ultilize behavioral loopholes (using the rocket projectile as a shield, the rocket projectile itself replaces BFG ball), or stock pickups are instead reserved for advanced effects. Such wizardry is not very viable in GZDoom environment, hence the behavioral translation to ZScript to ensure stability.

So for the most part, surface-level Dehacked editing should generally work well. It's when you start diving deeper (with advanced tools like DecoHack or DEH9k) that things can get murky on advanced ports.

Spoiler

This is just my attempt of giving a layman answer, so the examples here merely serve to illustrate my points. More experienced Doomheads, Dehacked sorcerers (or the makers of KDiKDiZD itself) may explain your question better than I did!

 

Edited July 8, 2023 by TheNoob_Gamer
add clarification

[Gregor]

12 minutes ago, TheNoob_Gamer said:

So for the most part, surface-level Dehacked editing should generally work well. It's when you start diving deeper (with advanced tools like DecoHack or DEH9k) that things can get murky on advanced ports.

 

What would you define as "surface-level Dehacked editing?" Again, most of the fancy modding in KDiKDiZD seems to work just fine through DeHackEd with GZDoom. Jumpwad and Valiant (both MBF) also use a separate DECORATE patch, so i guess it's not just vanilla. One weird case is 1000 Lines 3, which is vanilla but made with Decohack - it defines a lot of its DeHackEd content in the DECORATE patch, but some things like the flametowers in MAP30 are left out but still work anyways. Also a lot of its custom decorations are left out while several others are defined.

 

I guess there isn't a way to really know beforehand and one simply has to test each custom change for compatibility with GZDoom.

[TheNoob_Gamer]

5 minutes ago, Gregor said:

What would you define as "surface-level Dehacked editing?"

Anything that can be accomplished by a GUI editor like WhackED; swapping around weapon/enemy/entity frames; modifying strings, entity data (HP, height, mass), etc. MBF wads have quite a few features, but their effects can generally be achieved with enough tool knowledge.

 

Advanced tools allow for advanced effects manipulation that GUI editors lack, or would be very tedious to do. For example, you could create a smooth weapons pack without sacrificing (apparently) anything and maintains demo compatibility. Ultimately, what is "surface-level" depends on your modding finesse. 

9 minutes ago, Gregor said:

it defines a lot of its DeHackEd content in the DECORATE patch

It's probably worth noting that DecoHack is still at its core Dehacked - the program merely converts DECORATE-like text scripts to appropriate Dehacked files. And IIRC, certain effects of 1k Lines 3 are meant only for ZDoom-based ports?

[Gregor]

14 minutes ago, TheNoob_Gamer said:

 IIRC, certain effects of 1k Lines 3 are meant only for ZDoom-based ports?

No, 1000 Lines 3 is fully vanilla compatible. You might think of something like Auger;Zenith, which is limit-removing and has extra light effects and breaking glass in ZDoom ports. In 1000 Lines 3, the DECORATE patch actually lacks some definitions from the DeHackEd patch; it certainly doesn't add anything. So i guess content definitions in DECORATE overwrite DeHackEd definitions in GZDoom?

 

17 minutes ago, TheNoob_Gamer said:

Anything that can be accomplished by a GUI editor like WhackED; swapping around weapon/enemy/entity frames; modifying strings, entity data (HP, height, mass), etc. MBF wads have quite a few features, but their effects can generally be achieved with enough tool knowledge.

That's a bit scetchy though, since there is really nothing editors like decohack or even DEH9000 do that can't be done with Whacked or even input by hand in a DeHackEd text file if you really wanted to. It would be VERY tedious in some cases, sure, but still possible. That's why I never got the whole hype around DEH9000 - it's neat to have but i don't see what it does that Whacked can't do as well. It doesn't give you new codepointers for vanilla for instance or the ability to edit things that DeHackEd normally can't access. So it's more about the ease of use for someone who's familiar with Python scripting and it's easier to keep track of things. At least that's my impression.

[TheNoob_Gamer]

6 minutes ago, Gregor said:

No, 1000 Lines 3 is fully vanilla compatible... So i guess content definitions in DECORATE overwrite DeHackEd definitions in GZDoom?

Then I must have confused the thing with Eviternity, a project that also uses OTEX and has advanced ZDoom features. And yes, regarding the latter part, DECORATE modifications take precedence with the DEHSUPP lump; not sure about ZScript.

 

8 minutes ago, Gregor said:

That's a bit scetchy though, since there is really nothing editors like decohack or even DEH9000 do that can't be done with Whacked or even input by hand in a DeHackEd text file if you really wanted to.

Most (emphasize on most) cases of Dehacked modifications I'm aware of never make use of editing the files themselves. I see that the resulting patches are merely the data presented in a readable format, not to edit. My point is, with how archaic Dehacked gets nowadays, using an editor is the recommended default method; but using a command line toolset allows you to have a deeper look into its inner workings and hence the ability to better fine tuning features and more efficient ways to execute loophole tricks. In DEH9K's case, it makes certain tasks easier and more likely to succeed - for example, making an "enhancement" mod in Dehacked without breaking demo compatibility is not something you are likely to do with editors alone. 

[Xaser]

This proposal is trying to do too many things at once. At it's core, it's two big chunks of stuff that should really be split apart:

1. a new patch format
2. a collection of extensions to existing features (e.g. new codepointers)

#1: new patch format

 

This is a tall order. For better or worse, there are two major common-ish Doom actor-definition formats: DeHackEd (specifically the text-based BEX patch format) and DECORATE, so everyone and their brother (including me) is immediately going to ask: why not one of these? Doing something completely-new should really only be considered if there's no other alternative, i.e. one of the existing formats is a dead-end and can't be extended as desired -- UDMF was needed, for instance, since the Hexen map format wasn't extensible and had significant limitations as-is.

 

What does this new format do that baseline DeHackEd Patches cannot? To clarify, I'm specifically talking about the format -- is there anything feature-wise being proposed here that can't be done by either adding new addressable things (i.e. codepointers and new blank weapons) or brand-new blocks (e.g. a "[PLAYERSTATES]" section)? BEX isn't exactly the cleanest format around from either a human- or engine-side perspective, but DecoHACK basically solved the human-facing problem for us and every relevant port has already implemented a parser for it. If there's something important here that's difficult or impossible to express in the patch format, then maybe that'd be a reason to do not-DeHackEd -- but is there?

 

I also don't understand the desire to make the new format a binary blob. This requires writing a brand-new tool to let humans actually make use of the format. Sure, DECOHACK to DeHackEd is technically this, but that was only a desirable route because the BEX format is grandfathered in (i.e. changing it is an uphill battle). I don't understand the purpose of creating something brand-new that inherently has the same limitation. If the concern is filesize for some reason, people distribute their wads in ZIPs, text compresses well, and there's no way a reasonably sane mod would generate enough text for filesize to become an issue anyway. :P

 

At any rate, the previous two paragraphs ask the question "why not DeHackEd?", and if there's a good reason for that, then I must ask: why not DECORATE then? What would this new format do that DECORATE couldn't? Is there some Thing in zdoom-style DECORATE that's difficult to support in a more vanilla-ish engine? Why even try a new format if the sample mod's "source" files are already DECORATE to begin with? These are legit questions, not rhetorical. Well, maybe that last one is. :P

Finally, I do have some issues with specifics about the format, but I don't feel they're worth bringing up because that whole approach is just barking up the wrong tree IMO.
 

#2: feature extensions

 

New codepointers and other things that could conceivably be added to MBF21-flavour DeHackEd (i.e. using the `args` table) are much much lower-hanging fruit than the rest of the stuff being pitched here, and there are some really cool features being shown off in that demo wad. There's some gold here, like the idea of having custom pickups be define-able via a Touch state and a set of codepointers.

 

The most practical way forward in general is to pitch the new codepointers and extensions separately and help kickstart a "MBF23" (realistically, "MBF24" at the earliest probably :P ). There's a few things that people have brought up but so far not really much of a dedicated effort to collecting these all into a proper wishlist. The stuff you've done here seems like a great starting point.

tl;dr on all the above: for the format, stick with DeHackEd or go all the way with DECORATE. All the codepointery extensions are a separate thing to be pitched alongside whatever approach is done here, and has less barriers for entry for anyone wanting to implement this.

[boris]

Except for weapons and ammo, what does this proposal offer that's not possible with DEHEXTRA/DSDHacked/MBF21, or that the latter could be extended to do? DEHEXTRA offers a bunch of new states, things, sounds, sprites. DSDHacked effectively lifts those limitations altogether. MBF21 adds new functionality, including parameterized code pointers. And that's already supported by a bunch of ports as of today.

 

https://doomwiki.org/wiki/Dehextra

https://doomwiki.org/wiki/Dsdhacked

https://doomwiki.org/wiki/MBF21

[Gez]

5 hours ago, Gregor said:

So i guess content definitions in DECORATE overwrite DeHackEd definitions in GZDoom?

Not necessarily, but they can.

 

Say you have a DEHACKED patch that changes the imp. Now add a DECORATE lump that replaces the imp. In GZDoom, the imp is changed by the former but also replaced by the latter, so you don't get to see the changed imp.

4 hours ago, TheNoob_Gamer said:

And yes, regarding the latter part, DECORATE modifications take precedence with the DEHSUPP lump

The presence of a DEHSUPP lump will prevent DEHACKED parsing in GZDoom. It's not its original meaning, but it can be read as "DEHacked SUPPression" now.

 

Also one thing to keep in mind is that it's not just ZDoom/GZDoom. It's also Skulltag/Zandronum. A few projects have had this trick used not because the DEHACKED caused problems in ZDoom or GZDoom, but because they caused problems in Skulltag or Zandronum.

[Gregor]

44 minutes ago, Gez said:

Also one thing to keep in mind is that it's not just ZDoom/GZDoom. It's also Skulltag/Zandronum. A few projects have had this trick used not because the DEHACKED caused problems in ZDoom or GZDoom, but because they caused problems in Skulltag or Zandronum.

Does that mean it's actually Zandronum support that necessitates the inclusion of a DECORATE patch in most of those mods where they use both DECORATE and DeHackEd to do the same thing?

Edited July 8, 2023 by Gregor

[fabian]

12 hours ago, kgsws said:

it contains tree-like structure encoded using CBOR

 

I'm afraid this is where most people will stop reading. 

[hobomaster22]

I don't see a good reason for this. Developers can already develop their mods in a sane way with DECOHack. This eliminates the need to work with the underlying file format directly and allows users to write decorate like code already. Practically all ports support dehacked and it would be more logical to extend it. Finally, proposing a new format in binary is a huge step backwards. Binary is brittle, hard to debug, and any potential performance benefits are so minimal it's not worth it. Even worse, they can end up being slower to parse and more bloated than something like JSON. There is a good reason binary formats are mostly dead and JSON has taken over the world.

[Individualised]

I'm interested in the reasoning for using a binary format. It's not the first time that OP has proposed a new binary-rather-than-text-based format for Doom (they also proposed a mapping format for ACE Engine that could also be used elsewhere).

Edited July 8, 2023 by Individualised

[Dimon12321]

I was always dreaming of having vanilla Doom with more than 9 weapons.

[Doomy__Doom]

Oh I didn't even pay attention to the binary thing at 3 AM.

 

Why? There already is a two point failure scheme with editing tool (decohack/whacked/...)->patch format (plaintext)->in-game effects (parsing the patch). Binary is doing nothing here but a) making the tool mandatory for any editing at all, b) making it a hurdle to verify tool outputs and thus debug first point of failure.

 

Loading times of actual slow/heavy wads are not because of parsing a couple text lumps, it's because of everything else. Most of my wads/ folder content is >10mb and the biggest outlier dehacked patches I can find with a quick look are 62kb for Vesper and 82kb for Corruption. That's... nothing, really.

 

 

39 minutes ago, hobomaster22 said:

Even worse, they can end up being slower to parse and more bloated than something like JSON

True that. As an anecdote, a different game series I'm involved in modding with uses an in-house binary format for content definition (unit/item/weapon/etc stats, game parameters, model references, map placements and tons of other things) which has about 60% bloat to 40% actual game data. Unpacked into plain CSVs structured by type, space required is half the original file size :)

[kgsws]

6 hours ago, Xaser said:

This proposal is trying to do too many things at once. At it's core, it's two big chunks of stuff that should really be split apart:

1. a new patch format
2. a collection of extensions to existing features (e.g. new codepointers)

That is actually true.

 

 

6 hours ago, Xaser said:

What does this new format do that baseline DeHackEd Patches cannot? To clarify, I'm specifically talking about the format -- is there anything feature-wise being proposed here that can't be done by either adding new addressable things (i.e. codepointers and new blank weapons) or brand-new blocks (e.g. a "[PLAYERSTATES]" section)? BEX isn't exactly the cleanest format around from either a human- or engine-side perspective, but DecoHACK basically solved the human-facing problem for us and every relevant port has already implemented a parser for it. If there's something important here that's difficult or impossible to express in the patch format, then maybe that'd be a reason to do not-DeHackEd -- but is there?

 

4 hours ago, boris said:

Except for weapons and ammo, what does this proposal offer that's not possible with DEHEXTRA/DSDHacked/MBF21, or that the latter could be extended to do? DEHEXTRA offers a bunch of new states, things, sounds, sprites. DSDHacked effectively lifts those limitations altogether. MBF21 adds new functionality, including parameterized code pointers. And that's already supported by a bunch of ports as of today.

 

It offers cumulative loading. It is internally structured in a specific way that you can load multiple DOOMHACK lumps without any collisions.

Things are addressed by name instead of index to array. States are relative to specific DOOMHACK lump.

As with DECORATE, you can load two independent mods at once. With DEHACKED/BEX you will probably get a ton of collisions with states and things.

 

DEHACKED / BEX i just outdated thing, i think. Any new extensions seem like a format hacks to me.

 

 

6 hours ago, Xaser said:

go all the way with DECORATE

I agree that DECORATE would address this. It's just that it is too intrusive to implement.

 

6 hours ago, Xaser said:

I also don't understand the desire to make the new format a binary blob. This requires writing a brand-new tool to let humans actually make use of the format. Sure, DECOHACK to DeHackEd is technically this, but that was only a desirable route because the BEX format is grandfathered in (i.e. changing it is an uphill battle).

Exactly, editing is often done using external tools. So "binary" format should not be an issue.

 

6 hours ago, Xaser said:

I don't understand the purpose of creating something brand-new that inherently has the same limitation.

 

1 hour ago, hobomaster22 said:

Finally, proposing a new format in binary is a huge step backwards. Binary is brittle, hard to debug, and any potential performance benefits are so minimal it's not worth it. Even worse, they can end up being slower to parse and more bloated than something like JSON. There is a good reason binary formats are mostly dead and JSON has taken over the world.

 

1 hour ago, fabian said:

I'm afraid this is where most people will stop reading.

 

10 minutes ago, Doomy__Doom said:

Oh I didn't even pay attention to the binary thing at 3 AM.

 

What is the issue with CBOR? What are supposed limitations? Dead end?

CBOR is basically JSON in binary form.

It offers everything what JSON does, but is easier to parse.

There are no limitations like with BEX. Since it supports object with key / value, like JSON does, it can be extended with anything in the future.

If you specifically don't like binary representation of tree-like structure, we can use JSON directly.

CBOR->JSON or JSON->CBOR are (almost) directly convertable. (CBOR specifies 'binary string' which does not exist in JSON, but i do not use that in my proposal)

 

So again. CBOR is basically JSON. If you think parsing JSON is somehow better then we can use JSON. Literally nothing about internal structure i proposed would change.

Edited July 8, 2023 by kgsws

[dsda-dev]

As already mentioned, it feels like a lot of the proposed features are already covered by or could be added to the existing mbf21 + dehextra format. Keep in mind though that not every port adopted those changes, and further few would adopt these, so you won't be able to have something that is as cross-compatible as dehacked. Every generation of changes leaves some ports behind, and a complete format change will leave more than a simple extension.

 

The thing with decorate is that not all features can be injected into a port in isolation. Let's say I do define 32 weapons. How does the user access them? Should there be 32 weapon hotkeys? What does the hud look like? There are usability problems that can be addressed in the advanced ports because they can also customize the interface significantly.

 

Now a point about the general approach. If you come in with a solution already planned out, it adds a lot of baggage to the conversation and there are many assumptions that are more difficult to pull apart and address. Creating new standards is way more a matter of lobbying and communication than it is about technical details or how it might look in crispy doom. When I started thinking about mbf21, the first thing I did was open a thread asking creators (not software developers!) about what their immediate headaches were. What kind of problems do they face regularly that prevent them from achieving their goals? Focusing on the problem itself and refining it in a wider discussion will make sure there is a basic alignment already among the people involved before even thinking about a solution.

 

If / when we start thinking about the next mbf2x, I will begin by asking creators where they are running into walls and can't quite do what they want to. I know a few have started to write down different actions that are missing for instance that would open up a lot of flexibility. I'm not sure it's the right time to have that conversation though - there is something I will call "standard fatigue" or that there needs to be a cooldown between standard changes where we can recalibrate and see exactly how things are. By that I mean mbf21 is getting a lot more attention right now, but I wouldn't say it's "fully explored", which means a conversation about its limitations might also be premature. There is also just the hard reality that bugs come up over time that need to be addressed and we want a stable baseline before stacking new things on top.

 

In terms of dsda-doom specifically, I'll say I don't have much interest in implementing a comprehensive new standard like this right now. There are plenty of existing stable standards that still need to be supported, and I would rather focus on them.

[ChopBlock223]

14 hours ago, kgsws said:

"CustomWeapon":

{

"Slot": 8,

"AmmoType": "Cell", "AmmoUse": 1,

"StateReady": 12, "StateDeselect": 13, "StateSelect": 14,

"StateFire": 15,

"StateFlash": 20, "SelectionOrder": 290

}

I want to preface with how I don't intend to shit over your work here or anything, but I have a lot of questions.

 

Speaking as a dumbass who bangs rocks together to try to make things work (and as may be evident by how I fucked up the formatting of the quoted example code), I don't know what a lot of these values mean.

It's a weapon, it uses cells and has a selection order, but StateReady being 12 tells me absolutely nothing obvious at all, I have no idea what 12 entails and I would need added outside context to hope to understand this.

 

Meanwhile, looking at Decorate, every single individual frame in every state for every actor is written out in a pretty clear way, in a sequence, and every actor flag and function has a spelled out name. You would need added context too, but you'd pretty quickly learn what belongs to what if you poke through the .wad or .pk3 for a while, and it's straightforward.

 

You describe how you would "compile" code in advance using a more easy to read substitute language, similar to DecoHack, which would be workable, but that also makes me wonder what this would largely offer over DecoHack with MBF21's own extended DeHacked stuff? That has new functions too and is unreadable once converted into state table balderdash, where you'd "compile" from a readable DecoHack code.

 

To me, it would feel like it'd be more practical if more functionality was added onto that existing system itself, rather than adding a second system like it. It may be different under the hood, of course, and I remember sourceport devs expressing significant frustration about adding support for DSDHacked and MBF21's new DeHacked stuff because DeHacked is a bunch of crude hacks which is supported only as legacy.

 

 

12 hours ago, kgsws said:

It allows you to add new weapons and ammo types. It adds new code pointers with more flexibility, including sector movement.

It also allows cumulative modding, so you can like create mod for other mod.

 

12 hours ago, kgsws said:

Yes. I am actually attempting to propose a fresh start. Without old limitations but also something easily extendable in the future. And importantly, easy to implement. 

These parts sound very good, though I feel it'd be more useful for users like me if you didn't need to "compile" readable code into a mess, rather just developing a directly readable and writable language akin to Decorate.

Of course, this may be much more work to actually develop and then implement, but I'm just speaking as an end user with limited understanding.

 

12 hours ago, kgsws said:

It would be cool if every source port supported DECORATE. 

That would make things a lot easier. I have heard vague suggestions over the years of partial/modified Decorate support for less advanced ports, or a stripped down interpretation of it, either which would be demo compatible.

I have no idea what the status of any such thing is, or if it's actually in the cards at all.

 

7 hours ago, Xaser said:

The most practical way forward in general is to pitch the new codepointers and extensions separately and help kickstart a "MBF23" (realistically, "MBF24" at the earliest probably :P ).

That'd be cool.

 

7 hours ago, Xaser said:

There's a few things that people have brought up but so far not really much of a dedicated effort to collecting these all into a proper wishlist. The stuff you've done here seems like a great starting point. 

I can think of all kinds of things. A_Wander, A_JumpIf, and whatever that one was called where it made a jump based distance to target.

However, with some resistance in terms of building onto DeHacked, I again wonder if something more like Decorate proper would be in better order.

[Bauul]

2 hours ago, ChopBlock223 said:

whatever that one was called where it made a jump based distance to target.

 

Just FYI A_JumpIfTargetCloser is already part of MBF21, at least according to the spec document.