DSDhacked [unlimited everything]

[Gez]

7 minutes ago, ChopBlock223 said:

I'll once again preface with my ignorance and lack of knowledge, but what about forking Decorate

That doesn't really make sense, because there's no such thing as DECORATE. It's not some sort of standalone module bundled with GZDoom. Stuff like worrying about RNG and PRNG is moot.

 

Now what you can do is actually define some common baseline specifications that would be a subset of what GZDoom supports. Basically if you cover only what can be done with MBF21+DSDHacked, putting aside stuff like runtime expression evaluations and anonymous functions, then you end up with something that's just a more convenient modding interface for defining new stuff.

[Graf Zahl]

Yeah, that's precisely what I'd be aiming at. The expression evaluator may not be totally out of scope, though, as it is fairly contained, it merely needs better encapsulation of the state's arguments. But surely not for a first version.

[Kinsie]

I'm pretty sure Eternity is already closest to cornering the market on "high level actor definition language that's also demo compatible", and outside of one or two notable projects like Heartland that rely on Eternity's mapping features, it doesn't seem to get a whole lot of use. I haven't delved into that port, so I don't have any insight into why. If anyone here does, I'm all ears.

 

I feel like DSDhacked is trying to find some middle ground between GZDoom's Jenga-Tower-O-Features at the expense of demos and other vanilla purity (that someone replacing the Zombieman with 40+ randomly chosen Zombieman replacements probably isn't too fussed about), and DEHACKED's universal compatibility with pretty much every source port dating all the way back to the Ming Dynasty (provided you're willing to dash yourself against the rocks of frame-juggling). I'm not entirely convinced that middle ground actually exists, at least in a way where anyone would seriously use it when there's already enough ways to redefine monsters to fulfil most workflows.

 

If this is just a fun, hobby-project way to keep a programmer busy and off the streets, though, then sure, that's fine. :)

[ChopBlock223]

And then using simple and straightforward formatting like Decorate (or something similar) as a model for that?

 

EDIT: In response to Gez

Edited September 15, 2021 by ChopBlock223

[Graf Zahl]

1 minute ago, Kinsie said:

I feel like DSDhacked is trying to find some middle ground between GZDoom's Jenga-Tower-O-Features at the expense of demos and other vanilla purity (that someone replacing the Zombieman with 40+ randomly chosen Zombieman replacements probably isn't too fussed about), and DEHACKED's universal compatibility with pretty much every source port dating all the way back to the Ming Dynasty (provided you're willing to dash yourself against the rocks of frame-juggling). I'm not entirely convinced that middle ground actually exists, at least in a way where anyone would seriously use it when there's already enough ways to redefine monsters to fulfil most workflows.

 

 

You know, such attitudes are incredibly discouraging to us port authors. It boils down to "why improve things if people made do with the old?" and is the epitome of the anti-innovation that has kept the classic source ports in their niche for 20 years because no developer of these ports ever tried to branch out. DSDA is the first port that actually makes an attempt to remove these barriers while at the same time trying to avoid building its own niche, and it should get all encouragement that can be gotten to continue with that.

 

And please do not use Eternity as an example for making it sound pointless. Eternity mods only run on the Eternity Engine which means that this mere fact alone may serve as discouragement, whereas these new *documented* standards are meant to implemented by more than one port, so they won't cause any sort of port lock-in, provided that sufficient ports care. If they find adoption, even those ports which initially take a wait-and-see approach may later follow suit.

 

 

[GoneAway]

The time investment in this for dsda-doom was in releasing limits on all the entities involved, which would need to be done regardless - i.e., it doesn't actually have anything to do with the format that communicates the entities. I'm skeptical how requiring a new format to be parsed versus a very slight extension to dehacked would be better for all the ports that support dehacked but not anything else (the only real change here is the sprite / sfx name setting).

 

For anything further, that's a topic that's totally open right now. But that would be so distant that it's not super important to worry about right now. We won't even know what the priorities are for a new standard until modders use mbf21 for a while and figure out what causes the most pain and where the important gaps are. That in turn would inform what we want to achieve out of another iteration and what format that would come in.

[GoneAway]

Something else to keep in mind: let's assume this next iteration discussion happens some time in 2022. Let's be optimistic and guess that by then dsda-doom probably supports (some subset of) the "in hexen" format, acs, decorate, and perhaps some kind of udmf. What would that mean for the conversation? Would it cover the niche of people that want to make more advanced things but still have them playable in a prboom+-derived port? Would there still be the same demand for mbf22? Basically these are all possibilities in my head that might change the conversation and the form of any standards that come out, so it's hard to plan so far ahead.

[Graf Zahl]

22 minutes ago, kraflab said:

 I'm skeptical how requiring a new format to be parsed versus a very slight extension to dehacked would be better for all the ports that support dehacked but not anything else (the only real change here is the sprite / sfx name setting).

 

It may have been "very slight" to you - but surely not for me! Implementing the spec meant lots of jumping through hoops and having to solve problems I'd have preferred to not care about. So, assuming that other authors of more advanced ports may face the same issue, these things may have an impact in the end and hamper adoption if modders feel they are limiting their work by using these features.

 

This is also what may discourage other port authors, i.e. having to adapt a spec aimed at comparatively primitive implementations to more refined and better abstracted implementations which may be a poor fit to the spec's needs.

 

10 minutes ago, kraflab said:

Something else to keep in mind: let's assume this next iteration discussion happens some time in 2022. Let's be optimistic and guess that by then dsda-doom probably supports (some subset of) the "in hexen" format, acs, decorate, and perhaps some kind of udmf. What would that mean for the conversation? Would it cover the niche of people that want to make more advanced things but still have them playable in a prboom+-derived port? Would there still be the same demand for mbf22? Basically these are all possibilities in my head that might change the conversation and the form of any standards that come out, so it's hard to plan so far ahead.

 

If you got "all that" (good luck trying to make it work within a single year... :P) there would be no discussion. I think when having the choice of defining new actors in a high level language like DECORATE vs. Dehacked it's a very small minority that'd choose Dehacked. Most would only do it if they have no choice. So all addition that'd need would be to expose some features that could be retroactively grafted on the stock actors.

 

 

Edited September 15, 2021 by Graf Zahl

[GoneAway]

It's a good point about it being annoying to make changes to primitive stuff in advanced ports. For me this was a logical conclusion to dehacked that would hopefully prevent the "let's add another x of y to dehacked" iterations from happening multiple times (dehextra part 2 was definitely already gaining traction for people), which would mean less times something like this needs to happen. In other words, a "let's finish this once and for all" feeling.

[ketmar]

the main problem with DeHackEd and its derivatives/extensions is that it is still based on the idea of a big frame table. and this is not how most advanced sourceports work these days. DECOLITE (or DECORATE stripped, if you want) can be easily compiled to some extended DeHackEd internally by the sourceport, but converting DeHackEd back to DECORATE/DECOLITE is much harder task (and often it is simply impossible).

 

supporting good old DeHackEd already turns the engine code into a freakin' mess, and adding even more mess on top of that is something i, for example, won't do, ever.

 

besides, it is much easier to work with actors and inheritance. no need to know the exact frame indicies in the vanilla, no need to manuially counting frames, and so on — built-in DECOLITE compiler can do it all for you.

 

from my PoV, there is simply no reason to keep building things on top of DeHackEd ideas these days. the support for the original DeHackEd is required to run old mods, but please, let's try to avoid that hack for new mods!

 

p.s.: k8vavoom simply rejects mods with dehextra and MBF21 dehacked extensions. this is as much support as i want to implement for those things.

Edited September 15, 2021 by ketmar

[Astronomical]

I don't know anything about programming so this probably will sound really dumb.  But isn't the whole issue here created because we are trying to build off of a previous standard?  So it does seem misguided that the solution would be to build off of Decorate?  I like the idea of a high level scripting language like Decorate becoming a standard but I would prefer if it could be independently implemented and didn't rely on anything previous.  We could easily blow away anything done with the dehacked standard if it was included.  The only issue that I could see with the plan is that Decorate has been updated to a point that some mods don't work between gzdoom versions, as well as many mods not working in zandronum. 

[ketmar]

it's much more than that: basically DeHackEd and DECOLITE require completely different internal engine architecture. the thing is that DECOLITE not only can be compiled (read: implemented) for the older engines (implementing DeHackEd in more advanced engines is a PITA over PITA over PITA), but it is also much easier to use for modders. and i mean MUCH easier.

 

we are not talking about supporting the full GZDoom DECORATE (that's why i keep calling the thing DECOLITE, to avoid such confusion), but about the base extension architecture to build upon. the exact specs and features of DECOLITE can be fleshed out later.

[GoneAway]

2 hours ago, ketmar said:

supporting good old DeHackEd already turns the engine code into a freakin' mess, and adding even more mess on top of that is something i, for example, won't do, ever.

I don't really see how, but I'm not familiar with k8vavoom. Is it a specific problem in how dehacked support is implemented? A translation layer should cover the abstraction between reading the file and what happens inside the engine, so it shouldn't impact it much. If anything, supporting more dehacked and more games in dsda-doom has made the engine itself cleaner over time because it heavily favors replacing the old stiff code with nice abstractions :^)

[Graf Zahl]

No, it's not that easy. For our advanced ports Dehacked is some really messy legacy baggage that blocks us from genuinely cleaning up the engine. Had it not been for the continued need to support Dehacked, DECORATE would look a lot different than it does because much more information would have been moved into subclasses in the actor hierarchy. As an example, for a simple decoration all those properties an actor has to support being used as a monster are not needed, it could be made with half, or even just a quarter of the original size. However, since any decoration can be turned into a monster using Dehacked, we have to support this, meaning we cannot strip down the actor base class to be better suited for simple stuff and use an extension for actual monsters. This not only inflates data size, it also forces the main thinking function to be all-encompassing and we cannot provide a simpler one for simpler objects.

 

As another example, in GZDoom there is a strict actor hierarchy, normally each actor should only be able to execute its own states and its ancestors'.

Dehacked requires that it can run *every* state. Which means we cannot do state scanners that scan the hierarchy - we need to use ones that scan all actors in the game. This not only complicates the setup but also makes it a lot harder to do proper consistency checks.

 

With unlimited states it gets even more problematic because there is no good way to assign them to any actor at all, so gross hacks are the only way to go forward because now all places where states get used need additional handling.

 

I also had some issues with the sound extension because Dehacked allows referencing a sound before it is defined. So I ended up with a situation where defining the sound had to happen before I knew what it is.

 

In all these cases the core problem is the same - Dehacked works on arrays that no longer exist and that often oversimplify matters, so each time new references get added to the spec it forces us to add yet another workaround for it that needs to circumvent the abstractions that were built into the engine.

 

I fully understand why Ketmar does not want to deal with this, but so far it has not been an insurmountable obstacle for GZDoom, but I simply cannot guarantee that this will remain so. It depends on what kinds of stuff will get added in the future.

 

 

[ketmar]

@kraflab yeah, exactly what Graf wrote, every word there is a sad truth.

 

there is no more "common state/frame table", and sharing state between actors is a PITA that basically breaks every assumption the engine could made about the code, ruining all kinds of sanity checks and optimisations. and all other things from Graf post apply too.

 

while it is possible to implement dehextra/MBF21, i won't mutilate the engine for that: it is tortured more than enough even for simple DeHackEd support.

 

p.s.: GZDoom and k8vavoom are not really different in the internal design, they both built upon entity class hierarchy. so we have a common set of problems with DeHackEd too.

Edited September 15, 2021 by ketmar

[GoneAway]

My experience is that hard things usually aren't as hard as people think. The things you mentioned don't ultimately sound like that big of an obstacle to me. Of course, you always have to make a choice whether to adapt, hack, or ignore, and each choice has its own tradeoffs.

[Gibbon]

18 minutes ago, kraflab said:

My experience is that hard things usually aren't as hard as people think. The things you mentioned don't ultimately sound like that big of an obstacle to me. Of course, you always have to make a choice whether to adapt, hack, or ignore, and each choice has its own tradeoffs.

I agree with that.  I implemented DEHEXTRA and mbf21 onto ReBOOM Experimental in 8 hours.  However it was a sledgehammer implementation, as a POC.

 

Now if that was in C++ with a strict hierarchy and radically different internals, I can see where the issue lies.  It isn't necessarily in architecture terms but in "do I want to maintain this code forever" kind of thing.  I definitely get where they are coming from.

[Graf Zahl]

24 minutes ago, kraflab said:

My experience is that hard things usually aren't as hard as people think. The things you mentioned don't ultimately sound like that big of an obstacle to me. Of course, you always have to make a choice whether to adapt, hack, or ignore, and each choice has its own tradeoffs.

 

The problem are not that they are a big obstacle, but that this requires a lot of messy cruft in the code to support.

Just have a look at GZDoom's dehacked.cpp, the last 500 lines of it are virtually all fudging to force some totally alien concepts into the engine so that Dehacked works as intended.

And here's where the problem lies. The more features Dehacked gets the more cruft accumulates. Of course the big damage has always been there from the start. The mere presence of Dehacked made it impossible to  define a properly constructed class-based actor hierarchy.

 

 

[ketmar]

@kraflab of course it is possible. the end result will be a messy code with alot of corner cases and unnecessary checks all over the place. or, in other words, i have to undo most of the work on the engine code cleanup and sanity checks to support something that was designed around a concept of changing some internal tables in the binary code, the tables that don't even exist anymore. they are not "hidden" in any sense, they simply aren't there in any way. the very concept that DeHackEd was built around is not there anymore. everything is different, literally.

 

yet DECOLITE solution can be easily adapted to both engine types, and it is much easier to use by modders. it is superior in every possible way, not only for advanced engines, but for more vanilla-like engines too. and it would be even possible to reuse some simple ZDoom DECORATE mods in vanilla-like engines with it, with no or very little changes.

 

the only thing we need now is somebody willing to implement it. then we can create the specs, and once people will see how easy to mod the game with DECOLITE, they will never ever want to use DeHackEd anymore.

[ChopBlock223]

7 hours ago, kraflab said:

Would it cover the niche of people that want to make more advanced things but still have them playable in a prboom+-derived port? Would there still be the same demand for mbf22?

Speaking personally, it would well cover my needs/desires, I like doing advanced Decorate stuff with GzDoom (well, advanced for me, anyway), but Boom as a map format does most of what I need, and MBF21 improves on that format in helpful ways. A big appeal to me is that it's possible to record demos on the maps I make, which is what people want for speedrunning, and I want to make some speedrunnable maps.

 

I'm definitely looking forward to your future developments on these things. Improved DeHacked has been helpful for projects I'm involved in, and I appreciate that a lot even if I'm personally kind of intimidated and hesitant to try to work that kind of code on my own.

 

4 hours ago, Plank_Guy_89 said:

So it does seem misguided that the solution would be to build off of Decorate? 

I think that anything can work as long as it's sensible to implement for the engine, and that it's easy for a typical modder to read and write.

 

I suggested Decorate or something like it because I know it and works well for me. I may have had the idea that going that route would make implementation easier for ZDoom family ports which already support regular Decorate, though I'll freely admit that I have no idea if that notion makes any actual sense.

4 hours ago, ketmar said:

we are not talking about supporting the full GZDoom DECORATE (that's why i keep calling the thing DECOLITE, to avoid such confusion), but about the base extension architecture to build upon. the exact specs and features of DECOLITE can be fleshed out later. 

Yeah, exactly!

 

[GoneAway]

51 minutes ago, ketmar said:

@kraflab the end result will be a messy code with alot of corner cases and unnecessary checks all over the place.

It doesn't have to be that way. It never needs to be that way. The only limits are those you place upon yourself 🤠

 

Since I'm not about to implement it myself in these engines though, it's a pointless discussion. 🤷

[ketmar]

52 minutes ago, kraflab said:

It doesn't have to be that way. It never needs to be that way. The only limits are those you place upon yourself 🤠

sure. with infinite time and resources, i could rewrite the whole thing. but i don't know where to download neither infinite time, nor infinite resoures… and we are talking about the base architecture upon the whole engine built, not about some extra feature here.

 

52 minutes ago, kraflab said:

Since I'm not about to implement it myself in these engines though, it's a pointless discussion. 🤷

we simply tried to explain why we are so against DeHackEd, in any incarnation. but yeah, i guess that we're going to start a new circle, so it looks like a time to drop this off-topic.

 

i'd like to implement initial DECOLITE support in PrBoom+/UM, but sadly, i don't know when i'll have enough time and resources to do it. and i bet that Graf is even more busy. this makes me sad ketmar…

Edited September 15, 2021 by ketmar

[ChopBlock223]

1 hour ago, ketmar said:

i'd like to implement initial DECOLITE support in PrBoom+/UM, but sadly, i don't know when i'll have enough time and resources to do it. and i bet that Graf is even more busy. this makes me sad ketmar…

Any support at all for things like these from devs makes me optimistic, even if still just hypothetical!

[GoneAway]

Considering PR+UM is basically in maintenance mode and superseded by woof & dsda-doom (depending on what flavor of port you like), it probably wouldn't be the ideal place to put such a thing. At least, this doesn't seem like the right time to do such a thing. Decorate is on my to-do list for dsda-doom (probably the next thing after my current project), and most likely what I end up doing there would be portable to woof without too much headache, as that tends to be the case (if woof wants it).

[Graf Zahl]

If you port DECORATE please be aware that this needs to work on the actor/state data without Dehacked modifications. In ZDoom Dehacked is done afterward as DECORATE is also used for the stock actors. This may become a bit tricky with the unlimited stuff inserting itself at places you may not know when parsing DECORATE first

[LexiMax]

48 minutes ago, kraflab said:

Decorate is on my to-do list for dsda-doom (probably the next thing after my current project), and most likely what I end up doing there would be portable to woof without too much headache, as that tends to be the case (if woof wants it).

 

Would this be portable to ports that have a fixed-size state table, so long as the table was big enough for the mod you wanted to play?  What about non-Boom ports like Crispy Doom?

[ketmar]

1 hour ago, kraflab said:

Considering PR+UM is basically in maintenance mode and superseded by woof & dsda-doom (depending on what flavor of port you like), it probably wouldn't be the ideal place to put such a thing.

that's exactly why it may be a good place to start, to avoid chasing a moving target. and it could be portable to other ports too. but anyway, i don't think that i will have time to do it in the near future, so you will prolly finish implementing it way before i will even start. ;-)

[Syn]

I'm beyond thankful that I came into the community at such a golden age for modders. It really does inspire me to actually work towards getting better at all this stuff...

[GoneAway]

4 hours ago, AlexMax said:

 

Would this be portable to ports that have a fixed-size state table, so long as the table was big enough for the mod you wanted to play?  What about non-Boom ports like Crispy Doom?

Tbh I just don't know enough to answer something like this definitively right now 😅

[Graf Zahl]

7 hours ago, AlexMax said:

 

Would this be portable to ports that have a fixed-size state table, so long as the table was big enough for the mod you wanted to play?  What about non-Boom ports like Crispy Doom?

 

The state table's format is mostly the same in all ports up to DSDA, so it should be no problem to implement the same resizing mechanism.

 

But if you want to keep it static, what should a reasonable upper limit be? At some point the table will become too large anyway, so "unlimited" isn't really a thing. For example, some modder may set their index to something ridiculously high like 65535.

 

This is something I can deal with in GZDoom because I use a hashmap to index my states during parsing, but with a static table you now will have to allocate all 65536 states including all the unused gaps. The same applies to the other resources. For actors and sounds I actually cheated - I just define an entity like ~SomeGenericName~#index which I then look up by that name instead of having a huge array of unused stuff. But again, with a table based format you have to fill in all the gaps in the list.

 

With that in mind it should work if you use reasonable limits to avoid the reallocation but rest assured that sooner or later some map will break your limit - either by being careless or being too ambitious. I'd still expect most modders to just count up from the first free one and not put their things somewhere far away from the beginning. But you never know.

 

In any case, since this is all new and has not been put to use yet, maybe we can make it a little bit nicer and reduce the potential problems?

The thing about this spec that bugs me most are the sound and sprite arrays because they introduce an easily avoidable indirection.

Maybe we can just do away with them entirely. If the parser could be changed to optionally parse these as strings in addition to numbers we could just write

"sprite = POS2" or "Attack Sound = poshot" and remove the [SPRITES] and [SOUNDS] blocks entirely and just dynamically add content to the arrays when a new name is found. This not only would avoid arrays with gaps, it would also make the whole thing a lot easier to handle for those ports which no longer have these tables.