ACE Engine v2 - DOS Doom II [vanilla++?]

[Graf Zahl]

Just now, kgsws said:

That is what i'm worried about. I think new formats would have to come first, with conversion tools. Then editor developers could catch up.

 

I propose https://en.wikipedia.org/wiki/CBOR, it would allow any extensibility required while being easier to parse. And it offers features like knowing array size before parsing each element.

Also, most importantly, all the map data would not be shoved into a single lump, be it binary or text.

 

CBOR would also be used for states and actor definitions.

 

I'm even less in favor of binary formats for actor definitions than for maps. We should be beyond the state where data needs to be compiled before use.

It would make sense if the performance gain would be significant, but as things stand, the same applies here as for maps: The actual parsing is not the main driver in execution time here. Processing the data into internal structures is.

 

 

Just now, kgsws said:

Well, i did try adding UDMF support to ACE Engine - i had map loading sorted out, without BLOCKMAP.

While this might be caused by my questionable text parser, it more than doubled map loading time.

 

Well, parsing of the actual map data may be twice as long, depending on the quality of the parser.. But like I said, loading a map is not merely parsing. You won't be able to accelerate the rest.

 

 

Just now, kgsws said:

Overall it's painful to work with since i don't even know how many elements i have allocate (linedefs, sidedefs ...).

 

That's where dynamic arrays normally come in. You know, they are quite standard these days.

 

 

Just now, kgsws said:

It's not change for the sake of change. Well, for GZDoom, it is. But other source ports are still stuck in DEHACKED era.

 

Yes, they are, and on top there's DECOHACK to make it nicer to edit.

Which all has the nice property of being really universal. What you propose will require a gargantuan effort by virtually every player involved.

 

 

Just now, kgsws said:

To sum it up, i want to propose new, enhanced, editing feature-set that could potentially run on old hardware.

 

Then please define a baseline of where you want it to run.

 

13 minutes ago, OpenRift said:

Leave it to Graf to shit all over good ideas.

 

It doesn't strengthen your point if you just ignore or dismiss opinions that don't fit your view of things.

Just a little hint: If you don't get the port authors on board and listen to their feedback you will end up empty-handed.

 

 

[RataUnderground]

25 minutes ago, OpenRift said:

Leave it to Graf to shit all over good ideas.

Why is there always so much hostility towards him every time he appears in a thread? He just made a constructive comment with his opinion on what is practical and realistic. He is not discouraging anyone, And this is something I only see happening with him.

[OpenRift]

4 minutes ago, RataUnderground said:

Why is there always so much hostility towards him every time he appears in a thread? He just made a constructive comment with his opinion on what is practical and realistic. He is not discouraging anyone, And this is something I only see happening with him.

"Overall, I don't think this will work out," doesn't sound like being constructive. He just doesn't want his incumbent standard to be challenged, so he opts for faux-skepticism. 

 

He's also done this in of kgsws's earlier ACE works. Not a shred of "oh shit, this is actually really cool" or any sort of props to the crazy shit he's pulled off. No, he immediately goes into the logistics of if it's a viable standard. 

 

His whole outlook really is the antithesis of the Doom community's "we do it because we can" attitude to modding and does nothing but waste everyone's time with directionless criticism. It's not constructive at all.

[Andrea Rovenski]

I, for one, think this shit rules and it's motivating me to actually finish my dos computing setup :D 

[kgsws]

42 minutes ago, Graf Zahl said:

I'm even less in favor of binary formats for actor definitions than for maps. We should be beyond the state where data needs to be compiled before use.

Is there any issue with that? I mean, maps need compiled data (nodes), ACS has to be compiled, old texture definitions have to be "compiled".

Every modern doom editor can do all this.

 

42 minutes ago, Graf Zahl said:

Well, parsing of the actual map data may be twice as long, depending on the quality of the parser.. But like I said, loading a map is not merely parsing. You won't be able to accelerate the rest.

I'm not gonna defend quality of my text parser, no. But it sill is a time that could be reduced.

 

42 minutes ago, Graf Zahl said:

That's where dynamic arrays normally come in. You know, they are quite standard these days.

I know, and i want to avoid those. Again, for possibility of old hardware support.

 

42 minutes ago, Graf Zahl said:

Yes, they are, and on top there's DECOHACK to make it nicer to edit.

Sounds to me like something that has to be "compiled" from text to text.

 

48 minutes ago, Bauul said:

Out of curiosity, how much longer is "doubled map loading time"?

That depends on map size. I tested that a few moths ago so i can't say exactly, but it was like 3 seconds for older version of ACE00 (in my demo wad).

Edited June 8, 2023 by kgsws

[kgsws]

51 minutes ago, Graf Zahl said:

What you propose will require a gargantuan effort by virtually every player involved.

I'm not sure. Seems simple enough for me. But maybe it's just because i already have an idea how to implement it all.

 

51 minutes ago, Graf Zahl said:

Then please define a baseline of where you want it to run.

Well, since you are reading in "code execution for Doom2", it pretty match says something like 486 PCs (high end, though).

 

25 minutes ago, OpenRift said:

He's also done this in of kgsws's earlier ACE works. Not a shred of "oh shit, this is actually really cool" or any sort of props to the crazy shit he's pulled off. No, he immediately goes into the logistics of if it's a viable standard.

I mean, some things are better not to be touched. Like emulating code execution ... no way.

And, TBH, i would also not bother supporting something like lilith.pk3, so i understand that to a degree.

 

 

EDIT:

Maybe i should just code a demo for some source port and make conversion tool to show my proposed map format.

Edited June 8, 2023 by kgsws

[OpenRift]

1 minute ago, kgsws said:

I mean, some things are better not to be touched. Like emulating code execution ... no way.

And, TBH, i would not bother supporting something like lilith.pk3, so i understand that to a degree.

Of course. It doesn't need to support it. 

 

The way I see it, an ACE engine standard would exist as what I like to call "half-step" standards. A good example of this kind of standard would be the extended but not removed limits of EXE hacks like Doom32, where all the core static limits are raised, allowing you to play limit removing stuff like Scythe 2, just not insane stuff like Auger Zenith. It still has a place as a standard, but just not like a pillar standard like vanilla, Boom, MBF or MBF21. 

[JustAthel]

It would be kind of cool if there was something that could "emulate" the effects of what ACE does in the ports we have now that would need it (i.e. Chocolate-Doom having an ACE Engine compatibility option in its setup or a certain flag that pops up in any of the modern Boom ports if it detects any of the ACE lumps). There's no way that any of this isn't going to be used by some aspiring mod creator, even though the way we have it now kind of is a bit tacky for both the creator and end user. Even though a lot of what's here can actually run in a ZDoom port, having this run under pure vanilla with nothing other than a -file command is actually super cool and kind of stretches what we mean when we say something is "vanilla compatible".

[msx2plus]

this is really amazingly cool stuff, kgs. congratulations on the achievement.

 

55 minutes ago, OpenRift said:

Of course. It doesn't need to support it.

i believe what they mean is that if somebody took advantage of errors in their work in a similar way, they wouldn't feel the need to keep supporting that utilization of those errors, not that they are literally going to support lilith. in that way, they are acknowledging graf's viewpoints.

[Revae]

2 hours ago, Graf Zahl said:

we

So Graf is on board then!

[Darkcrafter07]

3 hours ago, kgsws said:

I am not sure why do you need that. ACE Engine does not support ZIPs.

If you want to separate maps, you can just create new WAD file in SLADE and copy only map data.

 

That, to me, looks like it's complaining about PNAMES/TEXTURE1 exploit. Indeed, TEXTURE1 indexes out of bounds.

Make a copy of my WAD file, then delete PNAMES and TEXTURE1. Just note that it will no longer work in DOS as those lumps contain code execution exploit.

Thank you for a quick response and a possible solution! Well, the zip is there so you could look at it.

 

[Redneckerz]

6 hours ago, kgsws said:

So, do you think there would be enough interest in new, radically different standard? I really mean radically different.

If you really want to set a standard here, you would have to talk to the people i mentioned previously. Kraflab was only able to get MBF21 going on due to having a significant part of the veteran community behind him and working with him.

 

For your idea to succeed you would have to do the same to see widespread support. Because you want to target older hardware, you have to define how old - You made ACE Engine for DOS. That puts your idea firmly in the vanilla school of philosophical thought - without ACE being actually vanilla.

 

You may want to get rid of that stigma. ACE is incredible, but to define a standard requires broader support.

4 hours ago, Gez said:

The problem will be getting both editor developers and port developers on board, and I don't really see that happening.

Gez, humor me. What would be the reason to doubt this? Kgsws is a known force in the community and repeately has proven itself to come up with new and innovative ideas that challenge the status quo.

 

MBF21 was built with the help of a seasoned crew of veterans - people who have deserved their spurs in this Doomy wild west - But so does Kgsws.

 

However, part of what makes MBF21 charming is how it has a base set of parameters defined - Choco Doom won't run MBF21, for instance. This ACE-defined standard seems more rooted towards vanilla and legacy parameters than source ports like Boom or MBF. It targets a different niche.

 

3 hours ago, RataUnderground said:

Why is there always so much hostility towards him every time he appears in a thread? He just made a constructive comment with his opinion on what is practical and realistic. He is not discouraging anyone, And this is something I only see happening with him.

It stems from years of happenings against GZDoom and Graf speaking out against other ports. As it stands Graf's opinion's have been strong in the past and are strong now, which inevitably leads to people tying that to character.

 

Not the best of ideals really. I wish i could change his onlook at certain things but at the end of the day, Graf's Graf.

Edited June 8, 2023 by Redneckerz
I can't type. This edit is proof.

[OpenRift]

@Redneckerz I couldn't have said it better myself. Well said.

[kgsws]

1 hour ago, msx2plus said:

i believe what they mean is that if somebody took advantage of errors in their work in a similar way, they wouldn't feel the need to keep supporting that utilization of those errors, not that they are literally going to support lilith. in that way, they are acknowledging graf's viewpoints.

Yes, exactly that. I would not support any mod deliberately abusing obvious bugs. Though, there's a thin line between bad abuse and good use.

 

38 minutes ago, Redneckerz said:

If you really want to set a standard here, you would have to talk to the people i mentioned previously. Kraflab was only able to MBF21 going on due to having a significant part of the veteran community behind him and working with him.

I understand, but it was pretty much required. DEHACKED alone was too limiting.

Now, the question is: do we need something better than constantly extending DEHACKED? Something that would be more on par with DECORATE, but much easier to parse and with clearly defined behavior?

 

43 minutes ago, Redneckerz said:

Because you want to target older hardware, you have to define how old - You made ACE Engine for DOS.

Well, that is intended to be a possibility. But yes, i would port this *something* to DOS, using my code execution. It would not be ACE Engine anymore.

 

49 minutes ago, Redneckerz said:

Gez, humor me. What would be the reason to doubt this? Kgsws is a known force in the community and repeately has proven itself to come up with new and innovative ideas that challenge the status quo.

TBH, i don't think i am that known. I am pretty inactive. Once in a while i come up with a crazy idea, do something, and then go inactive for years.

 

52 minutes ago, Redneckerz said:

MBF21 was built with the help of a seasoned crew of veterans - people who have deserved their spurs in this Doomy wild west - But so does Kgsws.

I mostly do "wow" style hacks, but are those always practical for everyone? Most things i did are only in ZDoom (and derivatives) code base anyway.

 

54 minutes ago, Redneckerz said:

However, part of what makes MBF21 charming is how it has a base set of parameters defined - Choco Doom won't run MBF21, for instance. This ACE-defined standard seems more rooted towards vanilla and legacy parameters than source ports like Boom or MBF. It targets a different niche.

I don't think i understand this.

What i am proposing is no longer ACE related. I am proposing new standard that goes beyond Boom and MBF. Closer to ZDoom, while making it easier to implement and with a fresh start to avoid compatibility bloat. (that does not mean i want to remove old features)

Yes, i would use code execution to port this to DOS, but that's not what it stems from.

 

 

So, basically, main question is: do we need DEHACKED successor?

Any ideas how to get opinions of veteran community?

[Dark Pulse]

3 hours ago, RataUnderground said:

Why is there always so much hostility towards him every time he appears in a thread? He just made a constructive comment with his opinion on what is practical and realistic. He is not discouraging anyone, And this is something I only see happening with him.

Because it's "cool" to shit on someone's hard work when you don't know dick about how much work it would take to make a cool idea actually functional.

 

I might not agree with Graf on everything, but shitting on him just because he goes "no, it's not that simple" is just blind and stupid. People should at least listen to what he says and have a counterpoint if they really think strongly enough about an idea to actually press him on it.

 

And either way, in the end, it's also his port. He doesn't owe anyone anything - he's going to work on what he thinks is cool to do.

Edited June 8, 2023 by Dark Pulse

[LexiMax]

4 hours ago, kgsws said:

I propose https://en.wikipedia.org/wiki/CBOR, it would allow any extensibility required while being easier to parse. And it offers features like knowing array size before parsing each element.

 

If we're bikeshedding over a standard, I think that something with a schema like Protobuf3 would be a much better fit.  The availability of schemas actually makes writing serialization code much easier for the supported languages and platforms, but the protocol is also simple enough to be parsed by hand without one.  There are a few things I'm not a fan of (missing fields are assumed to be 0, varint encoding is not fast as designed), but i tried writing my own protobuf-alike for a work project and in retrospect wish I hadn't.

Edited June 8, 2023 by LexiMax

[kgsws]

59 minutes ago, LexiMax said:

If we're bikeshedding over a standard, I think that something with a schema like Protobuf3 would be a much better fit.  The availability of schemas actually makes writing serialization code much easier for the supported languages and platforms, but the protocol is also simple enough to be parsed by hand without one.  There are a few things I'm not a fan of (missing fields are assumed to be 0, varint encoding is not fast as designed), but i tried writing my own protobuf-alike for a work project and in retrospect wish I hadn't.

I have good experience with CBOR but it is just an option i like. I have never worked with protobuf so i don't know all the features and limitations.

 

Here are rough ideas how would new format work like. Everything is a subject of revision.

Also, i am gonna represent CBOR in readable JSON format as conversion between those formats is ideal.

[
	{
		"name": "newMonster",
		"health": 1234,
		"speed": 8,
		"comment": "Anything will do ..."
	}
]

Values not required can be ignored. Like comment field in my example.

It is trivial to add new fields in future versions. Old versions would simply ignore unknown fields.

 

Now, what i like on CBOR is that i can store part of it for later. This example represents a single frame:

{
	"sprite": "TNT1",
	"frame": 0,
	"tics": 4,
	"action": "do_some_stuff",
	"arguments":
	[
		"whatever": 42,
		"more": true,
		"even": "text is possible",
		"math":
		[
			"health",
			"*",
			8,
			"+",
			7
		]
	]
}

What i can do when pre-processing frames is to find out exact bounds of arguments array (first and last byte) and store this binary for later.

This stored portion alone is still a valid CBOR.

Functions could then extract required arguments on-the-fly, when called. (i know, it is technically slower, but that is just an example)

And yes, result of math variable would be calculated dynamically.

 

I think all this allows great flexibility and extendability for future. And potentially custom, port-specific, feature set.

 

There are possible optimizations.

Keys could be pseudo-hashed and stored as integer for faster deserialising.

Various values can be stored as integers in final CBOR, but presented as strings in "scripting" UI. For example, TNT1 could be stored as 827608660.

And, of course, function arguments could be allocated specifically for each function and parsed right away.

 

Again, those are just ideas at the moment. There are many things to discuss before settling for a new standard.

I could code everything right away in *some random* form. But i would rather make a good standard with community support.

 

Maybe we should move out of "ACE Engine" thread with this ...

[Blzut3]

4 hours ago, JustAthel said:

It would be kind of cool if there was something that could "emulate" the effects of what ACE does in the ports we have now that would need it (i.e. Chocolate-Doom having an ACE Engine compatibility option in its setup or a certain flag that pops up in any of the modern Boom ports if it detects any of the ACE lumps). There's no way that any of this isn't going to be used by some aspiring mod creator, even though the way we have it now kind of is a bit tacky for both the creator and end user. Even though a lot of what's here can actually run in a ZDoom port, having this run under pure vanilla with nothing other than a -file command is actually super cool and kind of stretches what we mean when we say something is "vanilla compatible". 

Once arbitrary code execution is achieved it's no longer "pure vanilla."  Otherwise everything is vanilla and "vanilla compatible" loses all meaning and would be impossible to achieve (due to possibility of conflicting features).  At least not without being DOSBox.  This is why it's out of scope for Chocolate Doom.  Now other ports could choose to implement ACE engine compatible features, but this was always possible just as ACE engine has implemented GZDoom compatible features (thus far).

 

I am impressed with how much kgsws managed to implement in a relative short period of time though.

[Professor Hastig]

My thoughts about this proposal:

 

As the most important thing to define here, what's the proposed minimum hardware specs it is supposed to work on? This alone will set up lots of important boundaries of what can be done and what can not.

Creating a new map format because UDMF parsing is to slow? My suggestion would be to optimize the parser, and if you cannot do it by hand use a tool like re2c to help create one. It is very versatile and especially for a structurally simple format like UDMF might help a lot speeding things up. This would kill two birds with one stone. Not only won't you have to define something new, you get editor support for free by doing it. Any new format, either text or binary, will inevitably face an uphill battle to gain sufficient support, especially when combined with a new line action system as it was proposed.

 

The most interesting part here is definitely the actor definition system. Currently we do not really have any good cross-port options here. Dehacked is quite the mess, and even with DECOHACK we are limited by what Dehacked can achieve.

 

In my opinion redoing the line action system sounds interesting on paper but probably won't go anywhere in a world where even Hexen-style parameters are frequently viewed as inconvenient. How are these supposed to look? Without any ideas to discuss it's all a bit abstract.

[Untamed64]

Am I doing something wrong?

Do I have the wrong version?

[Gez]

9 hours ago, Redneckerz said:

Gez, humor me. What would be the reason to doubt this? Kgsws is a known force in the community and repeately has proven itself to come up with new and innovative ideas that challenge the status quo.

If you want to define a standard, you need to convince everyone. That's the problem.

 

As you know, the Doom community history is full of proposed standards that went nowhere and were forgotten. UDMF is one of the only success stories we have and that was mostly because it was backed by both Graf and Quasar on the port front, and CodeImp on the editor front. There had been text-based map formats before, as early as 1994, but they were ignored. As an amusing aside, Doom was released in December 1993, UDMF was finalized in May 2008, and we're now in June 2023. The UDMF standard (and so the computers it was meant to run on) are now as old today as the original Doom (and its target hardware) was when UDMF was designed.

 

MBF21 is not a brand-new standard, it's an extension of a de facto pre-existing standard. Any port that supports DEHACKED, and that's all the ports worth mentioning, can support MBF21 by extending a bit their existing DEHACKED loader. And the success of DEHEXTRA, which was a previous extension of DEHACKED with much more limited ambitions, showed that there was an appetite for DEHACKED extensions.

9 hours ago, kgsws said:

Closer to ZDoom, while making it easier to implement and with a fresh start to avoid compatibility bloat.

Problem is that it would induce more compatibility bloat because now there's a brand-new standard to support.

[Professor Hastig]

12 minutes ago, Gez said:

MBF21 is not a brand-new standard, it's an extension of a de facto pre-existing standard. Any port that supports DEHACKED, and that's all the ports worth mentioning, can support MBF21 by extending a bit their existing DEHACKED loader. And the success of DEHEXTRA, which was a previous extension of DEHACKED with much more limited ambitions, showed that there was an appetite for DEHACKED extensions.

 

 

I think MBF21's big appeal is that it didn't require extensive changes, so it was feasible even for conservative ports.

A well defined scope is often the first step to success.

 

 

13 minutes ago, Gez said:

Problem is that it would induce more compatibility bloat because now there's a brand-new standard to support.

 

Yeah, that was my first thought as well. And history quite clearly shows that often this won't work and the new "standard" ending up a footnote in history because it didn't bring anything compelling to the table. The ideas being proposed here sound interesting but the chance of success would be magnitudes higher if reinventing the wheel wasn't at the core of them.

 

[Dexiaz]

1 hour ago, Untamed64 said:

Am I doing something wrong?

Do I have the wrong version?

 

ace.wad? Wasn't it named ademo0.wad? I mean, what is your command line, since you've changed the wad name?

[Untamed64]

3 minutes ago, Dexiaz said:

 

ace.wad? Wasn't it named ademo0.wad? I mean, what is your command line, since you've changed the wad name?

There's ace.wad and ademo1.wad

Aren't you meant to run both or something? Couldn't find instructions, works fine on GZDoom, but wanted to try it out in vanilla

[Redneckerz]

11 minutes ago, Untamed64 said:

There's ace.wad and ademo1.wad

Aren't you meant to run both or something? Couldn't find instructions, works fine on GZDoom, but wanted to try it out in vanilla

It is mentioned in the release post. Just doom2.wad -file ademo1.wad.

[Untamed64]

7 minutes ago, Redneckerz said:

It is mentioned in the release post. Just doom2.wad -file ademo1.wad.

Ok, tried that

[LuciferSam86]

Yeah , would be nice having a GitHub with the draft of this idea so we can discuss there 

[Darkcrafter07]

@Untamed64 you should download doom2.exe v1.9 here and try again: 

 

[kgsws]

5 hours ago, Untamed64 said:

Am I doing something wrong?

Do I have the wrong version?

Just from a screenshot i see it's a wrong exe version. Correct version has title text in RED.

Also mentioned, you have to run doom2.wad -file ademo1.wad command.

 

 

6 hours ago, Professor Hastig said:

In my opinion redoing the line action system sounds interesting on paper but probably won't go anywhere in a world where even Hexen-style parameters are frequently viewed as inconvenient. How are these supposed to look? Without any ideas to discuss it's all a bit abstract.

How inconvenient? I find hexen-style lines way better than doom style lines.

 

6 hours ago, Professor Hastig said:

The most interesting part here is definitely the actor definition system. Currently we do not really have any good cross-port options here. Dehacked is quite the mess, and even with DECOHACK we are limited by what Dehacked can achieve.

Yes, i think that has priority over new map format right now.

[Untamed64]

42 minutes ago, Darkcrafter07 said:

@Untamed64 you should download doom2.exe v1.9 here and try again: 

*snip*

 

2 minutes ago, kgsws said:

Just from a screenshot i see it's a wrong exe version. Correct version has title text in RED.

*snip*

Cheers, it works now