DOOM relies on undefined link behavior

[Quasar]

kb1 said:

Yeah, that's basically how to handle it. You create a structure of game option variables. Then, make a array of those, and use one element for the engine's running set, and one for the menu set. When a demo's running, you use the menu set. When in a game, use the live set. When the demo quits, copy the 'menu' version to the live version. When a demo starts, set the live version as needed for the demo. It's not that bad to manage. Beefing up this structure can lend to easy option saving/loading, as well as live option changing across network games.

In fact the BOOM system of having default_* versions of the sync-critical variables was intended to make this very kind of thing possible. It's just that fraggle kind of threw it by the wayside while writing the SMMU console >_>

However I have already managed to change this. As of a couple nights ago, Eternity can now edit sync-critical options while demos are running, and even displays the default values rather than the current values when the variables are queried via the console or displayed on the menu.

[myk]

printz said:
But hopefully with this freedom it will have more time to implement original features, rather than mimic (bad word... let's say implement) ZDoom's features and stay in shadow.

Actually, since it wouldn't have an objectively different focus (versatility plus compatibility instead or ZDoom's more pure versatility) it would mimic it even more. Especially considering that, other than for this basic difference (backed by their licenses*) the engines tend to try to support or reproduces each other's capacities. As it stands, Eternity's charm and niche is some guarantee of purity, and the ability to combine more modern features with old functionality, such as having a console and running a vanilla demo, or loading a level dynamically and playing it with the more traditional consistency, as well as supporting old and new editing features with the least conflict possible.

* ZDoom's license mix is looser and incompatible with that of the main "purist" ports (PrBoom, Odamex and Chocolate) while Eternity's is compatible with those and more formal or strict in order to ensure it's free software.

[Quasar]

myk said:

Actually, since it wouldn't have an objectively different focus (versatility plus compatibility instead or ZDoom's more pure versatility) it would mimic it even more. Especially considering that, other than for this basic difference (backed by their licenses*) the engines tend to try to support or reproduces each other's capacities. As it stands, Eternity's charm and niche is some guarantee of purity, and the ability to combine more modern features with old functionality, such as having a console and running a vanilla demo, or loading a level dynamically and playing it with the more traditional consistency, as well as supporting old and new editing features with the least conflict possible.

I think you really get it :)

[Gez]

kb1 said:

No need to feel sorry if you don't agree. But, my statement still stands.

No it doesn't. Sure, if your port in inherently vanilla-demo-compatible, you can use vanilla-demos to find that bugs exist. However, if you think just running a demo is enough to figure out where the bug happens, and that it would save long sessions of comparing code with older versions, you are sorely mistaken. You just gotta read the SVN log for Eternity when it was being implemented to see it can be frustrating.

And even then, it's only a useful tool when running in demo-compatible mode. Say that "something feels wrong" but only when you play with a number of options set to different values from vanilla-demo-compatibility standards. What do you do, run vanilla demos which work perfectly and tell you your program is therefore 100% correct and flawless?

Relying on demo compatibility is not useful for ports which are developing new features and fixing old bugs, whether they keep a demo-compatibility mode or not.

[Quasar]

Gez said:

However, if you think just running a demo is enough to figure out where the bug happens, and that it would save long sessions of comparing code with older versions, you are sorely mistaken. You just gotta read the SVN log for Eternity when it was being implemented to see it can be frustrating.

You'd probably be referring to the time a brief while ago that I restored partial 2.02 demo compatibility to EE ;) And yes that was, relatively speaking, quite a nightmare. However, some of the "gadgets" I developed during that episode, such as the mobj position log, should make busting future problems of the same nature significantly less hassle :)

[kb1]

Gez said:

No it doesn't. Sure, if your port in inherently vanilla-demo-compatible, you can use vanilla-demos to find that bugs exist. However, if you think just running a demo is enough to figure out where the bug happens, and that it would save long sessions of comparing code with older versions, you are sorely mistaken. You just gotta read the SVN log for Eternity when it was being implemented to see it can be frustrating.

And even then, it's only a useful tool when running in demo-compatible mode. Say that "something feels wrong" but only when you play with a number of options set to different values from vanilla-demo-compatibility standards. What do you do, run vanilla demos which work perfectly and tell you your program is therefore 100% correct and flawless?

Relying on demo compatibility is not useful for ports which are developing new features and fixing old bugs, whether they keep a demo-compatibility mode or not.

I never said it was a "catch-all" answer to solve all problems. But, it's still Doom, even if a number of options are not set to vanilla. It's still going to be Doom - with some changes. The process of getting your port vanilla-demo-compatible forces you to build tools and functionality that becomes essential above and beyond it's original purpose. Try it.

Quasar said:

You'd probably be referring to the time a brief while ago that I restored partial 2.02 demo compatibility to EE ;) And yes that was, relatively speaking, quite a nightmare. However, some of the "gadgets" I developed during that episode, such as the mobj position log, should make busting future problems of the same nature significantly less hassle :)

Yes, I built a very similar thing for my port, that tracked XYMovement, Spawn, and, most important of all, the changes to the random generator index. By far, this is the most important check of all. If you pass a unique indetifier to the P_Random functions, it tends to provide a log of where the code goes, very efficiently.

[Gez]

kb1 said:

I never said it was a "catch-all" answer to solve all problems. But, it's still Doom, even if a number of options are not set to vanilla. It's still going to be Doom - with some changes.

You're not really making much sense. Either that or you're not replying to the point I had made.

kb1 said:

The process of getting your port vanilla-demo-compatible forces you to build tools and functionality that becomes essential above and beyond it's original purpose. Try it.

No. I don't like watching demos anyway.

Perfect compatibility demands that the efforts invested go above and beyond, too. Chocolate Doom, for example, has code to emulate access violation issues. If you want perfect vanilla-demo-compatibility, it doesn't just force you to build tools and functionalities -- it also force you to have the port's design go in one direction, and all others are barred. It prevents you from doing certain things that would break compatibility forever. And given that Choco and PrB+ already do a practically perfect job at this task, why bother competing with them?

[Graf Zahl]

kb1 said:

The process of getting your port vanilla-demo-compatible forces you to build tools and functionality that becomes essential above and beyond it's original purpose. Try it.

What a pile of nonsense. I have to concur with Gez. Maintaining demo compatibility requires a completely different approach to code maintenance than an engine with modern features as main focus that doesn't care much about demos.

Up to a certain degree it can be mixed but once you cross a certain threshold you have two options:

- dump demo compatibility in favor of your features or vice versa
- add duplicate code paths for certain functions: one that maintains the original behavior and one that fixes the quirks that make it unsuitable for expansion to be used with the more modern stuff.

Demo compatibility won't ever help you to make the engine more robust. On the contrary. To have demo compatibility you have to maintain each bug, quirk and glitch in 100% original condition - a nightmare when these get into the way of making the engine more robust.

I think the subject of this thread is just a perfect example: Here we have some undefined behavior that should be plugged with a proper fix. A proper fix, however, would necessitate rewriting a crucial part of the inner engine workings - a big no-no for demo compatibility.

[myk]

Gez said:
You're not really making much sense. Either that or you're not replying to the point I had made.

It made sense to me. How do you think ZDoom gained all those new compatibility options recently (as well as some of the previous ones)? It's mostly work coming from the more "pure" ports, the coders that studied the stuff and the players that used them or vanilla. Without them, Graf would have had neither the demand nor the input to fix them. Even if your argument about taking up more time due to compatibility is true (which is not clear, because as kb1 implies, one way to steer clear of breaking "Doom behavior" is to have it clearly established in code and in mind) it doesn't invalidate the effect of that compatibility. It should be worthwhile even to take a bit more time, if the result is more satisfactory.

And given that Choco and PrB+ already do a practically perfect job at this task, why bother competing with them?

It's not competing, it's cooperating and getting something from them. Quasar doesn't need to worry about all the work going into compatibility. He may find some new insights or report occasional bugs, but otherwise all he needs is to implement suitable parts of massive work done along-side what Eternity has that those engines don't, which is a nice combination. You seem to underestimate how much people like to have a single engine to meet most of their needs (see Dragonsbrethren's post) and fail to notice that without this vanilla-friendly trait, it would more or less be competing with ZDoom's role, which is already filled quite well. Instead, it is also cooperating with ZDoom, by having a somewhat different function.

[Graf Zahl]

myk said:

You seem to underestimate how much people like to have a single engine to meet most of their needs.

Understandable, yes. It sure makes things easier but in the end a setup with one engine to fit most of the playing needs and another one to fit all the demo needs works as well for me.

Where it does get annoying is with the people who demand one engine to fit all their needs though. And it's mostly these who work up an unreasonable hatred towards those engines which have to make sacrifices to do what they want to do.

[Gez]

myk said:

Without them, Graf would have had neither the demand nor the input to fix them.

In other words, there would not have been a solution to a problem that would not have presented itself. :p

[myk]

That's like saying purists wouldn't have problems with ports had the source never been released :p

Sure, some extra work needs to be done to accommodate all our different preferences and tastes, but the purists pushing for more vanillaism in ZDoom also helped locate various bugs that allowed ZDoom to play yet more of the older WADs without issue, for example.

What Graf says about player demands is true, it can cause tension and annoyances, but devs are also capable enough these days to set limits to what they support, depending on the overall focus of their engine.

[Quasar]

Graf Zahl said:

- add duplicate code paths for certain functions: one that maintains the original behavior and one that fixes the quirks that make it unsuitable for expansion to be used with the more modern stuff.

I am toying with the idea of creating a comprehensive compatibility overlay system which will remove most of the grievous cases of branching on demo_version and replace them with calls into a function-pointer-bearing structure. This would certainly offer a much cleaner approach to supporting versioned behaviors than what BOOM team came up with.

Graf Zahl said:

I think the subject of this thread is just a perfect example: Here we have some undefined behavior that should be plugged with a proper fix. A proper fix, however, would necessitate rewriting a crucial part of the inner engine workings - a big no-no for demo compatibility.

Right, and it is still a dissonance for me. To have things potentially corrupt critical linked lists is bad. But fortunately the behavior in question rarely comes into play, as it could only be triggered in a demo via use of a bad DeHackEd patch, which almost certainly would have crashed the vanilla engine too in some circumstances.

Also I did find a satisfactory solution in EE. As long as we're running with mobj reference counting going, the things cannot be deleted while mobj_t's are still using them in a blockmap traversal, so nullifying the links is actually unnecessary and creates more problems than it solves (and simply saving the next pointer during the iteration is insufficient, as two or more things could potentially remove themselves from the list simultaneously from the iteration's point of view - fixing this would require adjusting an external iterator as is done for the master thinker_t list).

So, it's enough to simply set NOBLOCKMAP|NOSECTOR on things in P_RemoveMobj so that they cannot corrupt the blockmap and sector lists by calling P_UnsetThingPosition twice in a row. Though for BOOM, old_sectorlist must still be nullified so that the msecnode freelist doesn't get thrown away. This doesn't cause any side effects that I am aware of, however.

[kb1]

Gez said:

You're not really making much sense. Either that or you're not replying to the point I had made.

No. I don't like watching demos anyway.

Perfect compatibility demands that the efforts invested go above and beyond, too. Chocolate Doom, for example, has code to emulate access violation issues. If you want perfect vanilla-demo-compatibility, it doesn't just force you to build tools and functionalities -- it also force you to have the port's design go in one direction, and all others are barred. It prevents you from doing certain things that would break compatibility forever. And given that Choco and PrB+ already do a practically perfect job at this task, why bother competing with them?

Yeah, I drift a bit sometimes. Anyway, there's nothing preventing a vanilla-compatible port from doing many great non-vanilla things, it's just that, it requires a dicipline and a mindset to be able to cleanly separate the new and the old. Difficult, of course, but quite possible, and not without good benefit. Eternity, PrBoom (+) and others proves that.

Graf Zahl said:

What a pile of nonsense. I have to concur with Gez. Maintaining demo compatibility requires a completely different approach to code maintenance than an engine with modern features as main focus that doesn't care much about demos.

Up to a certain degree it can be mixed but once you cross a certain threshold you have two options:

- dump demo compatibility in favor of your features or vice versa
- add duplicate code paths for certain functions: one that maintains the original behavior and one that fixes the quirks that make it unsuitable for expansion to be used with the more modern stuff.

Demo compatibility won't ever help you to make the engine more robust. On the contrary. To have demo compatibility you have to maintain each bug, quirk and glitch in 100% original condition - a nightmare when these get into the way of making the engine more robust.

I think the subject of this thread is just a perfect example: Here we have some undefined behavior that should be plugged with a proper fix. A proper fix, however, would necessitate rewriting a crucial part of the inner engine workings - a big no-no for demo compatibility.

Hmmm. "...an engine with modern features as main focus that doesn't care much about demos".

How about this: "...an engine with modern features as main focus that does care about demos". Anything wrong with that? Is that "a pile of nonsense" too?

The point I'm trying to get across is that it's possible to do both. And there are benefits with that approach.

A question for those posting: This isn't a "favorite port" discussion is it? Seems like it might be.

For the record, I have respect for any and every port that will at least run, say, Episode 1 reasonably well. If your port doesn't show the E1 demos, and that's a deliberate choice, I can see past it. And yes, it does feel a bit incomplete to me. But, I understand it, and the reasonings behind it.

Quasar mentioned his frustration about having to rely on misbehavior. I agree, it's frustrating. Some others commented that continuing to maintain demo sync was not necessary, and that they were actually encouraged by the prospect of the pending new features. In light of that discussion, I simply wanted to offer my honest opinion that, by ditching compatibility, something would be lost.

My experience starts with a non-compatible codebase. Originally, I thought that compatibility was too limiting, too-difficult, too messy. Gradually, the codebase started to feel less and less like Doom.

My ideology slowly started to shift towards embracing the benefits of the discipline of demo compatibility, and, with it the nostalgia returned. That's a cool feeling when you get it right, and difficult to describe on a forum. I hope I did it justice.

Quasar: Interesting idea on the function pointer structure. Where would you make the split? In other words, you could go function for function, all the way to 2 completely separate code paths with their own renderers if you were feeling particularly sadist... :)

Of course, the larger the split, the more efficient, to a point. I'd be inclined to consider code cleanliness above efficiency at some point.

[Graf Zahl]

kb1 said:

How about this: "...an engine with modern features as main focus that does care about demos". Anything wrong with that? Is that "a pile of nonsense" too?


The point I'm trying to get across is that it's possible to do both. And there are benefits with that approach.

What benefits aside from the fact that you need to maintain crufty code that has to do a lot of compatibility checks, code duplication and other strange things to satisfy both sides?

I know what hoops Quasar had to jump through and in a few situations it also meant not doing some non-critical fixes that would benefit the modern features in favor of demo compatibility to keep the code manageable.

In the end demo compatibility and new features are things that are constantly at odds with each other.

kb1 said:

My ideology slowly started to shift towards embracing the benefits of the discipline of demo compatibility, and, with it the nostalgia returned. That's a cool feeling when you get it right, and difficult to describe on a forum. I hope I did it justice.

You still haven't pointed out the benefits aside from the fact that the engine can play old demos.

[myk]

Graf Zahl said:
In the end demo compatibility and new features are things that are constantly at odds with each other.

They require some extra work (which may be of use later, and may be something a developer enjoys in the end) to be put alongside each other, but the idea that they're opposed befits those people who are constantly trolling each other on whether new features or purism is better and should be "the way of the community".

You still haven't pointed out the benefits aside from the fact that the engine can play old demos.

I don't know if he said it, but this also implies you can run old WADs and offer traditional game play. Looser compatibility (which,as stated, comes about only because the stricter one is developed) often works for WADs, as well, and satisfies some game play concerns, but not as much.

Not sure what's making you want Eternity to be another ZDoom. I would argue Eternity's greater care over these aspects is beneficial to you as a developer since it helps define ZDoom's role in the community, instead of having two ports trying to do the same thing, conceptually. It's not like you have to do the same thing in G/ZDoom. Some dude may come to you once in a while and say "nuh uh, Eternity is better because it can run vanilla demos"; just remind them that, while it's up to them what they prefer, that is a reason why ZDoom can add features quicker and more freely.

Keep in mind Eternity is the way it is also because Quasar likes this sort of thing and combination. You can even see him hacking away at the beta binaries in another thread, and he's stated admiration for Killough, who kick-started the idea of a port that could add extra features yet maintain a high degree of compatibility.

[Graf Zahl]

myk said:

They require some extra work (which may be of use later, and may be something a developer enjoys in the end) to be put alongside each other, but the idea that they're opposed befits those people who are constantly trolling each other on whether new features or purism is better and should be "the way of the community".

Keeping demo compatibility prohibits you from doing certain reorganizations and optimizations, even those that have no real impact on how the game runs.

One example: ZDoom orders its thinker list differently in order to ensure that a few things work in a well defined fashion - which is needed to avoid glitches. With an unordered thinker list that's required for demo compatibility that's simply not possible - unless you duplicate all the related code which is also not a good option here.

Of course keeping demo compatibility ensures that things continue to work like you want but that's beside the point. ZDoom has some things that wouldn't work that well with Doom's original thinker list - at least not without some performance hit.

[myk]

Yes, that explains why ZDoom is the way it is. I'm not saying you're one of those people (that troll each other) but that, with some compromises and workarounds, the two stances complement each other, either appearing focused in different specialized engines separately (Chocolate vs. GZDoom) or more or less merged within one with demos and all (PrBoom or Eternity being good examples to different degrees.)

Graf Zahl said:
Of course keeping demo compatibility ensures that things continue to work like you want but that's beside the point.

Indeed, I don't personally need more ports to do what I can already do. I just see what Quasar is onto. It's not like I think ZDoom should add vanilla demo compatibility. Some people may have added it to their ZDoom wish list, but I think it's pointless considering ZDoom's direction and construction. Eternity has had Doom compatibility since day one, on the other hand. Randy Heit got rid of it almost immediately, a dozen years ago!

[Moriarti]

I feel like pointing out that the released Doom source code was not compatible with v1.9 demos, either. It took some time before we started seeing ports that could play them back without desyncing, and only because somebody went through the trouble of figuring out what had been changed between v1.9 and v1.10. So a new port starting out in late 1997/early 1998 did not automatically have demo compatibility either.

[kb1]

myk describes my stance rather eloquently. I would add a couple points:

Vanilla-compatibility probably would hinder ZDoom at this point. Dismissing compatibility early allowed a great deal of streamlining, and, in fact, complete re-engineering of major critical systems, to the point that tacking on compatibility code at this point might be very ugly indeed.

And, there are efficiency issues with compatibility code. A mis-predicted branch on modern CISC processors with deep pipelines is murder, and, of course, branching is what were talking about. Going the "function pointer" route may alleviate some of that, and may not.

My unreleased port actually started it's life as ZDoom 1.11, from which I proceeded to add all manner of funky features and options, with no regard to compatibility. That continued for years, until I ended up with some really fun shit, but it wasn't exactly Doom.

I finally came to the conclusion that, if I wanted the game to feel right, I needed to "get back to the roots". To do this, I had to work backwards - putting back compatibility code...to get the old and new features to feel right!

To state that another way, I propose that, making your game vanilla-compatible will assist in making your new features feel right. A strange statement, indeed, but that's been my experience.

Graf started a thread asking for suggestions on which ZDoom behaviors could be improved to make ZDoom feel more "Doomy". But this would be largely unnecessary had ZDoom adopted a vanilla-friendly mentality early on. IMHO, ZDoom is just fine as-is - it is what it is, and there's plenty of nice, fun ZDoom mods that rely on it's behaviors.

Graf Zahl said:

You still haven't pointed out the benefits aside from the fact that the engine can play old demos.

To summarize the benefits (beneficial for me, at least):
. Play old demos. duh.
. Play old wads as intended (whatever that means)
. Creates a framework that, by it's very nature, forces unintended gameplay changes to rear their ugly heads, as demo desynchs
. Creates a framework that, by it's very nature, encourages the programmer to supply configuration switches to toggle intended gameplay changes on/off.
. Creates a "self-documenting", code-searchable catalog of custom gameplay changes. (Search for "if (compatibility ...)
. Encourages new feature creation to be written in a demo/network compatible way.
. And, it plays old demos!!

So, yeah, probably doesn't make sense for ZDoom. But, if ZDoom were vanilla-demo-compatible, I guess all the programmers would delete their code and just use ZDoom. :)

[printz]

Gez said:

And given that Choco and PrB+ already do a practically perfect job at this task, why bother competing with them?

Why not bother? With a few more compatibility optimizations, such as getting rid of the last remaining Boom-born inconsistencies (weapon autochanging having priority over further ammo updates that make it unnecessary, for instance), and implementing a compatibility system akin to PrBoom's and ZDoom's, Eternity could completely cover & replace Choco and PrBoom(+) for me. Oh, and it should be crossplatform and have a Mac build too :>