Doom and Strife source code restoration

[OpenRift]

47 minutes ago, nukeykt said:

relative to DOBUILD.BAT. In my setup I have 2 folders: one for watcom and one for doom.

 In watcom folder I have 9.5b + TASM.EXE in BINW folder. In doom folder I have 2 folders: doom tree and dmx.

 Watcom is mounted on C drive and Doom on D drive.

 Thus DOBUILD.BAT is located at D:\DOOM\DOBUILD.BAT, dmx_r.lib is at D:\DMX\DMX37\LIB\DMX_R.LIB

I've quadruple checked over doing these exact instructions and I'm still getting the same error.

 

Are there any settings I should have set in Watcom or certain parameters I should use when mounting either drive? I'm using DOSBox-X for this btw if that's any help.

[nukeykt]

I used vanilla dosbox 0.74-3 myself. Maybe you need reset disk cache in dosbox? (ctrl+f4)

 

EDIT:

btw, which revision of Doom you're building?

Edited February 8, 2022 by nukeykt

[OpenRift]

9 hours ago, nukeykt said:

I used vanilla dosbox 0.74-3 myself. Maybe you need reset disk cache in dosbox? (ctrl+f4)

  

EDIT:

btw, which revision of Doom you're building?

I've been trying to build The Ultimate Doom's EXE. Maybe it has something to do with your DOSBox settings?

Edited February 8, 2022 by OpenRift

[nukeykt]

6 minutes ago, OpenRift said:

I've been trying to build The Ultimate Doom's EXE. Maybe it has something to do with your DOSBox settings?

I don't recall changing any setting in my config file. DM me in discord, you can find me in chocolate doom server

[NY00123]

35 minutes ago, OpenRift said:

I've been trying to build The Ultimate Doom's EXE. Maybe it has something to do with your DOSBox settings?

 

One thing which should be done is preparing the relevant environment variables for using Watcom, like %WATCOM% or %LIB%. I probably assumed that writing the details was out of scope of the documentation for gamesrc-ver-recreation submodules, as it's more related to getting Watcom ready for use. On a second thought, I won't be surprised if out of the ones experimenting with the code, a relatively large percentage will try Watcom for the very first time (at least in a while).

 

Secondly, looks like I missed this:

 

On 2/6/2022 at 8:49 PM, Gibbon said:

Still. I think a GPL port should at least be buildable with open source and free compilers. It kind of defeats the point of it if you need to use proprietary and 'questionably' obtained compilers to build it.  DJGPP and it's assembler would be nice and the porting process to it wouldn't be too difficult.

 

I can see the desire in using a GPL-compatible toolchain for building a GPL-compatible codebase. The scope of gamesrc-ver-recreation implies that the original tools have to be used (for the purpose of near-perfect binary exe recreations), so there isn't a great deal of choice; Actually, there's often no choice at all.

 

The sources are still there, so anybody who's interested can try to adapt them for a different toolchain. There's also Open Watcom. It's currently still not available under GPL-compatible terms, albeit I've recently found the following discussion with a proposal to change the license: https://github.com/open-watcom/open-watcom-v2/discussions/271

 

On 2/6/2022 at 8:49 PM, Gibbon said:

Also, doesn't the original EULA apply here?  Decompiling the original exes (which are still sold) I think is not totally legal.  A clean room reverse engineer is better.  I mean, this would be like decompiling and making a GPL binary for Doom 2016.

 

Blzut3's answer should probably cover most of it. That's also one of the reasons why, ignoring a few exceptions, I generally restricted gamesrc-ver-recreation to not more than recreations of other versions of code bases which were already open-sourced.

 

I'll quote a part of one of my posts in the more general DW thread about gamesrc-ver-recreation:

 

Quote

Additionally, Nuke.YKT is now a part of gamesrc-ver-recreation.

 

There might still be restrictions on what's uploaded to gamesrc-ver-recreation. For instance, a reversed engineered game is generally not covered. Exceptions are still a possibility. For instance, after Blake Stone: Planet Strike was open-sourced by Apogee, it was stated that the Aliens of Gold sources were assumed to be lost, thus explaining their lack. Therefore, I didn't mind building upon Blzut3's earlier reverse-engineering efforts, and later uploading reconstructed sources for the game.

 

Edited February 8, 2022 by NY00123
Typo fix

[OpenRift]

1 hour ago, NY00123 said:

 

One thing which should be done is preparing the relevant environment variables for using Watcom, like %WATCOM% or %LIB%. I probably assumed that writing the details was out of scope of the documentation for gamesrc-ver-recreation submodules, as it's more related to getting Watcom ready for use. On a second thought, I won't be surprised if out of the ones experimenting with the code, a relatively large percentage will try Watcom for the very first time (at least in a while).

All good now, Nuke helped me on Discord :)

[NY00123]

Another tidbit to add:

 

17 hours ago, Blzut3 said:

It is good that you think about these things though, since far too often I see people shaming leaked source code and praising non-clean room reconstructed code despite that from what I can tell they should basically be the same legally.

 

Here is how can one look at it. Leaked source code should generally not be accessible to the public. On the other hand, reconstructed code, even if not clean room, can be made using at least one exe which was properly released. That is, it can be a derivative of files which were made available in expected manners. Of course, it's possible the work was done using debugging information mistakenly left in one of the exes, but it's still different (or at least, technically more difficult and time consuming) from outright copying leaked sources.

 

In practice, we can see that even with leaked code, things apply on a case-by-case basis. From what I've seen, Witchaven as available from Steam and GOG.com is claimed to be bundled with enhancements from EGwhaven. Additionally, WitchavenGDX and TekwarGDX exist.

Edited February 8, 2022 by NY00123

[Blzut3]

4 hours ago, NY00123 said:

In practice, we can see that even with leaked code, things apply on a case-by-case basis. From what I've seen, Witchaven as available from Steam and GOG.com is claimed to be bundled with enhancements from EGwhaven. Additionally, WitchavenGDX and TekwarGDX exist. 

Don't really want to turn this thread into a discussion on RE morality so I'll skip commenting on your first paragraph.  Just wanted to say that the handling of Witchaven is exactly the kind of thing I was talking about.  In that case no cyber crime was involved with obtaining it, so in that case the only problem is that no copyrights were granted.  The community treated it like poison.  (Probably not helped by the fact that those games aren't popular so there wasn't a lot of incentive to say "screw it.")  Then other games got non-clean room RE'd and those efforts were praised as if it was any different besides the level of effort.  Granted these days we have those ports now so it's all moot.

[NY00123]

It might indeed be a good idea to make a partial topic change, and see if anybody inspected the sources for the purpose of comparing different versions. Looking for mentions of APPVER should lead to most changes, if not all. Unless I did a mistake, I was surprised to find out the version as shown in the textual loading screen was the only difference in code between versions 1.7 and 1.7a of Doom II. The DMX code was exactly the same in v1.7, v1.7a and French v1.8; Basically dmx34a with fmfix. English v1.8 and later (including Chex Quest) used dmx37.

 

EDIT (Feb 10 2022): Nuke.YKT realized I was a bit wrong with the dmx version for 1.7(a) and French v1.8; It's still based on dmx34a with fmfix, but also had other minor changes to the fm code. I renamed the expected directory name in doom/makefile and pushed this change.

 

On 2/9/2022 at 3:00 AM, Blzut3 said:

Don't really want to turn this thread into a discussion on RE morality so I'll skip commenting on your first paragraph.  Just wanted to say that the handling of Witchaven is exactly the kind of thing I was talking about.  In that case no cyber crime was involved with obtaining it, so in that case the only problem is that no copyrights were granted.  The community treated it like poison.  (Probably not helped by the fact that those games aren't popular so there wasn't a lot of incentive to say "screw it.")  Then other games got non-clean room RE'd and those efforts were praised as if it was any different besides the level of effort.  Granted these days we have those ports now so it's all moot.

 

At the least, what I wrote in the other post could explain the differing reactions of people, at least partially; Albeit it probably mostly comes down to the differing levels of interests that the games have, as hinted by you.

 

Another thing which might have an impact is the presence of a source port. If reverse-engineered code or a modification of leaked sources still targets a platform like DOS, the interest might be significantly smaller than a port to modern platforms.

Edited February 10, 2022 by NY00123
Clarification related to 1.7(a) and French v1.8

[Redneckerz]

On 2/6/2022 at 11:56 PM, NY00123 said:

I'm not sufficiently familiar with idgames' history, and I currently still have a total of 0 direct uploads to idgames, so I'm not that familiar with specific rules. Is the restriction of distributing original Doom exes still in effect?

It generally isn't, and its generally considered that id Software doesn't care if you do this, because it requires a WAD file to function anyway.

However, technically it is a propetiary executable, so you could technically face charges if you did. But this has never happened since 1997's source code release.

On 2/6/2022 at 11:56 PM, NY00123 said:

Also, generally speaking, I think that people will still prefer to let a mod be available with more than a standalone exe, be it a .deh file or source code.

Obviously, that is ease of use. I am just saying a recreated GPL executable lends itself better to executable hacks because the possible legality issue as discussed above is circumvented.

 

In fact, there is already a project that actively makes use of your work (Gibbon's Doom128, forthcoming)! :)

On 2/6/2022 at 11:56 PM, NY00123 said:

If it's about the Wiki, Nuke.YKT recently add a stub via this URL: https://doomwiki.org/wiki/Gamesrc-ver-recreation

I might create an account and add on my own, as there's information that might otherwise not really be known.

This definitely needs a lot more expansion and recognition. The size of this project is unique and is already an inspiration to certain folks and there are multiple applications for it. Please, feel free to add on it and ill happily approve anything you suggest there :)

On 2/7/2022 at 1:23 AM, OpenRift said:

This is true, but also if you look in DEHACKED.INI near the bottom, you'll see something like this:

I'm not entirely sure, but my guess is that you could theoretically set the file size, and various offsets for the different data sections manually and it could work with a compile of the recreation. I could be wrong though.

Seeing as exe hacks are confirmed to be working, this does have potential.

[OpenRift]

5 minutes ago, Redneckerz said:

Seeing as exe hacks are confirmed to be working, this does have potential.

That's a bit different, because all the core sections are all in the same places in an EXE hack. But in the case of the recreations and variations of it, these may have different offsets and may not even have the same string lengths. I'll have a closer look when I get back from class.

[OpenRift]

21 hours ago, OpenRift said:

That's a bit different, because all the core sections are all in the same places in an EXE hack. But in the case of the recreations and variations of it, these may have different offsets and may not even have the same string lengths. I'll have a closer look when I get back from class.

Okay, so an update on my findings. Here's the data section offsets for the various EXEs, including that of NEWDOOM's. converting these from hexidecimal to decimal, I plugged the offsets into dehacked.ini and attempted to load the new EXE in dehacked. It created doomhack.exe, but immediately crashed DOSBox. I suspect this has something to do with the offsets for the EXE's codepointers, which cannot be set manually in the INI file. Basically, you'd have to recompile dehacked with native support for the new EXE.

[NY00123]

3 hours ago, OpenRift said:

Okay, so an update on my findings. Here's the data section offsets for the various EXEs, including that of NEWDOOM's. converting these from hexidecimal to decimal, I plugged the offsets into dehacked.ini and attempted to load the new EXE in dehacked. It created doomhack.exe, but immediately crashed DOSBox. I suspect this has something to do with the offsets for the EXE's codepointers, which cannot be set manually in the INI file. Basically, you'd have to recompile dehacked with native support for the new EXE.

Good to see you going through these, thanks. Yeah, I'm not sufficiently familiar with dehacked modding, but I still suspected that this would be the case; There are offsets which are hardcoded in dehacked and depend on the version of the Doom exe.

 

As a consequence, as I wrote beforehand, there's no practical reason to support applying .deh files directly on top of newly built DOS exes with (a modified) dehacked. Better implement a proper Vanilla Doom compatible .deh file loader in a source port (even if the port is actually still running under DOS). Alternatively, change the sources directly.

[OpenRift]

5 minutes ago, NY00123 said:

Good to see you going through these, thanks. Yeah, I'm not sufficiently familiar with dehacked modding, but I still suspected that this would be the case; There are offsets which are hardcoded in dehacked and depend on the version of the Doom exe.

 

As a consequence, as I wrote beforehand, there's no practical reason to support applying .deh files directly on top of newly built DOS exes with (a modified) dehacked. Better implement a proper Vanilla Doom compatible .deh file loader in a source port (even if the port is actually still running under DOS). Alternatively, change the sources directly.

I think what might be worth looking into is what kind of settings give the vanilla EXEs the arrangements that they have, and how that could be replicated to some rough degree.

[NY00123]

17 minutes ago, OpenRift said:

I think what might be worth looking into is what kind of settings give the vanilla EXEs the arrangements that they have, and how that could be replicated to some rough degree.

 

I think that this is where what I wrote here earlier still holds:

 

On 2/6/2022 at 8:08 PM, NY00123 said:

I didn't even think about applying .deh files with DeHackEd itself to any exe made via gamesrc-ver-recreation. From what I was reading, DeHackEd detects the game version by the exe size, so it's already expected to fail due to the lack of DMX or the relevant DOS/4GW stub. Even if these weren't a concern, there's more that could break. As long as everything the .deh file tells DeHackEd to change is exactly at the expected location in the exe, I think that patching will generally work, but this isn't necessarily the case. For instance, as written beforehand, global variables which aren't explicitly initialized in the C code might be ordered differently. 

 

Generally speaking, I thought about gamesrc-ver-recreation more in terms of education, as well as the ability to use the code in ports to modern platforms, mostly ones inspired by Chocolate Doom.

 

Thanks to Nuke.YKT's efforts, almost all Doom exes currently covered may theoretically be recreated with more-or-less the same layout, including how are global variables ordered. That's, of course, under the assumption that matching DMX versions (which might've been modified on their own) are used. From my understanding, a matching DOS/4GW stub will also be required for compatibility with .deh files.

 

Currently, this will still not apply to the Doom II 1.666 prototype; Even for the rest, any little difference in the process can lead to a bit different exe, and this is sufficient for making existing .deh files incompatible; This, at least from the way I see it, might be enough for giving up on the idea, and doing something different might work better (e.g., adding a proper .deh file loader to a source port, if there isn't any).

Edited February 11, 2022 by NY00123

[OpenRift]

Okay, so today I've been experimenting with Dehacked's source code and I managed to compile a version that is... sort of compatible?

 

I tested it with Rowdy Rudy 2 and it looks like everything loaded up properly, but it seems like there's some weird visual glitches going on in-game with the patch applied.

It's basically like slime trails but much, much worse.

 

On top of that, the dehacked interface seems to be displaying some weird things as well, with text fields looking like this:

 

Instead of looking like this:

 

For context, here are the offsets that were replaced in my version:

https://docs.google.com/spreadsheets/d/1Q1NnqmFj2-cMoJkuZpGFrtMecMPJCLddskV-Ddr7uDM/edit?usp=sharing

 

And here's the source itself

deh128.zip

 

Anyone have suggestions on what I can do?

Edited February 17, 2022 by OpenRift

[maxmanium]

What did you change about the source code? Or did you just edit the offsets in the ini?

[OpenRift]

14 minutes ago, maxmanium said:

What did you change about the source code? Or did you just edit the offsets in the ini?

I had to change the offsets in the source code because you can't specify the codepointer offsets in the INI, unfortunately.

[Redneckerz]

11 hours ago, OpenRift said:

Okay, so today I've been experimenting with Dehacked's source code and I managed to compile a version that is... sort of compatible?

 

I tested it with Rowdy Rudy 2 and it looks like everything loaded up properly, but it seems like there's some weird visual glitches going on in-game with the patch applied.

It's basically like slime trails but much, much worse.

 

On top of that, the dehacked interface seems to be displaying some weird things as well, with text fields looking like this:

 

Instead of looking like this:

 

For context, here are the offsets that were replaced in my version:

https://docs.google.com/spreadsheets/d/1Q1NnqmFj2-cMoJkuZpGFrtMecMPJCLddskV-Ddr7uDM/edit?usp=sharing

 

And here's the source itself

deh128.zip

 

Anyone have suggestions on what I can do?

You may want to add @xttlhere as DeHacked Resident. 

 

Also he just posted so that's why i tagged, haha

[NY00123]

Maybe I should start by asking if a build of the unmodified DeHacked sources works as expected, without the aforementioned problems shown in the interface. If the problems are reproduced even without modifying the sources then they're probably not related to the changes.

 

In retrospect, this might be a good example of possibly less intentional or planned manners in which gamesrc-ver-recreation code is used, which can be a good thing.

 

In the current state, it sounds like it'll still require a custom DeHacked build for each new exe, unless something like an appropriate extension of the .DEH format (or any of my preceding suggestions) will be the way to go.

[Gez]

2 hours ago, NY00123 said:

In the current state, it sounds like it'll still require a custom DeHacked build for each new exe, unless something like an appropriate extension of the .DEH format (or any of my preceding suggestions) will be the way to go.

Shuffling the codepointer offsets to the ini should be sufficient to make a single dehacked build? It'd be just a custom ini for each new exe, but that's unavoidable.

[OpenRift]

8 hours ago, NY00123 said:

Maybe I should start by asking if a build of the unmodified DeHacked sources works as expected, without the aforementioned problems shown in the interface. If the problems are reproduced even without modifying the sources then they're probably not related to the changes.

  

In retrospect, this might be a good example of possibly less intentional or planned manners in which gamesrc-ver-recreation code is used, which can be a good thing.

 

In the current state, it sounds like it'll still require a custom DeHacked build for each new exe, unless something like an appropriate extension of the .DEH format (or any of my preceding suggestions) will be the way to go.

I've used unmodified dehacked with it and it locks up dosbox and crashes it, which is why I had to make a source modification.

 

As for each new EXE, the good thing about dehacked is that it has support for multiple EXE versions, so I can just replace the original offsets with the corresponding ones found in the recreations. That's what I've done so far and it works for the most part, but I think certain data sections aren't as long as they would be in the original EXEs.

[NY00123]

3 hours ago, OpenRift said:

I've used unmodified dehacked with it and it locks up dosbox and crashes it, which is why I had to make a source modification.

 

Hmm, is it possible that I should maybe clarify what I wrote earlier?

 

Basically, I was wondering if you could build DeHacked from unmodified sources, and then use it with any of the original Doom exes from the 90s (rather than gamesrc-ver-recreation or any other exe). This way, if there's still a problem (say in the UI), then it's probably in the compilation process; Unless there's an actual problem with the original DeHacked sources that I'm not aware of, but after many years, I'm currently having doubts.

[nukeykt]

Update to the Strife restoration

A couple more revisions of the Strife executable are covered now: registered v1.1(aka v1.0) and registered v1.2. Both reconstructed EXEs are identical to the original EXE files (up to garbage data between string literals and differences due to the __LINE__ macro). Thus gamesrc-ver-recreation now covers all known registered versions of Strife.

 

The next obvious step is to try to cover the demo versions of the Strife, but I expect much more differences because both demo versions use much earlier revisions of the executable, so I guess I'll leave this for later.

Edited February 20, 2022 by nukeykt

[nukeykt]

Bumping a thread with some news.
Recently I started reverse engineering of the Doom 95 exe. Wanted to do this for a long time actually, mostly because it is directly derived from the DOS doom code and its exe was build using the Watcom compiler(same compiler used to build DOS Doom) and it included debug symbols(!!!). Now when we have perfect DOS doom sources recreation, it can be used as basis for this effort. Reverse engineering process is pretty starightforward: compile dos doom C modules with the right compiler settings and compare to the doom 95 code disassembly, then modify C code until compiled code matches byte by byte (or at least by behaviour). And of course windows/directx related code needs to be written purely based on disassembly. Today I finished first pass of reverse engineering and finally built exe. With some more bugfixing it is mostly working now.

[NY00123]

I was surprised to hear that Doom95 was built with Watcom back when you initially told me about these efforts. It's impressive that you managed to get it to the current state as of Doom's birthday.

[OpenRift]

15 hours ago, nukeykt said:

Bumping a thread with some news.
 Recently I started reverse engineering of the Doom 95 exe. Wanted to do this for a long time actually, mostly because it is directly derived from the DOS doom code and its exe was build using the Watcom compiler(same compiler used to build DOS Doom) and it included debug symbols(!!!). Now when we have perfect DOS doom sources recreation, it can be used as basis for this effort. Reverse engineering process is pretty starightforward: compile dos doom C modules with the right compiler settings and compare to the doom 95 code disassembly, then modify C code until compiled code matches byte by byte (or at least by behaviour). And of course windows/directx related code needs to be written purely based on disassembly. Today I finished first pass of reverse engineering and finally built exe. With some more bugfixing it is mostly working now.



Glad to see you're back!! Very exciting stuff, perhaps one day we'll finally get the Doom95 that we should've had!

[THEBaratusII]

On 12/10/2022 at 1:40 AM, nukeykt said:

Bumping a thread with some news.
Recently I started reverse engineering of the Doom 95 exe. Wanted to do this for a long time actually, mostly because it is directly derived from the DOS doom code and its exe was build using the Watcom compiler(same compiler used to build DOS Doom) and it included debug symbols(!!!). Now when we have perfect DOS doom sources recreation, it can be used as basis for this effort. Reverse engineering process is pretty starightforward: compile dos doom C modules with the right compiler settings and compare to the doom 95 code disassembly, then modify C code until compiled code matches byte by byte (or at least by behaviour). And of course windows/directx related code needs to be written purely based on disassembly. Today I finished first pass of reverse engineering and finally built exe. With some more bugfixing it is mostly working now.

 

EPIC! Would love to see more limits raised (iirc VISPLANES was increased for 640x400) or removed along with various fixes because I loved playing Doom 95 during my childhood and was a shame it didn't age well.

[nukeykt]

Doom v1.2 restoration WIP

https://bitbucket.org/gamesrc-ver-recreation/doom/src/doom12/

[nukeykt]

To celebrate upcoming doom's 30th anniversary gamesrc-ver-recreation now covers original shareware v0.99/v1.0 release

https://bitbucket.org/gamesrc-ver-recreation/doom