EXE hacking

[uhbooh]

Is there any documentation on modifying the original EXE files? I'd like to know how to edit things dehacked doesn't. Par times, static limits or codepointers for example.

[printz]

To modify constant tables, you may look at the Doom source code. For example, par times are defined in g_game.c of the Doom source as sequences of integers, like this:

// DOOM Par Times
int pars[4][10] = 
{ 
    {0}, 
    {0,30,75,120,90,165,180,180,30,165}, 
    {0,90,90,90,120,90,360,240,30,170}, 
    {0,90,45,90,150,90,90,165,30,135} 
}; 

int is four bytes, so you can find them in DOOM.EXE (or DOOMP.EXE on my computer) by looking for this hex representation of 30 and 75 (E1M1 and E1M2): "1E 00 00 00 4B 00 00 00".

Static limits have already been increased in Doom+ (which is what DOOMP.EXE really is), see the Doom+ homepage.

Codepointers are much more complex, and Greg Lewis managed to figure out how to point to them in Dehacked. As for what they actually do, I suppose they point to machine code, and you can replace machine code as long as you don't need to increase the file size. I'm not 100% sure on this, so please ask guys who have managed to reverse-engineer a similar executable, that of Strife. Those guys are Kaiser and Quasar. Mainly, what tools did they use to disassemble the programs? And are they free? Disassembling it would go a long way to making you understand what is going on.

[uhbooh]

printz said:

Codepointers are much more complex, and Greg Lewis managed to figure out how to point to them in Dehacked. As for what they actually do, I suppose they point to machine code, and you can replace machine code as long as you don't need to increase the file size. I'm not 100% sure on this, so please ask guys who have managed to reverse-engineer a similar executable, that of Strife. Those guys are Kaiser and Quasar. Mainly, what tools did they use to disassemble the programs? And are they free? Disassembling it would go a long way to making you understand what is going on.

Thanks for the help, I'm suprised vanilla dehacked patches don't change par times given how easy it is. As for codepointers, let's say you wanted to modify a_cposattack to play the unused DSCHGUN instead of DSSHTGN (or whatever the sounds are called). This would probably be very possible with hex editing. Another possibility whould be changing a_fireshotgun to make 10 hitscans instead of 7(iirc).

[xttl]

printz said:

Static limits have already been increased in Doom+ (which is what DOOMP.EXE really is), see the Doom+ homepage.

Limit extension in Doom+/Doom2+ was done by enlarging the size of the uninitialized data section (or "bss section") from the LE header, then patching eg. all visplane related code to use the empty memory space now available at the end of that section for arrays much larger than original. The space that was reserved for the old smaller arrays will still also be there, unused.

printz said:

Codepointers are much more complex, and Greg Lewis managed to figure out how to point to them in Dehacked. As for what they actually do, I suppose they point to machine code, and you can replace machine code as long as you don't need to increase the file size. I'm not 100% sure on this, so please ask guys who have managed to reverse-engineer a similar executable, that of Strife. Those guys are Kaiser and Quasar. Mainly, what tools did they use to disassemble the programs? And are they free? Disassembling it would go a long way to making you understand what is going on.

The best disassembler is IDA Pro which unfortunately has a price that's not so nice if you wanted to buy it purely for non-commercial hobbyist use. (I've also heard some rumors that Hex-Rays doesn't even really sell it to just anybody who has the money...though also claims to the contrary). An old version for Windows only with some features removed is available for free though: https://www.hex-rays.com/products/ida/support/download_freeware.shtml

If you use the old freeware version, you can't load Doom or any other Watcom 32-bit DOS programs that have the DOS extender attached into the main EXE (vs. included as a separate file, "dos4gw.exe", "dos32a.exe" or such) properly. You have to strip the extender out yourself while still leaving Watcom's realmode stub (and the 32-bit "LE" Linear Executable itself) in. Basically, using a hex editor look for the second "MZ" DOS EXE header from the beginning, that should be Watcom's stub which would try to load DOS4GW from a separate file. Then cut everything before the second "MZ" out from the file and you should be able to load it in older versions of IDA.

As for pointers, any references to code or data addresses in memory that aren't relative (like call/jmp immediate form instructions), the address that is inside an actual function in the code section, or an actual variable/array in the data section (like the state/frame table with its "code pointers") isn't really used. Instead, the LE executable loader will on startup always rewrite all such references based on a table of so called "fixup records" (relocation records) in the LE header. This is done because the LE sections aren't guaranteed to be located at the same protected mode address on every computer configuration. So, if you want to modify where they point, you have to modify the relocation records instead of the code/data itself (possibly better to modify both, but so far I have never seen the addresses from the actual code/data section get used...). You of course also have to do this if you want to move an instruction which references a memory location even if you aren't changing the location it references.

A few years ago I made a tool that can dump the whole relocation table to a text file and then rebuild it from that but it's really embarassing code, so I'd rather not distribute it (at least not the source).

[Linguica]

xttl said:

If you use the old freeware version, you can't load Doom or any other Watcom 32-bit DOS programs that have the DOS extender attached into the main EXE (vs. included as a separate file, "dos4gw.exe", "dos32a.exe" or such) properly. You have to strip the extender out yourself while still leaving Watcom's realmode stub (and the 32-bit "LE" Linear Executable itself) in. Basically, using a hex editor look for the second "MZ" DOS EXE header from the beginning, that should be Watcom's stub which would try to load DOS4GW from a separate file. Then cut everything before the second "MZ" out from the file and you should be able to load it in older versions of IDA.

Maybe I'm dumb, but this means you wouldn't actually be able to use such a hacked executable, right? You'd have to, like, remember all the offsets and then add back in the size of the DOS extender block or whatever?

[xttl]

You should be able to run it if you have dos4gw.exe in the same directory. You can get that from the Open Watcom distribution. (or the distribution package for any program compiled with 32-bit Watcom whose author didn't have DOS4GW Professional)

edit: also if that doesn't work, try this command line: dos4gw your.exe

[Linguica]

OK, but you couldn't, like, use IDA Pro to create a doom2.exe patch for whatever you were doing and have it work right? You'd have to have some other program to change all the memory offsets back to fit with the original EXE?

[xttl]

I don't know, I've never tried it. I've always used a hex editor to do smaller patches and NASM to assemble replacement functions for larger or more complicated patches. Also I just noticed that the free version of IDA actually doesn't load Watcom LEs even if you strip the extender. :( However if you manage to acquire a paid for version of IDA Pro (eg. through your employer), the status bar should show you the on-disk offset for any data/code you're currently browsing (it is the first hex number on the left).

edit: also the most recent versions will load Watcom LEs even without stripping the extender.

[uhbooh]

I didn't expect this thread to get this much attention, neat!

Anyways, to capitalise on what i posted yesterday...

void A_CPosAttack (mobj_t* actor)
{
    int		angle;
    int		bangle;
    int		damage;
    int		slope;
	
    if (!actor->target)
	return;

    S_StartSound (actor, sfx_shotgn);
    A_FaceTarget (actor);
    bangle = actor->angle;
    slope = P_AimLineAttack (actor, bangle, MISSILERANGE);

    angle = bangle + ((P_Random()-P_Random())<<20);
    damage = ((P_Random()%5)+1)*3;
    P_LineAttack (actor, angle, MISSILERANGE, slope, damage);
}

This is the codepointer for the heavy weapon dude's attack. Let's say you wanted to change this...

S_StartSound (actor, sfx_shotgn);

...to this

S_StartSound (actor, sfx_chgun);

Wouldn't this be possible in hex by changing a few bytes?

Another thing i'd like to ask, is it possible to change the players view height? Some mods like Batman Doom would benefit from this.

[Graf Zahl]

uhbooh said:

Wouldn't this be possible in hex by changing a few bytes?

Of course. The bigger problem would be finding those 'few bytes' first...

[Randy87]

Doom 2 1.9
69B01:
mov edx,2 -> mov edx,56
BA 02000000 -> BA 56000000

[uhbooh]

Graf Zahl said:

Of course. The bigger problem would be finding those 'few bytes' first...

You're right, that'd be pain the ass. One solution would be to compile two different EXEs, one with the new sound and one unchanged. Then using a hex editor, find what bytes changed. The problem with this is the linux doom and 1.9 dos doom EXEs aren't identical.

Randy87 said:

Doom 2 1.9
69B01:
mov edx,2 -> mov edx,56
BA 02000000 -> BA 56000000

Ninja'd!

Another problem is this

{ "chgun", false, 64, &S_sfx[sfx_pistol], 150, 0, 0 },

DSCHGUN is rerouted to DSPISTOL even if the lump is present. All of the other sounds have their third value set to 0, so this might be possible to change too.

[xttl]

If you have to make do with NDISASM instead of IDA, here's some info that may be useful:

For regular Doom(2) v1.9 EXE, code section begins at 0x42014 on disk and is 0x44F71 bytes long. Initialized data section begins at 0x88014 and is 0x254FC bytes long.

For Ultimate Doom v1.9 EXE, code section begins at 0x42214 and is 0x45181 bytes long. Initialized data section begins at 0x89214 and is 0x258D0 bytes long.

For original Final Doom EXE, code section begins at 0x42814 and is 0x45451 bytes long. Initialized data section begins at 0x89814 and is 0x26EB0 bytes long.

For anthology Final Doom EXE, code section begins at 0x42814 and is 0x454A1 bytes long. Initialized data section begins at 0x89814 and is 0x26EB0 bytes long. (so same as unfixed original Final Doom due to 4kB page alignment padding)

So first copy the code section by itself to a separate file, then do "ndisasm -u -o 0x10000 cs.bin > cs.asm". -u means disassemble as 32-bit and origin 0x10000 is specified here because the unrelocated CS base in memory is always 0x10000 for Doom binaries. This way you get the same code offsets in the disassembly listing as someone who's using IDA (useful if you need to eg. talk to such a person about something).

Then copy the data segment to another file (eg. ds.bin), and if you see eg. "0002B65D B8AC040200 mov eax,0x204ac" in the listing generated by ndisasm, you load ds.bin in your hex editor, go to 0x204AC and you'll see what data is referenced, assuming it's not uninitialized data (in this example, it's the "genmidi" string for call to WAD lump lookup function in DMX init code from Final Doom anthology EXE). Note though that the unrelocated CS base in memory is 0x70000 for Doom binaries (so you have to add that to the offset in ds.bin if you want to get the unrelocated in memory address).

edit:

uhbooh said:

DSCHGUN is rerouted to DSPISTOL even if the lump is present. All of the other sounds have their third value set to 0, so this might be possible to change too.

Apply the patch Randy87 gave you, then (addr: oldbyte newbyte):

00040F95: B0 C8
00040F98: B0 00
00040F99: 48 4E
00040F9A: 01 02

(overwrites relocation record with a duplicate of the previous one, so the next patch doesn't get overridden by the LE loader)

0009D4C4: B0 00
0009D4C5: 48 00
0009D4C6: 01 00

(overwrites link pointer in sound table with NULL)

After this it should use DPCHGUN/DSCHGUN for chaingunner's attack sound.

[Randy87]

Beat me. I used to have this down, but it's been awhile and my notes are obscure and make no sense.

Edit:
Thanks for these notes, it can be hard to find info like this.

[uhbooh]

I've figured out why DSCHGUN redirects to DSPISTOL, in the vanilla exe, it changes the pitch of the sound! This unused behaviour isn't emulated in any sourceport.

[xttl]

to be quite honest,

Linguica said:

I appreciate your research into this, and it's of historical interest to learn that it wasn't the DMX library that randomly broke pitch shifting but actually id itself who somehow did it. Nevertheless random-pitched sound effects must be destroyed.

Also the sound isn't used in vanilla at all anyway unless you patch the game.

Btw. here are the (unrelocated, in memory) entry points for all 4 v1.9 Doom binaries (in case you're using ndisasm, IDA will give you these automatically):

regular: 0x50B48
ultimate: 0x50D58
old final: 0x51028
anthology final: 0x51078

so you can start mapping the code from there. (if for some reason you don't want to start mapping it from some function you recognize from the initialized data references)

[uhbooh]

I'm aware that early versions of the EXE had random pitch shifts in the sound effects. What i meant was that, in all of the original EXEs, not just the early ones, DSPISTOL is played slightly faster and at a higher pitch when DSCHGUN is called. Source ports don't do this because they don't use the DMX sound library.

EDIT:
I've found most of the bytes related to monster codepointers, as well as all of the par times. I'm still searching for stuff like player codepointers, player view height, A_BossDeath behavior and SSG ammo usage. If someone can point me in the right direction, that would be very nice.

[Jon]

uhbooh said:

I'm aware that early versions of the EXE had random pitch shifts in the sound effects. What i meant was that, in all of the original EXEs, not just the early ones, DSPISTOL is played slightly faster and at a higher pitch when DSCHGUN is called. Source ports don't do this because they don't use the DMX sound library.

That's an interesting point, and it's something we should fix/emulate in chocolate doom. I worked around the strange link for the chaingun sound in the code when working on the pitch-shifting code but neve really understood it.

I've opened a bug against chocolate-doom to track this.

I've found most of the bytes related to monster codepointers, as well as all of the par times. I'm still searching for stuff like player codepointers, player view height, A_BossDeath behavior and SSG ammo usage. If someone can point me in the right direction, that would be very nice.

It might be worth poking at the dehacked source code (assuming it's available, I can't remember), if you haven't already: it worked by manipulating the code pointers tables, etc., so figured out much of this stuff.

[VGA]

A little note to dos exe hackers. The DOS/32a extender comes with some utilities, one of them can remove the embedded dos4gw extender or replace it with dos32a.

Also, has anyone used the Dosbox debugger? From the screenshots it looks like SoftIce, heh.

[xttl]

I have used it a bit sometimes. I recommend you get the heavydebug version and also remember to disable dynamic CPU core. When you want to eg. edit code/data in memory or place breakpoints, you will first have to figure out the relocated base addresses for the code and data sections of the Doom executable (or "objects" as the LE spec calls them). After you have them, they will fortunately always stay the same unless you change your memory configuration in dosbox.conf. I'd guess that changing DOSBox versions will possibly affect relocation too but I haven't tested that.

On another note now that I logged in to post this, here's a work in progress. It is Final Doom Anthology exed patched with MAXVISPLANES, MAXDRAWSEGS, MAXVISSPRITES, MAXLINEANIMS and MAXOPENINGS patches adapted from Doom2+. No MAXPLATS patch yet, and no longtics demo support. However, it supports a new command line parameter "-maxzone" which allows you to specify the size in megabytes the game will at most try to allocate for its internally managed memory pool (default is 8MB like in original exe, 2-32MB allowed). It also has patched savegame code which uses Z_Malloc/Z_Free to allocate a 2MB buffer for creating the savegame (instead of temporarily using the screen buffers for max 176kB saves like original). This may cause Z_Malloc to run out of memory in some maps (eg. Plutonia MAP30 -skill 4) with the default maxzone size of 8MB so I may yet change it to use regular malloc instead (like Doom+/Doom2+ does) and/or add a new "-maxsave" command line parameter for specifying the save buffer size (defaulting to 176kB like original). Unfortunately Doom doesn't have code in the executable that would write save games directly to disk like Heretic and Hexen have (just never used by default if they can alloc SAVEGAMESIZE bytes). :( That would make things so easy.

I was also thinking about poassibly adding some command line parameters to break teleports and Lost Soul bounces or to enable the pre Ultimate Doom boss death behaviour so you could use this as a sort of "ultimate vanilla EXE". This is also why I based it on the Anthology version even though DHE doesn't support it (but I might take a look at the DHE 3.0 sources later to see if it can be updated to support Final Doom Anthology version).

[VGA]

Well, you are deep in the code. Make sure you keep notes, backups etc heh

[Randy87]

I made IDA databases for Doom 2 1.9 and The Ultimate Doom a while back.
I never got around to finishing them, but I've found them very useful.
I can provide IDC/map files if anyone is interested.

[Linguica]

Could you upload them anyway? Someone in the future will undoubtedly be interested and it's always aggravating to find an old forum thread where a person has done the exact thing you are looking for but never actually shared it.

[Randy87]

On 3/1/2016 at 9:47 AM, Linguica said:

Could you upload them anyway? Someone in the future will undoubtedly be interested and it's always aggravating to find an old forum thread where a person has done the exact thing you are looking for but never actually shared it.

Good point, I'll edit this post with a link when Mediafire stops misbehaving.
 

Doom_IDA.7z

Doom_IDA.7z

[uhbooh]

Jon said:

It might be worth poking at the dehacked source code (assuming it's available, I can't remember), if you haven't already: it worked by manipulating the code pointers tables, etc., so figured out much of this stuff.

Thanks for the advice! I just checked the source and using it as a guide, found the weapon codepointers! Speaking of DeHackEd, when i finish finding all of these bytes, i'll release a new version of DeHackEd that can edit codepointers, par times, and a bunch of other stuff. I'm suprised it hasn't happened yet since people have been using the same patches for 20 years.

EDIT:

Jon said:

That's an interesting point, and it's something we should fix/emulate in chocolate doom. I worked around the strange link for the chaingun sound in the code when working on the pitch-shifting code but neve really understood it.

While messing around with weapon codepointers, i've found that DSCHGUN doesn't have a pitch shift when the player calls it. This might be a bug, because monsters play it just fine.

[Linguica]

Randy87 said:

Good point, I'll edit this post with a link when Mediafire stops misbehaving.

Doom_IDA.7z

mirrored: http://www.doomworld.com/linguica/mirror/Doom_IDA.7z

[xttl]

uhbooh said:

Thanks for the advice! I just checked the source and using it as a guide, found the weapon codepointers! Speaking of DeHackEd, when i finish finding all of these bytes, i'll release a new version of DeHackEd that can edit codepointers, par times, and a bunch of other stuff. I'm supred it hasn't happened yet since people have been using the same patches for 20 years.

Add support for Final Doom too while you're at it. Both versions since DHE 3.1 source with support for non-Anthology Final Doom wasn't ever released...

Another thing you could do is make it possible to assign code pointers to frames that do not have them in the original EXE. DHE does not allow this because NULL pointers aren't relocated so the code pointers for those frames do not have corresponding entries in the LE relocation table. You will have to rebuild the relocation table if you want to do this though. (and possibly also extend/rebuild the LE, there's some unused space in the area reserved for relocation records in the header but I don't know if it's enough for assigning code pointers to every frame/state in the table, probably not).

You could even make it patch in some startup code that will load the state table (or any of the tables really) from separate files or WAD lumps. This would allow adding more entries. I made a patch like this for the Doom Press Beta EXE once because I thought I'd try to add the boss monsters to it but in the end I never got around to do that because binary hacking Doom all day started feeling like too much waste of time for relatively little gain in life, haha.

I may actually have this patch somewhere still and also some other stuff for the beta. I at least used to have press beta patches for configurable keys + option for always running, make it not crash when you walk into special sectors of unknown type (it prints a warning in the status bar instead) so converted retail maps work better, sped up lifts and also making zombies drop clips and shotguns (because otherwise you always run out of ammo if you play retail maps, lol).

[TimeOfDeath]

Off-topic, but since this thread is up, does anyone know where to change the weapon keys in 1.9 with a hex editor?

[Randy87]

On 3/1/2016 at 2:33 PM, TimeOfDeath said:

Off-topic, but since this thread is up, does anyone know where to change the weapon keys in 1.9 with a hex editor?

Not so simple, but this might work.
VanillaCustomWepNums.7z

VanillaCustomWepNums.7z

[TimeOfDeath]

Cool thanks, it does work. Is it only possible to change weapon keys with other weapon keys?