GENMIDI

[shadow1013]

I was thinking of random ideas and it occurred to me, the FreeDOOM genmidi lump sucks... Like really badly, and it is of no fault of Fraggle, who found it from the BSD kernel, but a better idea came to me afterwords "What license is allegro under..." I checked and it is basically a freely redistributable license, which matches FreeDOOM's goals. So why not produce a GENMIDI lump from the allegro patchset?

[fraggle]

CSonicGo is working on an improved patchset which is pretty damn awesome so far from the sample I heard. Unfortunately the patches are in the .a2i AdlibTracker format which is a pain to decode. I was going to take a look at writing a script to generate a GENMIDI lump but I haven't got round to it yet.

[shadow1013]

Oh, well if I knew the structure of the format I could probably write up a short little program for it. And where can I hear these samples? I'm excited at the prospect of a new patchset

[fraggle]

CSonicGo gave me a MP3 recording a while back of Doom's E1M1 music played with the new instruments, though that was in AdlibTracker (Doom's MIDI playback will likely sound different / not as good).

The a2i patch format used by AdlibTracker is unfortunately non-trivial to decode. In fact it's one of the most fucking stupid designs I've seen in a while. The files are compressed (despite being on the order of 50 bytes in size or less) using an obscure compression algorithm from "Jibz's aPLib compression library" - which is apparently written in assembler.

You can find out more if you check out the AT2 source code - there's a technical document in there that describes the file format. But prepare yourself for some serious facepalming.

[shadow1013]

Well I downloaded the source package and was unpleasantly surprised to see it was in Pascal, which I have a hard time following, but what I did pick up is that it seems to be using lzss and lzw, both compression schemes which aren't that uncommon. Can't AT2 save to sbi instruments though?

[fraggle]

shadow1013 said:

Well I downloaded the source package and was unpleasantly surprised to see it was in Pascal, which I have a hard time following

You and me both. It's actually worse than that - the relevant decompression part is a Pascal file containing inline assembly. The AT2 developers were clearly twisted, sadistic bastards.

Can't AT2 save to sbi instruments though?

I think I asked CSonicGo the same thing and his response was that it isn't possible.

[shadow1013]

Ohhh I saw that part... That's just lzss, there's an implementation for that in the dmutils package. I think. And inline assembler doesn't bother me actually... I enjoy fiddling around with it.

[fraggle]

It looks like the aplib source code package includes a C implementation!

[shadow1013]

That is awesome! Now parsing the .a2i format is all that's left :)

[Csonicgo]

I'm still working on it, now that I have some christmas time!

[fraggle]

Great! Hopefully I'll find some time to look at writing the conversion script.

[fraggle]

After some work I now have a script that can decode a2i files. It should be fairly straightforward now to generate GENMIDI lumps

[fraggle]

Good news, everyone! I've been working on the holidays and I now have a build system that constructs a GENMIDI lump out of individual SBI/A2I instrument files. I just need to check a few things and then I'll merge it into the main Freedoom tree.

EDIT: I've written up a guide (HTML version) for how to edit instruments and help improve the GENMIDI lump.

[andrewj]

fraggle said:

EDIT: I've written up a guide (HTML version) for how to edit instruments and help improve the GENMIDI lump.

It's a good document, however it's not actually the MIDI standard which defines those 128 instruments. MIDI itself deals with the electronics and on-wire protocol, whereas the instruments are defined by the "General MIDI" standard (GM for short).

[shadow1013]

I like this new build system, it provides for easy replacements. Do we know the state of CSonicGo's patches? I really hope they come out good. I'm actually quite excited for a FreeDOOM which moves ahead closer to completion

[fraggle]

Thanks for the feedback. I've made a few more changes which are on my github repo.

Earlier last year there was a thread by horncomposer who made some improvements and modifications (by hand) to the old GENMIDI lump. He sent me a copy of it by email a while back, so I've imported his changes into the new build system. One particularly nice thing about this is that we now have a good number of the percussion instruments.

andrewj said:

It's a good document, however it's not actually the MIDI standard which defines those 128 instruments. MIDI itself deals with the electronics and on-wire protocol, whereas the instruments are defined by the "General MIDI" standard (GM for short).

Okay, fixed :-P

shadow1013 said:

I like this new build system, it provides for easy replacements. Do we know the state of CSonicGo's patches? I really hope they come out good. I'm actually quite excited for a FreeDOOM which moves ahead closer to completion

He told me by email that he's planning to do some more work on it in the new year.

[shadow1013]

I played through episode 1 on this new genmidi, and I found it awesome! It sounds already a million times better than the existing one we had. One thing I did notice is that Jute's Dave3D music is generally much lower volume than most of the other music, and we should decide on one general volume. One of the biggest offenders is the music for e4m1, which is *VERY* loud, almost obnoxiously so.

[fraggle]

Adding percussion certainly makes a lot of difference. A lot of it still sounds pretty terrible, though. Hopefully we should see some further improvements soon :-)

[shadow1013]

Yes I certainly hope so :)

[Csonicgo]

Don't worry guys, I've been in hospital, I should be back on track next week.

[jute]

The problem of dynamic levels in different MIDIs has been discussed before. Someone suggested writing a small program that would "normalize" all the MIDIs' note velocities to fall within some predefined range.

[fraggle]

Csonicgo said:

Don't worry guys, I've been in hospital, I should be back on track next week.

Okay, hope you're okay!

I've just added a hack feature to Chocolate Doom what I hope should be useful for you and anyone planning on doing any GENMIDI hacking. Run the game with -opldev on the command line and you'll get an on-screen display showing the instruments in use on the current MIDI channels and recent percussion effects:

[shadow1013]

Wow, hope you get better! and -opldev looks nice. A little off topic, but does choco-doom support the DMXOPTIONS variable to get opl3 sound?

[fraggle]

Not yet, no. It needs more research to support.

[fraggle]

My GENMIDI changes are now in the main git repo.

I thought I'd do a quick braindump of the things that I think need doing to improve the current GENMIDI:

• Complete the set of percussion instruments. There are still a few left that need to be added. You can see these in config.py listed as "NullInstrument".
  
• Tune some of the existing percussion instruments. Percussion instruments are played at a fixed MIDI note which is specified in the GENMIDI lump. At the moment, instruments 70-81 are just using 'E' in octave -5 because I wasn't sure what else to use. Judging by the names of these instruments this probably isn't a very good note to be using.
  
• Fix the warnings with the current instruments due to use of waveforms that OPL2 doesn't support:

  instruments/instr007.sbi: Modulator uses waveform 5: only 0-3 supported
  instruments/instr007.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr008.sbi: Modulator uses waveform 5: only 0-3 supported
  instruments/instr008.sbi: Carrier uses waveform 5: only 0-3 supported
  instruments/instr021.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr021.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr022.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr024.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr024.sbi: Carrier uses waveform 5: only 0-3 supported
  instruments/instr029.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr030.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr031.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr034.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr035.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr037.sbi: Modulator uses waveform 6: only 0-3 supported
  instruments/instr041.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr041.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr042.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr042.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr043.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr043.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr063.sbi: Modulator uses waveform 5: only 0-3 supported
  instruments/instr063.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr065.sbi: Modulator uses waveform 5: only 0-3 supported
  instruments/instr066.sbi: Modulator uses waveform 5: only 0-3 supported
  instruments/instr067.sbi: Modulator uses waveform 5: only 0-3 supported
  instruments/instr072.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr080.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr105.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr105.sbi: Carrier uses waveform 4: only 0-3 supported
  instruments/instr106.sbi: Modulator uses waveform 4: only 0-3 supported
  instruments/instr111.sbi: Carrier uses waveform 5: only 0-3 supported
  instruments/instr127.sbi: Carrier uses waveform 5: only 0-3 supported
  

• Improve the current guitar instruments. I think this will make a big difference to a lot of the songs - listen to E1M1 for example.
  
• Improve / replace percussion instrument #40 (Electric Snare) - there are a bunch of Doom tracks where this seems to sound weird.
  
• Probably not a huge priority since this is for Freedoom, but this file gives some statistical information about the instruments used in the original Doom music, which might be useful in choosing targets for improvement.
  

[shadow1013]

Listening to this has always given me the impression that the guitar sounds like a piano instead of a guitar

[Gez]

fraggle said:

• Probably not a huge priority since this is for Freedoom, but this file gives some statistical information about the instruments used in the original Doom music, which might be useful in choosing targets for improvement.
  

Only covers Doom, though, not Doom II, which has different MIDI settings (GENMIDI and DMXGUS lumps differ) because they were optimized for a different selection of instruments. Given that most classic Doom mod soundtracks were written for Doom II megawads...

[fraggle]

fraggle said:

Fix the warnings with the current instruments due to use of waveforms that OPL2 doesn't support

I went through the instruments that were generating warnings and fixed a bunch of them. I tried to rework them so that they sound the same as before, but I'm frankly I'm not an expert at this, and gave up on a bunch of instruments that I couldn't get to work.

Interestingly, shadow1013's comment above that "the guitar sounds like a piano instead of a guitar" may actually be caused by the guitar instruments using OPL3 waveforms. As a result when these are played back in OPL2 mode they get mapped onto a different waveform and sound wrong. My tweaks have fixed instruments #30 (Overdriven Guitar) and #31 (Distortion Guitar), and some tracks, like E1M1 and the Doom 1 intermission music, sound noticeably improved. With this in mind it may be a good idea to pay particular attention to fixing as many of these warnings as possible.

These are the remaining instruments with warnings now:

instruments/instr008.sbi: Modulator uses waveform 5: only 0-3 supported
instruments/instr008.sbi: Carrier uses waveform 5: only 0-3 supported
instruments/instr021.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr021.sbi: Carrier uses waveform 4: only 0-3 supported
instruments/instr024.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr024.sbi: Carrier uses waveform 5: only 0-3 supported
instruments/instr041.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr041.sbi: Carrier uses waveform 4: only 0-3 supported
instruments/instr042.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr042.sbi: Carrier uses waveform 4: only 0-3 supported
instruments/instr043.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr043.sbi: Carrier uses waveform 4: only 0-3 supported
instruments/instr063.sbi: Modulator uses waveform 5: only 0-3 supported
instruments/instr063.sbi: Carrier uses waveform 4: only 0-3 supported
instruments/instr065.sbi: Modulator uses waveform 5: only 0-3 supported
instruments/instr066.sbi: Modulator uses waveform 5: only 0-3 supported
instruments/instr067.sbi: Modulator uses waveform 5: only 0-3 supported
instruments/instr105.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr105.sbi: Carrier uses waveform 4: only 0-3 supported
instruments/instr106.sbi: Modulator uses waveform 4: only 0-3 supported
instruments/instr111.sbi: Carrier uses waveform 5: only 0-3 supported

Gez said:

Only covers Doom, though, not Doom II, which has different MIDI settings (GENMIDI and DMXGUS lumps differ) because they were optimized for a different selection of instruments. Given that most classic Doom mod soundtracks were written for Doom II megawads...

Very good point. It would certainly be prudent to pay more attention to making sure the Doom II tracks sound good. It should be plausible to write a script to generate similar data for Doom II's tracks as is found in this text file.

[shadow1013]

I've run FreeDOOM on vanilla (with my mapset ;) under dosbox with the DMXOPTIONS variable set to -opl3 (and I do get stereo so...) and it still sounded like a piano...

[fraggle]

I've run FreeDOOM on vanilla (with my mapset ;) under dosbox with the DMXOPTIONS variable set to -opl3 (and I do get stereo so...) and it still sounded like a piano...

It's possible that DMX masks out the top bits of the waveform registers and it might do it even in OPL3 mode. I believe the -opl3 and -phase options were only ever experimental so maybe it's a bug.