Disappearing history

[Quasar]

Despite our efforts to preserve it, especially at the wiki, bits and pieces of Doom history and community culture are vanishing at an increasing rate.

In the past couple of years (mostly this year), the following resources have all disappeared:

• NewDoom and its forums
  
• raven-games.com, including the hosting of a number of Heretic/Hexen series mods (especially some unique things for their sequels)
  
• altdeath forums
  
• files.drdteam.org
  
• rome.ro/johnromero.com, including the backup it had of Lee Killough's classic Doom information page. For a while, this was not even accessible through archive.org due to a redirect present at the domain.
  
• soulsphere.org, including several historically interesting files and pages, such as "Free Hissy" (fraggle may care to answer to this one)
  
• doom3world.org, previously the heart of Doom 3 modding
  
• doom3.filefront.com - while still "up," this site which hosted a ton of Doom 3 mods is now blocking downloads behind privacy-invading ads and surveys, making them effectively inaccessible
  

If that's not enough of a list to make an impression, we're also currently having trouble locating source or binaries for some historically significant source ports, such as DosDoom v0.2 (from which almost every other port derives), and Boom v2.00 (the first initial release).

Why are we not more proactive in creating permanent archives and why are our best efforts where they exist still insufficient, too slow and too short in reach, to prevent so much information loss? It's troubling to me.

[Linguica]

and ^{yet doomworld soldiers on}

[Coraline]

Yes, it is quite frustrating. I can relate solely because I've been trying to hunt down DOSDoom versions for the 3DGEWiki and there's an alarming rate of missing binaries and source packages. For such a historic port, even before it became EDGE, it's a little depressing to not find much trace of it anywhere. I scoured the ends of the web and found next to nothing.

Someone said something in an earlier thread -- we were not concerned with maintaining history in the 90's. Unfortunately, that turned out to be correct.

With all of the other community websites vanishing without much of a trace either, all we can do is try our best to crawl whatever is left to maintain some sort of footprint of what once was. Perhaps the collective thought process was, "Doomworld is still around, that'll never go!" - despite a few close calls.

[Tompig]

We just need to be more vigilant in future.

And just try our best in general that if someone asks for a vintage mod/file we try and source it too them asap.

[printz]

I think the community is going to concentrate on forums (like Doomworld and the ones specialized on source ports or other utilities) and the permanent /idgames repository. Everything else is temporary and dependent on maintainer's money. If you have any valuable data, please upload it to /idgames for posterity.

[Obsidian]

Dunno if anyone cares, but The Abyss forums are also shutting down soon. In case anyone's wondering, it's pretty much where a good portion of the NewDoom regulars ambled off to when the whole situation with WMull started going overboard. It's kind of empty nowdays though. :/ I'll be sad to see it go, it's a nice little corner of the internet.

[ETTiNGRiNDER]

What more do you propose? I'm not sure exactly how archive.org decides on when to make another copy of sites on its own but there's a way to prompt it to immediately archive a particular page, under https://archive.org/web/ (see the bottom right corner). I don't see any reason why source port files couldn't be uploaded to idgames by whomever has them. The Fusion port and site might be a good one to ensure the safety of if people haven't already.

This site (home of the Fusion port, mods thereof, some observations regarding mechanics in Hexen, etc.) comes to mind as one with some stuff that might not be around elsewhere: http://s91291220.onlinehome.us/

Since you mention in passing the Hexen 2/Heretic 2 stuff, trying to preserve Hexen 2 maps/mods and stuff is also of interest to me and if you have anything special in mind for that I'd appreciate a PM.

Edit: Couple other pages with maps/mods that seem to be disconnected from the mainstream community -
http://kenonserver.narod.ru/load/
http://www.hexenmagetower.com

[fraggle]

I'm concerned that a lot of really cool mods are going to disappear because people seem not to be uploading them to idgames. If the strict copyright rules on idgames are a problem we should set up a parallel archive.

Please give me a list of the soulsphere.org things you're missing and I'll try to locate them.

[Hekksy]

I was very disappointed when the Skulltag forums went down just because I would spend my spare time in-between games browsing the ancient history of the community on them. I learned so much browsing them back in 07 and 08.

Even more disappointing was that when they were up for the previous breaths there was a bug in the software that removed old posts each time a new post was created. So even if Carnevil allowed access to them again the good history is probably gone forever.

RIP Skulltag forums.

[Linguica]

Quasar said:

we're also currently having trouble locating source or binaries for some historically significant source ports, such as DosDoom v0.2 (from which almost every other port derives)

It's not exactly the same obviously, but PDoom, which we *do* have the source for, is a lightly edited DOSDoom v0.2.

Actually, here's a full diff between the original Linux release and PDoom, for fun (well, diff -r --ignore-file-name-case -i)

[Technician]

Linguica said:

and yet doomworld soldiers on

Inplying the cracks aren't already forming.

[Tompig]

Technician said:

Inplying the cracks aren't already forming.

I'd like to hope that if the whole doom community rallies under the doomworld banner AND we are given warning of any upcoming threats, we should be able to weather any storm.

[Coraline]

Linguica said:

It's not exactly the same obviously, but PDoom, which we *do* have the source for, is a lightly edited DOSDoom v0.2.

Actually, here's a full diff between the original Linux release and PDoom, for fun (well, diff -r --ignore-file-name-case -i)

!!! Awesome find! I've been looking for PDoom!

[Urban Space Cowboy]

Wall-o'-text imminent...

Quasar said:

Why are we not more proactive in creating permanent archives and why are our best efforts where they exist still insufficient, too slow and too short in reach, to prevent so much information loss?

This is a very good question, not just for the Doom community but for the Internet in general. Unfortunately, the inherent instability of the "files hosted on a single server on the other end of a string of routers" model seems to have caused Internet users to develop a sort of learned helplessness, disguised behind 2edgy4u "and nothing of value was lost" smarminess every time something happens like AOL user pages being nuked without warning in 2008, or Geocities being shut down in 2009.

I think the problem's going to keep getting worse because it gets better, for the following reasons:
1. Hosting on THE CLOUD. Old geezers may recall when "the cloud" was a semi-disparaging term because it referred to remote servers that were unspecified and unknowable (thus unreliable). Now, "hosting on THE CLOUD" and "THE CLOUD computing" are selling points?
2. Over-reliance on scripting/AJAX. Ever visited a site where the source code is near-empty and the page is blank until some scripts have a chance to run? Assuming they do run? Sucks, doesn't it?
3. Social networks where you can upload all you like but (unless specifically enabled on an individual basis, or you find a workaround) you can't download anything!
4. The increasing use of phablets to browse the web, run apps, and do basically everything. If you're an Android/iPhone user, you're expected by Baron Google or Lord Appleton to lock yourself into the Google/Apple walled garden, use only Google/Apple approved apps bought from Google/Apple stores, and give up all your private information for the greater glory of Googlistan/Applovia. And then there's the troublesome trend of Windows 10 following suit, and before you post "you can opt out lol", look up "tyranny of the default".

That's the problem. So how do we solve it? I can think of a few guidelines that might help:
1. Remember that profit motive is unhealthy for servers. If a service fails to turn "enough" profit (whatever "enough" may be), it may get disappeared a la Google, or made into a money-grubbing perversion of its former self a la SourceForge. I'm not sure how to address this; co-op ownership? There've been a few attempts at decentralized web hosting: WebTorrent, ZeroNet, IPFS, but none seem to have caught on.
2. Keep in mind that nothing lasts forever. Facebook has gone down three times in as many weeks. More close to home, Doomworld itself had a very close call recently, and the idgames archive's future was in doubt when Ty Halderman (R.I.P.) was hospitalized. Assume that an Internet server could easily be here today and gone tomorrow, and act accordingly.
3. If you're a server admin, then for goodness's sake don't depend on scripts/AJAX to render pages, and keep a local copy of any scripts you use! Go over your client's head if you have to, I won't snitch. ;) Think about this: if jquery.com dies or googleapis.com gets ownz0red, all your beautiful scripts that rely on them will be so much electronic dead weight.
4. If some host's demise is imminent and you don't have the energy/space/time to archive it yourself, get in touch with people who do! The Archive Team is the one group I know of. Any other suggestions would be appreciated. Apathy and smarm would not.

[ETTiNGRiNDER]

fraggle said:

I'm concerned that a lot of really cool mods are going to disappear because people seem not to be uploading them to idgames. If the strict copyright rules on idgames are a problem we should set up a parallel archive.

I think there are multiple things that might or might not turn people off to idgames besides the conservative copyright policy, such as the file size limit, the filename length limit, or the anonymous commenting system (even if the last isn't technically a part of the archive itself).

I myself have a released-elsewhere-but-not-on-idgames wad that I didn't upload because I'm not particularly proud of it (it was for a speedmap compo). I suppose I can send it on up to the idgames upload area if people think I should.

[Xaser]

fraggle said:

I'm concerned that a lot of really cool mods are going to disappear because people seem not to be uploading them to idgames. If the strict copyright rules on idgames are a problem we should set up a parallel archive.

Though the vanishing sites mentioned in the OP are indeed problematic, this is the issue that hits close to home for me me -- and I'm most certainly one of the offenders here.

This is maybe not the thread for this, but since it's tangentially related, the main reason is because the /idgames upload process is hoop-jumpy enough that I feel like I'm negotiating/signing a contract, with the user limits and upload delay and e-mail verification... heaven help me if I make a mistake somewhere or notice a bug after upload. The end result is I'm not comfortable putting a wad on it until it's 100% done, which rarely happens for any of my solo stuff. :P

[Quasar]

fraggle said:

I'm concerned that a lot of really cool mods are going to disappear because people seem not to be uploading them to idgames. If the strict copyright rules on idgames are a problem we should set up a parallel archive.

Please give me a list of the soulsphere.org things you're missing and I'll try to locate them.

    http://soulsphere.org/apocrypha/e1m1.html is linked from E1M1: Hangar (Doom)
    http://freedoom.soulsphere.org/ is linked from Freedoom
    http://www.soulsphere.org/ is linked from Talk:Doom source code files
    http://www.soulsphere.org/ is linked from User:Fraggle
    http://www.soulsphere.org/ is linked from Simon Howard (Fraggle)
    http://www.soulsphere.org/ is linked from Doom Wiki:Central Processing/2007
    http://www.soulsphere.org/freehissy/ is linked from Hissy
    http://www.soulsphere.org/projects/smmu/ is linked from SMMU
    http://www.soulsphere.org/projects/smmu/ is linked from Sfdoom
    http://www.soulsphere.org/projects/smmu/textdoom/ is linked from Text Mode Doom
    http://www.soulsphere.org/random/Search_logo.pngis linked from Doom Wiki:Central Processing/2006
    http://www.soulsphere.org/random/am_oids.zip is linked from Automap asteroids
    http://www.soulsphere.org/random/boomref.txt is linked from Talk:ZDaemon editing
    http://www.soulsphere.org/random/doom-hashsums.csv is linked from Doom files
    http://www.soulsphere.org/random/doom-md5sums.txt is linked from Talk:Doom files
    http://www.soulsphere.org/random/doom-md5sums.txt is linked from Talk:DOOM1.WAD
    http://www.soulsphere.org/random/doom-rpg-bnf.txt is linked from Talk:Doom RPG
    http://www.soulsphere.org/random/doom-rpg-bnf.txt is linked from Doom Wiki:Central Processing/2006
    http://www.soulsphere.org/random/slidoor.zip is linked from Door
    http://www.soulsphere.org/random/wads/my-first-levels.zip is linked from Doom Wiki:Central Processing/2006
    http://www.soulsphere.org/stuffage/bsp/ is linked from Doom rendering engine
    http://www.soulsphere.org/toke/ is linked from Dylan McIntosh (Toke)
    http://www.soulsphere.org/toke/tokedm.zip is linked from Dylan McIntosh (Toke)

[fraggle]

Should mostly be fixed now. A lot of those links already worked (they're just the old non-https: versions)

Only thing I can't find is the Free Hissy page. I think I deleted it a couple of years ago because I didn't realise anyone still cared about it. I expect I probably have a backup somewhere.

[Clonehunter]

Lets not forget that fact that we can't find photos of the Doom II gaming convention display that was covered in dildos.

[Revenant100]

fraggle said:

I'm concerned that a lot of really cool mods are going to disappear because people seem not to be uploading them to idgames. If the strict copyright rules on idgames are a problem we should set up a parallel archive.

I wish to echo this sentiment. We're at a point where many modern Doom releases are at a very real risk of being lost as users are choosing to only upload their files to temporary or unreliable hosts. idgames isn't even on the radar for these individuals, and this schism only continues to grow. Of course, this action is a double-edged sword as such volatile hosting solutions only ensure a guaranteed loss in the future, but I'd say that the archive's cumbersome and archaic uploading restrictions have already done much to discourage new users from even considering it as a viable option.

[Linguica]

Agreed... if only the /idgames maintainer(s) frequented these forums and we could discuss these issues...

[MrFlibble]

Quasar said:

If that's not enough of a list to make an impression, we're also currently having trouble locating source or binaries for some historically significant source ports, such as DosDoom v0.2 (from which almost every other port derives), and Boom v2.00 (the first initial release).

I'm sorry if this has been solved already, but being a fan of Boom I did a bit of Googling and found what seems to be binaries for v2.00 and v2.01 right here:
https://soulsphere.org/random/competn/sda/tools/

The same file (v2.00) can be found here. Or is that not the v2.00 you're missing?

[chungy]

The binaries are a nice find, but source is the actually interesting part that's still missing. If you can find it, that'd be great :)

[Andy Johnsen]

I should probably go through my Doom collection some day and try to salvage some bits I know are not available elsewhere. I'd upload the entire thing somewhere if it was not for the sprinkled bits of personal Doom related stuff in the mix (emails, logs unfinished projects and such).

I started meticulously archiving anything doom related back in the early 2k's, but unfortunately stopped around 2005-6, most of it is readily available through the known archives, but some is certainly not (like all of the Challenge-TV .zdos - painfully grabbed file by file before the site went down). I also have a bunch of old doom sites backed up, but none of them too significant.

Did a quick HD search for hissy for kicks, and this came up:

[Xeriphas1994]

To "learned helplessness" I would add "learned overabundance".  A dozen awesome megawads fall from the sky each year; why should I care if I can't find everything in the Top 100?  If I have a mapping problem, I get instant help on a forum; does it really matter if there are WAD specs and a wiki with collected instructions?  It takes quite a mental leap to even realize there's an issue.

(This is an overall cultural thing, not a Doom thing.  Nobody says, "There aren't enough cable channels.  I wonder what movies/sports were like in 1982."  Well, I've said it, but I'm a bit weird.)

For forums in particular, you'd have to find someone with sufficient resources who wants to preserve all their posts as a teenager.  :>

One (cheaper) action item might be to learn wget and make periodic backups, on physical media, of the archive.org material, as people have long done for /idgames.

[ETTiNGRiNDER]

Linguica said:

Agreed... if only the /idgames maintainer(s) frequented these forums and we could discuss these issues...

I think "won't use idgames" attitudes are far more common in the ZDoom and Zandronum circles than here, though.

[Gez]

The idgames archive also has a problem wrt. history in that when there were multiple versions of a mod uploaded in different places, the older ones would get deleted after a while. Which can be annoying when said older versions were very different and some content was entirely replaced and you want to check out the older version...

[Gentlepoke]

I can dig through my archives to see if I have anything of note. I believe when Geocities went down I did by best to save as much of the old Doom stuff that I could so there might be some things in there for example that might be of interest.

[Tompig]

ETTiNGRiNDER said:

I think "won't use idgames" attitudes are far more common in the ZDoom and Zandronum circles than here, though.

I'm a lil out the loop, why would anyone refuse to use idgames, just makes sense to put it all into one place dosent it?

[Arctangent]

Tompig said:

I'm a lil out the loop, why would anyone refuse to use idgames, just makes sense to put it all into one place dosent it?

Well for one, I've heard that it can be frustrating to actually connect to it and be able to put the file onto it, because it can take several attempts. I've never done it myself, though, since I don't have an FTP client installed.