Closing Wad Archive

[smeghammer]

It's gonna take a while to get all the data files, though I did find a spare HDD to put my downloads onto.

 

For anyone who wants to see this data themselves, you will need MongoDB community, CLI tools, and (probably) the GridFS extension (not got all the data yet, so haven't tried setting up the binary data in a collection yet...). For MongoDB installation, I suggest using the zipped download rather than MSI - you just need the download directory and to add a /data/ directory for the - er -data. Also, download the tools and put them in the /bin/ directory too:

[root]/mongodb/bin/             <-- the unzipped download in here

[root]/mongodb/data/           <-- add this

 

Start it like so from within the bin/ directory above:

 

>mongod --dbpath c:\mongodb\data\

 

(you can add to PATH if you like of course, so you can start from anywhere)

 

If you get something like

C:\mongodb\bin>mongod --dbpath c:\mongodb\data\
{"t":{"$date":"2022-08-05T10:56:27.628+01:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true}}}
{"t":{"$date":"2022-08-05T10:56:27.630+01:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}

...

you are good to go.

 

Once it is running, open a shell (again from /bin/, unless you have set PATH) by:

 

>mongo

 

The last part is using the CLI tool mongorestore, to create a new collection from the JSON files. I used the lumps.json:

 

C:\mongodb\bin>mongoimport --db=wadarchive --collection=lumps --file=C:\Games\Doom\WADs\wad-archive\lumps.json

 

If all went OK, you can now browse and interact with the data in this new database/collection, either via the CLI shell (mongo.exe) or with Studio3T:

 

 

 

 

20 hours ago, WadArchive said:

Yes it is dump from mongodb, have a look at README.MD. Yea lumps.json is a biggie which is why I uploaded it compressed.

 

 

Thanks for this - I have lumps.json imported as a mongo collection OK.

Are the other JSON metadata files available please (wads.json,  filenames.json, readmes.json, additional.json and idgames.json)? Or are they included in some of the other ZIP files?

Edited August 5, 2022 by smeghammer
mongodb vs mongo... typo

[WadArchive]

17 hours ago, smeghammer said:

Are the other JSON metadata files available please (wads.json,  filenames.json, readmes.json, additional.json and idgames.json)? Or are they included in some of the other ZIP files?

They are all there, have a look at the archive listing (the 'Show All' link).

 

I have also upload a CSV that lists all the WAD SHA1 hashes and the filenames: https://archive.org/download/wadarchive/wads-list.csv

[smeghammer]

2 hours ago, WadArchive said:

They are all there, have a look at the archive listing (the 'Show All' link).

 

Doh! I'm an idiot!

 

Thanks.

[Doomlover77]

@WadArchive  I’m gutted. Love using wad archive. I’ve found it so useful for looking for old and new wad collections plus forgotten wads. I’ve collected the wad pictures from the wad archive to use for my doom wads database on my iPhone. @Doomlover77  

Edited August 6, 2022 by Doomlover77

[Ani]

We'll never forget you and your site man. Please take things nice and easy out there.

[RjY]

I am really sorry to hear this. I have used this site a great deal, it has been invaluable for identifying wads (if I guess the original slot of a map in a compilation wad, there is a good chance that I can compute the original wad file's md5sum, and then look it up in your database. This worked surprisingly often, e.g. E13.WAD E1M8 is an old version of Outpost Quark, DOOMPL2.WAD E1M7 is NETBEAST.WAD) and checking Guessthemap answers (it is much quicker to check the automap view of a map in your database than it is to download the wad, unzip it, and look at it in Eureka -- if indeed it is in a format that Eureka can open at all).

Many heartfelt thanks for running it over the now-seemingly short years it was available. Gone too soon! Now I understand how people who use Google services must feel :).

[smeghammer]

So I resurrected my WAD Downloader python app, and did a little bit of tidying. NOTE! I have not yet updated the Windows batch files!!!

 

I already wrote code to extract Doom WADs from WAD Archive, and I've tested it - it is still good.

 

If you are of a techie mindset, this might be a good alternative to downloading and extracting the WADs from the Internet Archive terabyte dump.

 

To use, you will need:

• An available mongoDB server
• python3 configured in PATH 
  • extra python packages: lxml/html5lib (for scraping); pymongo (for mongoDB interfacing); bs4 (BeautifulSoup - for scraping), Flask (for templates and API)
• optionally, an IDE (I use Eclipse, with pydev extension, which you can also get from the Eclipse marketplace via one of the IDE menus)

To run, you will need to do this:

 

For the crawler (collects links and puts in the database as a queue), you can run:

 

>./crawler.sh [DATABASE IP] [DOWNLOAD SOURCE]

where DATABASE IP is the IP address of your Mongo instance, and DOWNLOAD SOURCE is the string key for the site to crawl. These are currently:

 

• D (doomworld/idgames)
• DS (doomshack)
• W (wad archive)
• T (The sentinels playground)
• DWS (doom wad station)
• C (camoy)
• R667 (realm 667)

For the fetcher (fetches the WADs specified by the links in the database queue), you can run:

 

>./fetcher.sh [DOWNLOAD SOURCE]

 

This will find any entries in the queue flagged as NOTFETCHED for specified [DOWNLOAD SOURCE] and download the WAD/PK3 or whatever, to the wad-downloader/downloads/[DOWNLOAD SOURCE] directory. This can get quite big, quite fast...

 

The string key for the site as the same as for the crawler.

 

By all means look at the source code as well - I made the crawler code modular, class-based and uses class inheritance - hopefully to make creating new crawlers easier.

 

 

 

 

 

 

 

 

[Ralphis]

13 hours ago, smeghammer said:

So I resurrected my WAD Downloader python app, and did a little bit of tidying. NOTE! I have not yet updated the Windows batch files!!!

 

I already wrote code to extract Doom WADs from WAD Archive, and I've tested it - it is still good.

 

If you are of a techie mindset, this might be a good alternative to downloading and extracting the WADs from the Internet Archive terabyte dump.

 

To use, you will need:

• An available mongoDB server
• python3 configured in PATH 
  • extra python packages: lxml/html5lib (for scraping); pymongo (for mongoDB interfacing); bs4 (BeautifulSoup - for scraping), Flask (for templates and API)
• optionally, an IDE (I use Eclipse, with pydev extension, which you can also get from the Eclipse marketplace via one of the IDE menus)

To run, you will need to do this:

 

For the crawler (collects links and puts in the database as a queue), you can run:

 

>./crawler.sh [DATABASE IP] [DOWNLOAD SOURCE]

where DATABASE IP is the IP address of your Mongo instance, and DOWNLOAD SOURCE is the string key for the site to crawl. These are currently:

 

• D (doomworld/idgames)
• DS (doomshack)
• W (wad archive)
• T (The sentinels playground)
• DWS (doom wad station)
• C (camoy)
• R667 (realm 667)

For the fetcher (fetches the WADs specified by the links in the database queue), you can run:

 

>./fetcher.sh [DOWNLOAD SOURCE]

 

This will find any entries in the queue flagged as NOTFETCHED for specified [DOWNLOAD SOURCE] and download the WAD/PK3 or whatever, to the wad-downloader/downloads/[DOWNLOAD SOURCE] directory. This can get quite big, quite fast...

 

The string key for the site as the same as for the crawler.

 

By all means look at the source code as well - I made the crawler code modular, class-based and uses class inheritance - hopefully to make creating new crawlers easier.

 

 

 

 

 

 

 

 

 

Using this method, have you happened to find know how much of the 1TB are the wads themselves?

[smeghammer]

28 minutes ago, Ralphis said:

 

Using this method, have you happened to find know how much of the 1TB are the wads themselves?

No idea but looking at some of the zip files, there are many many screenshots so I suspect considerably less than 1TB for just the WADs. 

[Sneezy McGlassFace]

The king is dead, long live the king! 

[smeghammer]

On 8/7/2022 at 6:46 PM, smeghammer said:

This can get quite big, quite fast...

I filled my 200GB disk... 

 

I mean zero bytes free filled. SSH and mv to the rescue.

[DeathWalkerGT]

The end of an era...

I still remember searching WADs to play for hours, oh those were the days...and these days shall not be forgotten.

[BobJohnson]

Hey everyone, I have stumbled upon wad-archive is shutting down. Nice to see that everything is archived thanks to the owner.

I have created torrent for the DATA files in the archive: 

Quote

magnet:?xt=urn:btih:A46E74F337D2D1CD6BA4F2C2DDA292399B1367CB&tr=udp://tracker.opentrackr.org:1337/announce

 

I have capacity to get a small website going which would serve as a mirror with search and listing for this archived data. I will create separate thread related to it.

 

-- Edit --

MAGNET LINK UPDATED.

Edited August 23, 2022 by BobJohnson

[WadArchive]

You could just use the torrent provided by archive.org:

 

[Doomlover77]

Well I used the wad archive yesterday. Was upset to find out it had closed. RIP Wad Archive. It Was excellent using you on a day to day basis. 

[Maximum Matt]

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO

[Zero Master's Clone]

Is it just me or have a good number of good websites been getting shut down lately.

 

at least it will be archived that's one reason I love computers because you can store files that contain your nostalgia and make copies of them.

Edited October 12, 2022 by Zero Master's Clone

[Wadmodder Shalton]

Now I'll have to update both my Shareware CD PWAD Documenting and Compilation MegaWAD breakdown threads to include links to Internet Archive's ISO contents explorer in most cases.

Edited October 14, 2022 by Wadmodder Shalton

[XK 47]

On 8/4/2022 at 6:03 PM, Biodegradable said:

That's a shame, but uploading everything to Archive.org was the best move. Thank you for your service, @WadArchive.

You see pal, He may or may have not included All of the IWADS when Archiving. At least all the Important IWADS. Here's what I mean by that:

[UmbrellAssassin]

On 8/22/2022 at 4:00 PM, BobJohnson said:

Hey everyone, I have stumbled upon wad-archive is shutting down. Nice to see that everything is archived thanks to the owner.

I have created torrent for the DATA files in the archive: 

 

I have capacity to get a small website going which would serve as a mirror with search and listing for this archived data. I will create separate thread related to it.

 

-- Edit --

MAGNET LINK UPDATED.

Hello, I am getting an invalid url error  

[ludicrous_peridot]

Ooohhh, that's sad news, but I imagine for @WadArchive this an open opportunity to do something even more glorious with the archive no longer being a burden. So all good wishes and godspeed with whatever you are up to!

Does this mean however, that there's no publicly available search tool (for end users like me) based on index of LUMPS and not WADs. This was an immensely useful feature of the good old archive.

 

 

[CBM]

I recently lost all my doom stuff but I am looking for a wad of mine that I uploaded to wadarchieve

 

it was for a cp and the cp leader has asked me for another download link now that wadarchieve is gone

 

my file was called speedhorror.wad

 

it can be seen in the wayback machine version of the site but cant be downloaded from there

 

this is the link I had that is now invalid:

 

https://wadhosting.com/Wad/Download/E27E8B77D94E3A4C6FAE0F0310D36F881484C951

 

maybe a tech savvy person help me retrieve my wad?

 

I tried downloading the archieve stuff but I cant make heads or tails of it

 

my wad contains custom assets like textures and stuff and one of the textures is named something like cacoeye something I think

 

please someone help me find this wad.... my laptop got stolen recently and I have already lost two years of hard works worth of doom projects and unreleased stuff because I dont make backups often enough

 

 

[Herr Dethnout]

On 8/8/2022 at 7:10 AM, Sneezy McGlassFace said:

The king is dead, long live the king! 

idgames didn't died yet.

 

Spoiler

heh

 

[Herr Dethnout]

1 hour ago, CBM said:

I recently lost all my doom stuff but I am looking for a wad of mine that I uploaded to wadarchieve

Sadly the only way to find it is through the Archive.org backup. And it's almost 1tb of pure unnamed wads. You can search the wad through the link code (E27E8B77D94E3A4C6FAE0F0310D36F881484C951) but it will take a eternity and idk if this actually works lol

Did you at least give the wad to another person, or released in the forums so theres a chance that somebody has a backup?

[WadArchive]

Have a read of README.MD, it will tell you the structure files. You have the hash already (E27E8B77D94E3A4C6FAE0F0310D36F881484C951) so won't take forever. It will be in DATA/e2.zip

[Herr Dethnout]

OOOOOOO so the I was thinking that the 0x-Fx files was just a hexadecimal numbering, now I understand. lol

[smeghammer]

I can get that. See my thread at 

it will be later today. I can get it from the wad filename if you have that...

[smeghammer]

Hmm.

 

that GUID (7E8B77D94E3A4C6FAE0F0310D36F881484C951) is not inside the /E2 archive. When was it uploaded, and was the archive dump done before that? If so, it won't be in there.

 

Thereis also no record of it in the filenames database collection:

db.getCollection("filenames").find({'filenames.0':'speedhorror.wad'})
db.getCollection("filenames").find({'filenames.1':'speedhorror.wad'})

 

both return none.

 

It doesnt appear to be there at all:

 

db.getCollection("readmes").find({'_id':'E27E8B77D94E3A4C6FAE0F0310D36F881484C951'})
db.getCollection("lumps").find({'_id':'E27E8B77D94E3A4C6FAE0F0310D36F881484C951'})
 

also return none. Sorry.

 

[CBM]

3 hours ago, Herr Dethnout said:

Sadly the only way to find it is through the Archive.org backup. And it's almost 1tb of pure unnamed wads. You can search the wad through the link code (E27E8B77D94E3A4C6FAE0F0310D36F881484C951) but it will take a eternity and idk if this actually works lol

Did you at least give the wad to another person, or released in the forums so theres a chance that somebody has a backup?

sadly not so if it didnt get a backup made with the rest of the wads then its lost forever

 

@smeghammer

thanks for trying

 

it was uploaded on the 7th of october 2022

 

----

 

edit

 

my file is on the list when looking at the site using wayback machine but wayback didnt save the actual wad file

 

http://web.archive.org/web/20221010224535/https://wadhosting.com/

 

 

 

Edited November 23, 2022 by CBM

[Herr Dethnout]

20 minutes ago, CBM said:

sadly not so if it didnt get a backup made with the rest of the wads then its lost forever

 

@smeghammer

thanks for trying

 

it was uploaded on the 7th of october 2022

 

Welp, unfortunely this means that your wad is lost now. The archive backup was made on July, 10. :/

Edited November 23, 2022 by Herr Dethnout