Closing Wad Archive

[CBM]

20 minutes ago, Herr Dethnout said:

Welp, unfortunely this means that your wad is lost now. The archive backup was made on July, 10. :/

thats life I guess

[smeghammer]

Thats a shame. 

[Xyzzу]

Super late but rip Wad-Archive - the gift that kept on giving.

(nsfw)
 

Spoiler

 

[xX_Lol6_Xx]

On 10/24/2022 at 1:42 AM, XK 47 said:

You see pal, He may or may have not included All of the IWADS when Archiving. At least all the Important IWADS. Here's what I mean by that:

I know it's a month old commend, but I must say that has been there long before wad archive even closed (First uploaded in 2020 as a matter of fact). Obviously I won't post the link but you can look it up with a couple of keywords

[smeghammer]

So I got a Flask/Python thing working to get at the named WAD files inside the zipped data files, as I have been tracking in this thread.

 

It's kind of basic in the sense that it ONLY exposes the download of the WAD files, and shows the relevant readme and images if they are found. I can see from the database that there is a bunch of other information about the WAD itself that could be shown.

 

So...

 

If any of you guys are interested in using this yourselves, what additional info would it be useful to extract and display in addition to the download and images/readmes? And would detailed instructions on how to set up the components be useful?

 

Note it is not hosted anywhere online (the source web app files are hosted on my GH site) so it does need to be set up and run locally.

Edited November 27, 2022 by smeghammer

[Doomlover77]

Hi. Is it possible to access the wad archive through the internet wayback machine ? @Doomlover77  

[smeghammer]

Oh! Looks like you can yes.

[Doomlover77]

@smeghammer can you still download stuff? @Doomlover77 

update - Apparently you can only access wads through the categories and only download through Doomworld. 

here’s the link for anyone interested. 
 

https://web.archive.org/web/20220702112047/https://www.wad-archive.com/ dated 2/7/22. 
 

https://web.archive.org/web/20130712094006/http://www.wad-archive.com/ dated 12/7/13 - 9 years ago !!

 

You can see all snapshots of the website here from July 2013 to the present day by clicking on the year and choosing a snapshot on the following link:-

https://web.archive.org/web/20130615000000*/Http://www.wad-archive.com. Happy browsing. @Doomlover77  

Edited November 28, 2022 by Doomlover77

[Tom2007]

Rip wad archive is was the best places to get wads from :( 

[Doomlover77]

I’ve been going through the wad archive back up on archive.org this week and have managed to download a .gz file in the 01\ directory for Dragon1.wad, a doom level. I’m sure I’ll find some more obscure wads. @Doomlover77  

Edited March 18, 2023 by Doomlover77

[smeghammer]

On 11/28/2022 at 12:58 PM, Doomlover77 said:

@smeghammer can you still download stuff? @Doomlover77 

update - Apparently you can only access wads through the categories and only download through Doomworld. 

here’s the link for anyone interested. 

 

Not sure if you saw, but the place to get the most recent data dump is here:

https://archive.org/download/wadarchive

 

You will see that is dated from last year. Don't confuse the auto-archived stuff that archive.org will do periodically, with the active push that  @WadArchive made of the whole thing. My utility uses the /DATA subdirectory as the source of the archive files, and the JSON metadata to translate the GUID wad names to actual wad names.

 

Hope that helps.

[nmzz]

Figured out how to search for files (wasn't super clear from the thread):

 

Open filenames.json, search for the wadname then the first two hex numbers (example 1d) are the chunk number from the DATA/ folder ( https://archive.org/download/wadarchive/DATA/ ) you can then click "View Contents" of say "1d.zip" and download only that wad without downloading any of those ~4GB files.

Edited July 18, 2023 by nmzz
rewrite

[smeghammer]

3 hours ago, nmzz said:

Figured out how to search for files (wasn't super clear from the thread):

 

Open filenames.json, search for the wadname then the first two hex numbers (example 1d) are the chunk number from the DATA/ folder ( https://archive.org/download/wadarchive/DATA/ ) you can then click "View Contents" of say "1d.zip" and download only that wad without downloading any of those ~4GB files.

 

Yes. That is a proper PITA to do manually.

 

That JSON file is actually a MongoDB dump, so if you rebuild the database, you get searching for free... 

 

If you haven't seen it already, I reverse engineered this to rebuild the full database; a browser with filtering; map list per WAD with any images associated and and a downloader for the selected wad file (see here for details). The listing will present you with the actual filenames (rather than the UUIDs they are inside the zip archives).  It'll save as the proper filename as well, not the UUID.

[nmzz]

Would it be possible to host that freely on github pages or something?

[smeghammer]

No. GH Pages is for flat files (like my smeghammer site).

 

It would need to be something like an AWS instance, or maybe Atlas? Either way, the web hosting would need at least 4TB disk space (for the archives), Python3 with required packages, Mongo database, and a proper webserver (cherrypy is fine for local hosting, but probably wouldn't cut it for in-the-wild hosting - you'd need Apache or IIS etc. as well as a proxy from CherryPy to Apache).

[LoatharMDPhD]

So... I Ask... Who Among You, Are so Bold as to pickup that Torch, and Run With it Again...

[WadArchive]

23 hours ago, nmzz said:

Figured out how to search for files (wasn't super clear from the thread):

 

Open filenames.json, search for the wadname then the first two hex numbers (example 1d) are the chunk number from the DATA/ folder ( https://archive.org/download/wadarchive/DATA/ ) you can then click "View Contents" of say "1d.zip" and download only that wad without downloading any of those ~4GB files.

Try reading the readme https://archive.org/download/wadarchive/README.MD

[Wadmodder Shalton]

Someone should make a project similar to "The Maximum Doom Reference" document that provides links to each of the wads that are on the package available on Internet Archive.

[nmzz]

 

On 7/18/2023 at 3:24 PM, smeghammer said:

No. GH Pages is for flat files (like my smeghammer site).

 

It would need to be something like an AWS instance, or maybe Atlas? Either way, the web hosting would need at least 4TB disk space (for the archives), Python3 with required packages, Mongo database, and a proper webserver (cherrypy is fine for local hosting, but probably wouldn't cut it for in-the-wild hosting - you'd need Apache or IIS etc. as well as a proxy from CherryPy to Apache).

I'm thinking this could be much simpler, all we could need is some client side javascript providing a simple search functionality to generate links directly to the archive.org data.

[smeghammer]

Yes, I have been thinking about that.

 

You would still need to manipulate the big /filenames.json file to retrieve the filename mapping to UUID actual filename. Generating the link to the /data/xx/bigassuuidfilename.x should work in theory though. It might also be possible to link/use the image data too with some client-side code.

 

Doing it this way would save downloading a bunch of data of course.

 

I might have a crack at doing this anyway. If so, I'll add it to my smeghammer site.

TBH my python thing was an exercise in was it possible...

[Doomlover77]

I’ve copied the information from wad-list.csv into a spreadsheet and put in all the game info i.E. map replacement and the number of maps in each wad. This is still ongoing. What do you think? @Doomlover77  

Edited July 20, 2023 by Doomlover77

[smeghammer]

@nmzz- I have a PoC to show... It's a bit limited ATM because the downloaded filename is still the UUID. There may be a way to change that. Also, doing this client-side with GH Pages is limited by the large filesize limit (100Mb) and some of the JSON files exceed that.

 

Go to

 

https://www.smeghammer.co.uk/test3.html

 

and add a ?term=[your search term]

 

onto the URL ( e.g.: https://www.smeghammer.co.uk/test3.html?term=poison).  If there is a match with the filename, the relevant link(s) are built, and anchor tags rendered in the page. They are still gzipped, and have the long UUID-based filename, but you can extract and rename as you wish. One of the advantages of the python-based code is that the server process extracts from the gzipped archive and renames the file back to the actual original filename (as found in filenames.json) before offering to save. 

 

This is a PoC mind - there may be a way of doing some of this with CSJS. Certainly, adding a simple form, rather than having to manually add to the URL will be done. Just wanted to show the PoC first.

 

and @Doomlover77, that info is very useful!

 

Just FYI - if you do manage to get my code running, I do something similar with the database back-end - those JSON files are just mongo collection exports - so I query the database for each file to get the map info, the ID Games text file, the screenshots etc.

 

Edited July 20, 2023 by smeghammer

[Doomlover77]

@smeghammer an excellent job. Well done. @Doomlover77 Also I’ve constantly worked on my spreadsheet since July (it was commenced in March).  It will be completed by Christmas or the new year or possibly after that.  It’s coming on rather well. 

Edited September 17, 2023 by Doomlover77
A picture of my wad archive wad database with some wad information.

[smeghammer]

TADAAA!!!

 

https://www.smeghammer.co.uk/wad-archive/

 

Much better. 

 

You still need to manually extract the file from the archive, and it is still a UUID based filename, and I'd quite like to attach a keypress handler to the go button so you can just hit [enter], but hey it's a start...

 

Have fun!

[nmzz]

smeghammer: Yeah, that's exactly what I had in mind. Great job in making it happen and so quickly to boot!

 

Hopefully this gets pinned somewhere as this provide somewhat easy access to the archive.

[smeghammer]

Thank you! I will tinker with it a bit I think, as per above. I'd also like to make the page a bit nicer - an image or something... 

 

Spoiler

But definitely note the limitations (can't set content-disposition header, don't think I can change the filename to save as, can't auto-extract...)

 

Edited July 20, 2023 by smeghammer

[smeghammer]

OK tinkered with...

 

https://www.smeghammer.co.uk/wad-archive/
 

Spoiler

 

 

It will now show a graphic for the IWAD (if specified in the data), a text icon for the idgames readme, if present and an overlay actually showing the readme if you click on the icon. The overlay will stay sticky if you scroll and will open in the viewport regardless of how far you have scrolled. Also, at the moment Doom2, TNT and Plutonia are flagged with the Doom2 logo. I'll try and find suitable icons for these, and Final Doom.

 

 

 

[enter] will trigger a search, and [esc] will dismiss the readme overlay. 

 

You can use the form, or use ?term=[term] on the URL.

 

It might be quite slow, as the source data files are quite large, and all processing is client-side of course.

 

Spoiler

I still need to ensure any term on the URL is put into the search field, and vice versa. I also need to minify the javascript files...

 

I also want to do something similar to the README icons above, for screenshots, automap images, TITLEPICs etc. that are also often associated with each WAD as well. The principle will be the same - build links to the archive.org dump - but that is for another time.

 

 

Edited July 23, 2023 by smeghammer

[smeghammer]

>>> UPDATED! <<<

Now with screenshots!

 

So I added some code to display a paginated view of screenshots, if present in the source metadata. 

 

Absolute links are built to the archive, and local img objects are preloaded on selecting the screenshots icon from the search result. If more than one is found, we see a paginated display showing the first image, with links to display each of the subsequent ones.

 

The URL parameter and the text field are now in sync, and the overlay can be triggered with [enter] and dismissed with [esc].

New column indicating screenshots:

Spoiler

 

 

 

 

First page of paginated list of images:

Spoiler

 

 

 

 

And image 12:

Spoiler

 

 

 

 

 

 

Now I have the model, I can do the same for the MAPS and GRAPHICS, if found, too.

 

Note - you will likely need to hard refresh to update locally cached CSS and javascript files

Edited July 30, 2023 by smeghammer

[Doomlover77]

@smeghammer  this work is excellent. The website is better than your previous version. Keep it up. @Doomlover77 

[peido]

Hi.

 

I have had an idea that maybe can be done in the future to improve the archive. I'm not saying it is easy nor am I saying it is possible, I don't know, this is just an idea I had right before falling asleep in bed.

The idea is to have AI to complete every wad (it doesn't have to be speedrun) and to save the demo. Then, if the archive had a demo player, people could more easily see obscure wads before downloading them.

 

Now that I have written the idea down, it really seems kind of a dumb idea, but maybe in the future with the improvements of AI.