How to find all wads by year on idgames?

[Memfis]

I tried searching for "zip" in the textfile and sorting results by date: https://www.doomworld.com/idgames//index.php?search&page=485&field=textfile&word=zip&sort=time&order=asc

 

But that doesn't seem to include everything. For example I don't see this wad in the results linked above: https://www.doomworld.com/idgames/levels/doom2/m-o/nevermo2 It's from 06/16/04 so it should be between 06/10/04 and 06/17/04 but it isn't?

[Memfis]

Oh I know, it's because "zip" isn't actually in the textfile... What should I do then?

[Ribbiks]

I'm assuming the api still works? https://www.doomworld.com/idgames/api/

 

shouldn't be too hard to write a script that crawls through the metadata of every upload, then sorts by date added to archive. there'd probably still be some weird cases or slight inaccuracies, but yeah...

[Ribbiks]

Hi. so I ran this. I attached a sorted list of files, everything in the /levels/ directory, excluding the deathmatch/ sections.

 

Spoiler

import os, time, urllib2, json
import cPickle as pickle

BASE    = 'https://www.doomworld.com/idgames/api/api.php?action=getfiles&name='
D1      = ['levels/doom/','levels/doom2/']
D2      = ['0-9/','a-c/','d-f/','g-i/','j-l/','m-o/','megawads/','p-r/','Ports/','s-u/','v-z/']
PORTS_D = ['0-9/','a-c/','d-f/','g-i/','j-l/','m-o/','megawads/','p-r/','s-u/','v-z/']

if not os.path.isfile('allDat.p'):
	allDat = []
	for d1 in D1:
		for d2 in D2:
			if d2 == 'Ports/':
				for d3 in PORTS_D:
					myDir = d1+d2+d3
					myCmd = BASE+myDir+'&out=json'
					resp  = urllib2.urlopen(myCmd)
					allDat.append(json.loads(resp.read()))
			else:
				myDir = d1+d2
				myCmd = BASE+myDir+'&out=json'
				resp  = urllib2.urlopen(myCmd)
				allDat.append(json.loads(resp.read()))
	pickle.dump(allDat,open('allDat.p','wb'))

#
allDat = pickle.load(open('allDat.p','rb'))
sorted_dat = []
for i in xrange(len(allDat)):
	if 'content' in allDat[i]:
		for file_dat in allDat[i]['content']['file']:
			sorted_dat.append([time.strptime(file_dat['date'],'%Y-%m-%d'),file_dat['date'],file_dat['url']])
	else:
		print allDat[i]
		exit(1)
sorted_dat = [[n[1],n[2]] for n in sorted(sorted_dat)]

for n in sorted_dat:
	print '\t'.join(n)

 

 

sorted_dat.txt.zip

[Memfis]

Man, this is so useful, thanks a lot!

[Nevander]

I'm not familiar with this kind of thing but what is the deal with the pickles? Importing pickles, pickle loads and even pickle dumps!

[paymentplan]

This is neat! I'm gonna download all the WADs from my birthday across the years and play 'em next time it rolls around!

[Ribbiks]

@Nevander pickles are just serialized python objects. it's a way to immediately save and reload arbitrary variables from disk without having to parse anything or having to use a specific format. They're only in that little script because my initial thought was that grabbing all the metadata from /idgames might take awhile, and I wanted to save it to disk so that I wouldn't have to do it more than once. it turns out that was completely unnecessary because it takes all of 10 seconds to harvest it all, but I didn't feel like cleaning up my code snippet before posting it so you get to see the sloppy version :p

[PVS]

Ribbiks, you can make the same sorted list for Heretic/Hexen wads?
 

[Grazza]

The archive contains a ready-made and constantly updated fullsort document in the root directory.

 

Quote

I tried searching for "zip" in the textfile

You should be searching for zip in the filename.

Edited August 12, 2017 by Grazza

[PVS]

Grazza thanks, this full sort list is useful to me.