DSDA api

[GoneAway]

I've added an endpoint here to query records:

 

https://dsdarchive.com/api/demos/records?wad=heretic&level=E1M1&category=Pacifist

 

Response: {"player":"Kimo Xvirus","time":"0:27","category":"Pacifist","level":"E1M1","wad":"heretic","engine":"Heretic v1.3 + vv","date":"2009-02-22T03:06:50.000Z"}

 

(I can add more details to the response, it's just a proof of concept.)

 

This could be used, for instance, with irc / discord bots, for pages on other sites to not need manual updates, etc.

 

What would you like to see out of a dsda API?

[Andrea Rovenski]

I like this. Maybe wad specific stats like total time, total demos, first demo (date, demo), last demo (date, demo), etc

[Tristan]

Nice, this should be a godsend for the Doom Wiki - I tried on a couple of occasions to push along the maintenance of the speedrunning tables there, but with now over 5000 map articles and counting it's just an impossible task, no one has that much time on their hands. For Pr+ demos would it be possible for the response to show the demo's complevel?

[Nine Inch Heels]

2 minutes ago, Eris Falling said:

For Pr+ demos would it be possible for the response to show the demo's complevel?

I'm not coding this, but afaik the complevel always gets baked into the demo file, so I suppose it can be read.

[GoneAway]

1 hour ago, Eris Falling said:

For Pr+ demos would it be possible for the response to show the demo's complevel?

That info is included in the engine column right now, e.g.,

 

https://dsdarchive.com/api/demos/records?wad=enirvana&level=Map 19&category=Pacifist

 

{"player":"4shockblast","time":"9:30.37","category":"Pacifist","level":"Map 19","wad":"enirvana","engine":"PrBoom+ v2.5.1.4 cl9","date":"2017-07-28T20:51:56.000Z"}

Edited May 8, 2019 by kraflab

[Xymph]

On 5/8/2019 at 9:29 PM, Eris Falling said:

Nice, this should be a godsend for the Doom Wiki - I tried on a couple of occasions to push along the maintenance of the speedrunning tables there, but with now over 5000 map articles and counting it's just an impossible task, no one has that much time on their hands.

Adding/updating DW demo tables via a XymphBot script has been on my to-do list for a long time, pending API availability, so yes, this is most welcome.

 

I don't really have any insight yet what data will be needed to accomplish that, so perhaps someone familiar with DW demo coverage (Eris?) can advise what API features would be useful?

Beyond that, desired endpoints and attributes will probably not come up until I would start building a script, which is quite a ways off due to (non-DW) things higher on my list.

[Tristan]

I'm neither a technical wizard or a speedrunning guru, but it looks to me like the API gives us all the info we'd need - I didn't have any plan to change the format of the tables on the wiki either aside from adding -nomonsters to the main table https://doomwiki.org/wiki/User:Eris_Falling/Sandbox#Speedrunning

[GoneAway]

I guess one thing missing is a link to the file.

[Xymph]

33 minutes ago, kraflab said:

I guess one thing missing is a link to the file.

Obviously. And a notes field?

 

Edit: query by wad ID (e.g. 947 for Doom IWAD) would be most helpful, since I already have those in XymphBot .ini files.

Edited May 17, 2019 by Xymph

[GoneAway]

The id for doom is "doom" in the new dsda; I don't have any plans on preserving those numbers 😛

[Xymph]

1 hour ago, kraflab said:

The id for doom is "doom" in the new dsda; I don't have any plans on preserving those numbers 😛

Please reconsider that. Wad IDs (and optionally map IDs too) are used in the dsda template and only the template would need to be updated to point to the new site and WAD/map pages.

With a different ID system, some 4500+ DW articles would need to be updated which, even with a bot script, would be painful and time-consuming -- as would manually looking up the 380+ new IDs and updating the bot .ini files.

Edited May 17, 2019 by Xymph

[GoneAway]

Considering the scope of this project, updating links on the wiki doesn't strike me as a big task anymore, since it should be automatable. 🙃

 

The template has a title, which could be used to make the links instead, but only if there aren't duplicates, so I don't know if that's doable.

[Xeriphas1994]

FWIW: What about an API query that takes the md5 (or 8.3 filename perhaps) of a WAD and returns its ID?

 

For the bot, that means a one-time code update to replace Andy's IDs, then subsequent runs make two passes, one to retrieve the IDs and one to get the records. If the IDs later change again (backend upgrade or whatever), it keeps working.

 

As a bonus, it could prevent associating a demo with the wrong version of a megawad.

Edited May 18, 2019 by Xeriphas1994

[GoneAway]

Not seeing any reason to add an extra step in between wad and id, especially one that needs to be changed every time a wad is updated.

 

What version of a wad a demo belongs to is a separate topic and a nonissue, so I don't see that being a relevant benefit either.

[Xymph]

9 hours ago, kraflab said:

Considering the scope of this project, updating links on the wiki doesn't strike me as a big task anymore, since it should be automatable. 🙃

Despite automation, it is still a big task to me. Bot account actions aren't autonomous; a bot script is run on each .ini file separately, under human supervision so it can be aborted in case a malformed edit comes along (e.g. due to a non-standard entry that the script didn't take into account yet). No-one is going to review thousands of bot edits afterwards, so I believe that monitoring is important. While running across multiple .ini files can be automated further, monitoring updates on 4500+ pages (the other 1000-odd map articles don't have DSDA links) is still very time-consuming.

 

And before that, updating the .ini files is a time-consuming task in itself. Could you at least provide a mapping list of old numeric ID to new alphanumeric ID so I can try to script those updates? It should not be hard to create that as a by-product of the import phase from the old site.

 

9 hours ago, kraflab said:

The template has a title, which could be used to make the links instead, but only if there aren't duplicates, so I don't know if that's doable.

The optional title field determines how a link is shown in the article, and cannot and should not be used to control the external link.

 

The whole point of this template (and many like it) is that the external links can easily be updated in a single place if the IDs are preserved, and it'll be frustrating if that won't happen here.

[GoneAway]

Ya, I'm sure we can work out some kind of redirect path that takes legacy IDs.

[Xymph]

16 minutes ago, kraflab said:

Ya, I'm sure we can work out some kind of redirect path that takes legacy IDs.

Thanks, I'd appreciate that very much.

 

For new WADs without old ID, we can create a new dsda template and use that in future map articles.

[Xymph]

On 5/8/2019 at 10:31 AM, kraflab said:

What would you like to see out of a dsda API?

I started to explore approaches to script-adding speedrunner/demos information to the wiki. My initial API wishes:

 

- https://dsdarchive.com/api shows available routes and parameters as a HTML help page

- https://dsdarchive.com/api/players lists info like https://dsdarchive.com/players does

- ditto for https://dsdarchive.com/api/players/adam_hegyi like https://dsdarchive.com/players/adam_hegyi

 

Based on your example in the initial post, I tried constructing the endpoint for "some" map but the API request returns 'not found'. Where's my error?

[Andromeda]

27 minutes ago, Xymph said:

Based on your example in the initial post, I tried constructing the endpoint for "some" map but the API request returns 'not found'. Where's my error?

Seems to work with "level=Map 01" instead of "level=MAP01".

[GoneAway]

My eventual goal would be to have all of the content of the current site exposed as a json api, so that's certainly in line with the long term plan. Probably set up documentation with swagger or a similar tool.

[Xymph]

On 5/18/2019 at 1:21 PM, kraflab said:

Ya, I'm sure we can work out some kind of redirect path that takes legacy IDs.

Now that Andy's site has shut down, this has become vital for the wiki, which had 9100+ links pointing into the void. I've archive.org-ed links in relevant templates but demo zips are typically not archived there, so the templates should really point to the new site, with the numeric wad/player IDs getting redirected to the new names. That would only be the short-term solution of course.

 

The longer-term solution is to provide two mapping tables (in .[ct]sv or .json format?) of old IDs to new names, so that all old dsda* templates on the wiki can be changed into new dsda2* ones via XymphBot scripts.

 

And then the API would need to be finished and documented, so that record tables (with zip links) can be updated script-wise, like I've already done for all Compet-n record tables.

Can I ask to please prioritize this for the wiki's sake?

[GoneAway]

I'm gonna be completely honest with you - this is not a high priority for me right now. I understand that it's a big issue for you, but there is a stack a mile high of things that I need to do. The mapping is a bit complicated now because players have asked for name changes, some wads have been reorganized, etc. I don't think we can completely avoid manual labor, but some things can be done automatically. What you could do in the meantime is take the wad file names (e.g., scythe) and make a request to the site to see if the page exists. If it does then add a link. That would hopefully automate and cover most of the wads, and then there would be a small list of things to find manually.

 

As far as the tables, I really struggle to see duplicating the data on dsda as being vital, to me. Once again I totally understand why this is important to you and it's something I plan to work on long term but it's just not high on my list.

[Xymph]

I see. Perhaps we need to rethink whether to maintain record tables per map article, but that's a wiki discussion. The WAD links need to be updated regardless, but I momentarily blacked out on the fact that I already mapped the player IDs/names, at least for the speedrunners covered in the wiki. So that part of the request is dropped. Meanwhile...

 

6 hours ago, kraflab said:

The mapping is a bit complicated now because players have asked for name changes, some wads have been reorganized, etc. I don't think we can completely avoid manual labor, but some things can be done automatically.

... I'd really want to approach the WADs from the ID angle, which are configured separately in the bot .ini files for this very purpose, and not filenames or anything else. If you could just export that one table in any format that's easy for you, without manual reorganizing on your end, then I can first automatically update/verify templates and second deal manually with any stranglers/mismatches. Then I'll let you off the hook on the redirects and the API. :-) Please do at least this one little thing for the wiki?

[GoneAway]

By table, do you mean old dsda ID in one column and new dsda address in the other? Like "44 scythe"?

[Xymph]

9 hours ago, kraflab said:

By table, do you mean old dsda ID in one column and new dsda address in the other? Like "44 scythe"?

Yup. Except the old ID of Scythe is 789. :)

Edited November 14, 2020 by Xymph

[GoneAway]

I don't think we have that information anywhere, so it would require scraping archived pages, which would be the same work for either of us. I'll check if someone on the team has this though.

[Xeriphas1994]

On 7/13/2019 at 3:12 PM, 4shockblast said:

The DSDA is an archive at its core, sure, but it would be incorrect to say that it isn't regularly used as a leaderboard.

 

a new player or casual viewer won't really care about the minor details of HUD vs not or 2-key strafe50; they will just want to know what the best non-TAS time is

 

Would it be easier to implement a URL parameter displaying records only? (I think AdamH had one.)

 

On 11/13/2020 at 8:47 AM, kraflab said:

I really struggle to see duplicating the data on dsda as being vital

 

In purely database terms, that's true, but the wiki seems to accumulate redundant content to combat entropy -- Doomers work for free after all, so we try not to take others' projects completely for granted. Or perhaps the reader has a slow/metered connection and won't always click links. Whatever the reason, wiki users maintained DSDA links manually for a decade, so someone must have thought it was a great idea. :>

 

Presumably, thousands of broken file links also affect Google rank. I'm not fanatical about that but others are.

[Keyboard_Doomer]

@Xymph

Not sure what format is best for you so I just did old full link and new full link separated by a tab. Let me know if you want me to generate it in a different format.

I didn't do any checks if the links to the new DSDA are correct or even exist so it does have the caveats kraflab mentioned - removed/merged/renamed wads or usernames. I corrected a few that I know of manually but there still will be plenty to fix.

 

dsda_mappings.zip

[Xymph]

1 hour ago, Keyboard_Doomer said:

@Xymph

Not sure what format is best for you so I just did old full link and new full link separated by a tab. Let me know if you want me to generate it in a different format.

I didn't do any checks if the links to the new DSDA are correct or even exist so it does have the caveats kraflab mentioned - removed/merged/renamed wads or usernames. I corrected a few that I know of manually but there still will be plenty to fix.

 

dsda_mappings.zip

I can take it from here, thanks.

[Xymph]

On 11/16/2020 at 5:20 PM, Keyboard_Doomer said:

@Xymph

Not sure what format is best for you so I just did old full link and new full link separated by a tab. Let me know if you want me to generate it in a different format.

I didn't do any checks if the links to the new DSDA are correct or even exist so it does have the caveats kraflab mentioned - removed/merged/renamed wads or usernames. I corrected a few that I know of manually but there still will be plenty to fix.

 

dsda_mappings.zip

Yes, some two dozen wad paths (relevant to the Wiki) needed updating, including most Master Levels entries.

Oddly, https://dsdarchive.com/wads/diamonds is missing (for https://doomedsda.us/wad1598.html) - why is that?

 

Thanks again, here is my fixed list: dsda_wads-fixed.zip

Edited December 2, 2020 by Xymph
Update wads list