How should I go about downloading everything from idgames quickly? >1 tranfer at a time just causes constant login prompts.

[deflatedhissy]

This is in coreftp - only client I know of that allows multiple files at once and/or multiple connections per file.  Or is this just how ftp is, and I'll have to put up with slow, sequential downloads?

[Xymph]

1 hour ago, deflatedhissy said:

This is in coreftp - only client I know of that allows multiple files at once and/or multiple connections per file.

At gamers.org, eating up many/all available connections with one client is considered rude. The limit is 1 connection per IP. On mirrors, YMMV.

Edited August 7, 2021 by Xymph

[deflatedhissy]

21 hours ago, Xymph said:

At gamers.org, eating up many/all available connections with one client is considered rude. The limit is 1 connection per IP. On mirrors, YMMV.

 

When someone updates a version of their level that's on idgames, do they use the same filename?  I'm a bit confused on how to get a ftp client to overwrite only files that have been changed since the 2020 mirror I just downloaded, and skip the rest.

[Xymph]

3 hours ago, deflatedhissy said:

When someone updates a version of their level that's on idgames, do they use the same filename?

That happens, but alternatively if the filename contains a version number (often the case for tools) then the name is updated too.

 

3 hours ago, deflatedhissy said:

I'm a bit confused on how to get a ftp client to overwrite only files that have been changed since the 2020 mirror I just downloaded, and skip the rest.

FTP is a poor protocol for that sort of thing, but you could grab the fullsort.gz file, filter on new files and download only those.

 

However, I don't quite understand why any individual user would want to download an entire 46 GB archive when the master archive and all its mirrors will never disappear -- at least until the climate crisis really goes out of bounds or Earth is hit by a massive meteorite. ;-)

[deflatedhissy]

15 hours ago, Xymph said:

That happens, but alternatively if the filename contains a version number (often the case for tools) then the name is updated too.

 

FTP is a poor protocol for that sort of thing, but you could grab the fullsort.gz file, filter on new files and download only those.

 

However, I don't quite understand why any individual user would want to download an entire 46 GB archive when the master archive and all its mirrors will never disappear -- at least until the climate crisis really goes out of bounds or Earth is hit by a massive meteorite. ;-)

 

I'm not all that familiar with wget, can you remember a specific wad that hasn't changed filename after an update, so I can check whether my client skipped it?  And I guess I want a mirror because it's a bit silly to download a frequently used wad multiple times when you have a local copy.

[Xymph]

Archive locally what you download, then downloading multiple times can be avoided. Also, man wget. ;)

[deflatedhissy]

On 8/9/2021 at 4:58 PM, Xymph said:

Archive locally what you download, then downloading multiple times can be avoided. Also, man wget. ;)

 

I've noticed some wads that are attached to a post instead of linked to on idgames, what happens when the forums go down?  Not ones that are hosted elsewhere, ones that have an actual attachment in the post with the doomworld domain name.

[Gez]

Forum attachments are a special thing, each user has a limited quota of virtual disk space for their attachments and once that quota is reached, they can't attach more stuff unless they delete old stuff. So any attached mod is at a threat of disappearance at their uploader's whims and/or necessity.

 

Personally, the only maps I've ever attached to posts are very basic test/demonstration levels to show how a feature works (or fails to work), so they don't have much value outside of the context of the thread they were posted in...

[DoomGater]

On 8/8/2021 at 3:49 PM, Xymph said:

(...)the master archive and all its mirrors will never disappear -- at least until the climate crisis really goes out of bounds (...)

Ok, I guess I better start my personal backup now...

[deflatedhissy]

How about backing up https://allfearthesentinel.net/?  It's huge compared to idgames and I recall there being an old version of the archive going down.