Automated way to check if a wad has unmodified id resources?

[Doomkid]

A lot of wads get rejected from idgames on the first attempt due to using unmodified id resources. It seems there's some automated process by which each lump is checked against the lumps in the iwads, which totally makes sense, however it results in a wad being rejected if one tiny lump is accidentally included without alteration.

 

Is there some automated way to check this? I think it would result in less headaches all around.

[ketmar]

you just gave me a nice idea for an utility.

[Mordeth]

I too would be interested. I've been told Slade does this, but attempts to make this trigger by intentionally including a vanilla patch didn't generate a warning.

[Gez]

SLADE has a feature for that: Archive->Maintenance->Remove entries duplicated from IWAD. AFAIK it's what the idgames maintainers use to quickly check.

 

To work, it needs you to set the IWAD as the "Base Resource Archive" so that it has something to which it can compare the entries.

 

Note that maps are excluded from the comparison, because it compares whether a lump has the same name and checksum as one from the reference IWAD, and maps are made of more than just one lump.

[Doomkid]

That is an excellent feature. Is there some way you can get it to identify duplicated lumps without necessarily deleting them in case it's an instance where the lumps are necessary? For example when sprites are partially replaced in a vanilla-compatible wad. If so, what I need already exists in full!

[Gez]

Not at the moment, sorry. I suppose it could be changed to show a confirmation list with checkboxes like the maintenance feature to remove unused textures.

[ketmar]

anyway, i already wrote a simple utility to do such checks. now i only have to add some options, and build windows binary. so maybe i'll release it somewhere in october, 2057.

[Bloodshedder]

4 hours ago, Doomkid said:

That is an excellent feature. Is there some way you can get it to identify duplicated lumps without necessarily deleting them in case it's an instance where the lumps are necessary? For example when sprites are partially replaced in a vanilla-compatible wad. If so, what I need already exists in full!

You can just copy and paste the list it gives you into Notepad or something and then don't save the changes to the WAD afterward.
 

[ketmar]

wadcheck v0.000000 ;-) README is inside the archive. sources will eventually be put in k8vavoom repo. done.

 

wadcheck_540413.7z

 

p.s.: it is not limited to iwads: it works by creating a database of known lumps, and then checks if lump is in database. i.e. you can create a database for any wad/pk3, and check any other wad/pk3 against it. of course, database can contain info for more than one "known wad", and the utility will show you properly attributed lump names.

 

as it uses hashing internally, there is some small chance of false positives. in practice, it should be so rare that it is irrelevant.

 

p.p.s.: the utility is quite dumb at the moment. maybe i'll add checking for "borrowed maps" later (i.e. it will check map geometry and report matches for it), some additional flags to ignore some lump types and such. dunno if somebody will need that, tho. anyway, enjoy.

Edited April 13, 2019 by ketmar

[Doomkid]

Thank you, this is a very nifty tool

[ketmar]

you're welcome. it wasn't heavily tested, so don't hesitate to report any bugs or things you'd like to see added/improved. i won't making any promises for some complex changes, but who knows... ;-)

[ketmar]

put sources to k8vavoom repository. not that anybody will be able to build it out of the box, but anyway...

[insertwackynamehere]

It would be kind of cool if it could also use a locality sensitive hash function in order to detect derivative works (recolors, altered sprites, sector copy and pastes, etc) probabilistically. That would require a bunch more work though and probably not be very popular in its application.

[ketmar]

i thought about that, bit it wasn't requested. it can be fun statistics, but tbh i don't see alot of other value in it besides being fun. and for sheer fun factor it is too much code, sorry. in the current form i just took some code from k8vavoom, and that is basically all of it.

Edited February 4, 2019 by ketmar

[insertwackynamehere]

Hahah yeah the fun factor is all I had in my head when I mentioned it 

[Doomkid]

Sorry for the bump, but I kind of think ketmar’s tool should be mentioned in the “how to idgames” thread from 2005. A veritable fuckton of wads seem to be getting rejected lately for containing iwad resources, a readily available solution to this problem should be made obvious so as not to deter newbie modders from using idgames. Just a thought!

[ketmar]

then i prolly should upload update with small bugfixes. will do that soon.

[ketmar]

ok, i replaced old version with slightly updated one. fixes are mostly cosmetics (but zip/lzma unpacker had bugs too), database format wasn't changed. output format slightly changed, tho. if you wrote some tools to parse old output format, use "--long" option to restore old output. have fun.

[Beezle]

8 hours ago, Doomkid said:

Sorry for the bump, but I kind of think ketmar’s tool should be mentioned in the “how to idgames” thread from 2005. A veritable fuckton of wads seem to be getting rejected lately for containing iwad resources, a readily available solution to this problem should be made obvious so as not to deter newbie modders from using idgames. Just a thought!

On the subject of rejected idgames uploads, is this where it shows em all? http://www.gamers.org/pub/idgames/REJECTS

[Jon]

I might add this as a feature into HWadTools