Simple linear format

[Quasar]

I think it would be useful to have a simple headered linear graphics format, as linear data without a header is liable to be misinterpreted, and/or its details may be forgotten, rendering the image difficult to extract or reuse in the future. I've looked at various existing formats and all have trappings that make them unattractive for various reasons.

Plenty of paint programs on the other hand can load .raw, skipping an optional number of header bytes.

Here's how I'd define such:

#pragma pack(push, 1)
struct linear_t
{
   uint8_t  hdr[8];   // Offs  0  - Must be "DLINEAR\x08"
   uint16_t width;    // Offs  8  - Raster width
   uint16_t height;   // Offs  A - Raster height
   int16_t  topoffs;  // Offs  C - Top offset
   int16_t  leftoffs; // Offs  E - Left offset
   uint8_t  transp;   // Offs 10 - Transparency flag
   uint8_t  maskcol;  // Offs 11 - Mask color if transp != 0
   uint16_t resv1;    // Offs 12 - Reserved, must be 0x0000   
   uint32_t resv2;    // Offs 14 - Reserved, must be 0xFFFFFFFF
};
#pragma pack(pop)
// assert(sizeof(linear_t) == 24)
// Raster data follows structure immediately.
// All values are little endian.

Now I'll wait to hear how this is unnecessary, redundant, etc. even though PNGs are super-bloaty for small paletted graphics and no external utilities can read patches.

[Csonicgo]

in b4 graf and danj for 3 pages

[Tristan]

inb4 "I didn't scroll to the right"

[DaniJ]

Effectively a doom-Texture with magic bytes?

ZDoom already implements a textual TEXTURES.

EDIT: Ah I see, an image format. I can't imagine you'd talk the likes of Adobe into supporting it, so where is the benefit in us supporting another adhoc image format given there will be no art packages which can save it?

[Graf Zahl]

So...

Why a new format?

Sure, PNG may not be the most efficient way to do this stuff - but it has one huge advantage over some home brewn stuff: There's lots and lots of software that can directly manipulate or create it.

For this you'd have to ask each tool developer to add support for it.


To sum it up: No, I see no real point in this, especially since ZDoom already defines a format along these lines (check imgztexture.cpp, you can ignore the compression thing in there if you are dead set against it, as it's optional)

Not surprisingly, the only thing it's ever been used for is some decals in zdoom.pk3, all made by Randy.

[Quasar]

Graf Zahl said:

So...

Why a new format?

Sure, PNG may not be the most efficient way to do this stuff - but it has one huge advantage over some home brewn stuff: There's lots and lots of software that can directly manipulate or create it.

For this you'd have to ask each tool developer to add support for it.


To sum it up: No, I see no real point in this, especially since ZDoom already defines a format along these lines (check imgztexture.cpp, you can ignore the compression thing in there if you are dead set against it, as it's optional)

Not surprisingly, the only thing it's ever been used for is some decals in zdoom.pk3, all made by Randy.

PSP and Paintshop can both manipulate .raw formats, you just have to tell the first (at least) how many header bytes to skip over. This allows you to, at least, load and then save them into a different format. Converting *to* the format would require saving in .raw and then filling in the header bytes manually (it'd be better if an editing util such as SLADE were to handle that kind of thing importing an arbitrary graphic, admittedly).

Where's the spec on the ZDoom format? Is it already supported by any external tools? Does it also share the benefit of being .raw loadable?

EDIT: Found it. Couple of issues I see are assuming palette index 0 is transparent (fuck your graphics if it's not) and no documentation on required values of the reserved area (this is important for format verification purposes).

[Csonicgo]

RAW has been supported since the win3.1 days with PSP.

[Gez]

IMGZ is kinda specialized since it is meant for graphics which only have an alpha channel. Specs are, naturally, on the wiki. External tool support is included in SLADE 3 though it's read-only. (Adding conversion to this format wouldn't be too hard, but nobody ever requested it.)

Speaking of formats supported by SLADE 3; there's what I dubbed "arah" which is a format found in the Doom Alpha which is r and has a header.

As for the bloatiness of PNG for small graphics, it's not that bad if you use an optimizer. Look at, for instance, unkna0.lmp in zdoom.pk3: 222 bytes. Convert it to Doom format: 840 bytes.

Edit: speaking of PNG optimization and bloatiness, Eternity's startup.wad:

01:26:21: PNG FRAME size 48193 =PNGCrush=> 0 =PNGout=> 40670 =DeflOpt=> 40337 =+grAb/alPh=> 40337
01:26:21: PNG DOOMSW size 46885 =PNGCrush=> 0 =PNGout=> 44324 =DeflOpt=> 44312 =+grAb/alPh=> 44312
01:26:22: PNG DOOMREG size 48171 =PNGCrush=> 0 =PNGout=> 45724 =DeflOpt=> 45709 =+grAb/alPh=> 45709
01:26:22: PNG UDOOM size 48274 =PNGCrush=> 0 =PNGout=> 45818 =DeflOpt=> 45805 =+grAb/alPh=> 45805
01:26:23: PNG DOOM2 size 45275 =PNGCrush=> 0 =PNGout=> 42678 =DeflOpt=> 42670 =+grAb/alPh=> 42670
01:26:24: PNG TNT size 32100 =PNGCrush=> 0 =PNGout=> 29590 =DeflOpt=> 29575 =+grAb/alPh=> 29575
01:26:24: PNG PLUTONIA size 35588 =PNGCrush=> 0 =PNGout=> 32463 =DeflOpt=> 32461 =+grAb/alPh=> 32461
01:26:25: PNG HACX size 36391 =PNGCrush=> 0 =PNGout=> 34368 =DeflOpt=> 34362 =+grAb/alPh=> 34362
01:26:26: PNG HTICSW size 23039 =PNGCrush=> 0 =PNGout=> 20517 =DeflOpt=> 20517 =+grAb/alPh=> 20517
01:26:26: PNG HERETIC size 29009 =PNGCrush=> 0 =PNGout=> 26577 =DeflOpt=> 26569 =+grAb/alPh=> 26569
01:26:27: PNG HTICSOSR size 37272 =PNGCrush=> 0 =PNGout=> 35112 =DeflOpt=> 35103 =+grAb/alPh=> 35103
01:26:27: PNG HEXEN size 32803 =PNGCrush=> 0 =PNGout=> 29799 =DeflOpt=> 29792 =+grAb/alPh=> 29792
01:26:28: PNG STRIFE size 18557 =PNGCrush=> 0 =PNGout=> 16645 =DeflOpt=> 16631 =+grAb/alPh=> 16631
01:26:29: PNG FREEDOOM size 29052 =PNGCrush=> 0 =PNGout=> 27287 =DeflOpt=> 27278 =+grAb/alPh=> 27278
01:26:30: PNG ULTFD size 29640 =PNGCrush=> 0 =PNGout=> 27874 =DeflOpt=> 27858 =+grAb/alPh=> 27858
01:26:30: PNG FREEDM size 28661 =PNGCrush=> 0 =PNGout=> 26867 =DeflOpt=> 26849 =+grAb/alPh=> 26849

[Quasar]

An 800 byte 9x9 DOOM patch turns into a 10 KB PNG. I'd call that bloaty.
Sorry that was wrong: it's a 157 byte patch that turned into a 1 KB PNG.

[Gez]

Look at sgrna1.png in zdoom.pk3. 8x8 PNG, 258 bytes. 39 times smaller than the 10 KB figure you advance.

[Quasar]

Gez said:

Look at sgrna1.png in zdoom.pk3. 8x8 PNG, 258 bytes. 39 times smaller than the 10 KB figure you advance.

Paletted or RGB? I've been told RGB PNGs are much smaller. The disadvantage they have is that you must remap/quantize the colors on load, so translations cannot be reliably applied.

As far as the example image above, that's no go considering it's a font letter.

[DaniJ]

Obviously a PNG is going to be more "bloaty" even in paletted mode - the palette is included in the image file.

[Graf Zahl]

Gez said:

IMGZ is kinda specialized since it is meant for graphics which only have an alpha channel. Specs are, naturally, on the wiki.

Actually, no, it can also be a paletted image using the game palette - it's just that nobody has ever used it that way.

DaniJ said:

Obviously a PNG is going to be more "bloaty" even in paletted mode - the palette is included in the image file.

... and obviously that's unavoidable in a self-contained format.

[DaniJ]

Exactly. As far as size on disk goes, you're going to be hard pressed to beat the original format (though you could apply some RLE or quadtree based compression if you're really desperate to shave a few further bytes).

[Quasar]

And so now you guys maybe grasp the motivation of a simple linear format. No palette needed, no compression features that are mostly redundant to the preferred container format (.zip), and ease of loading at runtime (just cache the lump and go, mostly).

[DaniJ]

I get the format. I just don't see any point in redesigning what we already have to effectively add some magic bytes and (domain-specific) xy offsets.

[Gez]

And so, what about the arah format?

It uses the same header as a patch (width/height/x and Y offsets), but the image data itself is raw. Drawbacks: transparent index is always 255, no room for flags or identifier string, no reserved stuff for future expansion. Advantage: even smaller, support already exist in certain utilities, reuses pre-existing Doom structures.

Quasar said:

Paletted or RGB? I've been told RGB PNGs are much smaller. The disadvantage they have is that you must remap/quantize the colors on load, so translations cannot be reliably applied.

A PNG's palette doesn't need to contain 256 colors, so if you're only using, say, 8 colors total, then the palette data will only take 24 bytes.

[DaniJ]

I thought I should expand on my previous comment because reading the thread back, I know I would feel somewhat despondent if it were I who proposed the format.

Now, I'm going to state the obvious because I think a reality check is needed. This is not 1993 and we aren't all sat in a developer meeting with John Carmack, Romero et al deciding how to improve the DOOM data structures for the benefit of source ports which we know will exist twenty years from now.

What you propose is a slight revision of a format which has been in active use for nearly two decades. Regardless of whether a new format is introduced now, we must still continue to support the existing one.

As such the format can really only be considered "ready-to-render" for those ports which implement a uni-paletted renderer. Consequently, the performance drawing aspect of this is already moot to many ports.

So, what are the other merits? The first obvious one is more reliable recognition. However as this isn't 1993 this one is also moot because as said, we we must continue to support the original format.

The only remaining benefit that I can see is the presence of width and height properties, meaning we won't have to guess at or adopt commonly used logic to attribute them automagically.

All things considered, I don't feel this is worth introducing yet another bespoke format for.

[Quasar]

Gez said:

And so, what about the arah format?

It uses the same header as a patch (width/height/x and Y offsets), but the image data itself is raw. Drawbacks: transparent index is always 255, no room for flags or identifier string, no reserved stuff for future expansion. Advantage: even smaller, support already exist in certain utilities, reuses pre-existing Doom structures.


A PNG's palette doesn't need to contain 256 colors, so if you're only using, say, 8 colors total, then the palette data will only take 24 bytes.

And I still don't know what indices of the actual palette they correspond to, or that they even really exist in the actual palette. So I have to do nearest color matching just as if the image had been 24- or 32-bit.

That'd be fine if the palettes of all 4 target games didn't contain duplicate colors, but even just the DOOM palette itself is absolutely rife with them. This would require one of two things to be done:

• Translations would have to manually remap all duplicate colors. This places an unnecessary burden on the end user.
  
• Translation building logic in the game engine would have to automatically remap all duplicate colors. This then *prevents* their application as actual separate colors intentionally. This places an unnecessary restriction on the end user.
  

BTW you didn't link me to a header for the original id raw format and I'm not familiar with it so I can't really comment either way. Strife also contains unused code for a linear format which has only width, height, topoffset, and leftoffset in its header. Why they added this and then didn't use it at all, I have no idea.

[Quasar]

As a perfect example of the trouble you can get into because of this duplicate palette index crap, if I convert EE's linear 256x32 CONCHARS lump to PNG format using Paint Shop Pro, it destroys the drop shadow, which ptoing painted using index 247 (index 0 is currently always transparent).

Why PSP insists on mauling images whose palettes have duplicate colors, I have no idea, but there's no way to stop it.

[Blzut3]

Maybe I missed something in the thread, but what is the point exactly? What do you benefit from having a raw image format?

Even if we were to agree to a new format no one would bother to use it in mods, so it would be code just as used as the IMGZ format. The code required to load a "bloated" image format takes negligible time, so the only benefit a raw format like this would have is if you were trying to port EE to some system which stream data directly off a cartridge or something. Even then I don't you would want the transparent index changing, and you would probably want to keep it 0 in order to shave CPU cycles. (Speaking of which the Jaguar graphic format is a raw image (2 - 32 bit presumably) with a 16 byte header. 8 of which are unused and could be allocated to offsets.)

[Graf Zahl]

.. and again, even if it's deemed to be necessary - why something new? Why not using something that already exists?

I had another look at ZDoom's IMGZ and the only thing currently undefined is transparent color - but since to this day it hasn't been used for anything except alpha maps it should be expandable, since the header still has multiple unused bytes.


But in the end I'd still have to question the motivation. This won't help getting rid of the Doom patch format so in the end nothing is gained. For that we already have all the tools and there's also some code floating around that performs some integrity validation so that corrupt patches can be detected.

Even in ZDoom the only reason IMGZ was added had nothing to do with making a better format but with evading some obnoxious 'features' of WinTex, as a comment suggests:

//==========================================================================
//
// An IMGZ image (mostly just crosshairs)
// [RH] Just a format I invented to avoid WinTex's palette remapping
// when I wanted to insert some alpha maps.
//
//==========================================================================

[DaniJ]

Even then its redundant. Simply save as greyscale and swap the channels as you read the image if necessary. Heck, you can even use greyscale format images for paletted graphics if you want to by interpreting the luminosity values as palette indices.

[Graf Zahl]

I guess the problem was with WinTex corrupting stuff due to making bad assumptions. It's well known to do such shit.

This was long before the addition of PNG support so today it's indeed just a redundant leftover from bad old times.

[fraggle]

I'm also not convinced at all that this is necessary. Disk is cheap nowadays, so is bandwidth. Just use PNG. Does anyone care about a few extra bytes used for storing the palette?

The only possibly-legitimate reason to define a new format might be so that you can include the offsets for the image, and it looks like you could probably even store that information in PNG files if you care enough.

[Blzut3]

fraggle said:

The only possibly-legitimate reason to define a new format might be so that you can include the offsets for the image, and it looks like you could probably even store that information in PNG files if you care enough.

Which Doom editing tools already support.

[Xaser]

If I'm reading the specs in the OP right, this is talking about a paletted graphics format, right? If so, I see literally no benefit to this at all as an end-user (modder) since SLADE3 can convert to and from Doom Patch format at will. Export as PNG -> Edit in any image editor -> Reimport as PNG -> Convert to Doom Palette. Done. :P

I haven't mucked around with RAW images in PSP7 in a while, but IIRC it requires you to input the dimensions as well, since that can't be inferred given the (lack of) format. Utilizing this newly-proposed format with my preferred choice of editor would thus be much more trouble than it's worth, since I'd have to manually enter dimensions for every image I load into it. It's faster to convert to/from PNG as outlined above because you can do it en-masse with existing tools.

Hope I'm not sounding offensive (promise I'm not, Quas :P ), but I almost feel like I'm missing something obvious/important here since the whole proposal seems like an inadequate solution for a nonexistant problem.

[Enjay]

I afraid that it also strikes me as something unnecessary and unlikely to be used. It would be a faff-around to use as a modder and would give no real advantage. So it would be pain for no gain.

There may be advantages for port/tool authors. I'm not qualified to evaluate that. What I would say, however, is that all the currently supported formats would need to still be supported so, really, where is the benefit? Add to that the fact that I suspect very few modders would adopt the format, this side of things also seems like pain for no gain.

I feel that another Doom engine format would just serve to confuse things. There is already confusion out there about the current patch format. How many modders think that the Doom format is BMP simply because that's what the input format of the lump manager they use takes? They make assumptions and sometimes errors resulting from this mistake and eventually they have to find out about the Doom graphics format. Now another format for WADs-only (etc) is proposed? I really can't see that helping.

As for PNGs being bloaty, so many mods are released these days with huge chunks of sound or music data that a few extra bytes from having paletted PNGs seems trivial. I certainly haven't felt myself worrying about the size of files (either to download or taking up space on my drives) because of their PNG content and I haven't had any problams running PNG heavy mods either. One of the advantages of using PNGs is that they are paletted - so if the WAD is run in a port that supports it, the patch will use its own palette but use the Doom one in a port that doesn't support individually paletted patches (assuming that other ports do what ZDoom does and fits the colours in the graphic to the best matches in the game palette).

Also, I note the comment that no external utilities can read [PNG] patches. That's simply not the case. SLADE has excellent PNG support. I believe that DB2 also does. Even DeePsea supports them to a certain extent (though I tend to use Doom format ones during editing and then put the PNG versions into the WAD when I'm done). There are also a collection of little utils (eg some of BagHeadSpidey's Doom utils) that support PNGs and, of course, any graphics editor worth its salt also supports them. In other words, there are utils that support PNGs as patches (and other graphics). The tools concerned are pretty much the utils being actively worked on and maintained (and used by the majority of the community I suspect) and so, if any utils were to support the new format, it would only be the same utils as currently support PNGs.


Short version - To me it seems like an unneeded addition to address problems that aren't particularly significant to most people actively using Doom and I suspect it would serve mainly to muddy the water and add to the confusion that may already exist rather than actually solve any problems in practical terms. This would be compounded by a likely low uptake of the new format by end users. Sorry.

[Quasar]

Enjay said:

Also, I note the comment that no external utilities can read [PNG] patches.

PNGs != patches. But the rest of your comment is fairly relevant. I've abandoned the idea and will just suffer the suboptimal loading and conversion overhead of .PNGs in the areas where this format would otherwise be useful.

[Graf Zahl]

Quasar said:

...where this format would otherwise be useful.

Where would that be? Is there any place where it'd provide a genuine advantage over either PNG or standard patches, outside of integrity validation?

In any well written implementation there should only be one place where any format validation is done: when the texture is initialized for the first time. After that you should only work with internal data where any 'suboptimalities' shouldn't come into play anymore. The increased processing time for PNG sure shouldn't be an issue anymore these days.

Sorry, but I can't see a single usage scenario where this new format would offer something of real value.