UDMF v1.0 RFC

[Quasar]

Eternity's implementation of line ids has not yet seen an official release or any documentation and therefore IF there is actually a problem, I still have a chance to fix it.

So with that said, all this talk about compromise not being possible is hooey. Unless it is you, Graf, who is unwilling to do any compromising. Let's keep everything on track and not get derailed over something like this that should really be simple.

You didn't reply to my previous post at all, where I offered the different options. I need your opinion on those options and reasons they won't work (if they won't), as well as an explanation of ZDoom's behavior in this respect. This is the only way I can satisfactorily understand what it is that I need to do with the spec.

Eternity's mess with ids/tags is my worry alone and shouldn't leak into this equation.

Hexen's args[0] is functionally identical to Doom's tag. We have agreement on that at least, because it's not something that is up to argument - it is a simple fact.

Hexen also provides the line id, something Doom did not have.

So. In Hexen maps, leave everything alone. Nobody has at any point spoken of changing the way that Hexen behaves. The only thing UDMF offers to Hexen in this respect is the ability to have line id's start out at positive integers rather than at -1 (EE's ExtraData offers the same ability).

In maps using Doom specials, all we need is to put the tag *somewhere*. I don't so much care about where it goes as I do about being able to find it.

In Eternity we can have Doom and Hexen-like specials in the same map. This is a simple consequence of having assigned the Hexen-like specials to non-conflicting numbers. But I don't see how this really actually becomes a problem for the tag/id issue. If I must, I will implement a gigantic table of line numbers that tells me how to handle the id or tag for each when loading UDMF maps in Eternity namespace.

This is a simple problem really and it is being inflated into something it is not. Let's get 'er done.

[CodeImp]

Option A, except just call the field 'tag', because it has the same purpose as a sector's tag, except it's for lines and things.

This 'tag' field defaults to -1 in Hexen namespace and defaults to 0 in Doom namespace. This is more something for me to remember while writing the editor configurations. You could also decide to default it to 0 in hexen namespace (meaning that 0 cannot be used as a tag), but I don't care really.

This is what I do care about :
I am highly against 2 different fields for the same purpose, or reusing the arg[0] for this purpose (which changes its meaning: now it becomes an identification of the element instead of something that refers to a tagged element).

[Graf Zahl]

Quasar said:

Eternity's implementation of line ids has not yet seen an official release or any documentation and therefore IF there is actually a problem, I still have a chance to fix it.

Good to know. I didn't know that.

So with that said, all this talk about compromise not being possible is hooey. Unless it is you, Graf, who is unwilling to do any compromising. Let's keep everything on track and not get derailed over something like this that should really be simple.

I'm more concerned with how any compromise will affect the spec. Under no circumstances should it confuse anyone wanting to implement the spec.

You didn't reply to my previous post at all, where I offered the different options. I need your opinion on those options and reasons they won't work (if they won't), as well as an explanation of ZDoom's behavior in this respect. This is the only way I can satisfactorily understand what it is that I need to do with the spec.

None of the options you mention make any difference to me. Internally ZDoom has the line ID and the 5 args. Any Doom format linedef is translated by storing the tag in the ID for Boom compatibility and also in the arg representing the tag for the line special.

So obviously, since both values are completely separate internally my preferred solution would be to keep the values separate and when converting a Doom format map the tag value for a Doom format line special should be stored in both the ID and arg0 field.

If that can't be done for whatever reason dictated by some other port I only need a clear definition of what to do and then use the proper value appropriately.

Eternity's mess with ids/tags is my worry alone and shouldn't leak into this equation.

As it should be. ;)

Hexen's args[0] is functionally identical to Doom's tag. We have agreement on that at least, because it's not something that is up to argument - it is a simple fact.

Hexen also provides the line id, something Doom did not have.

So. In Hexen maps, leave everything alone. Nobody has at any point spoken of changing the way that Hexen behaves. The only thing UDMF offers to Hexen in this respect is the ability to have line id's start out at positive integers rather than at -1 (EE's ExtraData offers the same ability).

In maps using Doom specials, all we need is to put the tag *somewhere*. I don't so much care about where it goes as I do about being able to find it.

At least we agree on this one. But see my remark above.

In Eternity we can have Doom and Hexen-like specials in the same map. This is a simple consequence of having assigned the Hexen-like specials to non-conflicting numbers. But I don't see how this really actually becomes a problem for the tag/id issue. If I must, I will implement a gigantic table of line numbers that tells me how to handle the id or tag for each when loading UDMF maps in Eternity namespace.

The only problem I see is that the Boom specials referencing lines by tag would clash with having the line ID in the base spec for all namespaces, even those where this might lead to potential data collision. The following scenario is something ZDoom can't handle:

- The spec allows both ID and tag to be specified with some specials/scripts etc. referencing lines by tag and others by ID.

So how about the following:

The line ID is not part of all namespaces but instead an additional key only applicable to the Hexen namespace and port specific ones that can use it. Then the tag could be stored in arg0, you could define the ID according to your own needs in Eternity and ZDoom would just inherit the semantics from Hexen. There'd be no potential clashes of values anymore and all is well because compliant parsers would have to ignore it in all namespaces where it might cause problems.

[CodeImp]

Ok, I think I misread the post where you put in the options and I notice the examples with the options are in Doom namespace. So how about this?

"arg0" (or any other arg, depending on special in Hexen namespace) REFERS TO another element. While "tag" IDENTIFIES an element with a tag number.

HEXEN

namepsace = "Hexen"

linedef
{
    special = 123;
    arg0 = 7;        // Refers to sector, line or thing (depending on special) with tag 7
    arg1 = 152;
    arg2 = 4;        // Refers to sector, line or thing (depending on special) with tag 4
    
    tag = 5;         // Identifies this line with tag 5
                     // Leaving this out, defaults the tag to -1 (no tag at all)
}

DOOM

namepsace = "Doom"

linedef
{
    special = 34;
    arg0 = 7;        // Refers to sector with tag 7
    
    // "tag" field MAY be left out here and default to 0,
    // because Doom has no specials that refer to linedefs anyway.
    // However, including the "tag" field here, although useless,
    // should not cause any problems.
}

[Graf Zahl]

CodeImp said:

Ok, I think I misread the post where you put in the options and I notice the examples with the options are in Doom namespace. So how about this?

HEXEN

    // "tag" field MAY be left out here and default to 0,
    // because Doom has no specials that refer to linedefs anyway.
    // However, including the "tag" field here, although useless,
    // should not cause any problems.

Unfortunately this is not correct. Boom defines several line specials that reference lines by tag (e.g. scrollers or silent line teleporters.)

And these are the ones causing the problems. As long as they reference the line ID there'd be no issue. However, from Quasar's previous posts I gathered that they still reference the tag, despite the presence of a line ID that'd be better suited.

And that *is* a problem because if the tag was used like this and a line ID defined on top of it it would not work with ZDoom anymore and due to the translation to Hexen format specials there'd be no decent way to resolve this.

If Eternity wants to keep it like this the only option would be to prohibit the use of line IDs at all in common namespaces having such specials (meaning 'Doom'.) or using ID as tag and define proper semantics.

[CodeImp]

Well then, we give Doom linedefs their tag back? Like this?

HEXEN (same as above example)

namepsace = "Hexen"

linedef
{
    special = 123;
    arg0 = 7;        // Refers to sector, line or thing (depending on special) with tag 7
    arg1 = 152;
    arg2 = 4;        // Refers to sector, line or thing (depending on special) with tag 4
    
    tag = 5;         // Identifies this line with tag 5
                     // Leaving this out, defaults the tag to -1 (no tag at all)
}

DOOM (updated)

namepsace = "Doom"

linedef
{
    special = 34;
    arg0 = 7;        // Refers to sector with tag 7
    
    tag = 5;         // Identifies this line with tag 5
                     // Boom specials can refer to linedefs by this.
                     // Leaving this out, defaults the tag to 0
}

Please don't tell me it is impossible for Boom specials to read the reference from arg0 and lookup the reference by the tags. And if so, that could never be a lot of work to make it so.

[Graf Zahl]

CodeImp said:

Please don't tell me it is impossible for Boom specials to read the reference from arg0 and lookup the reference by the tags. And if so, that could never be a lot of work to make it so.

If the 'tag' is the same as the line ID that's exactly how ZDoom works.

All the confusion comes from the fact that I understand Eternity's implementation so that these specials use the tag to reference lines but everything else uses the ID - which would be a different value.

As long as the Boom specials actually reference the lines by their line ID (which you prefer to call 'tag') - there'd be no problem at all.

[CodeImp]

"tag" in my example is what was "tag" originally in Doom, except it is no longer used to REFER TO ANOTHER element, that is what the args are for.

Either call every field that identifies the element "id", or call them all "tag" (which I prefer, it's better Doom terminology IMO), but don't call it "tag" when it's on a sector and "id" when it's on a line, that just confuses everyone more.

[Quasar]

Let's lose the term "tag" for UDMF entirely since it is causing confusion. Hexen refers to Line and Thing #'s as ids and that is a better term than tag IMO.

Graf said:

So obviously, since both values are completely separate internally my preferred solution would be to keep the values separate and when converting a Doom format map the tag value for a Doom format line special should be stored in both the ID and arg0 field.

I can't see any problem with this. It seems like a perfect solution to me. Doom lines end up with the "tag" in arg[0] and their id being equal, so regardless of where you get the info from, specials should continue working.

I would however add the stricture that this is only done if the line's tag is non-zero, to avoid the problem of having "id = 0" appearing for every DOOM line. BOOM's line-referencing specials such as the translucency transfer are coded to not allow zero tag activation anyway, AFAIK.

Now if someone *were* to tamper with the map and set the arg0 differently from the id I'd expect it to break. So what would the consequences be if the standard simply forbids this for Doom-style specials? It might limit what you can do with such lines in Eternity's namespace, but I can live with that when all the Hexen-style features are now easier to use without the restrictions of the Hexen map format.

I'd rather see the Doom style specials begin to fade into obscurity in favor of the power that parameterized lines offer. UDMF only supports them at all in order to allow reversible, lossless conversion of all vanilla maps, and for ports that don't happen to support Hexen in the first place.

[CodeImp]

Quasar said:

Let's lose the term "tag" for UDMF entirely since it is causing confusion. Hexen refers to Line and Thing #'s as ids and that is a better term than tag IMO.

Oh damn you! But at least renamed them ALL to "id" then, no "tag" in sectors either!

Why does the id and args have to be the same in Doom namespace? Because Doom originally only had one tag? I see no reason why not to improve this. It makes the system more logical and gives a little bit more flexibility for map authors. Look at my example, it is perfectly possible for a line having an id 5 and referring to line with id 7.

namepsace = "Doom"

linedef
{
    special = 14424; // Some Boom special
    arg0 = 7;        // Special refers to sector or line (depending on special) with tag 7
    
    id = 5;          // Identifies THIS line with tag 5
}

[Quasar]

Well, let's be careful not to muddle up issues.

For Hexen, Eternity, ZDoom, and any other port that supports it, the tag and id CAN be different, and in fact must be allowed to be different. For example a line may apply a Door_Close special to tag 7 when you walk over it, but also have id #52 for purposes of having its blocking property changed.

For old DOOM maps, there was no id. Doom namespace maps should not specify ids different from the line tag, or they are not really functioning as Doom namespace maps.

I am willing to also sacrifice the ability to have differing tag/id on Doom-like specials *in EE's namespace* in order to keep this consistent, if that is necessary.

BTW earlier when I said that zero tags shouldn't be made into 0 ids, what I MEANT to say was that lines with *zero specials* and zero tags shouldn't have their tags made into 0 ids. If there's a special on the line, even a 0 tag might be important. Otherwise, it shouldn't matter in any case other than a special trying to erroneously apply to the entire map, which I don't think BOOM allows anyway for stuff like the translucency special.

[CodeImp]

Quasar said:

For old DOOM maps, there was no id. Doom namespace maps should not specify ids different from the line tag, or they are not really functioning as Doom namespace maps.

We don't have to add that same limitation. As long as this format does not break existing maps converted to UDMF we're good. Adding this ability to have a seperate identification number over the referencing number does not break old maps, it only makes it more flexible for new maps. We don't have to add an old ass limitation, because old ass doom sourceports (including vanilla doom/boom) are not going to read this format anyway.

[Graf Zahl]

The result of this discussion hinges on one single question, which only Quasar can answer and of which I am still unsure because it isn't really clear yet:

Imagine a line using Boom's silent teleporter, translucent line or any other line affecting special and independently setting the ID from the tag (as in special argument) is allowed.

How are lines to be marked that this linedef should affect?
By setting arg0 or ID?

If the answer is ID, there is no problem because that's how ZDoom works and whatever solution we decide upon will work.

If the answer, however, is arg0, we need some restriction because the arg is just that for ZDoom and not preserved as a reliably accessible value. Depending on the line special it can end up in any of the args or if there is no special using it even be dumped completely so obviously it can't be used to reference a line. In this case, at least in the 'Doom', 'Heretic and 'Strife' namespaces we need an explicit restriction that restricts this so that there can only be one value that is used for both. But this restriction must be part of the spec in no uncertain terms to avoid problems. Otherwise we get some mess of maybe's and canbe's that will be impossible to handle and at work I've got enough experience with this. J2ME (Java for cell phones) is full of this shit and is a nightmare to develop for.


This particular issue is the only thing that's preventing us from finding a solution that is compatible with both ZDoom and Eternity.

[CodeImp]

I do not think specials and args should set the identification of the element they are on, not in any way (hexen's SetLineIdentification was a retarded solution). Specials and their args should only REFER TO an element. The id/tag field of the element is what identification is for.

[Quasar]

Using the silent teleporter example:

In a BOOM map:

  x------------>x tag: 10  special: Teleport Line-Line 


  x<------------x tag: 10  special: Teleport Line-Line

If I understand your post on the ZDoom forum correctly, ZDoom turns these into Teleport_Line, giving both lines the id #10.

Then let's just say that the tag always becomes the id, if that's all that needs to be done. Seems simple enough. Eternity can adjust as needed to make this happen.

So the UDMF looks like this:

namespace = "Doom";
linedef { special = 243; id = 10; }
linedef { special = 243; id = 10; }

When loading the map, the implementing source port can do what it wants with id. If the implementor has no Hexen support, then id just becomes a tag again, and nothing is altered. If the implementor translates all maps to a virtual Hexen format, then it treats the id the same as it would treat a tag in a binary Doom map (if you need to copy it to args[0] conditionally, you do that now the same way you always would). If the port is messed up like EE, then you do I don't know what yet ;)

I seem to recall however that CodeImp previously objected to the idea of a field's default value varying on the namespace involved. If that's still the case, then we are stuck with Doom/Heretic/Strife maps being required to specify "id = 0" on almost all lines. Doesn't seem like justifiable bloat to me; I'd rather the default be dependent on the namespace as was previously suggested.

[CodeImp]

That is no longer the case, I can live with different defaults.

But why oh why? do you not want to understand my point about the difference between REFERRING TO and IDENTIFICATION? Args (which are used by the special) should be for REFERRING TO something, and the "id" field should ONLY IDENTIFY the element.

Yes, a converted map would duplicate the tag to two fields. Like this:

namespace = "Doom";
linedef { special = 243; arg0 = 10; id = 10; }
linedef { special = 243; arg0 = 10; id = 10; }

BUT, with the UDMF format it would now also be possible, though not required, to refer to elements that have a different id than the element using the special. Like this:

namespace = "Doom";
linedef { special = 243; arg0 = 6; id = 10; }
linedef { special = 243; arg0 = 5; id = 6; }
linedef { special = 243; arg0 = 10; id = 5; }

Yay, 3 way teleports :P And it doesn't break compatibility with old maps, because their conversion just copies the same value in both fields, like in the first example.

[Graf Zahl]

Quasar said:

I seem to recall however that CodeImp previously objected to the idea of a field's default value varying on the namespace involved. If that's still the case, then we are stuck with Doom/Heretic/Strife maps being required to specify "id = 0" on almost all lines. Doesn't seem like justifiable bloat to me; I'd rather the default be dependent on the namespace as was previously suggested.

I really don't care anymore how this is resolved, as long as it is resolved. Just define something you can live with.

Personally I have no issue with different defaults for different namespaces. Handling this in the parser would only require minimal overhead.

But if this is an issue, always writing the ID may be bloat but at least it's safe.

But I fully agree with CodeImp. If you changed this so that Eternity uses the ID properly for these specials a lot of these technicalities would just go away. It's not that any project would adversely suffer from it, considering that both of Eternity's showcase projects still seem to be an eternity (:P) away from release.

[Quasar]

CodeImp said:

That is no longer the case, I can live with different defaults.

But why oh why? do you not want to understand my point about the difference between REFERRING TO and IDENTIFICATION? Args (which are used by the special) should be for REFERRING TO something, and the "id" field should ONLY IDENTIFY the element.

Yes, a converted map would duplicate the tag to two fields. Like this:

namespace = "Doom";
linedef { special = 243; arg0 = 10; id = 10; }
linedef { special = 243; arg0 = 10; id = 10; }

BUT, with the UDMF format it would now also be possible, though not required, to refer to elements that have a different id than the element using the special. Like this:

namespace = "Doom";
linedef { special = 243; arg0 = 6; id = 10; }
linedef { special = 243; arg0 = 5; id = 6; }
linedef { special = 243; arg0 = 10; id = 5; }

Yay, 3 way teleports :P And it doesn't break compatibility with old maps, because their conversion just copies the same value in both fields, like in the first example.

I have no problem with this at all. If it works for Graf, it works for me. If it does not work for Graf, then I'm going to use the id-only idea so that we can move on with this :) It's so close to being finalized except for this one point ;) But it's better to take time NOW to make it work than to try to fix it later after maps have already started using it and we find some kind of problem. In that respect, I'm not regretful at all that's taken time and discussion to decide how to handle this. After almost 15 years with the binary map format, a week or two of deliberation on UDMF is like a drop in the bucket :)

[Graf Zahl]

CodeImp's examples are an exact representation of ZDoom's implementation so it'd be the perfect solution for me and if it works for you I'd say go for it.

Coincidentally it's also the option which offers the most mapping flexibility. :)

[andrewj]

What CodeImp is saying about the difference between referring to other objects (in arg#) and identifying each object (in ID) makes perfect sense to me -- the penny finally dropped :-)

I don't like the term "id" because the word "identification" often means it is for one particular unique object, like a social security ID is for one person only, whereas in a map format multiple objects may share the same number.

That's why I think "tag" is a better term, the meaning of applying to more than one object has already been established by DOOM mapping.

[CodeImp]

Yea same here, but if Quasar really wants to call it 'id' that's fine by me, there are more important things to argue about than this :P But at least be consistent and call the sector tags 'id' as well.

[Graf Zahl]

Ajapted said:

What CodeImp is saying about the difference between referring to other objects (in arg#) and identifying each object (in ID) makes perfect sense to me -- the penny finally dropped :-)

I don't like the term "id" because the word "identification" often means it is for one particular unique object, like a social security ID is for one person only, whereas in a map format multiple objects may share the same number.

That's why I think "tag" is a better term, the meaning of applying to more than one object has already been established by DOOM mapping.

Who cares...? ;)

Now let's wait for the next revision of the spec which hopefully is final. Then I can start adding ZDoom specific features.

[CodeImp]

Do not rush, there is no editor for it until DB2 comes out, and although I work hard on it, that won't be any time soon.

Slayer is working on a new editor also, I think he also agreed to implement UDMF, but that editor also won't be out any time soon.

[andrewj]

Graf Zahl said:

Who cares...? ;)

It's a minor point but helps make the spec easier to understand. If you really don't care then shut up already :-)

[RTC_Marine]

If anyone is interested, I created a doom wiki article for the specification: http://doom.wikia.com/wiki/UDMF

[DaniJ]

FWIW, can I suggest a slight change? It seems to me that it would be logical to do something like this with regard to the line special:

linedef { id = 10; special { num = 243; arg0 = 1; } }

[Quasar]

It would be, except the grammar does not currently allow for sub-blocks. It is also mostly a shallow distinction, that is, it adds more syntax but does not change the semantics at all. I'd tend to prefer less syntax where it is possible.

With that said, here is the final candidate spec:
http://doomworld.com/eternity/engine/stuff/udmf10.txt

Changes:
* Fixed error in grammar not allowing empty blocks - an assignment_expr may be "nil". However, no currently defined blocks can be legally empty due to containing required fields that cannot default.
* Added syntactic comments (// and /* */) with caveat that editor utilities are not required (but are allowed) to preserve such comments when rewriting the data.
* Changed names of known Strife flags and eliminated ones that are verified to be completely unused.
* Changed sector tag to id and thing tid to id.
* Specified spac/tag semantics for Doom/Heretic/Strife namespaces.
* Specified unit of degrees for mapthing angle.
* Changed mapthing height to float type.
* Cleaned up defaults for flags by specifying them all at once.
* Placed spec under GNU FDL. This will allow it to be copied verbatim to most wikis.

Once again, unless this generates complaints or more suggestions that are agreed to, it will become the final 1.0 spec.

[Graf Zahl]

@DaniJ:
It would make the parser more complex without adding anything of real use. Right now it's possible to parse this with a primitive parser that only can handle minimalistic token processing and doesn't need to handle any kind of nesting in both the parser and data organization.

This suggestion would only be of genuine unefulness if more than one special was allowed per line - but that would require adding semantics for such a feature to the spec - which simply is beyond its scope.

Keep in mind that this isn't really supposed to be human-readable. It's only text due to the extensibility.

@Quasar:

This looks mostly good. There's 2 minor issues left:

1. The line transparency and thing translucency flags don't describe fully what they do in Strife. I don't know if this is necessary in the spec but the line translucency is 75% and the thing translucency is 25% in Strife.
2. Regarding the ID/Arg0 issue, the spec should define clear semantics how they are used.
The remark at the end is unclear whether it should only apply to map converters or as a general rule. The current spec suggests that the same value has to be stored in both fields but doesn't say how they are to be used properly. I think it should be explicitly stated what the ID value is used for and what the Arg0 value is used for because if this is left to interpretation and such vague definitions historically have caused incompatible implementations as I have experienced countless times during my work on J2ME.

[CodeImp]

Yea it should definitely state this. The result of an old map conversion to UDMF would put the same number in both fields, but a UDMF implementation uses the special&args ONLY FOR REFERRING TO ids and a UDMF implementation uses ids ONLY TO IDENTIFY the elements.

[Quasar]

Spec is updated, same URL.

Changes:
* Fixed mistakes: omitted change of class0/1/2 to class1/2/3, changed Strife "transparent" line flag to "translucent", and fixed an instance where I typed "Hexen" but meant "Strife".

* Added blurb about interpretation of id/arg0, where such a distinction applies.

* Added a copyright date.