New (cross-port?) scripting languages for Doom

[Graf Zahl]

 

What you don't seem to understand that it doesn't matter shit what you think what's the best way to do this or what I think about it or what any other port developer thinks. As long as there's no consensus it's all noble ideals that will lead nowhere.

    
What DOES matter is what you can sell to the people that matter, that'd be me (as GZDoom developer), Coraline (as 3DGE's lead developer) or Team Eternity. If you do not get the most popular on board - well - read my last post.I can almost guarantee that you'll end up as a single port out there trying to compete with the rest of the world (i.e. other Doom ports) and failing to find an audience, if you go that route.

 

I am proposing ACS because this is a lowest common denominator that I feel has a chance to get accepted. UDMF wasn't created by going all out for the most encompassing and perfect mapping solution but by finding a compromise everybody could live with. I see no compromising here, just "I want it all".

 

[esselfortium]

15 minutes ago, Ladna said:

But I've said pretty much everything I want to say, so I'll try and avoid that here and bow out.  Please don't make ACS the standard Doom scripting language.

I think you're many years too late to sway anyone with this sentiment, regardless of merit.

 

(Full disclosure: I am personally 100% for wider adoption of ACS.)

[Jon]

why are we still banging on about cross port scripting standards when it has been ruled out of the "boom+" effort? It would be much more interesting to discuss potential future tech without getting mired in port adoption discussions.

[Arctangent]

I don't think it was even ruled out of the Boom+ effort, just ruled out as part of the first stage of it.

 

Also, it's kinda moot to discuss stuff without adopting port adoption, because ... I mean, if there's no port that adopts it, there's literally no way you can make use of it in the first place.

Edited July 4, 2017 by Arctangent

[Quasar]

If you want to aim for something experimental then it's better to just "do" than to discuss. If you want to see, for example, a Lua-powered Doom port in action, then just grab Choco, Crispy, or PrBoom-Plus and start hacking. It shouldn't even take a whole day to have the basic embedding functional.

[Graf Zahl]

9 hours ago, Jon said:

It would be much more interesting to discuss potential future tech without getting mired in port adoption discussions.

 

Would it?

We've seen how that will end. People get carried away with grand ideas that never see the light of day.

 

The problem both here and in the 'Boom+' thread is that crazy stuff gets tossed around, but there's nobody who actually makes a concerted effort to design a spec from the collection of ideas. That's just not going to work out and at the end we'd still be where we are now: We got several different ports, each doing its own thing with Boom as the common denominator.

 

[Jon]

32 minutes ago, Graf Zahl said:

 

Would it?

We've seen how that will end. People get carried away with grand ideas that never see the light of day.

 

The problem both here and in the 'Boom+' thread is that crazy stuff gets tossed around, but there's nobody who actually makes a concerted effort to design a spec from the collection of ideas. That's just not going to work out and at the end we'd still be where we are now: We got several different ports, each doing its own thing with Boom as the common denominator.

 

 

This thread is pretty young. It's too early to write it off as going nowhere. Yes similar such thread often do get a lot of ambitious suggestions from non programmers that are pipe dreams, but that hasn't happened here yet, most of the participants (yourself, myself, ladna) are programmers and have some idea as to what is possible.

 

I think there's still hope for this thread and the boom+ one which is slowly coalescing. I plan to add some concrete info (linedef numbers, links to example implementations) to the wiki spec page sometime this week (assuming kb1 doesn't take it on first).

[Master O]

13 hours ago, Jon said:

 

This thread is pretty young. It's too early to write it off as going nowhere. Yes similar such thread often do get a lot of ambitious suggestions from non programmers that are pipe dreams....

 

Can your new sourceport make waffles? [/joking]

[LexiMax]

On 7/1/2017 at 1:28 PM, Graf Zahl said:

I generally concur, it's just too bad that ACC is a) a really shitty piece of code (and compilation depends on its shitty design) and b) comes with a useless license.

What's wrong with alternative ACS compilers like https://github.com/DavidPH/GDCC and https://github.com/wormt/bcc ?

[Linguica]

Since they compile down to the same bytecode, they all boil down to the same problems, right?

[kb1]

1 hour ago, Linguica said:

Since they compile down to the same bytecode, they all boil down to the same problems, right?

This is the HUGE point that many don't seem to grasp. To have any chance of multi-port scripting support, there will need to be pre-built powerful Doom-like functions, that work exactly the same in every port. This suggests a certain number of functions, with a certain number of arguments. This suggests a byte-code VM. And, this suggests the only byte-code VM in mass use in Doom ports today. The only idTech1 standard: ACS. The source language doesn't matter - it gets compiled to byte-code.

 

Now, original ACS was limited. It contained only what Raven needed to make Hexen. But, the nice thing about VMs is that they can be expanded. ZDoom proves that. (I wish Randy hadn't added that encryption mess to it, along with the different formats - yuck). But, this thing is already built, documented, and proven to work. A prototype (without the encryption/multi-format stuff), with current extensions could be mocked up fairly easily. But, ports will need to come up to speed, before any scripting happens. Again, that's what a part of the Boom+ spec is moving towards. It will, first, create a more-compatible environment. Included is the managing of linedef/sector types, which is a big hurdle to Heretic, Hexen, and Strife support (even if your port does not plan to implement these games). This opens the door to Hexen-style map support, which, in turn provides the additional variables required to get the VM functions to be able to work across those ports.

 

Once that happens, ACS scripting can be dropped in. It is premature to be deciding on cross-port scripting support. This should be obvious: Ports must be compatible enough, before you bolt on a scripting language. You must be able to do common things in an exact way, across ports, within the port's native languages (C/C++), before you can expect a bolt-on language to be able to those things.

 

So, regardless of the exposed "script language" syntax, internally, these should all be able to compile to a common byte-code, to have ANY chance at cross-port support that actually provides identical functionality across ports. Otherwise, you end up with some mickey-mouse half-compatible mess that's hardly worth the effort.

 

With a VM back-end like ACS, the code in the front-end could be C/Java-like, BASIC-like, whatever. It all compiles to common byte-code. It's a win-win.

 

Now, I honestly don't know how good ZDoom's flavor of ACS is. At one time, I remember that strings were a bit wonky with ACS, but I think ZDoom has resolved a lot of that. I do know that it is rich function-wise. But a new compiler/front-end could resolve any such remaining issues, cause the compiler can do the dirty work involved in dereferencing strings, or whatever else is currently weak (if anything). A new compiler can accept any syntax you wish. The source may look different to ZDoom ACS veterans if a new source language is built. But, the VM would stay the same, and that's the beauty of the VM approach. And, it's the only way you're going to get cross-port compatibility, in my opinion.

 

One more observation: Language building is hard. Compiling is hard. Doing these properly is more difficult than the game itself. It requires many hours of deep concentration, a unified effort, research, trial-and-error, massive testing, etc, OUTSIDE of a game, before you ever try to hook it into the game.

 

[Albertoni]

46 minutes ago, kb1 said:

At one time, I remember that strings were a bit wonky with ACS, but I think ZDoom has resolved a lot of that.

They still are a bit wonky. No dynamic declaration, concatenation, etc., they're just pointers to a string table:

Take str x = "ABC";

Afterwards x+1 would return "BC", right? Nope, returns the next string you declared. "Hi, I'm" + "trying to concatenate" would return the string at the added index of those two.

 

It's not a huge problem, but it's annoying.

 

Edit: > StrParam will create a new string formatted based upon the same method for print or log. This string will only exist for 1 tic, so a delay will nullify it. As of revision r4295, strings generated by StrParam, GetCVarString and GetUserCVarString will last for as long as they need to, rather than for 1 tic.

 

Nevermind. Still unwieldy, but I know very well how supporting different types is terrible, I've tried to make a compiler before. Though I guess someone should update the data types page on the ZDoom wiki, heh.

Edited July 6, 2017 by Albertoni

[Arctangent]

I don't think they actually implemented a proper string type, pretty sure they basically just made the string table more dynamic. You're still just dealing with indices to it.

[LexiMax]

3 hours ago, Linguica said:

Since they compile down to the same bytecode, they all boil down to the same problems, right?

The specific problem I was responding to was the fact that ACC was hard to embed for automatic compiling because it was both terrible from a source code point of view and still under an unfriendly license.  Presumably, the other alternatives I outlined might be free of those problems.

 

BCC in particular is pretty interesting because it supports an additional language called BCS, which is like ACS but contains additional niceties, compiles down to the same bytecode, and is 99% backwards compatible with ACS (the only breaking change it makes is making boolean operators short-circuit).  Shamelessly stolen from the README:

strict namespace SampleCode {
   script "Main" open {
      static str basket[] = { "apples", "oranges", "pears" };
      foreach ( auto fruit; basket ) {
         Print( s: "I love ", s: fruit, s: ( fruit == "oranges" ) ?
            " very much" : "" );
      }
   }
}

 

Edited July 6, 2017 by AlexMax

[Agentbromsnor]

I don't really get the hate towards ACS in the modding community, though I'm not a code-wizard myself by any means. The way I see it, Hexen contains the most advanced version of idTech 1 (besides Strife, obviously). With that in mind, it only seems natural for me to use and expand on ACS. This way, the resulting source-ports should still feel and behave much like the original tech. Perhaps there's a lot more to it though.

[VGA]

I think people just don't want every port to become ZDoom.

 

Anyway, ACS doesn't make much sense if the map format is vanilla or boom, so from what I understand port authors must agree on a new map format where you can tag things etc. Like Hexen or UDMF.

[Jon]

8 hours ago, Linguica said:

Since they compile down to the same bytecode, they all boil down to the same problems, right?

That depends on where the problems are. For me the licensing issues on the raven compiler are a big deal, so the alternatives are great.

[Agentbromsnor]

8 hours ago, VGA said:

I think people just don't want every port to become ZDoom.

 

Anyway, ACS doesn't make much sense if the map format is vanilla or boom, so from what I understand port authors must agree on a new map format where you can tag things etc. Like Hexen or UDMF.

I get that, but from a scripting standpoint wouldn't it make more sense to go for the language that is already supported in a version of the engine?

[Gez]

There's not a lot of preexisting choices:

 

- enhanced ACS; ZDoom, Eternity

- FraggleScript: ZDoom, Legacy

- RTS: 3DGE

- PascalScript: DelphiDoom

 

I think that's about it.

[kb1]

19 hours ago, Albertoni said:

They still are a bit wonky. No dynamic declaration, concatenation, etc., they're just pointers to a string table:

Take str x = "ABC";

Afterwards x+1 would return "BC", right? Nope, returns the next string you declared. "Hi, I'm" + "trying to concatenate" would return the string at the added index of those two.

Yeah, that's fugly. But it's the compiler's job to interpret '+' to mean string concatenation. In your example, the compiler is writing byte code that treats string + string as addition of 2 indices. Actually, what it currently does is a valid thing to do, it's just not very helpful.

 

18 hours ago, Arctangent said:

I don't think they actually implemented a proper string type, pretty sure they basically just made the string table more dynamic. You're still just dealing with indices to it.

Nothing really wrong with that, especially in a script language. I would argue that the low-level details of complex types should be hidden from the scripter in most all cases, especially in a Doom script language. But, yeah, concatenation should work! There's a distinction between 'scripts' and 'programs', which is a blurry line. To me, scripts should 'just work', and be relatively safe from crashes and odd side-effects. Programs should provide all the low-level goodness, along with performance, and enough rope to hang yourself :)

 

@AlexMaxInteresting! And, a perfect example of multiple compilers/frontends, generating byte-code for the same underlying VM.

 

17 hours ago, Agentbromsnor said:

I don't really get the hate towards ACS in the modding community, though I'm not a code-wizard myself by any means. The way I see it, Hexen contains the most advanced version of idTech 1 (besides Strife, obviously). With that in mind, it only seems natural for me to use and expand on ACS. This way, the resulting source-ports should still feel and behave much like the original tech. Perhaps there's a lot more to it though.

It is a natural progression. A safe assumption is that most Doom-only ports eventually want to support the other idTech1 games, which includes Hexen, which requires some ACS support. Maybe the community will make something better than ACS, or BCS. But a lot can be learned from ACS, and ZDoom has done a lot to enhance ACS. It would be wise to, at least, become familiar with original and ZDoom ACS. Personally, I would not even consider a language that did not compile to byte-code or native (EXE/DLL), simply for performance reasons.

 

16 hours ago, VGA said:

I think people just don't want every port to become ZDoom.

 

Anyway, ACS doesn't make much sense if the map format is vanilla or boom, so from what I understand port authors must agree on a new map format where you can tag things etc. Like Hexen or UDMF.

I don't even know what it means to 'not become ZDoom'. It's healthy to have differing opinions of how to implement things, but such a blanket statement serves no positive purpose.

 

I suppose much of ACS does require Hexen. But much of it does not, as well.

 

Here's how I see it:

If we do NOT use ACS: A new Doom/Hexen port now needs to support 2 script languages: ACS and "ScriptX". And it will not be compatible with ZDoom maps using enhanced ACS at all, so, to support that goal, you'd need support for 3 languages, where 1 or 2 would suffice.

 

For me, it's going to take a good argument to offset all of the benefits of continuing with Raven's or ZDoom's ACS standard, and I want to make my own scripting language! Thank goodness that is far down the road...

[Albertoni]

19 hours ago, VGA said:

I think people just don't want every port to become ZDoom.

 

Anyway, ACS doesn't make much sense if the map format is vanilla or boom, so from what I understand port authors must agree on a new map format where you can tag things etc. Like Hexen or UDMF.

I don't see a problem with that, as long as such implementation in prboom-plus does one very clear thing: Keep demo compatibility, which is the one main quality of prboom-plus.

 

The idea is to allow people to write in a clear, readable manner what they want their voodoo doll scrollers to do. Without Hexen map format, ACS would be a simpler way to do so. Though I can't imagine what kind of ugly hack would be needed for passing arguments...

[Graf Zahl]

About strings, let's be honest here: Building a VM that can safely handle complex string operations is no easy task. Original ACS had no string handling at all - the only thing you could do is assign a string index to an integer variable.

 

In ZDoom, the parts of ACS that required complex string handling were those where ACS was mostly used as a crutch to stand in for actual engine scripting - for pure map scripting there is very little need to have this feature, provided you got a Print function that can do more than just outputting a string constant.

[Coraline]

I can imagine ACS will come next for 3DGE. We already have proper support for Hexen/ZDoom format maps - the thing that I'm thinking about is that if its going to be worth expanding much more of [ACS] past whatever Hexen needs, since RTS in 3DGE will always come first. Not sure if it will be worth the effort or what I'll have to do to support both it and RTS. .

[Graf Zahl]

The ACS VM is a relatively contained piece of code. You should be able to add it without any genuine impact on the rest of your code. Of course you shouldn't try to implement all of ZDoom's features at once (e.g. HUD messages are better left out until you know what to do with them.)

ZDoom's p_acs.cpp only got so large because it contains all the functions that ever got defined.

One piece of advice: Do not even think about using Hexen's original ACS VM, you won't get very far with just that because both ZDoom and Eternity implement an optimized bytecode format which requires quite different setup and many features cannot be implemented with that.

[Blastfrog]

To be frank, I don't see much use in debating whether or not to use ACS, the conclusion is obvious. If the point is to both be solid and widely adopted without issue, there is no other option.

 

There is absolutely nothing wrong with ACS, its purpose and scope is clear, it's excessively easy to work with and it does not have glaring issues or limitations like Fragglescript does for example.

 

Scripting does not need to be advanced in any way, it just needs to manipulate the levels and events during gameplay per the map author's intentions. I don't care about complex string manipulation or any kind of external filesystem read/write access when all I'm trying to do is print a message and raise a set of platforms in a certain order.

 

Anything outside of such a simple scope is better off as a new supplementary feature than to replace an existing and well-established one.

Edited July 7, 2017 by Blastfrog

[LexiMax]

Forgot to respond to this...

On 7/4/2017 at 11:38 AM, Jon said:

I am strongly considering picking up where AlexMax left off (with Lua on top of Crispy Doom) which is why I was looking for useful use-cases of interesting things that are difficult or impossible to do with the tools we already have today. They'd be the first things I'd work on.

Don't.  When you look at the code in earnest, you're likely to be pretty disappointed.  All I managed to do was integrate the interpreter into the build system, start it, and maybe (it's been so long I've forgotten) do some simple HUD modification with it.

 

^{If you are serious about contributing to that sort of project, though, I do have something on the back burner that I do intend to eventually return to that you might be interested in.  Said project does not fit within the scope of this thread at all, however.}

[JPL]

Lots of context to this discussion I probably lack, but I think any potential scripting system needs to be evaluated just as much in terms of its user interface as its capabilities and engine integration. ACS and ZScript are very powerful, as a veteran designer-programmer it's a super fun toolbox to play with and almost everything on my wishlist for those could (and probably will) be done pretty easily and incrementally.

 

However, when I try to imagine what would make the biggest difference re: People Doing New Cool Stuff With Doom, it's on the tools + UI side. Here are things I think form barriers to entry for creative dev-literate / dev-curious users:

• Engine and editor are wholly separate. Changes to maps/mods must be saved in-editor and then launched in-game.
• All assets - textures, sprites, sounds etc - require an engine reload to see changes.
• ACS changes require a recompile with an external tool.
• If you're doing heavy level scripting with ACS, you must flip back and forth between not just a text editor and a map editor, but between the text-based mindset of scripts and TIDs and the visual/spatial world of sectors, things, and lines.
• Editors offer basic visualization of connections between objects and map features (ie showing which sectors a line's special affects) but manipulating this data is still done by changing numbers in fields. This is a bit worst-of-both-worlds: none of the code-like benefits of text-based scripting and none of the visual/spatial benefits of in-world scripting. This might be easily solved with good editor viz, eg color-coding sectors by tag, but that may not scale to more complex map work.

I realize all of these would be quite a lot of work to remedy, and having worked with the full spectrum of code-based (Lua/ZScript), GUI-based (Bioshock/GameMaker), and purely visual (Unreal4 Blueprint) scripting I'm well aware of the challenges of each - don't take my points above as belief that visual scripting is inherently better - but I do think it's worth asking where the big gaps still are for Doom's dev experience.

It's also a question of values: where do people want Doom to be in 5-10 years? Many are probably happy with the current size, composition, and output of the community. I think growth could bring many positive effects. I think there are tons of people out there who'd love designing Doom levels (or standalone games using Doom-like tech) and can grok sectors, lines, and things but find the vanilla scripting system a bit too arcane and ACS too much like "real programming".

 

If anyone is interested in a system whose benefits require editor UI work, also note that I believe any form of non-code-based scripting should have an underlying representation that IS code-based, so it can be stored sensibly, diff'd and merged, processed with proven methods like parsers and lexers, etc. So all the above discussion of a standard language / concept vocabulary seems useful.

[Arctangent]

13 minutes ago, JPL said:

 

I kinda feel like this is an entirely different conversation. One strictly aimed at those who make the devs of map and lump editors, rather than the dev of source ports. There is still the important part of "does ACS or any other potential language translate well into visual coding," sure, but most of the discussion around it would need to be done by an entirely different set of people.

 

Though,

 

14 minutes ago, JPL said:

• ACS changes require a recompile with an external tool.
• If you're doing heavy level scripting with ACS, you must flip back and forth between not just a text editor and a map editor, but between the text-based mindset of scripts and TIDs and the visual/spatial world of sectors, things, and lines.

both of these are pretty well minimized by GZDoom Builder and I'm pretty sure Doom Builder as a whole for a pretty long time. There's a text editor built right into the program for the sake of writing ACS scripts and those scripts are compiled by an included version of ACC ( which can be updated manually, but is usually also straight-up updated with each version ) upon doing standard actions that would require updating the compiled lump.

[Jayextee]

8 hours ago, Arctangent said:

I kinda feel like this is an entirely different conversation. One strictly aimed at those who make the devs of map and lump editors, rather than the dev of source ports.

 

Surely, and I'm talking idealistically here I know, both these theoretical devs should be working together and toward the same goals? One hand feeds the other, and vice versa.

 

Fact is, any new modding feature only has value if there's a way to implement it -- for example without WadAuthor's customisable config files and Hexen support way back when, would ACS even have taken off as the de-facto standard scripting language? Or would something with more immediate application (such as RTS; although limited it is literally as easy as typing that shit up in notepad, saving the textfile, and then inserting it as a lump in the .wad) have taken off instead? I don't think this is a useless speculative question, either; there's a reason many ambitious features in ports like Vavoom never became the standard -- if the average (and I don't mean middle of the ability scale here, I'm talking lowest-common-denominator largest-bulk-of-folk) mapper has difficulty utilising it, then it generally won't be used.

 

As someone who has processing difficulties (read: dyspraxia) that hamper my understanding of code, visual scripting sounds well up my alley. Even 3DGE's RTS provided a few roadblocks (as evidenced by some of my more 'frustrated' comment lines in SLaVE's scripting...).

[Arctangent]

That doesn't exactly change the fact that if no editor devs have any interest in implementing a visual scripting GUI thing, there's literally no way for it to gain ground unless some new dev pops out of the woodgrain and decides to go that path.

 

Plus, source port devs only really need to chip in if this requires a new language to support. The current idea, after all, is a visual scripting editor that piggybacks on top of an already established scripting language. There's ... really not anything for a programmer that works on a source port without really touching any editors to even do, aside from being around to act as a consultant on the language's specifics.