The Force Engine (Jedi Engine Port/Replacement)

[VGA]

Ohhh, nice

 

 

[Arno]

Thanks for taking the time to put together that detailed blog post on 3D object rendering. I always like to read about how various 3D effects were achieved under strict performance limitations.

[lucius]

5 hours ago, Arno said:

Thanks for taking the time to put together that detailed blog post on 3D object rendering. I always like to read about how various 3D effects were achieved under strict performance limitations.

I'm glad you liked that type of content. :)

 

At some point, I have to continue the Dark Forces DOS rendering series.

 

@VGA - I recently added GIF recording to the engine, so I decided to use it to show the 3D object rendering in the Classic Renderer, as you see here. So it is now almost as easy to record GIFs and it is to take screenshots.

[VGA]

10 hours ago, lucius said:

I'm glad you liked that type of content. :)

 

At some point, I have to continue the Dark Forces DOS rendering series.

 

@VGA - I recently added GIF recording to the engine, so I decided to use it to show the 3D object rendering in the Classic Renderer, as you see here. So it is now almost as easy to record GIFs and it is to take screenshots.

Ah, nice. But isn't it big? The filesize for a few seconds of gif action? I am not at my PC so I can't easily check the dimensions and such.

 

Also, I realised I buried your update post in the previous page, here it is again LOL

 

On 12/27/2020 at 11:52 PM, lucius said:

I posted a final blog entry for 2020, the Classic Renderer is finally complete - just in time to end 2020.

 

TFE Classic Renderer Complete

The 3D Object rendering system, derived from the original DOS code, is now working in both 320x200 (fixed-point sub-renderer) and at higher resolutions and in widescreen (floating-point sub-renderer). This completes the Classic Renderer reverse-engineering and integration project, allowing me to move on to reverse-engineering the INF system.

 

. . . .

 

Read the post here: https://theforceengine.github.io/2020/12/27/TFE-Classic-Renderer-Complete.html

 

[lucius]

GIFs can be big, yes. But it is nice for some quick animations that play well in the browser - great for forum or blog posts. Obviously, GIF recording is no replacement for proper videos. Currently, the engine is set to record at 15fps and I usually record them at 640x480 in order to manage the size.

Edited December 29, 2020 by lucius

[Annunakitty]

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

[taufan99]

15 minutes ago, Annunakitty said:

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

Lucius has said that he's entirely focusing on Jedi Engine for now.

[[McD] James]

18 minutes ago, Annunakitty said:

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

 

Have you seen Daggerfall Unity? It's going to be amazing. https://www.dfworkshop.net/

[Annunakitty]

28 minutes ago, Ajora said:

 

Have you seen Daggerfall Unity? It's going to be amazing. https://www.dfworkshop.net/

I hadn't!  Thanks for bringing this to my attention!

[lucius]

3 hours ago, Annunakitty said:

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

The Force Engine is focused on the Jedi Engine games - so Dark Forces and then Outlaws. The Force Engine will remain a focused project and I will not be adding support for any other games,etc. until Dark Forces and Outlaws are completely playable and the modding tools are ready. And for these kinds of projects, TFE will remain the sole focus - I don't have enough bandwidth for multiple large-scale projects at once.

 

I haven't given up on the possible idea of a proper reverse-engineered "X Engine" port in the future (Future Shock, Daggerfall, etc.) - but that is a different project and in a distant future (if it happens). That said, I think Daggerfall Unity will work well for you. :)

Edited December 29, 2020 by lucius

[lucius]

In other news, I have moved on to INF reverse-engineering. But before I did, I decided to add optional perspective-correct texturing support to 3DO rendering. It uses "affine spans" to speed up the calculations in software, similar to Quake and similar engines. The length of these spans is computed based on the current virtual resolution. For example, at 200p perspective correction only occurs every 8 pixels whereas at 1080p it is every 32 pixels and at 2160p every 64 pixels.

 

Here is a short GIF showing the comparison:

 

I tested the results in one of the larger mods, Dark Tide 4, and took some screenshots and GIFs. These are running in the software renderer at 1080p with perspective correct texturing enabled for 3DOs:

 

 

And a large GIF:

 

 

If you haven't read it already, the most recent blog post is here: https://theforceengine.github.io/2020/12/27/TFE-Classic-Renderer-Complete.html

 

You can also see a list of archived blog posts with recent news here: https://theforceengine.github.io/archive.html

Edited December 29, 2020 by lucius

[lucius]

TFE Update and 2021 Plans

As the first blog post of 2021, this will cover several topics. First, an update regarding the progress of The Force Engine. That will be followed by TFE plans for 2021 - the year of the first fully playable release (hopefully), and finally the first “experimental” build and what that is all about.

. . . .

 

Read the post here: https://theforceengine.github.io/2021/01/18/TFE-Update-2021-Plans.html

[lucius]

INF System and TFE Progress

 

In the last post - TFE Update and 2021 Plans - I talked about the INF & Classic Renderer release which was to be followed by a “Player Control & Physics” release later in the year. Due to the way the renderer, INF and player controller are connected there has been a change of near-term plans. This post will go over those changes, talk about how the INF system works internally and provide a general update regarding the progress towards this year’s goal of reaching the 1.0 release.

. . . .

Read the rest of the post here: https://theforceengine.github.io/2021/03/02/InfSystem.html

[Doominator2]

Oh wow, I didn't even know the XL engine was still a thing. I remember religiously checking the forums everyday for progress on the XL port of Daggerfall around 2013-2014 until it just went silent for a while and then Daggerfall Unity started to catch some steam. Glad to know that the XL engine is still a thing, I'm guessing the Daggerfall and Outlaws ports are a thing of the past now? There was another port that was being worked on too but I forget what it was, might have been another Lucasarts fps?

[lucius]

2 minutes ago, Doominator2 said:

Oh wow, I didn't even know the XL engine was still a thing. I remember religiously checking the forums everyday for progress on the XL port of Daggerfall around 2013-2014 until it just went silent for a while and then Daggerfall Unity started to catch some steam. Glad to know that the XL engine is still a thing, I'm guessing the Daggerfall and Outlaws ports are a thing of the past now? There was another port that was being worked on too but I forget what it was, might have been another Lucasarts fps?

You might want to read https://theforceengine.github.io/ and https://theforceengine.github.io/2020/05/05/firstpost.html to get an idea what The Force Engine is. You can see the blog archive here: https://theforceengine.github.io/archive.html.

 

In summary, TFE is a fresh start, focused on accuracy using reverse-engineering, and is focused only on "Jedi Engine" games - Dark Forces and Outlaws. Dark Forces is the current focus but Outlaws shares a lot of the same code. The goal is for TFE to be a complete DosBox replacement for Dark Forces by the end of the year (meaning it is accurate to the original but does not use emulation).

[Rudolph]

Okay, now I want to see a Dark Forces remake of Shadows of the Empire.

[Gez]

Quote

A Note on Timing

Dark Forces quantizes time to 145 ticks per second.

Interesting. I wonder why the LucasArt devs chose this value -- I know Doom's value of 35 ticks per second was based on video refresh rate on the usual hardware of the era (70 Hz for 320x200 in 256 colors), as a simple way to keep game updates and video updates in sync.

[lucius]

1 hour ago, Gez said:

Interesting. I wonder why the LucasArt devs chose this value -- I know Doom's value of 35 ticks per second was based on video refresh rate on the usual hardware of the era (70 Hz for 320x200 in 256 colors), as a simple way to keep game updates and video updates in sync.

The strange thing is that Dark Forces uses the same 70Hz video mode. I think the reason it isn't tied directly to framerate is that the Jedi Engine uses a "Task" system, where tasks can yield, run at specific rates, etc.. Anyway, game and INF code is split into these tasks with the idea that not everything needs to run every frame. In the case of the INF system, it doesn't really work - I mean the main update task runs every frame in almost every map, as does the texture animation task, but I think this is one of the reasons they chose more granular timer settings. For TFE, this is being greatly simplified since it doesn't really buy anything but the timing itself has to stay the same.

 

That said, I didn't believe the 145 ticks / second thing at first, thinking I made a mistake. But you see it over and over again in the code, it is no mistake as odd as it is. :)

[lucius]

I put up a new blog post regarding the state of the project:

 

TFE Foundation and Test Release

 

The Jedi Renderer and INF System have been completely reverse-engineered and the collision detection system is almost complete. But a new problem looms on the horizon.

. . . .

Read the post here: https://theforceengine.github.io/2021/03/19/Foundation.html

Edited March 20, 2021 by lucius

[lucius]

Closing the Game Loop

 

The previous blog post talked about refactoring the code so that the reverse-engineered INF system, collision system, and renderer can all use the original, fixed-point level data. Another goal was to enable caching for systems, like the floating-point high resolution renderer, that need to transform the data in some way. It illustrated the connected nature of the core game and engine code

. . . .

Read the post here: https://theforceengine.github.io/2021/04/12/GameLoop.html

Edited April 12, 2021 by lucius

[JPL]

Very cool! For possible future project news posts, I'm curious to know what your process for reverse engineering is. Do you use a disassembler, monitor the running game in DOSBox, something else, or a mix of techniques?

I've never done this kind of programming before and it's where lots of "reimplement old game" open source projects seem to hit a hard wall after getting the formats figured out and the world rendering.

[lucius]

19 hours ago, JPL said:

Very cool! For possible future project news posts, I'm curious to know what your process for reverse engineering is. Do you use a disassembler, monitor the running game in DOSBox, something else, or a mix of techniques?

I've never done this kind of programming before and it's where lots of "reimplement old game" open source projects seem to hit a hard wall after getting the formats figured out and the world rendering.

Honestly, there is no magic to it - just a lot of tedious work, reading documentation, and puzzle-solving. :)

 

It is basically a combination of disassembly + debugging in DosBox (for DOS-based games anyway, you can use Visual Studio for Windows programs). Most of the work is figuring out patterns, keeping track of all of your functions, variables, and structures, and constantly updating them with good names and types as you figure things out.

 

For example, take this function:

void obj_setupAnimation()
{
    Logic* logic = s_curLogic;    // eax
    logic->u68 = (logic->u68 | 1) & 0xfd;
    logic->nextTick = s_curTime + logic->delay;

    SecObject* obj = logic->obj;    // ebx
    obj->anim = obj_getAnimationIndex(5);    // eax -> edx
    obj->frame = 0;
}

 

You can see that most elements of the logic structure have already been figured out but 'u68' is unknown. So I have the name stubbed out (u = unknown, 68 = at offset 0x68 in the structure). Later I will see logic->u68 in action and figure out what it is used for, then I will go back and update the name. By updating the name, that will enable new understanding in existing code which will allow me to figure out other elements of the code (for example '1' is likely a flag, what does it do? Why are some flag bits cleared (the 0xfd)).

 

Another big component, once you figure what the low-level code is doing is figuring out why. For example, take this function:

s32 vec2ToAngle(fixed16_16 dx, fixed16_16 dz)
{
        if (dx == 0 && dz == 0)
        {
            return 0;
        }

        const s32 signsDiff = (signV2A(dx) != signV2A(dz)) ? 1 : 0;
        // Splits the view into 4 quadrants, 0 - 3:
        // 1 | 0
        // -----
        // 2 | 3
        const s32 quadrant = (dz < 0 ? 2 : 0) + signsDiff;

        // Further splits the quadrants into sub-quadrants:
        // \2|1/
        // 3\|/0
        //---*---
        // 4/|\7
        // /5|6\
            //
        dx = abs(dx);
        dz = abs(dz);
        const s32 subquadrant = quadrant * 2 + ((dx < dz) ? (1 - signsDiff) : signsDiff);

        // true in sub-quadrants: 0, 3, 4, 7; where dz tends towards 0.
        if ((subquadrant - 1) & 2)
        {
            // The original code did the "3 xor" trick to swap dx and dz.
            std::swap(dx, dz);
        }

        // next compute |dx| / |dz|, which will be a value from 0.0 to 1.0
        fixed16_16 dXdZ = div16(dx, dz);
        if (subquadrant & 1)
        {
            // invert the ratio in sub-quadrants 1, 3, 5, 7 to maintain the correct direction.
            dXdZ = ONE_16 - dXdZ;
        }

        // subquadrantF is based on the sub-quadrant, essentially fixed16(subquadrant)
        // which has a range of 0 to 7.0 (see above).
        const fixed16_16 subquadrantF = intToFixed16(subquadrant);
        // angle = (int(2.0 - (zF + dXdZ)) * 2048) & 16383
        // this flips the angle so that straight up (dx = 0, dz > 0) is 0, right is 90, down is 180.
        const s32 angle = (2 * ONE_16 - (subquadrantF + dXdZ)) >> 5;
        // the final angle will be in the range of 0 - 16383
        return angle & 0x3fff;
}

Where:

signV2A is: signV2A(x) { return (x < 0 ? 1 : 0); }

 

You can imagine, before the comments and some functionally equivalent changes this looked like a string of random math operations. Some things, like abs() hide a lot of underlying complexity in the assembly. But if you want to be able to fix bugs and extend the code in the future it is not enough to figure out what the code is doing, but you need to figure out why it is doing it. So having a good background in game/engine programming helps and of course some knowledge of assembly, though even then you will have to look up some of the more esoteric instructions every now and again.

 

So it is a mix of tedium with spikes of puzzle-solving and head-scratching. After a while, you just start reading the assembly and just start typing the equivalent C-code. :)

[Scrabbs]

this is awesome, whatever it is your doing it's working phenomenally keep it up and do your best

[JPL]

Thanks for the explanation Lucius! I'm familiar with that style of gradually filling in a disassembly. Your commenting style is very good and makes it very easy to understand what's going on in that excerpt, I strive for that with my own work. Looking forward to playing the results, keep up the great work.

[lucius]

1 hour ago, JPL said:

Thanks for the explanation Lucius! I'm familiar with that style of gradually filling in a disassembly. Your commenting style is very good and makes it very easy to understand what's going on in that excerpt, I strive for that with my own work. Looking forward to playing the results, keep up the great work.

Thanks. To be clear, though, comments in most of the code aren't as dense. Generally, it is preferable to write "self-documenting" code and be a little more sparse with comments (since comments can go out of date over time and reduce the amount of code you can see at once). However, comments should always be used when the "how" and "why" of the code are non-obvious, as is the case with that particular function.

[Arno]

21 hours ago, lucius said:

Generally, it is preferable to write "self-documenting" code and be a little more sparse with comments (since comments can go out of date over time and reduce the amount of code you can see at once).

This is a good practice and one of the software engineering guidelines that Robert C. Martin a.k.a. "Uncle Bob" advocates in his book Clean Code. That book was an eye-opener for me.

But indeed, your code sample is the kind of algorithm where comments do have value.

[Dzierzan]

Hey there! I wanted to say that I am the author of that Dark Forces Voxel Pack. I personally find it crazy that I started the damn project without having any warranty that The Force Engine will be finished. Anyway, FYI, my pack is nearly completed... or.. it depends how you look at it. When it comes to props and items, only 5 voxels are missing. I also intend to fully voxelize at least 3 droids (3 of them are started) and maybe voxelize human props like Jane. But all in all, it should be a cool mod to play Dark Forces with :).

 

Here's an example of my latest work:

 

 

Ohh and I would also love to make Smooth Dark Forces mod using TFE, here are examples:

 

 

But I won't be stealing all the show ^^.

Edited May 6, 2021 by Dzierzan

[ReX]

@Dzierzan: That voxel model of Palpatine is phenomenal! The smooth weapon-firing animations are cool too.

[DuckReconMajor]

Those smooth weapons look so good they should be illegal

[JPL]

For folks following this project, it does look like great progress has been made since the last update, in this branch of the project's git repo: https://github.com/luciusDXL/TheForceEngine/commits/release

Basically lots of game sim being implemented, as Lucius described in the last update: https://theforceengine.github.io/2021/04/12/GameLoop.html