The Force Engine (Jedi Engine Port/Replacement)

[lucius]

1 hour ago, JPL said:

For folks following this project, it does look like great progress has been made since the last update, in this branch of the project's git repo: https://github.com/luciusDXL/TheForceEngine/commits/release

Basically lots of game sim being implemented, as Lucius described in the last update: https://theforceengine.github.io/2021/04/12/GameLoop.html

Thanks for posting about this here, JPL.

 

Process

I haven't posted any new blog posts since the last one, mainly because that work is still ongoing. The way the process works is that the raw reverse-engineered code goes into a private project (for various reasons, though still on GitHub) and then gets integrated into TFE. The code integrated into TFE is what you see on GitHub and what goes into any release. This means there is a chunk of code that is fully or partially reverse-engineered and still waiting to be integrated - this is done so the integrated code can be cleaner and more organized and so I can remove things like variable and function addresses while still keeping that information around in case I need to revisit or double-check some code in the future. Anyway, at some point, all of the reverse-engineered code will make it over, except for any elements replaced by modern equivalents, library functions, or the TFE framework (things like file access, OS-level access, low-level input, etc.). Any systems derived from reverse-engineered code will be called out in the source, so it is clear what parts of the code are derived from the original source and which parts are part of the TFE framework.

 

The goal is to have the next test release, with the core game loop complete, within a few weeks (this month hopefully). And the overall goal of TFE version 1.0 this year remains.

 

Current Work

(As posted on Discord)

Recently I have been converting "back" to using "Tasks" as in the DOS version. I had moved away thinking I could use the raw task functions with some minor modifications since the Task system is honestly over-engineered for the game. However, due to some complexities in some logic functions and yield() based timing, I have been converting back from raw task functions to proper tasks.

 

Originally the DOS code used a hand-written "coroutine" like system written in assembly. Full Coroutines are still more than I want to implement (the complexity to implement and maintain on different platforms doesn't warrant the benefits) - so the Task system will be tweaked slightly to avoid this requirement. To put it simply, this means adding a slight amount of complexity to a few of the task functions. However, this allows other task functions to become simpler - so it is a net win overall.

 

And avoiding full coroutines means 1) I avoid either writing per-platform assembly code to support the feature or 2) I avoid complex/hard to follow template meta-programming which is a nightmare to debug and maintain (and I'm not a huge fan of templates except in very specific cases, like STL containers).

 

Here is an example, this is from the original code but tweaked - the Task system has some features of coroutines but you have to define your local state. This makes the engine code much simpler, even if it does rely on some C-style macro tricks behind the scenes.

void superchargeTaskFunc(s32 id)
{
    // First define the local context structure. If no persistent state is needed, this can be skipped.
    // task_begin_ctx assumes the structure is called 'LocalContext'
    struct LocalContext
    {
        s32 i;
    };
    // A task function must begin with task_begin; or task_begin_ctx;
    task_begin_ctx;

    // Setup supercharged.
    s_superChargeHud = JTRUE;
    s_superCharge = JTRUE;
    
    // Returns and resumes the function in about 45 seconds.
    task_yield(4664);    
    // Then loops 5 times when the powerup is about to run out.
    for (taskCtx->i = 4; taskCtx->i >= 0; taskCtx->i--)
    {
        // Plays an alarm sound, turns off the HUD highlight and than waits 0.6 seconds.
        // Note the function exits at task_yield() and then resumes in 0.6 seconds, hopefully after many frames
        playSound2D(s_superchargeCountdownSound);
        s_superChargeHud = JFALSE;
        task_yield(87);    

        // Turns the HUD highlight back on and then waits 0.6 seconds.
        s_superChargeHud = JTRUE;
        task_yield(87);
    }
    // Finally turns off the super charge.
    s_superCharge = JFALSE;
    s_superChargeHud = JFALSE;
    // Then clears the supercharge task since the task will be deleted after calling task_end;
    s_superchargeTask = nullptr;

    // Removes the task and frees its memory.
    // If instead task_loop was called, then the task would continue starting from task_begin.
    // A while() loop can be used assuming there is a yield.
    task_end;
}

 

The good news is that the final task function code is very close to the original DOS code (mainly adding some syntactic sugar, such as the "taskCtx" to hold "persistent" locals so the functions are re-entrant rather than the stack being auto-allocated and managed by assembly code) and should be functionally equivalent. This means that timing for AI and certain types of potentially problematic Logics, like above, should be accurate.

 

Note, however, that the original Task code acts a bit like threads - but is not necessarily thread-safe (there are cases where I know it isn't just by looking at it). Meaning the logic all happens in the same thread, despite the structure of these functions. Interesting note, I believe this system evolved into the Sith engine's fiber system used by Cog scripts in Jedi Knight.

[lucius]

Game Loop Progress

 

The shape of the core game loop is now clear, so this post will go over its structure and also talk about any changes being implemented for TFE. It will also talk about progress towards the next test release and what has been accomplished so far.

 

. . . .

 

Read the post here: https://theforceengine.github.io/2021/07/14/GameLoopProgress.html

[Dzierzan]

Hey there again!

 

Gonna hijack this thread ^^ (as I see no point making a seperate one and it's related to TFE). As I mentioned before, I am making custom voxels for Dark Forces. Right now I can write that all items, all props and all switches are voxelized! I still have to finish my droids and I also intend to voxelize humans like Jan or burnt dead bodies.

 

Here's all stuff (I didn't show animated voxels, but you can be sure they are done as well):

 

[lucius]

[Progress Update And Plans]

 

As The Force Engine (TFE) approaches the Core Game Loop Completion milestone, I thought it was time for another update and to talk about plans for reaching TFE version 1.0 and beyond.

. . . .

 

Read the post here: https://theforceengine.github.io/2021/08/30/ProgressUpdateAndPlans.html

[lucius]

The Core Game Loop release has been a long time coming. Most of that time has been occupied reverse-engineering the original code, but the integration process (integrating the code into TFE, testing it, refactoring, fix bugs) has been proceeding quickly.

 

The plan is for the "Core Game Loop Release" to be available in late September, which will complete the core loop - meaning being able to start a new game and play through all of the levels with full weapons, enemies, and level functionality.

 

The original placeholder code has been removed, this video shows The Force Engine (TFE) running the reverse-engineered Dark Forces code, starting with main. It shows using save game data, launching a level, and playing until the end. AI is not yet functional but should start coming online next week.

 

 

Here are some of the features and systems shown:

0:00:04 Agent Menu, using the original save game data.

0.00:15 Sliding on Ice, and water with a current.

0:00:18 Weapon fire and hit effects.

0:00:28 Hitting F7 to disable and enable the HUD.

0:00:39 More ice physics and the Cleats.

0:01:00 Running and walking (moving slower than normal).

0:01:12 Pickups with effects, changes to the HUD, and associated on-screen text.

0:01:25 Water current.

0:01:35 Conveyor Belts.

0:02:13 Shootable Switches.

0:02:36 Damage Floors.

0:02:52 Crushing Damage.

0:03:08 Standard Switch.

0:03:34 Cycling Weapons.

0:03:45 Red Key Door.

0:03:52 Me fumbling to enter the LAUNLOCK cheat to get the red key. Note that the message wasn't displayed because the Red Key message has a higher priority.

0:04:18 Damage Sector.

0:04:23 Gas Mask.

0:05:05 Night vision googles.

0:05:10 Head lamp.

Edited September 13, 2021 by lucius

[Murdoch]

I just realized something. Now when anyone posts about problems running these games on modern systems people can reply "use the force (engine)".

[QuaketallicA]

3 minutes ago, Murdoch said:

I just realized something. Now when anyone posts about problems running these games on modern systems people can reply "use the force (engine)".

 

DOSBox is still my #1 choice...just make sure to edit the config file to disable the awful normal2x upscale that is enable by default and makes everything super pixelated.

You can even run any soundfont you want if you run the setup.exe and have the coolsoft viritualmidisynth software with soundfonts downloaded.

[Dzierzan]

4 hours ago, QuaketallicA said:

 

DOSBox is still my #1 choice...just make sure to edit the config file to disable the awful normal2x upscale that is enable by default and makes everything super pixelated.

You can even run any soundfont you want if you run the setup.exe and have the coolsoft viritualmidisynth software with soundfonts downloaded.

 

I guess this will change once TFE hits ver 1.0 and beyond. I mean if TFE allows for playing 320x240 resolution with modern controls then it will always beat Dosbox, knowing also that Lucius aims for faithfulness. Of course I wanna use only HD resolution, as I am tired of pixelation.

Edited September 13, 2021 by Dzierzan

[VGA]

Original in Dosbox hurts my eyes, it seems more pixelated than Doom. I think it's because of the textures and the sprites. Doom has less color diversity and is darker. Dark Forces is a pixelfest.

[lucius]

1 hour ago, VGA said:

Original in Dosbox hurts my eyes, it seems more pixelated than Doom. I think it's because of the textures and the sprites. Doom has less color diversity and is darker. Dark Forces is a pixelfest.

I think Dark Forces looks really nice at HD resolutions. The video is showing the original 320x200 resolution because that is the current state of the release - I'm focusing on finishing the core game loop release and fixing accuracy issues (i.e. reverse-engineering mistakes). But the actual release will support higher resolutions and widescreen.

[j0lt]

Oh wow, that's awesome! Essentially giving it the GZDoom treatment?

[lucius]

43 minutes ago, j0lt said:

Oh wow, that's awesome! Essentially giving it the GZDoom treatment?

There will be a GPU-based renderer available in the future, though the original software renderer will always be available. One key point is to preserve the original code and functionality and then add optional enhancements.

[j0lt]

27 minutes ago, lucius said:

There will be a GPU-based renderer available in the future, though the original software renderer will always be available. One key point is to preserve the original code and functionality and then add optional enhancements.

Awesome! Can't wait to see it progress!

Is it going to feature modern controls and keybindings as well?

[lucius]

5 minutes ago, j0lt said:

Awesome! Can't wait to see it progress!

Is it going to feature modern controls and keybindings as well?

It already does - including controller support. :)  I was using mouselook and WASD controls when capturing the video.

[kwc]

I’m currently playing through Dark Forces on dosbox, so I’m looking forward to this release. 
Seems like there is quite a modding scene over there, can’t wait to check out community maps with this port.

 

[j0lt]

On 9/13/2021 at 1:24 PM, lucius said:

It already does - including controller support. :)  I was using mouselook and WASD controls when capturing the video.

Well that's exciting!

[lucius]

I have just finished integrating and testing all of the weapons, including secondary fire. This means it is finally time to move on to the AI agents. This weekend I hope to get at least the Mousebot and exploding barrels done, and hopefully, the rest of the AI can be integrated next week.

 

In the meantime, I recorded a short video showing the weapons in action:

 

[lucius]

There are a few bugs that have been pointed out - some fire rate issues and the fusion cutter not firing left to right and then right to left (that one has already been fixed). If you see something, you can post about it here: https://the-force-engine.freeforums.net/board/4/bugs

 

Bugs aside though, the weapons are all in so you should start seeing some AI early next week. :)

Edited September 18, 2021 by lucius

[lucius]

Several bugs were fixed, so weapons are done* - and it's time to start integrating the AI code. So hopefully the next video will feature things shooting at me. :)

 

Bugs fixed:

• Fusion Cutter, not alternating fire direction from left to right and then right to left.
• Repeater primary mode firing too fast.
• Assault Cannon's primary firing rate being slightly off.
• Not being able to fire the Repeater faster by mashing the button versus holding it down (it is only slight but noticeable).

* done, for now, more bugs may be uncovered during testing.

[lucius]

The Core Game Loop Release gets closer with the Integration of Dark Forces Actor/AI system. Each AI Actor consists of a number of shared components, running independently and communicating over callbacks. This video shows several AI agents - the "Troopers" (Stormtroopers, Officers, and Commandos), Mousebots, "Scenery" (animated bits in the environment that can be destroyed like the red lights), and not shown but fully working - exploding barrels and pre-placed landmines.

 

Note: there are several known issues that will be visible/audible in the video:
* lack of iMuse (future release)
* Kyle not going down properly with elevators (falling).
* I need to double-check the threshold for the landing sound.
* audio falloff doesn't quite match DOS (future release)
* there is an INF bug related moving walls/doors that will be addressed for this release.

 

I expect to see the rest of the AI should come online quickly. Unless there are hidden surprises, I expect to be finished before the end of next week.

 

 

[Rathori]

Damn, this looks so good.

[lucius]

I mentioned this on the Discord, but the Interrogation and Probe Droids, Ree-Yees (both variants since they are technically different enemies), Trandoshans, and Gamorrean Guards have been integrated since this video was published. This required integrating the flying actor support and thermal detonator AI aiming. Which basically completes the AI system, for basic enemies I think (there is a bit more for Remotes and boss enemies though). So it is "just" a matter of integrating the rest of the setup functions, which should take a few hours (I still have to do a bit of reverse-engineering there since I skipped most of the enemy setup code before, but it is really quick since they all follow the same patterns).

 

The main issue is that there is a movement issue with Ree-Yees that I noticed in Ramsees Hed, so I will need to fix that before finishing the basic enemies. But I expect them to be done this weekend. Then I can spend a few days on boss enemies and finally start wrapping up this release.

Edited October 2, 2021 by lucius

[Kinsie]

Really excited about this! Looks great so far.

[lucius]

All of the basic enemies and a few of the bosses have been integrated - leaving only three bosses to finish.

 

So I decided to take a step back and deal with other loose ends - Level Completion, VUE Animations, Enemy Generators, and fixes for various bugs and issues. This video shows this work by running through Talay, where the VUE animations, Enemy Generators, and other features can be seen.

 

 

Edited October 15, 2021 by lucius

[Somakarma]

Any explanation or fix for game running hyper fast? I have loaded your mod but its as if too many CPU cycles are running on DOSBox.

[lucius]

On 11/6/2021 at 1:43 PM, Somakarma said:

Any explanation or fix for game running hyper fast? I have loaded your mod but its as if too many CPU cycles are running on DOSBox.

Most likely you have forced Vsync off in your GPU driver's control panel. Currently TFE assumes that it is on, since it turns it on and the driver than overrides it behind the scenes. When vsync is enabled, TFE rounds time steps to the nearest vsync interval (or 1) in order to eliminate microstutters. This works great when the API actually does what the code tells it to, but not when it is externally overridden. (If you can't tell, I really dislike this feature - the driver overriding GPU behavior behind the game's back with no knowledge of why decisions were made can often break games, but of course, only big games get the appropriate driver fixes).

 

Anyway, you have two options:

1. Enable vsync for TFE in the control panel.

2. Wait for a future build that will have an option to turn off vsync. Then you can turn it off in TFE so it matches.

 

I will make an announcement here when the full "Core Game Loop" Release is available - when the full game is completable and graphics options are back such as widescreen and higher resolution support, vsync, etc.

Edited November 8, 2021 by lucius

[lucius]

TFE hit a major milestone recently - as of build 0.02.001-107-g5977a8a, available on the downloads page, all enemies have been integrated and Dark Forces is playable from beginning to end using The Force Engine. Note that if you have already played Dark Forces in DOS, TFE will copy over your agents/save games. If you download, you should obviously pick the latest build, which will be at the top.

Currently, support for higher resolutions and widescreen is being re-added (current builds are still limited to 320x200), since that was previously removed due to refactoring and changes due to integrating the reverse-engineered code. Once complete, I will make an official Core Game Loop build - which will complete that milestone. This should be finished sometime this week, with the build being released early next week. Once this build is released, widescreen and higher resolutions will be supported, such as 1080p.

The next milestone, with work starting next week, will cover Cutscenes, In-Game UI, and Mission Briefings. While the current build allows you to play through the game, you will need to look up key card images online on two levels, because you cannot bring up the PDA. The next big milestone will fix that. I will post more information about that once work begins.

 

The most recent builds also include the UI to remap controls, adjust mouse and controller sensitivity, as well as inverting each axis.

Edited November 11, 2021 by lucius

[lucius]

High resolution and widescreen support is now working. The following video shows high-resolution support that updates the rendered view as you change it in the UI, widescreen support, HUD adjustment, and color correction. I "LASKIP" through a couple of levels to show what they look like in high resolution and widescreen and compare to 320x200.

 

 

The next feature for the "Core Game Loop" release is public mod support. Once that is finished, I will consider the milestone done - though there are still some bugs - and I will move on to the next milestone: cutscenes, mission briefings, and in-game UI (mainly the PDA).

[Yhe1]

when is texture filtering going to be done?

[lucius]

40 minutes ago, Yhe1 said:

when is texture filtering going to be done?

Not until after version 1.0 is released - this is still using software rendering. While moving to higher resolutions required me to write a floating-point renderer, it is derived directly from the original fixed-point renderer, so it has the same glitches and features. If you select 320x200, it uses the original fixed-point renderer though, to preserve the code.

 

Anyway, a hardware renderer is not planned until enough reverse-engineering of Outlaws is complete to properly integrate the enhanced features (slopes, dual adjoins, vertical adjoins, etc.) - I want the full core renderer to be complete before moving on to hardware support. In other words, there won't be any texture filtering until early next year, since that is when work on Outlaws will begin in earnest. The rest of this year will be spent finishing version 1.0 (Dark Forces support complete).