[GZDoom-Software] Fog Boundary Issue with 3D floors

[Rachael]

So - here's my white whale. I've been trying to tackle this for months, and figured I might as well give this forum a try - there might be someone (well, honestly, a lot of people) who knows the software renderer better than I do - particularly ZDoom's, and in this case, hopefully, particularly ZDoom's 3D floor code.

 

As a lot of you might know, there has been a bug in ZDoom's 3D floor code that if fog is present, it overdraws the fog boundary which causes it to not be properly clipped to the walls.

If you attempt to run Unloved in the GZDoom software renderer, there are a few places you can see this bug very prominently.

 

Here's a screenshot of the bug in action:

So - yeah, I've been trying to tackle this one for months, and may have managed to make a break-through with this commit: https://github.com/raa-eruanna/qzdoom/commit/add0acc0c6c1c97a9998d568c070475735be2344

Here's a screenshot of the same scene after this commit is applied:

That should be all she wrote, right? Not only that, it fixes the fog boundaries appearing way too dark, as well. Bonus!

So - off we go packing, problem solved and everything, right? Not quite....
 

There's still some overdraw occurring, and I think it might help to have a fresh pair of eyes look at the code to see what I could be missing - or if there's a better way than this admittedly gross hack (which pales in comparison to what it originally took to get 3D floors working in the first place, in this case, to be honest) to fixing the problem.

Any help would be appreciated. The room where I am primarily testing this is the room that leads to MAP04 inside MAP01, though MAP04 right at the very start works, too.

 

Edited June 19, 2017 by Rachael

[Rachael]

This problem is fully solved now. All I had to do was disable the fog boundaries after they were redrawn once in the "renew" pass.

 

(In the unlikely event anyone is actually curious - it was done here -> https://github.com/coelckers/gzdoom/commit/e290274fb769a6059656842c8cf8d8398436a0c5)