GZDoom and Custom Shaders

[illYay1337]

I've been working on a custom shader and I figured things out from here: https://zdoom.org/wiki/GLDEFS

 

I have my custom shaders working but I noticed some issues.  First of all, the surfaces with my shader appear brighter but only at odd angles.  90 degree walls and ceilings are OK, and my map is currently using the default shading behavior on 90 degree walls.  I haven't tried even lighting or smooth lighting mode yet.

 

Spoiler

Slopes are also brightened.  It also seems to do flat shading a bit while non-material surfaces use the sector brightness directly without shading.  My rocky terrain ends up looking worse. (However notice how there's no ugly transition from flats to walls thanks to triplanar mapping 8D)

Spoiler

 

Also there's this artifact that's especially visible on slopes.  It's harder to see on a still image so I modified my shader to output solid red.

 

Spoiler

 

These are issues that appear to be out of my control.  I tried both using a material and a hardware shader with identical results.  When using a material all I'm outputting is the Albedo color so lighting is all taken care of already by the engine and should just be using sector brightness.  Hardware shader has identical behavior visually.

I have quite a lot of experience with computer graphics and GLSL.  I've seen this pinprick artifact in the past when I made a mistake calculating the specular highlight in a custom engine.  However here this pinprick is not created by me.  Surfaces that are just assigned a texture render completely fine.

Here are the material sources for reference.  It's essentially to solve the ugly transition in a cave environment between the sidedefs and the sloped floors by doing away with texture mapping altogether.  The UV's are determined by world coordinates.

 

/**
 * Shader that does triplanar mapping which should be great for natural terrain textures
 * The UVs are procedurally determined based on world coordinates so caves, mountains, etc automatically look good
 * This can be very useful for Doom where it's otherwise difficult or nearly impossible to align walls and flats
 * At the moment it's built to take the passed in texture and triplanar map it.
 *
 * Made with the help of this article: https://www.martinpalko.com/triplanar-mapping/#The%20Theory
 *
 * Future versions or variations might get more advanced, and you could make snowy mountains for example, by having top facing normals use a snow texture
 * 
 * Usage: Add to gldefs as explained here: https://zdoom.org/wiki/GLDEFS
 *
 * You can add either a worldScale define or uvScale define
 * By default it acts as if you passed worldScale 1.0
 * worldScale 1.0 will try to map the texture as closely to how doom would map it as possible in doom's world coordinates
 * Other values will multiply the scale
 *
 * uvScale won't take world scale into account and will just base UV's one to one with world position, so you may want very low values like .001 to look good.
 * uvScale is slightly more optimized since there's no division by texture size
 *
 * e.g.
 * material texture "textures/ogrodtex/OGRIDRST_triplanar.png"
 * {
 *   shader "shaders/triplanar.fp"
 *   define worldScale = 2
 * }
 */

Material ProcessMaterial()
{
    Material material;

    vec2 uvScaleActual =
#ifdef uvScale
        vec2(uvScale);
#else

#ifndef worldScale
#define worldScale 1.0
#endif
        //to sample the texture in "DooM" world units we divide by its size
        1.0 / textureSize(tex, 0) * worldScale;
#endif

    vec3 normals = normalize(vWorldNormal.xyz);
    vec3 blendWeights = abs(normals);
    
    // Z normal faces north/south
    // Y normal faces floor/ceiling or up/down
    // X normal faces east/west

    // Floor / Ceiling need to sample in the -v direction
    // Walls sample in the -v direction as well, but also flip u depending on facing direction to more roughly match doom's texture mapping

    vec2 northSouthUVs = pixelpos.xy * vec2(sign(-normals.z), -1.0) * uvScaleActual;
    vec2 floorCeilUVs = pixelpos.xz * vec2(1.0, -1.0) * uvScaleActual;
    vec2 eastWestUVs = pixelpos.zy * vec2(sign(normals.x), -1.0) * uvScaleActual;

    material.Base =
        // North / South
        getTexel(northSouthUVs) * blendWeights.z
        // Floor / Ceiling
        + getTexel(floorCeilUVs) * blendWeights.y
        // East / West
        + getTexel(eastWestUVs) * blendWeights.x;

    // In the future, if more textures are used for other material components, sample them in a similar way
    // Future versions can also sample a totally different texture for Floor, for example, to add snow or grass to tops of hills
    // You could also sample different textures based on elevation in world coordinates, so high peaks are snowy while low areas are grassy

    return material;
}

 

Edited November 19, 2022 by illYay1337

[illYay1337]

Ah it's actually hilarious what a mistake I made.  It was brighter because of a mistake in the triplanar shader not because the engine was making things brighter.

 

I tried regular solid colored textures and the same odd artifacts were visible for me.  The brightness came from this:

return
        // North / South
        getTexel(northSouthUVs) * blendWeights.z
        // Floor / Ceiling
        + getTexel(floorCeilUVs) * blendWeights.y
        // East / West
        + getTexel(eastWestUVs) * blendWeights.x;

I was over brightening the pixels which happens with any angled surface but never with 90 degree surfaces.  BlendWeights came from a normalized vector, but the sum of the components wasn't 1.  I somehow forgot that a normalized vector is of length 1 but the sum of its components is not 1, which caused the 3 colors to blend incorrectly resulting in an overbrigthened pixel.

Had to add this earlier in the shader.

    // make it so the sum of all components is 1
    blendWeights /= blendWeights.x + blendWeights.y + blendWeights.z;