Update:

I'm a dummy and forgot that drawing a rect and drawing an image are two different things and will need two different pipelines with their own shaders.

Unless there's any objections I'll leave this here in case anyone finds the contents useful.


Howdy,

I'm currently setting a custom pipeline for a Graphics2 instance (on HTML5) so that I can mask a filled rectangle (to draw some health bars with rounded corners without resorting to 9-slice images).

I copied the default g2 image shaders:

#version 450

in vec3 vertexPosition;
in vec2 texPosition;
in vec4 vertexColor;
uniform mat4 projectionMatrix;
out vec2 texCoord;
out vec4 color;

void main() {
    gl_Position = projectionMatrix * vec4(vertexPosition, 1.0);
    texCoord = texPosition;
    color = vertexColor;
}

(modifying the fragment shader to apply the mask):

#version 450

uniform sampler2D tex;
uniform sampler2D texMask;
uniform vec2 maskResolution;
in vec2 texCoord;
in vec4 color;
out vec4 FragColor;

void main() {
    vec4 texcolor = texture(tex, texCoord) * color;
    texcolor.rgb *= color.a;
    vec2 maskCoord = gl_FragCoord.xy / maskResolution;
    maskCoord.y = 1.0 - maskCoord.y;
    vec4 mask = texture(texMask, maskCoord);
    if(mask.r < 0.5) {
        discard;
    }
    FragColor = texcolor;
}

I created a custom pipeline as follows:

var structure = new VertexStructure();
structure.add("vertexPosition", VertexData.Float3);
structure.add("texPosition", VertexData.Float2);
structure.add("vertexColor", VertexData.Float4);

maskPipeline = new PipelineState();
maskPipeline.fragmentShader = Shaders.mask_frag;
maskPipeline.vertexShader = Shaders.mask_vert;
maskPipeline.inputLayout = [structure];
maskPipeline.blendSource = BlendingFactor.BlendOne;
maskPipeline.blendDestination = BlendingFactor.InverseSourceAlpha;
maskPipeline.alphaBlendSource = BlendingFactor.BlendOne;
maskPipeline.alphaBlendDestination = BlendingFactor.InverseSourceAlpha;

maskPipeline.compile();

maskLocation = maskPipeline.getTextureUnit("texMask");
maskResolutionLocation = maskPipeline.getConstantLocation("maskResolution");

Then try to draw as so:

g = // ... Graphics2
g.pipeline = maskPipeline;
// I added the `get_g4()` function to `Graphics2` to get access to the underlying Graphics4 in order to do these calls
g.get_g4().setTexture(maskLocation, mask);
g.get_g4().setFloat2(maskResolutionLocation, mask.width, mask.height);

g.begin(false);
g.color = Color.red;
g.fillRect(/*...*/);
g.end();

When I do this, the mask works perfectly. Also, any images I draw are masked and drawn in proper colour. However, whenever I try to fill a rect with a given colour (g.color = Color.red; g.fillRect(/*...*/);), it is drawn in pure black—i.e. that implies that the vertexColor attribute in the vertex shader either isn't being set or is being set to black. I've dug through the Graphics2 implementation and figure out why this would be happening. The same behaviour is seen if I completely rip out any of the masking and just straight copy the shaders and set the pipeline as above.

Any ideas?

Thanks!