Compositing gave me a ground-truth alpha, but the truth came from a model

For a course project I wanted numbers for video background removal, not just squinting at frames. SAD, MSE and the rest all need a ground-truth alpha for every frame, and nobody hand-paints that for real footage. The usual trick is to build the clip backwards: take a foreground whose alpha you already know and composite it onto a background yourself. It worked, with one catch in what that "truth" actually is. Everything below comes from one synthetic clip, 90 frames at 540×960, one machine, one run.

Compositing turns a known alpha into a test clip

The formula is the plain over operation, I = αF + (1−α)B. Choose α and F yourself and you know the right answer for every pixel before any method looks at it. My foreground came from a public-domain VOA Learning English green-screen clip on Wikimedia Commons. I took frames 90 to 179 (3.0 to 6.0 seconds, the stretch where she waves), ran them through an AI matting export to a transparent WebM, and decoded the RGBA frames back out with FFmpeg, naming the libvpx-vp9 decoder explicitly and before the input. Without that every alpha value decodes as 255. A small numpy script then put those frames over two backgrounds I generated. One is flat green, RGB(0,177,64). The other is a cluttered canvas of 420 random blocks, circles and lines, green and skin-tone patches included, sliding one pixel left per frame to fake a slow pan. Both get fresh Gaussian noise (σ=2) every frame, and the script wrote out all 90 frames at 540×960.

Sample: VOA Learning English, Wikimedia Commons, public domain; trimmed and composited, modified. Backgrounds self-made.

Both people are foreground in the alpha panel. In the trimap the gray edge band traces both outlines, from the hair down to the shoes.

Swapping only the background moved every number

Scored against itself, the ground truth got 0 on every metric, so the pipeline adds no error of its own. The AI matting I then scored is the compatible tier in ImgIng https://imging.ai/ , run in the browser on an Apple M4 Mac. On the green version its average SAD over 90 frames was 2.89. Over the cluttered background, same everything else, it was 7.44. Edge-band flicker, measured only where the true alpha stays still between frames, went from 2.205 to 11.984 (mean frame-to-frame change ×1000). Only the background changed, so the background is the cause. Real footage never gives you that.

The same bench caught a chroma key failure. FFmpeg's chromakey at similarity 0.10 scored SAD 3.88 on the green version. At 0.25 it jumped to 93.37, and only 40.65 of that sat inside the edge band, which covers about 15.8% of the frame. Most of the rest came from dark clothes and shoes that the key treated as green. A reference shows you where it got worse, not just that it did.

The ground truth came from a model

This is the part that's easy to get wrong. A known alpha is not the same thing as a correct one. Mine came from a matting model, RVM ResNet50, not from someone refining hair by hand. A method that got a strand of hair more right than the model would be scored as wrong. So the composite tells you how far a method is from a known alpha. It doesn't tell you how far it is from the real edge of her hair, and I have no hand-made matte to measure that gap.

The compatible tier runs RVM MobileNetV3, a smaller sibling in the same family as the reference model. My guess is that siblings make similar mistakes, and similar mistakes cost nothing here. I can't measure that favour, so the AI numbers may be optimistic.

The foreground still carries green spill

The second limit lives in F rather than α. The foreground colours come from the original green-screen footage, so fine hair pixels carry a green tint from the cloth behind her. That puts chroma keying at a disadvantage along the edges, and that comes from my clip, not the method. Meanwhile my synthetic green is one even colour plus light noise, unlike the real cloth, which covered only the middle of the frame and wasn't evenly lit.

I use the clip as a bench and not a judge

The synthetic clip answers one-variable questions well: same method over a different background, a different similarity value on the same background, or whether the error sits in the edge band. It's bad at "which method is better", because the reference was made by a close relative of one contestant. These values also don't belong next to a paper's table: resolution, region and normalization all differ. Next time I'd change the reference first: hand-refine a few frames, or build F and α from footage that never saw a green screen. I haven't figured out how to do the second one yet.

Sample credit: "Filming with the green screen" by VOA Learning English, Wikimedia Commons, public domain. Backgrounds generated by me.