Compositing, or putting characters in front of a fake background, has been around since the beginning of cinema, and it has always been a tedious process. Netflix has a new method that uses machine learning to automate some of the labor but uses harsh magenta lighting on the actors.
Chroma keying has been the go-to technique for basic compositing for decades. In this technique, actors stand before a brightly colored background (often blue or green) that can be easily identified and replaced with anything from a weather map to a battle against Thanos. The “alpha” channel, which is transparent, is modified along with the red, green, and blue channels to create a “matted” effect in the foreground.
While quick and inexpensive, there are a few drawbacks to consider. Things with transparency, small details like hair, and anything that shares a color with the background may not come out as expected. However, it’s usually good enough that attempts to replace it with more advanced and costly approaches (such as a light field camera) have stalled.
However, Netflix researchers are trying it with a hybrid approach that could result in easy, flawless compositing but requires a hellish lighting setup on set.
According to recent research, their “Magenta Green Screen” achieves remarkable effects by encasing the players in a sandwich of light. Behind them was a sea of green light (natural lighting, not a backdrop) and a riot of red and blue in front of them.
Even a seasoned post-production artist would probably cringe at the resulting appearance on-site. Although they may need a bit extra oomph here and there, an actor’s natural in-camera presence is best achieved by lighting them brilliantly with pretty natural light. However, that appearance is drastically altered if lit with only red and blue light since natural light does not have a significant portion of its spectrum eliminated.
The technique is also ingenious since it facilitates easier foreground/background separation by using only red/blue for the foreground and green for the backdrop—standard cameras, which ordinarily record such hues, record red, blue, and alpha instead. Due to not having to separate a full-spectrum input from a limited-spectrum essential backdrop, the resulting mattes are exceptionally accurate.
Of course, it looks like they merely replaced one problem with another: Compositing is now straightforward; however, fixing the green channel on subjects originally illuminated in magenta is challenging.
Since themes and compositions vary, this must be done methodically and adaptively; taking a “naive” linear approach would result in a washed-out, yellowish appearance. How can it be done mechanically? The help of AI is here!
The researchers used their own “rehearsal” takes of similarly lit “normal” situations to train a machine learning model. A convolutional neural network can quickly and intelligently restore the missing green channel by comparing full-spectrum image patches with magenta-illuminated ones.
Even while it’s possible to restore the color in the post (to the point where it’s “virtually indistinguishable” from the ground truth captured on camera), the problem of the awful lighting for the actors and set remains. Many actors already find working in front of a green screen weird; now, picture them doing so while light in a cruel, inhuman way.
The report proposes a solution in the form of “time-multiplexing” the lighting, which would involve repeatedly cycling the magenta/green lights on and off. To do so at the framerate at which most films and television are shot—24 times per second—is distracting (even dangerous). However, if the light is switched on and off quicker—144 times per second—it appears “nearly constant.”
However, this calls for intricate timing with the camera, which must record just when the scene is magenta. And they have to factor in motion gaps as well…
This, as you can see, is still in the testing phase. However, this novel technological solution to an ancient challenge in media production is intriguing in and of itself. This wouldn’t have been conceivable even five years ago, so it’s worth experimenting with; it may or may not catch on in the studio.
