Netflix, the popular streaming platform, is revolutionizing the process of compositing in filmmaking. Compositing involves placing actors in front of a background that doesn’t actually exist, and it has always been a challenging task. Traditionally, chroma keying was the simplest method used, where actors stand against a brightly colored background, typically blue or green, which can be easily replaced with different visuals. However, this method has its limitations, such as issues with transparent objects, fine details like hair, and objects that share a similar color with the background.
Netflix researchers have recently introduced a groundbreaking technique called “Magenta Green Screen,” which combines old and new methods to achieve seamless compositing. This technique involves lighting the actors with a garish magenta color against a bright green screen. Although this unconventional lighting setup may seem daunting to post-production artists, it simplifies the process of separating the foreground (actors) from the background.
Ordinarily, actors are lit with natural light to achieve a more authentic look. However, exclusively lighting them with red and blue light distorts their appearance, as normal light contains a wide spectrum of colors. Nevertheless, this technique ensures that the foreground is captured in red, blue, and alpha channels, producing highly accurate mattes without the artifacts commonly associated with full-spectrum inputs and limited-spectrum key backgrounds.
While this new approach simplifies the compositing process, restoring the green channel to the magenta-lit subjects presents a challenge. To address this, the Netflix team trained a machine learning model using their own training data. The model, a convolutional neural network, compares patches of the full-spectrum image with the magenta-lit ones, developing an intelligent process to quickly restore the missing green channel. The results are remarkably close to the ground truth captured in-camera.
Although the color restoration can be achieved successfully in post-production, the issue of actors and sets being lit in this unconventional manner remains. Many actors already find it unnatural to work in front of a green screen, and doing so under harsh and inhuman lighting exacerbates the challenge. A possible solution presented in the paper is “time-multiplexing” the lighting by alternating between the magenta and green lighting multiple times per second. However, this requires intricate synchronization with the camera and accounting for motion blur.
It is important to note that this new technique is still in the experimental phase. Nonetheless, it offers an intriguing solution to an age-old problem in media production, leveraging cutting-edge technology. Five years ago, this approach would have been unimaginable. Whether or not it becomes widely adopted is yet to be seen.
The whytry.ai article you just read is a brief synopsis; the original article can be found here: Read the Full Article…
