Shooting Plates and Virtual Production
November 10, 2022
In the last few years, the development of very large, very high-quality LED and OLED video walls, such as Sony’s Crystal LED, when combined with advances in computer game rendering techniques has led to the emergence of a new film and video production process known as Virtual Production. This harnesses the graphics processing power of a computer game engine such as Unreal to generate and render, in real time, a virtual background that is displayed on a large video wall.
Actors then play out their roles in front of the video wall, where they can see and interact with the virtual background while motion trackers on the camera move the background 3 dimensionally. The final finished image is captured without the need for post production compositing.
But there are many ways that these Virtual Production studios with their large, high quality LED walls (often called an LED Volume) can be used, from self-illuminating green screen to rear projection.
In rear projection, the film projector displays a reversed image of a pre-recorded background (called the plate). This technique made its debut in 1930 with the movie Liliom which won a technical Oscar in 1931. Also known as background projection and process photography, the concept relies upon synchronizing camera and projector shutter speed.
Rear projection transformed
Many productions blend Virtual Production and rear projection to the extent that some people see them as the same thing, even if technically they’re quite distinct. Each has different benefits and both are gaining considerable traction as production companies seek ways to shoot smartly and to control costs.
Shooting background plates (film or video footage of a location) and then using those plates as a background for a scene is nothing new. The technique first started to be used in the 1930’s when cameras and projectors that could be accurately synchronised were developed. Initially they projected onto the rear of a semi-transparent screen and the process became known as “rear projection”. It also sometimes comes under the wider category of shots called “process shots” which covers many different types of techniques where separate background and foreground elements are combined.
A common use of rear projection is for driving shots where the actors sit in a car placed in front of the projection screen to give the impression of driving along a road. But it’s always been a challenge creating a background image with enough sharpness, brightness and contrast to deliver a truly convincing result.
In the 1960’s as science fiction films became more popular, new techniques using projectors that projected on to the front of screens helped to improve the quality levels and front projection was used in films such as 2001 and for many car chases in the Bond films.
Front projection is a refinement of rear projection where background imagery displays background imagery being projected onto both the performer / car and a retroreflective background screen. Front projection was invented in 1955 and first used in movies such as Matango (1963) and 2001: A Space Odyssey (1968).
Chroma key compositing relies upon post-production to combine two video streams. Typically, a green screen background is made transparent in post and substituted with a separately filmed background. An NBC TV broadcast first used the ‘Chroma-Kay’ technique in 1957 and Petro Vlahos’s refinements won him an Academy Award in 1964. More flexible than rear projection, it became a staple of cinematic special effects. Star Wars Episodes 1-3 (1999 – 2005) made extensive use of post-production compositing with state-of-the-art digital sets and characters.
New technology, new realism
Fast forward to today where modern LED video walls can approach real world brightness and contrast levels and it becomes much easier to blend the background scene to your foreground cast and objects. Once again, shooting background plates and displaying the plates as the background for your shot or scene is back in vogue as it can make impossible shots possible or offer a cost-effective way to create a true to life impression of being in a specific location.
Using an LED Wall for the background doesn’t eliminate the need to light the foreground but the light from the screen can be allowed to spill into the foreground adding moving and changing light patterns such as true to life reflections on glass and shiny car bodywork. All of this makes the final shot appear more convincing than before. This added level of realism makes the use of LED walls and video plates far more attractive for a wider range of shots than earlier rear production techniques.
A LED Volume Wall can display pre-shot video footage in a modernised alternative to rear projection with brightness and contrast levels that approach real life. It has many of the same benefits as ‘true’ virtual production; giving cast and crew an immediate sense of place while reducing post-production compared to chroma-key technology. Pre-shot or pre-rendered footage reduces spontaneity in camera movement but shows such as The Mandalorian (2019-) often switch between techniques as needed for a specific shot.
Shooting plates
As more productions are now delivered in 4K and HDR, in order to provide a suitably high resolution image for the LED wall it is important that you shoot your plates at the highest possible quality. Curved LED walls allow the background to wrap further around the foreground making camera moves that would have been impossible with flat rear projection possible. So, as well as shooting your plates at a very high resolution you will often need to shoot a very wide view of the background.
Additionally, a tracking device can be fitted to the studio camera so that your background plate can be moved and reframed to match the way the camera is moved. But it should be noted that unlike a true Virtual production, while the observable area of the background plate can be moved in sympathy with the camera’s movements, the perspective and three-dimensional aspects of the background plate will remain the same as when it was shot.
Sony’s VENICE 2 camera is an excellent choice for working with LED Volumes. When filming an LED wall, whether it’s a virtual set or real-world plates, it is essential that the camera has a stable Genlock system. In addition, a high refresh rate is desirable as this minimises screen refresh or skew issues, especially important if the background is in motion such as with a moving vehicle shot. These same VENICE benefits extend to capturing plates for rear projection and one recent test shoot put this to the test.
This virtual production set-up features a Sony Crystal LED screen displaying a 3D background generated on a VFX workstation and output to screen via a ZRCT-300 Display Controller. The workstation also synchronises lighting to match the virtual scene, while the VENICE 2 camera has a tracking system continuously relaying XYZ positioning, direction of movement, lens information and more. Virtual production originated in part with Rogue One in 2019 and has since been intensively developed with ever greater refinements.
Future developments
As more studios with LED Volumes are built, the use of plates for rear projection style production will almost certainly increase. Plates can deliver true to life images quickly and easily and for some productions it will be faster, simpler, and cheaper to use plates rather than a Virtual set.
When you have an extremely high-quality video wall such as Sony’s Crystal LED B-Series and top quality, very high-resolution plates as may be shot with VENICE 2, the seamless blending of the foreground and background becomes easier. Rear projection allows you to bring your locations to your cast and crew, saving on travel and minimising the risk of weather disruption. It’s quick, easy and a convincing alternative to true Virtual Production.
The industry expects a rapid resurgence in the use of background plates, very often mixed with Virtual Production as each process has distinct benefits but equally each can complement the other. It’s simply a matter of understanding the subtle differences between both and then choosing the method that best fits your needs and budget.
