Virtual production is an umbrella term for a variety of concepts. But more accurately what is referred to as virtual production is a virtual set or location displayed on a volume of LED panels linked together to form a large screen. Those images combined with camera tracking deliver an all-new production experience which can be creatively and financially more efficient.
There is no need to take an entire cast and crew to a remote location and there is no issue with filming permissions, crowd control, local laws, environmental impact and so on. With virtual production you bring the location to you. Other benefits include controlling the entire environment, including the weather and time of day. With VP you can spend the entire day shooting in sunset, whereas before you would have to shoot over multiple days.
The trouble with a green screen is the actors cannot see what they are interacting with. With virtual production the actors can directly see the environment they are in, leading to a better performance. And it’s not just the actors, the entire crew can see what the shot will look like, everyone from cinematographers being able to frame and light how they want, to hair and make-up who can see how something will look instantly and immediately. Everyone benefits.
The other benefit of virtual production is you control the lighting: the screen is not just an image to be viewed but it is also light source. Reflections of the virtual set on foreground surfaces and objects instantly add to the reality rather than having to be added (or removed) in post.
In addition, historical problems with fine detail in front of green/blue screen such as hair, transparent items or smoke are negated as there is no need to key around the elements. The blend is perfect because the background is, in a sense, really there as far as the camera is concerned.
Technically the main elements are camera, camera tracking, an LED screen and a game engine (typically Unreal) to manipulate content. DMX control of lighting is also increasingly important and can be driven directly by Unreal Engine. There are many other moving parts but these are the main ones.
Virtual production has brought about the creation of entire new departments, virtual art departments, asset teams and dedicated volume teams such as a tracking team, and an Unreal team. Typically, the emphasis of a production will lean more towards pre-visualisation rather than post-production. New workflow changes could be colour and chroma effects, performance capture and also editorial changes fed back to the screen after viewing dailies etc. Metadata plays an even bigger role as gyroscopic data and precise lens information is essential to ensure accurate distortion and depth of field characteristics are replicated.
Lens choice is quite a bit more restrictive; you can only use certain focal lengths and you have to be very selective about distance to screen and subject to avoid moiré and aliasing issues. You must carefully manage your depth of field, too shallow and the screen becomes too defocussed making it no longer possible to see the depth rendered in the virtual 3D set. This makes the background appear as a single flat plane, rather than a scene with depth. Framerates are more limited not only due to processing power required to run the game engine at high rates, but also refresh rates of the LED panels.
The main things to look for in a panel is the size between LED emitters (known as pixel pitch), colour gamut, refresh rate, brightness, contrast and reflectivity. Low contrast screens make it very difficult to match foreground and background black/white points. Low brightness screens can’t replicate practical or real-world light sources adequately and highly reflective screens are problematic as foreground lighting washes out the background blacks. In addition, if shooting High Frame Rate (HFR) content then bright screens are essential to provide enough light.
Pixel pitch refers to the distance between each individual LED. The smaller the pixel pitch the more resolution a screen will have but will also require more rendering power. Distance to screen plays a huge role, for closer distance (either for close ups or using small screens) a a small pixel pitch is needed as on a larger pitch the individual LED’s can be seen.
No. While many virtual production studios will have a very large screen designed to completely fill the entire background of a scene, smaller screens also have a role to play. For product shots or more intimate work a smaller volume can be used. Smaller screens can be used to fill in small parts of a scene. For example, to provide the view that would be seen out of a window. A smaller screen with a smaller pixel pitch will support you in getting closer to the screen without any negative effects like moiré or aliasing.
Curved screens require less studio space than flat. Generally, a curved screen needs less floor space to accommodate the same fields of view as a flat screen, so for the same or similar shots a curved screen installation will be smaller than a flat screen. Additionally, the modular construction of virtually every screen system currently in use means they can configured as either curved or flat and the cost is the same for either. Often immersion will be better with a curved screen as straight lines wrap around the scene in a way that is closer to how we perceive in reality and provides better viewing angles for cameras.
A top screen is typically only used for light and reflections and is rarely shot at directly. Often, for this reason it is a wider pixel pitch which with typically brighter LED’s. However, the importance of a ceiling should not be underestimated as the light and images it projects into the foreground can be a vital part of the overall composition.
As well as, or as an alternative to a ceiling screen it is often possible to control the foreground lights from the computer render farm or Unreal Engine. This allows you to include lighting effects such as fire, the sun passing through trees or buildings, lightning etc in the foreground to help blend the foreground with the virtual background.
More often than not you will still want a foreground set of some type. The foreground set might be very simple, perhaps just a few props, or may be the full contents of a complete room or structure. A foreground set gives the cast a 3-dimensional space to move in and objects to interact with. Foreground objects add depth making the entire scene more convincing. Depending on what you are shooting you may need some foreground objects to disguise the join between the floor and the screen.
Yes, you can. The ability to move the camera and have the background move in sympathy with your camera moves is one of the things that really separates virtual production from pre shot plates. You can use a dolly, track, jib etc just as you would on a normal shoot provided you don’t pan or tilt beyond the edges of the screen.
Tracking is incredibly important. For the camera to interact with the virtual world, you must feed the game engine or playback software with precise camera positional data. There are two types of tracking, inside out and outside in. Inside-Out uses a tracking camera positioned (usually) on top of your shooting camera that looks for key features/data points to provide positional data. Outside-In uses cameras based around the set pointing towards the camera which will have a tracking node on the top. There are a variety of methods from IR/star point tracking to feature tracking.
The camera must be synchronised to the screen to avoid issues such as rolling light/dark bands or flicker, this is normally achieved via Genlock. Additionally, it is normally beneficial to use a camera with a fast sensor readout speed to minimise skew issues.
xR (extended reality) is a mixture of the real and the virtual, most often by the means of set extension. You most often see this in live broadcast industry such as News and Sports. The presenter would be on a smaller sized screen and through set extension and mixed and augmented reality, would appear to be in a much larger area, often interacting with the screen showing graphics etc.
These two terms are often confused, while virtual production is strictly the use of a computer or game engine generated virtual displayed on the LED wall, the use of pre shot background plates shot on the LED wall is a modern alternative to traditional rear projection. Typically, this would not necessitate camera tracking, but still benefits from cast and crew being able to see the environment along with truer to life brightness and contrast. A common application is driving scenes.
The biggest value of virtual production comes from bringing what is typically only seen in the post production phase right through to the pre-production and production phase. You can imagine any location, control the time of day, the seasons and the weather, immediately display CG assets and more. Everything from scene planning, actor performance, creative control and more all benefit from this creative and immersive technology.