We’ve recently started working with a new set of technologies called virtual production. Virtual production involves shooting video content in a volume set made up of large LED walls. Backgrounds and other content are displayed on those LED walls so that they can be captured in-camera, along with the cast and smaller set pieces. This methodology provides numerous benefits over shooting in front of green screens or going on-location, and it’s becoming increasingly viable for smaller productions.
To understand virtual production it’s helpful to think about its predecessors, especially chromakey. Chromakey or “green screen” technology is commonly used in VFX for background replacement, allowing the subjects to be placed into a different environment or replacing elements of a set with something else. In this pipeline the shoot is done first in front of a background that is a flat solid color, usually green. Then the VFX work is done in post-production, where VFX artists use compositing tools to replace the green with whatever imagery is desired by the filmmakers. Chromakey has been around for decades and is used by everyone from your local weather girl to kids making viral videos to blockbuster movies with 9-figure production budgets.
But chromakey has some issues. Chief among the issues is lighting – since the post-production elements are completely disengaged from the principle photography, lighting conditions may not match, causing the final scene to lack in realism. Even worse, having a larger green wall behind the set reflects a lot of green light back onto the set. This can make it difficult to cleanly composite out the green screen, especially if the foreground contains light colored or reflective elements like, say, a shiny tabletop. We call this green spill, and it can greatly complicate the post-production process for small productions, especially if the on-set production team was not well versed in shooting for chromakey. We’ve seen such mistakes double or triple a post-production budget or even require a reshoot.
Virtual production reverses this process and allows the production team to develop all of the VFX visuals in pre-production rather than post. Having all of this work done ahead of time allows the filmmakers to see how all of the elements come together as they are shooting on-set, and to adjust elements of the shoot accordingly. This is a more organic process that inherently reduces the risks of requiring costly reshoots that sometimes can be necessitated by disparate elements failing to come together as originally envisioned.
Shooting within a volume set that is displaying the final environment also allows the cast to better understand the scene and respond to elements of the scene that they would have to only imagine when shooting for chromakey. This can be especially useful when working with non-professional talent like a CEO giving a keynote or a musician making a music video. And since the actual environment is displayed around the volume set, the light spill and any reflections match up perfectly and no linger have to be corrected in post.
In some ways, shooting on a volume set is return to an even earlier moviemaking technique – rear projection. Developed as early as the 1930s and commonly used in early Hollywood films, this technique involved projecting the desired background content onto a screen behind the actors. You’ve almost certainly an old movie in which the characters were sitting in a car and the scenery passing by outside the car was projected in this way. The primary problem with rear projection is that the camera must remain fixed or else the illusion will be broken. Virtual production solves this problem by controlling the background and adjusting it in real-time as the camera is moved.
Virtual production is quite new – the techniques were first developed a few years ago for big Hollywood movies. Director Jon Favreau has done some of the most interesting high-production work using this technology, first on The Lion King and more recently in The Mandalorian. But while virtual production is very new, we’ve been working with many of the component technologies for many years – real-time graphics, animation, game engines, CGI and even large LED walls are all basic tools in our digital experiential toolbox. Virtual production simply combines these elements in new and innovative ways.
The current pandemic situation has greatly complicated shooting anything on-location, making virtual production drastically more appealing almost overnight. The production teams that pioneered these techniques had to build their own volume sets, at great expense. For big-budget movies where the alternatives would be large on-location shoots and multi-million dollar VFX budgets, these investments were worthwhile, but of course that’s not the case for most productions.
Happily there are now a handful of volume set facilities available to rent, opening this technology up to a broader range of productions. The continuing challenges of Covid-related safety protocols, and the other benefits of virtual production will likely drive further investment in such facilities. We’ve already seen this technology used for event content and we expect that trend to only accelerate. Salesforce used virtual production when creating case study videos for AT&T and Adidas as part of its content creation for Dreamforce in December.
Sony has already announced new LED products specifically targeted for use in volume sets, and we expect to see other manufacturers follow suit. In a time when audiences are going to be expecting ever-improving production values from all of the media they consume, marketers and other creators of digital media will need to add virtual production into their repertoire sooner or later.
Images: Stills from volume shoots for AT&T and Adidas by Salesforce