Getting Started: Virtual Production

1. The Journey to Virtual Production

The transition to creative technologies, particularly virtual production, has been driven by a desire for more immersive and engaging experiences. For instance, the rise of Virtual Reality (VR) in 2016 allowed creators to merge it with motion capture, offering a more dynamic and interactive medium. This shift was not just about adopting new technologies but also about leveraging existing knowledge. Market research, for example, provided a foundation for understanding both qualitative and quantitative aspects of audience engagement, allowing for real-time adjustments based on feedback.

2. Understanding Virtual Production

Virtual production, though a buzzword in recent years, has been around for over two decades. Initially, there was a misconception that virtual production was solely about making movies using VR headsets. However, it's much more than that. Virtual production has evolved to include in-camera VFX, which is essentially the process of integrating visual effects in real-time during filming. This method has been in use for over a century, with techniques like matte paintings and miniatures. Modern iterations include the use of LED walls and real-time game engines to create immersive environments.

3. Pioneers in the Field

Several projects have marked significant milestones in the evolution of virtual production. The TV series "The Mandalorian" stands out as a game-changer, especially with its use of game engines for real-time rendering. Prior to this, movies like "Gravity" and "Oblivion" utilized LED walls to create realistic environments, eliminating the need for post-production compositing.

4. The Future of Virtual Production

While sci-fi genres and period pieces might seem like the obvious choices for virtual production, given their need for unique and often inaccessible locations, the potential applications are vast. As more artists and innovators experiment with the technology, we might see genres like rom-coms being reimagined in virtual spaces. The key is to push boundaries and explore new narrative possibilities.

In conclusion, creative technologies, with virtual production at the forefront, are reshaping the entertainment landscape. As tools and techniques continue to evolve, the possibilities are endless, promising a future where stories are not just told but experienced.

Virtual Production for Dummies

Virtual production is the big, bad monster of film production: desirable, but terrifying and complicated. Always being dubbed the “future of film”, but never being taught to young creatives and filmmakers has made virtual production seem like a superpower. But, if I, a 20-something year old with no ties to film production at all, can learn the ins and outs of “the future of film”, so can you.

To make what is sure to be a confusing topic more digestible, let’s break virtual production up into three parts: what it is, what it does, why it’s important.

What it is:

What is virtual production? Most simply, virtual production is the real-time rendering of scenes and environments, mixed with physical elements like actors and props, to create immersive and adaptable sets and shoots. In essence, virtual production is comparable to shooting a movie in a real-time game engine.

WLab films a short video using real-time rendering on the LED screen and an actor.

Now, what does it mean to be rendered in real-time? To be rendered in real-time simply refers to the constantly generating graphics of the scene. Take any first person video game; as you move and and interact with the game, the surrounding scenery changes and adapts to your movements and actions. Virtual production works in a very similar way, but instead of being in a game, you are in front of a large screen and your immediate movements determine what is being rendered on that screen, just like they would if you were a character directly in the game engine. That’s what virtual production is like; one large, dissected, customizable video game.

Of course, it’s all much more complicated than that, as it takes even practiced individuals considerable time to match lighting and shadows and prop placement to reflect what is being displayed on the screen. But, once all is set, the camera shouldn’t be able to differentiate between the tangible props, and the real-time renderings on the LED screen.

You may have already seen these techniques in practice and didn’t even know. Are you a fan of the Mandalorian? What about movies like Avatar: The Way of Water, Thor: Love and Thunder, or The Batman? They all use virtual production technologies. Pretty cool, right? It gets cooler.

Because virtual production works off of what is essentially a game engine, there are infinite possibilities to what can be displayed and rendered. Want a beach with a plane crash and rocky cliffs? Sure! Nevermind, give me a dingy house with a turned on lamp and two large windows. Done. With knowledge of Unreal Engine, the software used to create these virtual settings, anything can be created, changed, or destroyed on the LED wall. You have complete control.

What it does:

Now that we have established that virtual production is, let’s discuss what it does, what it is used for. How, you may ask, is this virtual production stuff any different from using a green screen? Or plopping in CGI material in post-production. In function, it’s not too different. The end goals are both the same: create scenery and visually enticing backgrounds or settings for a film. But, virtual production is the simpler, and quicker alternative to classic green screen technology.

In films of the past, CGI and post-production work was developed separately from live-action scenes and inserted into the scene after-the-fact. This method worked for decades and has given us beloved films. But, the separation of live-action footage and visual effects is time consuming and adds significant time and resources onto post-production.

Additionally, with CGI being rendered independently of live scenes, there’s more room for error and less flexibility in editing and changing these effects. In virtual production scenarios, a director could completely change a background or remove a significant visual effect in seconds, something that would take months to alter in CGI post-production.

NYU Tandon @the Yard combines real-time virtual production technology and motion capture.

The creative freedom and limitless possibilities of virtual production lend themselves usefully to creating dynamic and immersive environments and sets for media of all kinds. On WLab’s own LED wall, we have filmed short films, music videos, experimental media, and TikTok, just to name a few.

That brings us to my last point…

Why is it important?

Virtual production isn’t being heralded as ‘the future of film’ just for shits and giggles. No, it really is changing the landscape of filmmaking and film production.

The extensive creative ability and the efficient post-production capabilities of VP are making it an important film technology.

The combination of the quick post-production turnover time, and ability to make quick, even drastic, changes mid-scene would cut down on resources needed to produce high quality, visually complex films.

Directors no longer need to scout for locations, as they can make up any environment they want and project it into the wall. This is more environmentally friendly as well, as large crew sets are not disrupting natural environments. In addition, filmmakers can make on-the-fly decisions, utilizing virtual productions’ flexibility and creativity to the fullest extent to change and experiment with different virtual elements.

Not to mention the cost efficiency; without teams of post-production specialists, productions save time and money and still end with a visually stunning and immersive film experience.

But, above all, virtual production is an opportunity for innovation and experimentation within the film industry. The limitless possibilities and ever-developing technology ensures that filmmakers can make their visions, no matter how ostentatious, come to life before their very eyes.

Now that you know the basics of virtual production, would you consider trying it out? If you still want to learn more about what virtual production is, you can read the VP field guide here.

Maybe you would rather listen to industry professionals and creatives talk about their experiences in virtual production? I have that too.

Or, better yet, maybe you live in New York City and want to use a virtual production lab for a creative project of your own? Yeah, we can arrange that.

Article By Lea Filidore