Small studio, big cinema
Smart Virtual Production with Unreal 5 for a better actor experience
Size does not matter (anymore)
As a small film studio, physical space and financial budgets are limited. Virtual production helps a LOT with these issues, but the studio is still limited by the size of its premises when it comes to shooting scenes where actors have to walk long distances.
Together with Filmstudio Basel, we built a smart IoT device and unreal blueprints that links the actor’s movement on a treadmill to the position of the camera in Unreal 5, creating the illusion that the actor is walking through a vast environment. By overcoming the studio's size limitations, we enabled Filmstudio Basel to create big cinema productions with a small budget.
What is Virtual Production?
Virtual Production is a technology in which LED panels or back-projections are being used as a backdrop for a film set, on which video or computer-generated imagery is being displayed in real-time.
This allows studios to shoot a scene in literally any environment without actually getting the entire crew there. This makes film production much more cost effective and sustainable. For an impressive example have a look Epic Games tour of “The Volume”, a Virtual Set that was used to film The Mandalorian.
A currentless treadmill
In other Projects, we already successfully partnered with Walkolution. The company produces motor-less treadmills that are the perfect base so that actors can walk naturally without actually changing their location. We were happy to bring this amazing product back for this project. The key feature of the treadmill is that it is purely muscle-powered, this way solving a major problem we saw early on: a traditional electronic treadmill would rob actors their self-determination in acting, as it runs at a fixed speed.
With the Walkolution treadmill, the actors can decide when to start and stop walking, or even walk/run at different speeds, increasing the actor experience tremendously.
Walkolution GmbH
It was a great pleasure to work with Centigrade again.I was very excited to utilize our product in such a creative and experimental environment and it gives our company the opportunity to further expand the use case of the walkolution.
The Challenge: Digitising the Treadmill
Of course, a muscle-powered treadmill would not be enough. The virtual scene rendered by Unreal 5 had to move in perfect sync with the actor... and this was where things got interesting. First, we built a sensor to measure the actor's walking speed, using a good old-fashioned Lego wheel attached to a rotary sensor that's driven by the Walkolution surface.
In order for Unreal 5 to be able to communicate with the treadmill, it needs input parameters that it can actually understand. The Walkolution is not digitised at all - which is the strong idea behind it - but we still needed digital input data for the game engine. For this, we connected the sensor to an Arduino Leonardo microcontroller. Now this is where it gets REALLY interesting!
A Treadmill as a Game Controller
Before you read on: This is for the nerds among us. The Unreal game engine needs to interpret the output of the rotary sensor as a movement input to reposition the virtual camera. To solve this, the microcontroller emulates an Xbox gamepad for Unreal 5. The velocity information from the treadmill is mapped to the gamepad's analogue sticks.
To make this clearer: Slow walking on the treadmill now corresponds to pressing the analogue stick of a game controller
slightly forward, while fast walking corresponds to pressing the stick all the way up.
With a bit of tweaking, we managed to link the treadmill's movement to the Unreal Engine's camera in real time.
Real-World and Virtual Camera Synchronization with nDisplay
One of the key components for this project was Unreal's nDisplay technology.
It’s support for virtual reality components allows us to synchronise cameras on set with virtual cameras in the engine.
Using motion tracking devices such as the HTC Vive or Oculus Rift, we can track the movement and orientation of the
real-world cameras and replicate that movement in the virtual environment. The result is a seamless integration of
real world and virtual footage, creating a more immersive and believable final product.
Prospects: Enhancing the Actor Experience
Freeing yourself from studio size limitations is on its own something small studios will look forward to. But the benefits and future possibilities of this project don’t stop here. In the traditional world, any visual effect that is supposed to go along with the action of an actor would need to be synchronized manually as a post-processing step. This can be a source of errors and quickly become expensive.In order to keep post-processing effort at a minimum, actors must be able to determine for themselves, when which visual effect is supposed to happen as they have the best feeling for the timing of their acting.
Getting back to our setup: what if, the actor could trigger events in the scene with his feet? An explosion or something suddenly flying above the actor's head: The technology we built in this project with Film Studio Basel is a step towards that direction, both in engine and the real world. We will keep you posted!
European Star Cinema
Working in a small studio may limit your technical options, but it doesn't stop you from coming up with great ideas. With Centigrade's creative approach, we were able to develop a solution to our space constraints and improve the experience for our actors. All this in just two days while having a lot of fun — hackathon style.
