audio tracking for Live VR – Orange Lab
As innovation hubs where new technologies are developed, Orange Labs work to improve telecommunications services and digital experiences. It is on the Orange Atalante site in Rennes that a project aimed at improving the live virtual reality (VR) experience was undertaken.
The objective is to be able to track the artists as they move for spatialized audio mixing in real time and to illuminate them thanks to automated tracking. This project is particularly relevant for live music, sports and theater applications in VR, as well as television and podcasts in Dolby Atmos, Binaural or Ambisonic formats.
Technologies used
- Naostage’s K SYSTEM tracking system: K SYSTEM allows artists to be followed without requiring tags on the performers or in their pockets. Tracking is performed using cameras only.
- LSpat Revolution software from FLUX::: Used for real-time spatialization of singers.
- Equipment from Kariba Productions and Virtual Audio: The performance was filmed in 12K VR and recorded with high-definition audio quality.
The project involved artists June Caravel and Gwendolyne Coronado, whose positions were sent to Spat Revolution for real-time spatialization and to the lighting system for automated tracking. The main stages were as follows:
1. Studio Capture:
- K SYSTEM was used in the studio at a height of 3.5 m and at least 3 m from the sound sources.
- Data was transmitted via OSC to Spat Revolution.
- The performance was captured using 360° VR cameras for ambisonic mixing.
2. Production and Recording:
- The performance “My Star” was filmed in 12K VR by Éric Vidal of Kariba Productions.
-The audio was recorded by Eric Munch of Virtuel Audio.
Results
The result was clear: effective tracking and seamless integration.
Indeed, as expected, the tracking of the artists allowed real-time audio spatialization without tags and without any hassles from installation to the final result. Additionally, Naostage's OSC worked perfectly with Spat Revolution, ensuring optimal coordination between sound and light.
& Acknowledgements
The project carried out at Orange Lab in Rennes demonstrates the effectiveness and innovative potential of markerless tracking for live VR performances. Through the integration of several technologies and the collaboration of multiple experts, the team was able to create an immersive, real-time experience, opening up new possibilities for live performances and VR recordings.
Many thanks to Marc Emerit, Christophe Daguet and Arnaud Lefort from Orange Lab; Éric Vidal from Kariba Productions; Alexis Reymond and Nathan Van de Hel from Naostage; Nicolas Erard from FLUX::; and artists June Caravel, Gwendolyne Coronado and Vincent Thevenot for their participation and commitment to the project.
