Back in 2017, Korey Pereira undertook the audio post-production for his first Virtual Reality project with 360 Video. He settled on Ambeo VR from Sennheiser and Facebook’s Spatial Audio Workstation for mixing in Pro Tools. In this article, Korey brings us up to date with his current VR workflow. Find out what has stayed and what he does differently 3.5 years on.

Back in 2017, I was working on my first ambisonic project and exploring the current options available at the time. I ultimately settled on the Ambeo VR from Sennheiser for audio capture and the Spatial Audio Workstation from FB360 (formerly Two Big Ears) for mixing in Pro Tools. You can read about my experiences in the article here on Production Expert Developing a 3D Audio Workflow For Virtual Reality In Pro Tools.

What Has Changed Since 2017?

I am happy to report that three and a half years down the road, I am still using the same core technology, but let’s start with what has changed in the realm of 360 sound. In 2017 I was using my Sound Devices 788T-SSD as my recorder. Since then, I have undertaken a LOT of ambisonic recordings. To help streamline my setup – especially while travelling – I picked up a Sound Devices MixPre 6 specifically for use with the Sennheiser Ambeo VR. When combined with the MixPro bag from K-Tek, it makes for a compact solution for capturing sound in the ambisonic format.

If you are new to the world of ambisonics, then check out this great blog post from Waves that covers all the basics: Ambisonics Explained: A Guide for Sound Engineers, this really helped me when I was getting started.

Nugen Audio Halo Upmix Added

As far as mixing for 360 video, the Spatial Audio Workstation, part of the Facebook 360 package, that comes with Pro Tools Ultimate is still my platform of choice for creating ambisonic mixes. I have since added the Halo Upmix plug-in from Nugen Audio to my workflow for up-mixing music and some stereo ambiences into the ambisonic format. Halo is already a regular in my 5.1 and ATMOS workflows but has proven just as useful for 360 video since they added support for ambisonic formats.

Monitoring And Quality Control Evolved

For monitoring and quality control, the equipment has evolved drastically. When I started on my first project back in 2017, I was using an Oculus DK2 Headset. It was a clunky setup but had the benefit of working natively with our Mac Pro, enabling me to mix and monitor video on a single system! Setting it up involved connecting a physical cable with HDMI and USB ports from the headset to the computer as well as mounting a head-tracking sensor via an additional USB located at the front of the room.

The video quality was horrendous. The ‘screen-door effect’ when viewing videos was laughably bad compared to what is available today.

Fast forward to 2020. Last year I worked on a fairly large project creating sound design and ambisonic mixes for 39 one-minute 360 videos as part of a video gallery in the recently released video game, Medal of Honor: Above and Beyond.

New Computer

For spatialization (setting the direction and distance from the viewer), I am still using the Spatial Audio Workstation from Facebook and am also still running Pro Tools on a Mac. I have since upgraded (after a system failure of my good-ol’ 2009 Mac Pro) to a 2020 Mac mini. If you want to read more about this upgrade, check out my recent article: Switching From Apple Mac Pro 2009 To Mac Mini 2020 For Dolby Atmos Post Production.

New VR Headset

The most significant upgrade relates to how I view the 360 video. For this, I upgraded our Oculus DK2 to an Oculus Quest.

The video resolution is significantly better, going from “1080x960” to “1600x1440” pixels per eye on the Quest. The video refresh rate is also much better, going from 30fps to 72fps, making for a much smoother viewing experience. The Quest comes with internal head-tracking sensors, internal speakers, hand controllers, and the ability to operate with (or without) a computer! The wired connection has evolved from an HDMI+USB umbilical cable to a single USB-C cable.

I have also moved to a dedicated PC for running our VR headset. While building a computer can save some money, I opted for a pre-built Dell G5 “gaming” computer. To guarantee wired support for the Oculus Quest via the ‘Oculus Link’, which is still in beta, I made sure the computer would meet their requirements, which can be found on the Oculus support pages. 

I went with a Dell G5 computer with these specs…

• Intel Core i5-9400 (6-Core/6-Thread 9M Cache,4.1GHz)

• NVIDIA GeForce GTX 1660 Ti 6GB GDDR6

• 256GB M.2 PCIe NVMe Solid State Drive + 1TB 7200 rpm 3.5" SATA Hard Drive

• 32GB DDR4 2666MHz

Oculus Cabling

While there are a number of off-brand USB cables available on Amazon, I opted for the official fiber optic Oculus Link cable. Although $80 is a bit steep for a cable, when you are working professionally, the last thing you want is a cheap cable taking your rig down.

Hardware And Software Allocation

Pro Tools and the Spatial Audio Workstation run on the Mac mini. I then load the 360 video using the FB360 Video Player on the VR PC in “slave” mode and select the Mac as the timecode source for the player. With this configuration, when I press play in Pro Tools on the Mac and the Quest connected to the VR PC starts playing back in sync. When I move my head around in VR on the Quest, it sends the head-tracking information back to Pro Tools and plays back the audio based on which way I am looking on the headset. Pretty cool!

Keep in mind that for best results, you should convert your videos to the Facebook recommended settings, which can be found in the Facebook 360 documentation. Although they recommend 30fps, I was able to get stable performance at 60fps for this last project.

While I typically leave the Oculus Quest tethered to the VR PC, I was able to get it working wirelessly! Once we are able to host clients back in the studio, this will definitely come in handy. To make this work, I purchased and downloaded the Virtual Desktop app for the Quest. I then used SideQuest Virtual Desktop Patch to allow the same setup to work wirelessly to the Quest. I was then able to view the video in sync with Pro Tools on the Quest with head-tracking being fed back to Pro Tools the same as my wired solution!

The last ‘trick’ to my setup involved putting a small piece of painter’s tape over the head sensor on the Quest. Normally when you take the headset off, the player would lose connection to the headset. By tricking the Quest into thinking it was still on my head, it continues to play video in the headset as well as the TV in the room and continue to feed head-tracking back to Pro Tools while not on my head. This lets me turn the headset on the desk and hear the perspective shift in my ears.

Monitoring The 360 Audio Mix

So how do I monitor a mix in 360? My go-to headphones for VR/360 are my Sennheiser HD25. They sound great and are easy to wear over the Quest. I wanted to be able to easily audition how my mixes would sound on an Oculus Rift S’ internal speakers, so I purchased a replacement Rift S headband from Oculus and custom-wired it to my trusty Presonus HP4 headphone amp. This lets me quickly switch between the two while working in Pro Tools.

Once I have all the sounds in place and am happy with how the mix sounds on both the HD25 and the Rift S headband, I use the FB360 Video Encoder app to mux (or combine) the mix to an MP4. I have found that a 3D 3840x3840 60fps video will not playback smoothly on the Quest with an ambisonic mix, however, I found that creating a lower resolution (2048x2048) video at 60fps plays back smoothly on the Quest in the Oculus Gallery app.

This also gives me the option to check the mix on the Quest’s built-in speakers, a pair of cheap Amavasion earbuds as well as the HD-25.

Once you have the mix sounding the way you want it, the FB360 Video Encoder gives you an array of export options both video and audio-only.

3 Years On

While three years ago VR felt like a fad, it has been gaining some major traction. Especially with the standalone Oculus Quest 2 being released at an all-time low entry-level price of $299, it is easier (and cheaper) than ever to get started in VR. 

A number of game developers are now releasing AAA-quality games exclusively for VR. Companies like Netflix and Amazon have released apps that allow users to sit in a virtual movie theatre and watch their favorite movies or tv shows on a big screen.

So that is where I am at now. I’ll likely be back in another three years to see where the next chapter of content creation for 360 and VR takes me!