As the technologies for VR and 360 video rapidly advance and become more accessible, media creators are realising the crucial role that sound plays in achieving realism. Sound designers are exploring this new frontier of 3D audio at the same time that tools for the medium are being developed and introduced. When everything is so new and constantly evolving, how does one learn where to start or decide where to invest time and experimentation?

To better understand this process, Andrew from Pro Sound Effects caught up with Korey Pereira who is a sound editor and mixer based in Austin, Texas - who has started to work and develop a workflow in the virtual reality and 360 audio world based around the Ambisonics format.

Andrew: Can you provide some background info about who you are, the work you’ve done, and what you’ve been up to lately?

Korey: I’m the Owner and Creative Director at Soularity Sound, an Austin-based post production company. We primarily work with Indie filmmakers, but also do some television and ad work. In addition to my work at Soularity, I also work as a sound editor and mixer at a few other Austin post facilities including Soundcrafter (my credits with them include Linklater’s Boyhood and Everybody Wants Some, as well as TV shows such as Shipping Wars and My 600lb Life).

Andrew: I reached out to you recently because you purchased our NYC Ambisonics library, and I was curious if you were using the library in any VR projects. You responded, “I have been loving the library!! Huge fan of the ambisonics format and what it can do...I am actually working with a filmmaker that is creating a lot of VR content. I am currently in the process of developing a workflow and would like to integrate ambisonic content if possible.” Who is the filmmaker you’re working with, and what kind of VR content are they working on?

In the coming months I plan to start creating audio content for VR with a local content creator, Deepak Chetty. Deepak is an amazing filmmaker. Over the years we have collaborated on a number of projects. Most recently I worked on his stereoscopic 3D Sci-Fi/Action epic film, Hard Reset, which won the 2016 "Best 3D Live Action Short" from the Advanced Imaging Society. The film premiered at the iTV Fest on October 8th where Deepak took home the “Best Director” award. 

I love Sci-Fi as a genre, because there really are no rules. It lets you really go for it as far as sound. Deepak has been shifting his creative focus towards 360 content and are hoping to start working together in that aspect in the near future.

Deepak: The content I am working on now is mostly non-fiction documentary based content in 360 - mainly environment capture with a throughline of audio storytelling that serves as the backbone of the piece. I am also looking forward to experimenting with fiction based narratives in the 360 space, especially with the use of spatial audio to enhance immersion for the viewer.

Andrew: Prior to meeting Deepak, did you have any experience working with VR / 3D audio?

Korey: No, this is my first venture into the world of VR Audio or 3D Audio. I have been mixing in surround for over a decade, but I am excited about the additional possibilities this format brings to the table.

Andrew: What have been the most helpful sources for studying up and figuring out a workflow?

Korey: The internet! There is such a wealth of information out there and you kind of just have to dive in. The benefit of 360 audio, being a relatively new format, is that people are still willing to talk openly about it. There are a number of sites and user forums and groups that have helped point me towards a workflow that will work for my needs including Designing Sound, AudioVR, Road to VR, and the VR Audio & Immersive Sound group.

Andrew: Was there anything particularly challenging to get used to or wrap your head around?

Korey: In a lot of ways designing audio for VR is not that different from traditional sound mixing for film. You start with a bed of ambiences and then place elements within a surround space. I guess the most challenging part of the transition is anticipating how the audience might hear your mix. If the viewer decides to watch a whole video facing the surrounds, how will it sound?

Andrew: Can you describe the workflow you’ve established so far? What are some decisions you’ve made regarding DAW, monitoring, software, plugins, tools, formats, order of operation... 

Korey: I am a Pro Tools guy, so my main goal was finding a solution that works seamlessly inside the Pro Tools environment. As I started looking into different options, the Two Big Ears Spatial Workstation really stood out to me as being the most intuitive and easiest platform to hit the ground running with. Even better news is that as a result of Two Big Ears joining Facebook, Spatial Workstation is now available for free from Facebook.

Basically you install a Pro Tools plugin that works as a 3D Audio engine and gives you a Pro Tools project with all the routing and tracks laid out for you. There are object-based tracks that allow you to place sounds within a 3D environment as well as ambience tracks that allow you to add stereo or ambisonic beds as a basis for your mix.

The coolest thing about this platform is that it includes a 3D video player that runs in sync with Pro Tools. There is a binaural preview pathway in the template that lets you hear the shift in perspective as you move the video around in the player. Pretty cool!

In September 2016 another audio workflow for VR in Pro Tools entered the market from the Dutch company, Audio Ease, with their 360pan Suite. Much like the Spatial Workstation, the suite includes 360pan which is an object-based panner that when placed on every audio track, allows you to pan individual items within the 360 degree field of view. The 360pan Suite also includes the 360monitor, which allows you to preview head tracking within Pro Tools. Where the 360pan suite really stands out is with their video overlay function. By loading a 360 video inside of Pro Tools, Audio Ease adds an overlay on top of the Pro Tools video window, letting you pan each track in real time, which is really useful. For the features it offers, it is relatively affordable. The suite does not come with its own template, but they have a quick video guide to get you up and going fairly easily.

Andrew: Are there any aspects that you’re still figuring out?

Korey: Delivery is still a bit up in the air. You may need to export in multiple formats to be able to upload to Facebook, YouTube, etc. I was glad to see that YouTube is supporting the ambisonic format for delivery, but I look forward to seeing workflows become more standardized across the board.

Andrew: Any areas in which you see the need for further development, and/or where the tech just isn’t there yet?

Korey: I think the biggest limitation with VR is the lack of affordable and easy to use 3D audio capture devices. I would love to see a super portable ambisonic rig that filmmakers can easily use in conjunction with shooting 360 video. Especially as media giants like YouTube are gravitating towards the ambisonic format for delivery, it would be great for them to be able to capture the actual space in the same format.

In January 2017, RØDE announced the VideoMic Soundfield - billed as the world’s first on-camera ambisonic, 360-degree surround sound microphone - though pricing and release dates have not yet been made public.

One new product that is already available, that I am really excited about, is the Sennheiser Ambeo VR Mic which is around $1650 or £1200. That’s a bit pricey for the most casual user once you factor in a 4-track recorder, but for the professional user that already has a 788T, the Ambeo VR mic offers a nice turnkey solution. I like that the mic looks a little less fragile than some of the other options on the market. It has a built-in windscreen/cage similar to what you would see on a live handheld microphone.  It also comes with a Rycote shockmount and cable to 4-xlr, which is nice.

PTE: Do check out the Zoom F4 and F8 which are so much more affordable than the Sound Devices 788T that Korey mentions. Watch our overview with team member Alan Sallabank shot at BVE 2017.

PTE: You can check out an interview with Ambeo user Tom Lowe and an overview video shot at BVE 2017 where Sennheiser's Andy Lillywhite gave Mike Thornton an overview of the Sennheiser Ambeo mic and some ambisonics workflows using plugins from NoiseMakers.

Andrew:  Some leading companies have recently selected ambisonics as the standard spatial audio format - can you talk a bit about how you utilize ambisonics for VR?

Korey: Yeah, I think this is a great decision. I like the “future proof” nature of the ambisonic format. Even in traditional film mixing, I like having the option to export to Stereo, 5.1 or 7.1 depending on the project. Until ambisonic becomes more standardised, I like that the Two Big Ears / FB 360 encoder allows you to export to .tbe B-Format (FuMa or ambiX/YouTube) as well as quad-binaural.

I am now a huge fan of the ambisonic format in general. The Pro Sound Effects NYC Ambisonics Library (and now Chicago & Tokyo as well) was my first experience using the format and I was totally blown away. In a traditional mixing environment it adds another level of depth to the backgrounds. I really look forward to being able to bring it to the VR format as well.