An interview with Dave Sorbara, Partner and Chief Creative Officer
1. Over the past 17 years, Grayson Matthews has specialized in music and sound for film, television and radio. Recently you started exploring new mediums, primarily around the creation of immersive content. What sparked your interest in immersive media and spatial audio?
Back in the late 1990s, we were in recording studios with giant consoles and tape machines that were very expensive – it was the only place where you could do professional recordings. Then the computer came along, and I remember being really fascinated with what you can do with a computer versus an analog console. I was interested in the ability of computers and technology to do a lot of things that analog technology in recording studios had been doing. That was a thread in our existence that always led us to the next transition in terms of the evolution of our company; it was really about technology.
As we progressed, I discovered spatial audio and virtual reality (VR) and what was happening behind the scenes with a lot of tech-heavy audio people, who are really more mathematicians than audio people. It’s a whole new way to think about sound and requires a new set of technologies. Suddenly, the possibilities of what you can do with sound began to open up again and I started getting the same feelings in my bones about sound. It was all about solving new problems and hearing things you never heard before. I felt reinvigorated in thinking that this was another wave of interest that can drive us forward over the next decade or so.
2. How is immersive content changing the way we tell stories and experience audio?
Immersive content is all about elevating the engagement with the individual audience through interactive stories. It’s that idea that you want to pull the audience into the story more so than having them sit there and watch the narrative. For example, my four-year-old son loves watching videos on YouTube, however his biggest obsession isn’t watching the video itself, it’s being able to change very quickly from video to video – he won’t watch the whole thing. At one point, I showed him a 360° video and once he realized you can pan the camera around and engage with a piece of content, static video in comparison was boring.
For younger people who have grown up in the age of the iPhone and the Internet, being engaged with content more so than being a passive player is really important. A friend of mine’s son, who is 12 years old and loves hockey, would never sit and watch hockey on television. He would rather play hockey in a hundred different forms such as hockey video games, street hockey, fantasy league hockey, etc. You can engage with the love of that sport, instead of passively watching a Maple Leafs game on TV.
3. What are some of the best immersive experiences you can think of to date so far? Who’s doing this well?
I think the best example of immersive content today is mixed reality experiences where you’re combining physical objects in a virtual free roam space. Toronto companies such as The Void, Globacore and one of our partners Seed Interactive do this very well. It is remarkable how you can transport your mind and body to a different planet via these different technologies. To feel like you’ve been transported instantly and have your brain fully believe it, is pretty amazing.
Mixed reality (MR) experiences are definitely cutting-edge – there’s not a lot of people doing them. We are in very early days, but there are some great VR games out there. One that I love playing is SUPERHOT, which is an amazing example of such a simple concept that only works inside of an immersive experience. Another huge one for me is music, because it is completely different in the context of immersive content. What I believe is the most important part about music these days is music performance. As computers advance in music production, live music will be crucial to its longevity in existence and in popularity. Capturing a music performance using 3D capture technology for audio and video and making someone feel like they are actually there is one of the best immersive experiences. There’s such a different emotional connection you make to the music in a performance inside of a headset or in a connected immersive piece of content.
4. What VR/AR gear would you recommend someone who is new to the immersive world?
At this moment in time, I would hesitate to buy anything VR related that connects to a large computer. Vive’s next iteration will be coming out this year and a lot of the gear is moving away from being tethered to computers. The next generation consists of stand-alone, no computer required, high frame-rate and high-resolution headsets that will be able to deliver what we’re delivering today on an Oculus connected to a powerful computer. The only difference is that there is ease of portability and use and it requires less setup.
In terms of augmented reality (AR), if you haven’t experienced what you’re capable of doing now with your AR tool kit on the iPhone, that’s the first place to dig into because there’s so much showing up there. Check out the IKEA Place AR app and you’ll get a feel for where this medium can go. It’s a great way to get people familiar with the ideas of what the possibilities are. Also, AR headsets will start making their way into the marketplace. Microsoft is coming out with a new iteration of their HoloLens that’ll be high-resolution. And the super secretive company, Magic Leap has finally shown off their first MR headset.
5. Tell us about your experience designing 360° soundscapes for SESQUI’S spatial dome?
I have to say that designing 360° soundscapes is interesting because you take all the learning you’ve done for stereo and surround and you think you can apply it, but then you realize it’s totally irrelevant and you have to start again from the beginning. The amazing thing about the process of starting fresh is going back to that creative instinctive of just using your ears, your feeling and your heart and ask yourself, “Does this sound cool? Is this interesting?” Surprisingly, you have this infinite number of creative possibilities in a 360° space. You’re creating environments and coming up with ideas on the spot and you want to experiment with all of it because you have no idea whether they’re going to work or whether it’s right or wrong. What ends up happening is you go back to that instinctive child in you who’s just playing to find something. In a way, that’s what a lot of the SESQUI project was really about.
The most eye-opening thing about creating sound for these type of experiences is that you have to throw convention out of the window and start from scratch.
Starting from scratch is a beautiful thing because it’s freeing and scary. You thought you knew something and suddenly, now you don’t. It was an amazing experience for that.
6. What can we expect to see on Grayson’s mobile application?
What we’re trying to do is push the possibilities of music and sound. This is an application where we can show people where it can go. We’re limited a lot of the time by the mediums with which we’re sharing content, so we’ve developed an app that will demonstrate the most cutting-edge version of what we’re trying to do. We aim to find a natural place where sound can go that resonates the way music does emotionally.
7. What are some challenges you have faced in building and creating immersive music experiences artificially?
The biggest challenge is creating the feeling of what were trying to achieve for the end users. It’s a technical challenge because a lot of the 3D spatial audio that we hear today is a synthesized mathematical equation and it makes sense and it works, but it’s missing that natural sound that connects with the individual. Although it localizes sound, it doesn’t sound real.
For example, HEAR 360’s 8ball microphone tries to capture that feeling versus an ambisonic microphone which captures a mathematical version of what it’s hearing. I think the next frontier is getting people to emotionally connect to sound like they do to a piece of music. We’ve had 75 years of working with technology and understanding it in order to get to a place to make people feel something when they hear it.
8. Why is it crucial to run tests before integrating 360° audio into live-streams?
The tests are important because there’s no precedent for what you’re going to hear and what it’s going to sound like. Since everything is new, almost every project is an R&D experiment where you have to solve more problems. The technology we’re using is new and always changing so every time you deploy new gear you have to understand what you’re going to get out of that gear. Testing and prototyping is a huge part of the process because there’s so many variables. You need to do the R&D and testing to ensure you are going to get something usable when you are shooting.
9. What are your greatest findings in experimenting with immersive technologies?
The scary thing that I’ve learned is immersive technologies can transport human beings pretty easily. Realising how easy it is to take someone’s brain and convince them that they are doing something else, are somewhere else or feeling something, is pretty powerful. It has the ability to educate exponentially quicker than other methods of technology and it can make things feel more real than anything else. Recognizing how powerful that is, you know it’s here to stay, it just needs to be taken seriously.
10. Where do you think it’s all going?
We live in this experience economy, where experiences are the most important things to humans. The ability for new technologies to make people experience things they typically wouldn’t have access to or have not felt before is becoming more evident. Education is going to change drastically. For example, you can take a nine month technical training course and shrink it down to two weeks because of technology like this. Entertainment is going to play a large role in the immersive world. In the near future, you’ll be able to sit courtside and watch a basketball game while looking at six degrees of freedom cameras, which will give you the ability to walk and move through the space. AR has given us the ability to overlay information and content and integrate it into real space. For example, AR tape measures, which can be downloaded in the App store, are a game changer. Businesses that mass produce tape measures are now competing with AR technology because an app gives everyone the convenience of virtually carrying around a tape measure in their pocket. The AR tape measure apps are more powerful, efficient and have more features than a physical tape measure. These types of advancements are constantly going to happen in the near future.