Spatial audio might be the first lightweight augmented reality that many users will experience, helping to, um, visualize how “real” AR can get.
Today, AR is mostly experienced through our phones or bulky (and expensive) goggles like Hololens or Magic Leap.
We can hardly walk around the neighbourhood with a Hololens on. And a phone or iPad can deliver some pretty stunning experiences – but you’re still “separated” from that experience by the screen.
Until lightweight goggles come along, we can experience rich and immersive AR today – which will still come at the cost of a pair of AirPod Pros and an iOS device.
What Is Spatial Audio?
Spatial audio, unlike say “surround sound”, is audio that responds to the movement of your head.
Imagine you hear the sound of a dripping tap. You turn your head slightly to figure out where the sound is coming from. You’re able to ‘place’ the sound because you have two ears.
Your head has 6-degrees of freedom:
Spatial audio can react to these movements and recalibrate where a sound is “coming from”.
If done well, spatial audio can give the absolute illusion that sounds are in a particular point in space: a river on the right, the sound of birds in the trees up and to the left.
Apple and Spatial Audio
Apple announced spatial audio support for its AirPod Pros. It was positioned mostly as something to be used with other devices, like an Apple TV or iPad. A sort of ‘advanced surround sound’.
As The Verge reported:
“Additionally, Apple announced a new “Spatial Audio” feature, which will only be coming to its AirPods Pro earbuds. The new feature will offer 3D, surround sound-style audio on your AirPods Pro to replicate a “movie theater experience” and will constantly recalibrate based on the position of your head to whatever device you’re using.“
But behind the scenes, developers could also access APIs to “read” the motion of your head while wearing AirPod Pros.
Apple Patent and USDZ
Apple recently received a patent for spatial audio.
Without getting too geeky about it, the patent covers the “flow” from capturing audio to placing it in a simulated environment. (Apple loves the term Simulated Reality over Augmented Reality, a discussion point for another day):
“Embodiments of a file format for spatial audio as herein described enable augmented SR application developers to compose sounds for use in SR applications in which the audio data that is encoding a sound capable of being composed into a SR application is stored as an audio asset including asset metadata describing not only how the sound was encoded but also how a listener in a SR environment that has spatial audio rendering capability experiences the sound.”
Which cross-links to Apple’s proposed standards for USDZ.
In other words, Apple is working to create a production flow for spatial audio from capture through to “placement”.
Developers Play With 6DoF Tracking
We’re starting to see more experiments with the feature. Ukaton, for example, demonstrates how precise the tracking is of your head movements:
This validates how precise the data is that comes from your AirPods, and opens up some interesting use cases around head tracking alone.
Warren Moore ran a similar test. The GitHub repository lets you check it out for yourself, and see how you can use the data streams from your AirPods.
Spatial Audio Experiences
SpatialBliss has taken spatial audio and released an app. If you have AirPods Pro, you can download it and check it out:
In their case, the audio “surrounds you”. It creates a sense of realism because that audio is spatial. They provide a nice library of soundscapes, mostly focused on relaxation.
One of the interesting things about spatial audio is the possible cross-over to the ambisonics community. OK…I had no idea what that was either. But as I started to dig in, I discovered a massive community of people recording soundscapes.
The Soundfield library gives you a snapshot of the types of soundscapes that people capture.
But recording the sounds of the world can have a deeper meaning. Check out this astonishing video about Recording the Sounds of Extinction.
All of which points to some deeper possibilities for spatial audio: the ability to “play back” environments, to be transported to somewhere else (and somewhere “real”).
Augmented reality won’t just be about visuals. It will include audioscapes that transport you, guide you, direct you through a store, or just help you enjoy a movie at home or a moment of meditation.
It may well be the first lightweight augmented reality that we never see.