We've just released Steam Audio beta 2, and one of the improvements we've made is to the performance of bilinear filtering of HRTF data. This blog post describes this feature in more detail, with examples and benchmarks. For the full release notes for the beta 2 release, click here.

Wait, What Is This About Again?

In Steam Audio, when you attach a Phonon Source component to any Audio Source in Unity, the following options appear:

Under Direct Sound, there's a drop-down labeled HRTF Interpolation. There are two settings here: Nearest and Bilinear. This post explains what each of those settings does, and the performance vs. quality trade-off involved in selecting the best option for your sound source.

For those of you using the Steam Audio SDK directly, note that HRTF effects are applied using the iplApplyBinauralEffect function:


IPLAPI IPLvoid iplApplyBinauralEffect(IPLhandle effect,
IPLAudioBuffer inputAudio,
IPLVector3 direction,
IPLHrtfInterpolation interpolation,
IPLAudioBuffer outputAudio);


There are two possible values of the interpolation parameter: IPL_HRTFINTERPOLATION_NEAREST and IPL_HRTFINTERPOLATION_BILINEAR. These are the same thing as the Unity drop-down options mentioned above, and the performance vs. quality trade-offs discussed here apply as-is.

Querying HRTF Data

Here's how HRTF data is measured. A person (or a mannequin) sits in the middle of an anechoic room. A microphone is placed in each of their ears. A speaker is positioned at one of many pre-determined directions around the listener, and a special sound is played through it. The sound recorded by the microphones is (after some post-processing) the HRTF for that specific direction. The HRTFs for all the measurement directions together makes up the HRTF data set for the listener.

So HRTFs are measured only at some specific directions. But in a game, the source can move freely, and the direction from the listener to the source may be something other than one of the measurement directions. Given all the measured HRTFs, how can we estimate the HRTF for the source's actual direction? There are two common ways of doing this:

Nearest Neighbor
Just pick the measurement direction that's closest to the source's actual direction -- its nearest neighbor. If the measurements are regularly-spaced, this approach is super-fast. On the downside, as the source moves, the nearest neighbor will change, and you may be able to hear the sudden change in the HRTF.

Bilinear Interpolation
For regularly-spaced measurements, calculate a weighted average of the HRTFs at the four measurement directions that are closest to the source's actual direction. This is slower than the nearest neighbor approach, because extra calculations are needed, and care must be taken because the averaging involves complex numbers. With this approach, you're less likely to hear noticeable changes in the HRTF as the source moves around.

Comparing Nearest Neighbor and Bilinear Interpolation
For many kinds of sound, nearest neighbor works well in that sudden changes are not noticeable when the source moves around. For others, the changes are noticeable, and bilinear interpolation should be considered.

For example, here's white noise, with nearest neighbor, followed by bilinear interpolation:

https://youtu.be/KNtDOceuliE

And here's speech, with nearest neighbor, followed by bilinear interpolation:

https://youtu.be/65QwgoqPGoQ

In general, noise-like sounds, like machinery, helicopters, etc. benefit the most from bilinear interpolation. Sounds like speech and music tend to sound good enough with nearest neighbor.

Performance

To compare the CPU overhead of nearest neighbor and bilinear interpolation, we use a small benchmarking program that directly uses the Steam Audio SDK. We measured the performance of the iplApplyBinauralEffect function, which performs all of the computational work of HRTF-based binaural rendering, including either nearest neighbor or bilinear interpolation. To measure performance, we used the std::chrono::high_resolution_clock class, which is part of the C++11 standard library.

Terminology
Before discussing any further, let’s quickly recap some basic terms related to audio processing.

• Sampling Rate. The number of digital samples used to represent 1 second of audio. For example, a sampling rate of 48 kilohertz (48 kHz) means 48,000 sample values are used to represent 1 second of audio.
• Frame Size. The number of digital samples that are processed at the same time. This is usually set by your audio engine. Typical frame sizes are 1024 samples or 512 samples.
• Frame Time. The duration of audio represented by a single frame’s samples. This can be calculated as (audio frame time) = (audio frame size) / (sampling rate). For example, with a 1024-sample frame at 48 kHz, the audio frame time is about 21 milliseconds (ms).

Metric Measured
With our test program, we measured the Theoretical Max. Sources that can be achieved on a single CPU core. This is the maximum number of HRTF-based 3D audio sources that can be processed on a given device, with one CPU core maxed out. To calculate this metric, we measure the time taken for a single execution of iplApplyBinauralEffect, and use the following calculation: (max sources) = (audio frame time) / (process time for 1 source). We do this because all audio processing for a single frame must complete within the audio frame time, or else the user will hear artifacts.

Device Specs
Here are all the devices on which we measured the performance of Steam Audio:

Benchmark Results
Below are the benchmarking results for each of the devices we tested. All numbers are the theoretical max. number of sources that can be rendered by using 100% of a single CPU core:

Conclusion

In conclusion, bilinear filtering is a great way to ensure that moving binaural sources sound smooth as they move. While there is a performance hit relative to the default nearest-neighbor filtering, the absolute performance remains high on PC platforms. On the other hand, for many common kinds of sounds, nearest-neighbor sounds acceptable, so you don't need to spend CPU cycles on bilinear filtering. You should experiment with the various sounds you're creating and using in your game or VR experience, and enable bilinear filtering only on the sounds where the difference is noticeable.