One of the biggest issues Project Heartbeat has faced is the Godot audio stack, while sufficient and good enough for most games it really isn't suited for use in rhythm games.
For this reason I spent this week working on a new audio engine specifically for Project Heartbeat, it's called the Shinobu engine. Keep reading to learn how you can test it and why it's very important for the future of PH!
This new audio engine is based uses the excellent miniaudio sound library. If you need a good library for all your audio needs miniaudio is the king, it's beautiful.
TL;DR: What it means for you
- Lower latency
- Better performance
- No more desyncs!
- More smooth note movement
- Ability to adjust your buffer size
The technical side
One of the most important features Shinobu brings to the table is the ability to select your own internal buffer size.
Audio just like video, has frames, these frames are composed of instaneous samples for all possible channels, unlike video however the frequency is much higher, while a typical video might have 30 frames per second a typical audio file might do 44100 frames per second, which is quite a lot..
(In case you are wondering the reason why it's ~40 kHz is because of Shannon-Nyquist telling us that "If a system uniformly samples an analog signal at a rate that exceeds the signal’s highest frequency by at least a factor of two, the original analog signal can be perfectly recovered from the discrete values produced by sampling." and as human hearing caps at around 20k kHz it is enough for us).
This means that there has to be an intermediary buffer that gets filled with a bunch of frames, this is the main source of audio latency, as telling the engine to play say a note SFX as soon as possible might take a bit until the OS asks for samples again, reducing this buffer is important to reduce latency, however this must be done with care as CPU usage goes up the lower the buffer size is (for example, at 10 ms the mixing happens 100 times a second).
Timing is then done by using thea mount of samples consumed by the audio system every time it asks for it.
Another issue is your operating system, sometimes your audio driver might not allow smaller buffer sizes, Shinobu will try the lowest possible buffer size the operating system gives it (but always bigger than the buffer size you set in audio->buffer size).
As the audio buffer might sometimes be higher than the frame time i.e it might be >16 ms worth of samples timing from that might be jittery, this is why it now uses an internal high precision clock to interpolate the time between updates. This is an alternative to just lowering Shinobu's internal buffer size which increases CPU usage.
This buffer size setting might actually be unnecessary, I am not sure: we will have to test over the following months, but if it works we might just target the lowest latency the OS gives us and match the internal buffer to that and interpolate away the rest.
How to test shinobu
Go to your Steam library, right click the game's cover art/name and select properties, click on betas and enter the super duper secret password MARINSTHIGHS, then click the blue button to opt into the bet branch! If you want to return to the normal game just set it back to "None".
What's left to do
The only major thing that isn't implemented in Shinobu is the spectrum analyzer the visualizer used, I will try to implement it using Godot's spectrum analyzer as a reference.
Source code
As it's tradition, shinobu's source code is available in our modified version of godot as a module:
https://github.com/eirteam/godot/tree/0.14/modules/shinobu