Low Latency Audio in iOS

(This post was originally published on Medium.com)

I’m developing an iOS and Android musical instrument app and found that achieving low audio latency is not a simple thing.

Latency is simply delay. In my case, it’s the delay from an external event to a sound being played. Since sound travels at 1400m/s or (1.4m/ms), every ms of latency is the same as if the source of the sound was 1.4m away. So audio/music products should have as low a latency as possible.

Latency can be measured easily by making a sound recording with your phone, and then examining the resulting audio file with software such as Audacity. I made an example app that plays a sound when a button is pushed. When I make a recording of using this app and examining the waveform, you can see the initial blip from the button push preceding the sound being played. The latency is the highlighted section in the Audacity screenshot below.

I’m using Flutter, a cross-platform app development platform, to develop my app. One advantage of Flutter is that it has a large ecosystem of plugin packages that make it possible to add features, such as audio, with a few lines of code. Here’s an example:

import 'package:soundpool/soundpool.dart';

    Soundpool pool = Soundpool(streamType: StreamType.notification);

int soundId = await rootBundle.load("sounds/dices.m4a").
      then((ByteData soundData) {
return pool.load(soundData);
      });
int streamId = await pool.play(soundId);

I tried a number of Flutter audio packages with varying amounts of latency:

flutter_ogg_piano (Android only)
audio_players
soundpool

So it seems that while Flutter packages are easy to use, there isn’t one available that gives low playback latency on iOS. This led me to investigate some alternatives:

Writing native code: this negates Flutter’s cross-platform advantage and would require a separate codebase for iOS and Android. With a larger team and budget, this may be the way to go.
Using a 3rd party library. I investigated a couple of libraries:
– Superpowered: -no Flutter support, -high annual cost
– BASS: -no Flutter support, +one-time license fee scales with commercial use case, +well supported

It seemed that BASS is the preferred choice, so I investigated how to integrate an external library using Flutter’s FFI (foreign function interface) capabilities. BASS is provided with precompiled libraries for iOS and Android as well as a header file defining the API. I used a Flutter package called ffigen to generate Dart code for calls to functions in the library.

I made a Flutter package to help others use BASS in Flutter. See flutter_bass. It includes an example app that has a comparison of BASS to soundpool and flutter_ogg_piano. Here are the results of measuring the button-press-to-sound-play latency with the example app on both iOS and Android devices comparing BASS to soundpool and flutter_ogg_piano. Note that these latencies include the time for Flutter’s gesture recognizer to sense the button press.

Latency Comparison

Sound library/plugin,Device,Mean Latency (N=5),Latency Stdev (N=5)
BASS 2.4,iPhone 6,115ms,11.3ms
Soundpool,iPhone 6,238ms,12.3ms
BASS 2.4,Samsung Galaxy S7,153ms,5.6ms
flutter_ogg_piano,Samsung Galaxy S7,158ms,5.9ms

(Note: I have this table in a Github gist, but haven't figured out how to either embed a table using NimbleBuilder, or directly include a Github gist.)

Some things to note from these results:

BASS reduced the latency compared to soundpool by 123ms!
BASS is only slightly better than flutter_ogg_piano
The standard deviation of the latency are similar for each platform, suggesting that this is due to characteristics of each OS.

To achieve the Android BASS result above, some tweaking was required to get from the first working result of 267ms (yikes!) to the improved result of 153ms:

setting the frequency to 48000 (instead of 44100) reduced the mean latency by 68ms. With the original setting of 44100, this error message resulted: W/AudioTrack(29616): AUDIO_OUTPUT_FLAG_FAST denied by client; transfer 1, track 44100 Hz, output 48000 Hz.
With the modified setting of 48000, the message became: [ +6 ms] I/AudioTrack(29616): AUDIO_OUTPUT_FLAG_FAST successful; frameCount 480 -> 480
setting BASS_CONFIG_DEV_BUFFER to 10ms reduced the mean latency by 45ms.

Lessons Learned During Development

how to use ffigen to create a Flutter class from a .h file.
that Flutter root bundle assets can’t be accessed as Files
when using BASS, SetConfig won’t have any effect when called after Init.
for Android, set the BASS_CONFIG_DEV_BUFFER low e.g. 10ms
for Android, set the frequency to 48000 instead of 44100 for low latency

I look forward to your comments. You can learn more about what I’m developing at happyfeet-music.com.