[MUSIC PLAYING]
DOM TURNER: My name is Dom.
I work in the
Android DevRel team,
and I'm here today to talk
to you about low latency
audio on Android.
So why is audio
latency important?
Well, there are a
whole raft of use cases
where the audio
latency is directly
proportional to the
user experience.
So this can be in games
where you tap on the screen
and you hear a sound,
particularly in rhythm games
like Guitar
Hero-style games where
you're tapping in response
to some rhythmic event
and you need to have
audible feedback as quickly
as possible-- like, the longer
the delay between tapping
on the screen and
hearing a sound,
the worse the user experience.
Also, in DJing apps, you
are tapping on the screen
and manipulating audio, and you
expect that audio manipulation
to happen in real time.
Karaoke, where you've got an
input, which is your voice.
And generally, you're listening
to a backing track and also
your own voice.
So if the delay between
you singing and hearing
your own voice is too
long, then it sounds awful.
Also, in VR, we
have objects which
make sound in a virtual world.
And if that sound
doesn't follow you around
as your head moves
in this environment,
then it kind of
distorts the reality.
And lastly, of course,
there's a whole load
of apps for live
performance-- synthesizers,
drum machines, looping apps.
Anything which you press
a key and make a sound,
you need low latency audio.
So with this in mind,
we built a library
to help developers build
these kind of apps.
It's called Oboe, and it's
available on GitHub now.
We just launched Version 1,
so it's production-ready,
able to be included
in your apps today.
And the way it works
is, under the hood,
it uses the AAudio API
on API 27 and above,
which is the new
high-performance, low latency
audio introduced in Oreo.
And on older devices,
it uses OpenSL ES.
So basically, the
idea is provide
a simple, easy-to-use API
which works across the widest
range of devices.
So rather than me
talk about this API,
I thought it'd be fun to try
and build a musical instrument
in 17 minutes and 23 seconds.
So before I start
that, I'm just going
to explain the
architecture so it kind of
makes sense when
I'm in the code.
So we have an app, and I'm
going to build the audio engine
part of this app.
This audio engine is going to
be responsible for creating
an audio stream.
That's provided by
the Oboe library.
And we're going to be
passing audio frames of data
into this audio stream.
Ultimately, this stream is
responsible for putting data
out of the audio device.
In this case, it will be
the 3 and 1/2 mil jack
on this Pixel XL phone.
Now, every time the audio
device needs more information,
it's going to give
us a callback.
So we get this callback loop
of, hey, I need more audio data,
and our audio engine is going to
be passing frames of audio data
into the audio stream.
For some form of
control, we're going
to monitor tap
events on the screen.
So when you tap down,
the sound will come on.
When you lift up,
sound will go off.
This worked about 50% of
the time in rehearsal,
so let's see what happens.
OK.
First problem, I need to log in.
Can you see my screen?
Fantastic.
OK.
So I'm just going
to run the app.
So I've kind of got a very
simple shell of an app.
It doesn't really do
anything at the moment,
but it just has a
few little shortcuts
that make it possible
for me to do this
in a very short amount of time.
So I will just run the
app on this Pixel XL.
And hopefully, you'll be able
to see that it does nothing.
So here we go.
Here's the app.
And when I tap on the
screen, nothing happens.
No sound comes out.
It's non-functional.
I just want you to
know that there's
no smoke and mirrors there.
It is genuinely live.
[LAUGHTER]
Thank you, Glenn.
OK.
So we're in our
main activity here,
and we're going to talk to
the Oboe library via JNI.
So I have a couple
of native methods
here which I'm going to use,
and I'll implement those methods
in a second.
So we're going to
create an audio engine,
and we'll start by
calling start engine.
So let's just jump
into our JNI code.
So this is in
native-lib.cpp here.
So I'm going to define
an audio engine up here--
just call it engine.
And then, I'm going
to call a method
on my engine called start.
Now, I've already created
the header and implementation
files--
just the blank files for
this audio engine class.
So I'll go ahead now
and write the class.
So AudioEngine, and I'm going
to have one method called start.
OK.
Now, I can use
Option-Enter to generate
the definition for this.
I'm in my implementation,
and I'm in the start method.
Before I can start
using the Oboe library,
I need to do two things.
Number one, I need to
include the Oboe header.
There we go.
And the other thing
I need to do is--
this kind of just makes
it easier for me-- which
is to use the Oboe namespace.
And this just avoids me having
to prefix all of the Oboe
objects with the word Oboe.
So in our start method, we
want to create an audio stream.
To do that, we use an
audio stream builder.
[? Call ?] [? up ?] [? b. ?]
And the builder allows us to set
properties on the stream.
So that's things like
the format of the stream.
Now, when I set
the format, there
are two choices I
can choose from--
either 16-bit
integers or floats.
I will use floats.
I can also set the
number of channels.
So that would be two for
stereo or one for mono.
And I can also set
properties which
inform the stream of my
latency requirements.
So the most important one
here is set performance mode.
And there are a number of
options, but the one I want
is, obviously, the
low latency one.
The second thing I can do is set
the sharing mode on the stream.
So let's just set that.
So I'm going to set this to
an exclusive sharing mode.
And what that means
is that I'm requesting
that the audio device give
me an exclusive stream.
That means that my
app's audio won't
be mixed with any other
audio on the system.
So if the device
supports it, I can
avoid having my audio
mixed with anything else
before it goes through
the speaker or headphones.
And by doing this, I can cut
a few milliseconds of latency
off the output.
So that's all I need to
do to open the stream.
So I can go ahead now
and call open stream.
This takes a reference to
an audio stream pointer.
Again, I can use
Option-Enter to create
a new field called stream.
So back in our header, it's
done this automatically for me.
OK.
Once the stream is open, there's
one final step I need to take,
which is to set the
buffer size on the stream.
So I can do this by doing
set buffer size in frames.
Now, to get the smallest
possible buffer size,
we have to interrogate the audio
device for the minimum amount
of data that it will read in one
operation-- so a discrete chunk
of audio data.
And we call this a burst.
So we want to obtain the burst
size from the audio device.
We can do this using stream
get frames per burst.
And that's the minimum
possible buffer size
that we can set
our stream to have.
But we actually don't
recommend that you
use this absolute minimum.
We recommend that you use
double this because it provides
a good protection
against underruns,
and it's a good trade-off
with latency and underrun.
OK.
So that's all I need to do to
create a low-latency stream.
So now, I can go ahead and
start the stream, which
will do nothing
because we haven't
found any way of putting
data into the stream.
So to get data into the
stream, we use a callback.
So I'm back in the
builder, and I'll
use the set callback method.
This wants to take an audio
stream callback object.
And to make this
simple, I'll just
use my this object, which means
that my audio engine needs
to implement this interface.
So we'll have it do this.
Now, I can use Control-O
to show me the methods
that I can override
in this interface.
The one I want is onAudioReady.
This is the method
that will be called
every time the audio
device needs more data.
So inside here, let's
just take a quick look
at what this method
signature is.
So onAudioReady is called.
It tells me the stream
that wants more data.
And it also gives me
a container array.
So this container array,
which is of type void star
because it can be either 16-bit
integers or floating point
samples, is something
that I'm going
to write my audio data into.
So I write into
it, and then that's
passed out to the audio device.
Lastly, we have num
frames, which tells me
the number of audio frames
which need to be populated
in this container array.
So I need an audio source.
And I'm going to cheat
a little bit here.
I have created an
oscillator in advance.
And let's just take
a quick look at it.
And it's just going to
generate a square wave here.
So that's a periodic signal
varying between two points
to create a square wave.
So let's now create
an oscillator.
Let me just do oscillator.
It's a templated object, so
I need to tell it what type.
And I'll just include
the oscillator header.
So now that I have
an oscillator,
I can do osc render--
OK.
So Android Studio is complaining
about this method signature.
That's why it's got
a red underline here.
I'm just going to
go ahead and build,
and that normally sorts it out.
Yeah, there we go.
OK.
Ignore the errors.
OK.
So on my oscillator, I
have a render audio method,
which is going to put the audio
frames from the oscillator
into this audio data array.
So first thing I need to do is
cast it to an array of floats--
audio data-- and pass
in the number of frames.
So the last thing I need
to do in onAudioReady
is return a result. And this
can be either to continue,
where the callbacks will
continue, or it can be stop
and the callbacks will stop.
So in my case, I'm
going to continue.
Right.
The final thing I want to
do is set some things up
on my oscillator.
So I'm going to set
the amplitude, which
is kind of the volume.
I'm also going to
set the frequency.
Set that at about 80 Hertz--
nice bass frequency.
And I also need to set
the sample rate, which
tells the oscillator how
frequently these samples should
be produced.
And I can get that
from the stream--
get sample rate there.
OK.
I know you're all
desperate to hear a sound.
There is one very final
thing I need to do here,
which is I need to
respond to tap events.
So I'll just go back
into MainActivity,
and I'm going to override
the onTouchEvent so that,
if the event is
down, then I'm going
to call this native method tap.
Make that true-- else.
Otherwise, if the event is up--
I'm lifting my finger
off the screen--
then I'll pass in False.
OK.
Let's just have a look
at this tap method.
This needs implementing.
So I'll just pass in
this true or false value.
Create the new method.
And I'll just call
osc setWaveOn,
and that's going to
pass that in there.
Now, moment of truth.
OK.
I'm going to run this.
And in theory, you
should hear a sound.
So when I tap on the
screen, we should
hear an 80-hertz square wave.
The pressure.
[VIBRATING SOUND] There we go.
[APPLAUSE]
[VIBRATING SOUNDS]
So you can see it's the
lowest possible latency here,
and it's actually
pretty responsive.
So we have a musical instrument.
Admittedly, it's not the
best musical instrument
in the world.
A little bit to be desired on
the sound and control front.
So what I actually
thought would be nice
is if we could add a
little bit more control.
So for the last four
minutes and 30 seconds,
I'm going to tie
the motion sensor
to the pitch of the oscillator.
So to do this, I'm going
to cheat, uncomment
some code that I wrote earlier.
And what this does is
register a listener,
which will listen to the
rotation vector sensor.
So to listen to
these events, I need
to implement the sensor event
listener interface, implement
the methods onSensorChanged.
So what I want to do
is set the frequency
based on the event
values of the x-axis.
And I also need to
scale this value.
So I want to have it from,
let's say, 20 Hertz--
the limit of human hearing--
and let's go up to like--
this should give me up
to around 100 Hertz.
Yep.
That looks good.
Again, I just need to
implement this frequency.
OK.
So we'll just go
osc setFrequency.
And there we go.
OK.
So we're good to go on that.
Brief interlude--
very brief, in fact.
Has anyone heard
of the Amen break?
OK.
One person.
Yes.
So the Amen break comes from a
song in the '60s by the Winston
Brothers called "Amen
Brother," and it's
four bars of the most
incredible drum solo.
And it's the most
sampled loop in history,
but no one's heard of it--
evidently-- apart from one guy.
So I thought it
would be nice if I
could play my new musical
instrument over this loop.
So here's the loop.
I need to run the app.
Let's give it a go.
Let's just make sure it's here.
OK.
So with a bit of luck.
[MUSIC PLAYING]
Here we go.
[VIBRATING NOISES OVER DRUMBEAT]
Right.
[APPLAUSE]
OK.
So that's about it from me.
If we can just go
back to slides?
And yeah.
So the library is
available on GitHub.
There's documentation,
code labs, all sorts
of good stuff on there.
We'd love you to
start using it, start
building amazing audio
experiences on Android.
And I'll be around afterwards
if you have any questions.
So thank you all very much.
[APPLAUSE]
[MUSIC PLAYING]
