MALE SPEAKER: Welcome.
It's my pleasure to
introduce Gil Weinberg.
He's a professor at Georgia
Tech for the Center of Music
and Technology.
And I went to Georgia
Tech a long, long time ago
and we didn't have a music
department back then.
So when I visited recently, the
folks I was meeting with in ECE
sent me over to
meet Gil and he's
doing some really cool stuff
with interactive computing
and musicianship.
So building machines that
you can interact with,
not just from a
dialogue perspective,
like the kind of thing
I'm thinking about,
but more from a
music perspective
where you want to
jam and do some jazz
improvisation with a robot.
And he's going to talk about
that in many other things
today.
So welcome.
GIL WEINBERG: Thank you.
[APPLAUSE]
Thank you.
Thank you for coming.
So I'm going to talk about
three or four main projects.
Only three of them I
showed in these slides.
The last one is a surprise.
We did the project which we just
finished and I have some very,
very fresh new slides.
The reason I started
to be interested
in robotic musicianship
is that I'm
a musician before
I became interested
in computation or in robotics,
I was a musician and still am.
And I was always fascinated
by playing in a group.
By being constantly ready
to change what I'm doing.
By trying to build
on what other people
are playing in real time.
Improvising in a group, a visual
cues, auditory cues, of course.
So when I started to be
interested in robotics,
I wanted to capture
this experience.
Before I show you some of
the efforts that I did,
maybe I'll show a short clip
of me playing with a trumpet
player and trying to see
the kind of experiences
that I was trying to recreate.
[VIDEO PLAYBACK]
[JAZZ MUSIC]
[END PLAYBACK]
So I think you've seen
a lot of eye contact,
trying to build a motif that
I'm not sure what is going to be
and trying to create something
interesting with them
back and forth.
And the first robots that
I tried to develop it
was building on these ideas.
But what I wanted to do is to
have the robot understand music
like humans do.
The big idea that
I started with,
was to create robots
that listen like humans,
but improvise like machines.
Because I felt that if I
want to push what music
is about through new,
novel improvisation
algorithms and new
acoustic playing,
I first have to have connection
between humans and robots,
and that's why there is the
listening like human part.
Only then would I be able
to start to make the robot
play like a machine in order to
create this kind of connection
and relationship.
I'll start with
something very simple
that probably many of
you are familiar with.
If I want a robot
to understand me,
maybe the first simple
thing that I can make him do
is understand the beat
of the music that I play.
And here we use auto-correlation
and self-similarity algorithms.
This is a piece from Bach.
You see the time is both
on the x and on the y.
And you see that it's
symmetric and by comparing
the [INAUDIBLE] to
the algorithm you
can try to find sections
that are similar
and detect the beat from that.
But what we try to do is to
actually have it in real time.
And you see here my
student Scott Driscoll
used this algorithm based
on Davies and Plumbley
from Queen Mary.
And you see that's
it becomes much more
sophisticated because it's not
just analyzing the beat in Bach
or in the Beatles-- Scott is
playing in real-time so he
is trying to get the
beat, but then with Haile,
start to play with it.
Scott is trying
to fit what he is
doing to what Haile-- Haile is
a robot-- and back and forth.
So it's a little more
complicated you see.
Sometimes they escape from the
beat and get back to the beat.
And here's a short example.
Got the beat.
Now Scott will start
faster and slower.
Faster , got it.
So as you can see, it
loses it, it gets it back.
I think in a second, it
will play slower which
shows how the system fails.
The next thing that
I was trying to get
is to have the robot understand
other things at a high level
musically.
Not just the beat but
concepts that we humans
understand such as
stability and similarity.
And basically we
had a huge database
of rhythm generated almost
randomly-- with some rules,
some stochastic rules.
And then we had a coefficient
for stability and similarity.
Whenever Haile listened
to a particular beat
there's some settings
and coefficient
for stability and similarities.
At some point the robot
actually decided by itself.
First the human on the side
can change the similarity
and stability and create some
output to bring rhythm back.
And this similarity is based
on Tanguiane from 1993,
basically looking
at the overlapping
onset between beats.
I can get more into this
if you want, maybe later.
And this is as
similarity algorithm
based on Desain and Honing.
[INAUDIBLE] between
adjunct intervals
is what set how
stable the rhythm is.
This is based on music
perception studies
that I've been doing.
And basically, there is
a mathematical procedure
where you compare
each one of the notes
to the note that comes after it.
And at some point, after giving
preference to one and two,
which are stable, you can get
for every particular rhythem--
for example, this one,
the quarter quarter,
two-eighths quarter-- a
particular onset stability
by combining all of the ratios
and getting something stable.
And here's a short
example of Scott
playing with Haile
and Haile trying
to understand the stability
of Scott's rhythms.
And then based on a
curve of similarity,
starting most similar
then going less similar.
And basically a
human put a curve.
But I can see a scenario
where Haile could come up
with a curve by itself,
trying to start with something
that Scott understands.
Scott is playing seven
quarters, and then slowly
introduce new ideas.
And you can see how Scott
actually listens to the robot
and at some point, building
on what the robot is doing.
So Haile is building on what
Scott is doing, obviously,
by looking at the
stability and similarity.
But at some point, Scott
is almost inspired.
Maybe inspired is
too big of a word,
but that's a goal that Scott
will come up with an idea
that he would come up with
if he played with humans.
[VIDEO PLAYBACK]
[DRUMMING]
That's almost a new idea.
Scott is building on it.
[END PLAYBACK]
And this is a darbuka
drum player concert.
You will see how the
professional darbuka
player, actually from
Israel, is kind of surprised.
But I think his facial gestures
were interesting for me
because I think he was
surprised for the better.
And at some point you'll see
how all of us are playing
and Haile tries to get to beat.
So we combine the
stability similarity
and beat detection
into a drum circle.
[VIDEO PLAYBACK]
[DRUMMING]
This is call and response.
Later it will be simultaneously.
[DRUMMING]
And now, what it does, it
listened to these two drummers,
and tricked the pitch
from one drummer,
and the rhythm from another.
And the other arm,
most from pitch
and the timbre of
the two players.
[END PLAYBACK]
So we played the rhythm
that one player played,
and the pitch-- well, it's not
really pitch, it's in the drum,
but this is lower and this
is higher next to the rim,
and tried to create
something that is really
morphing between these two.
Again, things that humans cannot
do and maybe shouldn't do,
but here something
interesting can come up.
Another thing that I was
interested in is polyrhythm.
It's very easy for a robot to
do things that humans cannot.
Sometimes I'll ask
my student to clap.
I will not ask you to clap.
I'll give you an example.
I think there is two
main reasons here.
[SPEAKING RHYTHM]
This is nine.
[SPEAKING RHYTHM]
I don't ask you to clap
but sometimes I would.
It was [SPEAKING RHYTHM] seven.
And then I asked
my students to do
the nine in one
hand and the seven
in another hand which I
will definitely not ask you.
But see how Haile
here captured--
decided to record the rhythm.
So it records the nine.
He choose the nine and the
seven, and at some point
he introduced them in
polyrhythmic, interesting
rhythms.
[VIDEO PLAYBACK]
[DRUMMING]
And we add more and
more rhythms over it.
[END PLAYBACK]
And I don't know if know,
but Pat Metheny had a project
that he used robots in.
He came to our lab and
I explained it to him,
I showed it to him.
I said, this is something
no humans can do.
And he said, sure, my
drummers can do it.
And he can also do the four
with his leg, and another three
with another leg.
So I think everyone can do
it except maybe Pat Metheny's
drummer.
And here is something that's
at the end of concert,
it's obviously, you
know, fishing for cheers.
We just have a little
short MIDI file where
we playing with it together.
Unison always works,
so I'll just play this.
We grabbed the nine and seven.
[VIDEO PLAYBACK]
[DRUMMING]
And that's from a performance
in Odense in Denmark.
They had a robot festival.
[END PLAYBACK]
So the next project was Shimon.
And actually I have
an story about this
because I put a video of
Haile and Scott playing,
the first one, on my website.
It was before YouTube, or
before I knew about YouTube.
And someone grabbed it
and put it on YouTube,
and then CNN saw it and they
asked to come and do a piece.
And when they put a
piece, the next day
I got an email from the
NSF, from the NSF director
who said we saw your piece,
please submit a proposal
and continue to develop that.
Rarely happens.
Never happened since.
I tried.
I put so many videos--
And this is the next
robot that we came up
which adds multiple things.
The main thing that it adds
is the concept of pitch.
It plays a marimba.
And the second aspect
it adds-- we're
talking about the personal
connection, gestures,
visual cues-- is the head.
And many people ask
me, why the head?
It doesn't play any music.
In a second I'll show
some of the utilization
of what the head does.
And of course it has a camera.
That's the first
question you get
about any head, robotic head.
So the name is Shimon, and
here is the first example
of my post-doc, Guy Hoffman.
He's now a professor in Israel.
[VIDEO PLAYBACK]
[PLAYING PIANO]
It shows how a headless
Shimon responds
to dynamics, to expression,
and to improvise.
Look at Guy, how
he looks at him as
if his kid did something nice.
[END PLAYBACK]
[LAUGHTER]
So when Guy played fast,
the robot responded.
Simple, just tempo detection.
And then loud and
louder, and tried
to get some
understanding of tonality
and of giving the right codes.
And we had some
studies done about it.
If you remember the first
code was [SPEAKING CODE]
Three codes.
And we tried to see if,
by looking at the robot,
at the gestures, with
the arms before the head,
if the synchronization between
the human and the robot
is better.
And you see that the first
code is maybe easier.
The second code, very
difficult to synchronize
in terms of milliseconds.
The third code, you
kind of adjust yourself.
So it's a little better.
And we have multiple conditions,
one only visual, one audio,
and one with synthesized sound.
And you can see that
the visual is better
because it's less
milliseconds of delay
between the human and the robot
in both of the conditions,
precise and imprecise.
Especially in imprecise,
you'll see it's better.
For imprecise, the robot
could play 50 milliseconds,
before or after.
And just looking
at the robot helped
it become a much
more synchronized,
tight performance.
And it works much better at
slower tempos, less than 100
BPM as in more than that.
Then we added some
other aspects.
We said, OK, we have a head,
let's do what the head can do.
Actually we didn't
think of this way.
We thought of what the head
can do and then built the head,
but for the story.
And what it can do is
multi-modal detection.
We had a camera
there and we tried
to detect the beats
of the hip hop artist
to coordinate with the beat
detection of the music.
And you see how the
multi-modal detection work.
And also just to help
them feel together--
and this is Guy Hoffman's
work-- and develop gestures
that will make everyone
feel that they are
synchronized with each other.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
So that's just the audio.
[MUSIC PLAYING]
Now double-checking with
the beat of the head.
And of course
blinking never hurts.
The blinking is random.
[END PLAYBACK]
All right.
OK.
So that was a little bit
about listening like a human
and maybe grooving like a human.
And then I started to think more
about creativity, or machine
creativity and what it means.
And actually, this is the
first time I've showed it.
I hope it's not streamed,
I think it's not streamed.
This is a TED Ed video
that I just finished doing.
I didn't put in online
yet, so it's a part of it.
I didn't show it yet.
But it shows one
of the ideas I had
for creating machine creativity
using genetic algorithms.
And the idea was that if have
a population of motifs, maybe
random motifs, and then
I have a set of mutations
and cross-breeding
between these motifs,
and have a fitness function
that is based on human's melody,
then I can create
new, interesting music
that has some aesthetics
for humans in them.
Because it's easy
to create melodies
that were never created before.
Most of them will
not be listenable.
The question is, how do you
make meaningful and valuable
melodies?
And in the video, you'll
see a little reference
to the Lovelace Test,
which probably some of you
know about maybe I'll just say
that the Lovelace Test is based
on Ad Lovelace, the
first programmer,
or is considered to be
the first programmer,
trying to-- instead of
intelligence like the Turing
test, try to determine
when the robot is creative.
And she said something like,
a robot will never be creative
before it generates ideas that
its own programmers cannot
understand how they came to be.
And then in 2001 there
was a group of scientists,
some of them continued to
develop [INAUDIBLE] that
came up with this Lovelace
test, which is mostly
a theoretical thought
experiment of creating something
like that.
But even if we do
something like that,
I didn't think that the music
that would come be valuable.
And I'll show this
little TED Ed clip
and see how I went about it.
[VIDEO PLAYBACK]
-So how can evolution make a
machine musically creative?
Well, instead of
organisms, can start
with an initial population
of musical phrases.
[MUSIC PLAYING]
GIL WEINBERG: They
put the drum beat.
It's not mine.
-And a basic algorithm
that mimics reproduction
and random mutations by
switching some parts,
combining others, and
replacing random notes.
Now that we have a new
generation of phrases,
we can apply selection using
an operation called a fitness
function.
Just as biological
fitness is determined
by external,
environmental pressures,
our fitness function
can be determined
by an external melody, chosen
by human musicians or music fans
to represent the ultimate
beautiful melody.
[MUSIC PLAYING]
GIL WEINBERG: Chopin.
-The algorithm can then compare
between our musical phrases
and that beautiful melody,
and select only the phrases
that are most similar to it.
Once the least similar
sequences are weeded out,
the algorithm can re-apply
mutation and recombination
to what's left, select the most
similar or fitted ones, again
from the new generation and
repeat for many generations.
[MUSIC PLAYING]
The process that got us
there has so much randomness
and complexity built
in, that the result
might pass the Lovelace Test.
More importantly, thanks to
the presence of human aesthetic
in the process,
we'll theoretically
generate melodies we
would consider beautiful.
GIL WEINBERG: All right.
[END PLAYBACK] So
here's an example
of how this is
used in real-time.
Chopin is not [INAUDIBLE]
with fitness function
as you see here.
But actually, the
saxophone player and myself
playing piano-- I programmed
Haile with multiple motifs
and every time it listened
to one of our musical motifs,
it used dynamic
time warping to see
what's most similar and
come up with a variation.
And it stopped after
something like 50 generations
because, of course,
if it ran more than 50
it would become
exactly what we played.
So we just stopped one
time to create something
that is similar to what we
did, but a little different
and based on the, let's call it,
original DNA of the population
that we started with.
And [INAUDIBLE] Haile will
never use it with Shimon.
This is Haile when we first
tried to have Haile with pitch.
So the whole idea of
creating rich, acoustic sound
is not there.
We put the little toy
vibraphone below Haile.
So excuse the
quality of the sound.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
And listen to how
all three of us
are building on
each other's ideas.
Saxophone, piano, and robot.
[MUSIC PLAYING]
[END PLAYBACK]
Yeah.
It's difficult to evaluate it
because of quality of sound.
But that was the idea
of trying to put it
in a real-time scenario.
Another idea where
I tried to have
it play more like a
machine was style morphing.
And as you can understand by
now, I play jazz, I like jazz,
I like Monk on the left
and Coltrane on the right.
And I thought,
wouldn't it be awesome
if we tried to morph
between Coltrane and Monk
and create something that
they themselves will never
be able to create.
They cannot morph
into each other.
And maybe something
interesting will come up
that is just a combination
of both their styles.
And we used HMMs here,
basically analyzing
a lot of improvisation by
both of these musicians
and trying to see what the
likelihood is that at any given
note, what the next note would
be, what the next rhythm would
be, just a second order.
And then with an iPhone I
can play a little melody.
[VIDEO PLAYBACK]
And this is a little app
that we built regardless.
And at some point when
the melodies that I play,
of course it can be
piano too, is ready,
I will send it to
Shimon, and then I
will be able to morph
between my own melody, Monk's
style, Coltrane's style and
with two styles with saw
a different kind of
balances between these two
improvisations and myself.
[MUSIC PLAYING]
If you know Monk's
syncopation in clusters.
If you know Coltrane, you
know the crazy, fast licks.
And this is
somewhere in between.
And of course, one doesn't
play saxophone over piano,
so it doesn't sound
like any of them.
[END PLAYBACK]
But at least the notes are
based on the improvisation.
Some other quick examples
of things that we tried.
Here we had the
connect because we
wanted to get more
than just what
this simple camera
in the head had.
And we tried to have
motion-based improvisation.
You see the Atlanta symphony
percussionist, Thomas Sherwood,
doing some tai chi-like
movement and trying
to have Shimon improvise.
And actually Tom is playing
an electronic marimba
and Shimon has a system
in which sometimes it will
repeat what is listened to.
Sometimes it's
[INAUDIBLE], sometimes
it will introduce new ideas,
a stochastic system that's
supposed to inspire
Tom to play differently
than he would otherwise.
[VIDEO PLAYBACK] So
that's the tai chi part.
[MUSIC PLAYING]
And it wasn't easy for it to
repeat, exactly this pattern.
It's very limited with the four
arms and where they can be.
I'll show some path
planning strategies later.
So not every motif was
captured perfectly.
And now it starts to
introduce new motifs.
It will repeat and
then create some kind
of a musical interaction.
[END PLAYBACK]
This is my student, Ryan
Nikolaidis, the entire motif.
[VIDEO PLAYBACK]
And Shimon is
developing this motif.
This is part of the US
Science and Technology Fair
in Washington.
[MUSIC PLAYING]
[END PLAYBACK]
And for something else,
some of them just, you know,
they love dubsteps.
And the machine learning
maybe interesting or not,
but dubsteps are there.
So it's just a dubstep piece
that one of our students
developed for Shimon.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
So it can do all kinds of stuff.
[END PLAYBACK]
Here we used vision-- and
you see this green ball
on the conductor's
baton-- and tried
to combine the
input from conductor
and from the audio analysis
from the orchestra to improvise,
based on the chords that
the orchestra is playing.
[VIDEO PLAYBACK] That's the
Georgia Tech student orchestra.
[MUSIC PLAYING]
[END PLAYBACK]
All right.
And I think this is
my favorite clip.
This actually was a PSA of
Georgia Tech for three or four
years.
You know that we
come from Georgia
where dueling banjos are a big
thing, at least in some movies.
So we tried to do dueling
marine buzz if you like,
and present kind of the
dueling aspect and Shimon
with the gestures--
and this is again,
Guy Hoffman's work--
project some personality.
Hopefully better than the movie.
[VIDEO PLAYBACK]
[CALL AND RESPONSE MUSIC
 PLAYING]
[END PLAYBACK]
I think I have two
more clips before we
go to the next project.
I just got the request, how
about having a robot band?
They listen to each other
and create something.
I'm sure robotics people
here are being asked about,
will robots replace
humans often?
And in my case it's
really painful sometimes
because they come to me
and tell me, how dare you?
Music, really?
Like, OK cars, OK
cleaning carpets,
but you're going to
take human from us
and let the robots play it.
So I never tried, I'm not
interested in having robots
playing together by themselves.
It's only and always have
been about inspiring human
and creating something
interesting for humans.
And here you can
see a quick clip
where you see bit detection
and [INAUDIBLE] improvisation
between two students
and two robots.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
He's not so good with
the beat, by the way,
so-- strictly for
Haile to detect it.
But it got it.
And now it start to
improvise with the other arm.
What you hear here is
a student that we don't
see, playing the synthesizer.
[MUSIC PLAYING]
And now Shimon,
before we had a head,
actually we used
someone else's CGI head.
Andrea Tomaz from Georgia Tech.
I'm dropping back and
forth in time here.
[END PLAYBACK]
All right.
So that's a robot-robot
interaction for you.
I promised to talk a little
bit about path playing.
There are multiple ways
in which these forums
can be next to the piano
or next to the marimba.
In this case I would
just put a piano.
And we have state machines
that represent each one
off these states.
Given any melody, what would
be the most reasonable path
to take?
And the inputs that
we have is a curve
that humans play for
tension-- melodic tension
or harmonic tension,
or rhythmic stability.
And I'll show it here.
This is some the
pitch or note density.
And using the Viterbi algorithm,
actually [INAUDIBLE] Viterbi,
we're trying reduce
the space and try
to come up a reasonable path
to create improvisation,
following these curves.
So it's basically
developing, as I say,
an embodied musical mind,
where not only of the music
it has to play but
also it's own body.
The whole idea of
embodiment is a bit part
that robot musicians have
and that software musicians
or machine musicians don't have.
Only robotic musicianship has.
So we are trying to come up with
these algorithms, and actually,
my student, Mason Bretan
programmed really extensive
jazz theory into Shimon.
So now it can listen to
chords that Mason, in a second
will play, and then
try to improvise,
using this both music
and physical limitations
that he has using Viterbi.
He tried some other, but the
Viterbi worked pretty well
and this is his improvisation.
This clip actually went
viral kind of for our fill.
There were around 50k
viewers last month.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
Mason fed all the
chords offline.
All the calculation of
what's going to be play
was done offline.
[MUSIC PLAYING]
[END PLAYBACK]
So I don't know if it's
got [INAUDIBLE] quality
but it's not bad.
I was happy.
All right, so we all think
it's very important, the path
planning and
creativity and so on.
But it's important to
realize that this can
be taken in a different light.
And I think if you're being
made fun of for what you do,
this guy is probably the
best guy to make fun of you.
Some people tell
me, don't show it,
but I think-- I'm proud of being
made fun of by Stephen Colbert.
[VIDEO PLAYBACK]
-Jazz robots, which
combines two of the biggest
threats to our nation--
jazz and robots.
He's been programmed
to dig it too.
[LAUGHTER]
Bobbing his head to the
beat and checking in
with this fellow players.
Meaning that long after
the robot uprising kills
all humans on earth,
there will still
be someone on the planet
pretending to enjoy jazz.
[LAUGHTER]
[END PLAYBACK]
OK.
Shimi was an effort to
take Shimon and make
it a little smaller.
Actually it almost
became a product.
There was a failed
Kickstarter there.
And it focused on the
gestures, not so much
from the acoustic
reproduction of sound,
which is part of the reason that
I went into robotics because I
kind of grew tired by sound
from speakers, no matter
how great your hifi system is.
I was missing the
acoustic sound which
is one of the reasons
I went to robotics.
So here we don't
have acoustic sound,
but we have a bunch
of other applications.
We call it a robot companion
that can understand music.
It can connect to your Spotify.
It can dance, it can
do the bit detection.
It has a set of algorithms
to get data from the audio
and respond accordingly.
And here's a clip.
[VIDEO PLAYBACK]
[MUSIC COLDPLAY, "VIVA LA VIDA"]
GIL WEINBERG: In this case it
takes the music from the phone,
but we also have an application
that takes it from Spotify.
And the camera and the phone
can see if you don't like it.
And better to say no,
because the music is loud.
[TAPPING "DROP IT LIKE IT'S
 HOT"]
[MUSIC SNOOP DOGG, "DROP IT LIKE
 IT'S HOT"]
GIL WEINBERG:
[INAUDIBLE] speaker dock.
You can see the
dancing speaker dock.
[MUSIC SNOOP DOGG, "DROP IT LIKE
 IT'S HOT"]
[END PLAYBACK]
Here is a couple of other
applications, some using
speech, and some
using audio analysis.
That stuff is audio analysis.
Looking for a beat in
what Mason is playing.
[VIDEO PLAYING]
[PLAYING GUITAR]
GIL WEINBERG: No, there
wasn't anything there.
[PLAYING GUITAR]
You get it.
It has a system of gestures.
[END PLAYBACK]
GIL WEINBERG: And here,
using Google Voice.
Mason is connecting to Spotify.
[VIDEO PLAYBACK]
-Look at me.
Can you play something happy?
GIL WIENBERG: And we
have some algorithm
to determine happy, disco, rock.
[HAPPY MUSIC PLAYING]
-Can you play disco?
[MUSIC BEE GEES, "SATURDAY NIGHT
 LIVE"]
GIL WIENBERG: Of course this.
So a different set gestures
for different genres.
All right.
[END PLAYBACK]
Annie Zhang, another student,
worked on the Shimi band,
trying to create collaborations
between three Shimis.
And she looked at a lot
of [INAUDIBLE], music
[INAUDIBLE] elements you
would get from the audio.
Anything for the beat detection
to the beat to the arm
is for the energy.
MFCC for vocalist to see if
there's vocals there or melody.
And then this is her interface.
And she then came up from
the low level features,
detecting some high
level features,
beats, energetics,
melodics, vocalness,
percussiveness of the sound.
And then she actually went
and studied [INAUDIBLE] system
and tried to create
a dancing grammar
for all of these gestures
that will connect
between how Shimi is dancing.
And here are some of
her ideas for that.
And a whole system of different
ways in which three Shimis can
dance together-- solo,
having the wave, two
in synchronicity from both
sides, and the main one
is different.
So a full system that later
allowed her to have this.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
And every time it moved
different, the gestures.
[END PLAYBACK]
And even in a very noisy
environment in a demo--
[VIDEO PLAYBACK]
[MUSIC PLAYING]
[END PLAYBACK]
Here too, we try to have
an embodied musical mind
and have an understanding of
what are the important music
notes in a piece.
Tonally, maybe
they're more stable,
maybe it's the end of a
phrase that goes up and so on.
And then combine it
with some understanding
of what are the
positions in which what
can move to particular
places, being
aware of its own
limitation in terms
of the time of the gestures.
So here you see
the red dots that
represents a
particular end point
in gestures that try to be
based on the important notes
melodically.
Again, the combination of
being aware of the embodiment
and of musical theory
and creating a system
that, I think Mason
here put some titles
that you can read as he plays.
[VIDEO PLAYBACK]
[PLAYING GUITAR]
[END PLAYBACK]
And as you can see, Mason
is extremely talented,
and he is looking for an
internship this summer.
[LAUGHTER]
Just to end the
Shimi part-- maybe
I'll just play this little clip.
I couldn't make a Shimi without
letting it dance to this song.
[VIDEO PLAYBACK]
[MUSIC STAR WARS, "CANTINA
 SONG"]
[END PLAYBACK]
And that's the last project
that we just finished last year.
I actually took a break.
I took a sabbatical year off.
And just when I was about
to go to the sabbatical
I got an email
from Jason Barnes.
He's an amputee drummer.
He lost his arm, now it would
be like three years ago.
When we met it was two.
And he saw some of
the videos online
and he was really devastated.
There's a clip online.
You can see him talking about
how he thought he had nothing
to live for if he cannot drum
when he lost his arm when he
was electrocuted in some
weird accidents on a roof.
And then he saw these video
and he emailed me and asked me,
can you use some
of the technology
to build me a robotic arm?
And, of course, that was
very intriguing for me.
What he really wanted was to
be able to control the grip.
Because he had the arm up
all the way up until here so
he can still hit.
But for a drummer, it's
very important for them
to have control of the
grip, tight or loose
to create different
kinds of drum strokes.
And that was very
interesting for me.
And the people I
usually collaborate
with couldn't come
through, so I-- I
don't know if anyone
here is from Meka,
but I did contact Meka
who built this arm.
And that was great but because
I'm interested in improvisation
and creativity, I really
wanted to have this aspect
of my research in this arm.
So I asked Jason if he
minded that we had two
different sticks, not only one.
One was all about grip, so
you can have your expression
and play it as he wants.
The other one actually
would listen and improvise
and will listen to the music
and listen to other musicians
and try to maybe inspire
you to create something else
and become better or try
to create music that he
would never create otherwise.
Let me show you
a couple of clips
first of the basic interaction
of just controlling the group.
And what we discovered
after Phillip from Meka
built the arm is that we don't
need just to hold the stick,
we can actually use
it to generate hits.
And after some work
on noise reduction,
we were able to some
pretty quick responses,
very low latency
so Jason could just
flex his muscle, in the
muscle and actually create
a beat, in addition
to using his own arm.
So there's some
aspect of a cyborg
there because part
of the gesture
is his own body and part
of it is electronically
based on his muscle and
maybe other muscles,
not only the forearm muscle.
So actually, this
took eight months.
It was amazingly fast,
again, thanks to Meka.
And this is the first
time Jason tried it.
MALE SPEAKER: Next.
Next.
Next.
GIL WIENBERG: And
that's where we really
got-- I didn't expect that,
that we can create 20 hertz.
And you know, my favorite
polyrhythm games.
I'll show more of the
musical context later.
But let's first--
see, here we didn't
have the EMG working yet.
It's a little knob that changes
between tight and loose,
but just to see the effect.
This was a tight hit.
[VIDEO PLAYBACK]
[DRUMMING]
Assuming the holding
of the stick tight,
this is a loose hit.
[DRUMMING]
And this is a role tight.
[DRUMMING]
[END PLAYBACK]
Many more expression
than he had before.
He couldn't do all of this.
And actually, he was in
tears the first time.
He said, I didn't play this
well since I lost my arm.
Which was the first time I
actually worked with people.
And it was pretty powerful.
But, here, you can
see the second stick.
The kick is very short.
Look here, we have
an accelerometer
and when he wants--
only when he wants,
he can lift the arm quickly and
the second stick will pop up.
It will be really fast.
OK.
And then we can start
with the improvisation
and the creativity,
the machine creativity.
Before we do that we
can just have a computer
generate strikes.
So nothing to do with him.
Even though it's
part of his body,
suddenly someone else is
generating hits for him.
And even Pat Metheny's
drummer cannot do.
Not with one arm.
And this is, again,
playing with roll,
with tumble, because it's so
fast it becomes actually color
and not even rhythm.
And this is what I call
cyborg drumming where
you combine some of the
computer generated hits
with his own hits.
He decides where to put the
arm, where not, what to choose.
Create a human
machine interaction
and musical collaboration.
And at some point-- actually
I wrote an NSF proposal
about it-- we want to create
some human brain machine
interaction.
But now some of the
most sophisticated
stuff we're doing with pedals.
With his leg, it controls
dynamics and expressive
expression elements.
And when this first
clip went online,
it also kind of became viral.
And I got a lot of
emails and one of them
was from a guy called
Tom Leighton who actually
played with Richard Feynman.
I didn't know that Richard
Feynman was a drummer,
playing hand drums.
And he said, I have a bunch of
recordings from Richard Feynman
and I would love for
you to play with it
and see if you could
do something with it.
And of course I jumped
on this opportunity.
And what we did here is
we put some of the rhythm
that Richard Feynman
played in his arm,
so he had a chance to play with
Richard Feynman in a weird way.
And first you'll see him
hovering over the drum,
just listening to the
rhythm, and then he
starts to decide what drum, when
he whats to lift the symbol,
and so on so on.
So it created all
kinds of interaction
with someone who is
not with us anymore.
And I think it also was a
powerful experience for him.
[VIDEO PLAYBACK]
[DRUMMING]
That's the rhythm by
itself, just hovering.
And now he adds to it
with the other arm.
[DRUMMING]
Now he starts to
decide where to put it.
He's hovering but
now we drew this
as part of his
gestures and movements.
[END PLAYBACK]
Here we have analysis
of the chords
and we try to create
similar connections,
a very intimate connection
between the guitar player,
Andrew, and Jason.
Different chord that he plays.
We analyze the chord
and change the rhythms
that Jason is playing.
So suddenly he has
to respond, not only
to the arm that is
generated by the computer
or generated by his muscle,
but generated or controlled
by a different musician.
And you see the whole eye
contact and the whole,
trying to create
something together,
happening between them two.
[VIDEO PLAYBACK]
[DRUMMING AND GUITAR]
It was one of the
first times that he
did it and it's not tight,
but it becomes tighter
with every rehearsal
and every performance.
[DRUMMING AND GUITAR]
[END PLAYBACK]
And here he is, in context,
trading four for those of you
who play jazz.
You play four bars
and then someone else
improvises four bars
and back and forth.
Here, the arm listens to the
improvisation for four bars
and then Jason
gets to improvise.
And the arm is responding,
improvising, [INAUDIBLE]
classic manipulation of what it
heard in the four bars before.
[VIDEO PLAYBACK]
[JAZZ MUSIC]
[END PLAYBACK]
And I have longer clips
for all of this online.
This is the latest project.
I just received this videos
from Deepak, my student.
Just yesterday he just
finished doing that.
And if you may
have noticed, a lot
of the expressions that
if you are a drummer
you probably know that you can
hit in much more versatile ways
than just a hit or not a hit.
There's a lot of
multiple bounces
and a lot of expression in it.
Here is Deepak who is a drummer.
My students need
to be musicians,
and in this case roboticist.
Actually he was just accepted
to a robotics program
at Northwestern.
The first master student
from music technologies
that actually got into
a robotics program.
It shows the different
kind of drumming techniques
that he has.
[VIDEO PLAYBACK]
So all of this richness and
expression, we couldn't have.
So Deepak had to go and
learn PID control, which
again, is not one of
the classes that we
offered in music technology.
[END PLAYBACK]
And he tried to come up
with a model for the talk,
taking into account gravity.
And something he had to know,
it's a very complex system.
The bounciness of the
drum, for example,
is not being taken into account.
But he tried to put
everything in simulation.
And with some coefficient
of restitution
here is 0.85 just as a constant.
And he tried to create all
kind of bounce behaviors.
Some of them are just based on
humans and bouncing balls kind
of techniques.
But some of them are actually
bounces that humans will not
create or are not likely to.
You see this is much faster.
At its end, it's lifting
for the next hit.
And here you have a high to
low force profile you see.
This hit is softer
than the next one.
So you really have to change--
it's time varying talk changes.
And he put it into a model
and created a PID control.
And this is the first clip.
He just did it yesterday
morning and sent it to me
just before I left.
[VIDEO PLAYBACK]
So hopefully this will
be embedded into some
of the new performances.
And we're going to
France to perform in May.
Hopefully Jason can play a much
more wide range of expression.
This is some of the future works
that the NSF proposal that I
have wrote is going to address.
Am I'm working with
a neuroscientist
from Northwestern, [INAUDIBLE].
And we are trying to
get some understanding
about anticipation.
Because there are
latencies in the system,
and no one isn't focusing
on volition detection, which
is basically-- your
brain is shooting,
sometimes seconds
before your motor cortex
actually knows that
you're about to hit.
In a way, you know you're
going to do something
before you know you're
going to do something,
or a part of your
brain knows about it.
I tried to get this data.
We did some experiments
with 36 EEG electrodes,
and there maybe some
way to get information
before the hit is going
to be made and then
better anticipation algorithms.
So if he hits with
both hands, or hits
with both times-- this
hand hits by itself,
it will be on time because
we have some information.
We did some machinery to detect
different kind of strokes
and we saw that the
sum-- it's promising.
It might be tricky and
we have some outliers,
but we can pretty much
detect single strokes
versus double strokes, which
is something else that we
can try to implement.
And this is one way it will
probably never look like.
But this picture
is in the proposal.
Justin is an example.
The trick here, when
you create such cyborgs
is how not to interfere with
what you're doing anyhow.
But as you can see, I'm jumping
into third arm scenarios.
So half of the proposal
would be working with Jason
on his own arm, creating this.
But the second part is, let's
try, available to anyone,
whether you are disabled or
abled, and have a third arm.
And the promise, the
spiel is that if we
have anticipation algorithms
and timing and synchronization,
that works in music-- and music
is a very, very time demanding
and medium.
You know, you hear
five milliseconds
that what you heard is two.
So if we can pass
synchronization in music,
we can anticipate and help
in all kind of scenarios--
on operation rooms or fixing a
space station with a third arm.
So how to put it on
the body in a way
that you can be
completely-- first,
the cognitive load,
how to control it.
So we thought about the
brain, the organ of muscles
that we can still try to hit.
But the muscles are used
for legs for the drums
and for arms for the drums.
This is a different kind of
design for the future brain
machine control.
And I'll be happy to go
for questions right now.
Thank you very much.
[APPLAUSE]
AUDIENCE: I think
you alluded to this,
but can you say
more about why you
think the physical manifestation
of a striking thing
is important.
Are there other artists
actually getting information
from the motion of the robot
or is it just for looks.
GIL WIENBERG: So one
study showed that it
helped with synchronization.
That's the graphs that I showed
when you see the arm moving,
when you see the beat.
You'll better synchronize
your gestures with it.
I'm a big believer
in embodiment.
I believe that that's why
we go to concerts as opposed
to listening to music at home.
We want to see how people
interact, not only by music
but by gestures.
A drummer and a guitar
will end the song together.
And there's a lot about
interacting with gestures
and being aware of your body
as a musician that I think you
cannot do without it.
Of course, all of it can two
iPhones that play this music.
But I think we're going to miss
the whole musical experience.
It's something really deep
and strong in embodiment
in the musical experience.
AUDIENCE: So two sort of related
questions, if you don't mind.
The first is, have
you tried sort
of persisting some information
across several sessions for any
of these to try to
have the robot develop
some kind of style
more incrementally,
and sort of lock
into certain patterns
that it prefers systematically
over longer periods of time?
GIL WIENBERG: I
thought about it.
We didn't get to it.
It's a great idea
because musicians--
that's what they do.
There's length, there's
lifelong learning.
And we're doing
just small episodes.
There was a student that was
interested in starting to do
but then he decided that
a master's is enough.
He doesn't need a PhD.
But it's a great suggestion
and it's definitely
something important for music.
AUDIENCE: And then
related to that,
some composers or
musicians have--
you talked about
embodiment-- certain sort
of different bodily
features like, I don't know,
someone like Django Reinhardt
in jazz had only two fingers
to use on the guitar.
Rachmaninoff had very big hands.
So have you thought about
sort of creating robots
that have different variations
on what kind of motion
is available to them and seeing
how that guides things, or sort
of, how more
constraint can change?
GIL WIENBERG: So that's
the latest work by Mason
with the path planning.
And he tried to come
up with a system.
His PhD thesis will come up
with a model of other musicians.
That's one of his
main goals to continue
to develop the field of
robotic musicianship.
Other people who
build robots will
be able to use this kind
of path planning ideas
and put as a coefficient
the different limitations
that they have.
AUDIENCE: OK.
Thank you.
GIL WIENBERG: Yeah.
AUDIENCE: There was
only one instance
where you mentioned
a robot knowing
anything context-specific
or genre-specific.
For most of the
examples, it was instead,
everything that the
robot should know
about how to
generate the content
comes from the content
it's receiving.
But then there was one
where one of your students
programmed all of these
jazz theories into it.
Do you have an
aversion to telling
a robot genre-specific or
context-specific information?
Or do you want it to be as
blank of a slate as possible?
GIL WIENBERG: I don't have
aversion for anything.
I come from jazz.
Jazz is about improvisation.
A lot of what I'm trying to do
is [INAUDIBLE] improvisation.
So it doesn't necessarily
fit maybe other genres.
Like with classical, yes,
Bach can be improvised
and you can have all
kind of counterpoints
based on improvisation.
But I can see a scenario
when you're trying
to put classical theory in it.
I less see how it's going to
be used for improvisation.
I think that it should
be very interesting, yes.
AUDIENCE: Do you think you would
ever do it negatively and say,
OK, well, these things, if
you ever come across this,
this is not idiomatic?
You should avoid it?
I'm thinking, for example,
in jazz of course,
but also for the hand drums,
like, I don't know anything
about that style
of music but would
you ever want to say
like, in whatever mutation
or combination that comes
up, this particular pattern
is seen as ugly?
GIL WIENBERG: I don't
know if ugly, but--
AUDIENCE: Or not--
GIL WIENBERG: Not
part of the grammar.
Yes, we have it in
the jazz system.
AUDIENCE: OK.
Cool.
GIL WIENBERG: And
I think it will
be interesting to try
to see what happens when
we go to other genres,
with maybe a different kind
of tonal theory.
AUDIENCE: I have a question
back to the pathing analysis.
I'm curious if you have a
measure of how much less you
are able to do when you do have
these physical constraints,
versus if you could have the
machine improvise it and watch
other people, and it would
just have to produce tones.
Is there some sort of measure
as to the limitations,
qualitatively versus what it
could do, if it didn't have
the physical limitations
of forearms that
can't cross over one another?
GIL WIENBERG: I think
we can deliver--
I don't have a measure.
But I think a measure
can be developed
and I think a measure can
also be used for future robot
designers to build robots that
have less of these limitations.
And when we came up
with the four arms,
that this arm can never play
this octave, we knew that.
And we knew that, in a way, it
might play faster than human
and create 16 chords
simultaneously which humans
cannot do.
But it will also be more
limited than humans.
And I think when I play piano,
I do path planning in a way.
If I don't have the
fingers, if I just
to look at notes
with the fingers,
I constantly,
probably, do something
similar to Viterbi's beam search
job algorithm or something.
Creating a measure in terms of
coming up with different kind
of chords that you
can or cannot play--
different kind of speed of
phrases that you can or cannot,
you know.
They're what, if you noticed,
one for the black keys,
for the accidentals and one line
for-- so the limitation just
there and see if you
can play clusters.
Can it play a cluster
of C C sharp D E flat E?
It can't.
We don't have a measure
to really come up
with a coefficient, but
I think it's a good idea.
I think all of you have
a lot of good ideas.
AUDIENCE: Thank you.
AUDIENCE: First, awesome talk.
Thank you very much.
The question I have
has to do with,
kind of, wanting to play
with these robots myself.
There's a bit of a
bottleneck in that
your team is doing the hardware
and also all the software.
Have you thought about
moving towards a more
platform-based approach where
other labs could write code
against your robot and
be able to try it out?
GIL WIENBERG: That's a good one.
And by the way I'd like to
give credit when credit is due.
My students are mostly
computer science students.
The platform itself--
I have colleagues
come from the MIT Media Lab.
Guy Hoffman designed
and Rob [INAUDIBLE]
designed the robots and Meka
designed the latest one.
So this is not our research.
Probably the Meka robot
is the best candidate
to become a platform.
The others are too
flaky, and too specific.
But part of what
I'm interested in
is also the art and the
design and the coolness
of creating a new robot.
So it's just because I love it.
But I think it can be a good
idea to try to, for the arm,
for example, to try
to make a platform,
create code, create, I don't
know, an API for other people
to try to play with.
Definitely.
I think this field is
pretty new and there's
all kinds of [INAUDIBLE].
At some point, if enough
people think it's important,
maybe we will use a
robotic operating system.
But for now, we just focus on
creating cool musical one-offs.
AUDIENCE: Thanks.
AUDIENCE: In the
physical gesture robot,
you showed a picture
of a score where
you had analyzed important
points in the melody where
you might end a gesture.
But it was taking
as input, raw audio.
Is that right?
As opposed to a MIDI file
or a digital encoded score.
GIL WIENBERG: The
input was a chord
that Mason and played
from a MIDI piano.
So it got the symbolic
notes from the MIDI.
AUDIENCE: Oh, OK.
And he was playing a guitar I
thought at one point as well?
GIL WIENBERG: Yes,
that was different.
The guitar was only
played with shimmies.
Piano plays with Shimon.
The guitar for Shimi
to depict the right--
and yes, it did repeat--
actually, as I mentioned,
this is offline.
Mason played this
melody, it did it offline
and then it recreated it.
AUDIENCE: Oh, OK.
OK.
Thank you.
And one other thing.
About the auto-correlation
beat detection.
Does that infer any time
signature information?
Or how good is that at detecting
strange time signatures?
If at all?
GIL WIENBERG: It's not as good.
Four quarters is much easier.
A big estimation,
so it pretty much
focused on regular
time, three, four.
Others would be more difficult.
AUDIENCE: Thank you.
MALE SPEAKER: I
had one question.
I'm a classical musician and
this is about online learning.
I mean, yeah, online real-time
learning from the instructor.
So as a classical musician,
you sit down, you have a lesson
and you have a instructor
that teaches you,
based on what they're
hearing, giving you feedback
and then they correct or
improve your playing right then.
I don't know anything
about instruction
for jazz improvisation.
I guess I've seen a bit of that.
Wynton Marsalis has
a program right now
that you can watch where he's
working with some young jazz
musicians and he's
giving them feedback.
A lot of it's about
communication and listening.
Have you thought about
that sort of thing?
Not so much-- what
you're showing here
is the actual improvisation, but
if you wanted to give feedback
to the robot and say, you know,
hey, why don't you try this.
Or that's a little bit too busy.
You maybe should be kind
of simpler at this time
and then add some more
color later or something.
I don't know.
GIL WIENBERG: The
same students that
decided that MS is fine were
thinking about learning.
Starting from nothing and
learning getting some input
and reinforced.
And then develop a style
that will develop over time.
We were thinking about.
We didn't get into it.
But yes, in jazz too, I think
similar to classical music,
that's how it works.
First production of the sound.
You know, when I
first played piano,
it was a lot of hits on my
fingers and holding turtles.
If I have some
more students using
learning-- not in the
machinery applications of it,
but actually your robot that
knows nothing and starting
to teach it slowly.
It can be very interesting.
Maybe very interesting
new music perhaps that
wouldn't come this way.
MALE SPEAKER: OK,
thank you very much.
That was awesome.
GIL WIENBERG: All right.
Thank you.
[APPLAUSE]
