- All right, thank you
for the introduction.
My name is Mahdi Azmandian.
This work was done at my
internship at Microsoft Research,
in collaboration with Mark Hancock
from the University of Waterloo.
One of my favorite quotes is from a movie
called The Prestige.
It tells the story of two
rivaling illusionists.
One of them is reckless.
He's absolutely obsessed,
and he sacrifices everything,
and he stops at nothing to make
the best illusions possible.
On his death bed, his
archenemy asks him, why?
Tell me why you spent so much obsessing
over these illusions.
He said you never really
understood it, did you?
He said, the audience knows the truth.
The world is simple, but
if you could fool them,
even for a second, then
you can make them wonder.
Then you can get so see
something really special.
It was all the look on their faces.
Today, I want to talk about my obsession
with illusions and virtual reality.
Let's start with a very simple trick.
You're going to come into a room,
you're going to see
three cubes on a table.
You're going to put on
a head mount display,
a virtual reality head mount display,
and then you're going to
see three virtual cubes.
Everything seems nice and dandy.
Then without you knowing,
we're going to reach out
and remove two of those cubes,
leaving only one behind.
Then the experience starts,
reach out and grab the blue cube,
and you reach out and grab the blue cube,
then reach out and grab the green one.
Then you keep doing this again, and again,
and you're thinking this
is the most stupid demo
I've every experienced,
but you don't know what's
going on behind the scenes.
The whole time you're reaching out
and grabbing the same cube,
and you had no idea until
you take off the HMD,
and people's expressions
are absolutely priceless.
I'm going to go against
the illusionist's code,
and reveal our secret in this talk today.
This is something we
call haptic retargeting.
Haptic retargeting allows
physical props to be reused,
for haptics, by leveraging
the dominance of vision,
to retarget people's hand motions.
Essentially what this
means in practice is,
a single prop can be
repurposed to allow movement
and stacking of a virtual space,
with a multitude of virtual objects.
Why do we care about these illusions?
We're not in the business
of tricking people only.
Let's take a step back and think about
virtual and augmented reality experiences.
Normally the first thing happens,
especially in VR is that you put on HMD,
and your body's gone.
You completely lose
connection with your body,
and you can't interact with the world,
or with augmented reality,
you see your body,
but you cannot really
interact with the world,
and affect the world they
way you naturally would,
or sometimes you have a controller and
you can't really interact wit the fidelity
that you normally have an experience.
As Dan talked about, we
care a lot about haptics,
and people have looked at different ways
of bringing in haptics,
like the phantom, or
the cyber grasp glove,
but one other way of doing
this is using passive haptics.
Essentially there's a one-to-one map
with everything everything
in the virtual world,
and the real world, and you
can reach out and grab objects,
and it's a very compelling experience
when you can interact with
things the most natural way
that we're used to.
That works and it's all good,
but what if you have a
lot of virtual objects
that are not really present in the scene.
It's similar to the
notion of an x-reality,
but what do you do here?
Let's say you want to
make a mindcraft castle.
If you want to make a castle,
you're going to have to have
a castle of cubes around
and move those around.
This is not really practical
to do it passive haptics,
so we ask ourselves, how
can you make a single object
provide haptics again, and again?
Repurpose the haptics that
you're getting from the object.
The solution that we found for this
is to hack the human perception.
These kind of illusions have been around
for quite a while.
Our inspiration essentially
was redirected touching,
where the high level idea is
to warp the virtual space,
to map differently shaped virtual objects,
onto a single real object.
Their classic example you see,
you come up to a square
shaped panel facing you,
but in the virtual world
you see the same panel,
but it's rotated away form
you, and you reach out
and interact with it,
and somehow the haptic
from the physical object
is lining up with the virtual object,
or another example, you
see yourself interacting
with this curved surface,
and you're seeing your
hand moving up and down,
you're convinced that this
virtual surface is curved,
but in reality, you're actually
touching a flat surface,
and you're convinced because
of what you see overriding
what your body is telling you.
Another space that also
uses these kind of illusions
is redirected walking.
The problem here that they care about,
is being able to explore
large virtual environments,
and you have a small track space.
The kind of thing that
they do is they manipulate
the trajectory that you
have in the real world.
If you end up walking in a certain path
in the virtual reality experience,
and you're perceiving that,
but in reality, you're
walking down a different path.
An example that they had
is you can be walking
on a zigzag path, shown in blue here,
and perceiving that in
the virtual environment,
but in reality you're just
walking back and forth
on the same path.
The key idea that they use here
is something called rotation gain.
It's the idea of upscaling your rotations
in the virtual and real mode environment.
Essentially in this example here,
you can be rotating in place 180 degrees,
but in the virtual environment,
you'll see yourself
rotating only 9 degrees,
and as a result, you're going
to be walking on a path.
Overall you're going to
be having a trajectory
in the the real world that's
different from what you have
in the virtual world, so
you can do a lot of things,
change people's perception
of the shape of objects,
with redirecting touching,
or get people to think
differently about the space
that they're walking
in, with real walking,
but we asked ourselves,
what can you do with objects and props,
and grab them and move them around.
Before we start to talk about this,
let's talk about the
hardware configuration
that we use for our system.
One key thing that we really cared about,
was to somehow get our body
back into the experience,
and reproject body back
into the work space.
To do this, we mounted a
connect up from the ceiling,
looking downward,
essentially looking at
the real environment
to understand what's going on,
to be able to render that
in the virtual scene,
and we also had users put
on a head mount display,
an Oculus,
and interact with props in
the virtual environment.
We also needed to track a few things,
we needed to track our hand, a cube,
and a wand, and of course
the way we track things
is very simplistic expedient
method for our study,
but in reality you would probably
use something more viable,
and usable for a demonstration.
We also had a little simple knob,
the confirm button, to progress
the experience further,
and when you put that all together,
you get this video that we
saw this experience here,
that you see your own hand, the way it is,
the shape that it actually has,
can reach and grab objects,
and you can touch,
you can feel them, the
mapping is very compelling.
Let's talk about our first trick.
Our first trick is called body warping.
What this means is that
we take this virtual
representation of your
body, and we manipulate it.
As an example here, if
you have a physical cube,
and a virtual cube onto the side,
what you can do, is as the
user reach from their hands
towards the cube, you can
have their hands shift
virtually to side, and
align with the virtual cube
that they're trying to reach.
This was the key idea
behind the first trick
that we showed you,
where as users reaching
out to grab the objects,
let's say for the blue cube,
their hand is gradually
shifting to the right,
and lining up with the blue cube,
or for the red, one, they're
gradually shifting to the left,
to line up with the red cube.
Another trick that we have
is called world warping.
World warping is similar
to redirected blocking,
it's using the idea of rotation,
and scaling of motions,
so for instance in this case,
we can have a user doing,
say 90 degree rotation with their head,
but virtually the world is
also rotating around them
without them noticing,
and you can have virtual objects
line up with your physical
by using this technique.
To demonstrate this, we had a
ring of cubes demonstration,
that we made, where you
can have a physical cube
line up with a bunch of cubes on a ring,
by rotating the world around them,
so the user would look to the right,
and look at billboard, and then look back,
and another cube was available
to reach out and grab.
Essentially the reason we're having people
rotate their heads,
is to use these head motions to subtly
rotate the world around them,
and then get the cubes to
line up on a different one,
one at a time.
We use hand motions to
create body warping,
and then we leverage head
motions to have world warping,
and then we said, how can
we combine those two for a
hybrid mechanism?
The way it would work,
let's say when you want
to do the alignment,
you split the alignment
tasks between world and body,
so with your head motions,
part of the alignment
is done with the world warping,
then when you reach
out to grab the object,
the rest of it is taken
care of by body warping,
so potentially you can increase
the power of your alignments,
and things you can do when
you have an hybrid mechanism,
but the coolest thing we can do
with this kind of
manipulation, is stacking.
We wanted to get a sense
of grabbing an object,
and stacking on another object
that doesn't exist there,
and the way we did this was
we applied the same idea
of warping, but to the height,
so when you're moving the object,
when you reach up and grab it
to put it on a target location,
the height warp is applied,
so you put the cube on the table again,
but somehow you see your hand going up,
you're convinced that oh, this stacked.
I remember a person was
doing this in our experiment,
and he was like, well
should I just let it go?
I said yeah, have faith brother,
and he let go, and it
was like, whoa it stayed,
and he expected it to drop down,
but it stayed right there.
Now to put this all to the test,
we conducted a study.
The task involved our
motivating example of
being able to create a mindcraft castle.
The expense worked like this.
Essentially we guided
the users in the process
of creating a mindcraft castle.
It started with them
looking up the billboard,
to the side, to get the new instruction.
They look back an see a blue cube
that would spawn,
and then they put it in
the red target location,
then they press the button,
and the cube would solidify,
and then they look back
again, to the billboard,
and when they look back at the table,
they'd see a new cube magically spawning,
and then out of thin air,
they'd grab that and put
it in the target location,
and they do this nine
times to be able to create
the shape that we were giving them.
The conditions we had
were for each participant
experienced different
conditions, the same conditions,
we had different manipulation techniques.
The ones we mentioned.
We had only body, only
world, only hybrid work,
and then we also had baseline,
or use the virtual one,
which is essentially the case
where you would not have haptics in VR,
and you would grab along, reach out,
and click to grab a virtual object,
and move it around, and then
click again to release it,
and that was the baseline
for making this castle.
We had 20 participants that
first started with them.
We did an ocular dominance test,
and a spacial visualization
test before this study,
and at the beginning of each condition,
they could play around
with the cube a little bit
to get comfortable and
acclimated to it for a minute,
before we started the actual task.
The task involved
looking at the billboard,
getting instruction, putting the blue cube
at the red target location,
nine times to the end,
and at the end of the condition,
we would have them fill in
immerse your tennis shoes
question there to measure presence,
and we also track their hand location
to analyze the way
their hands were moving.
The details are in the paper,
but the most important thing
is that our hybrid technique
was significantly better
than the wand method.
And people claim the
presence that they had felt
from the hybrid technique,
was way more than wand,
but interestingly some people, did prefer,
at least one person did
mention that this is all nice,
but I prefer using the wand,
because of the simplicity,
and unencumbered nature of the wand.
It's interesting how
technology changes the way
what we call natural,
and what we're used to.
More details on the breakdown of presence
are in the paper.
We also looked at their hand motions.
We wanted to see how
their hands are moving,
so we plotted the
trajectory of their hands
when they'd reach out to grab a cube,
and we plotted them for
different conditions.
We noticed about 30% of the time,
with body warping, the
body warping condition,
what would happen would be,
they'd reach out and somehow miss,
and we'd see a kink in their trajectory,
and they'd have to reach back and adjust
and grab the cube.
This did not happen for the
world warping situation,
and what was going on was that,
so they'd see a virtual cube,
a visual target is onto the right,
and not the left, the physical cube
was actually located on the left,
so if you split the virtual
and visual representations,
what's going on is that
they'd see the visual target,
they'd do this quick ballistic
motion, to reach and grab it,
and then they'd see their hand miss
because of the shift that was applied,
and they'd come back and readjust,
and grab the virtual cube,
but if they move their hand slowly,
then they would gradually
see things shifting,
and then they'd trust their
vision, over their body,
and grab the object and grab it properly.
What did we learn from this?
Essentially body warping
definitely guarantees alignment,
because the user's
definitely going to need
to reach out and grab the object,
so there's hand motions that always,
that you could relay
on to do body warping.
On the other hand, you
would have to do it slowly,
because if you reach out you could miss,
and that could effect
people's performance.
World warping, on the other hand,
it requires you to do head movements,
which you can't necessarily
rely on always being present,
but if you do manage to get it,
things will line up perfectly,
before the users reaching in
would have no mistargets.
Hybrid warping is the happy medium
that's trying to bring
the best of both worlds,
and mitigate the artifacts they have,
that each of these
techniques used separately.
In the future what we want to do
is investigate detection threshold
to see for each of these techniques,
how much can you get away with,
without them noticing,
or to which point can you push them
without affecting and
ruining their performance.
The demo that we showed you here,
this study that we had,
the user was being guided.
We knew which cube they're going to grab,
we knew where it was going to go,
but we also had a few demonstrations
that we played around with,
trying to predict what
the user's going to do,
based on their hand
motions and their speed,
and try to have a predicted method
instead of just forcing
the user to carry out tasks
like a robot.
We also want to look into
being able to map different
types of objects to each other,
instead of only mapping things
that virtually and physically
would have the same shape.
We also want to look into advanced ways
of warping the world,
because we had access to the
mesh rendering of the user,
and we can essentially manipulate things,
and shift things around,
without them knowing.
In summary, again, hactic
targeting allows physical props
to be reused for hactics.
The key idea behind this
is we trust our eyes
over what our body is telling us,
and using this, we can have a single prop,
repurpose it to allow
movement and stacking
in a virtual environment.
If you haven't had a chance
to check out or interactivity demo,
definitely try it out, and I assure you,
the stacking is really magical.
Thank you.
(audience clapping)
- [Male] Hi, Rule Vertig,
Alquents University.
Very nice work.
Have you tried this while,
being ambulant, walking,
and what happened, because
you're kind of screwing around
with the x-probe perception,
so I would argue that
it gets a lot harder.
- When you're walking around?
- [Male] I understand
that this is hard to do,
with a physical object,
but you can imagine taking this task
into a more mobile space,
rather than being seated by a table.
- Actually, in the redirected
block, touching worked.
That was done.
They also looked at, what
if you can walk around
and redirect people, and come back in,
and grab the same object.
Does that make sense?
- [Male] Yeah, exactly.
- To line up things again, and again.
Yeah, so they haven't actually done that.
That required you to walk up.
We actually are focusing on the case
where you're having a
seated VR experience,
which is probably what
we're going to be seeing
for the near term future,
so they have looked at
doing a little bit of that,
but they didn't do the exact same type
of warping that we're doing.
It's actually our next step,
that we want to put it into a vibe,
and have you stand
around, and walk around,
and react with objects.
I mean if you're standing,
I don't think it should be a problem,
but when you start walking
we can use redirective walking techniques.
- [Male] When you start
shifting the world,
then you might not end up at
the same spot, for example.
- Yeah, well that's exactly
what redirective blocking does,
and there's nothing wrong
with it necessarily,
as long as you don't have people
walk into targets that they don't see.
- [Male] Right, thanks.
- Absolutely.
- [Male] Nice talk, I really enjoyed it,
and even I tried the
demo, and it was awesome,
but my question is, essentially
why doing the hand warping,
you need a delimiter action, right?
Is it possible to have
a continuous engagement,
and still try to do the same thing?
- What do you want delimiter, exactly?
- [Male] As in disengagement,
and disengagement has to be
there for you to do magic.
- Correct,
So the thing is the way I
imagine things being done
in the future.
Let's say you really want
to make a mindcraft castle
on your own.
I think you would have
say two, three, four,
physical cubes on the table,
and they reach out and grab this here,
and then for the next
one you want to grab,
you grab another cube, and
then that allows disengagement,
and then engagement again,
so being able to create
a castle on your own,
a single cube, I doubt that
will be really possible,
but if you have three or four cubes,
and if you make a castle
of 100 cubes out of that,
I think that would be pretty awesome.
Does that make sense?
- [Male] Hey, thanks for the talk.
I was just wondering, can
you give some intuition
how far you could go with radial movement
instead of tangible movement.
- Radial movement.
We actually did a perceptual
study before all of this,
to have a sense of how much
we could get away with,
the most sensitive people,
you could laterally move them
like eight centimeters, ten centimeters
without noticing,
but some people, I'm
talking about the average,
more proficient users.
Angularly speaking,
again it's sufficing, I
was like around 8% to 10%,
is also how much you
could do for proficient.
Somehow 8 and 10 is the magical number,
but I've gotten away with 20.
People didn't bother at all, but yeah,
I haven't gone beyond 20%,
and people definitely start
noticing once it's beyond 20%.
Something I didn't mention was
one thing you have to be careful about,
is how you're applying
these manipulations,
and how it's effecting other things,
like when we were applying certain shifts,
we made sure that the user
wouldn't see their hand
detach and go away from them,
so instead we would
rotate around their body,
so they would at least
see the continuity here,
so there's other things that
start to become a problem
once you want to do a
lot of manipulations,
even if they don't notice,
they're going to be like
okay why is my hand
coming out of my throat?
Do you know what I mean?
Or out of my mouth, do
you know what I mean,
so we have to be careful
at the kind of things
you can do with that.
