hey Ron its norm from tested and welcome
back to projections where this week
we're gonna be talking about augmented
reality I had a chance to recently try
tilt 5 these are the augmented reality
glasses made by Jerry Ellsworth and her
company and based on a technology that
she's been developing since she was at
valve software she took that tech and
launched a product called cast AR about
six years ago now and since then has
been iterating on the hardware on the
software and on experiences to now
launch a new product called tilt five
and I had a chance to chat with her
about what's changed since Cass a are
what's in this new hardware with type
experiences they're developing for
tabletop AR and all that good stuff
let's take a listen
well Jerry thank you so much for having
us here looking forward to this demo for
a long time now I know you've been
working on this for years and years and
years so the latest iteration of this
technology and what you're working on
it's an AR headset called tilt five I'll
say I'm with the company for people who
might not know and may have heard
rumblings about like how a our headsets
work how does tilt 5 work let's start
from the basics so our system is kind of
unique compared to other systems our
system actually projects out to a
special game board which is called a
retroreflector and that allows us to
make the headset really high performance
lightweight super wide field-of-view and
super comfortable for the user so part
of the the system is it projects out of
these silver lenses here down to this
route this retroreflector and it comes
back to each user and that allows the
focus of these pixels to be correct so
if I put my finger next to a virtual
object it's in focus and that's super
unique to our system a lot of other
systems have a fixed focus and you can't
focus on real-world objects and the
virtual objects and then also the system
is super lightweight so it's just 85
grams it just weighs nothing it just
folds up and slip it on so our main
objective is to have a system you fold
the game board out slip the glasses on
and just get right into fun you know no
calibrations no complicated setup no
sensors in the room it's a fundamentally
different approach
- AR yeah then what we've seen out in
the marketplace before where people have
used lenses essentially a projector is
on so two waveguides lenses which let
you put objects into real space but have
limitations like field of view yeah like
the accomodation vergence problem where
things don't look like they're in focus
or you have fixed and a plane right
there because you're using the
retro-reflective and essentially
projecting out and back end you solve
for a lot of it yeah exactly I mean we
like I mean I love the systems that are
like AR anywhere those are really cool
but they have limitations we decided to
do AR somewhere it's kind of our
internal way we think of it's like what
can we do and what can we just nail and
make perfect and so with this system
like playing with your friends like head
to head or play with your friends
virtually or even solo games like we can
nail this and it's gonna be a great
experience
it's omit the tabletop game board it's
yeah it's having this board here which
is about like 2 feet by 2 feet where
you're calling for with the tracking
markers around it can be any board game
yeah that you would buy the game store
that you were a gen con' recently people
were flipping out rare or it was awesome
you have D&D in there you kept our
defense you can have single-player games
multiplayer games action games is so it
doesn't have to be board games it can be
like you know never before have you been
able to sit across the table and play an
action like war game with your friends
and that make it really heightens the
experience so like if I like pull up
your base I get to see it in your eyes
yeah and that's that's pretty unique and
all that software is kind of defined and
parameterize by the hardware that you've
yeah iterated toward we talked about you
have this nice yeah I'm here view of it
it really helped helps illustrate I
think how this projection system works
they look at the projectors and it's
really novel that looking at here the
projectors actually go toward your eyes
bounce yeah yeah so these lenses here
these are actually really special lenses
I mean they just look silver and kind of
plain but there's actually a lot of
layers in there this is one of the
inventions I did pat on my back it's
like this actually controls polarization
and the way that the light
gets reflected on the inside so that you
get 85% efficiency when the light goes
out and back again where if you just had
a silver mirror you'd only get 25% of
the light so it's a very elaborate
high-tech one-way mirror system yeah
that doesn't work the polarization to
make sure that maximum light is
transferred to the user but it also
isolates each eye so you get this 3d
experience and it also helps isolate
between multiple users so you can have
any number of users around there and
then there's two projectors no the two
projectors shine down onto these lenses
and we've gone through like five
iterations on these and the new ones are
fantastic they're 110 degree field of
view super high fidelity and in really
small and light like even between this
iteration in this iteration we shrunk it
down even smaller it's like two sugar
cubes worth of like volume I'm looking
around you know your office you have all
these parts everywhere so a projector
like this it's not something that's
off-the-shelf like you know people
always think that we just get it off the
shelf like no this is like four years of
development to make this projector
because you can get small projectors but
there's you know very narrow field of
view they have a very shallow depth of
field and so we had all these
requirements we had to have like InFocus
over a large depth we had to have a
massive field of view had to be super
efficient so that we could run off you
know cell phones and right right when
people think of like pico projectors
like those would not work great for this
but now a lot of those projectors get
too hot thermal was a problem for us too
so we had to come up with solutions so
that we get the efficiency of the LEDs
up enough that we weren't heating the
LEDs up and heating the headset so yeah
a lot of work went into the projectors
and a projector like that's one
necessary work well as like no if you
point this at the wall it's only 0.65
lumens or point at your hand you can't
even see right but because of kind of
the amplification of the retroreflector
how it just takes all the light and
brings it back to each user you get the
super-bright image like we were demoing
at Gen Con and it was probably the most
harsh environment we've ever convention
lighting these huge like mercury vapour
lamps yeah it was super bright and
people are like wow I can't believe how
bright this isn't
yeah and the objects look opaque and
yeah really
are you talk about resolution and these
are HP so they're they're 720p
all right and then you're refreshing
them at a 180 frames per second 80
frames a second which then needs to talk
about how you're processing all that cuz
games then running at hundred eighty
frames no they don't have to no no
that's a really cool part of our system
and it took us a long time to get here
it's like I dreamed about this like
years ago when I worked at valve I'm
like in the future the game engine is
going to run asynchronous from the
actual projection system and in the
headset it's going to do reprojection to
realign the image and do all the
prospective transforms that need to be
done and so that's what we've achieved
we have a chip on here that receives
images over USB it lands in the chip and
then the chip upscales it to 180 frames
per second and does all the reprojection
so as you move your head back and forth
or your shift side-to-side the image
stays locked to the game board and it's
updating at this insanely high frame
rate in the VR world we hear about
reproduction all the time to compensate
for lost frames and it's very difficult
you know spatial projection for from a
lot of movement this is all hardware
base
yeah the loop the entire loop is in the
headset which is really unique like VR
systems are just brute forcing it by
having giant video cards so what this
means is that we can have multiple
headsets running off one PC and as long
as the framerate it's good enough and
you're happy with the animation right
it's running at 60 frames you could have
two instances of games running at 60
frames or four at 30 frames per second
and for your perspective as long as it's
comfortable as long as I'm used updating
and the reproduction again is updating
then you're usually oh yeah you won't
have no discomfort yeah you won't have
what's what's bad in VR and AR system is
when the tracking goes bad and you drop
a frame then awesome the image like goes
flying off somewhere because you had
head motion this happens to me all the
time I'll be sitting on my desk working
with unity and I'll crash unity and I
won't realize it I'll be looking at the
game board waiting for something to
update
it'll be like come on come on oh you
don't notice it because the last image
is right it's running basically at zero
frames
get the last one frame right but it's
still smooth it's just no I should have
shown you that we'll have to fire it
back up I'll show you that oh it's neat
very cool and then then the tracking
these are the markers oh you're doing
positional tracking of that way say
inside out and what's unique about our
system on the tracking is we have two
cameras that are right here so one of
them is 140 degree field of view
infrared camera so you can play at night
you can play full daylight so anywhere
in between because we illuminate the
game board with infrared light and
that's on a specific wavelength and we
filter the camera so the camera is
another custom-designed piece has a
bandpass filter to only respond to that
wavelength of light then we have a
second camera in here that runs on a
totally different wavelength of infrared
light that's for tangible so playing
cards and tracking complicated object
tracking like tracking miniatures on the
table and that recognize that's
basically computer vision yeah yeah
recognition you can be as simple as QR
codes shapes and your your one control
there yeah all that stuff is
positionally tracked because you have
that inside out track yep yep man it's
an 8 megapixel camera so it's super high
res so we can pick out features like it
really far distances with it and we
talked about the the tracking field of
view being wider than your visual field
with you which so I think you were
trying that out you were like clear off
the game board yeah still tracking it
yeah and then also you know you did a
thing where you brought a extension to
the game yeah you're also rendering
beyond this because you don't know the
user the system doesn't know where the
game wardens necessarily except that
it's somewhere within that kind of big
field of view and so I was able to look
in and see as far yeah into the world or
even extending the work yeah yeah if you
just put another piece of a game board
here like a taught another tile and your
your game board just extends and part of
our offering will be will have a base
kit which has this fold-out rectangular
board just easy gives you everything you
need but then we'll have the deluxe kit
that has like the tiled system that you
can do really fun stuff with like make a
rectangular board if you're doing D&D
right or if you want to have like lots
of vertical you can have a little
kickstand so you can tip it up and you
can have all this vertical height out of
the game board so that that's that's an
interesting thing because
extending it I get like you're having
more surface area which then you can
look deeper and more into it but having
an angled game board it doesn't matter
what angle it is because it's all about
we can't even tell if you angle this it
just it it's seamless
all you do is just get more vertical
it's it's a little difficult sometimes
to understand what you're gonna see when
you look at this system a lot of people
are like oh you can never project
anything above the game board which is
not true at all
you know us sitting here playing a game
side by side I can have objects like
this tall off of the game board and it's
great it's a great experience all about
the angle that you're looking at the
game board form kind of like you know at
a 3d movie things are still coming out
at you both in and out but because it's
wide enough that the angle works and you
can have sterile images that far apart
yeah yeah so on this system you can have
like I can't see anything taller than
that but I can see stuff this like in
the center of the game board it's like
this tall easily but if you had a
requirement where I wanted to have
something really tall I just took the
the pro board up so there's a sweet spot
in terms of where objects are placed
yeah right in front of my front which
tends to be where you're looking anyway
yeah and what's funny is people are like
well you know it's gonna clip on the
backside sometimes and it turns out like
we play video games all the time that
clip 3d stuff in the back and it's like
it it just works like your brains like
oh yeah that just went off the screen
and it feels right well in terms of
sweet spot - one of the things we
mentioned earlier is that accommodation
verges I was able to hold a cube up
focus on that cube and then focus on
other cubes on the game board and go
back look for and go back and forth
immediately you're nothing that's
changing the rendering what is it about
this projection system the retro
reflective material that allows for that
to feel real natural well it's a
combination things it's kind of
complicated but it's the aperture of our
projectors which send out really tight
ray bundles and then also retroreflector
these ray bundles will be going out in
like a cone and it'll be focusing but
since retroreflector brings everything
back again it refocuses them and it
brings them through these intersection
points which are correct so it's like
we're kind of through a clever optical
trick we're making an entire leg field
display without having to compute it and
that works specifically for what
you've experienced the sweet spot for
tabletop yeah you are at that you know
within certain number of meters and like
as close as you want those no minimum
focus distance at GenCon we actually had
people like bumping their head into the
table like Oh careful wait when you were
first experiment with this I mean people
imagine putting this everywhere yeah
right that technically what good work
yeah oh yeah yeah we are Maine kid it's
gonna be like something that you can
deploy in the home easy just flip the
game board open slip the glasses on grab
your wand and start playing but yeah
there's some folks that are taking our
system and doing really crazy things
like huge domes for location-based
experiences so you know all your friends
in this dome you can see each other have
a really wide field of you you know and
sharing this virtual experience but
again having the parameters of tabletop
that allow the developer to work yeah
they all get the same board it's a
language that exists if you understand
your four sides you can do multiple
players so what are some of the game
experiences oh well we're really excited
for our announcement hopefully this is
going out after our Kickstarter and
fantasy grounds is coming to our system
so for the DeeDee Pathfinder crowd
they're gonna love this so you can have
your whole virtual world laid out in
fantasy grounds the DM gets their own
unique view into the space the players
sitting around the table don't get to
see what's coming until the DM triggers
so the the DM will be able to do things
like an eerie mist settles in the valley
and that's they wave their hand over and
eerie mist will come out of their hand
and it'll be revealed this fog of war
will be lifted to the other players and
if your friends can't login or can't
join you around the table they can login
and you can connect your game board to
their game board and so in the DM waves
there and over they would see a virtual
representation of the hand and the eerie
fog you know they do a trigger and the
monster Springs out because that's true
asymmetrical play but yeah people are
using the same game board so the game
knows there's gonna be someone in that
position it actually doesn't really
matter where they're sitting it as long
as not rendered properly for them and
some of you can be playing with someone
yeah so the DM like could see like well
here's a trigger point and there could
be stats and there could be a monster
sitting here waiting to jump out and the
other players are oblivious it until
that moment you trigger it so it's gonna
be great for storytelling so we're super
excited about that partnership what's
also exciting about that is their system
is truly cross-platform so if your
friends don't have a tilt 5 system yet
you can play on your PC or tablet and
still participate there's the base the
underlying technology and those systems
are all the same yeah yeah roll they
bring your ass they have adventure packs
that you can download you can bring in
3d assets and work with them yeah that's
super exciting so that's the D&D crowd
we have a bunch of action games that are
going to be shipping with the system so
we have a bunch that we have which are
kind of party action games so you have
your friends sitting around the table
you have shooting blocks and trying to
grief each other and and do head to head
then we have some solo play experience
we have this racing game that's a very
Mario Kart like and then we also it has
a battle arena mode which is really fun
so you're all in this arena it's got
these spinning discs and stuff and you
try to shove your friend's car it's kind
of a rocket League like very cool yeah
and these are all essentially just like
PC games or a mobile game because you
can plug this into yeah Bowl device for
PC or yeah many of them will be on PC
and Android devices they just go to
Google Play or you go to steam and you
download the game and and you're good to
go
we have a couple developers working on
true card games so locomotives is one of
our partners that's working with our
brand spanking new hardware brave souls
and it's looking really good you got a
little sneak peek of that and so that
one's like a Wild West theme where you
can have up just I think six players
they can be either remote it's truly
cross play also so you can play on your
tablet or your PC or on the tilt five
and it's really up to your developers to
choose that they wanted to be really
graphically intensive something with a
lot of network play or something that's
very single-player focus that's kind of
up to up to them exactly
yeah we're just we're making a tool you
know and we want to put enough you know
tools in the toolbox for you know the
developers to make these rich
experiences so
that's why we thought like tracking
physical objects and hands and and cards
like super critical you know great
tracking great field of you have them
all target a game board so everyone has
the same experience no matter where
they're at you have all the input-output
you have speakers oh yeah we don't even
talk about eleven yeah and this is all
essentially whoever gets one of these
can be a developer yeah yes
so each kid is a developer kit so our
SDK is super simple like I'm just a chip
designer by trade and I'm writing unity
demos every one of those like kind of
cheesy demos that I showed you today was
just like a camera test demo that I put
together but our SDK you just take our
plug-in drug it drag it on to the camera
and unity or unreal and then that gets
your head tracking and then if you want
to use the wand
you know the buttons and triggers is
just standard unity input and then for
the six degrees of freedom you know it's
it's straightforward just hook that to
your game object like you can get up and
going in like 10 15 minutes with our
system so you got this software side the
Harvard sites being iterated on this is
kind of like using production molds now
yeah injection molded plastic you're
getting down to your ID
everything's getting finalized what's
the roadmap to getting this out
delivering on the cake for starters yeah
our timeline is like we already have
some developers working with our kind of
pre-production units we're gonna have a
canary group or kind of an early beta
test that's like q1 yeah yeah it's gonna
be a smaller group of people that love
bug hunting this way goes and then
general availability like the the bulk
of the Kickstarter units will go out q2
you know GDC time absolutely or a lot of
helpers gonna be interested it really is
one of those technologies that you have
to put on and yeah say that about a lot
of the RNA are technologies but like it
is don't know how good it looks until
you put it on so we're really anxious
about our Kickstarter video we're super
proud about it we're all but also
wringing her hands because we did
compositing in it because to show the
experience you kind of have to like
fudge it a little bit and like you have
ly slow camp camera angles and you have
like stuff flying off the board which is
true for the people that would have been
that are being
it's not necessarily true with him you
have a camera that's like five degrees
off the board or something yeah but what
we're doing is we're taking all of the
composited shots and we're doing like
side by sides of like here's film
through the glasses and here's the
composited shot yeah so hopefully people
will give us a little bit of a leeway
because we're trying to sell the
experience because it really is magical
when you try it and if you absolutely
and having used a bunch of VR headsets
it really is it's a different flavor of
it and I think the tabletop aspect is
super super cool yeah yeah
congratulations with the launch of the
Kickstarter I know it's been a long time
coming yeah and thank you so much for
having here to check it out thank you
that was a really enjoyable conversation
for me and I want to thank Jared for
being so open about the technology in
these tilt five glasses and what's
changed since they worked on cast a are
and the challenges going forward as they
finalize the hardware and also work with
developers on the software when they're
ready to push this out consumers early
next year now did get a chance to use
the latest prototype of tilt five for a
variety of tech demos that Jerry showed
me and first thing on the hardware it's
super light they're pointing this as
under 90 grams and while I didn't have a
scale to to measure it it's definitely
the lightest of any VR or AR headset
I've used in recent memory it fit
perfectly over my glasses of a plastic
like I said is looks polish and I think
in the version I had there was a little
band that could tie so the tighten it to
the back of my head and I wanted to do
that but once the glasses were on a
couple things I immediately noticed one
it's a are so I want to see the outside
world in the outside world is very
visible not only the field of view to
the outside world which some other era
headsets want to narrow that because
they want to hide the fact that you
don't have that much of an AR image
either but also the clarity of the
outside world you know yes it is a
little bit darker a little bit tinted
then looking through normal glass or
plastic but I could interact with people
I was looking at things and I didn't
feel the need to take off those glasses
to read something to look at something
you know it is how I think a
should be was also really comfortable
was the imagery itself and all the
things that Gerry talked about in terms
of the projectors being really bright
even though they're running on a very
low power because the retro-reflective
materials balancing that light directly
back into your eye the imagery of the
augmented objects was a vibrant was
colorful it was sharp and also had a
really nice frame rate and field of view
I did some tests where I moved my head
left and right so I was going beyond the
view of the game board and even when I
was looking off to the side of my head I
could see the game board
I actually still see the images so that
110 degree field of view for that those
projectors really helps and really is a
different type of AR experience if
you've used something like a magic leap
or a hololens the smoothness of the
motion of the the graphics and the
objects that were on the board in the
board or floating above the board also
made this really comfortable I know
Jerry quoted about a hundred and eighty
frames per second for the projectors and
I'm not sure if that's alternating
frames Nighy on each eye or if they're
actually 180 on both projectors but
regardless the imagery was was super
smooth and I after I interviewed her she
did give me that demo again where she
crashed unity and the image was locked
in place I almost couldn't tell that
there weren't any additional frames I
didn't notice any drop frames at once
that reprojection engine they have
inside the hardware to do some
essentially a time warp and space warp
and skewing of the images works now of
course if I move my head dramatically a
frame or did some crazy lateral movement
did look skewed and it wouldn't look
right so you're never gonna be running
games at like five frames a second but I
can imagine if you're running at 30fps
or even slightly lower the graphics will
still look rock-solid and the images
will still be locked and will never feel
like they're not a part of the real
world even though there's projected
images something I was also truly
impressed by is that seemingly the tilt
five technology
solves for the accommodation virgins
conflict the idea that these objects
wherever they may look like they're
floating in space will be in focus as
you focus your eyes on them not just
because of the stereo overlap I was
looking at these cubes floating above
the tabletop surface and as well as
those spread around and really move my
head around them and shifting my eyes
focus from one cue to the next and
everything looked like they were in the
right place it's an AR experience that
you don't get out of waveguide bass
headsets right now and I'm still trying
to wrap my head around exactly what type
of optical trick allows for this to
happen because they're not generating a
light field they don't have that type of
rendering power on the system but
apparently it's because of the way the
projector it's shoot all those rays of
light onto this micro surface of this
retro reflective material and the way it
bounces back in a way the combine at
different planes of focus allows you to
then shift and then see the objects at
those variable frames in that sweet spot
which they've designed for it was really
cool once one of those things that you
really need to try and it's a thing that
you don't even think about because
things look like the way they're
supposed to in the real world at least
in the right distances as the way
they're supposed to
now the compute stuff is also
interesting the demos I saw running off
of a laptop and as Jerry mentioned you
could run multiple tilt five headsets
off of a single laptop just based on how
much processing power you have on the
laptop what frame rate you want to run
the games at and also how many USB ports
it'll just be USB see right into the
headset and so the benefits are of
course you'd only need one computer
really to do multiple players it's gonna
be really open they're not doing their
own version of an app store just
download the games from Steam or the
Google Play Store and also the games can
run at different resolutions and then
some of the demos I noticed that yes
some of them weren't necessarily running
at native resolution I could see the
tell-tale signs of some anti-alias
saying some of the jaggies but again the
movement was so smooth and then some of
the games that did look like they're
running at native resolution that 720p
per I was really sharp in terms of the
games the demo
I did try were representative of some of
what they're going for so there was one
that was like a marble rolling down a
core so it was really cool to be able to
kind of look into the course and even
put some of the retro-reflective
material up and see beyond the tabletop
and see how much they're actually
rendering their example of a tower
defense game where I then it was able to
throw some tracking markers with QR
codes and then that use that to activate
or point with the wand controller and
activate things on that play field and
one of the cool things is because there
is computer vision there's optical
tracking of markers they could also do
printing of the markers that's
transparent are not visible to the naked
eye the visible spectrum so it can be
something like a CCG car you know a
magic card but underneath it there's a
layer of a QR code and so you're getting
real true mixed reality with things that
play in the real world but also work in
that salt 5 system their partnership
with fantasy grounds I think has a lot
of potential because that's an
established platform with licenses for
RPGs like Dungeons & Dragons and so
there's an existing player base so the
existing software stack and it's cross
platform so people with tilt 5 will be
able to play with people who are not in
the same room with them who may have
tilt 5 sets or may just be playing on a
traditional PC or tablet and so that's
gonna potentially expand the user base
and I think it's really smart of them
not to design an air system to try to
solve for a are everywhere you know with
the complexities of SLAM and mapping and
optics that work outside and inside
they're trying to solve for a our
designer a our system with these
parameters of it being a tabletop gaming
system or video traditional video gaming
system I would love to play a Diablo
like RPG or a gauntlet like RPG on this
2x2 game board and look deep into a
spiral staircase and and play with
friends in the room or not in the room I
can't wait for the system to come out
there's so me a little time before they
actually get this out to backers and so
they have time to work on those software
experiences working with their dev
partners it's a great a good launch
suite
games and experiences and also to refine
that Hardware and further I think
they're still tweaking I am you and the
software related to that but I really am
excited for this and can't wait to play
this when this comes out and I'll be
backing their Kickstarter and looking
forward to that release and if you have
questions about it please feel free to
post in the comments below but thanks
for watching we'll be back next time
with more coverage of VR and AR
technologies and I'll see you next time
