- So, as Michael said, my name
is Vinoba Vinayagamoorthy.
But the only person who
calls me Vinoba is my mother
and only when she's pissed
at me, so Vino will do.
I work at the BBC, as I said,
and my collaborator on this
paper was Maxine Glancy.
This project is actually a collaboration
between BBC R&D and IRT,
who is on the right there,
I've kind of pushed them outside.
Christoph Ziegler who is with
me here is also a co-author.
For those of you in the
crowd who don't know about
the BBC's research and development unit,
we essentially look at technologies to
future-proof the BBC.
And IRT does the same for
the public broadcasters
in Germany.
And this problem space is
essentially about putting augments
using AR technologies for television.
It is not a new problem space,
the idea has been around
for quite a while.
But where we defer is I
suppose the technology
has moved on to allow the
video frame synchronization
between a television
and a connected device
in your home network.
So, our general problem space has been
as a public broadcaster,
how do we let our audiences
watch, listen and experience
everything that we make anytime, anywhere
and in any way that they want.
So we have a background of
work that has been going along
where we are looking at a wide range
of platforms and and devices,
because we have to serve
pretty much everyone
with whatever devices they might have.
We like to create well-designed,
accessible user experiences
and we tend to select technologies
that might maximize these
services in a cost-effective way.
We also like to work in
partnership with others,
so IRT is a very good collaborator of ours
for a lot of projects.
And we tend to do, in addition
to the sort of paper writing
and distributing of our findings
across academic conferences,
we tend to do a lot of
standard specification work.
The DVB Project for instance,
is one for the digital
media broadcasting projects,
and we have worked with industry partners
to ensure that there's
a way for televisions
to talk to devices on a home network.
Through the companion screens
and screen specification.
This means that any TV generally in the UK
that has the Freeview Play patent on it,
has the potential to have
a profile of HbbTV 2.0
which will allow synchronization
between a connected TV and devices.
And so, as part of a new
project called 2-Immerse,
we wanted to see how far
we can push the protocols.
So, before I go on to the
actual thing that we did,
here's a video that shows
us having two video sources
on two separate devices.
And in a few minutes
you'll have a split screen
where you'll see that
the top left-hand side
shows the sort of timing
information there,
and it's pretty close,
it's frame accurate.
I have decided to spare
you the inconvenience
of listening to the beeps that goes with
this calibration video.
But it's very exciting stuff,
we had a standing ovation in
the office and everything.
It's also a video I've
shown over and over again
for the last two years.
Anyway.
So, in this particular project,
our objective was to see how
can we use AR technologies
to deliver something quite important
to the BBC Access Services.
Our sign language interpretation
for our television programs.
Now we are the R&D component,
so this is not going to
be a service for the BBC.
That is not at all what I am saying.
But we wanted to see if
this was a potential thing
that we could do.
So it's essentially a, you know,
passive non-interactive experience,
where you sit on our couch
watching our television program
and you have an optical
head-mounted display,
in this case a HoloLens,
not for any special kind of purpose,
the HoloLens was the first HMD we had
and IRT had a HoloLens as well,
so it seemed like a good device to use.
Our research questions were:
can AR be used to augment
TV programs using DBBCSS?
What should AR sign language
interpreters look like?
I mean, the idea is there
but no one has really designed
a sign language interpreter
outside the television.
And what do users with hearing impairments
people who would be
our target demographic,
what do they think of our AR
sign language interpreters?
And figure out the limits and potential.
So, this image hopefully gives an idea
of what we were working with.
The top column shows the
sign language interpretation
sort of invision,
which is essentially the traditional way
that sign language interpreters
are shown to the UK audiences.
The bottom right, on the far
end, has the German version.
We conducted this project
in a sort of balanced way
between the UK and Germany.
We also designed two AR interpreters,
each for each of the countries.
Our half-body interpreter,
that's the middle images on the slide,
and then the full-body interpreter.
They are both gender matched
because we had an English
program in Britain
and a German program in Germany,
and we had a female and a male narrator,
so we gender matched to the narrator.
Before we go on further,
I wanted to give this
talk with the sort of idea
of telling you things
that I've not actually put in the paper,
because you know, they're kind of funny
and not quite scientific,
but I kind of think it.
When we were designing the AR interpreter,
we got to play a bit of dress up.
Now this is an email that
I think I could send off
to my signer,
she was very instrumental
on trying to figure out
how to build a AR sign interpreter.
Now there are a lot of
guidelines in the UK
for how to build one for
the broadcast industry,
but no one has actually
thought about how to do it
if you are going to project
it on a wall outside the TV.
So, essentially I'm
telling her that, you know,
these are the colors that
we have a chrome-keying
technology which works
with the blue color.
If you have black hair,
that's gonna be an issue,
please don't wear white,
don't wear light blue,
don't wear navy, don't wear
jewelry, don't wear earrings.
There's a lot of information
that we had to get across
in a nice way, of course.
But Dominique was quite
instrumental, as I was saying,
and she was happy to ask questions
and come back and say, you know,
"Would you like me to
buy a couple of outfits"
"and come along?"
And so we had to have a
dressing area that she sat in.
We had a little bit of a mannequin play,
dress up kind of thing to
see the right sort of outfits
she could wear.
I don't think the guys
in IRT had this issue.
So they had a green screen,
as shown in this image,
and, you know, he just
didn't have to wear green.
He also doesn't have to tie his hair up
or no jewelry problems.
So it's a little bitter thing.
So I just wanted to show a little bit
of the conditions themselves,
essentially the stimulus
set up in Germany.
So this is the view of
the traditional invision
through the HoloLens.
Participants were not wearing the HoloLens
while they were looking at
the traditional invision,
but this is not an image
I can put on a paper,
so I thought I'll put it up here.
This is the half-body
interpreter in close-up.
As you can see, she's
sort of near life-size
and kind of fills in from
the top of the TV frame
to the bottom of the TV frame.
And if you're looking at
it through the HoloLens,
then she's sitting on the TV stand.
Doesn't look like it on this image,
but take my word for it.
And then this is the
full-body interpreter.
She is grounded on the floor
and she goes all the way
to the top of the TV frame.
So this is a video that we have.
As you can see it's very
similar to traditional signing.
There's a resting phase
where she's looking at the television
as if she's watching it with you.
And then when she starts interpreting,
she starts talking to the viewer.
And I love her,
I mean this is my favorite
condition I think.
She looks very much as if she's there.
I mean I don't use sign
language on a regular basis,
but the minute I saw
this I got to thinking
surely this is going to
be the best condition
out of all three.
Oh, sorry.
That video doesn't play
automatically because, you know.
Anyway, so this is the
half-body interpreter,
and she's also quite larger than life
and she's quite there, but cut off.
The full design of the
interpreters are available
on a work in progress
paper from TBX last year.
But this is a summary of
the guidelines that we used.
The heads of the AR
interpreters are aligned
to the top of the TV.
The bottom edge of the
half-body AR interpreter
is aligned with the
bottom of the TV frame,
almost sitting on the TV stand.
The feet of the full-body interpreter
are grounded to the floor
of the physical room.
The direction of the gaze
of the AR interpreter
has to be horizontally
facing towards the TV
while in resting phase
and has to be directly
facing towards the viewer
during the interpreting phase.
And positioned to the right,
just outside the TV frame
with a very slight overlap over the TV
in order to minimize distance between
the content on the TV
and the AR interpreter.
And these are all guidelines that we took
after talking with our pilot candidates
and also the sign language
interpreters themselves.
And then we designed the study.
The study itself again was mirrored
in both the UK and Germany,
so we had a training phase initially,
we don't have the weather
program that we used,
but we had a weather
program where participants
could sit with us and talk
about the AR technology.
They could whinge about how
heavy the HoloLens device was,
they could talk about how
they had to keep their body
quite straight,
essentially get their
frustrations about the technology
out of the way.
We really wanted to focus on
the design of the interpreters
and not on the limitations
of the technology.
So we had a training
phase where they could
kind of get used to
the whole aspect of it.
And it took a little bit of time,
longer than we would with people
who are more used to the technology.
But I think this was an important phase.
The programs matched the
languages of the country,
so you know, we had English
and British sign language
in the UK, BSL.
And German with Deutsche DGS in German,
I cannot pronounce it
but Christoph over there
will be able to do it.
I'm just going to call it
the German sign language.
The gender of the interpreter
was matched to the narrator,
and each participant who
came to us saw three clips.
The clips were themselves
on a continuous storyline.
So that was the first clip,
that was always the second clip
and that was always the third clip.
But the sign language
interpreters themselves
were randomized.
So people could, for
instance, see the traditional,
followed by the half-body,
followed by the full-body,
or traditional followed by
full-body and half-body,
and all the other permutations.
But the storyline was kept intact.
There we go.
Our participants all
had hearing impairments
and they were all regular TV watchers,
about more than an hour a day.
They had no biases against
factual documentaries
and they didn't hate public
service broadcasters.
I'm not sure about Germany,
but in the UK there are
folks who hate the BBC.
I don't know who these folks
are and I don't know why,
but they're there.
So we asked for people who did not
particularly have that feeling against us.
The folk who we invited in the end
stated that they had a
preference for subtitles
because it was perceived as more accurate.
And they used it to improve
their written language skills.
But they did think the sign content,
the signing made it a
more comprehensive thing
because you had a lot of
emotions and things like that,
which were not captured in subtitles.
And this was especially
the case if the participant
only had a basic command
of the written language.
You have to understand that sign language
is a language in itself
and it's the first language
for many of the folks
in the community.
All the participants
understood the sign language
and unsurprisingly participants mentioned
that they could only lip read on TV
if the subject was facing
outward, facing them.
All participants also reported
they were expert-levels
of familiarity with computers,
but only one of them in each country
called themselves an AR expert.
This is essentially the
demographic information
that we have.
It's nearly sort of gender balanced.
We have quite an older
sort of range of folks
who came to us.
And the folks also had different ways
of describing themselves.
So the profoundly deaf,
hearing impairments,
hard of hearing, all those are key terms
that they used to essentially tell us
what their particular impairments were.
We also have collected information
about how proficient or fluent or basic,
whether it was a first
language or the main language,
and all those sort of situations.
Extra details all in the
paper, so please have a look.
We then had some post
condition questionnaires.
After each of the condition
in the three conditions,
we had questionnaires to
ask them what they thought
of that particular condition.
And we also had a post study
questionnaire at the very end
to compare in between conditions.
And we had an interview.
But the interviews was
more conducted in parallel
to the interviews, the post
study questionnaire interviews.
So whereas we would have essentially got
folks to sit down and do a
questionnaire by themself
under normal conditions,
for this particular study
we chose to help them along.
So we would sit down there
and conduct the interview
and get their responses in
person and type it all out.
We had some findings of mild interest.
Again, it's in the paper.
But in the UK, the responses
on whether they liked it
was split three ways.
For the questionnaire data.
And in Germany, it was sort
of split nearly halfway
with preferences for the
traditional winning out,
with then the half-body,
and one person said they
liked the full-body AR.
The more interesting findings came
in the qualitative
information that we gleaned.
So novelty factors.
We always have factors that
they wanted to improve on.
So for instance, participants
always had opinions
or expectations of what they
call a hologram would be.
So "I've always wanted
to watch a hologram"
"and there has to be something"
"in which a hologram just happens."
"Something like a projection."
"Rather than something
you have to put on."
So there's an already an idea
of what the technology
should be capable of.
They would like to get things
like view closer to the videos.
There was also a question
of cost and barriers
to adopting the technology
if you had cochlear aids,
that was an issue.
So these are all factors that
we hadn't taken into account
and we should in future studies.
Physical discomfort.
"Having glasses on your
head for two hours? No way!"
"And I can imagine you
put them on and everyone"
"will look at you and they
will recognize you are deaf."
"I don't like showing the
outside world that I am deaf."
"I feel too vulnerable."
So there are also these personal feelings,
very human feelings of problems
that you have to consider
when working with this
particular community.
"For a longer film
holograms are unsuitable,"
"since that our eyes got tired fast."
"In this case subtitles are better."
"Maybe in the future
holograms could be displayed"
"without HoloLens or other glasses?"
"Might be difficult for women
with wonderful hairstyles?"
"I am also wondering about people"
"who wear distance glasses,"
"can those glasses
(fashionably bigger ones)"
"fit into the HoloLens. Like
Dame Edna for instance."
We did not have Dame Edna, but you know,
these were all very human
problems that came across.
Position of the interpreter.
They liked the fact that
in the traditional method
the picture in picture was
closer to the actual content.
Even though they were
impressed with the HoloLens,
so with things like you know,
with traditional method
you can see everything
within your eye-line.
You have to move attention
from one to the other
in the HoloLens, so the
distance was a bit of a problem.
Conditions like a game of tennis.
"I wanted the signer on the picture"
"even if it obscured the picture itself."
"I think the full-body
interpreter is better."
"However, it would be worth
trying to place the interpreter"
'half-body' over the TV"
"Because you will not be
so left or right heavy."
Same to do with the
clarity of the interpreter.
"Position and opacity of the
signer needs to be controlled"
"by the user to make a huge difference."
"If I could move the signer"
"and if I can change the
luminosity of the signer,"
"make her darker?"
"Same with subtitles - different
size, different colors."
So it went on with
essentially people telling us
that they wanted control over
the way they had signers.
In terms with traditional
and AR, for instance,
folks wanted to be sure
that they could figure out
which signer to use, where to place them,
whether they could add
subtitles in AR as well.
So, final discussions.
"Having control over
where to place the signer"
"would be good."
"People have different
preferences and it would be good"
"to adjust to type of program,
people's visual capability"
"and wallpaper."
"I don't have time to make
adjustments to such a system"
"and fine tune things, there
should be something universal."
So what came out very clearly
was that personalization
is key, because the preferences
are quite individual.
Some folks like the emotional
type of interpreters,
the other folks they
want more information.
So, again, all of these are in the paper
and I think I'm running out
of time quite quickly there,
so I'm gonna stop and show you this first.
We do have more pieces of work
that will be published in due course,
including production guidelines
and things like that.
Thank you.
I don't think we have time
for any questions, right?
(clapping)
- (offscreen male
speaker) So unfortunately
in this case I would invite you
to talk to Vino after, you
know, after the session.
So thanks Vino.
- Thank you.
