Hello everyone, my name is Jonah Texas.
Today I will present our research our CHI paper
Augmented Reality to Enable Users in Learning
Case Grammar from their Real-World Interactions.
This is from a project that I did together
with Audrey Labrie, Albrecht Schmidt, and Lewis Chuang
As a bit of background, here
is a picture of what people have always been
doing to learn languages: they've tagged their
surroundings with sticky notes explaining
vocabulary that they would then walk past
by and learn on the go. But it's quite an
effort to label everything in your surroundings,
so that's why artificial intelligence methods
could be interesting that automatically scale
the content by creating new sticky notes for
you that you then see in Augmented Reality.
But there are also other reasons why AR might
be useful for language learning besides making
content scalable. For example, it has been
shown that contextual relevance improves vocabulary
recall and it's also quite easy to show image
information besides textual and verbal information
which increases the associative strength of
vocabulary when you learn ist. In this case,
this is Multimedia Learning. Furthermore AR
can visualize spatial and temporal relationships
because it enables you to interact with your
environment and interactively find out about
objects and their relationship. AR has already
been used for language learning in a couple
of projects just to name two here, we have
ARbis Pictus and WordSense, two projects that
label objects in the learners' surroundings
with their corresponding translations and
in the case of WordSense also with example
sentences and some video material in addition
to it.
The question is though, does this - there
is more to language than vocabulary,right?
There are also structural concepts and grammar
aspects that we need to learn. So can we also
use AR in this case? What we looked at specifically
was German case grammar and we decided to
come up with a concept that would also label
objects in the learners' surroundings but
then use their spatial relationship to explain
German case grammar.
to make you understand what this is actually
about I brought some examples. So in this
picture you see "der Laptop" und "die Topfpflanze"
as two objects and if we want to describe
their spatial relationship, in English, we
would just say "The laptop is to the left
of the potted plant". So "the" still remains
"the".
But if we do the same in German we have the
sentence "der Laptop steht links von der Topfpflanze",
so as you can see "die Topfpflanze" changes
to "der Topfpflanze". This is necessary because
a preposition in this case demands a dative
case of this specific noun. This is quite
difficult for learners of German and could
therefore be an interesting case to look at.
Now as an AR concept, we would implement this
by annotating the real world and then by making
users, giving them the possibility to select
objects. And thus, we create sentences, example
sentences, in a handheld-AR application with
a purpose that is widely available on users'
normal smartphones. This is how the implementation
looked. So learners select first one object,
then a second one and based on this, a quiz
is created with multiple-choice options for
the preposition and article combination. Specifically,
the quizzes were constructed as follows: first,
we had a first object. That was the subject
of the sentence, then a verb that indicated
the position, either "stands" or "lied" depending
on the object, then we had the multiple-choice
options for preposition + article. This was
derived based on the real-world position of
the two objects, so "to the left", "behind",
"in front" and so on. And then we had the
second object. Now the question is does this
really help, is it good for learning, is it
more useful than traditional methods like
flashcard applications? That is why we introduced
the control of a snapshot application. This
basically showed the same images as in the
AR application but they created statically.
They were taken from the AR usage and then
introduced as static flashcards with the same
multiple choice options but no direct object
interaction. To summarize from a learning
respective, the AR app had the advantage of
context and direct object interaction. On
the other hand, Snapshot provided a familiar
type of interface and both had realistic,
photo-realistic content. The hypothesis that
we formulated was that case grammar improvements
with AR would be greater than for people that
used the Snapshot application. And we conducted
a user study with exactly this hypothesis.
We used a between-group study design with
24 participants. We had the procedure of first
using a pre-study questionnaire where we asked
about demographics and experience with AR/VR
and then did some vocabulary and grammar tests.
Then people used the applications, the respective
one for up to 15 minutes, we had an additional
vocabulary and grammar test, plus some usability
and feedback questionnaire and then finally
one week after, to allow for a bit of memory
consolidation, we had a second test on vocabulary
and grammar. We collected data such as interaction
logs, the responses to the System Usability
scale, and we asked about usage situations.
Plus, we had the knowledge tests and the grammar
tests that I mentioned before.
Results show that case grammar performance
did not improve a lot in either condition.
You have to note here that the performance
even went down a bit from the first and second
test in the case of AR but this was probably
due to type of test that we used, so the first
one was slightly easier than the second one.
And our hypothesis that we formulated could
not be found to be true. But there was actually
more evidence for the, the null hypothesis
that "AR was not better than Snapshot" in
this case. So we can see that Snapshot probably
also have a couple of benefits like familiarity,
it's easy to repeat material and you might
not get so easily to distracted by interaction
with your environment. So we can actually
say that Snapshot apps or flashcard apps are
probably still an integral part of mobile
learning.
The System Usability Scale showed some additional
proof for the fact that they were not really
different in usability either. Plus, we collected
a couple of statements here to show why some
of the ratings could have been caused .
The first one is for snapshot. People said
that arrows could have been useful to display
the actual spatial relationships. And they
also missed gamification elements like points,
a score or level system. Overall, the app
was easy to use.
The AR app had some technology issues regarding
the relative position detection, apparently
it wasn't always correct and apparently the
marker detection was slow in some cases. But
overall was interactive and fun and also easy
to use. As you can see, those are mostly technology
issues. Regarding the concepts, there was
actually not so much critical feedback, which
is an encouraging point. Now another interesting
point was the potential usage contexts. We
saw that the Snapshot applications would have
been used by people. They said they would
use it on public transport or at home, so
we have static contexts.
On the other hand, AR was also frequently
seen as a suitable use case at family's or
friends' places or even outdoors. Exploring
the environment would be the purpose that
people employ here. As a summary we can say
that we presented a novel AR app for learning
case grammar but using this app did not really
improve learning outcomes compared to the
traditional flashcard method for German case
grammar as a structural concept of language.
Our qualitative analysis showed some positive
and negative aspects of both technologies.
And for example, it showed that AR would be
likely, most likely used in exploratory settings,
too.
Now this leads us to the conclusion that hybrid
solutions could be interesting to look at
and I brought a small concept here of what
something like this could look like. So we
could have an AR component and a static component
and they are both regulated by some kind of
context sensing and attention sensing and
learner tracking module that would determine
the context of the learner and then encourage
them to either use an AR exploration method
in an exploratory setting or to look at some
static content when the setting is more stationary.
In the AR case we would first have some exploration
of the environment, detection of objects,
position tracking and then based on this AR
quizzes would be recorded that we could then
use as content for the static component. When
a static component is used, we have two use
cases depending on the attention level. Studying
with the AR and other flashcards would be
a use case for low or medium attention levels,
maybe on public transport when there is a
bit of distraction around. And when users
have potential full attention, then this could
be a perfect moment for introducing new content
from a textbook that requires a bit of focus
but is also important to cover the curriculum
that you are supposed to cover. Now this could
be a nice interplay of different components
that take advantage of different benefits
of situations. Okay that was it from my side
I hope you got a good overview of the topic
that we explored and I'm very happy to take
questions if you have any. Here you can see
my email address, don't hesitate to contact me
Thank you for your attention.
