When Trends Collide – The Uncertainty Principle of Revision Classes
Some years ago, when I was teaching, we ran
a series of revision classes for students
preparing for their mathematics exams. It
seemed obvious that this would be useful,
as the more you practise, the better you get.
Then we decided to do a little bit of analysis,
and discovered something that seemed really
strange. At first, we assumed we’d calculated
something incorrectly, but can you work out
what was actually going on?
There were 150 students taking the exam that
year, and the revision classes were optional,
so only those who felt they needed to attend
did so.
75 students attended the classes, and of those,
60 passed the exam. That’s a success rate
of 80%.
75 students didn’t attend the classes, and
of those, 56 passed the exam. That’s a success
rate of 74.7%
The teachers all patted ourselves on our backs,
said what a great job we’d done, and showed
the figures to the next year’s students.
We even added a bit of flourish to our explanation.
If you think about it, the students who attended
the revision classes were generally the ones
who were less confident about their maths.
You’d expect more of them to fail the exam,
but in fact, more of them passed, so the revision
sessions were obviously making a big difference.
All students should attend the revision classes!
We produced a nice comparative bar chart to
illustrate this fact.
We even resisted the temptation to distort
the scale to exaggerate the effect of the
sessions!
Now, it just so happened that 75 of the students
identified themselves as girls, and 75 identified
themselves as boys. Then 50 girls, but only
25 boys attended the sessions.
We knew that more girls passed their exam
than boys, so we decided to break the numbers
down by boys and girls, hoping to gain more
insight into the difference the revision sessions
were making.
For the girls, 45 out of the 50 who attended
the revision sessions passed their exam, that’s
a 90% pass rate.
But then we discovered that 23 out of the
25 that didn’t attend passed their exam,
and that’s a 92% pass rate.
So a higher percentage of the girls who DIDN’T
attend the revision sessions passed, than
those who did attend.
This was uncomfortable, but maybe it was due
to the fact that the 25 girls who didn’t
bother to attend the sessions had more self-awareness
about their abilities, and didn’t need to
attend.
Maybe the positive effect of the revision
sessions was even more applicable to boys,
which could be a powerful message, since only
half as many boys as girls had attended the
sessions, and a larger number had more to
gain!
So we looked at the numbers for boys. 15 out
of the 25 boys who attended the revision sessions
passed their exam, which is a 60% pass rate.
But 33 out of the 50 boys who didn’t attend
the revision session passed their exam, which
is a 66% pass rate!
This seemed like a paradox. Boys who attended
the revision sessions had a lower pass rate
than those who didn’t, and girls who attended
the revision sessions had a lower pass rate
than those who didn’t, but overall, the
revision class had a higher pass rate than
the non-revisers!
We checked the numbers, and they were right!
Superficially, we had a situation a bit like
Werner Heisenberg’s Uncertainty Principle
from quantum mechanics, but applied to revision
classes! If we don’t measure a student’s
gender, then we know that the revision class
will improve their chances of passing a maths
exam, but as soon as we know an individual’s
gender, then their chances of passing the
exam decrease! If you want to get maximum
value from the revision class, then don’t
let us know whether you’re a girls or a
boy!
How could this be? It’s clearly nonsense!
It turns out that this statistically strange
situation is relatively uncommon, but it does
happen from time to time, and it’s known
as Simpson’s paradox. Some people call it
Simpson’s reversal, or the Yule-Simpson
effect, the amalgamation paradox, or the reversal
paradox.
It’s when a consistent trend appears in
some data when you look at individual groups
separately, but a reversal of this trend appears
when you combine all the data together.
In fact, in situations like ours, where we
have three variables, each with two possible
outcomes at different rates: boy, girl, pass,
fail, attend revision, don’t attend revision,
it turns out that if we just chose numbers
at random, then we’d get a Simpson’s paradox
about a 60th of the time.
So, what’s going on?
Well, in our case, we were concentrating very
much on whether we thought the revision classes
were improving outcomes for students, but
if we just look at the data, there’s a huge
difference in the pass rates for boys and
girls which overwhelms any effect of the revision
classes.
Overall 116 out of the 150 students passed
the exam, which is a pass rate of 77.3%.
There were equal numbers of boys and girls,
but only 64% of boys passed the exam, whilst
90.7% of the girls passed the exam.
If we split up the students into two equal
sized groups at random, it seems pretty clear
that the group with more girls in it is probably
going to have the higher pass rate because
many more girls pass their exams than boys.
Here are the same numbers labelled Group A
and Group B, rather than Did Not Attend Revision
and Attended Revision, and let’s say that
the students were randomly allocated to the
two groups, and forget for the moment that
there were any revision classes.
Thinking that the group allocations were made
at random, we’re not distracted by preconceptions
about revision affecting outcomes. It might,
at first glance, seem a BIT strange that girls
and boys separately in Group A had higher
pass rates than the girls and boys separately
in Group B, but overall Group B had a higher
pass rate.
However, we can quite easily see that it’s
because Group B has more girls in it, and
the girls did a lot better than boys in the
exams overall.
If we NOW introduce the idea that students
were not in Group A or B randomly, but because
they had different revision regimes, then
we’d already be primed to spot the huge
difference in pass rates between boys and
girls, and that the total pass rate for each
group is influenced by how many boys or girls
are in each.
This would prompt us to consider boys and
girls separately, and we’d be focussing
on the observation that “attending the revision
sessions” seems to be associated with lower
pass rates for boys and for girls. And we
might then start looking at whether these
differences are statistically significant,
given our relatively small sample size.
This seems a more sensible way of approaching
the analysis – here are two groups, can
we find any evidence that one group performs
differently to the other given their mix of
boys and girls?
At this point I’d like to confess that I
made up these numbers, and this is supposed
to be an illustrative example which in no
way represents any actual revision sessions
I have ever been involved with.
Well, I didn’t MAKE UP the numbers – they’re
all real numbers, just not from revision classes,
and I carefully chose them to make a point
about causal analysis.
Whoever you are, revision will definitely
increase your chances of passing a maths exam!
Another example of this Simpson’s paradox
effect can be seen if we plot a scatter graph
of some bivariate continuous data.
Here we can see a scatter graph of people’s
age versus their score on a special aptitude
test.
Now, if we draw a line of best fit, it looks
like there is some positive correlation – although
there is a lot of individual variation, generally
older people get higher scores on the test.
But if we now look at sub groups we see different
patterns emerge. These people said their favourite
colour was green – and their data shows
a negative correlation. Older people tend
to get lower scores on the test.
These people said their favourite colour was
yellow – and again, their data shows negative
correlation. Older people tend to get lower
scores on the test.
And, these people said their favourite colour
was fuchsia – and again, their data shows
negative correlation.
So, overall the data shows positive correlation,
but when we know people’s favourite colour,
we see negative correlation. The overall trend
is different to the trend that each subgroup
shows.
So what have we learned in this video?
Well, basically, we need to be very careful
when trying to attribute a causal link between
variables!
If one thing looks like it causes another,
then you need to be very careful about what
you claim. Maybe it’s a coincidence, maybe
we’ve just got an idea in our head and we’re
interpreting the data as confirming our idea,
maybe there’s another variable lurking in
the background that’s causing the effect,
and then again, maybe someone just made up
the data!
But, whichever it is, good luck with your
maths revision!
When you subscribe to our channel, remember
to click the bell to ensure you get notifications
about our new videos, and for more mathematics
content, why not follow the link in the description
below to Nagwa.com?
