- So in this video, what
I'd like to do is build on
what we were talking about
in the last few videos
in terms of our approaches that we use
for propagating
uncertainty into forecasts,
to then think about
how we can break down those
uncertainties and analyze them,
in order to better understand
our models and our predictions
and to hopefully improve them.
One of the concepts I'd like
to emphasize here is what I call
the Model Data Feedback Loop.
The idea that we might
start from observations,
use them to calibrate a model,
propagate that uncertainty
into a prediction,
and then analyze that uncertainties
in terms of breaking them
down into the component parts
that go in.
So I might have a model
with multiple parameters
and I wanna know the contribution
of different parameters and processes
to my overall prediction.
I might have different covariates
and I wanna understand the
contribution of those covariates.
I might wanna understand
the relative importance
of parameters versus better constraining
the initial condition,
the initial current state of the system
in terms of how it might reduce
the forecast uncertainty.
Once I understand how different things
impact the forecast uncertainty,
that can help me set research priorities
that can direct, for
example, how I might go
and attempt to synthesize
our current literature and current data,
or set up plans for how I might
make new measurements
that specifically target
key sources of uncertainty.
One of things I wanna note here
is that in ecological systems,
most ecologists have an
abundance of hypotheses
about how all different
aspects of our system work.
So I don't view this idea of having
model data feedbacks prioritizing research
as any way in conflicting with the idea
of hypothesis driven research.
I think of it more as a way of
perhaps prioritizing the order in which we
tackle different hypotheses,
so that we can maximize
the information gain
and our ability increase
our predictive capacity.
To understand how
uncertainty analysis works,
let's come back to
the simple analytical case
of being able to propagate uncertainties
in terms of the analytical
or linear tangent approaches.
Where in that sense,
we saw that every component uncertainty
could be expressed in terms of
the uncertainties in an input,
whether that be an initial condition,
a parameter or a driver,
and then the sensitivity of the
forecast to that prediction.
So, what we then see is that
the overall forecast uncertainty
can be decomposed into
a sum of the direct impacts
of each of these uncertainties
and then the interaction terms
due to their covariances.
This essentially allows us
to say, what proportion of
the predictive uncertainty
can be attributed to each of those terms.
Furthermore, it allows us to understand
when a term contributes a large amount
to the predictive uncertainty,
is it contributing a large amount
because that is an inherently sensitive
parameter or process?
Or is that contributing a lot
because that is a poorly constrained?
All of us being equal,
if we're trying to reduce
uncertainties and forecasts,
we usually get more bang
for our buck by targeting
things that are poorly constrained
over things that are inherently sensitive,
but may already be well constrained.
If we take a step back,
one of the things that this tells us
is that if we wanna understand
the behavior of our systems,
and our, and their predictability,
we can't do that just with
sensitivity analysis alone.
This is something that we'll see a lot
in the literatures that people
use sensitivity analysis
to try to understand their models,
because that's only giving
us 1/2 of the picture.
It's telling us what's sensitive,
but not how that interacts
with uncertainties.
This flip side,
you know, often if we do
a calibration exercise,
we get an estimate of what's
uncertain in our model.
So what's the uncertainty
in the parameters
that also by itself is insufficient.
It's really the interaction
between these two things.
So if we have parameters
that are very sensitive
and very uncertain,
they're gonna contribute a lot
to our predictive uncertainty.
By contrast, if we have things
that are very insensitive,
and very well constrained,
they can contribute very little
to our predictive uncertainty.
And then in between,
things can be important
either because they're very sensitive,
or because they're very under constrained.
If we look at this more broadly,
a complex model to make
predictions may have
scores of parameters and inputs
and initial state variables
that we need to put into a
model to make a prediction.
So, as an example,
imagine a model had 100 different inputs
that needed to go in.
All else being equal,
we would then expect that on average,
each of those things would only contribute
1% to the predictive uncertainty.
In practice, it's very rare to
see all components of a model
contribute equally to the
predictive uncertainty.
But what we more often see
is there's going to be
a small number of things
that contribute a lot,
and a larger number of things
that contribute a little.
That is helpful because
it tells us there's a,
you know, potentially
large number of things
that are not going to be our priorities,
we can kind of take them as
given for the time being,
and really focus our attention
on a small number of things
that often drive our forecast uncertainty
and really prioritize those.
