Probably what we are going to try and do is,
I'm going to explain to you why I'm right
and then...
I'm joking.
I'm going to probably set up my stool for
the discussion about whether or not to defer
when someone's come in the community to discuss
epistemic modesty.
I guess Ollie will then...
Bring my perspective into it, contrast it
a bit, and that will be a starting point for
the whole discussion.
By way of introduction, there's many cases
where deferring to what other people think,
or people are who are maybe more informed
than you think, is pretty inarguable and pretty
obvious.
I think that's common knowledge amongst most
people here.
A common example given is, you know, if you're
taking your bins out or taking the trash out
on a particular day of a week and you see
none of your neighbors have left their trash
out on the same day of a week, you probably
think like, "I've made a mistake here" rather
than "everyone else's made a mistake."
And then change your mind because of that.
And likely if you read a textbook and you
see something in that it looks a little bit
surprising to you it's probably more likely
that you just don't understand it rather than
the textbook makes a mistake.
That doesn't mean textbooks are infallible,
but I mean you should probably like... if
you have to guess which one's right, you or
the textbook, you should probably guess the
textbook in most cases.
So that's all the really inarguable stuff,
which I think Ollie is going to be arguing
against very vociferously for the next hour.
Alas for me.
But where it gets a little more complicated,
there's lots of areas where we do often take
ourselves or license ourselves to disagree.
So like in politics, in various complicated
factual matters, in religion and all sorts
of other things besides.
And there we think, even if they're experts,
we still think we are within our rights epistemically
to go actually these guys are wrong and we're
right.
I guess my view on this is like, I'm probably
like a modesty radical, immodestly modest,
you might say, insofar as I think, in essence,
what you should always do in each possible
condition of things is you should hew your
own view towards an idealized expert consensus.
So you weigh up different expert classes,
work out which ones are more relevant, and
do all of that stuff.
And then you get an answer based on that.
And that's what you should think is actually
true.
So your credence should be driven by this,
and your own personal impression of where
the balance of reason lies counts in most
cases for virtually nothing.
It counts no more than another person similarly
well informed as you, coming to the view they
have.
And so that typically means in my day to day
life, I end up sort of deferring quite a lot.
And so, I don't have a particularly strong
political views or like, hot button topics
because the fact there is controversy means
I probably shouldn't be very confident that
one side is right or the other and so on and
so forth.
I guess the EA relevance of this is in many
cases, EAs do take views which like outside
side or often against expert consensus, and
how often is that a reasonable thing to do
or not reasonable?
And obviously given my own ideology here,
I tend to be leaning in favor of thinking
that the EA community should be a lot more
cautious and more deferential to existing
bodies of expertise.
So that's like me as it were setting out my
stool.
I look forward to Ollie pelting me with rotten
fruit, as appropriate.
I'll hand over to him for a bit.
I think ultimately a lot of what Greg said
is right.
I think there is an important sense in which
it is obviously really important to take in
due account other people's epistemic states
and other people's observations, and their
current opinions in forming your own opinions.
But I think there are various problems with
trying to do this naively.
I think the perspective that I'm taking, is
not necessarily have something deeply flawed
with something like the complete, philosophically
rigorous view of what Greg is saying, but
more something where I'm saying that if you
do naively the thing that Greg recommends
people do, you will end up running into various
problems, and I think these problems are kind
of illustrated by three of the topics we hope
to cover today.
There is the question of who are you going
to count as an expert, which is I think is
underlying just if I'm going to go by expert
consensus there is this key question of, who
I will let trust as an expert?
In questions of religion should I trust theologians?
They seem to have thought most about it.
Does mean that God definitely exists given
that most theologians seem to think God exists?
But then we are like, ah, maybe philosophers.
So we have this interesting question of who
do we trust as our experts?
Similar, a related question is when are you
yourself an expert?
At what point do you cross the threshold to
think that you are the world expert on a relevant
topic?
Maybe you are the person who has done the
most research in the world of all the people
that exist into a certain form of quantum
mechanics or in certain questions about online
discussion platform design (which I hope I
have achieved).
And in those situations, I think that is just
a question where I expect where if you do
the modest thing, and if you kind of carry
forward the attitude of modesty into that
situation, you will usually make bad decisions.
And third of all, there are important second
order effects, where if you have a group of
people who are generally primarily making
decisions based on other people's opinions,
you run into many, many information cascade
problems that we often see in markets, often
see in various real life situations, panics,
where everyone else is primarily paying attention
to what everyone else does.
And then you get stuff like bystander effect,
you get stampede and stuff like that.
And that is a problem that I think can be
avoided with very rigorous and careful thinking
about how other people came to their opinions.
But it's something that if you're going to
naively take the modesty stance, I think you're
going to be on average doing worst than I
expect most people in this room to be doing.
Okay.
I think those are my three big perspectives
and objections that I have with the broader
framework because I think philosophically,
I agree with a lot of things that Greg says,
but I think the question is more about, how
are we going to actually implement these insights
about epistemology and understanding.
That was of course in the details.
Can we talk about the first thing first, maybe?
Yes, let's go with the first one.
Okay.
Should I do it or do you want to do it?
Looks like we're both being modest, so I win.
So the question of what counts as expertise.
Right?
And so I often, in fact, I've written at length,
at great length on modesty online.
My idea is, I often like to use is that maybe
somewhat loosely, subject matter expertise
is basically the same thing as expertise per
se.
So to answer "is it good to have a death penalty"
look for people who work in criminal justice,
is a good to increase or decrease minimum
wage, maybe we should talk to economists,
I say as one enters the room just now.
If you want to know if you're sick or not
maybe consult a physician.
I think there were many cases where people
would agree that's a useful first pass heuristic.
But there's a key question as to, in many
cases it seems like this often co-locates
with social status.
So I used to be a doctor, doctors have this
quite good reputation, present company notwithstanding,
and that means that you're taken maybe more
seriously than you deserve to be.
There are like lots of stories actually in
medicine going further where people are overly
deferential to doctors who are doing really
stupid things.
Or like the nurse, who is not in the same
high status profession the doctor is, sort
of let people like me make terrible mistakes
and cause damage.
So that's obviously like where it can go wrong
and definitely what I would want to avoid.
The question is we have some data, some interesting
data Ollie wanted to cover.
So I'm stealing his thunder, on where very
simple scoring rules can beat commercial expertise.
So there was like reason rules from Kahneman,
there's prediction markets and all of this
stuff.
And so it seems you don't necessarily need
lots of subject matter knowledge, beyond being
an academic, being a professor, to actually
get better answers than the supposed expert
class.
Do you want to elaborate more on that, so
I'm not making your point before you can do
an even better job of what I'm attempting
to do.
I think like one of the most interesting questions
that illustrates this most is the super forecasting
literature, where you have this interesting
thing where you have the super forecasters
trying to make judgments about global politics,
trying to make judgments about technological
advances.
And those are people without subject matter
expertise.
They are people who usually are generally
well read, and can do Wikipedia searches,
and then just try to apply kind of good forms
of... good rules of reasoning, that are pretty
domain general, like they don't really depend
on a specific domain.
And in the super forecasting literature, that
has shown to very often outperform experts.
And now you have this problem where like okay
we now... we have the experts are forecasting,
which we can think of as kind of the experts
at answering certain forms of predictive questions.
And we have domain experts who are hopefully
good at answering domain expertise questions.
But when domains widows intersect, which is
quite a bit obviously, we are kind of stuck
if we try to do naive expert aggregation.
I think it's worth... maybe just on the second
point.
So it's obviously the case that if you're
trying to defer in the epistemic sense, your
criteria for expertise is just like predictive
accuracy.
So in a world where the super forecasters
and intelligent analysts are predicting something
and you know the super forecaster has got
a better track record, even if the analysts
have a higher social status, the modest thing
is to go with the super forecasters and not
with the intelligence analysts.
The problem is, of course, that we may find,
which we may go on to later, is that maybe
the general skills required to make good predictions
about various events is like this single key
skill, and doesn't really have very much to
do with particular domains of expert knowledge,
and so it shouldn't be like deferring very
much to these sorts of people.
I guess my own view, to like till my hand
preemptively, is that although it may be the
case for your super forecasting, you might
be able to sort of take issue with intelligence
services or similar things like that, if you're
a typical person in the audience, you're probably
not a super forecaster.
And so even if there's even better people
out there, you should be deferring to, instead
of an expert class, a typical subject matter
expert has got advantages over you, or so
I would hope, but that may lead us into the
next thing a little bit too early.
On which, more later.
I guess the other thing say, that Ollie alluded
to, it is a lot harder than it sounds just
to defer to the experts because, taking your
example further, like if you ask theologians
whether God exists, they all agree.
If you ask people in general, most people
believe God exists, worldwide.
That's so-called the common consent argument
for God's existence.
We also see, but then you go, yeah, fine,
but maybe smart people are expert class on
this.
So you stratify by IQ.
People with higher IQs trend atheist compared
to general population.
Okay.
That might be something.
But wait a minute, Well, what about philosophers?
Philosophers themselves also, philosophers
generally also trend atheist.
Philosophers of religion, though, who spend
lots of time thinking about whether God does
or doesn't exist, they actually trend strongly
theistic, at least in the US.
And then you go, well, who should we undercut?
You go to underlying features.
Maybe like people are really just naturally
religious gravitate to philosophy of religion.
They weren't persuaded by the arguments, and
you can have lots of very complicated discussions
over who counts as an expert, and like make
a rod for my own back.
There's a worry whereby you can liberally
gerrymander who you count as an expert to
get the answer you wanted to have in the first
place.
So, weigh these people slightly more highly,
and there's lots of fudge factor, so you can
twist the dials to, to get the answer you'd
like.
Obviously, epistemically virtuous people like
me would never do anything like that, but
you know, better off for the rest of you guys.
So whatever.
It's harder, it is very hard to do.
I would still insist or suggest, I'll suggest
it can still be better than trying to strike
out on your own and try and form your own
impressions of it, which is perhaps where
we begin to route back into a topic of disagreements.
Yes.
So I think one area that I'm curious about
kind of exploring, that I think could be really
relevant here, is I expect us to have some
disagreements about to what degree do we think
there are a set of key underlying, think of
them as general rationality skills, or you
could think of them as general philosophy
skills or something like that.
Where you can easily imagine a world, and
I like to always... there's a straw man mathematician
perspective where you study mathematics, and
mathematics has like eight branches.
You'll study algebra, you'll study linear
algebra, you study some of the more graph
theoretic stuff, and now you really have eight
models of the world and everything is really
just an application of this.
Like after you notice eight models, like voting
theory, no problem, you just apply some graphs
to it.
Database design, no problem, you just apply
some algebraic constructions to it.
And so, there's this idea that we might be
living in a world where there are very few
simple models that we can generally apply
to a lot of different things, and that ultimately
expertise in those models dominates the ability
to predict accurately in that domain over
this specific subject expertise.
I don't think this is truly as extreme as
the naive mathematician's view suggested,
but I think it has quite a lot of merit in
general, which we see both in super forecasting
domain, but also in the 
domain of thinking rigorously and reasoning
rigorously, which isn't necessarily only as
an are, a super forecasting domain.
I think in many ways I would describe it as
something like philosophical competence, where
you have a concrete example.
One of the big problems that I heard Open
Phil has often run into, in their discussions
with various experts, is they try to make
a decision of like how likely are various
risks, and they go to the relevant experts
and they asked them, “how likely are those
risks?”
And they answer them with “well very high.”
And you're like, “well what do you mean
with very high?”
"Well, like definitely high."
And you're like "is it 50 percent or 70 percent?"
It's like, “I don't know.
I can't really put a number on it.
High.”
And you end up kind of in this situation where,
I think there's an important sense in which
general skills about how you interact with
your beliefs, and even being able to make
precise statements about the world that are
general in the sense, like, could screen off
just the sense of domain expertise.
And I expected we probably disagree about
the degree to which there is specific domain
expertise in detail to these different areas.
Like can I just reach the summit of the epistemic
function with all creation arrayed before
you, is the only way you can make good progress.
So that's like, yes.
So I think you do anticipate correctly.
I'd be like, "well I'm not entirely sure about
that."
I mean, it seems the case that there's like
something which super forecasters have which
involves doing lots of different things.
They have to be able to predict.
There are specialist politics forecasters,
there's some of that.
They often look quite good in various domains,
so definitely the transfer capability set,
it's worth saying that they have some background
knowledge.
They're not usually doing things literally
from nothing but it's like not very much,
like something like a reasonably low time
level expertise, rather than spend like two
decades studying like dynamics of politics
in a certain area.
I guess there are also other areas you want
to look to besides super forecasting, because
like you could say there's like civilizational
inadequacy, whereby people aren't trying to
protect certain very important events.
And so maybe experts haven't really like been
trying to develop expertise yet in that area.
Well it's like expert performance.
It may be the case that it's quite widely
separated, so people who are really good at
Go on really good at Chess necessarily, or
like Poker, and things like that.
And we have pretty good evidence that actually,
when they can play each other.
There's lots of things you'd like to learn,
to become really good at chess, which aren't
just general skills.
Things like IQ and stuff help, but it helps
to the degree you'd expect.
So I guess my view is that lots of bits of
reality have this property whereby you have
to like pile in lots and lots of precise knowledge
to make good progress in an area.
We might just be like at a very early stage
of forecasting.
It's like in a field where you can become
an expert in.
Maybe the forthcoming and, one hopes, Tetlock-infication
of governments will lead more in that area
but who knows?
Maybe I'm wrong.
Yes.
So whenever we can find maybe a set of predictions
or something like that, where we completely
expect different groups of people to give
different quality of answers.
That would help us maybe tease out some of
the things.
Yeah.
So I wonder if we could find the edge of like
where I expect super forecasters to stop being
good.
Where just like you expect super forecasters
to stop being good, where maybe it works,
where like I can imagine that there is a world
where you...
Like let's say we go into the domain of quantum
physics or something like that, and where
I think I still need to reflect a bit more
about it, but naively would expect that, I
think like my trust in super forecasters would
extend further into domain of like narrow
quantum physics, and getting questions such
as like, "Is cold fusion possible?"
Something you talked about it in one of your
blog posts, and we ask physicists, maybe even
a pretty good physicist from a top university,
and we ask a super forecaster, and who would
we trust on that?
And I think I would be pretty torn.
I really do.
I think I would have a good chance, especially
if I have a super forecaster work on it for
a week, I would have a pretty good chance
that I would trust them more than the average
physicists.
At least PhD students.
I definitely agree with the direction of where
we differ.
I'm just not quite sure if we'd like can crispy
like, find out exactly where, because not
necessarily, if you have a fairly wacky thing
like cold fusion, physicists wouldn't necessarily
usually work on cold fusion.
Or have any knowledge of it, like they'd just
think it seems weird to me.
It's effectively outside my...
I agree in the general sense that obviously,
you'd want several key core skills, like having
good accuracy of beliefs.
You would expect these things to apply more
broadly than I would.
If you're trying to predict whether the answer
is like Ramsey 55, or Ramsey 77, I would be
much more willing to trust my intuitions about
super forecasters to give more evidence for
that, or similar things like that.
It would generally be nice, I guess...
One interesting area perhaps is like, data
on rationality training, like calibration
training.
There's not really crisp evidence of improving
like job performance is my understanding.
I'm not very acquainted with this literature,
but my standard understanding is, we do this
stuff.
They learn these cognitive biases, and yet
you haven't gotten better.
Now it could be a skill like this doesn't
transfer.
If you just drill it you might become smarter,
but it could be something like, it's not really
training the underlying skill.
But if it was improving the underlying skill
and yet you don't see improvement in performance
where we think expertise would matter, we
would give different predictions about how
likely that is to be the case.
But again this might be a little bit too woolly
to really pin down.
I actually think this is a very important
domain, and I think one of the things that
kind of underlines a lot of my edifice of
opinion here is, there is this interesting
question, where at some point in the 17th
century, humanity started being able to solve
problems that it was never be able to solve
before, remotely.
Like we started out like being able to kind
of, if we really tried, build a simple city,
and then 200 years later or 300 years later
we build rockets that flew to the moon.
We built computers, we built the Internet,
and it really seems like something... like,
we see something like the Flynn effect.
You can imagine maybe something like general
competence increased and people got more intelligent.
People had less like nutritional deficiencies
and maybe that's the reason why we can do
all of those things.
But I think the case for that is not strong
enough.
I think ultimately there is a strong argument
for something like, there was a quantification
and mathematification of a lot of these fields
of science, that allows us to build the methods
of science, build out physics as a domain,
and then start just solving lots and lots
of problems with this pretty small set of
tools.
And like, if you look at all of human knowledge
at the time, this at the time, if you look
at the 17th century, like what you would study
at university, it was primarily rhetoric.
It was primarily like lots and lots of theology.
And then you would maybe spend something like
10% of your time studying mathematics, maybe
15% of your time studying mathematics.
But that branch turned out, that kind of relatively
small aspect of human knowledge, turned out
to be very widely applicable, and to solve
a really large domain of problem.
And I think this highlights an important sentence
in which we have, I think we have an answer
to discretion of whether there are domain
general skills that allow you to get better
at solving problems in a large range.
And I think the answer is, a lot of like modern
quantified science and the evidence we have
for that is the scientific and industrial
revolution.
And I wonder if like...
I think there's part of the point I agree
with, but I'm not sure how much of the variance
it explains in the classical challenge, here.
I mean like inferring, I mean like, there's
a perennial problem in the field of like,
macro-historical judgment is very hard.
Too early to tell what caused the scientific
revolution.
It might be too early to say we're confident
with the scientific revolution was because
we discovered a maths, when there was like
lots of precursors.
Now we discovered calculus reasoning just
before, and I think it's quite hard to work
out things like intellectual traditions.
But I agree in the general sense.
We have like sort of really crisp historical
data showing we made this like advance in
thinking tools, and this great wealth of understanding
opened up in front of us.
I think that's all very favorable evidence.
It's like been, to a very large degree, these
very key skills you can apply to lots of stuff,
rather than my view where it's like, you have
like lots of like hedgehogs.
Hedgehogs or foxes?
I think hedgehogs.
You have to burrow you way into your area
to like, get good answers.
I'm just wondering.
We might be thinking of moving to point B?
You get the last word in, but...
No, I think that's pretty natural stopping
point here.
Okay, cool.
So next thing that we wanted to cover is when
are you an expert, and it covers a lot of
very similar ground.
I wonder whether you have like some initial
thoughts.
Yes.
I think like where we can go next is we can,
we can discuss often, in the general sense,
like how the different classes of cognizers
are like, and how much there's like a single
axis of thinking skills.
Like with people who were top, who are able
to get lots of things, or if it's like very,
very dimensional.
Like, if you're an expert at math, you can
make good predictions at math.
If you're an expert in politics, you can make
good predictions in politics.
I know there's lots of different things and
therefore if you're like, but then what do
you do, and if you're a typical person who
is not an expert in many things, or in my
case an expert in anything, what you can do
instead is say, if I'm like pretty mediocre
at everything, I can steal the knowledge out
people have got, and steal their predictions
and use that instead.
And that just seems like a strategy-stealing,
superior approach.
And then I guess I'd then say even if we...
even if we disagree or how much there's like,
one key skill, I'll say you can often maybe
like surpass what's a typical individual in
this community can do, if we defer more, rather
than saying I've read, I've looked into stuff
and rationality, I've read various works on
like how to by rational.
I see myself equipped and have very confident
pronouncements to make, on let's say macro
economic theory or public health and stuff
like that.
And that's the typical principle of mediocrity
rather than modestly.
The typical person, even in EA, who are very
clever people, of course you all are, generally
would always almost always do better if they're
deferring to someone.
Whether that's a super forecaster or a domain
expert is unclear.
But I do think we are unexceptional cognizers
in this whole sense, or at least not so exceptional.
We can always bank on ourselves to want to
stake out our own views constantly, but maybe
that's not quite getting to the crux of the
matter, as you might say.
So I think this is...
I feel like this is almost responding to your
previous point and I kind of wanted pull on
this, but it's kind of something that's been...
where I have this thing.
I have this feeling that we both agree that
the question of who is an expert and how should
we aggregate opinion, and something like when
exactly should we believe someone else's opinion
and how should we integrate with us?
Like if we both agree this is a very rich
and important domain, in a sense that it's
important that we figure out the general rules
of this space, we figure out how to aggregate
knowledge like that.
But I feel...
I don't know what it is, but I have sometimes
the feeling that when you advocate for modesty,
that you advocate not for the study of a rich
field.
I feel you're not advocating necessarily being
like, something that...
I think the thing that I would be definitely
deeply behind is something like: I think the
question of how to aggregate opinion of the
people around you is important, and the answer
is not obvious.
Like the answer is, here are some interesting
considerations that we have.
And then you would like maybe need examples,
and then somebody would want to start getting
expertise in this domain, meta-expertise.
That would be, let's look at the example of
super forecasters with physicists, analyze
the structure of that, and then come to some
conclusions, and then maybe try to extract
some general principles from that.
And then I would be like, yep, those are good
principles.
But like kind of making it clear that there
is an art and a skill here that needs to be
built up over time, where I'm worried that
when you usually say modesty, you're like
just, be less arrogant or something like that.
And I agree that maybe some people have like
an aura of arrogance, but it's like a very...
something like trying to tell a physicist
that they're not making a lot of progress,
because they always write too large in their
notebook, and so they run out of space before
they finished thinking on a current notebook
page.
And I'm like, I guess I can see how this is
maybe an error that someone is making, but
it really doesn't seem like the most important
dimension, or something like that.
Like arrogance or something feels almost like,
it's a relevant variable, that I think is
important to calibrate, but it doesn't strike
me as one of the key variables.
And I wonder whether you might disagree with
that.
I didn't know.
It may be a little bit hard to litigate the
examples I have in mind, which may not be
very helpful to discuss.
I think my view is, if maybe the case that
it's not necessarily straightforward, that
doing like the modesty thing well is in itself
a bit of a skill.
But I think it is also the case that maybe
I can't identify really clear mistakes.
So there's the case I got from a friend where
someone said by way I've disproved the continuum
hypothesis: I've god this notebook, would
you like to have a look, and it just feels
like in that sort of world, we answer, no,
I can just tell you with probability 1 minus
epsilon you're wrong about this, and you should
have thought the same because there's lots
of proofs of this, and the rest of it.
Yes.
So I see like I definitely feel like there's
like steps you make to do better.
I often see like sort of like really obvious
mistakes, and then the simple word, is please
stop being so epistemically arrogant.
But let me say what you do instead is a bit
more involved.
But as a crude rule of thumb, I often find
this seems to do better is I guess my view.
But also the other thing whereby like epistemic
immodesty, and immodesty in general are not
exactly the same thing, and they can co-locate.
You might get annoyed or people being like
a bit cocky.
So for me, I'm like cocky all the time, and
I'm like preaching epistemic modesty, so these
things do come apart in various respects.
But in the same way, it's important to stress
that one shouldn't poison one's epistemics
by personal dislike you have for a certain
communication style, or anything like that.
I may not always practice what I preach here,
but I do at least recognize I may be at fault.
Yeah.
I do think this is something that I probably
expect to be useful to clarify.
Like we both agree and we kind of had an earlier
conversation a bit about this, where there's
just this really important distinction between
what we think of as social expertise, where
there's like, I guess the degree to which
our gut judgment says that somebody is an
expert, and something like the actual epistemic
expertise.
The actual track record of how much will they
reliably get questions in this domain right.
And just that those are definitely two different
things, in different domains, that can get
completely different answers.
What made how completely might be another...
Yes, that might be the-
I might be more deferential.
It's like sort of cultural expertise.
Like okay, if I've got an expertise it's good.
If you're an expert in this area, even if
you're not a super forecaster, you might still
be worth deferring to in many cases.
Probably like obviously that disagreement
over how well we can correlate may not be
a large part of a wider discussion I guess.
Yes.
So I do think that...
I think the example that you mentioned, somebody
thinks they just proved the continuum hypothesis,
is very good, because also contrast like naturally
with somebody comes in and says...
I don't know, let's go back to like, when
was it, 2012?
Of like Iraq has nuclear weapons or something
like that, or chemical weapons.
Where there was this... there was a situation
where we had experts in the field, who was
the CIA at the time, who made predictions,
and now somebody comes and is like, you know,
actually, I Googled myself for a while for
the relevant evidence, and I think it's pretty
obvious that they don't.
And then we, I think in that case I wouldn't
go say probability one minus epsilon at all
that they are wrong.
I would probably say, if I don't know them
very well, like 30% that they're right at
maximum, maybe 20% that they are right.
But I would have a very different attitude
towards that kind of thing.
And I'm very worried that if we try to paint
this very broad brush of modesty, that we'll
put those in the same bucket as the continuum
hypothesis and mathematics, which we know
has... is in a framework, with several proofs.
And is in a framework in which we can expect
to be rigorous, and I think it's very important.
Like is it a framework that we've seen work?
Or something like that.
Where like, I do think mathematics is really
important.
We have seen mathematics work over and over
again, and so there's a sense in which when
somebody disproves a major mathematical theorem
that we expect to be true, or like at least
that many, many people have thought about
a lot, then we can trust that their effort
was something productive.
Whereas if we're talking about international
politics and situations where I have never
heard of a large class of experts who reliably
got those questions right.
And so in those situations I don't think we
should apply the same standard of modesty.
So I think you'll obviously fair to say that
picking mathematics is the easiest example
of my case.
We actually know the ground truth.
I think I would say though, but well, I would
go on to say something like this.
So suppose you're this person who Googles
for maybe like two hours, whether Iraq has
chemical weapons, whatever it was, and you
decide like probably not.
It seems to be the approach a typical person
would take.
It's like if you're like a third party to
this conversation, like somebody walks up
to you and says oh, I've done some Googling.
Actually Iraq doesn't have chemical weapons
after all.
That evidence to you should be like aggregated
with basically zero weight.
Because there's lots... like thousands of
thousands of people, with all of their views
saying it has or it hasn't.
Experts have spent loads and loads of time
on it.
And now it may be the case they are right.
Because it's very hard to tell as you say,
there's no really good, robust demonstration
of evidence.
But the mere fact that someone's telling you
"I've Googled this for a while" or similarly
cursory efforts, shouldn't really massively
update you one way or the other, I would allege.
And then you can sort of anticipate my next
idea, which is well if that's true, then by
indifference, you shouldn't care whether you
are that person or you're hearing it from
a third party.
Because if you're a person who spent two hours
Googling it or whatever, then you also shouldn't
be updating any more than if someone else
was doing, who's similarly situated to you.
And that, I would argue, would always lead
to considerable deference in most cases.
I think that really depends because, usually,
let's say I followed... at the time I don't
think I was doing this.
But let's imagine we, in the present day,
we have the same Iraq nuclear weapons question.
And I've been casually following the news
and the answer is like, okay, I've heard.
I've definitely heard the fact that there
are people who have opinions on both sides.
But I wasn't given any evidence about the
methods of the people on either side.
Like I wasn't given access to the methods
that they used to arrive at their opinion,
but now a friend of mine comes over and says,
I said no.
Now I agree that if we just count the number
of people who have opinions in here, this
might be, he might just be a very small drop
in the bucket, but if we assume that people
with the better methods reliably get better
answers, and I have some expectation that
my friend has better methods than the average
person, which I think I can expect in the
domain of policy forecasters, then this is
actually a major update.
Because now it is the first time that I heard
evidence, with the methods that were being
used.
And I think a really dangerous... like a really
important dimension of all of this, is that
in scientific fields it is very often extremely
hard to figure out what methods people actually
use to come to their opinions.
If I read a physics journal that describes
an experiment, I only really see the experiment
that they ran, and maybe some mathematical
analyses they did, but I didn't get any insight
into the thinking.
I didn't get any insight into how did they
go about hypothesis generation?
How did they go about... how many alternative
hypotheses did they consider and weigh?
And for my friends and for the people who
are very close to me, I have that evidence
much more often because I know their methods
and processes.
And so I actually think updating on that to
a large degree can very often be the correct
choice.
Ah, right.
So we do definitely disagree on that.
So I think where I go is, I think I would
suggest/allege/whatever, that it seems you
have a very parochial view of the typical
epistemic qualities of the people around you.
So I agree.
Like if someone, like if your friend like
comes up to you, somebody who you respect,
and they give you their reasoning as to why
we should think Iraq doesn't have chemical
weapons and you go, "I know this person is
telling me what they think, and they're definitely
very reasonable."
But you go, "look man, there's professors
of international relations."
Now you might go like, "okay but they may
not necessarily be super forecasters."
No, granted, but you can probably anticipate
if you manage to get to talk to one of these
people, they're like, you would hope, I'd
suggest you should expect, if you chatted
with someone who'd worked in security studies
in the Middle East, they'd give you a reasoned
judgment as to why we think it's more or less
likely that Iraq has chemical weapons or not.
And so it may be the case that you think...
if you think like the relevant, like meta-epistemic
qualities are very rare to find, such that
your friend who has them, you'd put them above
some relevant domain experts, we have like
implicit knowledge of what they think.
You retire what you see the US government
saying, you retire what you see academics
are saying.
You retire all these talking heads, or, well
maybe they aren't experts, but other people
who are experts on the news saying things,
and so on and so forth.
I'll say that you shouldn't necessarily go,
"but because I, I've seen with my own eyes
what they're thinking" in the case of your
friend, you should weigh that more heavily.
Maybe you can't have access to the generating
methods, but you should like offer like some,
you should be willing to trust these people.
They do actually have some there if you are
able to investigate, because that's where
I would like want to go I guess.
But I do think that like the big assumption
that was here was that an average chosen professor
of foreign studies at maybe a major university,
will they have methods that are at all reasonable
to come to good conclusions?
And I think the answer is like, no, in the
sense that I think the average professor of
foreign relations at a major university does
not have something like the calculus of reason
to actually make correct probabilistic statements
about that kind of stuff.
Like, it doesn't matter.
Even if you're exposed to a lot of the data,
if you don't have a sense... if you don't
reliably can keep apart causation from correlation,
if you don't reliably keep apart like the
social biases that you have and the status
hierarchies and the environment from your
epistemic beliefs, it doesn't...
I don't think it really matters how much precisely
you know about the specific foreign relations
of the country you're talking about, because
that evidence won't be like... those ideas
won't be able to build on one another.
You're not actually given me a reasonable
opinion that I can trust.
And I think, like, I want to go back to kind
of the mathematical reasoning, because I think
there's a strong analogy here of, you can
imagine you have, there is this question,
let's say we have...
I think, let's go into the domain of construction,
where only in the 18th or 19th century did
we really start to have any form of a systematic
method of building things.
Like we did some simple static analysis before
that, but mostly when we built things, we
stacked things on top of each other, saw whether
they fell down, noted which things didn't
fall down, and then built those things again,
and maybe combined them in smaller ways.
And so we had these people who definitely
were exposed a lot to the domain of building
things.
They architected many buildings with this
method, and now we have modern physicists
encountering them.
And they have access to the modern calculus
of physics and understand modern statics.
And now let's say they've only participated
in building a single building.
They've only seen a single thing constructed,
but they've kind of seen all the basic parts
of how it is to construct a building.
What I'm arguing for is that they will be
able to integrate that evidence much, much
better, because they will have the language
to describe those kind of things.
They will be able to describe statics, they
will be able to generalize, when they see
an upstanding arc, why that arc stands up
and how much weight it will have, and that
ultimately the next building is the one that
I would trust a physicist much, much more
than a person who just built a building.
So I think it's interesting because I think
where we differ is maybe like at a baseline
level or at a baseline difference between
two groups.
So I'm definitely not going to defend the
claim, that like a typical professor of foreign
policy has got really good like epistemic
norms, more than a typical person, because
we have lots and lots and lots of data on
heuristics and biases that apply to everyone,
no matter how allegedly degreed or prestigious
they are.
I guess where I differ is I'm not sure, although
you keep excellent company, whether a particular
friend of yours is in expectation going to
be much better than domain experts in this
area, such that you go, "even though this
person doesn't have the same level of expertise
as this person, we have a pro tanto reason
to trust them slightly more at least."
Because they have the other metacognitive
skills, I should count them much higher than
the prevailing view of an expert class, because
they are much higher than the average level
of like epistemic quality in this area.
I guess is like the thing.
If that could be made concrete whereas you'd
test people, and see how good they are at
predictions, if it turns out for example,
that bay area community is chock full of super
forecasters, then yes I'd that's obviously
like a really good reason to take their views
much more seriously than for example, what
you typically see expert groups doing.
I guess where I sort of differ is, although
I think we're great, I'm not sure we're that
great, in this sort of way.
I think I'm, I'm not arguing that precisely
the people I'm surrounding myself.,, I think
I hope that the people have chosen as my friends
are good at thinking; I think that is one
of the primary things I select them for.
But, I don't want to make the argument that
that is the thing that ultimately, it's not
the fact that they're my friends that's the
primary selection criteria here, but I'm saying
something like, it is, there are many, many
people who are not my friends who if they
give me advice, if they make a judgment in
a certain case, then I will trust him much
more than my friends.
I'm not making the argument that my friends
are...
Maybe like we can get more crisp idea.
So I agree it doesn't really matter whether
you like these people or they're friends of
yours, but I guess like I don't see what you
might think of as like...
If you're going to treat certain individuals
like, sort of epistemically... "oracle" is
more dismissive than I mean, but sort of like
people with very good judgment applied across
multiple domains, so you would actually have
good reason to trust them over what prevailing
experts seem to say, then it seems like you
can test, we can just look at their like track
record and say, "look, we just keep getting
stuff right, again and again and again, in
all these different areas."
But I guess I don't usually see this evidence,
or at least there is no public version of
this evidence, for when I see this often done
widely, and it may be the case for you do
have this pro-evidence, which is great.
I wonder, out of interest, how carefully you
try to investigate the track record of people
you take to trust so highly.
I also wonder in the typical use case of people
doing this stuff, how often they are doing
it as well.
Because maybe, you epistemic virtuous par
excelsis, are already good at this.
But a typical person might massively overweight
what their peer group perhaps thinks.
That that makes sense if someone they quite
like happens to say and so on and so forth.
So maybe even if the angels can do what you're
doing, maybe the great unwashed like myself
should be quietly deferring away.
So, I think the evidence that we have that
certain forms of reasoning vastly outperform
other forms of reasoning is not evidence that
is very hard to access.
I don't think like the fact that these gradients
exist, and seeing the rough structure of it,
and seeing that it's a very lopsided, kind
of very heavy tailed domain where we see certain
forms of reasoning massively outperforming
other forms of reasoning, I think it's something
that is fairly accessible just looking at
the trajectory of history.
I agree that historical judgments are hard
to make, but I do think that across the board
we've seen a few specific forms of reasoning
just show up as successful again and again.
So I don't think that it's about like the
degree to which I have like a special ability
to judge other people's judgment in some major
like...
I'm not saying you have special access rather
than like, do you in fact see these people
out there?
Is it the case that you can find them.
There's a few super forecasters, not many
super forecasters, and...
The thing that I'm saying is that the natural
variation does exist out there, and it has
existed over human history, and already makes
a massive difference.
I'm seeing that whether you... when trying
to answer a question, make simple quantitative
estimates or not is something we see massive
amounts of variation in the population.
Just like it probably is about like something
like 5% to 15% of the population that I think
very frequently if they try to answer a question,
we'll go and make a simple quantitative estimate
of the problem, and there's about 85% of the
population who expect will not do this.
This is a variation that we normally see.
It's nothing special, it's not an extremely
high bar of epistemic excellence.
So it's just like a pretty standard skill
that some people have and some people don't.
And I think we see reliably the degree to
which we make these short, small quantitative
estimates tends to be highly predictive of
the ability to make reasonable judgments in
almost all domains.
And as such, I felt that the thing you were
just saying is that there aren't that many
people who really live up to this high bar
of excellence.
And I'm saying no, I think even small gradient
degrees to which you are slightly better at
reasoning make a massive difference in your
ability to predict the future.
And those was a small enough that we see natural
variation in different scientific fields,
in different people, and so on.
Yes, okay.
So I guess like perhaps where I would go is,
I would want to like say something like this.
So it may be the case, per like the previous
topic, like the main variance in like how
good your predictions are is like how good
you are at these other skills you've been
highlighting, like making Fermi estimates,
or like metacognitive skills are a really
big factor.
But compared to that, often domain expertise
like has like a bit of a boost, especially
initially, that rapidly diminishes.
So it's often the case you must put your faith
in epistemic qualities over mere domain expertise.
But then there's also the question of how
the distribution of epistemic qualities goes.
I definitely agree, you do like a little bit
better by doing Fermi estimates, and things
like that.
I guess I'll say is that often we take the
expert class of people who work in academia,
for example, like implicitly higher IQ, and
we know IQ correlates rationality skills as
well.
And so it seems to me that even if we can
find people who are somewhat better, who are
really good at these sorts of skills, I think
the typical academic is somewhat better, if
not really, really good at these sort of metacognitive
skills.
There's a lot of room at the top for which
they could reach.
Now I want to say that maybe the room at the
top, there's not many people in this like
far tail of this distribution.
And so then, I want to make feet this observation
I made before about, how many of these people
do we have?
And so on, and so forth.
Yes.
My model predicts something like, the effect
of this continues to be pretty large, like
all throughout the board, up until, if I have
a single research organization, or if I have
an academic field that has these skills generally
much more widely distributed, even identifying
the individual people... at every given step,
I think it's highly valuable to identify people
who have those epistemic skills, and not.
And I'm worried that this gives rise to something
like...
I wouldn't necessarily say that we disagree
on data mentioned that much, on like if somebody
would put a gun to her head and like what
would we say in the moment, what the correct
answer is, but something like the degree to
which we expect additional evidence about
people's expertise to continue updating our
opinions in this space.
Where I think like my model is something like
if you give me a rough guess of, let's say
we go back does Iraq have nuclear weapons
question, and you tell me most university
professors with foreign studies, or let's
say like 70% of them think this is correct.
And like, okay, I guess I'm going to make
it like an update on that because that seems...
let's say they think they don't have weapons.
Okay, I'm going to make an update on that.
I don't know.
I feel that I would drastically change my
mind if I would now hear that a specific university
that I think has like larger track record,
let's say something like, I don't know, I
don't have much meta-domain expertise of like,
foreign relations.
But if I would see someone who I think has
demonstrated good judgment here, maybe this
would come up as a question in the good judgement
project, I would expect that this would drastically
change my opinion again, where I would expect
that it would change this quite a bit.
And as such I suspect that there's this common
phenomenon that I see with people taking the
modest approach, where they kind of saying,
just trust the average of the experts.
And then I'm like no, look, there's this sense
of confidence that comes with that, where
you're like, ah, the average of the experts
is as best as we can do.
And like, no.
The average of the experts is definitely not
the best that we can do.
The average of finding out the epistemic practices
of every single person in the field and precisely
understanding them, and also understanding
the subject domain is the best we can do,
and both of those are really interesting directions
to go, and there's much more work to be done.
You can't just stand here and say that the
problem is like as much as we can possibly
solve, solved just because kind of you have
made a rough expert survey.
Maybe we should move on the next topic.
Yes.
We have 15 minutes.
But I'm going to briefly defend myself from
your slander.
So it's definitely the case that it's not
going to be that all experts are created equal,
even through common sense with the epistemics
that had been previously cashed out in mind.
And I agree as you say, you can do much, much
better than the average of the experts, but
the average method may do much, much better
for your first order impressions of view.
And now it may be the case amongst like very
sophisticated cognizers, they end up sort
of getting stuck in the middle of being a
little bit too modest, a little bit too stuck.
But I think definitely in the general world,
even I think in the EA community, you see
much more, I have this impression, I read
this a little while, I'm pretty certain this
is the right answer.
And they don't really consult the experts,
and sort of ask what the experts think, and
all the rest of it.
And I always want to say, there's much further
you could do in terms of aggregate agreement
and things like that.
But I think that asking the experts the last
thing you want to do.
But probably I think often it's a step forward
among prevailing... as a fundamental thing,
in terms of weighing the variance, it's like
you can imagine sort of back tracing prediction
markets and betting markets, and see how big
an impact is like updating different releasing
information to have to update it.
And then if we buy that, that there's a stronger
sense of how elitist we should be in terms
of weighing different sorts of expertise,
is a hard question.
We mentioned earlier this idea of like gun
to your head versus second order.
So maybe I'll let you take it away, with respect
to that.
I think think one of the things that influences
my perspective on all of this the most, it's
kind of...
I think we've been so far answering the question
of okay, let's see.
You have to give an answer to what the likelihood
of Iraq having the weapons is, in the next
five minutes.
That's it.
Now you need to make a decision.
For some reason somebody is putting a gun
to your head and you need to give the answer.
Usually you're not in that environment.
I think usually you're in an environment,
hopefully, where you are trying to explore,
you're trying to understand the topic, and
I think there's the question of what attitude
should we have towards the topic while we're
trying to understand the relevant subject
matter?
And I think a classical example that I would
give here is let's say you're a mathematics
undergraduate, and you are trying to understand
the basic theorems of mathematics.
Let's say you're in linear algebra.
Let's say you're in real analysis or something
like that, and you can easily imagine a world
where you hear like there's Zorn's lemma,
or like kind of like one of the most highly
confusing axioms of real analysis.
You go okay, I guess lots of people find this
reasonable.
So I guess that's what I should believe.
And then you continue doing your proofs in
mathematics and you dutifully write your proofs
on.
But I don't think like this will actually
produce a great mathematician.
I think the great mathematician will be produced
who will react to that being like, what?
Why would you be able to have an infinite
chain of things that bottoms out at some maximum,
and be able to always be sure that it has
this weird attribute?
That doesn't make any sense!
Like I can imagine here that's also equivalent
to this other thing!
Like there's a sense in which I expect emotional
engagement with the topic, to heavily depend
the degree to which they learn, and that I
think can very easily be mistaken for something
like immodesty, where like when I studied
mathematics I will have emotions about it.
So like, there's no way this is true, this
is obviously false, and it has nothing to
do with my degree of...
I mean it's something like... maybe a part
of me has that credence, but it is an additive
to have while learning, and I'm definitely
worried that...
I really enjoy this aspect of scientific fields,
of physicists and mathematicians, and also
many people in this community are people being
able to have these engaged conversations.
Of feeling like they're really interacting
with their models, and I'm worried that you
are mistaking that for something like immodesty.
Or maybe we disagree, maybe disagree that
it's a good idea.
No, I think in this case it's a good idea.
I mean, we may disagree on how we interpret
it.
It's obviously the case that you could be
like an epistemic free ride with modesty,
so you spend all your time trying to weigh
up for the expert classes and never form object-level
beliefs.
You're just like, I'm really good like calibrating
expertise now, I do that all day long, and
I have like no object level views on anything
beyond what I can do.
Right?
And that might be a really useful skill for
the wider community.
It could be like a really useful thing to
do.
Like maybe what a super super forecaster could
be like, like a valuable trait for the community,
on which much more later, but it's obviously
the case that if you want to become a mathematician,
that's like not the right approach.
Your job is to actually do object level work,
like object level progress, and that requires
like actually engaging with the object level
matters of things, like full impressions about
things, make arguments about various bits
and pieces, and I think you can do this with...
You can marry this with modesty or, marry
this modestly to modesty, in the sense that
you could like, go well, in my own personal
view, I do philosophy occasionally in my spare
time.
I'll have a little bit of a look at this topic.
I'm probably not going to make anything groundbreaking
to argue in favor of this particular view,
and then hopefully come up with something
interesting.
Even if like you put a gun, what's the most
plausible view on this topic?
I may end up like just deferring... well,
philosophy is hard to defer to.
But, the pervading view.
Maybe a better example, is like Scott Sumner,
who's like a master economist, talks about
a two level view of approaches.
Well, from my own personal perspective, I
just spend all my time doing stuff like doing
object level work, like hopefully proving
my theory is right, but if you put a gun to
my head I would actually do the overall thing,
and then like if there's lots of like sort
of like back and forth, like testing out models,
then I'm pretty cool with that.
I guess I get the impression, like usually
people, if you put a gun to their head, they'll
say basically the same thing as they were
saying just now.
And it's like less of like the whole, less
of a two-level approach, which I see.
If they were doing lots of that, I'm really
happy.
Or like reasonably happy, I'm pretty curmudgeonly
anyway, but like more happy than I usually
am.
I guess I just don't see as much as I'd like
to see naturally.
I sort of get this impression with us, we're
overconfidently holding first impressions,
and there's like benefits like doing this
normal thing, but I almost feel, not for you
guys, but some people I've talked to other,
use it as like an excuse, like, well I'm just
like trying to be really cocky here, but excuse
this.
Yes.
I think there's like one, one perspective
where I expect we both, at least my model
of our two models, we'll both agree that one
of the things that is most valuable here,
that I think is definitely correct, which
is that like this habit and pattern of people
actually betting their beliefs, and then having
bets with one another that are evaluated one
way or another, and putting probabilities
on that, and having those bets proportional
to those probabilities, is like a norm that
I'm very excited about.
And like, would love to see spread much more.
I'm really excited about people doing more
like, prediction market-like things, especially
because I think it actually gives you the
evidence of whether we see people being overconfident.
I think we could have the conversation about
when we look at the track record of the people.
I think this is like a conversation probably
for another day just because it's...
I think we might want to go into questions.
Great.
We have a few minutes left for Q and A there's
been a lot of really great questions that
have been submitted, but unfortunately we
will have time to go through all of them.
So please see Ollie and Greg for office hours
right after this.
So to just get started very quickly, a couple
of related questions that have been asked,
talked about the incentive structure of experts.
So one question asked, how do we navigate
expert advice coming from domains plagued
by sub-optimal incentive structures such as
medicine or academia?
And similarly, does the instinct to trust
friends whose model you have access to over
experts make more sense when considering that
some of these experts may have perverse incentives
that lead them to promote certain beliefs?
So the answer is, like, doing it is really
hard.
I'm not saying it's easy to discount heavily
by like who has skin in the game.
Who has biases, and things like that.
I guess like I have like a very long discussion
of this sort of thing.
I feel like I can rehearse what I typically
say so it's like... so okay.
So you think someone's like, you think expert
classes... so economists may be biased, or
medics are often biased about various things
for a variety of reasons.
But a question like, hey, is this bias like
selectively toxic, like versus you or versus
your friend, because politics gets in the
way.
Why are your friends less likely to be biased,
from the outside view even you would be.
And that's like okay, fine.
So maybe it does count against them but it
seems like, unless they're really riven by
biases, maybe on balance they're more likely
to be right.
Because it's easy if you're like, a pharma
company doing something, it's easier to get
your drug to market if it actually works than
if it doesn't.
You can still do it, but at least it's harder.
So it's solvent in that sort of way.
So I guess I want to often say that, yeah,
I know, I'm not sure either sometimes.
But I guess my view would be in many cases,
that you can undercut expertise with claims
of bias, but I think often this selective
undercutting effect versus you or versus your
friends, is not so large as to outweigh the
other stuff.
Like oh, maybe I should study this for 20
years.
Like maybe that does count for something in
many cases.
So I'll say that often like, wins the day.
It doesn't always, but it often does.
And if you thing like the expert class should
be got rid of, maybe should just defer to
other people.
So sort of like, people who read really widely
in this topic on the Internet, or some other
group who might not have this bias but also
might have thought about it much longer than
you have.
My perspective on that is qualitatively, I
probably agree.
Quantitatively I kind of go much further in
the direction of like, the bias of relevant
expert fields.
I expect, okay actually I think I want to
give a slightly different answer.
A different answer is something like, there
is a very dangerous framing in thinking of
expert fields as being biased.
I think there's like a general problem that
we have.
This like this is named for the modern fallacies
and biases field, that alternatively it's
the field of modern cognitive heuristics.
And there's two framings of the same field,
that I think highlight very importantly different
perspectives.
Where, when you think of fallacies and biases,
you think of how are they deviating from a
rational actor who is perfectly epistemic
and doing everything right, and you're like,
what is that really like?
I don't know how to implement it on a human
mind.
But in a cognitive heuristic domain, I'm asking
myself what procedures are they actually executing
to get to the answer and, now you're not,
you're not trying to answer the question of
deviation, which I think is really hard, but
instead of just trying to answer the question
of, well, what kind of questions do we expect
that set of heuristics to reliably answer
correctly, which ones will it reliably answer
badly, and just trying to build a model of
how they actually come to their opinions.
And kind of similarly to how I much prefer
the framing of cognitive heuristics over the
framing of cognitive fallacies and vices,
do I also much prefer the framing of building
models where experts come to their opinions,
as opposed to asking ourselves the question
of how experts are biased.
I think trying to answer the question of how
experts are biased without first having built
a model of how precisely they will come to
their opinions, is the doomed endeavor, and
instead I would usually try to answer the
question of, well, let's just ask ourselves
the question of what questions are they incentivized
to answer?
What things like, what do they output given
those incentives?
And then ask ourselves whether that is the
kind of process that we would trust in outputting
the correct answer.
And have a very different framing.
Unfortunately, we're all out of time, but
again Ali and Greg will both be holding office
hours right after this.
So let's give them a round of applause.
