Thank you to all of you for
showing up for this talk.
I'd just like to tell
you about how GiveWell
got started, what we're
trying to do, and go
into a little bit of detail on
a couple of the organizations
that we've evaluated, how
they've become recommended,
and then what we're
planning for our future
and how it's different than
what we've done in the past.
And then hopefully I'll be
able to leave plenty of time
for questions.
So GiveWell started when
I was working in finance
after college.
And after being on the
job for a couple of years,
a few friends and I
wanted to give to charity.
We went to Google, I
think, and literally
typed in the word
"charity" to try
to find information
about what charities do
and how well it works.
And we were really surprised
and frustrated by how little
useful information was out
there to help donors like us who
wanted to get as much impact as
possible with our donations--
give effectively.
And so we worked
essentially part-time
doing research trying
to figure out where
to give for about six months.
And in that process, we learned
how little information was
available, but also
found ourselves
incredibly interested in the
research that we were doing.
So in the middle of 2007, Holden
Karnofsky, who's my cofounder,
and I-- we left our jobs,
and we started GiveWell
as a full-time project.
And the goal was to be the
sort of resource for donors
that we had been
looking for when
we were in the private sector.
And so what GiveWell
does is we aim
to recommend a short list of
highly effective organizations
that best meet our criteria
with a focus on evidence,
cost-effectiveness,
and the ability
to utilize additional funding.
As a function of
our criteria, we now
focus entirely on global
health and development.
When we got started--
and you can still see a lot of
this research on our website--
we looked both at US
domestic causes and also
international causes.
But after doing the
work for a few years,
we saw how much further
$1 could go overseas--
how much more cost-effective
those options were.
And now GiveWell focuses on
global health and development
exclusively.
We ask a lot of
the organizations
that we review in terms
of data and monitoring
and transparency,
and so we ourselves
are trying to be as
transparent as we can.
So one example is we
post audio recordings
from our board
meetings on our website
if you really want to get
into the nitty-gritty details
of what goes on at GiveWell.
But importantly, all the
details of our research--
what we recommend and
what we don't recommend
and why-- are all
fully available there.
And that's intended
to allow people
to consider the recommendations
we make for themselves
and decide whether or not they
agree with the methodology
that we used.
GiveWell itself it
a not for profit,
and so we raise
money from donors
who support our operations.
And those donors' goal is to
create better, more useful
information for the
charitable sector.
And I think their attitude is
that by supporting GiveWell,
they enable another
set of donors
to give more effectively.
So what I want to do
now is just explain
a bit about our
process and criteria--
which is how we
work-- before going
into a couple of
short case studies.
So the four key
criteria that we are
focused on when we're
looking at an organization
are what's the evidence
that its program works,
how cost-effective
is its program, how
and whether can use additional
funding that it receives
from donors, and then will it
be sufficiently transparent
that it will share the
information that we
need with us.
So just taking a step
back for a minute,
the way that our
process works is we
start by trying to cast
the net as wide as we
can to look for charities
or programs that
could meet our criteria.
And because we are
aiming to recommend
a short list of
organizations to donors,
we can only spend a lot of
time on the organizations
that we think are going
to be most promising.
And so the first
stage in our process
is taking this long list of the
thousands of organizations that
implement programs
in poor countries,
the hundreds of programs that
have some evidence behind them
and trying to predict and decide
which ones will ultimately
do best on the answers to these
questions which are criteria
and then spending the
most time on the ones that
seem like they're
succeeding the best.
So the first criterion
that we're really
focused on-- the first
question we're asking
is, what is the evidence
that this program works?
And the vast majority of
charitable organizations
just don't have information
that demonstrates
that their programs are
making a difference.
To give one example,
before GiveWell got started
and I was just a donor trying to
give a small amount of charity,
I was really interested in
the cause of water in Africa.
And I called up 20
organizations and asked them
questions like, how
many wells will you
dig with an additional
$1,000, and do check
that the wells remain
in use over time?
Does the water remain clean?
And if it does, what impact
does it have on people's health?
They thought I was crazy.
And they said, we don't
even get questions
like this from our
institutional donors.
Why are you asking
us these questions?
But that experience has
been essentially the same
from the time that I was
just a small donor to now
that we're at GiveWell.
Most organizations
aren't asked and aren't
ready to answer
questions like that.
And so because of that,
when we ask this question,
we tend to start with the
academic evidence for how
well programs are performing.
So one example of a
charity we recommend
is the Against
Malaria Foundation.
They're a group that distributes
malaria nets in Africa.
And there are a
series-- more than 20--
randomized controlled
trials that
were conducted in
developing countries that
show that when you give out
nets, people tend to use them--
that these nets do
prevent cases of malaria
and then reduce child deaths
when people use the nets.
And so we rely on that sort of
evidence to say, this program--
the distribution of
insecticide-treated nets--
is one that can potentially
have a very big impact.
Then we'll go out and try
to find the Against Malaria
Foundation and say,
do you have evidence
that you are effectively
able to deliver nets
and that people use them?
And so in their case,
they'll purchase
nets which are then
distributed to people
who are at risk from
malaria, and they'll
go back and conduct follow
up surveys 12 months later,
24 months later, 36 months later
to see whether the nets remain
in good condition and whether
people are still using them.
The second big question
that we're asking
is the question of how
cost-effective is this program.
Let's say we know
that malaria nets do
reduce cases of malaria.
But how much does it cost to
deliver some number of nets,
and how many deaths
does that avert?
And in this question,
we're trying
to get at how much impact
the organizations have
with every dollar
that they spend.
This type of modeling
is incredibly difficult.
There are a lot of
difficult judgment calls
that go into this.
And what we try to do is
put it all into a model
that we put on our
website to be as
transparent as possible
about the judgment calls
that we're making in this work.
But it is this question-- the
one of cost-effectiveness--
that drove GiveWell to focus on
global health and development.
In our first couple of years,
we saw that with a program
like malaria nets.
Our best estimate was
that $3,500 donated
to a group like the Against
Malaria Foundation averted
a child's from malaria.
At the same time, if you looked
at the charter school programs
that we were evaluating as part
of the US domestic programs
part of our work, that
cost about $15,000 a year
just to put a child
through a year of school.
And comparing those two
numbers to each other--
$3,500 per death
averted versus $15,000
for a year of
schooling-- really drove
us to want to focus on
international causes
where it seemed like the
dollars could go much further.
The third big question that
we asked the organizations
that we're considering
is, how will they
utilize additional funding?
And this question is
important in a few ways,
but there are cases where
organizations can't utilize
additional funds to
do more of the work
that they've done
well in the past.
And one example where this was
particularly salient for us
is--
we were interested in
researching organizations
that fund surgeries
in countries.
So these could be surgeries
that repair cataracts or cleft
palates, and the organization
would pay local surgeons
to perform these surgeries.
But they got to the
point where they
had exhausted the
existing surgical capacity
in the country.
And they weren't able
to take additional money
and give it to surgeons and
have them do more surgeries,
because the surgeons
were essentially
doing as many surgeries
as they could.
Now, there are certainly other
ways that one could address
that gap-- the gap of
limited surgical capacity
in a country--
but this organization's
experience and track record
focused on delivering
dollars to help
surgeons perform surgeries.
And that intervention was no
longer directly available,
because they had exhausted
the existing supply.
And so this question of how
marginal funds will be used
is one we ask of all
of the organizations
that we're considering.
And then the final question
that we're focused on
is the question of transparency.
The vast majority of the
charity-specific information
that we need is not
publicly available.
They're documents and
reports that organizations
have themselves,
and we need them
to be willing to
share them with us
and then be willing for us
to share them on our website
where the donor community
uses our research for us to be
able to work with them.
So this is the set
of organizations
that we recommend today.
They work on programs
that focus on malaria,
deworming treatment,
which is treating children
for parasitic infections
that are extremely
common in poor countries,
vitamin A supplementation,
which reduces child mortality,
and then two other programs
that I'm going to talk about
in a little bit more depth
because I think they
illustrate the type of process
that we have--
No Lean Season, which helps
with seasonal migration,
and GiveDirectly,
which is a group that
directly delivers cash to the
poorest people in the world.
So I'm going to
move on now and try
to go through these
two case studies--
first GiveDirectly and
then No Lean Season.
And I should say that in both
cases, what I'm going to say
is going to stay fairly
superficial just because
of time constraints.
But if there's anything I say
that you're interested in more
information on, I'm sure that
you can find the details of it
on our website.
There's nothing I'm saying
now that is not already
on our website.
And if you do have any
questions, always just
feel free to contact
GiveWell directly.
We're happy to answer.
So GiveDirectly
is an organization
that delivers cash directly
to some of the poorest
families in the world.
These are families living in
East Africa and are earning--
the average consumption
is less than $1 a day.
And GiveDirectly's
idea was, can we
just give out cash and allow
people to decide for themselves
how they will best be helped?
Now, before
GiveDirectly existed,
we at GiveWell had
long been interested
in the potential
for cash programs.
We wrote a blog post back in
2009 where it was called "Why
not just give out cash?"
And we were really
surprised by the fact
that we saw so many charities
delivering health supplies,
running other types of
job training programs,
but at the time, there
wasn't any obvious way
for an individual donor
to just give cash.
And in many ways, you'd
expect that cash is
the simplest form of charity.
And so when
GiveDirectly launched
and we got connected
to them, we were
really excited to move
forward, because we
have this strong
starting assumption
that cash was a
promising approach
to helping some of the
poorest people in the world.
And so the way
that GiveWell works
is once we decide to go
deep on an organization,
we are largely going
through a process
of asking a lot of questions
and trying to find the answers.
And you'll see this
same method laid out
in the charity reviews
that are on our website.
We lay out a bunch of questions,
and then we try to answer them.
And for any program, the
questions tend to focus first
on the intervention--
so in this case,
cash transfers in general--
and then at a second
level, on the organization.
How effectively does it
deliver that intervention?
So in the case of cash,
the first question we asked
was one I mentioned before,
which is, what's the evidence
does it work?
Does this program
actually have the effect
that we think that it might?
The types of
questions that we had
were first, what do
people spend money on?
Do they spend the cash that
they receive on what intuitively
seem like productive uses--
on shelter, clothing, food--
or do they spend it on so-called
temptation goods like tobacco
or alcohol where you might
expect that the cash wouldn't
have the same sort of effect?
And in looking at the
evidence of cash programs,
we very consistently saw
that people spend money on--
broadly speaking-- productive
uses and very rarely
spend money on the sort
of temptation goods.
Another question
we had in this case
is that GiveDirectly's
intervention
was different than
the way that many cash
programs had been run before.
The standard cash
transfer program
had been a small income transfer
of about $10 or $15 a month
to poor families.
GiveDirectly was saying, we
want to make a large lump sum
transfer with the idea that
this will enable people
to invest in assets or do things
that they otherwise might not
have.
And the evidence base for that
type of cash transfer program
was much more
limited at the time
that GiveDirectly launched.
They worked with researchers
and ran randomized control trial
of their program of their
implementation in Kenya,
and they found similarly
positive results
from their program
to the results
that we had seen in
the other trials.
The other big question that
we had about the program
is the question of
cost-effectiveness.
How does giving cash compare
to other programs that
are trying to achieve a
similar goal of increasing
people's long-term
income or their ability
to consume what they need?
The way that we've
done this is we've
tried to compare
GiveDirectly's impact directly
to other charitable
programs that we have looked
at that have a similar goal.
And so in GiveWell's
mind, the program
of deworming-- treating children
for parasitic infections--
it may have this
long-term effect
of increasing people's incomes.
That's a long story
which I'm very
happy to go into in question and
answer if anyone is interested.
But we compared the
cost-effectiveness of cash
transfers to the cost
effectiveness of deworming,
really focusing on this question
of how does the cost for some
amount of increase
in consumption--
how is that different?
What's the ratio between
what GiveDirectly achieves
and what the deworming
organizations achieve?
And if you look at the model
that we have on our website,
our best estimate now is that
the deworming organizations
are approximately
three to 10 times
as cost-effective as
GiveDirectly is for the impact
that they're able to achieve.
Now, I throw out those
numbers somewhat easily,
but there are best estimates.
But there's really a lot of
underlying difficult judgment
calls that go into it, and
so we don't see those numbers
as the final decision
on how cost-effective
these programs are
relative to each other.
But it's one of the components
that goes into our decision
making process--
is trying to take our
best estimate of what
that final number is.
Then the other set of questions
focused on GiveDirectly
as an organization.
Were they able to
effectively deliver cash
to the people who needed it?
Could they get people
cash, or was it
going to be stolen
along the way?
Would people have the
money stolen from then
after they gave?
And so GiveDirectly created
a process for monitoring
that was able to detect when
problems were occurring.
And so there were
two types of problems
that they have been able
to detect over the years.
One is they found some cases
where low level staff of theirs
in country have stolen
some of the funds
that they were
trying to distribute.
And that actually gives
us more confidence in them
as an organization, because
the amount of money stolen--
I think it was about $25,000 was
stolen at one point in Uganda.
They're an organization that's
delivering about $25 million
in cash transfers every year,
so it's a very small portion
of what they're delivering.
But the fact that they
were able to detect that
and then were willing to
disclose it and write about it
publicly shows us that their
monitoring system is actually
identifying the
sorts of problems
that it should be
set up to detect.
The other type of concern
we had about GiveDirectly
is their original model
focused on only giving cash
to households that were below
a certain poverty threshold
and not giving cash to
other households that
were above that poverty level.
And the way that they
determined poverty level
was by looking at the material
that household's roof was
made of.
So thatched roof houses received
cash, metal roof houses did not
receive cash.
And there's a lot
of concerns that you
could have about that process.
But one of the things that we
were most concerned about was
you literally had
neighbors, one of whom
would receive this
big infusion of cash
and the other one would not.
What they monitored
was the level
of conflict and disagreement
in the village as a result
of the cash transfer.
Again, they measured fairly
high rates of unhappiness
among non-recipients
of cash, but fairly low
rates of actual violence
or theft of money.
That, again, gave
us some confidence
that they were focused
on asking and answering
the questions that seem
most relevant to determining
whether or not they were having
the impact that they were
aiming to.
I think they've now
changed their model
and are distributing
cash to entire villages
to avoid this problem.
But even now, we still
have ongoing questions
about GiveDirectly
and cash transfers.
Some of the questions that
are most front of mind
for us today are, how long do
the benefits of cash transfers
persist?
Are these benefits that
allow people to purchase
more over the very
short term and then
those effects fade
away, or do they
set people on a
upward trajectory
that allow them to earn
more and more over time?
And I think this is
highly uncertain.
We've built in an assumption
into our cost-effectiveness
model, but it's certainly
something that could--
getting better information
about this question
could affect how we
view GiveDirectly.
And then the other big question
is, what are the larger scale
effects that these
cash transfers have
on the people who
don't receive them?
And this is a question
that hasn't been studied
particularly well to date,
but there's a large study
that GiveDirectly is
currently conducting.
And hopefully
there'll be some data
available in the next
few months that's
going to try to address this
question of what effects
is their program and having
on the broader economy.
The second charity that
I want to talk about
is a slightly different story.
So this is an
organization that we
recommended for the first
time at the end of last year.
It's called No Lean Season,
and it's a seasonal migration
program in Bangladesh.
And the idea here is
that the program provides
agricultural workers
with a small loan
during the lean
season-- so basically
the period between
planting and harvest
when they have
very few employment
opportunities at home.
And the idea is that
by providing them
with this small loan,
that incentivizes
them to migrate to
an area where they
might be able to earn
more money than they could
if they stayed home.
We first learned about this
program in 2011 or 2012
with research that was done
by Mushfiq Mobarak, who's
a professor at Yale.
And we saw this trial
and its results.
And we were really
excited about it,
because the impact on
consumption were very high.
Now, when we talked to Mushfiq
at the time, we asked him,
these results look really great.
But is this something
that exists?
Can we fund this program?
And at the time, it was
just a research trial,
and there was no opportunity
for donors to help scale it up.
But Mushfiq-- and then in
partnership with Evidence
Action and the head at the
time, Alex [? Huang, ?] and then
GiveWell in our
incubation grants program,
which I will talk a little
bit more about in a minute--
we gave Evidence
Action some funds
to see whether or not they could
create this as an organization
or create this program so that
it could be further scaled up
and continue implementing
the model in the field.
So again, here, we went
through a very similar process
of asking the key
questions that came
to mind to see how it
matched up against our other
recommendations.
So some of the
questions we asked
were you know very basic ones.
How much do people
earn when they migrate?
How much do they send back?
Do they migrate again in
future years with the incentive
and without the incentive?
And the answers to those
questions were fairly positive.
But other questions that we
asked as we've gone along
relate to, what is the--
I guess you could call it
the disutility of migration?
When the men-- and
it's always men
who are migrating in this
program from the rural areas
to an urban area.
What is their experience
like in that urban area?
Maybe they earn more
money, but maybe that
is counterbalanced by some
very negative experience
that they have in the
months that they're
working in this new location.
Similarly, what's the
effect on the families
that they left behind?
You can imagine a
family where the father
is leaving for three months.
Presumably that has some
sort of effect on the family
that he leaves behind.
How do we account for that
effect in our overall take
on this program?
And then finally, how does
this program's scale-up
affect the availability of
labor at the destination?
Maybe at very small scale,
there's plenty of jobs.
But all of a sudden,
you scale this up
to be a much larger
scale program,
and you could run out
of available employment
opportunities.
So in all of these
questions, I think
we got some of the
answers as the program was
growing from its first
days to where it is today.
But it's still a fairly
new, fairly small program,
and it's one that we in the
organization and researchers
are still watching
very closely and moving
fairly slowly with to see how
the answers to the questions
change as it grows.
Just in terms of
what it took to get
this program off the ground,
one of the big questions
that they had to answer was,
what implementation model
should they use in trying
to keep this program going?
In the original
trials, the researchers
just delivered a cash subsidy.
It was about the
cost of a bus ticket.
And as they tried to take
this program to larger scale,
they decided instead
to offer this as a loan
that someone could repay.
And I think the idea was
both that they partnered
with a local organization
in Bangladesh
that itself is a
micro-finance institution,
and delivering loans was
what that organization does.
But also, it had the
possibility of increasing
the cost-effectiveness
of the program
by reducing the cost that
it took to deliver it.
But this question of what
the implementation model is--
that remains a
very open question.
And so I think in this
case, the big open questions
that we continue to
have about this program
are, how will it change as
it scales up in the area
that it's currently in?
Is there promise to
take it to other areas?
And will the repayment
on these loans
mean that the implementation
model that they chose
is really the right one?
So I just want to
transition briefly
to talk a bit about
where GiveWell is going
and how that's somewhat
different than where
we've been in our past.
Most of what GiveWell
has done to date
is we look out at the world and
we say, what research and what
charitable organizations exist?
And then looking at
that, we determine
which groups should be
GiveWell recommended charities.
In order for groups to
make it to that stage
where they can be a
GiveWell recommend charity,
someone else had to provide some
funding earlier in the process
to help them get off the ground,
to fund the research that
helped them determine whether
their program was working,
to give support to build
up a monitoring system.
And as part of what we've called
our GiveWell Incubation Grants
program, we've delivered
funding to help organizations
go from where they are
to potentially become
future GiveWell top charities.
So the example of No Lean
Season that I talked about--
that was a grant that we made
out of the GiveWell Incubation
Grants program.
But other types of support
that we've provided have been
funding to other
startup organizations--
an organization
called New Incentives,
which focuses on conditional
cash transfers for immunization
in Nigeria--
a focus on trying to gather
additional research that
could better inform the
decisions that we're making.
So GiveWell Incubation
Grants made a grant
to a randomized
controlled trial that
was run via J-PAL for incentives
for immunization in India.
And then finally,
we've also tried
to give grants to help improve
the quality of the monitoring
that both our top charities
are currently doing
or the monitoring that
other organizations could
do to enable them to become
future GiveWell top charities.
The final piece I want to talk
about is just this question of,
how does GiveWell
evaluate ourselves?
And on one hand, a core piece
of how we evaluate ourselves
is the quality of our research.
But there's really
no objective way
to assess the quality
of our research,
and so we aim to
be as transparent
as we can, as open to
criticism as possible
so that we can make the
research as good as it can be.
But the research
only matters if it's
causing donors to give as
a result of the research
that we do.
And so what we track
is our money moved,
and that's the dollars, the
funds that go to the charities
we recommend as a direct
result of our research.
We do a lot of
work to track that,
both by encouraging donors
to give through our website
directly but also by going back
to the charities themselves
and asking them what they know
about where dollars have come
from that they've received.
And over the last
few years, we've
directed about $100
million each year
to recommended organizations.
And one of our goals
going forward on this side
is to try and increase these
numbers as much as possible.
So I want to pause here and
I guess just open things up
for questions from you.
I really appreciate
everyone coming out
and would just love to hear
whatever questions you have.
[APPLAUSE]
Excellent.
Maybe we can take a batch
of two or three questions.
I'll repeat them back
for our web audience,
and then Elie can tackle them.
My name [INAUDIBLE] I'm a
second-year MBA student.
I was question about from the
charities that are already
using an RCT-valid method.
When GiveWell is
evaluating them,
how much additional
burden of evidence
gets placed on the charity like
any other institutional donor
places the burden of evidence?
Any other question at this time?
Go ahead.
I was just going to
ask more generally.
So similar to your
beginning story, sometimes
friends of the family ask what
charity is best to give to.
And ones that maybe
not necessarily
are in your wheelhouse of
global health and development--
would you have
recommendations on how
to evaluate what charities
to give to and what metrics.
I know one metric that
people love to [INAUDIBLE]..
They look at overheard, and
they say, oh, this is not
an efficient charity, right?
So just any guidance on that
process and any thoughts
on things like--
I don't know if you've
heard of Charity Navigator
or these websites which try
to say which charities do well
and which charities
don't do well.
I'm not expecting necessarily
an answer to this.
It's more of a morality
sort of life view on this.
But how do you reconcile
the long-term and short-term
decisions on charity?
So you can perhaps save a
child's life in East Africa.
But if that child is
going to grow up and maybe
be part of a 56% unemployment
rate or part of a country
that has a 56% unemployment
rate, how do you reconcile--
obviously, saving a
child's life is important,
but how do you reconcile in
the long term a person's life
overall and how the impact of
a donation in the United States
that is an education
donation versus a donation
somewhere else that's a
life-saving donation-- how
that impacts the future of
a person's life long term?
So three questions I
guess our web audience
wasn't able to hear.
One is that what's the
additional evidence required
from charities that are
already implementing
an evidence-backed
intervention from an RCT.
The second-- what metrics
can we use in addition
to what you've shown so far
to determine good charities?
How do you weight overhead--
some of these other groups
like Charity Navigator.
And then how do you reconcile
long-term and short-term
outcomes?
All right.
Memorized all three of those.
So the questions that we ask
charities that are implementing
evidence-backed program--
very heavily depending on
the nature of the program
that they're implementing.
But there's no doubt that we're
imposing a significant burden
on them to engage
with us, and so we
try to set up our process to
be cognizant of that fact.
And so what I mean
by that is try
to have clear
milestones along the way
so that they can balance the
time that they're devoting
to sharing information with
us with the potential benefit
of getting funding as a result
of a GiveWell recommendation.
From talking to the charities
that have gone through
the process, I think that the
first evaluation that we do is
sort of similar in intensity to
what other institutional donors
are asking charities to go
through but very different
in kind, because we focus
primarily on a charity's past
activities-- what have they done
and how well it has worked--
rather than just
primarily asking
them to tell us how they
will do some new project.
After that first
year, though, where
they've reached that status
of being a top charity,
the burden of engagement
with us I think
is far less than it
is on an ongoing basis
with a donor like a major
foundation or a government
donor.
Second question-- OK.
This is like a test
that you get at MIT.
Can you remember
all three questions?
The metrics on
charities and thinking
about overhead and other
groups like Charity Navigator.
So overhead is the most
common metric that is used,
but I think it's a
very misguided one
to be so heavily focused on.
If you just imagine some
of the charities that--
you can think about a charity
that digs wells in Africa.
If they spend 100% of their
money on their program
but they dig wells
that don't work,
that's not an
effective organization.
What you actually care about
is whether the organization
is running a program that
actually makes a difference.
So then what can you do?
I mean, the first answer
is, there's no good answer.
And that's why I left my
job to start GiveWell,
because I don't
think you can answer
this question as
well as I'd like to
in a short part-time way.
It's something that I think
requires a lot of time.
But the tips I'd
give for people who
want to give effectively
outside of the recommendations
that we need--
the first is just be as
open-minded as possible.
A donor can say, is this
particular charity a good one?
On the other hand, let's say
that charity is a food bank.
They don't need to focus
on a particular food bank.
You could say, well,
what is the best food
bank in the Boston area?
And just opening up the sort
of scope of that question
I think is one way to have
much more effectiveness.
Another question is to
just ask the charities, how
do you know whether or not your
program is making a difference,
and see what answers
you get to the question.
And then the final
one is ask them
what they would do differently
with additional donations.
And check back a year later and
see whether they actually did
the things that they described.
We wrote a blog post that
lays out some of these.
It's called something like six
tips for giving like a pro--
nice clickbaity title.
Didn't get a lot of
clicks, but the post
is up there on our website.
And we'll go into that
in a little more detail.
[INAUDIBLE] Charity
Navigator and these
websites that try
to differentiate
between the [INAUDIBLE].
Yeah.
Sites Like Charity Navigator
and really any site
that's trying to rate a huge
number of organizations based
on basically publicly
available data
is just not able to answer the
question that donors really
need, which is, how effectively
are they helping the people
that they're trying to serve?
Charity Navigator is
trying to get to that.
But I think that's
the sort of question
that even when you go to the
level of depth that we have,
we still have lots
of open questions
about the organizations
that we review.
And a site like
Charity Navigator
that's looking at 10,000
organizations just
doesn't have the
ability to answer
that question effectively.
And then the question
on the outcome,
long-run, and reconciling
short-run and long-run
outcomes.
Yeah, I think there's actually
two separate questions here,
both of which are really
salient to GiveWell's work.
The first one you framed as
short-term versus long-term,
but you might also
look at it as easier
to measure versus harder
to measure outcomes.
This is a big challenge for us.
There's nothing that is inherent
to GiveWell's approach that
requires us to only focus
on the measurable outcomes.
But what is core
to our approach is
that we want to be able to
make recommendations that
don't default back to the
expertise that GiveWell has
or our subjective judgment about
what is good and what is not.
Instead, we want
to be able to make
a data-based case or at
least a reasoned-based case
that the outside world
could read, understand,
and disagree with.
And that means that
it's much easier for us
to engage with the more
measurable parts of the charity
world and not the
less measurable parts.
Our aspiration going
forward-- and we've
taken some steps
in this direction--
is to try and tackle the
less measurable activities.
But it will be really
challenging because
of that constraint
that we want to be
able to make that case
that doesn't ultimately
just rely on our judgment.
The other question you asked
about was moral values.
How do you weigh
increasing incomes
versus saving child's lives?
This is something that
we have no answer to,
but we are forced to
make that choice when
we try to allocate the funds
that we're responsible for.
And so every year, we
say, how much do we
give to Against Malaria
Foundation, which
is saving child's lives
versus GiveDirectly, which
is increasing people's incomes?
The way that we've
done this is we
first tried to look at
what research already
existed about how
policy makers themselves
try to make these trade-offs
between different types
of moral good.
And frankly, there's some
information about this
out there, but nothing that
is particularly reliable
or I would want to
just rely on ourselves.
The literal process
we've used to date
is staff members at GiveWell
will put in their values
for how they would trade off
between different moral values,
and those values ultimately
inform the recommendations
that we make.
We don't like that process--
that it's just this
small staff of ours
that's making those decisions.
And so what we're
trying to do is
find ways to gather more
information that could
help us inform this process.
And what we're doing
now is we're working
with a group called IDinsight.
They're a group that does high
quality evaluations of programs
in poor countries.
And with them, they're
working to gather more data
on beneficiary preferences.
How would the people whom we're
trying to help-- how would they
make the trade-off
between averting a death
and having additional income?
Great.
We have two questions from
the web that I wanted to ask.
One is, are there differences
in recommendations
that you make for
smaller donors as opposed
to larger donors such
as Open Philanthropy?
Is there a different
process that you use
to recommend programs for them?
And second is, in terms
of some charities,
smaller organizations
aren't ready for an RCT.
But to be able to make
a case that they're
doing effective work, what are
your top two tips for where
they should start
to generate evidence
or how they should start
to generate evidence?
Got it.
So our recommendations
are the same
whether they're for big
donors or small donors.
Good Ventures, which
is a foundation that
has given substantially
to our recommendations
over the last few years, was
started by Dustin Moskovitz
and Cari Tuna.
Dustin is one of the
cofounders of Facebook.
We make the same
recommendations to them
that we make to everyone else.
And so literally,
the $10 donor is
giving to the same place
as the $10 million donor.
In terms of the other
question, which is--
So some NGOs are not well
situated to set up a large RCT.
They want to begin
to demonstrate
evidence of effectiveness.
What would your top
two tips be for them?
Yeah.
I think the biggest thing--
the approach that
seems best to me
is really trying to set up a
system that could find failure.
A lot of the evaluations
that I've seen--
it was never plausible
that they would
have determined that the
program was not working.
And so in a sense,
GiveWell is one
of those small
organizations that's
trying to determine whether
or not we're having impact.
And so when we
evaluate ourselves,
we're probably still failing
in the ways that anyone might.
But having a goal of being
open to finding failure
and to really seeing
things go wrong I think
is probably the single
most important aspect
of the sorts of evaluations
that we want to see.
You talked a little bit
about scale effects,
and I wonder how you feel
about GiveWell's scalability.
At the moment, you're
kind of just moving
some charitable donations
around on the margin,
and I know that
[INAUDIBLE] the charities.
But if you could
snap your fingers
and tomorrow there was no
more domestic any donation
and it was all funneled
to international sources,
do you think that
would be a good thing?
And how does that play about
how you think about the future
if you become bigger
and more successful?
So just a question on
the general equilibrium
effects of contributing
to GiveWell
and moving money
potentially from domestic
towards international causes.
How do you feel about
GiveWell's scalability.
Yeah.
If we had to deal
with that situation,
we'd have to approach
it differently.
And so because we're
very far from it,
it's not a question that
really weighs on us.
Instead, the situation
we see ourselves in
is that collectively,
the groups that we've
recommended to date--
we think they could
effectively utilize.
Their collective room from our
funding and our terminology
is around $300 million a year.
We're only able to direct them
about $120 million a year.
So there's a vast amount
of funding they could still
receive before they start
having those significant
diminishing marginal returns.
If we were in some
other world where
we were directing
huge amounts of funds,
we'd have to take a
different approach,
because we would quickly
surpass the capacity
of the organizations
that we know
about that are effective
to utilize those funds.
How do you fund your research?
Do you have like a team of 20
analysts doing this research?
How are they paid?
Is it competitive?
Getting an idea
for how you manage
to run people well and
have an effective team.
Question on how GiveWell
funds its research.
I mean, we're a
nonprofit, and we just
need to raise money from donors
to support our operations.
We're a pretty small
team, and we operate on
a fairly small budget compared
to the amount of funding
that we direct.
And so we've been really
fortunate over time
that we really
haven't had trouble
raising our operating budget.
Our operating budget is
around $3 million a year.
And then our goal in terms
of how we compensate our team
is to make compensation
a non-issue overall.
And what I mean by
that is we don't
want to pay people so
much that they would want
to come to work for
us for the money,
but we also don't want
anyone to not take the job
because we're not paying enough.
And we definitely are not
trying to compete then
with what a tech company
could pay someone.
People do have to take some
discount to work for GiveWell,
but we don't want
someone to really
have a material
effect on what they're
able to do with their
lives because they
came to work for us.
So given the vast
amount of charities
that exist in the world, how
do you initially cast your net
and then start to narrow
it down as to which charity
you're going to evaluate?
So how does GiveWell
determine which
charities it's going to
invest how to do research on?
I mean, we literally start--
every US registered
charity files a tax form
with a little code that
says whether or not
it works internationally.
So one of the
first things we did
is we went through
10,000 tax forms
just to know what basic
activities charities undertook.
Now, a lot of the
groups we recommend
are not US registered charities.
There are international
groups, so we
tried to get lists of groups
that work on programs.
And I think I personally have
gone to more than 1,000 charity
websites just trying to ask
basic questions about what
do they do and what's the
evidence that it works.
And so we basically just
try to start very broadly
and then try to have some
you know basic heuristics
for narrowing down the
field and then slowly spend
more time with them over time.
And again, I think if
you go on our website,
you can see all of the
organizations whose websites
we went to and how they scored
on the different heuristics
that we use to decide
which ones to prioritize.
This is a question
that might be a bit
difficult as an
empirical question,
but I was wondering if you have
a sense at least anecdotally
thinking about
shifting donations
to international development
and specifically whether you
have a sense of your
objectives or your impact
as potentially crowding
out domestic donations
versus bringing in marginal
donors who aren't donating
because of the very
reasons that you mentioned
in your motivation for
founding GiveWell, which
is to say that on one
hand, you might be changing
the composition of
what people donate,
but you might also be
expanding the pool of donations
because people are now
willing to donate that weren't
otherwise going to donate.
So kind of a question
on to what extent
are we moving money from
domestic or international
to bringing in
additional or new donors.
Yes.
We survey our donors to ask
them the question of what
they would have done
in GiveWell's absence.
And so what they
tell us is about--
most of them are
either reallocating
from international organizations
to GiveWell's top charities,
and those tend to be big name
international organizations.
UNICEF, Red Cross,
Oxfam are the sorts
of names that they'll mention.
And then others will say that--
sort of as you described-- that
GiveWell's research gave them
the confidence to give more than
they would have in our absence.
These surveys are not very
methodologically rigorous,
and we don't really
have a sense of what
the true counterfactual is.
But it's a really
important question for us
in trying to have a better
sense of what the impact is
that we're actually having.
We have another question
from our online audience.
Do you ask that the
organizations you support
have one or two other
funders at the same time
too as to avoid dependency
on one major donor?
Does GiveWell more
likely to invest
in organizations that
are not dependent on just
GiveWell as a donor and
have other funding streams?
We often are trying
to make a risk reward
calculation in the
decisions that we're making.
And if we're the only
funder, then it's
a higher risk proposition,
because there's
some possibility that
the organization wouldn't
exist if we didn't
continue to support it.
But in cases where we think
that the upside is high enough,
we have been and we'd be ready
to be a group's only funder.
So in one case that
I mentioned earlier--
the group called New
Incentives, which
does conditional cash transfers
organization in Nigeria--
we're not literally
their only funder,
but we certainly account for the
vast majority of their funding.
But we think they're
so promising that we're
willing to take that
risk in their case.
I think we have time
for one final question.
Do you see GiveWell engaging
with policy makers or those
who manage aid at the
government or regional level?
Do you see that as a role
for your organization
or trying to transfer
your expertise there
or some sort of advising?
Or is that outside
of your circle now?
So to what extent does
GiveWell advise other donors
on their charitable
contributions?
Specifically those who
[INAUDIBLE] donations
[INAUDIBLE] at the
government level--
so advising a USAIE-type
organization as opposed
to a private organization.
So specifically bi-lateral,
multi-lateral contribution,
international aid.
Yeah.
We would.
We would very much like to
do that, because I think
if we were able to enable
those funders to give
more effectively, we could
have a lot of impact.
The challenge that we face
is that GiveWell was set up
around the idea of
serving donors that
were similar to
the type of people
that Holden and I were
when we started GiveWell.
Pretty much everything
that we've done
has been targeted at
helping individuals decide
how to give effectively--
individuals who don't know
a lot and don't
have a lot of time
to spend on their
charitable giving.
But I think at this point we
are hopeful that the knowledge
we've built up and the
approach we've taken
could be helpful to
others, and so we're
aiming to see whether or not
we can be helpful to people
in that position.
Well, on behalf of J-PAL
and the economics department
here at MIT, thank
you very much,
Elie, for joining us for D2P2.
Everyone give a
round of applause.
[APPLAUSE]
