Hello, everyone.
I'm Catherine Hollander, a research analyst
focused on outreach at GiveWell, and I'll
be talking about our incubation grants program
today.
I think when EA Global reached out about the
possibility of providing a talk about GiveWell's
work here, when I learned that the theme was
Stay Curious, our incubation grants program
felt like the perfect thing to talk about,
since this is really work that is intended
to expand the horizons that we're considering
at GiveWell.
Just briefly, I'm going to talk about what
GiveWell incubation grants are.
They're a newer part of our work.
The different types of incubation grants that
we've made, some of our past incubation grants,
as illustrative examples, I'm not going to
probably talk about all of the incubation
grants that we've made, but would be glad
to take questions about other grants if you're
aware of them, or curious to hear about other
possible areas that we've supported during
the Q and A, and I'm hoping we can leave about
10 minutes for that, and then just briefly
touch on where incubation grants fit into
GiveWell's plans for the future.
GiveWell, as many of you might know, is a nonprofit
that's dedicated to finding and recommending
outstanding giving opportunities.
We publish a short list of top charities every
year that represent some of the opportunities
that we think are extremely evidence-backed,
cost effective, transparent and in need of
additional funding.
We think that these are great opportunities
for many donors who are interested in using
our research.
Incubation grants are something that we started
to think about a few years ago.
GiveWell was founded in 2007, so we're a little
over 10 years old now.
Around 2014, we were starting to think, "Hey,
we've been recommending pretty similar types
of programs for a while now."
Our top charities list hadn't changed a lot.
It was looking like cash transfers, insecticide
treated nets to prevent malaria and deworming
pills continued to be the things that were
standing out among our criteria.
We were continuing to look for other things
that met those criteria but really not finding
a lot that was new.
We started to think, are there things that
we could be doing to help develop the pipeline
of potential future top charities?
One thought that we had was, as a recommender
and as a funder, we're really stepping in
at a fairly late stage in charities' life
cycles.
We're looking for things that are extremely
strongly evidence-backed, which means that
by the time they're getting to GiveWell's
recommendation, someone has probably already
funded them, such that they could develop
that track record that we need to look at
to see to make our recommendation.
Someone has funded the program research that
we rely on to say, like, a particular program
is very evidence-backed and we're interested
in finding charities that support that program.
The incubation grants program is us at GiveWell
stepping in as that earlier stage funder and
trying to specifically develop the types of
charities and the types of program research
that we think will ultimately make it to our
top charities list, where we can recommend
them to a large number of donors and direct
a significant amount of funding to support
their work.
This is the kind of goals of our incubation
grants program, developing the pipeline of
future top charities, and also improving our
understanding of our current top charities
since, particularly as GiveWell has grown,
we're directing a significant amount of funding
to the groups that we recommend and so we
want to make sure that we answer questions
that we have about them, and incubation grants
is one way that we can do that.
Finally, I mentioned us stepping in as a funder,
us making grants.
The funding that we are providing for the
incubation grants program comes from Good
Ventures, which is a large foundation that
we work closely with.
You might be familiar with them if you're
familiar with the Open Philanthropy Project.
They were started by Dustin Moskovitz who's
one of the co-founders of Facebook and Asana,
and his wife Cari Tuna and they're significant
supporters of GiveWell's top charity program
as well.
Right now they are the sole funder of GiveWell's
incubation grants work.
We're not recommending incubation grants to
the donors who use our research.
We are instead hoping that incubation grants
builds the list of things that we are ultimately
publishing for the public and recommending
to all donors.
I briefly touched on a few of the different
types of incubation grants, where we're primarily
thinking of our incubation grants program
in three different buckets.
The first is work to seed potential top charities.
This is, perhaps, fairly straightforward.
We're looking at organizations at an early
stage of development that would not meet our
current top charity criteria.
They don't yet have the kind of track record
that we'd like to see to point to them and
say, we think that there is a very strong
case for impact that we can make to donors,
but that we think have the potential with
additional funding and scale to become charities
that we would recommend to donors.
There are a few different grants that we've
made in this space.
One is to a group called Charity Science:
Health which came out of the EA movement.
Some of you here might be familiar with it.
They're providing SMS reminders for routine
immunizations in India, but they're a very
new organization, and so, not one that has
the kind of record that we'd want to see before
we recommend them as a top charity, but something
that we're interested in following as they
grow in scale.
The second type of incubation grant is work
to support monitoring.
Monitoring is really, really important for
GiveWell.
We're looking at charities where we want to
really feel that we can demonstrate that they
are having an impact.
We want to see very good information from
those charities, that indicates that they
are successfully reaching people with their
programs, that the people who are being reached
with the programs are the people that they're
intending to reach and that we're kind of
following any longer term impacts that we
might be interested to see.
One example of this type of grant that we
made was for work to identify cataract surgery
organizations where we might be able to help
them implement monitoring since, previously,
we were interested in the intervention of
providing cataract surgeries to improve visual
acuity due to cataracts, but we hadn't found
an organization that met our bar for having
monitoring, that was demonstrating who they
were reaching with their surgeries, how much
those surgeries were improving their vision,
and also just sort of what the counterfactual
story is, whether those surgeries would have
otherwise happened in the absence of that
charity's work.
Those were the types of monitoring questions
that we hadn't answered, and a type of incubation
grant that we made was to support work to
identify and build with a charity, a monitoring
system that would help us answer those questions.
That project is actually paused right now.
I'm happy to get more into that during the
Q and A, but that's sort of an illustrative
example of that type of incubation grant work.
The third type of incubation grant is to support
program research.
GiveWell's research process is sort of divided
into two different steps.
The first step for us is generally doing a
survey of the academic research, the independent
research that's been conducted on a particular
program to help people.
This is not research that's generally done
by charities themselves.
It's often done by academics, and that's our
first step in our process, is looking at the
academic research to see which programs we
think might be the most cost effective and
evidence-backed, such that we would be interested
in finding a charity that's working on that.
The reason we start there is just because...
There's two reasons.
One is that charities don't often, themselves,
do this type of research.
Running randomized control trials, for example,
is quite expensive and not necessarily the
sort of value added that most charities have.
The second is that some of the biggest drivers
in differences in program effectiveness and
cost effectiveness is just which program a
charity is implementing, so whether you're
delivering insecticide treated nets or performing
surgeries, that in and of itself will be a big
driver in differences among charities' cost
effectiveness, perhaps more so than looking
at two charities that are doing the same program.
So we're interested in that as a first step.
A type of incubation grant that we're excited
to make is one that looks at a promising area
of academic research, but maybe there's only
been one study that's done or we have open
questions and funding further program research
to support our understanding of that area.
Just a few examples of past incubation grants
that we've made.
The first is a group called New Incentives,
and I believe they were our first incubation
grant recipient.
If not first, second.
Back when I mentioned that we were starting
this program in 2014, or around then, we made
our first incubation grant to New Incentives
in January, 2014, I believe.
The program that they were implementing at
the time that we were quite interested in
was a conditional cash transfer program that
was intended to incentivize pregnant, HIV
positive women to deliver in a health facility,
deliver their child in a health facility in
order to prevent the transmission of HIV from
the mother to the child.
This looked to us like a promising program
but it wasn't one...
This organization was brand new.
We just didn't feel like we had enough information
to say "we really feel solid about the case
for this charity" such that we would recommend
it as a top charity, but we thought, this
seems like something that we would be interested
to learn more about, and seems like it could
become something that we'd recommend.
We made our first grant to New Incentives
back in 2014.
In 2015, they ended up realizing that they
weren't reaching enough HIV positive women
with their program, such that they decided
to expand the program to just be to incentivize
pregnant women to deliver in health facilities,
so they sort of shifted a bit there, and then
we spent the next year looking into the impact,
the research that we could find on facility
delivery on neonatal mortality, so, does delivering
a baby in a health clinic mean that it has
better neonatal mortality outcomes.
What we found at the end of 2016 was that
we didn't think there was sufficient evidence
to convince us that that was a program that
we should recommend.
Through this whole time we had been engaging
with New Incentives and we felt really positively
about them as an organization, so we decided
to make another grant to them to support a
shift that they were making into providing
conditional cash transfers to incentivize
routine immunization.
They've kind of gone through a few different
iterations through our Incubation Grants program.
What we're planning to do with them now is
to run a randomized controlled trial, which
I mentioned earlier as a type of evidence
that we're particularly interested in.
It's a type of study where you divide people
into two halves, a control group and a treatment
group.
You randomly assign them and then you look
at the differences in outcomes between those
two groups.
We're now working with another research partner
that I'm going to talk about momentarily called
IDinsight to run a randomized controlled trial
of their work and are now looking at them
as a potential top charity contender, something
like in 2020, so a few years out.
It's kind of an iterative process.
I think this illustrates that.
The timeline might be quite long from making
an initial incubation grant into joining the
top charities list, but we've certainly learned
a lot through the New Incentives process and
are continuing to follow them through our
incubation grants program.
The second group that I wanted to talk about
is another incubation grant that we made very
early on in the program.
This was another, kind of early 2014 incubation
grant, to a group called No Lean Season.
We became interested in the program that No
Lean Season is operating, which is a seasonal
migration program where they are providing
small subsidies to individuals in rural northern
Bangladesh to enable them to temporarily,
seasonally migrate during the time of year
when seasonal poverty is a large issue and
job opportunities are harder to come by.
The idea is that if you give people a small
incentive, that enables them to do something
like buy a bus ticket to travel somewhere
else where they can work, that can then have
beneficial income and consumption benefits
for their household.
In 2008, this crossed our radar a little later,
but the study was done in 2008, and it was
a randomized controlled trial that was done
by a Yale economist named Mushfiq Mobarak.
He ran a small trial of this program.
I think it was about 1900 households that
were involved with the initial trial, that
saw significant benefits for household income
and consumption as well as continued migration
the year following the program in the absence
of the incentive.
When we looked at that we said, this seems
like it has the potential to be one of the
most cost effective programs that we've come
across in a long time, and we're interested
in this, but 1900 households, this kind of
one study that was done in this very particular
area, we're not sure we would want to recommend
that quite yet as one of our top charities.
We made this initial incubation grant to Evidence
Action which is an organization that runs
the Deworm the World Initiative which is one
of GiveWell's longtime top charities, to scale
this program.
Evidence Action had also been interested in
this program, so we sort of came together
at that moment and provided them funding to
scale this work.
In the subsequent years, they conducted a
number of additional randomized controlled
trials.
They scaled from about that initial 1900 households
to, I think, around 100,000 households in
2017.
Last year we decided we were at the point
where we wanted to evaluate them under our
top charity, traditional top charity criteria
since we thought that they might now be at
the point of having that track record that
we'd want to see, and that there might be
sufficient evidence there where we'd say,
this is something we want to recommend to
donors.
We sort of put it through our normal process
last year.
We spent a lot of time talking to them about
how much funding they needed.
We built a cost effectiveness model of their
program which is a really important input
for us.
We looked at their monitoring and we did a
site visit to their work in Bangladesh.
Coming through that whole process, we ultimately
decided, at the end of 2017, that, in fact,
No Lean Season did meet our top charity criteria
and so we added it to our list of the nine
groups that we currently recommend.
This was our first incubation grant from the
beginning to the top charity list example
that we have and we're hopeful that this is
something we'll see again in the future.
This is currently the first and only that
has gone from that initial grant all the way
through to meeting our criteria.
The final example of a grant that I wanted
to talk about is a group that we're working
with called IDinsight.
Buddy Shah, who's the CEO of IDinsight is
giving a talk at EA Global tomorrow.
I think it's at 3:30 and I would recommend
it, because they're doing really, really interesting
work.
I mentioned that one of the things that we're
really interested in at GiveWell is this academic
research.
That's a really important first step of our
process.
What we found was that academics aren't always
incentivized to provide decision-relevant
information to folks that are implementing
programs on the ground.
What we're often curious about is how will
this program look when it scales, how will
it look in another area, and there's not always
the academic incentive to conduct research
that answers those questions.
IDinsight, we think, fills a really unique
role in the development space by specifically
targeting providing that information.
They'll run, sort of, low cost RCTs, randomized
controlled trials, and try to provide decision
relevant information to policy makers, or
people implementing programs to help them
really decide where the most effective places
are for them to work, how cost effective their
programs are.
This seemed really exciting to us.
We have a lot of those questions that we're
hoping to answer.
We've partnered with them through our incubation
grants program to work on a number of different
projects with us where we're hopeful that
they can help us better understand the spaces
that we're interested in.
I mentioned with New Incentives, that first
incubation grant recipient that I had, that
they were running a randomized controlled
trial, IDinsight is working with us on running
that randomized controlled trial to help give
us that information.
They're also working with us on our project
to help better understand one of our current
top charities, the Against Malaria Foundation.
The Against Malaria Foundation is a group
that's been on our list of top charities for
a very long time.
I think they initially joined our list in
2011, and in 2016 we realized that we had
some questions about the ways in which they
were conducting their monitoring of their
program.
The basic way that it works is that AMF, Against
Malaria Foundation, goes back to households
that have received insecticide treated nets
and it checks every six months for the first
few years after the implementation, or, sorry,
it checks a percentage of households that
received the nets every six months after implementation
to see whether they're hung and also what
the state of the net is, since we believe
that they degrade over time, like, develop
holes and become less effective.
I should say, the nets are things that you
hang over your bed.
They're treated with insecticide and they
both block mosquitoes from biting you and
also kill them, from the insecticide, so we
think they're very good for preventing malaria.
In 2016, we wrote about some of the questions
that we had about the way that they were conducting
surveys in the Democratic Republic of Congo
and in Malawi, and we partnered with IDinsight,
to help us both better understand their current
procedures for conducting those surveys by
having IDinsight observe them doing that on
the ground, and then also to make recommendations
for the Against Malaria Foundation to improve
how they're conducting those surveys.
This part of their work is really an example
of a type of incubation grant that's geared
toward improving our current list of top charities
rather than sort of building a new organization
to join it, but we think it's really important
to have this work as well, since we're directing
significant funding to the organizations that
we recommend in the tens of millions of dollar
range.
The final project with IDinsight that I wanted
to call out is another pretty different type
of work than the type of work I mentioned
before, which is that, I mentioned cost effectiveness
as a really important factor for GiveWell
in coming up with our list of recommendations
at the end of the year.
We're looking for things that we could make
a strong case are extremely cost effective.
Where this gets particularly challenging is
that a lot of the different programs that
we're interested in, and that exist in the
world, have different outcomes.
I mentioned cash transfers, deworming and
the distribution of insecticide treated nets
as some of the programs that were on our list
for a long time.
We think the benefits of those programs are
very different.
Distributing insecticide treated nets to prevent
malaria, we think reduces child mortality
from malaria.
Distributing deworming pills, we think, has
the potential for increasing the incomes of
children who receive deworming pills, later
in life.
Those are really different outcomes, and so
when you're trying to think, which of those
two is more cost effective, you have to come
up with some sort of comparison between the
two that you can make.
The way that we do this right now is that
we have staff members who work on GiveWell's
research team, informed by everything from
their own philosophical values to the World
Health Organization's assessments of the different
burdens of different diseases and other information
that we have to make those trade-offs, but
the piece of information that we think we
are most lacking to make those trade-offs
is how the recipients of the interventions
of our top charities would make those trade-offs,
how they would decide, "How much do I value
increasing income versus improving health?"
We have done reviews of the literature to
see if we can find information to inform that,
but we've generally concluded that that information
doesn't really exist and hasn't been collected.
A lot of the research that's done on how those
trade-offs get made has been in rich countries,
rather than in the poorest parts of the world
where GiveWell's top charities tend to operate.
The final project I wanted to mention that
we're working on with IDinsight is a project
to survey the beneficiaries of our top charities
to ask them and try to better understand how
they would make those trade-offs to inform
our cost effectiveness models.
This would be something that, again, I think
would fall under the category of improving
our current recommendations and strengthening
the information that we can provide to donors
who are interested in supporting these organizations.
Those are a few of the incubation grants we've
made.
This piece of our work has been ramping up
in the last few years so there's quite a few
that I didn't mention but I'm happy to kind
of talk about in the Q and A, but just kind
of wanted to close with a quick thought on
some of our plans for the future.
Especially for this audience, wanted to convey
that GiveWell is moving toward...
We've been around for 11 years.
We're moving toward being open to some of
the sort of riskier ideas and harder to prove
cases for impact than I think we have historically.
Incubation Grants plays a role there in giving
us the ability to try to answer some of the
questions that maybe have been challenging
for us in recommending things in certain spaces
in the past.
One example of sort of the riskier, harder
to prove types of areas that we're interested
in now that, in the past, I think we've largely
said, this is outside of our domain, is work
in policy advocacy interventions, where I
think we're interested in finding things that
might be significantly more cost effective
than our current group of nine top charities
focused on global health and development.
One of our best guesses for where we might
find that is interventions that are targeted
toward getting policies to pass that might
affect many, many people for a long amount
of time, that that might be a really cost
effective thing for us to do.
I think we'd be interested in supporting research
through incubation grants that will help us
better understand the causal impact of policy
advocacy interventions since I think one of
the biggest challenges there is you're not
just going to have a randomized controlled
trial of, like, the ability to pass a policy.
It's going to be a little harder to tell that
story, and so we want to support things that
will help us potentially consider those types
of areas in the future, but this comes with
the second bullet point which is that we're
still very interested in maintaining our level
of transparency.
What I mean by transparency is really being
able to make a case, publicly, that is vettable
by donors who are using our research for why
we think that something meets our criteria
and is very cost effective.
We want to be able to point to our rationale.
We want to be able to point to evidence and
monitoring that really demonstrates why something
has ended up on our list and why we are saying
this is one of the best opportunities we're
aware of, that meets our criteria.
Incubation Grants work is a tool that we're
using to help us do that, and particularly
as we move toward some of these areas that
maybe in the past have been harder for us
to assess by our traditional criteria.
I think we're very excited about this work.
Just to give you a scale of it, I think last
year we made something like close to $10 million
worth of incubation grants, and so it's a
big part of our work.
It's not the only part of our work.
We're still looking for organizations that
meet our traditional criteria, not through
the incubation grants program, but we're optimistic
about helping move toward building that pipeline
and bringing more organizations into our consideration.
I'm going to pause here for questions.
I think we're running low on time.
I'm around for office hours at 4:30, I believe,
and I'm also happy to follow up via email
with anyone who has questions after the talk
too, so, thank you.
Thank you so much. I really appreciated your talk.
Unfortunately we are a little brief on time
for questions, but I'll ask you one or two.
I think the kind of obvious one is, if you're
a person who's interested in doing program
research or helping an organization with monitoring,
where can you find the outstanding questions
that you'd like to see answered, or find the
organizations that you could tack on to?
Yeah, so, where would you work if you were
interested in incubation...
We have a list of all the organizations that
we're currently supporting through incubation
grants on our website, and we also have a
list of priority programs that we've identified
as promising areas, but we don't yet recommend
charities that are working in.
That's kind of one signal of where there might
be either opportunities to start organizations
or talk to us about that, and then I think
for thinking about where we might potentially
make incubation grants in the future, the
sort of core question that we have is, how
likely is this to become a GiveWell top charity?
We actually publish forecasts along with our
incubation grants so you can kind of see,
like, okay, we think that this grant has a
30 percent chance of resulting in a top charity,
and so we're trying to sort of come up with
initial rough models, think about how likely
something is to become a top charity, and
then make our decisions about what to fund
based on that.
Cool.
That's great.
Let's say someone decides that they're going
to run with a project, and now they want to
get GiveWell's attention and see if maybe
they can get an incubation grant and grow
into a recommended charity.
How do they get on your radar?
Yeah, I think just reaching out to us.
You can email me or email the info@givewell.org.
I think we want to have early conversations
before anyone spends a ton of time developing
something if they're trying to get incubation
grant funding.
We'd definitely appreciate the chance to kind
of check in and talk about what our interests
are and hear about what the potential program
is, but yeah, I would say just emailing us
is probably the best way to begin that conversation.
Cool, all right.
I think we're going to have to call it there,
but thank you so much, I really appreciate it.
