Welcome back to the NOC course on qualitative
research methods my name is Aradhna Malik
and I have been helping you with this course
and we are coming to the end of this course
these are the last few lectures Now so we
discuss the whole bunch of things and you
know Ive seen some feedback and we were gone
through a lot of things in this course now
what we are going to do today is we are going
to talk about the implications of whatever
we have studied the applications of what we
have studied to the social world so lets move
on with it
Today we are going to talk about in this lecture
we will try and understand how social programs
can be understood through qualitative evaluation
and how qualitative research applies to the
evaluation of social programs its very very
important in this day and age especially in
our country it is such an important thing
you know the government is investing heavily
in the development of India
The government is investing heavily in the
alleviation of the daily life situations of
the rural country especially people living
in rural areas through a whole lot of social
programs so it will be helpful for us to understand
how whatever we have studied till now applies
to understanding the social programs to evaluating
them to making them better and to making the
more fruitful
What do social program evaluators do Social
program evaluators aim to inform and improve
services programs policies and public conversations
at hand Social program evaluators try and
you know they engage in participatory action
research we talked about participatory action
research they go into the field they find
out what is required they do what is required
They get feedback from the field they bring
it back and then discuss amongst themselves
get the opinion of experts analyze whatever
they have seen and heard and observe and then
they improve on the current processes and
they go back and implement these changes these
improvements in the field in an attempt to
make their work better in an attempt to see
to improve the way social programs are progressing
so that is what they do
The contexts of program evaluation: these
are contexts about contested social policies
and programs about how and by whom resources
are allocated and about competing civic values
both in the global arena and local community
These are the context where you know it is
difficult to hold somebody accountable if
these are context in which you know a very
fine line has to be drawn between two competing
Paradigms
For example you know one needs to start something
new one needs to introduce new policies and
programs but critics say that till you have
improved till you have reached a certain level
of implementation with the programs that are
currently going on why should you invest anything
into something new and the governments say
well we cant really you know we are trying
to improve whatever we have going on
But it is also important to look forward it
is also important to invest part of what we
have in something new and then maybe the fruits
of our labor from investing something investing
in something new can later be fed into improving
what is already ongoing which will take up
a lot of time and energy
So that is where you know these are competing
values should I just sit and use all my resources
to improve something that is already going
on for example health health care now the
government has set up these hospitals everywhere
the whole countrys network with these healthcare
facilities and in many places unfortunately
in rural areas we dont have staff or we are
understaffed I should say we dont have stuff
we are understaffed in some areas
We have in some areas we have equipment that
people dont know how to use in some areas
the hospitals are located in areas which are
difficult for people to reach now these are
real problems the government did whatever
it could and they assessed the situation and
gave us this facilities Now they are also
trying to set up newer hospitals in areas
that were not reachable areas there also trying
to add to the facilities that are already
there in the current hospitals
And maybe some people would say well why dont
you first improve the infrastructure you have
why dont you first fully staff this hospitals
instead of setting up new hospital and we
say yes that is a very valid concern but we
also need to look forward we need to have
Super Specialty hospitals in rural areas in
smaller area in hard to reach areas we also
at the same time we also need to improve the
primary care health facilities within the
country
So we have to divide things we cant just focus
on one and completely neglect the other and
that is precisely what we are talking about
in the slides we are talking about competing
policies and programs competing civic values
who decide what to allocate How much too allocate
Who decide on what basis you know we will
allocate a certain proportion to all or how
do we make this decision is not who decides
what is how these decisions are made and that
is what social programs social program evaluators
facilitate okay
The factors influencing evaluation of social
programs: the first one is political power
that is number one policy making and within
political power we have policymaking we have
implementation we have value attributed to
the voice of the beneficiaries which means
that the people who are making these policies
have the power in their hands they are the
ones who are equipped who are empowered who
are knowledgeable or who have the how to develop
this policy they have the information that
the required to make these policies
Then they are also the ones you know that
there are people who are equipped who are
empowered to implement this programs so they
are the ones who has the power Then the value
attributed to the voice of the beneficiaries
that also add to the political power that
ultimately influences how social programs
are evaluated after all people who we are
doing all this for should also tell us or
should also be heard to find out whether what
we are doing is of used to them or not
So it is very very important to give a voice
to the beneficiaries it is also very important
to find out what people who are actually being
affected by this programs feel Credibility
and perceived role of evaluators are evaluators
qualified Are there what are they doing as
evaluators Are the assessing for the sake
of assessing only and writing reports and
being done with that or are they going to
do something about it
So that also affects how social programs are
evaluated or and also how you know these programs
are the evaluation that comes from these programs
is incorporated into further program implementation
okay Involvement of evaluators in the community
now we are talking about participatory action
research if the evaluators are actually participants
in the community
The evaluation will be much richer because
they are also stakeholders in the program
implementation they are also being affected
by the program implementation either they
are being affected as beneficiaries or they
are being affected as the implementers or
they are being affected as observers but they
are being affected so it just depends on how
much is state they have in the actual programs
Then the stakeholder interests what to people
want Are we giving them what they want The
power stakeholders have over the implementation
of social programs Many times stakeholders
actually convey they have the voice of or
they have the power to let the implementers
and policymakers know what they want and their
voice is really heard so all of this influences
how social programs are evaluated
If I have a stake in the program then I will
evaluate it very differently as opposed to
if I am just supposed to write a report and
submitted to the government without really
feeling the pinch or getting the benefit from
that social program So Im not seeing everybody
does it but my perspective will be different
if emotions are involved I will and it could
go the other way also
If my emotions of very heightened with the
program evaluation then I may not be able
to write an objective report I may not be
able to write a balanced report So the way
the program is evaluated depends on a lot
of things who we are Whether we are trained
Whether we are able to do it or not Okay
How does a qualitative program evaluation
fit into all this some contemporary genres
of program evaluation approaches: are postpositivism
Utilitarian pragmatism interpretivism and
constructivism and Critical Social Sciences
Let us look at each one of these
Postpositivism is oriented around the interest
of policy makers and funders postpositivism
looks at what policy makers and funders want
It involves causal questions about the degree
to which a program has attained desired outcomes
while retaining costefficiency compared to
its critical competitors
So postpositivism involves the degree to which
or how the outcomes have influenced the questions
or you know whether there is a cause and effect
relationship between the questions that were
asked and the outcome that has been produced
It addresses recurrent demands for accountability
in social expenditures So when we spend so
much of money when will look at when we evaluate
programs from a postpositivists point of view
we try to find out whether the money that
we used for these programs was actually used
properly or not
It prefers scientific logic because of its
perceived objectivity replicability and verifiability
So it is focused on logical explanation of
how things were done whether they were done
right or wrong the evaluation is very very
logical in this sense and evaluation is replicable
and verifiable okay
The primary values that are promoted efficiency
accountability costeffectiveness and policy
enlightenment the policies need to be such
that according to evaluate programs and policies
from postpositive point of view we want evaluate
the policies to be ironclad we want them to
be very systematic very logical and we want
to be able to very clearly see a cause and
effect relationship
The key audiences in this case are the people
who we write these evaluations are the highlevel
policy and decision makers the funders and
the social science community because there
for the research depends on such valuation
Preferred methods are quantitative: experiments
and quasiexperiments surveys causal modeling
costbenefit analysis plus minus one thing
meaning to another so the preferred method
of evaluating social programs from the post
positivist point of view is quantitative
If I say if you know A + B should be equal
to C it cannot be C and X and Z at the same
time that is what we are trying to find out
from this Typical evaluation questions are
intended outcomes attained and attributable
to the program did we get what we invested
into it or not is this program the most efficient
alternative or are there better alternatives
In which the outcome to input ratio is higher
okay
Utilitarian pragmatism: Addresses issues that
are not transferred well to real life social
contexts if refocuses the attention on the
needs of decision makers especially the practical
needs of onsite decision makers for program
information useful for management decisions
When we talk about utilitarian pragmatism
we are talking about the practical application
of things which may not be a plus minus game
sometimes the practical application of things
requires or inherently bills waste into it
So the cost benefit relationship may not be
very sound but it is implemented and the effects
are much more visible later okay
Primary values that are promoted: are utility
practicality and managerial effectiveness
even if the cost benefit ratio is not very
high as long as the people who are benefiting
from it feel good about it it is good as long
as the people who are stakeholders in the
process feel that it is of used to them it
is all right
Now the key audiences here are midlevel program
managers and on site administrators a friend
of mine once told me that when you implement
a program when you do something you must account
for or you must build at least 10 to 20% wastage
in that program that is what he told me this
is somebody who has been doing a lot of social
program work this is somebody who has been
out in the field for a long time and she said
everything doesnt happen on plus minus
If you spend 5 hours in the field you may
need to spend 2 out of those 5 hours just
building rapport with people its not about
giving them something letting them do something
going into the field you know building a road
and coming out you may need to build a rapport
with the people in the community you may need
to just sit put your things aside sit and
talk to the people in the community I will
give you a personal example at this point
Many years ago in the year 2001 in the summer
I went to Romania for three week internship
at an orphanage with a bunch of other students
and we were group of students and we went
to an orphanage and we went without lofty
plans and you know this was one of the very
few orphanages that had opened its doors to
foreigners so it was nice of them this was
in a place called Iasi IASI spelt as a Iasi
pronounce as Iasi this is in Romania
So we went there and this was such an enlightening
experience as far as the program implementation
is concerned we went to the plans we said
okay we are going to spend 8 hours there will
spend 4 hours in the morning go for an for
lunch for an hour go pack spend 4 more hours
guess what we went there we should our plans
to the administrator and she said close your
books what She said close your books
We do not need you outsiders students one
from India one from France one from England
one from Japan coming and telling us there
is one lady from Sweden she said we dont want
you people coming in telling us what how we
need to do a work better do you know what
we are facing We said no She said what we
need you to do is we need you to spend time
with these children this was an orphanage
the caretakers overworked
So she said you spend time with the children
and we will tell you what help us with what
we are doing and that will be enough service
for us and we said okay So initially some
of us had not done this we were confused we
said what we are here to do something substantial
we have to submit a report and she said with
you write whatever you want in the report
but we need you to sit and spend time with
children
So they were six of our students sanding we
had to host students each of us was adopted
one to two kids and all we did was play with
these children these children needed our time
they needed hands on affection some of the
children were abandon some of them were orphans
they did not have family they did not a people
to love them so we went we gave the time we
gave them affectionate in the process those
of us who knew how to help disabled children
we help them
There is a boy who cant walk properly which
he had polio and then its been so long you
know this example just came to mind so we
help the child we would just do is exercises
with him the whole day that was what was really
required now that was utilitarian pragmatism
a social program with the utility for the
beneficiaries really got our time and we give
children time
So the workers in the orphanage could do what
they had been hired for they were understaffed
and overworked so we give them some respite
with the result the overall performance of
that often is improved why because they were
doing their work we were just helping take
away the time or we were just filling in the
time that they wanted to spend with the children
that they could not that was not part of their
job description so that is how our time was
very fruitfully utilized
This is what this means the cost benefit analysis
would have yielded probably a negative ratio
but here the benefit at your son is got out
of our work was tremendous we spend three
weeks there it was amazing for everybody okay
primary values promoted utility practicality
managerial effectiveness The Key audiences
are mid level program managers and on site
administrators
The preferred methods of eclectic mixed structures
and unstructured surveys interviews observations
document analyses and panel reviews The typical
evaluation questions are which program components
work well and which need improvement How effective
is the program with respect to the organizations
goal and mission Who likes the program Now
the example that I just gave you which program
components work plans do not work
Onsite thinking and creativity work you go
there you assess the situation and then if
you need to plan you only plan for the day
making a plan ahead of time for three weeks
does not work will not work okay so how effective
is the program with respect to the organizations
goals and mission maybe from the perspective
of the organization that had sent us there
this was not there primary agenda the agenda
was to serve the community
So we say serve the community to my memory
if I remember correctly they that was not
very clearly detailed so we just have to serve
the community the community said we need to
be served this face we said okay here is our
server so we adaptable who likes the program
The kids loved us the children in that orphanage
loved us because we were just sitting and
playing with them and you know we adapted
what we had planned to do to bear needs
So they were happy they could do their works
for the overall output was great and that
said into our reports so that is how we did
things okay
Interpretivism and constructivism these approaches
seek to address interest and honor the experiences
of stakeholders closest to the program is
evaluated – namely program staff and beneficiaries
by giving voice to their contextualized program
understanding So when we evaluate programs
from the interpretivist perspective we can
understand how the stakeholders are interpreting
how they are feeling what they are feeling
about the program
The one before this was the practical applicability
who is happy Now we saying we are just trying
to find out how these people feeling
Primary values promoted pluralism understanding
contextualism personal experience so what
are we trying to promote we say okay within
a particular context one thing that may not
work in one context will work in another context
because the demands of different Key audiences
are program directors staff and beneficiaries
we are trying to see things from different
perspective and we are trying to understand
what is going on Why it is going on And you
trying to evaluate the programs from their
perspective
Need not necessarily be immediately beneficial
is it solving whole problems Is it giving
a something new to think about Is it distracting
us from our current problems So that we get
some respite while we you know we will get
a break from trying to solve problems and
then go back to those problems what is this
program doing Preferred methods are qualitative
now we are getting somewhere the methods in
this are qualitative case studies open ended
interviews and observations document reviews
and dielectrics
We are only trying to find out what is happening
so we go into the community and we try and
talk to people try and understand what they
felt so this is qualitative completely Typical
evaluation questionnaire how is the program
experienced by various stakeholders In what
ways is the program meaningful And that is
where qualitative research comes in we try
and get this descriptive data
We come back to analyze it and we say within
this context this is what makes sense within
that context that is what made sense they
may not be a very simple formula to understand
the cause and effect relationship in the situations
but this is really helping the community so
thats where qualitative explanations will
come in
Critical Social Sciences are openly ideological
that explicitly advanced a particular value
for agenda for example social justice empowerment
critical race consciousness or social change
Rationales for evaluation are the advocacy
of ideals and values and the answering of
certain program questions And advancing activist
ideology with grounding in a constructivist
philosophy We are trying to look at everything
from an ideological point of view
The primary values promoted are emancipation
empowerment social change egalitarianism critical
enlightenment with everything from a very
very very very focused very powerful microscope
we put everything under a very powerful microscope
and we say okay is this really contributing
to the business can we see something else
which is not really visible so we critically
evaluate ideologically evaluate from the point
of view of what a program should do in the
long run When we talked about ideology we
are talking about the higher goes
So key audiences here are program beneficiaries
and their communities the activists Preferred
methods are participatory action oriented
stakeholder participation in evaluation agenda
setting data collection interpretation and
action these are the methods this is purely
qualitative and the evaluation questions are
in what ways are the premises goes for activities
of the program serving to maintain power and
resource inequalities in the context
When we talk about critical theory we are
talking about identifying inequalities and
addressing them and that is precisely what
search program evaluations do Are be able
to identify those so we break up its first
decide how what with what idea in mind are
we evaluating it what are we going to do this
evaluation how are we going to use it and
that will help us determine the paradigm from
which we approach our evaluation and then
that will help us find out what kinds of tools
we use for evaluation whos going to be benefited
etcetera
The logic of justification for evaluations
conducted qualitatively Constructivism: constructivist
inquirers seek to understand contextualized
meaning to understand the meaningfulness of
human actions and attractions as experienced
and construed by the actors in a given context
So we are talking about contextual understanding
of meaning we try and presents the meanings
in contextualized manner
Constructivism is the emotional linguistics
symbolic interactive political dimensions
of the social world weve already talked about
constructivism and their meaningfulness or
lack thereof are all constructed by agentic
human actors These constructions are influenced
by specific historical geopolitical and cultural
practices and discourses and by the intentions
noble and otherwise of those doing the constructing
So these constructions are multiple and plural
contingent and contextual We look at them
within the context Im not going to go through
it because we discussed these terms ad nauseam
the task of the constructivist social inquirer
just a second let me see how much is left
probably have to get it spill over okay 
so maybe we can stop this discussion here
at critical social sciences thats all we have
time for today in this lecture we will discuss
the dress to wait in the next lecture thank
you very much for listening
