Female Speaker:
It's challenging to get people totally psyched
up to think about statistics for two hours,
and I'm very glad that you were game to do
so.
I'd like to introduce Dr. Sam Woolford, who
is a consultant to Abt Associates, which is
our statistical consulting company.
Sam has worked with both Bentz [spelled phonetically]
and I on -- working on the validity and reliability
of new scales that we've developed to do a
variety of different things.
And he is here today, in particular -- this
is the second workshop he's done.
He came two years ago?
Sam Woolford:
No, it was way longer than that --
Female Speaker:
It was way longer than that?
Sam Woolford:
Yeah.
Female Speaker:
At least six years.
And he taught us structural equation modeling,
which I can still say today, I can't do, but
I understand [laughs], which is really a nice
feeling.
So, this time, a research team -- and some
of them may be on the phone -- that I was
working with, and I was asked what the difference
between an exploratory factor analysis and
principal components analysis.
And I sort of answered boldly, like I knew.
And then we started looking it up, because
people weren't sure that was actually a distinction.
And in fact, we found discrepancies everywhere
we looked.
People were saying, you only use one in one
case.
And only one in another.
Then, it seemed to be a disciplinary difference.
And then, that fell apart.
And all of a sudden, we realized they were
described to be different things, but we had
no idea, really, what the argument or logic
was about when you would choose to use one
over another.
So, it was a very humiliating experience.
And we decided we need an expert to kind of
set ourselves straight.
So, that's the history of how Sam got here.
And so, we're really glad you're here.
And we desperately need you.
Sam Woolford:
Well, thank you.
Female Speaker:
So, thank you.
Sam Woolford:
Thank you.
Might be nice -- can we just go around the
room?
You can -- I promise I probably won't remember
your names, but it's just nice to have that
familiarity while we're together.
Male Speaker:
Hi, I'm Joel [spelled phonetically].
Sam Woolford:
Joel.
Female Speaker:
I'm Ruka.
Sam Woolford:
Ruka.
Female Speaker:
I'm Laurie.
Sam Woolford:
Okay.
Female Speaker:
Sophia.
Sam Woolford:
Sophia.
Female Speaker:
Janine.
Sam Woolford:
Janine.
Female Speaker:
Megan.
Female Speaker:
Allie.
Female Speaker:
Becksy [spelled phonetically].
Female Speaker:
Melissa.
Sam Woolford:
Melissa?
Female Speaker:
Angel.
Sam Woolford:
Angel.
[unintelligible]
Female Speaker:
Sharine [spelled phonetically].
Sam Woolford:
Sharine?
And we've got --
Male Speaker:
William.
Male Speaker:
Jeff.
Sam Woolford:
Okay.
I do this with my students every semester.
It takes me at least four or five weeks to
remember all the names and faces.
But, it's always a nice thing to at least
be able to talk to people.
So, what I did was take some of my own lectures
and pull them together to try and answer this
question, which I transformed into -- for
those of you, as I'm looking, are maybe old
enough to remember the movie -- was a Billy
Crystal movie called "Analyze This".
Which -- taking a little license on that one,
okay?
[laughter]
At any rate, there is a lot of confusion.
My students -- I mean, I've been teaching
this now for at least the last 10 years.
And this is a very hard differential for most
students to understand, because it's a little
bit nuanced.
And there's enough similarity in what goes
on when you do a principal component analysis
versus an exploratory factor analysis that
we lose track of what is the difference.
And maybe, the true difference is never really
set in until you start using it, and then
you forgot what those are.
So, let's make this as informal as possible.
I don't know exactly how long this is going
to take.
But what I've done is I'm going -- I'm calling
it an overview, but what I'm really going
to do is kind of walk through a lot of the
components of factor analysis that are common
to both principal components and exploratory
factor analysis.
Now, this is going to get even more confusing
because another common term for exploratory
factor analysis is common factor analysis,
okay?
So -- but that's not common, because it doesn't
apply to principal components.
So, we'll try and keep that straight.
If you get confused, again, stop me.
Let's talk about it, whatever.
But ask questions as we go through the material,
so that you make sure you get them answered.
I'm going to cover kind of the similarities
first.
And then, I'm going to look at some of the
specifics for both -- for each of principal
components and exploratory factor analysis.
And then, I've brought an example that's a
little bit -- it's not totally contrived,
because the data's real -- but it's contrived
in the sense that you probably wouldn't use
both of these methods on it.
But it provides a great example of what happens
if you use the different methodologies, and
you can really see how things are different,
which, hopefully, will resonate a little bit
better than if we tried something a little
bit closer to your world, okay?
And then, I'll summarize some of these similarities
and differences.
And then, if we have time -- which I'm not
sure we will -- but we can touch a little
bit to confirmatory factor analysis, which
is related, in a way.
But that's kind of a separate discussion,
okay?
All right.
So, just to get started, let's take a test,
okay?
So, let's just -- again, don't feel embarrassed.
Just, I'm going to ask you some situations,
and have you indicate whether you think this
is a situation for principal components, or
a situation where you'd use exploratory factor
analysis, okay?
So, if we wanted to reduce a large number
of variables to a smaller number of factors
that we were going to use in analysis later,
how many people think that's principal components?
Okay.
How many think that's exploratory factor analysis?
Oh, come on.
Everybody has to vote, now.
[laughter]
You can't have this --
Female Speaker:
I don't know.
Sam Woolford:
Okay.
How about reallocating the variation in a
large number of variables?
How many would think that was principal components
analysis?
How many would think that's exploratory factor
analysis?
Okay.
Female Speaker:
No one's voting.
No, we're getting some -- everybody's sneaking
one up, you know.
They don't want to show anybody else, but
they're letting me see it.
Okay.
How about creating an orthogonal representation
of the original variables?
Principal components analysis?
Okay.
Exploratory factor analysis?
Okay.
And that can solve problems with multi-collinearity.
Identifying underlying dimensions in the data,
such as constructs.
Principal components analysis?
Okay.
Exploratory factor analysis?
Okay.
And the rest of you are not making a commitment
here, one way or the other.
Okay.
Regression with many correlated variables.
Principal components analysis?
Okay.
Exploratory factor analysis?
Okay.
Creating a hypothesis for a confirmatory factor
analysis.
Principal components analysis?
Okay.
Exploratory factor analysis?
Okay.
That's probably the one we got closest to
getting correct.
Okay.
So, let's -- I mean, that's good, because,
you know, clearly there's some difference
of opinion here.
And that always makes for a more interesting
discussion.
So, for -- both of them work to reduce large
number of variables to a smaller number of
factors.
So, that could be principal components or
exploratory factor analysis.
Doesn't really differentiate.
Reallocating the variation of large number
of variables.
That is much more oriented towards principal
components analysis, okay?
And we'll talk about why.
Creating orthogonal representations of the
original variables.
Well, without the subheading there of solving
problems with multi-collinearity -- actually,
both of them do that depending on the rotation
that you might employ.
But really, principal components analysis,
the basic algorithm for that is really that's
part of the orientation of that algorithm,
okay?
So, it's a little bit more toward principal
components, but it also tends to happen with
exploratory factor analysis, okay?
Where are we?
I -- underlying dimensions and data constructs.
Most of you said exploratory factor analysis.
And that is probably why you would be driven
toward exploratory factor analysis.
Regression with many correlated variables.
That would be principal components analysis,
okay?
Again, regression is a variance based methodology.
You're trying to capture as much of the variations
as possible, and that's really what principal
components does.
And lastly, creating a hypothesis for a confirmatory
factor analysis?
Again, that's much more oriented toward exploratory
factor analysis.
Probably wouldn't use principal components
for that.
Okay?
But these are the questions that trip us up.
These are the situations.
So, let's try and figure out why this would
be the case.
And oh, we should say that factor analysis
in general -- both principal components and
exploratory factor analysis -- it's an exploratory
technique.
It's not a confirmatory or -- there's no significance
testing associated with it.
It is exploratory, the same way descriptive
statistics are.
So, what you get at the end is not that, “Okay,
this is the answer.”
What you get is something that you feel confident
that you can interpret, okay?
So there's no right or wrong here.
But ideally, what you're trying to do is reduce
the dimensionality of your data.
So, you're trying to take large numbers of
variables and reduce it to some smaller set
that you can then work with, okay?
All right.
So, overview.
So, these are common elements.
These are some of the key things that are
common to both principal components and exploratory
factor analysis.
So, you're always going to start off with
some number of variables.
So, these are the observed variables that
we get, whether it's from a survey or however
we generated them, okay?
Now, if there's no correlation between these
variables, there's no reason to do any kind
of factor analysis.
So, there has to be some sort of inherent
correlation.
And typically, each variable here, that you're
using in some sort of factor analysis should
have, at least, a correlation of some -- and
these are rules of thumb.
These aren't -- there's no theory behind this.
But typically, you want, at least, a correlation
of .3 with at least one other variable, okay?
Otherwise, probably that variable is going
to drop out in the analysis, somewhere along
the way, because it's not going to factor
well, okay?
So, we're starting off with some group of
correlated variables.
Ideally, we'd like to have some sort of conceptual
framework.
It is possible to go into a factor analysis
saying, “I really have no idea what variables
should, you know, kind of hang together here,
or what they should represent.”
If you're doing that, there's a good chance
you're going to end up with something that
is not justifiable, not supportable.
When you come out of a factor analysis, you're
going to come out with something that you
feel is interpretable, and you want it to
be something that you feel there's some justification
for besides the data, okay?
So you want to be able to point to some literature.
Point somewhere, say, you know, “This is
what kind of literature says should've happened,”
or “This is what some other study kind of
found, and we just added a little bit more
to it,” et cetera, so you have something
to hang your results on, besides just “I
think this sounds good, so I like this result.”
Okay?
So, you don't always have that.
Obviously, you also want to think through
some other issues that could impact the results.
So, if you have other extraneous variables
such as gender or some other kind of criteria
that might affect your data.
Might affect the factor analysis.
So, factor analysis might be different for
males and females.
Well, if you lump them together, you may not
get a definitive factor analysis.
It's kind of like dealing with a bimodal distribution
and saying, “Let me take the mean.”
Well, the mean doesn't represent either of
the populations that created the two, you
know, lumps in the data.
So, you end up somewhere that -- it's not
really part of either set.
And the same thing can happen with factor
analysis.
So, you do have to kind of think through,
what are some of the other variables that
could impact the results that you're going
to get, and “Do I need to have enough data
to be able to run them separately, so I can
tell whether that's a problem or not?”
Okay?
That's true, again, of either principal components
or exploratory factor analysis.
Sample size.
This is always an interesting topic.
There is, again -- it's not theoretically
determined.
These are rules of thumb, okay?
From people who had done lots of these.
Typically, if you're doing a factor analysis,
you want something like five to 10 observations
per variable, at a minimum.
These are all minimums, not maximums, okay?
Why is that?
Well, you're going to come up with some sort
of loading for each variable, on a factor.
So, the fewer observations you have in your
data set, both overall and per variable, the
more -- the less stable that loading's going
to be.
You'll still going to get a loading.
It will still run.
You'll still get an analysis.
But you have to ask yourself, how reliable
is that result?
So, these are all issues with the reliability
of whatever analysis you're running here.
I usually use somewhat the same four regressions.
So, if you're doing regression analysis, you
might want to think about having -- I usually
the 10 number, not the five.
But I've seen five to 10.
But if you're doing a regression, you know,
typically, you want something like 10 observations
per independent variable.
Now, if you have lots of independent variables
and not a lot of data, you start to run thin
pretty quick.
I've seen limits that say you want to have
at least 50 observations overall, but I've
also seen them that say you shouldn't run
a factor analysis with less than 200.
So, there's not a lot of consensus here.
But again, the range affects the reliability
of the overall model that you're creating.
So, the first number of observations, reliability
of the parameters that you're estimating.
Total number of observations is the reliability
of the overall model that you end up with.
And then, lastly, there's some suggestion
that you want to have somewhere between two
and five variables per factor that you have
in mind.
So, if you have an underlying idea, all of
the conceptual kind of foundation, that you're
looking for somewhere between three and five
factors, then, you might want to be thinking
about, “I want enough variables for each
of those factors, so that I get a stable factor
measurement.”
Okay?
Too few, you may not be capturing the essence,
the true nature of the underlying factor.
Too many, you can always throw some out, okay?
So, these are just -- yeah.
Male Speaker:
You said, for the 200, you heard that you
shouldn't run a factor analysis if you don't
have at least 200?
Sam Woolford:
I've seen suggestions that you want to have
at least 200, okay?
I'm giving you the range.
Again, it's going to depend how many variables
you have, and, you know, how many factors
you're using, et cetera.
There's -- that impacts the number as well.
But it's a little bit like power, if you're
familiar with statistical power.
If you really want to be able to identify
a pattern, if there is one, then you want
to have enough data to be able to get it accurately.
Since we aren't doing any statistical testing,
you don't have the same requirements for sample
size that you were going to do a T test, or
something like that, where you say, “Okay,
I need at least 30 to, you know, assume normality,”
or something along those lines, okay?
So, it's a little bit more variable, okay?
Any questions?
All right.
Oh -- I just went backward I think.
Right.
All right.
So, this is a little bit of a technical issue,
which you probably don't think about a whole
lot, because you just hit the button and run
your factor analysis.
But underlying the factor analysis, there
is the factor analysis uses the data in a
certain way.
And there are some impacts.
You have choices.
So, you can run it on a correlation matrix,
or you can run it on a covariance matrix.
Covariance matrix just means the data is centered.
It's not normalized.
So, it doesn't have standard deviations one.
Correlation matrix means they do, okay?
Now, if you use standardized variables, and
this has more of an impact on -- well, it
has an impact on both.
If you use a correlation matrix, all your
variables have, essentially, the same starting
weight, okay?
Because it's -- they've all been standardized.
If you use covariance matrix, variables that
have more variance associated with them are
going to have more weight in the factor analysis.
So, they're going to orient the results toward
those variables that have higher variation.
And you're going to see different loadings.
You do have this option.
And it is something that we will come back
to, when we look at the differences between
principal components and exploratory factor
analysis because this is where the change
is made to run the algorithms.
So, if you're using SPSS -- well, we'll talking
about it when we get there.
There are ways around it.
Okay.
So, I just want to point this out here that
you do need to be aware that you need to at
least think of when you go to run either principal
components or exploratory factor analysis,
do I want correlation matrix based or covariance
matrix based?
Okay?
And you can always run them both ways, and
see what happens with the results, okay?
Yes?
Female Speaker:
What are some reasons why you might choose
one over the other?
Sam Woolford:
Variables that don't have a lot of variation,
oftentimes, you're not all that interested
in.
And so, you don't want to make them have the
same weight as a variable that does have a
lot of variation.
Because there's -- variation is information,
okay?
Think of it this way.
If I have a variable, and every observation
is the same, doesn't really tell you anything,
right?
Because you only need one observation, and
you know everything.
But a variable that takes on a range of observations
has more variability.
You're going to learn more about, you know,
how does it change?
What's its relationship to other variables?
Et cetera.
So, there's more information in that variable
that there is in one that has a very small
variation.
As soon as you go to a correlation matrix,
you've eliminated that difference in information.
Now, they all have the same.
Female Speaker:
So, why would you do that?
Sam Woolford:
Why would you do it?
Well, in most of the studies that, you know,
I worked on with folks here, you aren't starting
out saying, “I have” you know, “I give
a survey.
Okay.
And I've got it planned out on a one to 10
scale.”
And I'm not expecting that necessarily, and
Barb can attest to this, because I always
tell her which variables don't have a lot
of range to them.
And oftentimes, we kick those out of the analysis.
But typically, you're expecting respondents
to answer across the range, and the range
is limited because you've defined your scale
in a limited way.
And so, you don't have a difference in range
on those kinds of responses.
Where you're more likely to see it would be
if you're taking blood pressure and heart
rate, okay?
Blood pressure has a longer scale, right?
So, it has more variability associated.
So, if you were looking for a factor that
was more oriented toward blood pressure, if
you standardize it with heart rate, then they're
going to be treated the same, and have the
same influence on the factor.
Versus if you deal with their covariance,
then it's more likely that blood pressure's
going to separate, to some extent, from heart
rate.
Does that help?
Female Speaker:
Yeah.
Sam Woolford:
Okay.
All right.
So, another area of consistency across these
is, how do you measure where your results
are reasonable?
Now, I didn't say whether they're significant,
because we can never test that, okay?
But we can at least talk about, did they satisfy
some general characteristics that we would
expect for data that would be useful to factor?
So, there are a couple of measures that are
commonly used.
One is Bartlett's test of sphericity.
So, most of your outputs will come up with
this.
How many people are familiar with this?
Okay.
So, Bartlett's test of sphericity is kind
of a -- it is a significance test that looks
at the correlation matrix and says, “Are
there any off-diagonal elements?”
So, you want to fail this test because it's
basically saying, it's your correlation matrix
and identity matrix.
If it is an identity matrix, right, then there's
no correlation.
So, there's no factor analysis.
So, this is a test of -- the null hypothesis
here is the correlation matrix and identity
matrix?
The alternative is if it's not.
So, you want to fail this, because you want
a matrix that has covariance in it, or correlations
in it.
That's one of the first things we said we
want to have in our variables, right?
So, if you've done this visually -- if you
looked at the correlation matrix, and you've
checked that all the variables have at least
a correlation of .3 or more with at least
one other variable, you should fail this test
without any problem.
So, this is just a double check.
Yes?
Go ahead.
Female Speaker:
If the null hypothesis is that it's an identity
matrix, then you're saying you want to fail
--
Sam Woolford:
You want to reject the null.
Female Speaker:
-- you want to reject it?
Sam Woolford:
Yeah.
Female Speaker:
Yeah.
Sam Woolford:
Right.
Female Speaker:
Well, I was just thrown off --
Sam Woolford:
Yes.
Female Speaker:
-- it's like the [unintelligible]
Sam Woolford:
Okay.
Yeah, no.
You want to reject, okay?
I have to be careful about my language.
All right.
There's some other measure that are also useful.
There's something measure of sampling adequacy.
How many of you are familiar with this one?
Okay, good.
So, there are two places where you want to
look for this.
So, there's an overall measure.
This is also the KMO, which is -- stands for
the -- I think Kendall something Olgram [spelled
phonetically], sometimes called the KMO, which
is three guys' names, and they took the first
initial of each of them.
At any rate, there's an overall measure which
says, as a data set, how well is your data
set oriented toward factoring?
Okay?
Again, rules of thumb.
These are not statistical tests.
Rules of thumb are that if you're at least
.6, it's adequate; ideally, you'd like it
to be higher, okay?
Won't go up above one.
So, it's kind of like correlation matrix measure,
okay?
But there're also measures for individual
variables.
Yes?
Male Speaker:
[unintelligible] what does the number one
[spelled phonetically] actually mean?
Sam Woolford:
Might mean that you've got a couple variables
that are really highly correlated.
I'm not sure.
I've never seen one that's above .9, I don't
think.
Male Speaker:
Just like theoretically, what would that --
Sam Woolford:
Yeah, I don't know.
Male Speaker:
Okay.
Sam Woolford:
Okay.
Each variable has an associated sampling adequacy
measure.
And again, you want to look at individual
variables the same way.
But you're going to run into problems, typically,
if you see any of those having a value of
less than .5.
And what is typically recommended is that
you take out the lowest one -- the one that
has the lowest value, just take that variable
out.
It's not going to work.
You're going to get low commonalities.
You're going to get low loadings.
It's just not going to fit anywhere.
It's going to probably screw up some other
things.
Take it out.
Rerun this part.
Get to this, and see if there are any more
less than .5.
If there are, take the next lowest one out.
So, you work at this one at a time.
You do not take out -- if you see four of
them less than .5, don't take out all four,
because you may be throwing out good data.
So, every time you take one out, all these
measures get recalculated.
And ones that had been lower than .5 may rise
above .5, okay?
So, it's an iterative exercise.
So, these are just measures that you always
want to look at and check to say, “Do I
see any problems", before you ever get into
rotations or extractions, or any of that stuff,
okay?
And it doesn't measure whether it's PCA or
exploratory factor analysis, okay?
So, again, what happens when you do one of
these analyses?
Typically, you want to think about the factors
and which ones you want to keep, right?
That's what we mean by factor extraction.
You have this issue in both.
So, the first is, how many factors do you
keep?
Right?
And there are typically three ways -- well,
let me step back and do one thing before we
talk about that.
There's also an issue of variability and how
you're measuring that.
So, when you think about correlation -- correlation
is shared variance, right?
If you think about a simple regression, one
independent variable.
If you run that on standardized variables,
what's the coefficient that you get?
What does that represent?
Anybody know?
It's the correlation between the independent
variable and the dependent variable.
So, your beta.
Your slope, all right?
If you standardize variables, there's no intercept.
So, all you get is a slope.
That slope is just a correlation between X
and Y, all right?
So, that correlation is -- we called shared
variance, and if you want to know how much
variance is shared, you square it.
And what does that equal?
It's your R squared, right?
It's the variance explained in the dependent
variable that's explained by the independent
variable, right?
Same idea here is that we've got correlation
in these variables.
That correlation represents shared variance.
The loadings that you get out of a factor
analysis represent correlations.
You square those, they represent components
of variance that is shared with the factor.
Same idea as regression.
No different, okay?
But you have to understand, when we get to
differentiating principal component and exploratory
factor analysis that there are different pieces
of variance.
So, there is common variance, and that's the
variance that you're saying is common between
the variable that you're measuring, and the
factor that you're computing.
There's also unique variance.
And that's variance that's specific to the
variable you've measured.
So, if you think about measuring intelligence,
okay?
We don't know how to measure it.
We have lots of variables that we use to get
an observation on it, so we could do an IQ
test.
We could do an SAT test.
We could look at your grade point average.
All those would be measures of intelligence.
And we would typically put those together
in a factor analysis, and hopefully, they
would all be measures of intelligence, okay?
We would come up with some new factor called
intelligence, all right?
That would be the idea.
But each of those variables has variance that
could be what we call common variance that's
shared with intelligence.
And in fact, if you've followed the literature
on some of these tests, if you've gone through
college, et cetera, you probably have seen
some of this discussion, that when you take
an SAT test, it's measuring socioeconomic
issues.
It's measuring all kinds of other things beyond
just intelligence.
So, if you're trying to measure the intelligence
factor construct, there's common variance,
which is common to intelligence.
There's unique variance, that's unique to
the SAT test that doesn't relate to intelligence,
okay?
But it causes the measure to vary.
Causes the SAT score to vary, independent
of how intelligent someone is, okay?
Now, there's a third set of variance, which
is also a measurement variation, which says,
you know, if I give the same person the same
SAT test, you know, and it's raining one day
and sunny the next day, they might get different
scores.
And that's just measurement error.
Oftentimes, you can't really differentiate
between unique variance and measurement error.
So, those are typically lumped together, okay?
But it's important to keep in mind these different
components of variance, because this is one
of the key differences that comes up to differentiate
principal components analysis and exploratory
factor analysis.
It's, what is the variance that you're really
measuring, okay?
And also, you need to keep in mind the objective
of the analysis.
So, principal components and exploratory factor
analysis both are based on measuring some
part of variance.
It's just whether it's the common variance
or all the variance.
And that's really one of the key differences.
Yes?
Female Speaker:
So, you're saying that factor analysis is
-- it will tell you the common variance definitively
[spelled phonetically]?
Sam Woolford:
Well, hold that question.
Once we look at the models behind each of
these techniques, it will become very clear
exactly what they're measuring.
You'll see it, okay?
But I just wanted to set the stage here by
saying, when you think about the variables
that you're measuring, and you're trying to
set up, in the background, some sort of factor
analysis, there is this sub-setting of the
variance within each of those measured variables,
okay?
Okay.
I think we might have missed something here.
Let me just -- no, I guess not.
Okay.
All right.
So, when we get to principal components, principal
components assumes that all the variation
is common, okay?
So, it's assuming that the unique variance,
the measurement variance, and the common variance
is all getting lumped together, and it's calling
all that just common variance.
So, all that is considered variance that it
shares with the factor, okay?
And in that case, and I'm just making this
point, that the diagonal of the correlation
matrix in the analysis that you run in principal
components is taken to be one, because that
is the proper variance in a -- if you standardize
the variables, right?
So, you're putting all the variance -- variance
of one -- into the analysis.
So, just keep that thought floating in the
back, because it's not important that you
understand, theoretically, what's going on,
but just that there is a difference somewhere,
and this is one of those places.
So, the assumption in principal components
is just that we can represent all the variation
in our measured variables through the factors,
okay?
And typically, then, the objective for principal
components analysis is going to be find the
minimum number of factors that capture the
maximum amount of variation.
So, the objective, typically, in a principal
components analysis is variance adjustment,
if you will.
It's moving the variance on the factors, trying
to get the smallest number of factors that
account for the largest amount of variance,
okay?
Now, when we get to exploratory factor analysis
-- yes, go ahead.
Female Speaker:
Sorry, this is from the last slide.
I'm just a little stuck on the assumption
that the factors can represent the variation
and the variables exactly.
What --
Sam Woolford:
Hold that question.
You're my best straight man, okay?
Just -- you're raising great questions.
As soon as I show you the model, it's going
to be really easy for you to see how this
happens.
Female Speaker:
Okay.
Sam Woolford:
Okay.
So, I'm just whetting your appetite.
So, exploratory factor analysis.
If we look at that -- also known as common
factor analysis, because it only looks at
the common variation, okay?
Or principal factor analysis is another term
that's often used.
And this leads to the confusion.
Because when you see principal factor analysis
versus principal components analysis, you're
going to say, “What's the difference?”
And this is why a lot of people don't fully
appreciate that there are two different models
in the background that are going on here.
So with this one, we're assuming that the
factors explain only the common or shared
variance.
So, it's assuming that you have unique variance
that is separately dealt with.
Principal components analysis lumps it all
together.
Says it's all the same.
We don't care to differentiate.
We're going to just move as much of it as
we can onto as small a number of factors as
we can.
Exploratory factor analysis says we understand
that there are different components to the
variance, and we only want to align the common
variance with our factors.
The unique variance and measurement error,
we're going to separate out and deal with
them separately, okay?
But our factors are only going to represent
that common variation.
Think of, now, come back to regression, okay?
Simple linear regression.
What do you have at the end?
I have Y equal beta naught plus beta one X
plus what?
Multiple Speakers:
Error?
Sam Woolford:
Error, right?
Epsilon.
So, what would that be more like?
Principal components?
Or would it be more like exploratory factor
analysis?
Multiple Speakers:
Exploratory.
Sam Woolford:
Right.
Because what's it doing?
It's separating out the unique variance, right?
Because that -- Epsilon represents a unique
variance that's not common to X and Y.
It's measure -- in that case, we call it,
typically, measurement error, but if you remember
your textbook, it also accounted for any X's
that you didn't include in the model, right?
So, in some sense, there's error there that
isn't part of what's in the X, okay?
Same thing.
Exploratory factor analysis is kind of saying
the same thing.
We're treating the factor as your Y variable,
in some sense, okay?
And we're saying well, we have all these measurement
variables that has variance in common with
the factor, but then, there's also some other
variance which, you know, we aren't trying
to say is part of the factor.
Principal components?
No.
It's saying, all the variation is part of
the factor, okay?
Measurement error, variable error, whatever,
okay?
All right.
So, keep that in mind.
Now, difference in exploratory factor analysis.
Remember I said that in the correlation matrix,
we put ones on the diagonal.
If you're going to look at exploratory factor
analysis, we do not.
We put a measure of the communality.
Communality is the common variance.
So, if you have standardized variables, variance
of each measured variable is one, right?
If not all that is being shared with the factor,
then some value less than one is the common
variance, right?
Everybody follow?
So, therefore, the diagonal -- if you put
the communalities there, the diagonal's measures
are going to be all less than one.
So the matrix that you're using to do the
analysis in an exploratory factor analysis
is not a correlation matrix, exactly.
It's actually, once you put those communalities
on there, it's equivalent to using the covariance
matrix, okay?
Pretty tricky.
Again, source of confusion.
Now, why do you need to know this?
Well, if you run SAS, and you want to run
exploratory factor analysis, you need to tell
it to put this on the diagonal.
If you don't, you get a principal components
matrix by default.
You get a principal components analysis by
default.
In SPSS, if you just run it without choosing
an extraction method, you're going to get
principal components analysis.
That is the default.
You don't have to worry about what's on the
diagonal because SPSS is just a little bit
nicer, and it allows you to just choose an
extraction method that takes care of all that
for you.
SAS is just a little uglier, okay?
All right.
But, it's at least something that you have
to think about.
Now, what's the objective intent here?
Typically, with exploratory factor analysis,
your intent is you're trying to look for some
sort of latent constructs.
Now, it's not always easy to define, what's
the difference between a latent construct
and a factor that would be a principal components
factor, okay?
Let me give you an extreme example.
Consumer price index is a principal components
analysis, all right?
Why?
What are they trying to do?
They're looking at price changes -- variation,
across lots of -- a market basket of foods,
right?
And they're combining all that to give you
one index that represents how variability
-- how much variability there is in all those
foods.
So, what do they do?
They took all the variation from all those
foods, pushed it all onto one factor, calls
it an index, becomes a CPI.
Now, what have we noticed recently, if you
follow any economic data?
When they report the CPI, they don't just
report one index anymore.
They kind of report several.
So, there's like a food CPI, and a total CPI,
and a fuel CPI.
Why?
Because it doesn't all come out on one factor
anymore, all right?
So, you have kind of some variables, like
the change in fuel prices, in heating oil,
and stuff like that.
Loads on a different factor, so it's a different
index, right?
Make sense?
So, you would use principal components for
that, because you're trying to push all that
variation in the prices onto as few a number
of factors as possible.
You're not trying to understand the essence
of a shopping basket, okay?
Now, the work that you guys do is more oriented
toward constructs.
You're looking at kind of concepts that can't
directly measured, but that we feel theoretically
are out there.
Intelligence.
Worry.
Hope.
You know?
Great concepts.
How do you measure them?
Who knows?
But you're not looking for an index, typically.
Now, you could be.
And if you are, then you're going to change
your mind and say, “Principal components
should be the way I go, because I'm trying
to create a hope index,” for instance.
For those of you who have used the depression
scale, which I know is pretty common, you
know, there are people that use that as an
index.
You'd analyze it differently.
You should analyze it differently, okay?
Not everybody does.
That's one of the problems.
I mean, you guys shouldn't feel that your,
you know, misconceptions about either of these
is unique to this room.
You can find it in literature, all over the
place, okay?
So, objective here, a little bit different.
Oftentimes, with exploratory factor analysis,
also, you're looking for kind of the input
to a confirmatory factory analysis.
So, ultimately, you want to go down that path,
you're typically not going to be using principal
components to get there, okay?
And in fact, principal components may lead
you down a false trail that may not work out.
Okay.
All right.
Now, last piece of commonality here is that
oftentimes, if you have lots of variables,
you'll get very similar results out of principal
components analysis and exploratory factor
analysis.
There are cases where your data set is set
up -- the data comes in in a way that if you
ran it either way, you'd get -- you know,
pretty much mirrored results.
And you'd say, “Well, why would I do it
one way over the other?”
And that's one of the reasons why people get
confused.
Because I've run a certain data set, and maybe
I didn't know what I was doing.
Maybe I did, I ran it both ways, and I still
got kind of the same interpretation.
That often happens.
But it's because the data set you're using,
not because the methodologies are the same,
okay?
So, you have to keep that in mind.
And confirmatory analysis -- factor analysis,
I just mentioned this now, is not an exploratory
technique.
So, remember, what you've done -- what we're
doing in PCA and EFA is exploratory.
We're trying to get a better understanding
of how the data hangs together.
If we really believe that we've got something
there that means something.
So, if we really think we're coming up with
a measure for intelligence, we need to shift
over to confirmatory factor analysis, and
do a different type of analysis where we can
actually do some statistical tests that say,
“This is a valid measure.”
Okay?
So, I'm just putting that in as a touchstone,
because we're not going to talk much about
it today, but that's kind of the next step,
after you've done either -- typically, an
exploratory factor analysis, okay?
All right.
So, let's look at where they're different.
So, now, we're going to look at the actual
models, okay?
So, let's look at the model for principal
components.
So, the idea behind principal components is
you're trying to find a linear combination
of the X's, the measured variables.
The X's, remember are the ones we said, these
are the ones you get survey measures on, or
whatever.
So, we want to find a set of coefficients.
Oh, I do want to make one comment, just because
I run into this too much.
This is not to you [spelled phonetically],
but others as well.
If you are thinking that you're going to do
a factor analysis at the end of the day, okay,
and you're doing a survey, the more levels
you can include on your survey, the more powerful
your scale is going to be at the end of the
day.
So, if you look at the literature for market
research, which oftentimes, you know, surveys
much of the -- many of the same characteristics
you do.
They've all moved to a 10 point scale.
They don't try and define every point on the
scale.
They may still only use five, you know, measured
points.
But A, it's even scale.
There's an even number.
So, somebody can't just go down the middle
all the time, okay?
They have to kind of register one side or
the other.
But also, you get much more variation.
Remember, we're talking about variation here.
If you only have four items on the scale,
it's really hard for people -- people may
be willing to differentiate their views, but
you haven't given them an opportunity to do
that.
And so, it all gets lumped together.
You've lost variance.
If you think about taking a variable like
temperature and saying, “I'm going to make
temperature into five buckets.”
Well, okay.
I've taken zero to 212.
And I've got five buckets.
So it's what, 30 degrees whatever in each
bucket.
So, I tell you you're in bucket one.
You don't know what the temperature is.
Same thing with Likert scales.
If you use a five point Likert scale, you're
saying, “Well, I could've used a 10, but
I really collapsed them, and so, I really
don't know whether you're a one, two, three,
four, five, six, seven, eight, nine, or 10.
I only know whether you're one through five.
And I don't know which ones got collapsed,
so I've lost a lot of information.”
Now, there's always a trade off with, you
know, beating people up and making them answer
tough scales, but, you know, you should start
thinking about making sure that you have enough
points in your scales.
Female Speaker:
So, that's really at odds with a lot of the
movement in social science --
Sam Woolford:
Yes.
Female Speaker:
-- right now.
Female Speaker:
Yeah.
Female Speaker:
And that's because it's hard for people to
understand a construct, in a way that they
can differentiate those degrees.
So, the argument -- the other argument -- nobody
would argue with the statistical argument,
because we want variance, is that you get
less meaningful data.
Because we can't really discriminate if we're
an eight or a seven, when we're trying to
figure out how hopeful we are, or how much
we trust something.
Something that's a very sort of mushy gut-level,
as much emotion as it is cognition, concept.
It may not be reliable to discriminate those
things.
Sam Woolford:
The -- well, typically, on the -- when you
ask your respondents, you aren't asking them
to say how hopeful are you?
You're asking them a whole series of smaller
questions --
Female Speaker:
Yeah.
Sam Woolford:
-- which is usually for a respondent to make
a judgment on.
So, whether -- you know, all I'm suggesting
is, think about it in whether you go to a
-- it seems to be we're standardizing our
scales at a smaller number.
And the rest of the --
Female Speaker:
Part of it is because of the studies where
people say they couldn't tell --
Sam Woolford:
And that's fine, because that is measurement
variation --
Female Speaker:
Yeah.
Sam Woolford:
-- versus if you lump it all together, you
don't know what's the signal, and what's the
error.
So, it's harder to differentiate.
Female Speaker:
I have a quick question --
Female Speaker:
[unintelligible] I just want to echo what
Barb is saying, because it is true that in
survey methodology, we place a lot of emphasis
on trying to be able to label the points on
the scale, so that -- what's a two to you
might be a three to you might be a four to
another person, if there are no labels.
But really, we're the same on the construct.
It's just that you interpret a two as what
I think of as a three.
And so, I encounter this a lot, a whole lot,
this tension between sort of, the statistician
says, “More points on the scale, more points
on the scale!”
And I'm saying, “But make sure they're meaningful
points, on the scale.”
So, I just really want to put a plug in for,
you have to balance those two things.
Sam Woolford:
No, and I would certainly agree with that.
But I think there's a lot of research, as
well, on consumer research, where it's pretty
much asking people for similar thoughts about
products, et cetera, where the scales are
not any better defined than a five point Likert
scale.
You know, so there are points that don't have
specific definitions associated with them.
And they're getting much better results.
So, you know, I'm just saying, there's research
out there.
Might want to take a look at it, and see what
some of the theory there is, or what some
of the experience there has been, and see
how it might work into what you're doing.
Female Speaker:
I just want to ask, so if we have a scale
that has been, you know, validated in the
literature, using a mid-point and a five point
Likert scale, is it acceptable to stretch
it out to a 10 point, in our own research?
Sam Woolford:
That's -- I mean, that's a good question.
I mean, at a pilot stage, might be something
you might look at.
The other thing is, you can always collapse
it.
You know, if you start out with 10, you can
always collapse back to five.
So, you're not -- it's not like you have to
-- you know, you're --
Female Speaker:
Use it?
Sam Woolford:
Right.
I mean, you can bring it back to the way it
has been used.
I would also suggest that even if you have
a scale that's been verified, unless you're
using it in almost the same situation, you
ought to be running confirmatory analysis
anyway, to make sure that it still holds,
right?
Because otherwise, you know, if you're applying
someone else's scale that they verified in
a very unique situation, it's not -- you can't
necessarily assume that that scale is going
to hold, in an experimental situation it wasn't
necessarily tested in.
So, and we have some experience here in the
room with exactly that happening.
So, yeah.
All right.
So, that was my own plug for -- because typically,
factor analysis is -- assumes that you're
dealing with quantitative variables.
So, a Likert scale, I think the common assumption
now, if you have a Likert scale with five
or less points on the scale, it's considered
discreet.
It's not considered continuous.
And we've been getting pushed back on reviewers.
Did we get a review -- did that come back
on our paper?
I can't remember.
But I know it did on one of ours, where the
reviewer came back and said, “You can't
run this using standard factor analysis.
You need to use a different type of software
that treats -- actually measures the correlations
in a different way.”
It does not assume that you can compute them
the same way you would with continuous data.
So, this is another thing to think about,
is that as you use these smaller scales, if
you're running exploratory factor analysis
through SPS and SAS, reviewers are getting
more savvy.
They're going to reject the paper, and say
you didn't do this right.
So, that's not a statistician's view.
In fact, I tend -- no, actually, we found
some significant differences, too.
If you run a five --
Female Speaker:
[unintelligible] Can I --
Sam Woolford:
Yeah?
Female Speaker:
-- can I ask a question?
And this, the whole polychoric versus sort
of a [unintelligible] correlation --
Sam Woolford:
Yes.
Female Speaker:
-- question?
Sam Woolford:
Yes.
Female Speaker:
Okay.
Sam Woolford:
And you do get different results.
Female Speaker:
Some -- yeah.
And on this situation that Barb and I had.
You can manipulate SAS to create polychoric,
and then run it on it, but you have to specifically
compute them, and then run it.
Sam Woolford:
Yeah, it's ugly.
There's other software that's more effective,
if --
Female Speaker:
Right.
Sam Woolford:
-- you want to do that.
But -- okay.
So, we're getting a little bit off track,
but good discussion.
All right.
So, anyway.
For principal components, notice this is a
set up that has all the X's on the right.
Your factor is on the left, right?
There's no error term involved here, so this
does not look like a regression equation.
And the idea is that you're estimating two
things here.
This is kind of interesting.
Factor analysis -- both exploratory and principal
components, both have a flavor of this.
So, you're -- in a typical regression, you
have your X's and your Ys, right?
Here, you've got equations that you're estimating
the coefficients for, but you only have X's.
So, not only are you estimating the Ys, you're
also estimating the slopes at the same time.
So, this is pretty slick.
So, if you didn't fully understand the slickness
behind factor analysis, this is pretty neat
stuff.
You're essentially running regressions here
where you don't have a dependent variable.
Now, you'll also notice something else in
this model.
This would look just like a regression model,
except for what?
There's no error term.
Because we said, with principal components,
all the error is captured in the X's.
It's all shared.
There's no unique variance, right?
So, there's no error term.
It gets measured exactly, okay?
Now, the way the algorithm for principal components
works is that the reason you can do all this,
you have to put other constraints on the way
that you do these estimations.
If you just went to do this like a regression,
you couldn't solve it.
So, you put some other conditions on.
Those conditions have to do with the variation.
So, the first one is -- the first component
you estimate, you pick the direction, which
means the A's, so that you maximize the amount
of variance in the X's that you share with
that Y one, okay?
So, the first component captures the maximum
amount of variation in the X's that it can
share, okay?
Then you go to the next component or factor,
and you choose the direction for that one
to capture as much of the remaining variability.
So, I've captured a certain amount of the
variability of the X's in my first factor.
So, I have some left.
I pick my next factor to be perpendicular
to the first one, and capture as much of the
remaining variance as possible.
So, again, I don't get it all, but I get as
much as I can.
I choose the A's, so that I do that, while
at the same time, giving me a direction in
space -- kind of the XY coordinate system,
that's perpendicular to the first factor,
okay?
So, this is my orthogonal factors.
Then, I keep going.
And I can keep doing this.
And then I do the third factor, the fourth
factor.
And in fact, I can get exactly P factors.
So, if I start out with P measured variables,
I can generate P factors, and account for
all the variance.
So, if I have five variables that I measure,
I can get five factors.
Those five factors will account for exactly
the same variance as I had in the first five
variables.
Now, the trick is, if I keep all five of those,
I haven't done anything, right?
I've gone from five variables to five variables.
So, ideally, I want to get less.
So, I got to choose which ones do I keep,
okay?
But that's the idea behind the model here.
So, all the variance in the X's is transferred
to the factors.
Now, interesting problem here.
If you started out with independent variables
-- so, if your X's were independent -- what
would happen if you ran the factor analysis?
The first variable, or the first factor, would
be equal to the variable that had the highest
variation.
The next one would be equal to the variable
that had the second highest variation, so
you’d just be equating your factors to the
variable, and this is why you don’t do factor
analysis since you’ve got independent variables,
because you don’t learn anything.
Nothing aggregates.
Okay?
So, all right.
Now, there’s one possible issue you may
run into, and that’s if you have very highly
correlated variables in your dataset, that
may impact the way this runs or the results,
so you also want to be aware of that.
There are different things you can do.
You might drop one, because if they’re very
highly correlated, then they’re probably
measuring the same thing.
Alternatively, you can average them, whatever.
You’re capturing all the common information.
So there are easy ways around that.
Okay?
So once we’ve got -- we run this, we understand
the model, now we have to analyze the factors.
This is where we look at how many factors
do we keep.
Yes?
Female Speaker:
Sorry --
Sam Woolford:
You want me to go back?
Female Speaker:
-- model.
Well actually, two questions.
Sam Woolford:
Yes.
Female Speaker:
The first is just a clarification.
You said you don’t do factor analysis when
your X’s are independent.
Sam Woolford:
That’s right.
Female Speaker:
You don’t do the whole component?
Sam Woolford:
You don’t do any kind of factor analysis.
Female Speaker:
Because --
Sam Woolford:
If they aren’t correlated, there’s not
going to be any shared variance.
Female Speaker:
I see.
Sam Woolford:
So there won’t be any common variance, so
you’ll still get -- you’ll be like running
regressions essentially.
Female Speaker:
Okay.
And then also, so given that the factors -- well,
I would assume that most of the time your
factors in a scale are not going to be orthogonal,
right?
Like if you’re doing a measure of hope,
then your different factors for hope realistically
are going to be correlated, right?
Sam Woolford:
Could be.
Female Speaker:
So then I don’t understand how -- wouldn’t
there then be an order effect if you’re
making them orthogonal, meaning like the first
component that it’s making would have higher
variance given just that it came first as
opposed to the next one?
Sam Woolford:
Well yeah.
Well first of all, if you’re doing something
like hope, you wouldn’t be using principal
components, okay?
So -- I mean this model wouldn’t be the
one we would be applying anything and so this
algorithm wouldn’t be used either.
Female Speaker:
Is it for that reason?
Because --
Sam Woolford:
Well, that’s a very good reason why you
wouldn’t use this.
Because when you’re trying to do hope, your
goal is not to move the variance.
It’s not to shift as much variance as you
can onto a smaller number of factors.
You’d be more oriented toward an exploratory
factor analysis where you’re trying to define
a construct which is only sharing common variance
with some of the X’s.
Now we’re going to have the same issue there,
so I’m not saying that you’re not going
to run into that problem, so let me deal with
your question you asked.
Is all of these algorithms that you’re running
in factor analysis, and there’s a whole
bunch of them, and I’m just giving you the
model and telling you what happens when you
run the base kind of default.
But we get an option to rotate, right?
The rotations can either be orthogonal or
non-orthogonal.
And those rotations, what they do is shift
the variance around, and they shift it around,
they capture the same level of variance and
they shift it around based on how you define
you want it, whether you want it orthogonal,
non-orthogonal, whatever.
So yes, there’s a very good argument why
you wouldn’t want orthogonal factors in
many analyses that you run.
From the point of view of developing a scale,
this is purely exploratory to give you a hypothesis
as to what you might push into a confirmatory
factor analysis.
Because one of the other things you’re getting
out of whether you run exploratory or principal
components, every measured variable has a
weight on every factor.
Even though your interpretation only looks
at the high weights, right?
All the other variables still have weights
on that.
When you want to shift over to a scale, you
want to get rid of all those small ones.
So when you go to a confirmatory factor analysis,
you’re only loading the high ones on the
factor.
So now you’ve got a completely different
problem than what you’re getting out of
exploratory factor analysis.
This is purely descriptive.
You can’t be basing, you know, selling the
farm for a scale based on the results you
get out of exploratory factor analysis.
That’s not what it’s built for.
It just gives you an idea as to which variables
might aggregate on factors that you might
be able to interpret in a way that makes sense
in terms of, you’re ultimately moving to
a confirmatory factor analysis.
So it’s a step in the process.
It’s not a definitive, "Yeah, I need this
result, and it has to be exact and," you know,
"it can’t be orthogonal," whatever.
There’s no way to test what’s right here.
You can run every different rotation you want
here, whether it be orthogonal, non-orthogonal,
with the idea of, am I able to interpret the
factors I get out of this better?
And if you can, then you’re going to say,
“Yeah, I like this one better.”
It doesn’t prove that it’s the right one.
There’s no proof in this analysis, that
whatever you come out with is the state of
nature.
It’s what you can build an argument for.
Okay?
So all these are tools.
They’re not answers.
You still have to create the answer out of
the output that you get.
So it’s not pushing a button and taking
a result.
It’s pushing lots of buttons, looking at
lots of results and trying to say, what makes
sense?
And we’ll look at an example so, you know,
this will become clear maybe in a few minutes.
Okay?
All right.
So, we have to figure out how many of these
factors do we keep?
We said if we run all the factors, you know,
if we have P variables, we get P factors.
We captured all the variance, but we’ve
still got the same number of variables.
Well I’d like a smaller number, please.
So I need to figure which ones to keep.
Three standard ways that people look at this.
Either you look at the variance of the factors
that you get, and you pick any factor that
has a variance greater than one.
Why one?
Anybody know?
Male Speaker:
One is how much variance there is in a single
variable --
Sam Woolford:
Right.
Male Speaker:
-- so it’s less than.
Sam Woolford:
So you want the factor to at least have more
variance than is in any one of the measured
variables that you’re using.
Remember, they’re standardized, so the variance
is one.
So ideally, you’d only -- you know, factors
less with variance less than one means I might
as well just use the variable, the measurement
variable, because I got more variance in that
then I do in the factor.
Okay?
That’s a very common one.
Oftentimes, people look at the percentage
of variance explained by the factors and say,
"Well, I want to make sure that I have at
least some valid percentage of the variance
captured, so I might keep a factor that has
a variance less than one because of that."
That could be a criteria I could use.
There is something called a Scree test which
I don’t pay a lot of attention to.
It’s a plot that you can look at that oftentimes
people use.
I think it’s oftentimes misleading.
Typically, I would argue that the variance
of the factors is probably the one that most
people would look to first.
So the next -- so, first idea, how many factors
do I keep?
Once I decide how many factors I keep, now
I can start thinking about how do I rotate
them?
Female Speaker:
Can I ask a quick question before you go on?
Sam Woolford:
Yep.
Female Speaker:
You mention percent of variance explained
as a possible.
Do you personally have any rule of thumb about
what’s a good enough percentage to force
retaining?
Sam Woolford:
Well this, again, depends a little bit on
the area of application.
So, you know, in social science research,
probably if you’re capturing something like
60 percent, you can make an argument for it.
If you start to capture less than 50 percent,
then you have to somehow work around the argument
that there’s more variance that you’re
not capturing in the factors than you are
capturing, and so the question is, what are
you throwing away?
But I haven’t -- there’s no -- I mean,
it’s all rule of thumb stuff.
So the higher it is, the more confident you
can be that, you know, the variables that
you’re using are measuring something relevant.
In the factors that you’re keeping, if there’s
a lot of variance that you’re throwing away
in the factors that you’re throwing away,
then you have to at least look at them and
say, “Do I not care about what’s represented
by those factors and they aren’t really
relevant for what I’m doing?”
Female Speaker:
So is that looking at total variance across
all the factors or per factor?
Sam Woolford:
Well, I mean the total variance across all
the factors is going to be 100 percent.
So, if you look at most outputs that you get
from principal components analysis, you’ll
see that the factors get defined and are lined
up in terms of the first factor and it will
show you how much variance and what percentage
of the total that is.
So it’s the percentage -- if you have 10
variables, the total variance is 10, because
each variable has variance one.
So if the first factor, you know, explains
three variance units, then that’s roughly
30 percent of the variance.
If the next factor, you know, explains two
variance units, then in those two factors,
you’ve got 50 percent of the variance.
If you feel like that’s not enough, then
you look at the next one.
If the next one has at least one unit of variance
in it, then it still meets the variance of
the factor being greater than one, and now
you’re up to 60 percent.
The next one is going to be less.
So, if you were hoping to get 70 percent of
the variance, you’re going to have to start
dipping into factors that have variances less
than one to start with.
Does that help?
Female Speaker:
That makes sense.
Sam Woolford:
Okay.
All right.
When you get your initial results -- so when
you say, “I only want to look at three factors,”
okay -- oftentimes those factors, you can’t
make any sense out of them, because maybe
every variable loads on the first factor,
and every variable loads on the second factor
and you’ve got all kinds of cross loadings
between factors, and you just say, “Well,
you know, I can’t interpret this.”
So we start rotating.
We rotate because once you’ve said, “I
want three factors,” you’ve set the communality.
So you’ve set here’s the common -- this
is the amount of variance of all these factors,
of all these variables that I’m capturing
in these, let’s say, three factors, which
is less than the total now.
So all these rotations keep the communality
values the same.
In other words, it keeps the amount of variance
explained in each of the variables common
to everything else.
That stays fixed.
But there are more than one solution that
will give you those communalities.
So all the rotations are doing is giving you
different solutions where either the factors
aren’t orthogonal, or they are devised in
a different way other than that algorithm
that I showed you that says pick the first
one, put the max amount of variance on that
one, and make it orthogonal to the next one,
put the maximum what’s left on that one.
So every different rotation is just a variation
on that algorithm, and it’s a variation
which says, all right, maximize the -- so
for instance, I think quartimax is something
like -- well, varimax says maximize the variation
of each variable so it only appears on one
factor, so it kind of maximizes on one.
That’s why people tend to use varimax rotation,
because each variable only loads highly typically
on one factor.
So it makes that factor a little bit easier
to understand and explain.
It’s not the only one.
There are other ones.
There’s quartimax, there’s -- I don’t
know, there’s about six or eight of them.
Some are orthogonal, and some are non-orthogonal.
Male Speaker:
And so just to be clear, with those rotations,
are the individual factors still orthogonal
to one another, or is it possible to make
them --
Sam Woolford:
If you use an orthogonal rotation, they stay
orthogonal.
If you use a non-orthogonal rotation, then
you give up on that.
Okay?
And so you built in -- you know, if you believe
that the factors you’re dealing with should
be correlated, you might be better off with
a non-orthogonal rotation, but you’d be
doing it in an exploratory factor analysis,
not principal components.
Okay?
For most of the work you guys do, okay?
All right.
So if you really want to explore those, most
of the documentation for whatever software
you’re using, somewhere in it has an explanation
of those algorithms and what the intent is,
okay?
Or you can probably Google and find it as
well.
Okay.
So we’re still doing analysis of factors.
The next step of this is really the interpretation.
So, we had to pick how many factors do we
keep in principal components?
We had to say, how do we rotate them?
We rotate them because we want to interpret
them.
Okay?
That’s the whole goal, is you want a set
of interpretable -- a smaller number of interpretable
factors from the dataset that you started
with.
Okay?
So typically, we look at the factor loading.
So now we’ve rotated it, okay?
And so now we look at what variables have
high loadings on which factor, and we tend
to define that factor in terms of those variables.
So that gives us some way of trying to build
a story for each of the factors.
And that’s all it is, is a story.
It’s your story, what you think that factor
represents.
Okay?
Plus or minus .5, some of that is dependent
on how big your sample size is.
Higher sample sizes, you’d want it to be
higher.
Smaller sample sizes, you have maybe .5, something.
You’ll find a range in the literature.
But the loadings you’re looking at -- I’m
sorry, did you have a question?
Female Speaker:
Oh, you might be answering it right now.
Sam Woolford:
Okay, so the loadings that you’re looking
at are just the correlations between the variables
that you’ve measured, your observed variables,
and the factor.
So when you say that the -- you’re saying,
you know, if you’re cutting off at .5 you’re
saying any -- I just need a correlation of
.5 or higher, and I’m going to include that
variable in my interpretation of that factor.
But if it gets lower than that, then I’m
going to say well it’s not really a high
enough correlation.
I’m going to assume that it doesn’t really
have a meaning here.
That’s interpretation.
It’s rules of thumb.
There’s nothing significant, statistically
significant about that number.
There is a suggestion that .7 or greater,
plus or minus, is maybe a better measure to
use, assuming your sample size is high enough,
because being a correlation, if you square
it, it’s shared variance.
So if you square .7, you get .49, which is
roughly .5.
So you’re saying I’m roughly sharing half
the variance of this variable with my factor.
Okay?
So that means that they’re really kind of
-- that measured variable means something
about that factor.
Again, it’s a rule of thumb.
It’s not a hard and fast issue.
So the communalities that you pick up, the
communalities are just the squared correlations,
squared loadings added up, give you the shared
variance between the observed variables and
a given factor and I’ll show you an example
of that.
Okay?
Now one of the things you may decide to do
is that if you have a variable that has low
communality, meaning it’s not sharing a
lot of variance with any of the factors, it
may not be worth keeping it in the analysis.
So again, this might be a point at which you
say, I’m going to take that variable out,
re-run my analysis, see if I get a clearer
picture, because it’s still taking up space.
The same way, if you don’t re-run a regression
with, you know, once you’ve decided the
variable is not significant, right?
If you take it out and re-run the regression,
you get all different estimates for all the
other variables.
Right?
If you leave it in, it’s taking up oxygen
from the other variables, so you may not be
getting a true read of the coefficients.
Okay?
Same deal here.
Yes?
Female Speaker:
So you said before you used varimax correlation
[spelled phonetically], and then it automatically
will be clear which one is like --
Sam Woolford:
I didn’t say it will automatically be clear.
[laughter]
Female Speaker:
But there will be a larger -- you know, there
will be like the largest --
Sam Woolford:
The tendency with varimax is to try and only
load each variable highly on one factor.
Female Speaker:
Okay.
Sam Woolford:
Okay?
Female Speaker:
Okay.
Sam Woolford:
It doesn’t always work.
Female Speaker:
So there might not be any that are lower than
.5?
Sam Woolford:
Yeah.
Well, if there aren’t, then you’ve got
other problems, okay?
[laughter]
Female Speaker:
Because -- okay.
Sam Woolford:
Okay?
Female Speaker:
Okay.
Sam Woolford:
All right.
So now let’s look at the difference in exploratory
factor analysis.
So here, remember we said that the variance
of the variables can be decomposed.
So here, exploratory factor analysis tries
to account for that.
So it looks -- deals with the common variance
separately from the error variance.
Okay?
And so it’s only trying to model really
the communality.
And here’s the model.
Now, notice how different this model looks
from the last model?
First of all, the thing that you might notice,
the observed variant in the principal components
model, factor was on the left, the observed
variable is on the right.
Okay?
This simple thing to point to on this one,
observed variables are on the left and the
factors are on the right.
Okay?
So here we’re saying, we’re measuring
each of the measured variables that we observed.
So the X’s are what we observe, and we’re
saying we can come up with some smaller number
of factors which we don’t know, right?
We haven’t measured them.
But through this analysis, we can get values
for them in a way that they will add up to
all the individual variables that we already
have.
Notice that there an error term on the right.
This looks more like a regression equation,
right?
There’s an epsilon which accounts for all
the unique variance in the X.
Same way in a regression, the epsilon accounted
for the unique variance in the Y, right?
So these -- does everybody see the difference
in these models?
Completely different.
So when you’re using them, you’re getting
something very different, depending on which
one you use.
Principal components versus exploratory factor
analysis.
You get different results because you’re
measuring something different.
Okay?
Now, so there are some -- what you get out
of exploratory factor analysis is you’re
actually reconstructing all the co-variances
between the X’s.
So that’s an actual outcome and that’s
how the coefficients are determined here.
So under these assumptions -- and again, these
should look somewhat like regression-type
assumptions, so the X’s and the epsilons
standardized.
Okay, that just takes out an intercept terms.
We’re assuming that the epsilons have mean,
zero, and uncorrelated with any of the X’s,
right?
Standard assumption and regression.
We’re also assuming that the epsilons are
uncorrelated between X’s.
Standard assumption, right, in regression.
So I mean, we’re not pulling in anything
that you haven’t seen before but it’s
a very different model here.
Okay?
So you’re finding the same set of epsilons,
or same set of zetas, the factors, same set
of factors are working in all these regressions.
So it’s kind of, again, a really interesting
thing to think about to say that when we do
regression, we have X’s and Y’s.
Well here, we only have Y’s, and somehow
we’re estimating the X’s and the coefficients.
So that’s kind of cool.
So all the correlation that you get between
the X’s is mirrored -- is what is driving
the estimation of the parameters here, which
is not what drove the principal components
analysis.
It was all variance driven, not correlation
driven.
Okay?
So the objective here, find a small number
of common factors that explain the correlation
between the original variables.
Not the same as what we had as the objective
for principal components.
Okay?
So, the way we get that is instead of putting
ones on the correlation matrix, we take the
ones out and we put in the communalities,
estimates of the communalities.
And as you go through the algorithm to estimate
these parameters, it keeps replacing the diagonal
in that matrix with better estimates of the
communality as it refines the estimates of
the factors.
Okay?
And in SPSS, this is known as principal axis
factoring.
So when you get to that table, and I’ll
show it to you in a minute, when you pull
down and say, "What’s my extraction method,"
you have to go to the dropdown list and you
have to find principal axis factoring if you
want to run exploratory factor analysis.
If you just run it as is, you’re getting
a principal component analysis.
If you want to do it in Sass [spelled phonetically],
you actually have to put in an option for
a prior.
So you have to put in prior equal squared
correlation -- I forget, SC something or other.
Female Speaker:
Is it SMC, the squared multiple correlation?
Sam Woolford:
Yes, thank you.
Squared multiple correlation.
So you have to put prior equal SMC to get
the exploratory factor analysis results, not
the principal components analysis.
Okay?
So we call it exploratory, which could also
comply to principal components because we
oftentimes don’t really have a hypothetical
model in mind.
We’re trying to see what comes out, and
use that as guidance to create a hypothetical
model.
There are arguments around that, but that’s
not unusual.
Okay?
Recognize the difference between this and
a confirmatory factor analysis, though.
When you run exploratory factor analysis,
every variable has a loading on every factor.
Okay?
When you run confirmatory factor analysis,
you’re eliminating that from the situation.
So you’re saying I only -- I’m hypothesizing
that I can measure each factor with only the
important variables for that factor, and I
don’t have to use the rest.
And then what you test in a confirmatory factor
analysis is whether the statistics bear out
whether that worked or not.
Okay?
All right.
So, for the analysis -- and this is again,
a point of confusion -- once you get to saying,
okay, I ran exploratory factor analysis, the
things you do are exactly the same.
You still have to pick out, how many factors
do I keep?
Okay?
I used the same methods to determine how many
factors I should keep.
I still have rotations that I can use to help
me identify a better definition of my factors
or make it clearer.
Use all the same rotations.
All you’re doing here is trying to come
up with an interpretation that you can make
sense out of.
So you’re looking for, which one can I build
a story around best?
Okay?
Whether hopefully your story has some basis
in theory, but --
Female Speaker:
Sam, could I ask a question on the determining
number of factors thing?
This is actually kind of related to what I
was asking before.
When you’re looking at choosing number of
factors by percent variance explained, but
in this case the variance is not going to
be the total variance.
It’s not going to be sort of thinking about
.6 out of the total of 1.
Am I correct about that?
Sam Woolford:
Correct.
Female Speaker:
And so do we have sort of criteria or rules
of thumb?
Sam Woolford:
Well, you’ll still have a total amount of
communal variance.
And so, when you keep a certain number of
factors, you’ll still be keeping a certain
percentage of that communal variance.
Female Speaker:
Okay.
That’s been my confusion --
Sam Woolford:
Yes.
And so that’s what that is measuring in
this case.
Female Speaker:
So that still makes sense as a criteria, even
though you’re not explaining all variance?
Sam Woolford:
Right.
Okay?
So again, the rotations basically do the same
thing.
You have the same rotations to choose from,
and your interpretation, all done the same
way.
So once you’ve run whichever one you want,
the stuff you do after that is all the same,
whether you’re using principal components
or exploratory factor analysis.
Okay?
No difference there.
Okay.
So let’s look at an example.
I’m just going to run through an example
using SPSS, so you can kind of see how things
differ.
And while it’s still fresh in your minds,
and so I’m going to switch out of this presentation,
and we’re going to go over here to -- so
here’s a dataset that is actually prices
for bread, burgers, hamburger, milk, oranges,
and tomatoes in 23 cities --
Female Speaker:
We can’t see it.
Sam Woolford:
Oh, okay.
What did I have to do?
Female Speaker:
I think it’s on the other model.
Female Speaker:
Display?
Female Speaker:
-- PowerPoint in a slide?
I think it’s how you programmed it for it
to work.
Sam Woolford:
I didn’t.
Male Speaker:
Can you mirror it in the desktop?
Sam Woolford:
Okay, so we’ve got to do that again.
Female Speaker:
Mirror display.
Sam Woolford:
Okay.
I can see it fine.
Female Speaker:
Can you drag it across, like past your screen,
to either side?
Sam Woolford:
I don’t know.
All right.
Female Speaker:
It might be a second screen [spelled phonetically].
Female Speaker:
Yeah, there it is.
Sam Woolford:
Oh.
So what do I do?
Female Speaker:
Drag it.
You can change your display settings.
Sam Woolford:
All right.
Somebody guide me here.
Male Speaker:
Can you just pull on the top and drag it to
the right.
Sam Woolford:
Oh, okay, all right.
So open the SPSS screen.
Okay.
So let’s do that.
So we’ll do this one.
Click and hold and drag to the right.
Male Speaker:
Maybe the left.
Sam Woolford:
Left?
[laughter]
Sam Woolford:
Stand up, sit down, fight, fight, fight.
[laughter]
Female Speaker:
It's there.
Sam Woolford:
Yep.
Female Speaker:
The only reason I know is because I did this
very recently.
Sam Woolford:
Feel free.
Female Speaker:
Okay.
Female Speaker:
This is what they had to help me with yesterday.
Female Speaker:
Okay, it's there.
Sam Woolford:
Wow.
You got that up a lot faster than I would
have.
Yeah, we did this earlier.
Female Speaker:
So we don’t want to extend it anymore.
Sam Woolford:
No, right.
Duplicate.
Female Speaker:
There we go.
Sam Woolford:
Yup.
And then apply.
We had to do that for the -- yeah, let’s
keep changes.
Yes.
Okay.
So does that work?
Okay.
All right.
So here’s the data.
So this is just prices in some measures for
these five items in 23 cities.
Okay?
Now what I’m going to do is run this analysis
as a principal components analysis, then run
it as an exploratory factor analysis.
You can see how they differ.
You probably would never use this in an exploratory
factor analysis, but it’s useful to see
it demonstrates the broad difference.
All right.
So I’m just going to also, since I have
some cheat notes.
So if we go to run this, so in SPSS you’d
go down here to dimension reduction in factor,
and we’d pick these variables.
Okay.
Now -- so I’m going to go through the various
options here.
So we might want correlation -- well, we definitely
want a correlation coefficient, significant
levels.
Here’s our KMO and Bartlett’s test of
sphericity.
We also want the anti-image, and I’m going
to ask for the reproduced because we’ll
look at that in a while.
The anti-image of the correlation matrix is
where you get the individual variable sampling
adequacies, measures of sampling adequacy.
Okay?
So we’ll do that.
Now, let’s look at the extraction.
Notice the method up here.
Default, principal components.
If I want to run exploratory factor analysis,
here it is down here, principal axis factoring.
Okay?
[laughter]
So, I’m going to use the correlation matrix.
We’ll take a look at the Scree plot.
I’m going to -- initially, because we’re
doing principal components, and I want to
make a point, I’m going to use all five
-- there are five variables, so I’m going
to get five factors.
Okay?
So, we’ll continue.
Rotation, I’m not going to do any rotations
right now.
We’ll add that in.
I’m not going to save the scores.
I can do that.
I am going to sort the loadings by size, which
just makes them easier to read.
And so let’s run this.
Okay.
All right.
So here’s our correlation matrix.
What’s the first thing you check?
Female Speaker:
Whether or not our variables are correlated.
Sam Woolford:
Right.
So you’d want to just check across and make
sure that burgers are correlated with at least
one other variable at .3 or higher, same for
the rest.
So oranges is starting to look a little iffy,
right?
Just sneaking over .3 with tomatoes, but the
rest of them seem like we’re probably okay.
All right?
Milk also is maybe a little bit low.
So right off the bat, you should be kind of
identifying I could have some problems coming
up, and those are the variables you’d want
to keep track of, okay?
Here’s your Kaiser-Meyer-Olkin.
Yes.
I knew Olkin was the last one.
[laughter]
The KMO sampling adequacy, okay?
We said it should be at least .6.
It’s .66 so it’s adequate.
Okay?
Not great.
And here’s our test of sphericity, and we’re
going to reject at .05.
So that at least is doing what we had hoped.
All right.
Here’s your anti-image matrix.
So this anti-image correlation, the diagonal
elements here are your measures of sampling
adequacy for each variable.
So we said those should be .6 or greater as
well, right?
So you see, they all are.
If one of these had been below .5 -- well
we said they really need to be above .5, right?
-- If one of those was below .5, rather than
going any further, we should go back and take
it out.
It’s not help -- it’s not going to help
our analysis.
It may screw it up.
Okay?
So you want to do all this.
This is all kind of upfront.
Make sure, you know, you’ve got a decent
set of data to work with.
Okay.
Here’s our communalities.
The initial communality that’s used in the
analysis and how much was extracted in the
factors.
Now, remember I used five factors.
I had five variables.
So the factors are mirroring all the variance,
so that’s why these are all ones.
They should be all ones.
Okay?
And here’s the variance for your first factor.
Okay?
It’s 2.4.
The second one is 1.1.
The third one is .7.
The fourth one is .4.
The last one is .2.
If you all these up, what do you think they
add up to?
Female Speaker:
Five.
Sam Woolford:
Five, yeah.
Okay?
Because that’s what -- when you standardize
these variables, and add up all the variances,
you get five.
Right?
I’ve kept all the factors.
They should capture all the variance.
So if you add these numbers up, they add up
to five.
Okay?
And this shows you the cumulative percentage
represented.
So the first factor counts for almost 50 percent
of the variance in all five variables.
The first two captures roughly 71 percent.
After that, you’ve got to make an argument.
You know, you could argue, “Hey, I’d like
to keep 85 percent, even though one of my
factors isn’t capturing at least as much
of the variance as any one of the variables
did.”
It’s an argument that you can make.
It’s justifiable.
Okay?
And again, if you can -- it if helps the interpretation
at the end, you may decide absolutely I’m
going to keep it.
If it hurts the interpretation, you’re going
to decide absolutely I’m going to throw
it out.
Okay?
Because it’s all based on what you can interpret
in the output.
It’s not what any of these measures are
at this point.
So, this is what the Scree plot looks like.
There’s an argument that says you look -- a
good Scree plot has an elbow, right?
So if you looked at this, elbow is kind of
like right here.
So you take the number of factors up to the,
and including the elbow, and drop the ones
after that.
This is just a plot of the variance of the
factors against the factor number.
Okay?
So this is showing you where the factor variance
starts to level off, and it’s not very big.
These are the kind of bigger ones.
All right?
That’s the logic behind it.
If you’re dealing with a lot of factors,
oftentimes these Scree plots are really useless,
because they’ve got multiple elbows, and
they, you know, they move around and they’re
really hard to interpret.
Okay.
So here’s our loadings.
Okay?
And if you look at, for instance, this gives
you the loadings on factor one.
They call it component one because you’ve
run a principal component analysis, so they’re
trying to help you.
Remember what you’re looking at.
[laughter]
Okay?
It won’t say the same thing when you run
exploratory factor analysis.
So the correlation between burger and component
1 is .89, pretty high.
And if I get that out of there, .78, almost
.8 for tomatoes, almost .8 for bread.
So this looks like bread, tomatoes, and burger.
Okay?
You might say, “Well how would I interpret
that?”
Well if I told you this data was captured
by, let’s say, McDonald’s, okay?
So they may be tracking these food prices,
which have an impact on their business.
So this might make a lot of sense for them,
because this is kind of like the lunchtime
food index.
Right?
Then you’d say, "Well okay, what about oranges
and milk?"
Well oranges you get kind of mediocre loading
on one, and you get a big loading on two and
you don’t get much else loading on factor
two.
Right?
So factor two looks like an oranges factor.
So maybe that’s the breakfast index.
And then milk, well, you know, that covers
a wide range.
That’s breakfast, lunch, whatever, milkshakes,
all that kind of stuff.
So that’s on the third factor.
So it looks like we have three factors.
But if you look at it a little bit closer,
you see that, well this kind of cross-loads,
so I’ve got tomatoes on four and one, and
that’s kind of ugly, because I don’t know
how to deal with that and you might say this
is kind of a cross-load, but in opposite directions.
So it’s not quite as pretty an interpretation
as we might have thought.
Okay?
Now I'll show you one other -- so that’s
our interpretation.
Remember, we didn’t do a rotation.
We kept all the factors.
So this was all we had left was to interpret
the result.
I’ll show you one other piece here.
These are the reproduced correlations.
Yes, question?
Male Speaker:
On a per-variable basis, it’s all the variance
of each variable contained across the five
--
Sam Woolford:
Yes.
Male Speaker:
Okay.
You squared all those correlations?
Sam Woolford:
You’d add up to -- yes.
Okay.
Good question, and thank you for asking it.
I meant to say it.
If you look at the reproduced correlations,
notice that we were using the correlation
matrix.
We’ve kept all five factors, so we’ve
kept all the variances.
And if you look at the residuals here from
the original correlation matrix, there aren’t
any.
This only works because we kept all five factors.
Okay?
So we’ll look at the difference that happens
when we only keep the ones we want.
All right?
So, we’ve looked at this.
What conclusion might we draw here on how
many factors we might want to keep?
What do you think?
How many say two?
Notice how I led you in.
How many want three?
The twos have it.
Okay?
[laughter]
There’s just more to think about.
So let’s look at what happens if we only
keep two, all right?
So that would be equivalent to saying we’re
just using the eigenvalues here.
Okay?
So, I’ll let you -- I can send the data
and you guys can play with this at home if
you want to try the three-factor solution.
So let’s go back to what we ran.
Okay?
And the only thing we’re going to do here
is going to change the extraction.
So we’re going to base it on eigenvalues.
So if the eigenvalue is greater than one,
we’ll keep it.
And, I’m also going to rotate the result,
okay?
And I’ll just use the varimax rotation.
We’ll see what we get.
Okay?
You probably guessed, I’ve looked at this
before, so -- it’s a good lawyer that never
asks a question he doesn’t already know
the answer to.
Okay.
Now, notice up front here, did any of this
change?
Why?
Female Speaker:
This is a correlation.
Sam Woolford:
What does it depend on?
It only depends on the measured variables.
It doesn’t depend at all on what type of
factor analysis you’re doing, or how many
factors you keep or what rotations you run
or anything.
All these things that we looked at up here
apply to all the measured variables, so they
aren’t going to change.
When you change your extractions and all that
kind of stuff, they all kind of stay the same.
So we don’t need to look at that again.
Oh, I’m looking at -- sorry, I’ve got
to come down --
Female Speaker:
I think you forgot to save.
Sam Woolford:
Did I?
Okay.
Thank you.
See how it all looks the same?
[laughter]
So let’s see, let’s make sure.
All right.
So we did this.
So we need to just say --
Female Speaker:
Extraction.
Sam Woolford:
Oh, did -- well I was --
Female Speaker:
When you --
Sam Woolford:
Oh, I didn’t save.
Oh, okay, okay, okay.
All right.
So let’s go back.
All right.
So let’s look at extraction.
Yes.
Thank you.
Continue, and okay.
All right.
So all this stuff stayed the same.
So now look at -- when you look at communalities,
all right, why are these communalities less
than one now?
Because we’re thrown away some factors.
Okay?
So this is all the variance that’s measured
across those variables in those two factors
that we’re keeping.
Okay?
If we look at -- let me move this over.
Can you -- yeah, okay.
So, this is the same picture that we saw before,
right?
This was the same result that we had for the
first two factors when we looked at all five
factors.
But now, because of the rotation, what we
get is we’re shifting variance across the
factors, so we’ve extracted them, so we’ve
kept the same amount of variation in these
first two factors.
But now what the rotation does is it shifts
it to satisfy the algorithm in that rotation.
So now you see more variation is going into
the second factor, and we’re losing some
out of the first factor, in the hope that
we can interpret the factors better.
Okay?
And notice that the cumulative percent, because
it’s based -- the communalities stay the
same.
So the cumulative percent stays the same.
It’s still 70.1, same as we have right here,
in the first two factors, right?
The only thing that’s changing is where
that variation ends up.
Scree plot looks the same.
So the only thing, this is the same component
matrix that we saw before, but it’s just
the first two columns of it.
So this is the same as the five one, but we’re
just throwing away the first three.
Okay?
So all these numbers here, all these loadings
are exactly the same as the ones we saw, when
we ran all five.
Where it changes, are in these two.
So here’s our rotated components.
So now what we’ve got -- notice these are
now different than the ones above.
So we’ve still got bread, burger, and milk,
but we’ve also got tomatoes.
Okay?
Now it might even better represent burgers,
because what do you put on a burger?
You put a tomato on it, right?
Female Speaker:
[affirmative]
Sam Woolford:
And then you have over here oranges very highly
loading on the second factor, but you’ve
still got this cross-loading here on tomatoes,
so it’s not great.
It still could be -- we might try some other
rotations, and see if we can get rid of the
cross-load, okay?
Otherwise we’ve got to build a story for
it.
So I’ve built a story for you.
I’m a McDonald’s statistician, and I just
came up with a way to understand my first
factor and a way to understand my second factor.
Female Speaker:
They put tomatoes in the orange juice.
Sam Woolford:
They could.
They could.
I mean, you’d look at their menu and say,
"Why is this happening," and maybe you’d
come up with something.
But it’s all based on the story.
It’s based on the interpretation.
There’s nothing in the statistics here that
says any one of these rotations is any better
than one of the others.
Okay?
Now -- so if I do this now and I still have
-- if I square these and add them up, I’m
going to get the new variance for the first
factor, and if I square it up here, I get
the new variance for the second factor.
If I square them and add them this way across,
I’m going to get the communality for bread.
If I square them and add them across the factors,
I get the communality for burger, et cetera.
So everything that’s up higher, you can
generate out of this matrix.
Okay?
All right.
So now, let’s say what happens to this solution
if I run it as an exploratory factor analysis?
So let’s come down.
So I’m going to run exactly the same analysis,
but I’m going to change my extraction method.
So instead of doing principal components,
I’m going to use principal axis factoring,
which is my exploratory factor analysis.
Okay?
So let’s see how everything changes for
this same dataset.
And I do want to point out on other output
that I just -- so if I just go back up here
first, I wanted to look at this reproduced
correlations.
Okay?
Remember the one above we reproduced when
we had all the factors.
We could reproduce the correlations exactly,
right?
If I only keep two of the factors, notice
what happens.
Okay?
My residuals start to be sizable for some
of these.
And so, we find out that there’s 70 percent
of these residuals have absolute values greater
than .05, which is an indication that you’re
not capturing the covariance.
When you run principal components, you don’t
care because you’re not trying to capture
the covariance.
I just wanted to show you what happens.
Okay?
Now, it’s going to be important to us when
we run exploratory factor analysis because
we are trying to capture the covariance.
So again, we want these numbers to be smaller
if we run it through exploratory factor analysis.
Okay?
So now we go down.
Here’s the exploratory factor analysis.
Notice again, the top three outputs, so the
KMOs, the Bartlett’s test, the correlations,
the MSAs all stay the same.
They aren’t dependent upon what you’re
running.
They’re just dependent upon the data you’re
using.
So we’re using the same independent -- the
same variables.
Communalities -- now, notice the communality
here.
Initial communalities, that’s -- remember,
the other one started with ones, because it
was using the correlation matrix.
This takes the correlation matrix and replaces
the diagonal with an initial estimate of the
communality.
So it’s not ones now for the initial communality.
But this is the final communality, okay?
And if you notice, these numbers are a lot
smaller than the ones that we saw right up
here.
So, keep these in mind, .7, .8, .5, .8, .7.
All right?
So the lowest one was milk, about .5.
We’re thinking that’s still roughly enough
to keep it.
If we come down here and we ran this as an
exploratory factor analysis, look what happens
to milk.
Milk is so low you’d probably delete it
immediately, because you’re not measuring.
It’s not sharing anything with your result.
You’ve got 1.8.
The rest are, you know, you’ve got a couple
0.6’s and the rest -- you know, this one’s
pretty -- oranges is pretty low, too.
Okay?
Now, why is that happening?
It’s happening because exploratory factor
analysis only looks at the communal variance.
It only looks at the variance in common.
Principle takes the total variance, and shifts
it.
This is only trying to find factors associated
with the common variance.
The common variance is less.
Each of these variables have unique variance
associated with it.
That’s what’s not getting measured here.
That’s not what’s not getting included.
Okay?
So that’s why these numbers are all less
than one.
So if we look down here at the factors, again,
you notice that we kept the first two.
In this case, the second one doesn’t even
have an eigenvalue greater than one.
So we’ve kept one that -- it actually -- this
was the same as what you ran the principal
components, right?
So that’s your initial starting point.
But once you start running the principal axis
factoring algorithm, okay, this is what you
end up with.
The second one doesn’t have an eigenvalue
greater than one, so you might not keep this.
You might only use one in this case, or it
might be telling you that you just haven’t
measured this factor or this second factor
very well.
You need more variables.
Cumulative percentage is down to 50 percent
now.
Notice the rotated amount is still exactly
the same because the rotation doesn’t affect
the communalities.
It just shifts the variance around.
So in the rotated version, we get a little
bit closer to one, because we’ve stolen
some variance out of the first factor.
Now again, if I was doing this, I would run
every rotation I could to see if I could get
a result that might look better.
Okay?
But typically you find the other rotations
oftentimes are even more difficult to try
and interpret so -- but it’s worth a try.
Scree plot stays the same.
If we now go over and look at the loadings,
okay, you notice they’re different as well.
Now it’s called a factor matrix, by the
way, as opposed to the component matrix.
So there are some hints here, to try and make
you realize which one you’ve run.
Okay?
But you see that now we’ve got -- we’ve
still got a bread, burger, and tomato -- now,
well, it’s different.
Before we had -- well yeah, these are the
ones we had before, I think.
We don’t really have anything loading very
highly on the second factor, so the second
factor you could almost argue here is not
measuring anything.
You could make an argument, it would be a
pretty weak one, for oranges, because the
loading is so low.
Okay?
I mean, you’re rounding it.
You get to .5.
But, you know, it might be good for exploratory
research purposes but it’s probably not
going to get you a paper.
Okay?
So you can see the drastic difference in the
two ways that you run this.
Now which one would be more -- oh, and then
if we look at the reduced correlations, remember
what happened when we had only the two factors
in the principal components, and we had 70
percent of the correlations were not reproduced
well.
Right?
Look at it here.
When you run exploratory factor analysis,
none of them, none of the residuals are greater
than 0.05.
So again, different model, different objective.
We’re reproduced all the co-variances here
in exploratory factor analysis.
We didn’t come close in principal components.
Completely different result.
Okay?
And here’s your rotated factor matrix which
is loadings if you rotate it.
And all this does for us is again makes tomatoes
look like it’s almost cross-loading, but
gives us a hint of a factor for fruit and
vegetables.
Well, it’s actually fruit because tomato
is a fruit and not a vegetable.
And then it has, you know, a hamburger factor.
Okay?
So the reason I did this was so that you could
see at the same time because typically you
would never do this, because you have in your
mind, when you start an analysis, which of
these two methods you’re going to use.
But I wanted to give you a picture of how
different the results are.
So if you’re trying to build a scale, you
do not want to be using principal components
analysis, because that’s probably not what’s
going to drive a confirmatory factor analysis.
You really want to understand the common variance
that’s in the latent constructs that you’re
trying to measure.
So that’s why oftentimes, if you use principal
components and you expect the results to look
a certain way, and then you put it in the
confirmatory factor analysis based on that,
and you find out it falls apart, and it’s
not supported, and you think, “Oh, what
the hell did I do?” and that’s the reason,
is you ran the wrong analysis to generate
your hypothesis, so your hypothesis didn’t
hold when you got to the confirmatory factor
analysis.
Okay?
So I’m going to stop here because I know
you guys are all probably fading into oblivion
and -- because I only had a few more slides
on just tying this to confirmatory factor
analysis, but we can -- I’m sure you’ve
heard enough for now.
So any questions that we haven’t hit along
the way?
Anybody?
It’s all clear now, right?
Clear as mud.
Yes.
Male Speaker:
Will we get a copy of the slides?
Sam Woolford:
I can provide a copy of the slides, absolutely.
Female Speaker:
That’d be particularly helpful for Holly
and Theresa on the phone.
Sam Woolford:
Okay, yep.
No, I will email those to you.
You can distribute them.
Okay?
Great.
Thank you guys all for being here.
[applause]
Now if I can get my computer to ever work
the same as it did when I walked in here today.
[laughter]
There are some other issues if you try and
use -- SAS has some ugly output for, I think,
exploratory factor analysis.
You get negative eigenvalues and things that
you have to interpret.
So I think the SPSS output is a lot easier
to kind of get in line with.
But if all you have is SAS, read the documentation.
[laughter]
That’s all I’m going to tell you.
It’s there somewhere.
Male Speaker:
Can I ask a question?
Sam Woolford:
Yes.
Male Speaker:
Explain to me AMOS [spelled phonetically].
Sam Woolford:
Well, AMOS is a component now -- it always
was a component, I think, of SPSS.
When SPSS owned it, you had to buy it separately.
You still buy it separately, but it’s all
through IBM, and they’ve actually embedded
it on the analysis dropdown, the analyzed
dropdown -- yeah -- and you can still use
it independently.
I’m not sure if you need the base SPSS system
to hold the data file.
Typically, AMOS is oriented more toward confirmatory
factor analysis and structural equation modeling
so typically what AMOS is very facile for
is that if you have, let’s say, the output
of an exploratory factor analysis.
So you’ve done -- you’ve collected your
data.
One of the other things we didn’t talk about,
but if you’re going to move to confirmatory
factor analysis, there are constraints on
the number of variables that you need for
each factor so if your initial analysis, or
if your initial data collection, if you said,
"Well I’ve got this factor that I think
I’m just going to measure it with two variables,
and then I want to create a scale," you may
be shooting yourself in the foot, because
you may get to confirmatory factor analysis,
and there’s a condition under identifiability,
where you don’t have enough measured variables,
indicators for all the factors to estimate
all the parameters, because basically when
you get to confirmatory factor analysis, what
you’re doing is equating the covariance
matrix.
So now we’re not doing correlation matrices
anymore.
We’re into covariance matrices, and you’re
linking the model that you hypothesize so
that creates a theoretical set of covariance
matrix, and you’re equating that to the
actual covariance matrix from the data, and
that’s how you estimate all your parameters.
So if you don’t have enough indicators for
each of the factors, you may have created
a situation where it can’t estimate them
all.
So you won’t be able to get a scale.
Assuming that you have all that, it’s easy
to build that model in AMOS, because it has
a gooey interface where you can actually draw
out your model, put in all your indicators,
easily attach the data from your data file
into that to make sure you’ve got your model
specified correctly, and then it goes through
all the different -- now we’re into a statistical
model as opposed to a descriptive model, and
when you get into confirmatory factor analysis,
there are statistical tests.
There are also a lot of rules of thumb, because
normally the statistical test isn’t satisfied.
That’s another discussion.
[laughter]
But you still get a number of tests on parameters
and things like that that you don’t get
in factor analysis.
And what you do, you might take the result
of an exploratory factor analysis.
When you move it to confirmatory, each factor
in exploratory analysis has every individual
variable loading on that factor.
When you move it to confirmatory factor analysis,
you are definitely making a hypothesis that
I only need to keep some smaller group of
those variables on the first factor, and that’s
going to measure the first factor well enough
for me, and I can put a different set on a
second factor and third factor, and I can
let them all be correlated, whatever, and
that will all be supported by the data.
So you’re really doing a test in confirmatory
factor analysis to say, does the data really
support a more definitive model than what
you’re generating in a pure factor analysis.
Okay?
Female Speaker:
Great.
Thank you all very, very much.
Male Speaker:
Thank you.
Female Speaker:
We are going to literally put this up eventually
on Genome News, where there’s educational
prompts and things, so if you want to go back
to it and listen to it, it will be freely
available --
[end of transcript]
