Okay.
This is the lecture on the
singular value decomposition.
But everybody calls it the SVD.
So this is the final and best
factorization of a matrix.
Let me tell you what's coming.
The factors will be, orthogonal
matrix, diagonal matrix,
orthogonal matrix.
So it's things that
we've seen before,
these special good matrices,
orthogonal diagonal.
The new point is that we
need two orthogonal matrices.
A can be any matrix whatsoever.
Any matrix whatsoever has this
singular value decomposition,
so a diagonal one in the middle,
but I need two different --
probably different orthogonal
matrices to be able to do this.
Okay.
And this factorization
has jumped into importance
and is properly, I think,
maybe the bringing together
of everything in this course.
One thing we'll bring together
is the very good family
of matrices that
we just studied,
symmetric, positive, definite.
Do you remember the
stories with those guys?
Because they were symmetric,
their eigenvectors were
orthogonal, so I could produce
an orthogonal matrix --
this is my usual one.
My usual one is the
eigenvectors and eigenvalues In
the symmetric case, the
eigenvectors are orthogonal,
so I've got the good --
my ordinary s has become
an especially good Q.
And positive definite,
my ordinary lambda
has become a positive lambda.
So that's the singular
value decomposition in case
our matrix is symmetric
positive definite --
in that case, I
don't need two --
U and a V -- one orthogonal
matrix will do for both sides.
So this would be
no good in general,
because usually the eigenvector
matrix isn't orthogonal.
So that's not what I'm after.
I'm looking for orthogonal
times diagonal times orthogonal.
And let me show you what that
means and where it comes from.
Okay.
What does it mean?
You remember the picture of
any linear transformation.
This was, like, the
most important figure in
And what I looking for now?
A typical vector
in the row space --
typical vector,
let me call it v1,
gets taken over to some vector
in the column space, say u1.
So u1 is Av1.
Okay.
Now, another vector gets
taken over here somewhere.
What I looking for?
In this SVD, this singular
value decomposition,
what I'm looking for is
an orthogonal basis here
that gets knocked over into an
orthogonal basis over there.
See that's pretty special,
to have an orthogonal basis
in the row space that goes over
into an orthogonal basis --
so this is like a right angle
and this is a right angle --
into an orthogonal basis
in the column space.
So that's our
goal, is to find --
do you see how things
are coming together?
First of all, can I
find an orthogonal basis
for this row space?
Of course.
No big deal to find
an orthogonal basis.
Graham Schmidt tells
me how to do it.
Start with any old basis and
grind through Graham Schmidt,
out comes an orthogonal basis.
But then, if I just take any
old orthogonal basis, then
when I multiply by
A, there's no reason
why it should be
orthogonal over here.
So I'm looking for
this special set
up where A takes these basis
vectors into orthogonal vectors
over there.
Now, you might have
noticed that the null space
I didn't include.
Why don't I stick that in?
You remember our usual figure
had a little null space
and a little null space.
And those are no problems.
Those null spaces
are going to show up
as zeroes on the
diagonal of sigma,
so that doesn't
present any difficulty.
Our difficulty is to find these.
So do you see what
this will mean?
This will mean that A times
these v-s, v1, v2, up to --
what's the dimension
of this row space?
Vr.
Sorry, make that V a
little smaller -- up to vr.
So that's --
Av1 is going to be
the first column,
so here's what I'm achieving.
Oh, I'm not only going
to make these orthogonal,
but why not make
them orthonormal?
Make them unit vectors.
So maybe the unit vector
is here, is the u1,
and this might be
a multiple of it.
So really, what's happening
is Av1 is some multiple of u1,
right?
These guys will be unit
vectors and they'll
go over into multiples
of unit vectors
and the multiple I'm not
going to call lambda anymore.
I'm calling it sigma.
So that's the number --
the stretching number.
And similarly, Av2
is sigma two u2.
This is my goal.
And now I want to express
that goal in matrix language.
That's the usual step.
Think of what you want
and then express it
as a matrix multiplication.
So Av1 is sigma one u1 --
actually, here we go.
Let me pull out these --
u1, u2 to ur and then a
matrix with the sigmas.
Everything now is going
to be in that little part
of the blackboard.
Do you see that this
equation says what I'm
trying to do with my figure.
A times the first basis vector
should be sigma one times
the other basis -- the
other first basis vector.
These are the basis
vectors in the row space,
these are the basis
vectors in the column space
and these are the
multiplying factors.
So Av2 is sigma two times
u2, Avr is sigma r times ur.
And then we've got a whole lot
of zeroes and maybe some zeroes
at the end, but that's
the heart of it.
And now if I express that in --
as matrices, because you
knew that was coming --
that's what I have.
So, this is my goal, to
find an orthogonal basis
in the orthonormal, even
-- basis in the row space
and an orthonormal basis in the
column space so that I've sort
of diagonalized the matrix.
The matrix A is, like,
getting converted
to this diagonal matrix sigma.
And you notice that usually
I have to allow myself
two different bases.
My little comment about
symmetric positive definite
was the one case where
it's A Q equal Q sigma,
where V and U are the same Q.
But mostly, you know, I'm going
to take a matrix like -- oh,
let me take a matrix like
four four minus three three.
Okay.
There's a two by two matrix.
It's invertible,
so it has rank two.
So I'm going to look
for two vectors,
v1 and v2 in the
row space, and U --
so I'm going to look for
v1, v2 in the row space,
which of course is R^2.
And I'm going to look for
u1, u2 in the column space,
which of course is also R^2, and
I'm going to look for numbers
sigma one and sigma two so
that it all comes out right.
So these guys are orthonormal,
these guys are orthonormal
and these are the
scaling factors.
So I'll do that example as
soon as I get the matrix
picture straight.
Okay.
Do you see that this
expresses what I want?
Can I just say two
words about null spaces?
If there's some
null space, then we
want to stick in a basis
for those, for that.
So here comes a basis for the
null space, v(r+1) down to vm.
So if we only had an r
dimensional row space
and the other n-r dimensions
were in the null space -- okay,
we'll take an orthogonal
-- orthonormal basis there.
No problem.
And then we'll just get zeroes.
So, actually, w- those
zeroes will come out
on the diagonal matrix.
So I'll complete that to an
orthonormal basis for the whole
space, R^m.
I complete this to an
orthonormal basis for the whole
space R^n and I complete
that with zeroes.
Null spaces are no problem here.
So really the true problem
is in a matrix like that,
which isn't symmetric, so I
can't use its eigenvectors,
they're not orthogonal --
but somehow I have to get
these orthogonal -- in fact,
orthonormal guys
that make it work.
I have to find these orthonormal
guys, these orthonormal guys
and I want Av1 to be sigma one
u1 and Av2 to be sigma two u2.
Okay.
That's my goal.
Here's the matrices that
are going to get me there.
Now these are
orthogonal matrices.
I can put that -- if I multiply
on both sides by V inverse,
I have A equals U
sigma V inverse,
and of course you know the
other way I can write V inverse.
This is one of those
square orthogonal matrices,
so it's the same as
U sigma V transpose.
Okay.
Here's my problem.
I've got two orthogonal
matrices here.
And I don't want to
find them both at once.
So I want to cook up
some expression that
will make the Us disappear.
I would like to make the
Us disappear and leave me
only with the Vs.
And here's how to do it.
It's the same combination
that keeps showing up
whenever we have a general
rectangular matrix,
then it's A transpose A,
that's the great matrix.
That's the great matrix.
That's the matrix
that's symmetric,
and in fact positive
definite or at least
positive semi-definite.
This is the matrix with nice
properties, so let's see what
will it be?
So if I took the transpose,
then, I would have --
A transpose A will be what?
What do I have?
If I transpose that I have V
sigma transpose U transpose,
that's the A transpose.
Now the A --
and what have I got?
Looks like worse, because it's
got six things now together,
but it's going to collapse
into something good.
What does U transpose
U collapse into?
I, the identity.
So that's the key point.
This is the identity and
we don't have U anymore.
And sigma transpose
times sigma, those
are diagonal matrixes,
so their product is just
going to have sigma
squareds on the diagonal.
So do you see what
we've got here?
This is V times this
easy matrix sigma
one squared sigma two
squared times V transpose.
This is the A transpose A.
This is -- let me copy down --
A transpose A is that.
Us are out of the picture, now.
I'm only having to choose the
Vs, and what are these Vs?
And what are these sigmas?
Do you know what the Vs are?
They're the eigenvectors
that -- see,
this is a perfect
eigenvector, eigenvalue,
Q lambda Q transpose for
the matrix A transpose A.
A itself is nothing special.
But A transpose A
will be special.
It'll be symmetric
positive definite,
so this will be its eigenvectors
and this'll be its eigenvalues.
And the eigenvalues'll be
positive because this thing's
positive definite.
So this is my method.
This tells me what the Vs are.
And how I going to find the Us?
Well, one way would be
to look at A A transpose.
Multiply A by A transpose
in the opposite order.
That will stick the
Vs in the middle,
knock them out, and
leave me with the Us.
So here's the overall
picture, then.
The Vs are the eigenvectors
of A transpose A.
The Us are the
eigenvectors of A A
transpose, which are different.
And the sigmas are
the square roots
of these and the
positive square roots,
so we have positive sigmas.
Let me do it for that example.
This is really what
you should know
and be able to do for the SVD.
Okay.
Let me take that matrix.
So what's my first step?
Compute A transpose A, because
I want its eigenvectors.
Okay.
So I have to compute
A transpose A.
So A transpose is four
four minus three three,
and A is four four
minus three three,
and I do that multiplication
and I get sixteen --
I get twenty five --
I get sixteen minus nine --
is that seven?
And it better come
out symmetric.
And -- oh, okay, and
then it comes out 25.
Okay.
So, I want its eigenvectors
and its eigenvalues.
Its eigenvectors will be
the Vs, its eigenvalues
will be the squares
of the sigmas.
Okay.
What are the eigenvalues and
eigenvectors of this guy?
Have you seen that two by two
example enough to recognize
that the eigenvectors are --
that one one is an eigenvector?
So this here is A transpose A.
I'm looking for
its eigenvectors.
So its eigenvectors, I think,
are one one and one minus one,
because if I
multiply that matrix
by one one, what do I get?
If I multiply that matrix
by one one, I get 32 32,
which is 32 of one one.
So there's the
first eigenvector,
and there's the eigenvalue
for A transpose A.
So I'm going to take its
square root for sigma.
Okay.
What's the eigenvector
that goes --
eigenvalue that
goes with this one?
If I do that multiplication,
what do I get?
I get some multiple of one minus
one, and what is that multiple?
Looks like 18.
Okay.
So those are the two
eigenvectors, but -- oh,
just a moment, I
didn't normalize them.
To make everything
absolutely right,
I ought to normalize
these eigenvectors,
divide by their length,
square root of two.
So all these guys should be true
unit vectors and, of course,
that normalization didn't
change the 32 and the 18.
Okay.
So I'm happy with the Vs.
Here are the Vs.
So now let me put
together the pieces here.
Here's my A.
Here's my A.
Let me write down A again.
If life is right, we should get
U, which I don't yet know --
U I don't yet know,
sigma I do now know.
What's sigma?
So I'm looking for a
U sigma V transpose.
U, the diagonal guy
and V transpose.
Okay.
Let's just see that
come out right.
So what are the sigmas?
They're the square
roots of these things.
So square root of 32
and square root of 18.
Zero zero.
Okay.
What are the Vs?
They're these two.
And I have to transpose --
maybe that just
leaves me with ones --
with one over square root of two
in that row and the other one
is one over square
root of two minus one
over square root of two.
Now finally, I've
got to know the Us.
Well, actually, one way to do --
since I now know all the other
pieces, I could put those
together and figure out what
the Us are.
But let me do it the
A A transpose way.
Okay.
Find the Us now.
u1 and u2.
And what are they?
I look at A A transpose --
so A is supposed to be U
sigma V transpose, and then
when I transpose that I get V
sigma transpose U transpose.
So I'm just doing it
in the opposite order,
A times A transpose, and
what's the good part here?
That in the middle, V transpose
V is going to be the identity.
So this is just U
sigma sigma transpose,
that's some diagonal matrix with
sigma squareds and U transpose.
So what I seeing here?
I'm seeing here, again, a
symmetric positive definite
or at least semi-definite
matrix and I'm
seeing its eigenvectors
and its eigenvalues.
So if I compute A A
transpose, its eigenvectors
will be the things
that go into U.
Okay, so I need to
compute A A transpose.
I guess I'm going
to have to go --
can I move that
up just a little?
Maybe a little more
and do A A transpose.
So what's A?
Four four minus three and three.
And what's A transpose?
Four four minus three and three.
And when I do that
multiplication, what do I get?
Sixteen and sixteen, thirty two.
Uh, that one comes out zero.
Oh, so this is a lucky case
and that one comes out 18.
So this is an accident
that A A transpose
happens to come out diagonal, so
we know easily its eigenvectors
and eigenvalues.
So its eigenvectors -- what's
the first eigenvector for this
A A transpose matrix?
It's just one zero, and when
I do that multiplication,
I get 32 times one zero.
And the other eigenvector
is just zero one
and when I multiply
by that I get 18.
So this is A A transpose.
Multiplying that gives
me the 32 A A transpose.
Multiplying this guy
gives me First of all,
I got 32 and 18 again.
Am I surprised?
You know, it's clearly
not an accident.
The eigenvalues of A A
transpose were exactly the same
as the eigenvalues of --
this one was A transpose A.
That's no surprise at all.
The eigenvalues of A B are the
same as the eigenvalues of B A.
That's a very nice
fact, that eigenvalues
stay the same if I switch
the order of multiplication.
So no surprise to see
32 and What I learned --
first the check that things
were numerically correct,
but now I've learned
these eigenvectors,
and actually they're
about as nice as can be.
They're the best orthogonal
matrix, just the identity.
Okay.
So my claim is that it
ought to all fit together,
that these numbers
should come out right.
The numbers should
come out right
because the matrix
multiplications use
the properties that we want.
Okay.
Shall we just check that?
Here's the identity, so
not doing anything --
square root of 32 is
multiplying that row,
so that square root of 32
divided by square root of two
means square root of
16, four, correct?
And square root of 18 is
divided by square root of two,
so that leaves me square root
of 9, which is three, but --
well, Professor Strang,
you see the problem?
Why is that --
okay.
Why I getting minus
three three here
and here I'm getting
three minus three?
Phooey.
I don't know why.
It shouldn't have
happened, but it did.
Now, okay, you could
say, well, just --
the eigenvector
there could have --
I could have had the minus
sign here for that eigenvector,
but I'm not happy about that.
Hmm.
Okay.
So I realize there's a
little catch here somewhere
and I may not see
it until Wednesday.
Which then gives you a
very important reason
to come back on Wednesday, to
catch that sine difference.
So what did I do illegally?
I think I put the eigenvectors
in that matrix V transpose --
okay, I'm going
to have to think.
Why did that come out with
with the opposite sines?
So you see --
I mean, if I had a minus
there, I would be all right,
but I don't want that.
I want positive entries down
the diagonal of sigma squared.
Okay.
It'll come to me, but, I'm
going to leave this example
to finish.
Okay.
And the beauty of,
these sliding boards
is I can make that go away.
Can I,-- let me not
do it, though, yet.
Let me take a second example.
Let me take a second example
where the matrix is singular.
So rank one.
Okay, so let me take
as an example two,
where my matrix A is going
to be rectangular again --
let me just make it
four three eight six.
Okay.
That's a rank one matrix.
So that has a null space and
only a one dimensional row
space and column space.
So actually, my picture
becomes easy for this matrix,
because what's my row
space for this one?
So this is two by two.
So my pictures are
both two dimensional.
My row space is all multiples
of that vector four three.
So the whole -- the row
space is just a line, right?
That's the row space.
And the null space, of course,
is the perpendicular line.
So the row space for this matrix
is multiples of four three.
Typical row.
Okay.
What's the column space?
The columns are all multiples of
four eight, three six, one two.
The column space, then, goes
in, like, this direction.
So the column space --
when I look at those
columns, the column space --
so it's only one dimensional,
because the rank is one.
It's multiples of four eight.
Okay.
And what's the null
space of A transpose?
It's the perpendicular guy.
So this was the null
space of A and this is
the null space of A transpose.
Okay.
What I want to say here is that
choosing these orthogonal bases
for the row space and the column
space is, like, no problem.
They're only one dimensional.
So what should V be?
V should be -- v1, but
-- yes, v1, rather --
v1 is supposed to
be a unit vector.
There's only one
v1 to choose here,
only one dimension
in the row space.
I just want to make
it a unit vector.
So v1 will be --
it'll be this vector, but made
into a unit vector, so four --
point eight point six.
Four fifths, three fifths.
And what will be u1?
u1 will be the
unit vector there.
So I want to turn four eight
or one two into a unit vector,
so u1 will be --
let's see, if it's one two,
then what multiple of one two
do I want?
That has length
square root of five,
so I have to divide by
square root of five.
Let me complete
the singular value
decomposition for this matrix.
So this matrix, four
three eight six, is --
so I know what u1 --
here's A and I want to get U
the basis in the column space.
And it has to start
with this guy, one
over square root of five two
over square root of five.
Then I want the sigma.
Okay.
What are we expecting
now for sigma?
This is only a rank one matrix.
We're only expecting a sigma
one, which I have to find,
but zeroes here.
Okay.
So what's sigma one?
It should be the --
where did these
sigmas come from?
They came from A
transpose A, so I --
can I do that little
calculation over here?
A transpose A is four three --
four three eight six times four
three eight six.
This had better -- this
is a rank one matrix,
this is going to be -- the
whole thing will have rank one,
that's 16 and 64 is 80, 12
and 48 is 60, 12 and 48 is 60,
9 and 36 is 45.
Okay.
It's a rank one matrix.
Of course.
Every row is a
multiple of four three.
And what's the eigen -- what are
the eigenvalues of that matrix?
So this is the calculation
-- this is like practicing,
now.
What are the eigenvalues
of this rank one matrix?
Well, tell me one eigenvalue,
since the rank is only one,
one eigenvalue is
going to be zero.
And then you know that
the other eigenvalue
is going to be a
hundred and twenty five.
So that's sigma squared,
right, in A transpose A.
So this will be the square root
of a hundred and twenty five.
And then finally,
the V transpose --
the Vs will be --
there's v1, and what's v2?
What's v2 in the --
how do I make this into
an orthonormal basis?
Well, v2 is, in the
null space direction.
It's perpendicular to that,
so point six and minus point
eight.
So those are the
Vs that go in here.
Point eight, point six and
point six minus point eight.
Okay.
And I guess I better
finish this guy.
So this guy, all I want is to
complete the orthonormal basis
-- it'll be coming from there.
It'll be a two over square
root of five and a minus one
over square root of five.
Let me take square root
of five out of that matrix
to make it look better.
So one over square root of five
times one two two minus one.
Okay.
So there I have -- including
the square root of five --
that's an orthogonal matrix,
that's an orthogonal matrix,
that's a diagonal matrix
and its rank is only one.
And now if I do
that multiplication,
I pray that it comes out right.
The square root of
five will cancel
into that square
root of one twenty
five and leave me with the
square root of 25, which
is five, and five will
multiply these numbers
and I'll get whole numbers
and out will come A.
Okay.
That's like a second example
showing how the null space guy
-- so this -- this vector
and this one were multiplied
by this zero.
So they were easy to deal with.
Tthe key ones are the ones in
the column space and the row
space.
Do you see how I'm getting
columns here, diagonal here,
rows here, coming
together to produce A.
Okay, that's the singular
value decomposition.
So, let me think what I want
to add to complete this topic.
So that's two examples.
And now let's think
what we're really doing.
We're choosing the right
basis for the four subspaces
of linear algebra.
Let me write this down.
So v1 up to vr is an orthonormal
basis for the row space.
u1 up to ur is an orthonormal
basis for the column space.
And then I just finish
those out by v(r+1),
the rest up to vn is an
orthonormal basis for the null
space.
And finally, u(r+1) up to is an
orthonormal basis for the null
space of A transpose.
Do you see that we finally
got the bases right?
They're right because they're
orthonormal, and also --
again, Graham Schmidt would
have done this in chapter four.
Here we needed eigenvalues,
because these bases
make the matrix diagonal.
A times V I is a
multiple of U I.
So I'll put "and" --
the matrix has
been made diagonal.
When we choose these bases,
there's no coupling between Vs
and no coupling between Us.
Each A -- A times each
V is in the direction
of the corresponding U.
So it's exactly the
right basis for the four
fundamental subspaces.
And of course, their
dimensions are what we know.
The dimension of
the row space is
the rank r, and so is the
dimension of the column space.
The dimension of
the null space is
n-r, that's how many
vectors we need,
and m-r basis vectors for the
left null space, the null space
of A transpose.
Okay.
I'm going to stop there.
I could develop
further from the SVD,
but we'll see it again
in the very last lectures
of the course.
So there's the SVD.
Thanks.
