And before that we will have to see something
known as Eigenvalue decomposition. So, the
answer to that was actually which I was hoping
all of you will give because all of you have
done two prerequisite, which is linear algebra
and machine learning, both of them teach you
principal component analysis. So, I was hoping
that you will give that answer.
Now, can you give that answer he already of
course, gave that answer, is that make sense
ok? So, we relate it to that so, but before
going to principle component analysis, we
look at Eigen value decomposition, how many
of you have seen Eigenvalue decomposition
before ok? Quite a few.
This is very straightforward. So, let u1 to
un be the Eigenvectors of a matrix A and let
lambda 1 to lambda n be the corresponding
Eigenvalues ok.
Now, I am going to construct a matrix U, such
that the columns of U are these vectors u1
to un, is that fine, what u looks like. And
now I am going to do this product I am taking
a the product of the matrix A with the product
of with the matrix U, where U is this right.
It is the all the Eigen vectors tagged one
after the other is this fine. The next step
I am just pushing the matrix inside. If you
know the 4 different ways of multiplying a
matrix you will know that this is correct
ok. Or else for now just thing that you can
just push the matrix inside ok.
Now, what is this I can replace them by the
lambda 1 u 1 lambda 2, because a u 1 is equal
to lambda 1 u 1 by definition ok. Now can
you write this again as a product of 2 matrices,
one is of course, the matrix U and the other
is.
Diagonal.
Diagonal. So, the diagonal matrix will come
first or the matrix U will come first? How
many if you say U will come first? How many
if you say the diagonal matrix will come first?
The sum is never one ok.
So, it is going to be like this ok. And you
can write this as U lambda. So, U is again
the vector the matrix containing the Eigenvectors
of A and lambda is a diagonal matrix where
every diagonally element is a corresponding
Eigen value.
Now, this is what we have so far A into U
is equal to U into lambda. Now suppose U inverse
exists, I will assume that U inverse exists
and later on I will tell you under what conditions
it exists, then I could write it as this any
of these 2 forms in one case. I am post multiplying
by U inverse in the other case I am pre multiplying
with U inverse ok.
So, this is known as the Eigenvalue decomposition
of a matrix. And the other way of writing
it is known as diagonalization of the matrix
right. You take a matrix apply some operations
to it so that, the result is a diagonal matrix
is this clear to all of you is very straight
forward ok. And again Eigen vectors play an
important role in this. Now the important
question is under what conditions would U
inverse exist. U inverse would exist if the
columns of the matrix U are.
Linearly independent.
Linearly independent, ok. Do we know the columns
of the matrix are linearly independent?
Yes.
Yes, because it is a.
.
Set of Eigenvectors and we already saw the
proof that the Eigen vectors are linearly
independent ok. This just follows whatever
I say ok. Now do we need proof for this I
slide 19 we did this ok, I did not realize
it fine.
Now if A is symmetric the situation is always
more convenient, why is it?
.
What would U be?
Orthogonal matrix.
What is an orthogonal matrix actually?
.
So the Eigenvectors are orthogonal. So, we
have this situation right. Suppose I want
to do U transpose U ok. This is how that operation
would look like ok. Now what is the ij'th
entry of the resultant matrix ?
Dot product.
It is the dot product between the.
ui and uj.
ui and uj. Everyone gets this right, the ijth
entry of this product is going to be the dot
product between ui and uj. This dot product
would be dash if i is not equal to 0 or j.
j.
J and there is no point in this. So, each
cell of the matrix Q ij is given by the dot
product and it is going to be 0 if i not equal
to j, and it is going to be 1 if i is equal
to j ok. So, U transpose U is equal to the
identity matrix; that means, U transpose is
the dash of U.
.
Transpose of U and of course, inverse also
ok. So, U transpose is the inverse of U. And
it is very convenient to calculate what is
the complexity of inverse? So now, you appreciate
that that is a that has high complexity and
in this case if the vector if the matrix is
orthogonal; that means, it is a collection
of orthogonal vectors and the inverse just
comes for free right ok?
So now given this situation, and do not read
the hint as if this is going to help, but
yeah, what can you now say about the sequence?
The same sequence that you saw earlier. So,
I have given you that the EVD of a is equal
to U sigma U transpose, where u is the collection
of the Eigenvectors and sigma is the Eigen
values the diagonal matrix containing the
Eigenvalues.
Now, what given this and ignoring the knowledge
of the first section of this lecture can you
tell me something about this series? What
would be the nth element of the series?
U sigma power n.
U sigma.
Power n.
Power n.
U transpose.
U transpose and you arrive at the same conclusion
right? Where I was talking about this operation
right. So, if we can say something about this
matrix then we can say something about this
series what can you say about this matrix,
if the largest Eigenvalue is greater than
1 as you keep raising it is power that value
is going to explode. And hence, the entire
product is going to explode less than 1. That
product is going to vanish and everything
else would be less than that right remember
is the dominant Eigen value ok.
So, everything would be less than that. So,
that product will vanish ok. So, the same
conclusions you can arrive at right. So, that
is why I want to do these sections again.
So, you would have done these in linear algebra,
but you would have not arrived at these conclusions
from a very different interpretation, but
I want to focus on the interpretations that
I care about. I do not, how many of you have
seen this series in the course on linear algebra?
You have ok, but I do not see why anyone else
would teach this is not required is only required
for some things that I want to do in the course
right, that is why; I wanted to do this section.
So, everyone is comfortable with Eigenvalue
decomposition it is a very simple stuff right
I mean there is no proof or anything involved
there we just use some properties of Eigenvectors
and Eigenvalues and do it ok.
Now, there is one more important property
of Eigenvectors, which well use today. So,
let us see what this means right. You have
a matrix A which is an n cross n matrix ok.
And your import interested in computing this
value, x transpose A x where x belongs to
Rn x belongs to Rn ok.
So, what am I trying to do here of all these
vectors possible in Rn, I want that vector
which maximizes this quantity. What is this
quantity scalar, vector, matrix, tensor?
Scalar.
Scalar ok, such that x is equal to1. This
is the problem that I have been given to solve
why it is not clear as of now, but suppose
this is a problem I am trying to solve, or
the inverse of this which is minimize the
same thing, of all the vectors in Rn find
the vector which minimizes this quantity,
subject to these constraints. Then the solution
for this is given by the smallest or largest
the solution is the smallest Eigen value of
A.
And x is the Eigenvector corresponding to
that. So, if you are trying to minimize and
the solution is a smallest Eigenvalue, we
need to clarify that if you are trying to
maximize and the solution is the largest Eigenvalue
is that clear and the value of x would be
the corresponding Eigen value. So, largest
Eigen vector is the same as something that
we have defined today dominant Eigen vector
right?
So, let me just repeat. So, that there is
no confusion. Let us focus on this problem.
The solution to this problem that is the x
which will give me the maximum which will
maximize this is the dominant Eigen vector
of the matrix A right, is that clear? Fine
ok. And if you want to minimize it is going
to be the smallest Eigen vector; that means,
the inverse of the dominant ok.
So, there is a proof for that I will not go
over the proof you can take a look at it at
your own leisure.
So, what has been the story so far. The story
has been that the Eigenvectors corresponding
to different Eigen values are linearly independent
ok.
If you are dealing with the square symmetric
matrix, which is something that we will deal
with soon. Then things are even more convenient
because the Eigen vectors are actually orthogonal
ok. And they form a very convenient basis,
and now we are going to put this to use when
we talk about principal component.
