We ended last week with the discussion
of the Power Method, the Inverse Power
Method, and the Rayleigh Quotient
Iteration. Now those methods allowed you
to find one eigenvector and one
associated eigenvalue.  In the case of the
Power Method,
it was the eigenvector associated with
the largest eigenvalue in magnitude.  And
in the case of the Inverse Iteration and
the Rayleigh Quotient Iteration, it was
an eigenvector associated with the
smallest eigenvalue in magnitude, or in
the case of the Rayleigh Quotient
Iteration that might actually be a
different eigenvalue all together.  Anyway,
what we are going to do this week is say,
"Well, yeah, but you may want to find
multiple eigenvectors and multiple eigenvalues."  And we may even want to find all
eigenvectors and all eigenvalues.  Are
there ways of leveraging these Power
Method related iterations and leverage
those to find all eigenvalues and
eigenvectors?  And the way we are going to go about that is we're going to sort of
say, "Well, if we want to do this, then
maybe we should modify the power method
in the following way."  And we will
slowly step you towards an algorithm
that's known as the QR algorithm.  And
then in the second half of this week, we
will discuss how to make the QR
algorithm into a practical algorithm.
Okay? Now for simplicity sake, we are
going to focus on the case where the
matrix is a Hermitian matrix.  And if you
don't like complex numbers, think of
it as a symmetric matrix.  And the reason
for that is that for the Hermitian
matrices the eigenvectors associated
with different eigenvalues we know are
mutually orthonormal, or orthogonal at least. And therefore certain parts of the
theory just fall into place a little bit more neatly.   These
methods can be extended to find the
eigenvalues and eigenvectors of non-
Hermitian matrices, of arbitrary square
matrices.  The theory for that is a little
bit more complicated, and those who feel
so inclined can always investigate those
on their own.  So what do we have here?
Well here we have our generic Power
Method.  And what does the Power Method do?  Well you start with some random vector,
you normalize it to have length one, and
then you start iterating where you hit
that vector with the matrix A.  And then
you go and you normalize that vector
back to having length one.  And you just
keep doing that.  And how did we see that
that then allow us to find the
eigenvector associated with the largest
eigenvalue in magnitude?  Well if you
took your original vector v_0 and you
wrote it as a linear combination of the
eigenvectors of matrix A.  And notice
that that vector actually has length 1
but, you know, that's not going to play
too much for the role here in our
discussion.  Then what do we know?  Then we
saw that if we have done k iterations of
this algorithm, the vector that we're
working with is roughly a linear
combination of these eigenvectors,
except that each of the individual eigenvectors has been hit by the
corresponding eigenvalue raised to the kth power.  Now how does normalization that
comes into play, makes things a little
bit more messy, but that, you know, let's
not worry about that part.  Okay?  And then what we could say is,  "Oh we can factor
out the lambda_0 to the kth power.
And then this term here becomes one or
this number right here.  And then each of
these become the eigenvalue divided by
the eigenvalue largest in magnitude.  And
then the idea was that since this ratio
in magnitude is less than one, raising it
to the kth power will eventually wipe all
of this out.  And therefore we end up with
a vector pointing in the right, in the
direction of the eigenvector
associated with the largest eigenvalue.
Now importantly, since we are restricting
ourselves to Hermitian matrices, we know
that the eigenvectors are orthogonal,
mutually orthogonal, and therefore this
linear combination of all of the other
eigenvectors that we're not converging
to, is inherently orthogonal to the
eigenvector to which we are converging, or the direction of which we're converging to.
Okay? And that's going to play an
important role in our discussion.
