we have been looking at conversions of iterative
methods for solving linear algebraic equations
and we have made some progress we have found
out the difference equation that we need to
analyze 
for solving the convergence problem for solving
the convergence problem we have looked at
so i will just take a brief review of what
we have done so we have this matrix a which
we write it as s minus t we split this matrix
well we wrote this in two ways one was l plus
d plus u and then using this l d and u we
formulated two matrices s and t and then we
showed that the iteration scheme essentially
any iteration scheme jacobi iteration scheme
gauss seidel iteration scheme relaxation iteration
scheme
it can be written as x k plus 1 equal to s
inverse t 
x k plus x s inverse b and finally we wanted
to converge to the solution let us say x star
equal to s inverse t x star the solution is
s star equal to s inverse t x star this iteration
scheme should converge to this fixed point
this is called fixed point of the equation
because it when you substitute it on the right
hand side you will get back the same vector
so x star is called a fixed point so we subtracted
this and then we got this equation e error
at point k plus 1 is s inverse t error at
k we got this equation where error was defined
as
error at iteration k was defined as x k minus
x star this is the distance from the true
solution well we do not know the true solution
we are going to reach it nevertheless the
error between the guess solution and the true
solution is governed by this difference equation
error at iteration k plus 1 so this tells
you how iterations progress you start with
some error and so we come up with the analysis
that ek is nothing
but s inverse t raise to k e at 0 error at
0 initial error and what we logically deduced
was if s inverse t raise to k this goes to
null matrix as k tends to infinity then if
s inverse t has a property you say it is a
nice matrix that when you multiply s inverse
t with itself multiple times the product travels
towards a null matrix it converges to a null
matrix and as k goes to infinity you get null
matrix
if that happens then error will go to zero
error going to zero which means the guess
converging to the true solution error going
to zero is a guess converging to the true
solution so now we need to analyze so seems
to be at the heart of this equation at the
heart of this equation and i am going to analyze
equations of this type i am going to analyze
equations of this type so i am going to abstract
it i am not going to exactly keep working
with s inverse t will apply results specific
to this particular problem i am going to look
at a generic problem
say z k plus 1 equal to b zk this is linear
difference equation z belongs to r n and initial
value is 
this is the initial condition so this is my
abstract problem for this i am going to generally
in today's class i am going to analyze behavior
of these kind of equations this equation if
you notice this equation is a special example
of this kind of equations i want to analyze
so what do i mean by i want to analyze i want
to come up with a judgment about the behaviour
of this solution
i want to come up with a judgment about how
solution behaves asymptotically as k tends
to infinity without requiring to compute i
do not want to actually compute the solution
if i want to compute the solution and then
later realize that it is diverging converging
i am not interested in that i want to analyze
without having to do explicit computations
well you might say this equation you know
i just i have done there if you start solving
if you know z 0 you can say that z at any
point k is nothing but b to power k z o
it is very easy to show that not a problem
just apply it multiple times this is the solution
computational solution if you go b matrix
if you know z 0 and this is how it will solve
in this particular case we do not know what
is z 0 right here if you map to this problem
i do not know what is the initial error if
i knew initial error i knew the solution then
the problem of doing iterations would not
arise so i want to analyze this without knowing
z 0 i just want to relate this to properties
of matrix b
if matrix b has certain nice properties then
i can claim that asymptotically this difference
equation behaves to the particular way that
is what i want to reach so for the time being
let us forget about this iterative methods
let us concentrate on this equation and let
us start insides into what happens how do
i analyze this this is a very very fundamental
equation linear difference equation it arise
in many many forms in computations it arises
in solving real problems engineering problems
and t is good to get inside into how this
behaves let us first look at scalar equation
and let see whether the inside that we get
from the scalar case how can we translate
that inside into the vector matrix case so
my first analysis is z k plus 1 equal to beta
zk where z belongs to r beta is a real number
and this is my difference equation i am staring
from my initial condition corresponds to some
z 0 well you can very easily show that zk
equal to at 
iteration k this will be beta to power k z
it is very very easy to show repeated use
of this equation
you can get this is it not i want to talk
about how zk behaves as k goes to infinity
will it depend upon beta or will it depend
upon z 0 it will depend only on beta will
it depend upon z 0 why i need support for
that why should it depend only on so suppose
z 0 is very large will it matters what really
matters what is changing with k z0 is not
changing with k what changing with k is beta
to power k
well if i say that mod beta is strictly less
than 1 suppose you have this additional information
then what will happen is if so 1 k is mod
beta is strictly less than 1 in that case
beta to power k goes to 0 as k goes to infinity
so what happens to mod zk tends to 0 as k
goes to infinity what is important is for
this is very very important for any z0 z0
is 01 z0 is 1000 z0 is million z0 is billion
whatever this has to happen why because mod
beta is less than 1 every time beta k shrinks
and then it goes to zero for any z0 so i can
look at this equation look at just beta and
talk about how asymptotic behaviour is going
to be
my second case 
is beta equal to 1 or mod beta equal to 1
so beta could be minus 1 beta could be plus
1 “professor minus student conversation
starts” what will happen in this case so
at any point in time mod zk will be equal
to mod z0 so zk will neither grow nor it will
decay and my third case is 
mod beta is greater than 1 what happens here
it will go to infinity whatever is z0 that
is another important thing that even if i
start again this is very very important
for any z0 even if i start with z0 to be 10
to the power minus 10 after sometime this
equation will actually diverge zk will go
to infinity even if you start with z0 which
is a small number your equation is not going
to converge to a small number just because
you are starting from small number same thing
is here even if you start from a very large
number if mod beta is less than 1 zk is going
to 0 after sometime
if mod beta is greater than 1 even if it is
slightly greater than 1 even if it is 10001
it might take longer time to go to infinity
but it will go to infinity as k goes to infinity
beta to the power k will go to infinity when
mod beta is greater than 1 and the solution
will what we say is that solution will diverge
so this qualitative behaviour of the solution
for this particular case i did not actually
i need not solve the equation for a particular
z0
i can just look at beta and say that well
if beta is like this this is what is expected
to happen so that would be my basis now how
do i extend this analysis to a matrix case
the tough part is that matrix b is a full
matrix and then we have a trouble there we
cannot do straight forward extensions because
what is matrix being less than 1 well we do
not know what is matrix being greater than
1 we do not have such notions
matrix is a array of numbers and then we have
to work out some tricks to come to use this
kind of analysis so what i am going to show
next is that in the attempt to solve this
problem to analyze this problem in the matrix
case what pops out surprisingly or not so
surprisingly is the eigenvalue problem in
fact you can see what is the origin of eigenvalue
problem when you start solving this linear
difference equations
so probably when it was taught to for the
first time you wondered why you know some
matrix into vector equal to scalar into vector
why did somebody think of this what is the
logical way of arriving at such an equation
where do you hit upon such equation see for
example we talked about positive definite
matrices so when you have first introduced
positive definite matrix you do not know i
mean why somebody call about positive definite
matrix
but when you understand the sufficient condition
for optimality you see that these are naturally
there is a need to define a matrix which are
the special property which is x transpose
a x is always greater than 0 then you know
the local point with the hessian is positive
definite then you will get a minimum and so
on so the same thing is here you will see
that what will pop out in the attempt to solve
this problem you will get eigenvalue problem
so now i am going to move to the vector case
and i want to keep these ideas these ideas
are very nice i want to use them to analyze
my vector case
let us start looking at a vector case here
the solution in the scalar case was beta to
power k into z0 some scalar let me take a
motivation from my scalar case and i propose
a solution for the vector case so my problem
now is z k plus 1 equal to bzk and i am proposing
the solution z at kth iteration is some lambda
some scalar to power k i am just taking motivation
from the scalar case i do not know what lambda
is right now some scalar * some constant vector
if you look at the scalar solution it was
a scalar raise to k into some number that
was z0 it turned out to be z0 well we still
have to worry about how z0 here will come
into picture but this is my guess solution
if this solution has to satisfy the linear
difference equation if this solution is to
satisfy linear difference equation i will
just substitute i will get lambda to power
k -1 into v
V is vector n into 1 vector lambda is scalar
and this is my solution at any iteration k
so lambda to power k minus 1 should be the
left hand side equal to b into lambda you
agree with me see look at this if this has
to be a solution to this problem then it should
obey the difference equation that is the first
criteria it should obey the difference equation
this equation i am just going to rearrange
and write
so i am writing this as lambda to power k
lambda v minus bv equal to 0 lambda is a scalar
lambda to power k into lambda will be lambda
k plus 1 the first term is covered b into
v lambda is a scalar so i can take it on this
side not an issue so lambda to the power k
i am taking it outside and i am not interested
in the trivial solution of lambda equal to
0 because then i will get 0 equal to 0 i am
not interested in the trivial solution obviously
so we rule out lambda equal to 0 next in general
lambda is not equal to 0 well there will be
some situation when lambda will be equal to
0 but we will talk about it little later
right now we are interested in the nontrivial
solution in general a general nontrivial solution
will be obtained when this equation holds
a nontrivial solution to the problem i have
just read it in this equation lambda cannot
be equal to 0 it will give you a trivial solution
i am interested in nontrivial solution of
this particular problem so this is lambda
i minus b into v equal to 0 have you seen
equations of this type where well not eigenvalue
problem i want to relate in some other context
matrix into vector equal to 0 when does this
happen a equal to 0 ax equal to 0 so when
does this ax equal to 0 has a nontrivial solution
so you have seen equations of this type ax
equal to 0 when you get nontrivial solution
when do you get a solution x is not equal
to 0 you get nontrivial solution for this
when a has nonzero null space or what can
you say about a rank of this a it should be
full rank if it is full rank then only solution
you will get will be zero right so it should
not be full rank a should not be full rank
what should happen when you get nonzero solution
columns of a should be independent is everyone
is getting this everyone with me on this if
columns of a are linearly dependent what will
happen you will get a nontrivial solution
x if columns of a are linearly dependent then
only you will get a nontrivial solution x
nontrivial means nonzero solution you will
get a nonzero solution for x equal to 0 only
when columns of a are linearly dependent when
the columns of a are linearly dependent what
do we call the matrix a to be singular matrix
when a is singular matrix now just compare
this equation with this equation
i want a nontrivial solution v a nonzero solution
v if v equal to 0 then 0 equal to 0 i am not
interested in that solution i am interested
in the nontrivial solution so when will i
get a nontrivial solution what is singular
lambda i minus b only lambda i -b is singular
then only i will get a v which is nontrivial
which is nonzero and you will give me 0 only
when lambda i minus b is singular only then
you will get a solution to this problem which
is nontrivial you will get a vector v which
is not equal to 0 vector only when this is
singular what is the algebraic condition for
this matrix to be singular 
determinant equal to 0 that is the origin
of your eigenvalue problem
so i am just continuing from there or determinant
of lambda i minus b equal to 0 normally when
you start studying eigenvalues and eigenvectors
we start with the point which is actually
the end point so normally your thought that
eigenvalues of a matrix b are defined as this
so why this this comes out because you are
trying to solve a linear difference equation
and this equation pops out if you want to
get a nontrivial solution for the linear difference
equation actually same thing happens if are
trying to solve linear differential equations
these equations pops out when you are trying
to solve linear differential equations look
at strang’s book equation it gives a beautiful
derivation for how it happens so this equation
is a fundamental equation it pops out and
that is why we keep studying eigenvalues eigenvectors
let us look at this equation little more detail
there is one more thing here at this point
how many unknowns are there another view point
the same equation
how many unknowns are there in this equation
three unknowns v is a n cross 1 vector you
do not know v you do not know lambda so how
many unknowns are there b is a matrix which
is known lambda is not known and elements
of v are not none we do not know what is v
how many equations i have i have n equations
these are n equations there are n plus 1 unknowns
to solve it exactly how many equations you
need n plus 1 equations and n plus 1 unknowns
is this is a linear equation or a nonlinear
equation linear think what are the unknowns
lambda and v does lambda multiply v is it
a linear equation my unknowns are v1 v2 v3
vn and lambda lambda multiplies you have n
nonlinear equations in n plus 1 unknowns here
n nonlinear equations in n plus 1 unknowns
and you have to solve it to solve it somebody
would have said i will use or something
we have used very very intelligent argument
we said a solution 
will exist only when this matrix is singular
how many solutions we should get here you
will get a nonlinear equation in general has
multiple solutions it will become evident
here that this nonlinear equation will have
multiple solutions so you have this additional
equation now this one more equation those
are n equations and n plus 1 unknowns
this is additional equation n plus 1th equation
this and that together you can solve what
do you get here now this you know when you
put this you get a polynomial of degree n
so this is actually it is a polynomial of
degree n and this has roots lambda 1 lambda
2 lambda 3 this has roots lambda 1 lambda
2 up to lambda n it will have n roots some
of them could be repeated some of them could
be whatever we do not worry about it right
now
so it has n roots it is the nth order polynomial
it will have n roots now corresponding to
each root you get one v vector because this
equation will hold for every lambda if you
pluck in lambda 1 here you will get 1 matrix
which is singular corresponding to that matrix
there will be a null space
corresponding to that matrix is a null space
and that v will belong to that null space
if you pluck in lambda 2 will be one more
similar matrix and v2 will be in that null
space and so on so for every eigenvalues this
root of this equation well this is called
a characteristic equation of matrix b and
roots of this equation are called eigenvalues
all that you already know now so what happens
is
lambda 1 i minus b v1 equal to 0 so v1 is
the first eigenvector v we call this this
is actually a vector in the null space of
lambda1 i minus b this is the singular matrix
for this value of lambda so there are n different
numbers which make this lambda i minus b singular
for each of those cases we have a vector v1
v2 v3 so i have this lambda 2 i minus b v2
equal to 0 and so on so i can write this n
eigenvector i get this n eigenvectors now
what did i start with i started with solving
my linear difference equation and i took a
guess solution
i started solving this z k plus 1 equal to
b zk this is my linear difference equation
i wanted to solve it and then i said i have
a guess solution zk equal to lambda to power
k * vector v it turns out that there are multiple
such lambda there are multiple such lambda
and there are multiple such vectors v not
just one vector so i have multiple solutions
i can call them as fundamental solutions or
eigen solutions so i have a solution which
i would call as let me call this as solution
number 1 which is lambda 1 to power k v1
this also obeys the difference equation then
let us call this z2k which is lambda 2 to
power k v2 this will also obey the difference
equation what is the first fundamental criteria
the solution should obey the difference equation
so this will obey this will obey there seems
to be n such solutions which obey the difference
equation in fact this is the linear difference
equation
you can show that any linear combination of
this eigen solutions are fundamental solutions
will also obey the difference equation any
linear combination so i can construct a solution
which is like this i can construct a solution
which is linear combination of fundamental
solution so i will let me use the same notation
that is here so linear difference equation
you can very very easily show that if each
one of these fundamental solutions obeys the
difference equation just pluck in and see
that a linear combination of these also will
obey the linear difference equation
so the first criteria satisfied what is the
second thing that we need to do there is one
more thing that we need to do is to relate
this solution to the initial condition because
a solution is found for a particular initial
condition z0 z0 has not come into picture
till now but do you agree that this guess
solution will exactly satisfy the linear difference
equation next thing is to match it with the
initial condition
now when i am writing an arbitrary linear
combination this is the general solution what
is unknown here alpha 1 lambda is known once
b is known matrix b is known i can write i
can get eigenvalues lambda 1 to lambda n are
known lambda 1 to lambda do not depend upon
initial conditions z0 they only depend upon
b they only depend upon b what about v1 to
vn will not you have a matrix eigen space
is fixed eigen directions are fixed
she might choose a different eigenvector she
might choose a different eigenvector but they
have same eigen directions values 1 minus
1 and minus 1 1 are same eigenvectors in the
sense that same eigen directions except so
eigen directions are fixed so i need to known
alpha 1 alpha 2 to alpha n and what we will
see is that alpha 1 to alpha n are determined
by the initial condition z0
so this solution is if this has to be a solution
then z0 should be equal to alpha 1 v1 plus
alpha 2 v2 alpha n vn everyone with me on
this i am just going to write this as v1 v2
vn alpha 1 to alpha n this is the first column
this is the second column these are all vectors
these are all column vectors i am just putting
them next to each other forming a matrix let
us for the time being assume that eigenvectors
are linearly independent let us assume for
the time being a simplified case
so this matrix this is a n * n matrix each
one of them is a n * 1 vector eigenvector
is n * 1 vector n of them are placed next
to each other this is a n * n matrix and this
alpha 1 to alpha n so you have to solve you
know z0 when you are solving the linear difference
equation i know the initial condition i know
z0 so it turns out that alpha 1 to alpha n
are determined by the initial condition that
is you have to solve this equation psi times
equal to z0 or i could rewrite this as if
eigenvalues of or if eigenvectors are linearly
independent i can simply write psi inverse
is everyone with me on this see what i wanted
to say is that look we stared solving this
equation then we came up with that guess solution
we found that there is not just one guess
solution there are n guess solutions what
are this n guess or fundamental solutions
lambda 1 to power k v1 v1 is the first eigenvector
lambda 2 to power k * v2 v2 is the second
eigenvector and lambda 1 lambda 2 lambda 3
are eigenvalues
these are eigenvalues and v1 to vn are eigenvectors
so this seems to be a fundamental property
of the given matrix so if i give you a matrix
for any initial condition this will be the
general solution i need to find out any linear
combination of this it is a linear difference
equation you can very very easily show that
if this equation is obeyed or this fundamental
equation is satisfied by the difference equation
any linear combination of this fundamental
equations is also satisfied by the difference
equation
so general solution is linear combination
of fundamental solutions general solution
is linear combination of fundamental solutions
now this general solution should match with
the initial condition initial condition will
give me these unknowns alpha 1 alpha 2 alpha
3 and so on how do i get the unknowns if eigenvectors
are linearly independent psi inverse z0 i
have completely solved the problem now next
question arises do i get insight by solving
it this way you would say well i could have
solved this always as
i know that this is the solution what do i
gain by writing it in such a complex way that
is what i want to come to now that how does
it relate to the analysis of this equation
now let me see how do we translate the wisdom
that we gain from scalar case to the vector
case this actually equation helps us in many
ways than b to the power k because it gives
you insight into what is happening internally
first of all if you look carefully here i
have written the solution a general solution
of this problem as linear combination of some
fixed vectors do you see this i am saying
any solution for any z 0 for any z0 the solution
is given by this so lambdas are fixed you
give me b matrix lambda is get fixed eigenvalues
get fixed eigen directions are fixed so v1
to vn is determined so what is specific to
initial condition is only alpha 1 alpha 2
how you combine the fundamental solutions
is the only difference
how do you create that linear combination
is the only difference otherwise the solution
always behaves according to this is that clear
there are fixed directions linear combination
of fixed directions and then this lambda 1
to lambda n is fixed it does not vary with
z0 only this multiplying constants change
now let us 
start looking at what is changing with k lambda
i to power k is the only term which is changing
with k because once i have matrix b eigen
directions are fixed once you give me initial
condition alpha 1 to alpha n gets fixed by
this equation because this depends only on
the eigenvectors
this depends on the eigenvectors so alpha
1 to alpha n get fixed from the initial condition
v1 to vn are eigen directions only thing that
is changing with k is lambda to power k i
need to now analyze how a scalar raise to
power k behaves i am very good at it b to
power k b is a matrix b is 100 cross 100 matrix
difficult to analyze 100 cross 100 elements
how they behave as a function of k here i
have reduced the problem to only analyzing
n numbers what are the n numbers eigenvalues
now let us try to get inside into lambda i
to power k
how does lambda i to power k behave can you
tell me something when lambda is less than
1 so it minus 10 also it will converge minus
10 is less than 1 modulus so this seems to
be important mod lambda i should be strictly
less than 1 if it is equal to 1 then you have
trouble so let us look at different cases
so my case 1 which is analogous to the scalar
case is mod lambda i is strictly less than
1 for all i equal to 1 2 n this is my first
case all eigenvalues are such well there is
one more additional things
that crops in here which is different from
the linear scalar case eigenvalues they do
not be always real eigenvalues can be complex
so actually when you are doing this analysis
your space is not real numbers they could
be complex numbers so in the complex plain
lambda plain what i am saying is that if i
draw this unit circle 
this is my lambda plane and this is my unit
circle 
this is imaginary this is real this is imaginary
axis this is real axis
what i am saying is that if all eigenvalues
of b are strictly inside the unit circle whatever
is the matrix 100 cross 100 10000 cross 10000
or 5 cross 5 if all eigenvalues are strictly
inside the unit circle go back and look here
if all eigenvalues are strictly less than
inside the unit circle what will happen to
mod zk norm zk now zk is a vector so norm
zk we need to look at now norm because what
will happen to norm zk well you have to do
the proper analysis
so norm zk will be norm alpha 1 lambda 1 to
power v1 plus alpha n lambda n to power k
vn which is less than or equal to mod alpha
1 mod lambda 1 to power k norm v1 and these
quantities this is not changing this is not
changing what is going to change is lambda
to power k if mod lambda i for all i is strictly
less than 1 what will happen the right hand
side will shrink and this left hand side is
less than the right hand side and this is
of course greater than 0 right norm is always
greater than 0
so if right hand side shrinks as k goes to
infinity then the left hand side also will
shrink so this will go to 0 because this will
go to 0 because mod lambda i is my next case
so if all eigenvalues are inside unit circle
why unit circle eigenvalues are complex numbers
if all eigenvalues are inside there is one
more step where it should show that this is
less than or equal to mod lambda i raise to
k and so on i have skipped one small step
in between but you can do that it is not difficult
what if even if one eigenvalue does not obey
this if you know there are 100 eigenvalues
99 of them are inside to the circle one of
them are on the unit circle what will happen
if one eigenvalue so the last eigenvalue is
on the unit circle what will happen other
components all the other terms will shrink
one term will not shrink so this will not
go to 0 this will be bounded it will neither
grow after some time nor shrink it will become
constant it is bounded above and it is bounded
below because this is between 0 and some value
even if one of them is on the unit circle
so it does not help me if the eigenvalue is
lying on the unit circle somewhere even if
one of the eigenvalue is lying on the unit
circle i have a trouble the trouble in the
sense well there is something that we have
achieved her right now let us not worry about
the convergence problem what we have achieved
is that just looking at the eigenvalues i
can tell you how the difference equation is
going to behave
i can talk about the qualitative behavior
of the difference equation by knowing relative
position of the eigenvalue in the complex
plane whether this is inside unit circle whether
it is outside unit circle whether it is on
the unit circle what if one eigenvalue is
outside the unit circle effect of other eigenvalues
will go to zero but that one eigenvalue will
keep growing and zk will go to infinity see
this fact for any initial condition it is
what i want to emphasize
see this analysis tells us how the solution
will behave for any initial condition so initial
condition will only determine alpha 1 to alpha
n even if one of them is on the unit circle
then and remaining are inside the unit circle
then the solution will be bounded as to infinity
it will not go to 0 and even if one of them
even if one eigenvalue is outside the unit
circle i am guaranteed so i can just look
at the eigenvalues
so my second case equivalently is that 
some eigenvalues are strictly less than 1
remaining are on the unit circle all that
i can say is that is bounded 
as k goes to infinity and my third case is
obviously when
some eigenvalues are such that lambda i mod
lambda i is strictly greater than 1 it could
be 1.0001 does not matter if it is outside
the circle it is outside the circle what we
can guarantee is that as k goes to infinity
norm vk will go to infinity it does not matter
few eigenvalues are inside the circle one
eigen outside the circle can make the solution
go to infinity at sometime or the other as
k goes to infinity
so this will become unbounded as k becomes
infinity so this analysis we could do without
requiring to solve for a given z0 i do not
have to solve for z0 i just take matrix b
i look at eigenvalues when eigenvalues have
certain properties well i am done i can tell
whether solution is going to converge or to
diverge so we started by looking at
ek plus 1 equal to s inverse t ek we started
by analysis this difference equation so what
is the model of the story eigenvalues of s
inverse t should be strictly inside unit circle
i should choose a equal to s minus t i should
choose this split in such a way that eigenvalues
of s inverse t should be strictly inside the
unit circle if that happens i am guaranteed
that convergence will occur wherever i stat
from i start from an absurd initial guess
my iterations will converge to the solution
so this is the key now we will start developing
from there because well you might say you
are transferring the problem from one difficult
problem to other difficult problem if you
have a 1000 cross 1000 matrix finding out
eigenvalues is also equally tough problem
finding out eigenvalue is not a joke you have
to solve a polynomial of order 1000 well can
do it but there is a limit
the matrix size starts going it also will
hit into a it is not a trivial problem to
get eigenvalues so we have a nice analysis
but still we have problem because this still
need lot of computations inside of the eigenvalues
you could have even solve for your z0 and
tried to see how the solution is behaving
so is there some more something else some
other properties i can use so we will look
at that in our next lecture
