so in the last few lectures we have been looking
at convergence of iterative schemes for solving
linear algebraic equations starting from the
basic equation for way the error evolves
so 
we have this iteration scheme to solve ax=b
and a was written as s-t and we said that
the error which is defined as 
iterative xk-the true solution x star this
evolves according to e k+1 this is a linear
difference equation 
it evolves according to this linear difference
equation
now from this we abstracted a linear difference
equation problem we said essentially we have
to look at equations of this type and z zero
is initial condition and then we wanted to
come up with the way of analyzing asymptotic
behaviour of the solution as a tends to infinity
so we came up with analysis based on eigenvalues
we came up with a condition that if rho b
is nothing but m a x/i lambda i that means
if lambda 
i are eigenvalues of matrix b we find out
its absolute value where eigenvalues can be
complex
so we find the absolute value and rho b this
is called as spectral radius and we showed
that the necessary and sufficient condition
if rho b that means spectral radius of matrix
be strictly less than 1 we said that if the
spectral radius of matrix b is strictly less
than 1 then the sequence zk norm of that we
tend to 0 as k tends to infinity and from
this we again connected to our original problem
we said which means that spectral radius of
s inverse t is strictly less than 1
then this is necessary and sufficient condition
for convergence of error okay the error between
the true solution and the guess solution will
diminish to zero if this condition is satisfied
okay i am just doing a recap of what we have
done till now so from this point we again
had some difficulty because we have to compute
eigenvalues so we said that eigenvalues computations
are difficult and then we used one more result
to come up with a sufficient condition
so we said that spectral radius of matrix
b is always less than or equal to any induced
norm of matrix b spectral radius of matrix
b is less than induced norm of matrix b what
is induced norm this is induced norm induced
by the norm defined on the rain space and
the domain space so this norm of matrix is
nothing but amplification power or something
like gain of a matrix if you can think about
it as a gain or amplification power
then we came up with a sufficient condition
that if induced norm is less than one obviously
spectral radius is less than one and convergence
is guaranteed if induced norm is greater than
one we cannot say anything okay if induced
norm is less than one we are sure so we had
another condition that if induced norm is
strictly less than one then spectral radius
of b is strictly less than one and then this
implies that asymptotically norm 
of iterate zk or difference equation zk will
go to zero as k goes to infinity
so this we can say without actually having
to solve it now in particular we talked about
infinite norm or one norm which are more convenient
to do calculations now based on this i wanted
to derive some results which is even more
simpler i do not probably have to even compute
the norm i can compute what is called as diagonal
dominance so in my last lecture i talked about
diagonal dominance
so i wanted to further cash on this result
that if the induced norm is less than one
then of course the spectral radius is less
than one induced norm is very-very easy to
compute particularly infinite norm as compared
to computing the spectral radius so checking
whether a particular iteration will converge
or not is very-very easy okay now let us move
back to the thing that we have done in my
last lecture
so this was overview of the entire stability
arguments that we have been giving but i wanted
to derive something more specific from the
previous results so we come back here we are
trying to solve for ax=b and a has been split
as s-t so for jacobi method in particular
i analysed jacobi method okay for jacobi method
s=d well we are also writing a=l+d+u this
is strictly lower triangular part of a this
is diagonal part of a and this is strictly
upper triangular part of a
so s=d and t=-l-u and 
then i defined the concept of strict diagonal
dominance 
if you take sum of absolute values of elements
of matrix a in a row except the diagonal element
and if that sum is strictly less than the
diagonal element then the matrix is called
as diagonally dominant matrix okay just to
give you a simple example
well you just write any matrix let us say
this is my matrix a these are the diagonal
elements here just have a look if i take absolute
sum of this this this it is smaller than this
i take absolute sum of this this this this
smaller than this okay i just look at this
matrix i look at its diagonal elements okay
this particular matrix will obey this condition
this is a strictly diagonal dominant matrix
just look at this this is 2+4+3+1 is always
less than 15 okay same thing here okay 5+3+1+9
is less than 23 so i am taking absolute values
okay so for this particular matrix can you
calculate what is going to be jacobi matrix
which is s inverse t can you calculate that
just do it what is s inverse t
well mind you again that jacobi matrix when
you actually do computations you never compute
s inverse you do row by row calculations okay
this is for analysis this is for getting insights
but what you will realise is that you just
look at the diagonal elements you look at
the sum of all diagonal elements
you can say whether the iteration are going
to converge or not which is very-very powerful
reason you do not have to actually solve it
and this is true of 5x5 for 10x10 for 1000x1000
if this condition holds iterations will converge
okay so you have guarantee convergence if
this condition is satisfied
so what will be s inverse t what will be jacobi
matrix let us call this jacobi matrix for
jacobi matrix s inverse t what will it be
this will be zero it will be -1 1 02/-5 -5
-5 then this will be 1 0 3 -2 2/9 9 9 9 okay
-l-u then what is this 2 4 0 3 -1 this divided
by 15 15 15 and 15 okay then the next one
is 1 -5 -3 0 and -9/-23 -23 -23 and the last
row is -1/5-1/5 1/5 0 0 okay what will be
the infinite norm of this matrix you take
absolute of rows so absolute of this plus
absolute of this plus absolute of this okay
absolute of these things all these numbers
is it always going to be less than 1 it is
always going to be less than 1 because this
matrix is strictly diagonally dominant okay
in the numerator this will appear okay actually
for a diagonally dominant matrix what you
know is that this divided by norm aii this
will be strictly less than 1 you can see here
you add absolute of each one of these rows
okay if each one of them is less than one
the maximum is also going to be less than
one what does it mean
spectral radius of this matrix is strictly
less than one so if a is diagonally dominant
the jacobi matrix which you get by s inverse
t has spectral radius strictly less than one
which means jacobi iterations will converge
without having to solve it for arbitrary initial
guess very-very important for an arbitrary
initial guess okay so any initial guess i
give even if it is completely wrong my iterations
will converge to the true solution okay if
diagonal dominance condition is met
so you can just check diagonal dominance of
a matrix very-very easily and then you know
whether the solution is going to be obtained
or not that is straight forward now there
are many more results of how do you analyse
the convergence behaviour and all of them
i am not going to prove i have stated those
results here and i am just going to state
them and show you how to apply them and the
proofs for each one of them or at least most
of them
some of them you can derive yourself for most
of them are included at the end of the chapter
notes in the appendix okay i do not want to
go over it in the class you go the philosophy
of how it is done and you have to look at
the proofs in the appendix to understand more
of this because we cannot spend time on this
beyond a certain point as long as you get
the philosophy it is fine now what are the
more results
there are some more results which exploit
the structure of matrix a one structure that
we exploit is diagonal dominance right the
other thing we will show is that if matrix
a is symmetric positive definite okay if matrix
a is symmetric positive definite then jacobi
and gauss-seidel method converges also you
can show that if matrix a is diagonally dominant
gauss-seidel method will converge okay
the proof is little more involved and you
should look at the proof given in the appendix
i have given details of the proof in the appendix
so if matrix a is diagonally dominant jacobi
iterations will converge it is also true that
if matrix a is diagonally dominant then gauss-seidel
iteration also will converge okay so you just
have to check for diagonal dominance you know
that gauss-seidel iteration will converge
and in fact is that most of the time gauss-seidel
iterations converge faster than jacobi iterations
okay
so if you know that a matrix is diagonally
dominant your preferred choice of using the
method should be gauss-seidel method not jacobi
method okay for other theorems; so the first
theorem that you should know about convergence
of iteration scheme is that if matrix is diagonally
dominant a matrix okay; then jacobi method
as well as gauss-seidel methods will converge
to the true solution okay
by the way remember this this is a sufficient
condition this is not a necessary condition
what does it mean that if this condition holds
jacobi and gauss-seidel methods will converge
you cannot say if this condition does not
hold you cannot say anything about convergence
you have to go back and check something else
you have to go back and check spectral radius
okay so this is only a sufficient condition
if this happens you are guaranteed convergence
will occur if this does not happen we do not
know we cannot say anything okay so this is
a sufficient condition not necessary condition
so the second result i would say very-very
important result so the symmetric and positive
definite seems to be something very-very nice
it seems to help us everywhere we go okay
now you might start saying well i have been
given this matrix a and you know only in very-very
special cases this a will be symmetric and
positive definite isn’t it
a is a square matrix i am not talking about
the square where we had a tall matrix i am
just talking about a is a square matrix and
the problem which is given to me is such that
a is not symmetric and positive definite okay
but i know that gauss-seidel method will converge
okay sufficient condition for convergence
is that if the matrix in my problem is symmetric
and positive definite is there something i
do to solve this problem to convert this problem
into symmetric and positive definite matrix
i just pre-multiply this equation with a transpose
so this gives me a transpose a okay i do not
have to solve for ax=b i can instead solve
for a transpose a=a transpose b okay i am
guaranteed convergence so i am using my theory
to change the problem in such a way that i
am guaranteed to get converge solution okay
i am going to solve this problem using gauss-seidel
iterations making use of this theorem okay
how do i make use of this theorem to modify
my calculations i pre-multiply both sides
by a transpose this becomes a symmetric positive
definite matrix okay now if i apply gauss-seidel
method to this matrix and this transform problem
i am guaranteed to get a solution this solution
is obviously a solution if it is a solution
of this it is also a solution of this you
have no problem with that so i could solve
this transform problem instead of solving
this problem i get a symmetric positive definite
matrix here okay i am using theory to modify
my calculations
i will just give you an example here so i
want to solve for ax=b and my a matrix=4 5
9 7 1 6 5 2 9 and my n vector is 1 1 1 okay
let say i want to solve this by gauss-seidel
iterations okay well what i will do is i know
this is not a solution procedure this is analysis
okay
from analysis what i know is that if i write
this matrix as a matrix if i call this s and
if i call this 
as t then doing gauss-seidel iterations is
equivalent then my s inverse t will be 4 0
0 7 1 0 this is my s inverse t if i am able
to use the row matrix a in this case the spectral
radius of s inverse t turns out to be 73 which
is strictly less than 1 okay if i use gauss-seidel
iterations iterations are not going to converge
because if i just choose the row matrix a
that matrix is neither diagonally dominant
just check it is diagonally dominant it is
not is it symmetric matrix it is not a symmetric
matrix forget about positive definite it is
not symmetric matrix but if i know this little
bit of information okay
if i do this transformation that is a transpose
ax=a transpose b okay then this a transpose
a matrix turns out to be 90 37 okay and this
is a transpose a a transpose b becomes 16
8 24 and now if i apply gauss-seidel method
to this transformed equation then spectral
radius of s inverse t turns out to be 096
okay so for the transform problem guaranteed
convergence of gauss-seidel method this is
a symmetric matrix just see this symmetric
matrix it is a positive definite matrix by
definition
a transpose a is always positive definite
even if a is not positive definite we have
seen this several times okay this is positive
definite matrix symmetric matrix convergence
is guaranteed just pre-multiplying both sides
by a transpose i can ensure that i will get
convergence by iterative method okay so in
the case where obvious things like diagonal
dominance are not there if you want to ensure
that you get convergence just pre-multiply
by a transpose both sides and then use gauss-seidel
you have guaranteed convergence very-very
powerful result
yeah for any given no matter how would always
so that spectral radius should be less than
one is necessary and sufficient condition
if necessary if the convergence occurs spectral
radius should be less than one if spectral
radius is less than one convergence will occur
okay "professor - student conversation ends”
but that is not the case with the norm if
induced norm is less than one convergence
will occur but if induced norm is greater
than one convergence may or may not occur
you may not know that is not the case with
spectral radius spectral radius is the absolute
measure which is necessary and sufficient
condition okay so it is possible to transform
there are more results of this type again
i am not going to go into the proof
for relaxation method we have this result
for relaxation method what you can show again
is the proof again given in the appendix you
should go and have a look at it if omega is
chosen between 0 and 2 well actually for relaxation
method we want to choose it between 1 and
2 because we showed that omega equal to 1
is equivalent to gauss iteration so we want
to choose between 1 and 2 but in general if
omega is between 0 and 2 okay this is a necessary
condition for convergence okay
so you know how to choose omega you have a
guideline here okay so again remember this
is only a necessary condition this is not
sufficient if you choose less than 2 that
does not mean convergence has to occur but
convergence occurs only when you choose omega
is less than 2 this is result 3 and the necessary
condition becomes necessary and sufficient
conditions if extension to this theorem is
another result this is for an arbitrary matrix
okay
now if a is symmetric 
and positive definite so if matrix a is symmetric
positive definite okay this necessary condition
becomes necessary and sufficient condition
okay now you know how to transform the problem
which is usually not symmetric positive definite
to symmetric positive definite matrix okay
so what i want to do the take home message
is that all these theorems are very-very useful
in shaping your calculations
you should know how to make sure that convergences
occur convergence is very-very important whenever
you are not sure in a arbitrary large scale
problem you are not sure of a matrix how it
is going to be if you want to use iterative
schemes for solving ax=b it is better to use
a relaxation method in which you transform
the problem because in general relaxation
method will converge faster than gauss-seidel
method
i will just show you a very small example
that jacobi method is the slowest to converge
typcially gauss-seidel method is faster and
if you choose omega properly then the relaxation
method will even converge faster okay now
how do you choose omega such that you get
very-very fast convergence it is very difficult
to tell the you probably have to compute eigenvalues
but that is not desirable you do not want
to really compute eigenvalues
so you have to develop some kind of experience
beyond the point you have use all these theorems
and understand the theory and then develop
experience to tweak with the calculations
that is very-very important okay
so i will just show you one simple example
this is taken from book but it is very-very
illustrative very simple problem so i want
to solve and such a simple problem of course
you do not need any of the iterative methods
2x2 systems you can solve it by hand so my
a matrix is 2 -1 -1 2 well we will say that
this is jacobi and gauss-seidel will converge
why symmetric diagonal dominant okay anyway
that is not the point the point is that for
jacobi method s inverse t will be 0 1/2 1/2
0 and the spectral radius is equal to 1/2
okay
for gauss-seidel method s inverse t this turns
out to be 0 0 1/2 1/4 and spectral radius
is s inverse t okay the spectral radius is
given by this actually spectral radius maximum
magnitude eigenvalue of s inverse t is an
indicator also of the performance okay now
there are two aspects it should be less than
one okay now how much it is less than one
how close it is to zero that also matters
in terms of the rate of convergence
whether the convergence is guaranteed or not
is decided by whether it is strictly less
than one okay that is the stability criteria
the performance is given by how much it is
less than one so this jacobi method in which
spectral radius is 1/2 okay converges slower
than the gauss-seidel method okay because
the spectral radius here is 1/4 in fact if
you start with calculations you will see that
one step of gauss-seidel will be almost equal
to two steps of jacobi okay
so the gauss-seidel can move much faster you
cannot show it for every matrix this is no
proof that gauss-seidel always converges but
in general gauss-seidel convergence faster
than and the reason is typically spectral
radius of s inverse t for gauss-seidel is
less than okay that is the reason now what
if i formulate the relaxation method so you
can almost show that because of this one gs
iteration is equivalent to 2 jacobi iterations
okay because spectral radius in this case
is even smaller
now for relaxation method 
s inverse t turns out to be inverse of this
matrix 0 omega -omega here 2 inverse 2*1-omega
omega 0 2*1-omega and of course we should
choose omega between 1 and 2 we want it to
be greater than one because if it is equal
to one it is nothing but gauss-seidel method
if we want to be greater than one now for
this simple case 2x2 matrix you can actually
find out what is the best value of omega that
will enhance the convergence what is the optimum
value okay
for different choices of omega you will get
different spectral radius okay you can actually
find out which value of omega this is just
again to tell you emphasis it this is only
to get insight in real problem i am not going
compute optimum omega by doing some i have
to tune give a guess for omega
so if you use some properties then you know
that lambda 1 and lambda 2 if these two are
eignvalues of s inverse t then that is equal
to determent of s inverse t which turns out
to be in this case if you take determinant
of that it will be 1-omega whole square okay
you know this property multiplication of eigenvalues
for matrix is same as determinant and then
what is the other property trace so lambda
1+lambda 2=trace of
so this will turn out to be 2-2omega+omega
square/4 now if you plot this that is if you
plot s inverse t versus omega if you plot
spectral radius using these two relationships
you can find out lambda 1 and lambda 2 and
spectral radius; and if you plot this you
will find that getting the optimum is not
very difficult
if you plot this you will find that the optimum
value for which the spectral radius is minimum
you know you will get a point where you will
get a minimum value of the spectral radius
so that value turns out to be omega optimum=107okay
and the spectral radius of s inverse t for
this omega is 007 okay i am skipping some
steps you can see here in the notes that is
not important what i want to point out here
is that if you are able to choose omega properly
then this is 007
so we had three situations; you know jacobi
method then gauss-seidel method and relaxation
method the s inverse t spectral radius in
this case was 1/2 this was 1/4 and this is
you know 07 which is almost 1/4 of this so
we said one iteration of gauss-seidel was
two jacobi iterations what can you say about
one relaxation iteration it is like one relaxation
iteration is like four gauss-seidel iterations
almost like eight jacobi iterations
so relaxation method can converge even faster
typically values close to 1 11 12 are used
this is thumb rule and not substantiated i
think gave some clue that you can use it close
to 12 but it is hard to say generally what
value of omega will make convergence very-very
fast okay so the tricks that you should use
is first of all make sure that either the
matrix is diagonally dominant if is not okay
to ensure convergence he should pre-multiply
both the side by a transpose that will make
it symmetric positive definite i have guaranteed
convergence okay
but i want convergence faster than gauss-seidel
gauss-seidel is better than jacobi so i will
apply gauss-seidel and i can make convergence
faster even going to relaxation method so
probably i should all these tricks and use
relaxation method to enhance my convergence
that is how i should proceed with arranging
my calculations so this brings us to end of
this analysis what is important here is that
there are many take home messages
one of the things is that eigenvalues is one
of the prime tools for analysing behaviour
qualitatively asymptotically i do not have
to solve that is the beauty of this tool i
do not have to solve the problem i can just
look at eigenvalues or in this case it turns
out finally that i can just look at diagonal
dominance i can see whether to convert the
problem to a symmetric positive definite matrix
i am guaranteed convergence of my iterative
scheme very-very powerful result
in fact eigenvalues are used for convergence
analysis in engineering literature in many
many many ways okay well most of you i think
have done the first course in chemical engineering
or process control and in process control
well you may not have connected it to the
eigenvalues in the first course but actually
what you can show is that if you write a differential
equation for local linear differential equation
for evaluation of the system dynamics then
the so-called roots of the characteristic
polynomial are nothing but eigenvalues of
certain matrix which governs the system dynamics
okay
if the eigenvalues are on the left of plane
and then you know what was nice thing about
eigenvalues there or roots of the characteristic
polynomial you do not have to solve you just
look at the roots whether they are lying on
this half of the plane or this half of the
plane you can tell how the system is going
to behave asymptotically without having to
solve okay same thing is here without having
to solve i can tell whether my iterations
will converge or not okay
also using necessary sufficient conditions
i can go and modify my problem to make sure
that convergence will occur okay this is more
important than the algorithm per se algorithm
you will learn to program it or nowadays i
think these algorithm will be available on
the net you might download it algorithm might
be very well written that does not mean that
does not mean that convergence is going to
occur okay
you should know why convergence occurs and
then make sure that you transform the problem
in such a way that convergence occurs that
is important okay there are two more things
that i need to do because we missed one lecture
some timetable is disturbed but i will try
to make up for it i will try to cover optimization
based iterative methods for solving ax=b okay
so till now i formulated iterations in one
particular way by splitting the matrix in
fact row by row calculations not really splitting
the matrix the way the iterations were derived
where doing row by row calculations
my next aim is going to be instead of doing
that can i iteratively solve this problem
well if i want to solve ax=b okay well if
i take a guess solution the true solution
is let us say x star 
okay and if xk is my guess solution then obviously
ek now my ek has a different definition ek
is b-axkokay this is not going to be 0 when
this is equal to x star it will be equal to
0 if xk is equal to x star this is equal to
0
the way i want to solve this problem is minimize
scalar objective function phi is defined as
e transpose e that is ax-b transpose ax-b
with respect to x okay i do not want to solve
this well you will say that if you apply it
on a necessary condition for optimality you
will get you know dou phi/dou x=0 and we will
give you a transpose ax if you apply this
condition dou phi/dou x-=0 then you will get
a transpose ax=a transpose b
i do not want to solve this i do not want
to go by this route i want to go iteratively
okay i want to guess x0 and by some method
i want to go to x1 then i want to go to x2
and so on and then i want to see whether this
iteration converges we are going to use what
is called as the gradient search okay one
of the fundamental methods in optimization
gradient based search so we will look at gradient
search and then there is one more method called
conjugate gradient search which we will look
at next okay
that is one thing which i want to do after
having done that we have talked about iterative
schemes for solving ax=b and then we move
on to a very-very fundamental issue matrix
conditioning which problems are inherently
ill-conditioned which problems are well-conditioned
how do i classify and say that this is ill-conditioned
problem whatever i do i am going to end up
into some trouble this is a well-conditioned
problem
if i am getting wrong solution i have made
a mistake so well-conditioned problems you
know absurd solutions you have made a mistake
ill-conditioned problems absurd solutions
you cannot do much how do you classify ill-condition
from well-condition is the next thing that
will bring this to end of this module
