We are considering solution of non-linear
equations. Last time we defined newton's method
and secant method and also bisection method;
so, for the bisection method we have seen
that the convergence is very slow. Now, today,
we are going to show that newton's method,
it converges quadratically or the order of
convergence is two; and for secant method
it is going to be better than linear convergence,
but less than quadratic convergence.
So, let me recall the definition of order
of convergence which we defined last time.
So, we look at a sequence xn of real numbers
converging to c. Then let en plus 1 be the
difference between c and xn plus 1, so c minus
xn plus 1; so, if modulus of en plus 1 divided
by modulus of en raise to p, so en is going
to be error at the nth stage, en plus 1 is
the error at the n plus first stage.
So, look at the quotient, modulus of en plus
1 divided by modulus of en raise to p; if
limit of this is equal to m where m is not
0, so limit as n tends to infinity modulus
of en plus one divided by mod en raise to
p; if it is equal to m not equal to zero,
then we say that m is the asymptotic error
constant, and p is order of convergence; so,
this p we are going to show that, in case
of fix point iteration it is going to be equal
to 1, in case of newton's method it is going
to be equal to 2, and in case of secant method
it will be about 1.6; so, better than linear,
but less than the newton's method, which is
quadratic convergence.
So, first let us show the linear convergence
in the fix point method. Now, in all these,
like for newton's method, for fix point iteration,
in order to show the order of convergence
we are going to use mean value theorem or
extended mean value theorem; and for the secant
method we will use the error in the polynomial
approximation.
So, here is the order of convergence, xn is
converging to c as n tends to infinity; by
en we denote the error xn minus c, and limit
as n tends to infinity modulus of en plus
1 by mod en raise to p, if it is equal to
m not 0, then p is the order of convergence,
and m is the asymptotic error constant.
So, let us first look at the fix point iteration
or Picard's iteration. So, we have got g to
be a map from interval a b to interval a b,
it is a continuous map, and modulus of g dash
x is less than or equal to K less than 1 for
x belong to open interval a b.
Then g has a unique fixed point 
c in interval a b; and our iteration is xn
plus 1 is equal to g xn, n is equal to 0,
1, 2, and so on, and x 0 is starting point
which is any point in the interval a b. So,
when I consider c minus xn plus 1, this is
equal to g of c - c being a fix point - minus
g of xn; and now, I use mean value theorem
to write the right hand side as c minus xn
g dash of dn.
So, this is our en plus 1, and this is our
en; look at our en plus 1 is en multiplied
by g dash d n. So, modulus of en plus 1 divided
by modulus of en is equal to mod g dash d
n, d n is between c and xn; our xn is going
to converge to c, so that will imply that
d n also will converges to c then assuming
continuity of the derivative, this will tend
to modulus of g dash c.
So, if this is not equal to 0 then we have
got mod en plus 1 divided by mod en is equal
to mod g dash c, so this will be our M, and
our p is going to be equal to 1; and does
for the fix point iteration the order of convergence
is going to be equal to 1. So, this is under
the condition that g dash c is not equal to
0; if our fix point c is such that g of c
is equal to c and g dash is equal to 0, in
that case we are going to get order of convergence
to be equal to 2. So, this part we will see
little later.
So, now from the fix point iteration let us
go to newton's method. Now, the newton's method
it need not always converge. So, first we
are going to show that if the iterates in
the newton's method, if they converge then
they have to converge to a 0, if the convergence
is there then it is to the zero of the function;
then we will look at an example where the
iterates in the newton's method they domain
not converge, but they will oscillate.
We will then consider sufficient conditions
for the convergence of newton's method. Now,
what we are going to do is, this newton's
method we will write it as fix point iteration
and we have got sufficient condition for convergence
of fix point iteration, so we will just translate
those, so that gives us a sufficient condition
for convergence of newton's method; then we
will look at the some other set of conditions
which also gives us convergence in the newton's
method and then we will look at the order
of convergence of newton's method.
So, here is newton's method, f has a simple
zero at c, so this is our assumption, that
means, f of c is equal to 0, f dash c is not
equal to 0, x 0 is our initial guess, xn plus
1 is equal to xn minus f xn divided by f dash
xn, n is equal to 0 1 2 and so on. And we
had seen yesterday interpretation of newton's
method as you look at the tangent to the curve
at point xn f xn, see where the tangent cuts
x axis, so the intersection of the tangent
to the curve at xn f xn and the x axis.
That is going to give us our next iterate
xn plus 1. So, we need the condition that
at no point the tangent should become horizontal,
that will be the case if at some point f dash
xn is equal to 0; so, that is why we our starting
assumption is f of c is equal to 0 f dash
c not equal to 0. So, if you are in neighborhood
of c f dash xn will not be 0 in any case at
present we are assuming that the iterates
are defined.
Now, suppose, iterates converge, so if we
have got, we have got iterates xn plus 1 is
equal to xn minus f xn divided by f dash xn.
Suppose, that all these iterates xn they lie
in the interval a b and function f and its
derivate, they are continuous on interval
a b.
Now, suppose xn is converging to d, so suppose
then xn plus 1 also will tend to d, because
it is the same sequence by continuity of f
f xn will tend to f of d f of xn or other
f dash of xn will tend to f dash of d. So,
we get c or rather d is equal to d minus f
of d divided by f dash d; so, that gives us
f of d to be equal to 0. So, if iterates xn
in the newton's method, if they are converging
then they have to converge to a zero of our
function. Now, let us look at an example where
the iterates they may not converge .
So, look at this example, f is defined on
interval minus 3 to 3 taking real values;
and the definition is fx is equal to root
of x minus 1 x bigger than or equal to 1 and
for x less than 1, the definition is minus
root of 1 minus x 1 can see that f of 1 is
equal to 0, and that is going to be unique
0. In the interval minus 3 to 3 the function
has only 1 0 and that is one now we need to
calculate the derivative.
So, f dash x is going to be 1 upon two times
root of x minus 1 if x is bigger than 1; and
1 upon two root of 1 minus x if x is less
than 1.
So, this function will be differentiable in
the interval minus 3 to 3 except at point
1. So, this is our f dash x; now xn plus 1
is going to be equal to xn minus f xn divided
by f dash xn. So, you have got xn plus 1 is
equal to xn f xn divided by f dash xn, so
f dash xn is in the denominator, so it will
go in the numerator and you will get minus
2 times xn minus 1 provided your xn is bigger
than 1.
If xn is less than 1, then it is going to
be minus root of 1 minus xn and then there
is this 2 into root of 1 minus xn again going
in the numerator; so, again whether xn is
bigger than 1 or xn is less than 1, f xn divided
by f dash xn is going to be 2 times xn minus
1, so this is equal to 2 minus xn.
From this relation I conclude that xn plus
1 minus 1 is equal to 1 minus xn. So, if I
start with x 0, if I take x 0 is equal to
1, and then of course it is we have already
found the 0. So, if x 0 is not equal to 1,
then our x 1 is going to be equal to 2 minus
x 0, it is going to oscillate between x 0
and two minus x 0.
So, we have got xn plus 1 is equal to minus
xn. So, you start with x 0, x 1 is going to
be 2 minus x 0, x 2 will be 2 minus x 1, so
it will be 2 minus 2 minus x 0 which is equal
to x0. So, x 2 is equal to x 0 that will give
you x 3 to be 2 minus x 3 will be equal to
2 minus x 0 and so on.
So, our sequence xn that is going to oscillate
between x 0 and 2 minus x 0; so, in this example
we had our function is defined on interval
minus 3 to 3, there is a single 0 and that
is 0 is equal to 1. Now, no matter how near
you choose your x zero to one no matter how
you are starting point is near to the 0, you
are the sequence which you get it is a oscillatory
sequence, this is a pathological example.
In general there are more chances of the convergence
of iterates provided your starting point is
near your 0. Now, in this example if you happened
to choose your starting point to be the 0
itself, then you will you have convergence,
you are going to get the constant sequence,
but otherwise it remains oscillatory and we
have no convergence. Now, look at the function,
it is continuous on the interval minus 3 to
3, but it lacks differentiability at an interior
point.
So, now, let us look at some of the sufficient
conditions for convergence of newton's iterate.
So, we have as I said we are going to try
to write newton's method as an fix point iteration
and then for the fix point iteration we have
got sufficient condition.
So, our newton's iterates are xn plus 1 is
equal to xn minus f xn divided by f dash xn.
So, if I define my function g to be g x is
equal to x minus fx upon f dash x, then I
can write the newton's iterates as xn plus
1 is equal to g xn. So, this i can look at...,
so this is newton's method; and the iterates
can be written as a fix point iteration for
this function g.
Now, for the convergence of Picard's iteration
what we had was g should map interval a b
to interval a b, it should be continuous;
and modulus of g dash x should be less than
or equal to k less than 1 for x belonging
to interval a b. So, we needed continuity
of our function g, and we needed differentiability
over open interval a b and the derivative
should be less than or equal to k, where k
is less than 1 for each x belonging to a b.
So, under these conditions we showed that
g has a unique fix point and no matter what
starting point x 0 you choose in the interval
a b.
The Picard's iterations xn plus 1 is equal
to g xn, they are going to converge to the
fix point our g now is g x is equal to x minus
fx upon f dash x. So, first thing we need
to assume is that, f dash x should not vanish,
so f dash x should not be equal to 0, so that
our function g is defined on the interval
a b. When we will look at the derivative of
g, the second derivative of function f will
come into picture, so our function f should
be twice differentiable; and then let us calculate
g dash x with g x is equal to x minus fx upon
f dash x, whatever condition we get we will
say that that should be less than 1; so, let
us write down this condition
g x is fx x minus fx upon f dash x; so, the
first condition is f should be 2 times continuously
differentiable on interval a b.
Second, f dash x should not be equal to 0
for x belonging to a b; third condition will
be look at g dash x, so g dash x will be derivative
of x is 1 and for fx upon f dash x let me
use the quotient rule, so it will be f dash
x square, then denominator into derivative
of the numerator, so it will be f dash x square
minus fx into derivative of the denominator,
so it is going to be f double dash x; so,
this is going to be equal to fx f double dash
x upon f dash x square; so, we want modulus
of fx f double dash x by f dash x square,
this should be less than 2 for x belonging
to a b; and an important condition is that,
g should map interval a b to interval a b.
So, if these conditions are satisfied, then
our newton's method it is going to converge;
and we have seen that when it converges it
is going to converge to 0.
Also these conditions they imply that g has
a unique fix point in the interval a b and
fix point of g is nothing but 0 of f. So,
we are going to have a unique 0 of function
f and the newton's method or the newton's
iterates they are going to converge; now,
this is one set of conditions; so, here we
had just translated, so let us see more say
geometric conditions for convergence of newton's
method.
And these methods are..., so this is as before
that f should be 2 times continuously differentiable,
then f a into f b should be less than 0; that
means, f a and f b they are of opposite signs;
f dash x not equal to 0 x belonging to a b,
this also was assumed in the earlier set of
conditions; f double dash x bigger than or
equal to 0 or f double dash x less than or
equal 0 on close interval a b; and modulus
of f a upon modulus of f dash a should be
less than b minus a.
And mod f b by mod f dash b should be less
than b minus a. Then for any x0 in a b the
Newton's iterates xn will converge to c with
f of c is equal to 0, f dash c not equal to
0. So, look at the first condition, f a into
f b is less than 0; so, by the intermediate
value theorem f has at least one 0 in the
interval a b; if you assume that f dash x
is not equal to 0 in the interval a to b,
along with this condition consider f double
dash x bigger than or equal to 0.
The second derivative tells you something
about concavity and convexity of the function.
Now, if f double dash x is strictly bigger
than 0, that will mean that f dash has to
be strictly increasing. If f double dash x
is strictly less than 0, that will mean that
f dash is strictly decreasing; so, this condition
f dash x is not equal to 0, it tells us that
f dash is going to be of the same sign, it
will be either bigger than 0 or it will be
less than 0; the fact that f a into f b is
going to be less than 0, that tell us that
there is at least one 0, then f dash x it
will be either bigger than 0 or less than
0; if f dash x is bigger than 0, f will be
strictly increasing; if f dash x is less than
0, it will be strictly decreasing; that means,
there is going to be unique 0 in our interval
a b.
Then the third is something about the convexity
and concavity; and the last condition that
those conditions are imposed to guarantee
that if you choose the starting point say
x 0 is equal to a or x 0 is equal to right
hand point b, then the next iterate they will
lie in the interval a b. Because what we want
is all our iterates they should be in the
domain of our f, f is defined on interval
a b; so, I am not going to prove this theorem,
but I am just going to show you that the last
condition implies that if I choose x 0 is
equal to a, then x one is going to be in the
interval a b.
And the proof is simple. So, our x 1 is going
to be equal to x 0 minus fx 0 upon f dash
x 0; let x 0 be equal to a, then we have x
1 is equal to a minus f a divided by f dash
a. So, modulus of x 1 minus a will be equal
to modulus of f a divided by mod f dash a
and this we are assuming to be less than b
minus a, so this will imply that our x 1 is
in the interval a b.
So, this last condition mod f a by mod f dash
a less than b minus a, that implied that the
first iterate x 1 is going to be in the interval
a b. And a if you choose a x 0 is equal to
b then the other condition will guarantee
that x 1 belongs to interval a b. Now, we
want to show that the iterates in the newton's
method they converge quadratically.
This is the advantage of newton's method,
like this is one of the plus point that is
why newton's method is so popular; that if
it converges, it is going to converge quadratically;
Picard's iteration it converges only linearly.
Now, let me show you that the newton's method,
it is going to converge quadratically; that
means, when I consider the error at the n
plus first stage, modulus of en plus 1 divided
by mod en square that will converge to a non-zero
constant as n tends to infinity, so because
mod en square, so that too is the order of
convergence.
So, our iterates are xn plus 1 is equal to
xn minus f xn upon f dash xn, xn's are converging
to c such that f of c is equal to 0, and f
dash c is not equal to 0. So, let me look
at f of c and write Taylor's series expansion,
so it is going to be f of xn plus f dash xn
into c minus xn plus f double dash at some
d n into c minus xn square divided by 2; so,
this is the extended mean value theorem or
truncated Taylor series for expansion. Our
f of c is equal to 0, so what I do is, I divide
throughout by f dash f xn and I take this
first two terms on the other side; so, when
I do that I will have xn minus f xn divided
by f dash xn minus c, so what I have did is
I am dividing throughout by f dash xn and
taking the two terms on the other side, so
i have xn minus f xn by f dash xn minus c
is equal to f double dash d n divided by 2
times f dash xn into c minus xn square; take
mod of both the sides, here xn minus f xn
upon f dash xn that is our xn plus 1.
So, this is modulus of xn plus 1 minus c,
so that is our modulus of en plus 1 is equal
to modulus of f double dash dn divided by
2 times f dash xn into c minus xn is en, so
it will be modulus of en square. Look at this
condition f dash of c not equal to 0 xn is
tending to c, so for n large enough f dash
xn is not equal to 0, so we have modulus of
en plus 1 upon mod en square is equal to....
So, you have mod en plus 1 is equal to mod
f double dash dn by 2 times mod of f dash
xn into mod en square, xn is tending to c,
d n lies between xn and c; so, assuming second
derivative to be continuous you get modulus
of en plus 1 divided by mod en square is equal
to mod f double dash d n divided by 2 times
mod of f dash xn, which converges to mod f
double dash c divided by 2 times f dash c.
So, this will be our asymptotic error constant
and our p will be equal to 2; so, we have
got quadratic convergence; so, this was for
the newton's method.
Now, we are going to look at secant method.
So, in the secant method what we do is, we
start with two points x 0 and x 1; in newton's
method we have got only one x 0, and then
we looked at the tangent to the curve at x
0 fx 0; for the secant method as the name
suggest we are going to look at two points
on the curve x 0 fx 0 x 1 fx 0, look at the
straight line joining them c where it cuts
x axis, that is going to be our x 2.
And then you consider x 2 and x 1, look at
the secant which passes through x 1 fx 1,
x 2 fx 2 c, where it cuts x axis that is going
to give us x 3 and so on, so this is the secant
method. So, as we showed that when the iterates
in the newton's method converge, they have
to converge to a zero of our function, same
thing we will show for the secant method.
Then we are going to show that our formula
is going to be symmetric in xn and xn minus
1; xn plus 1 in the secant method, the formula
is in terms of xn and xn minus 1 the values
of function.
So, we will show that it is symmetric and
then we will consider the order of convergence
in secant method; x 0 and x 1 they are in
the interval a b, what we are doing is f dash
xn in the newton's method is replaced by the
divided difference based on xn minus 1 and
xn. So, xn plus 1 is equal to xn minus f xn
divided by divided difference based on xn
minus 1 xn, so I substitute, it is going to
be xn minus f xn divided by f xn minus f xn
minus 1 divided by xn minus xn minus 1.
Suppose, xn converges to c, then xn minus
1 also converge to c, so it is the same sequence,
and continuity of the divided difference gives
us that this will converge to f of c c, but
our definition of divided difference when
the arguments are repeated it is f dash c;
so, you get c is equal to c minus f c divided
by f dash c; so, that will give you f of c
is equal to 0. So, whenever the iterates converge,
they are going to converge to 0 of our function.
Now, here is the symmetric like xn plus 1
is equal to xn minus f xn upon f xn xn minus
1; so, if I interchange xn and xn minus 1,
that means, if I consider xn minus 1 minus
f of xn minus 1 divided by divided difference
based on xn xn minus 1 I am going to get the
same result.
And this result is something expected, because
what we are doing is we are looking at point's
xn minus 1 and xn, these we have obtained
by the iteration process so far. Now these
two points I look at the corresponding points
on the curve, I join them by straight line,
so then whether I the order should not matter,
what matters is the two points xn minus 1
and xn; whereas, if you look at the formula
xn plus 1 is equal to xn minus f xn divided
by f of xn minus 1 xn the divided difference,
it is not evident, how I can instead of xn
I can write xn minus 1 and instead of xn minus
1 write xn.
So, one has to do a bit of calculation. So,
let us work out the details. So, we have got
xn plus 1 is equal to xn minus f xn divided
by the divided difference, I substitute for
the divided difference, so I have xn minus
f xn multiplied by xn minus xn minus 1 and
then divided by f xn minus f xn minus 1.
So, now, multiply this xn by f xn minus f
xn minus 1, so I will get xn f xn minus xn
f xn minus 1, then minus xn f n and then plus
xn minus 1 f xn; so, this xn f xn will get
cancelled, so you have xn minus 1 f xn and
minus f xn f of xn minus 1; so, we are writing,
what we are doing is, we are adding and subtracting
xn minus 1 f xn minus 1 from here, what I
have is xn minus 1 f xn, so it is this term,
then minus xn f xn minus 1 it is this term;
so, I am subtracting xn minus 1 f xn minus
1 and I am adding it.
When I do that I will get xn minus 1 minus
f xn minus 1 and this is nothing but the divided
difference based on xn xn minus 1. So, this
formula and this formula it is the same, its
symmetric in xn minus 1 and xn. So, now, we
want to look at the order of convergence in
the secant method; earlier what we did was
we looked at f of c is equal to zero for newton's
method, then for f of c we wrote the Taylor's
formula. Now, here what you will have to do
is, you will have to consider the error in
the interpolating polynomial and then the
remaining proof will be similar.
So, we have got fx is equal to f xn plus divided
difference based on xn xn minus 1 into x minus
xn plus this is the error term; so, this is
linear approximation; this is a polynomial
of degree less than or equal to 1 which interpolates
the given function at xn and xn minus 1; this
is the error term f xn xn minus 1 x x minus
xn x minus xn minus 1, so this is from our
polynomial interpolation.
So, the results from polynomial interpolation
we keep on needing them often like all our
numerical integration it was based on the
polynomial interpolation; numerical differentiation
also the polynomial interpolation it came
into picture. Now, for this solution of non-linear
equations, linear approximation when you consider
the tangent line approximation that means,
your interpolation point is repeated twice;
you get Newton's method when you take the
points xn and xn minus 1 and fit a polynomial
of degree less than or equal to 1 you get
secant method.
So, now, you have got this f x, so write 0
is equal to f of c, so I am substituting x
is equal to c, so it will be f xn plus c minus
xn, this is the divided difference plus f
of xn xn minus 1 c and then c minus xn into
c minus xn minus 1. As we did in case of newton's
method, let us divide by this divided difference,
so you are going to have it to be equal to
xn, I am going to take this term on the other
side and I will be multiplying by xn minus
xn minus 1.
So, as such what we have is, we have got 0
is equal to f of c plus or is equal to f of
xn plus c minus xn divided by c minus xn into
multiplied by divided difference xn xn minus
1 plus divided difference based on xn xn minus
1 c multiplied by c minus xn c minus xn minus
1, so I will divide by this divided difference.
So, I am going to have 0 is equal to f xn
divided by divided difference based on xn
xn minus 1 plus c minus xn plus divided difference
based on xn xn minus 1 c divided by f of xn
xn minus 1 and multiplied by c minus xn c
minus xn minus 1. So, from here I have got
this, this divided difference.
Now, if you take this on the other side, then
what I am going to get is, so I am going to
take this on the another side, so it will
be xn minus f xn divided by f of xn xn minus
1 minus c is equal to the right hand side
f of xn xn minus 1 c divided by f of xn xn
minus 1 into c minus xn c minus xn minus 1,
this is our xn plus 1.
So, we have got xn plus 1 minus c, take the
modulus, so you will have modulus of en plus
1 to be equal to modulus of f of xn xn minus
1 c divided by f of xn xn minus 1 and mod
en mod en minus 1.
So, here now we have got, if you compare with
the newton's method we had modulus of en plus
1 is equal to something into mod en square,
but now we have got this mod en and mod en
minus 1.
So, we will see next time that, this is going
to make the order of convergence to be less
than 2, it will be about one point six. Next
time we are going to consider one more method
which is known as regula falsi method; and
then we will compare these methods, what are
the advantages? What are the drawbacks? And
then we are going to consider iterative methods
for solution of system of linear equation.
So, thank you.
