The following content is
provided under a Creative
Commons license.
Your support will help
MIT OpenCourseWare
continue to offer high quality
educational resources for free.
To make a donation or
view additional materials
from hundreds of MIT courses,
visit MIT OpenCourseWare
at ocw.mit.edu.
TOM LEIGHTON: This
week we're going
to talk about recurrences,
how to set them up,
how to model a problem
as a recurrent problem,
and then how to solve them.
This stuff is really useful when
you get into algorithm design
and algorithm analysis.
6006 and 6046, we you
use this stuff a lot.
We're going to start with a very
famous recurrent problem known
as the Towers of Hanoi problem.
And I'm guessing
that many of you
have seen this problem
in one form or another.
In the Tower of
Hanoi problem, you've
got three pegs and n disks,
these little guys here.
All different sizes.
And the object is to start
with a stack like this
and move the disks
around so they wind up
with the stack on another disk.
Now, the rules are you can
only move one disk at a time.
And you can never put a big
disk over a little disk.
So that you can't do.
So that would have to go there.
Then you could do this.
And now I've gotten the
top two moved over to here.
And the goal is to take
the minimum number of moves
to get the whole stack
moved over to here,
and to even figure
out if that's doable.
Now, this was invented by a
famous French mathematician
named Edward Lucas in 1883.
And in the original
legend around this puzzle,
there were 64 disks.
This one has just seven on it.
And the story was at
the beginning of time,
God placed the 64 disks on
one of the three golden pegs.
And he got an order of
monks whose lifelong mission
was to move the disks one
at a time so at the end
all 64 disks were lined
up on an adjacent peg.
And according to legend,
when that task was done
the tower would crumble
and the world would end.
Sort of explains
why monks are never
smiling, because
they work forever
and then just
something bad happens.
Now, the question we
want to be able to answer
is, how long till
the world ends?
So we can start with
a simpler version.
Say let's take three
disks and start with that.
And let's see, is it doable?
Well, we could.
That's one, that's two,
three, four, five, six, seven.
So it's doable with three disks.
And we took seven moves.
Maybe there's a
better way, and that's
part of what we'll
try to study today.
In fact, what we'd like to know
is how many moves for n disks?
And is it even finite?
So we're going to define t
sub n to be the minimum number
of moves for n disks.
So to take a stack of
n disks and move them
to an adjacent peg.
So, for example, what's t1?
One.
I got one peg, I just move it.
That's easy.
t2, 3.
Move the little
guy, move the base,
move the little guy back on top.
That's the best you can do.
And we've seen that
t3 is at most 7.
Maybe there's a better way.
But let's try to first get
an upper bound on this.
Any thoughts about how we
might go about a general upper
bound for n for this problem?
Like say I had four disks.
Any thoughts about
a strategy where
we might be able to
analyze or get an upper
bound the number of steps
for four disks here?
Or more disks.
Any thoughts?
Yeah.
AUDIENCE: You can look
at the previous problem
and then add a certain
number of [INAUDIBLE].
TOM LEIGHTON: Look at
the three disk problem.
That's a good idea.
In fact, you know, I could
look at the three disk problem
recursively.
I got these three disks.
Say I move them over a series
of steps, seven steps at most,
to put them here.
What would I do next?
Move the lower one and then
do this recurrent problem
again back.
All right, so we can take
a problem with n disks
and solve it recursively.
Let's see what happens
when we do that.
So let's look at a
recursive solution.
And let's draw it for n.
So the first phase
that you described
is I start, here
are my three pegs,
and I start with this,
1, 2, 3 down to end here,
and nothing on these pegs.
And I'm going to use recursion
to take the top and minus 1
and move them out here.
So I leave the nth guy there.
Nothing here.
And I go 1, 2, to n minus 1.
How many steps does that move
take, that series of moves
take, that phase?
In terms of t.
t of n minus 1, because I'm
taking an n minus 1 disk stack
and moving it.
So that is tn minus 1 steps.
All right, now my next
move is to take this guy
and stick him here.
So we'll draw that over here.
So the nth guy moves there and
I leave all these guys here.
That's going to be phase two.
I move the largest disk.
And how many steps
did that take?
One step.
Just move one disk.
One step for that.
And then my last phase
is to take all these
and move them onto here.
And so then it looks like
the final result, 1, 2,
over n minus 1 over n.
And how many steps
this phase three take?
tn minus 1.
Good.
All right, so we've
got an algorithm now,
a recursive algorithm.
And the total time,
the number of moves tn
is at most tn minus 1 plus
tn minus 1 is 2tn minus 1
plus the one step up there.
For example, which
we already know,
t3 is at most 2t2 plus 1,
which is 2 times 3 plus 1 is 7.
So we already knew we
could do it in seven.
What's an upper bound on
the time to move four disks?
15.
OK?
All right, so t4 is at most 15.
But maybe there's a better way.
This is just one way to do it.
How many people think there's a
better way than this approach?
How many people think
there is no better way?
All right, very good.
Yeah, this is the
best you can do.
And let's prove that.
Let's get a lower bound.
All right, to see why this is
optimal, think about the step
where the nth disk moves.
At some point in our procedure,
the biggest disk has to move.
Now, it could move to
this peg or that peg,
doesn't really matter.
But when it moves,
what do I know
had to have happened
before that guy moved?
Everything on top of it had
to move to the other peg.
I started with 1
through n minus 1 here.
None of them can be here.
They all have to be here.
OK?
So that means that I have to
have had tn minus 1 steps.
So at least tn minus 1 steps
before the big disk moves.
All right?
Then there is the step
when the big guy moves.
One step for the
big guy to move.
Because it's got to
move at least once.
And then after it moves, I got
to get everything else, one way
or another, on top of it.
At some point after the
last time the big guy moves,
everything else
has to go on top.
And that's another
tn minus 1 steps.
So after the last time
the big guy moves,
you've got to take
tn minus 1 steps.
OK, so that means
there's a lower bound
that no matter what you do tn
is at least 2tn minus 1 plus 1.
And so in fact, it's equal.
tn equals 2tn minus 1 plus 1.
So t3 is 7, t4 is 15.
Any questions about that?
All right, well we'd like to
know how long till the world
ends, which is t64.
So we'd like to get
a formula for t of n.
We could just compute
it 64 times and see.
But much nicer just
to get a formula.
And that's what we're going
to learn how to do this week
is to get formulas,
closed form expressions,
to solve recurrences.
And there's a bunch of
different methods you can use
and we're going to
talk about them.
The simplest is to guess the
answer and then check it.
It's called guess and
verify, also known
as the substitution method.
Now, what's a good
guess for t of n?
2 to the n minus 1.
You got t of 1 is 1, t of 2 is
3, t of 3 is 7, t of 4 is 15.
You won't compute too many
before you guess that t of n
is 2 to the n minus 1.
Now, that doesn't
mean it's true.
So how are we going
to verify that guess?
What proof technique?
Induction.
It just keeps coming back.
You just never finish
with induction.
So we're going to verify
this by induction.
What's our induction hypothesis?
Any thoughts?
Right there.
p of n is going to be that
tn equals 2 to the n minus 1.
Very typical.
So the predicate is that t
of n is 2 to the n minus 1.
Base case.
We'll pick n equals 1.
T of 1 is 1 and that
is 2 to the 1 minus 1.
So the base case works.
Now we'll do the inductive step.
We're going to assume p of n.
We're going to assume that
tn equals 2 to the n minus 1
to prove pn minus 1, which
is tn plus 1 equals 2
to the n plus 1 minus 1.
So a very standard induction.
So let's look at tn plus 1 now.
tn plus 1 equals 2n plus 1.
Yeah, 2tn plus 1.
Now we plug in here.
That's just the recurrence.
Now we plug in using our
assumption from the induction.
This is 2 to the
n minus 1 plus 1.
Multiply the 2.
That's 2 to the n plus 1.
Minus 2 plus 1 is minus 1.
And so we're done.
All right, so we
guessed the answer.
It was a pretty easy guess.
Very simple to prove
it by induction.
Any questions?
All right, how
long would it take
me to do all seven disks
on this puzzle here?
How many steps?
127 steps.
How long till the
end of the world?
A long time.
2 to the 64th minus 1 is
18 billion, billion moves.
So going to keep those monks
busy for a little while
before the world ends.
Now in general, this method
is sort of the best method
for solving a recurrence.
We'll see lots of examples
where it works great.
The only problem is it
requires that divine insight
thing, that guess to
guess the right answer.
Obviously if we guess the wrong
answer, it's not going to work.
Guessing is many times
easy, but not always.
So you need other methods
when you can't guess it.
The next most common
method has a lot of names.
It's called plug and chug,
expansion, iteration,
brute force, exhaustion.
Let me show you how that
works in this example.
OK, so plug and chug.
So we'll solve the
same recurrence.
So we've got tn
equals, and I'm going
to write it just
reversed, 1 plus 2tn.
Because I'm going to expand
out the recurrent term.
That's why it's
called expansion also.
Well, that's equal to 1 plus 2.
Now I expand this out
using the recurrence.
1 plus 2 tn minus 1.
All right, now I do a
little chugging here.
So I plugged here.
Now I'll do chug.
That's 1 plus 2
plus 4 tn minus 1.
Now I'm going to plug in again.
What's that?
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: You're right.
You're right, I've got to get
minus 1 here and minus 2 here.
That is correct.
Very good.
I would have run into trouble.
That's good.
Now I plug in here.
1 plus 2 plus 4, 1
plus 2 tn minus 3.
Then we do the chug.
1 plus 2 plus 4
plus 8 tn minus 3.
All right, and now we're
going to try to observe
the pattern we're getting.
So there's a little bit of a
guess going on in this context
still.
But you see the pattern.
We've got powers of 2.
Pretty easy pattern.
So you might say if
I keep on doing this,
I'm going to get 1
plus 2 plus 4 plus
dot dot dot plus 2 to
the i minus 1 plus 2
to the i tn minus i.
Got 2 to the 3 tn minus 3 there.
And then I take this all the way
down to i equals n minus 1, 1
plus 2 plus 4 plus 2
to the n minus 2 plus 2
to the n minus 1 t1.
And t1 is just 1.
So I'm just getting the
sum of the powers of 2
up to the n minus 1.
And we know what that is
from last time, for sure.
That's just 2 to the n minus 1.
So we've derived the answer
with only a little bit
of a-- we had to observe
the pattern here,
which is often easy to do.
But now we've
derived the answer.
Now to be really safe,
we've got to go back
and still do verify,
just to make sure
that we didn't make a mistake in
how we did the derivation here.
OK?
Any questions about
that approach?
You just keep plugging it
back into the recurrence,
look at the pattern you get, and
try to guess it from that point
and then solve.
All right, let's do a little
more interesting recurrence.
This is a fairly famous
one that comes up
in a well known algorithm for
sorting called merge sort.
This is an algorithm
you'll look at in 6046,
one of the many
sorting algorithms
you'll study when you
take an algorithms class.
So you're given n numbers and
your task is to sort them,
put them in order.
And you're allowed
to do comparisons
between the numbers.
You can think of them as
you've got a list of names,
putting them in
alphabetical order.
How many comparisons do you
have to do to make that happen?
And this is one way to do
it that's very efficient.
So to sort n items.
In fact, we're going to look
at n being a power of 2.
Call the items x1, x2, xn.
n's going to be a power of
2 to make it simple for now.
The first thing
we're going to do
is recursively
sort the first half
of the items and
the second half.
So we use recursion to short
the first n over 2 items
and the last n over 2 items.
And then we're going to
merge those two sorted lists
together.
So we're going to merge
this list and that list.
So let's see how this works and
figure out how many comparisons
we have to do to sort the list.
For example, let's say
we're sorting this list.
10, 7, 23, 5, 2, 4, 3, and 9.
So I've got eight numbers
that I'm going to sort.
So the first step is to
recursively sort that list.
And when I do that, I produce,
so I'm sorting 10, 7, 23, 5.
And that produces, let's see,
I'm going to get 5, 7, 10, 23.
And then I recursively
sort these guys.
And that produces
2, 3, 4, and 9.
I've done comparisons,
but haven't counted them
because they're
in the recursion.
Now I'm going to use
comparisons to merge the lists,
And the way I'm
going to do that is
to look at the smallest item
in each list and compare them.
Because I know the
smallest item overall
is either this one or that one.
And when I compare 2 to
5, 2's the smaller one,
so I start with 2.
And I strike it.
What two items do I compare next
to get the next smallest item?
5 and 3.
Smallest remaining guys.
I get 3.
Scratch that guy.
And I compare 5 and 4.
Get 4, scratch him.
And 5 and 9.
5 is the smallest.
7 and 9.
7.
9 and 10.
And actually now I
don't really have
to do any more comparisons.
All I've got is one list left.
So I can just send
these guys down.
And I produced a sorted list.
Now, how many comparisons do
I have to do in the worst case
if this is n over 2 items here
and this is n over 2 here?
How many comparisons
do I have to do
to produce the n item
sorted list after I
have the sub list sorted?
What is it?
n.
Very close.
n minus 1 is right,
because every time I
do a comparison I pull an
item and stick it in the list.
But once I get down
to the very last item,
just one guy sitting there, I
don't have to do a comparison.
It was just the last guy.
I know he's the largest.
So it could go up to n
minus 1 in the worst case.
In this situation, I
actually did n minus 2
because I was left with
two guys at the end,
I just bring them down.
But it might have been that
the next largest thing to 23
was here and then I would
have done n minus 1.
Any questions?
Yeah.
AUDIENCE: [INAUDIBLE].
Would that make it faster?
TOM LEIGHTON: Well,
that's what's going
to happen with the recursion.
Because the first
thing we're going to do
is when we unfold the
recursion is you're
going to take the first
half, split it in half,
recursively sort them
and then merge them.
So the answer is
that's going to happen,
but we're sort of going to
hide it by the recursion.
All right, you didn't see
me produce that sorted list.
But I would have
done it by taking
10 and 7 sorting them, 5
and 23 and sorting them
and then merging them.
OK, any other questions?
All right, well let's
set up the recursion
to count how many
comparisons are
needed for the whole thing,
not just the last merge phase.
All right, so let's define.
And instead of t sub n,
I'm going to write t of n
as a function.
Makes no difference,
but you'll see people
do recurrences both ways.
t of n or t sub n.
It's going to be the
number of comparisons.
Use my merge sort
to sort n numbers.
In the worst case, worst case.
All right, so we already
know that merging
takes n minus 1 comparisons
in the worst case.
And you always want to
prepare for the worst case.
How many comparisons are
used in the recursive
steps to sort this list and
to sort that list, in general.
tn over 2 to do
this list times 2.
Because you've got two
lists of size n over 2.
Two sub problems
besides n over 2.
Each takes t of n over 2.
All right, so you've got
2 tn over 2 comparisons
for the recursive sorting.
That means that t of n equals
2 t of n over 2 plus n minus 1.
And we should say
what t of 1 is.
What's that?
What's the time to sort?
0.
You don't do any comparisons
if there's one item.
So we'll set that
as our base case.
All right, so let's
find the answer.
So the first thing to always
do when you see a recurrence
is to take a few values,
compute a few values,
and try to guess the answer.
So what's t of 2?
1.
Plug it in.
I get 2t of 1, t of
1's 0, plus 2 minus 1.
That's 1.
And we're only doing
powers of 4 here.
Sorry, powers of 2.
So I go to t of 4 next.
What's t of 4?
5.
Right, because I got double
this is 2 plus 4 minus 1.
That's adding 3.
So I get 5.
All right, let's look at t of 8.
That's twice 5 plus 8 minus 1.
17.
Not so clear what to guess here.
We can try one more.
T of 16 equals 2 times
17 plus 16 minus 1
equals 34 and 15 is 49.
Is that right?
34 and 15.
Any guesses?
You're not going to guess this.
That's not going to work.
That happens.
And a miracle could happen,
you get a divine insight
and guess it, but I
think it's not likely.
All right, so let's
go to plug and chug.
See if that works.
All right, so now plug and chug.
All right, so we
write down t of n
is-- and you always write their
current part last for plug
and chug to keep it simpler.
2 t of n over 2.
And then I substitute, I
plug in for t of n over 2.
So I have n minus 1
plus 2 n over 2 minus 1
plus 2 t of n over 4.
Then I chug.
So I have n minus 1
plus that's n minus 2.
Plus 4 tn over 4.
All right, then
we'll plug in here.
And I get n over 4 minus
1 plus 4 tn over 8.
And now we chug.
That's an n here minus a 4.
I did something wrong.
That should be 2.
4 times 2 is 8 tn over 8.
All right, can you start to
guess the pattern here maybe?
We probably should do
one more to be safe.
But you got an n minus 1 plus
an n minus 2 plus an n minus 4.
And if we do one more, what's
the next term going to be?
n minus 8.
And this is looking
pretty simple.
So we could guess now that
the pattern is n minus 1
plus n minus 2 plus
n minus 4 all the way
out to plus n minus
2 to the i minus 1
plus 2 to the i t
n over 2 to the i.
And now we go all
the way down where
i equals log n, base
2, and see what we get.
1 plus n minus 2 plus all
the way down n minus 2
to the log n minus 1 plus
2 to the log n t of 1.
What's t of 1?
0.
That's nice.
That goes away.
All right, so this is
not looking so bad.
This equals the sum i equals 0
to log n minus 1 of n minus 2
to the i.
All right, and this
will be good practice
maybe for towards the test.
Well, I can split that sum up.
i equals 0 to log n minus 1
of n minus the i equals 0 log
n minus 1 of 2 to the i.
Well, this is easy.
I'm just adding n up a
total of log n times.
So I have n log n.
This is just something
the powers of 2
which we know how to do.
So that's going to be
2 to the log n minus 1.
And so we get our answer
is n log n minus 2
the log n is just n
minus 1 is a plus 1.
Well, we derived an answer.
Let's just go back and
check some of the values.
n equals 1.
That's 0 and I get 1 minus 1.
It works.
n equals 2.
2 log 2 is 2 minus 2 plus 1.
Works.
Plug in 4, I get 4 log 4 is
8 minus 4 is 4 plus 1 is 5.
So it works for those examples.
Now, to be really careful we
should prove it by induction.
Because when you do
enough of these equations,
you make mistakes like I
invariably do even up here.
And even though it's
written down here,
I'll make mistakes
writing it on the board.
And so you want to
check by induction.
OK, so this is the answer.
This is how many comparisons
are done in merge sort.
And the nice thing is this
is growing nearly linearly.
It grows as n log n.
And that's why this algorithm
is used a lot in practice.
Much better than
comparing every item
to every other item, which would
be more like n squared steps.
So much better than a
naive sorting algorithm.
Any questions about
what we did there?
Just one rule of thumb when
you're doing the plug and chug.
Notice that I didn't try to
collapse terms along the way.
It makes it easier sometimes
to see the pattern.
I left it as n minus 1 plus n
minus 2 and then plus n minus 4
rather than saying,
oh, this is 3 n
minus 7, which might have made
it harder to guess the pattern.
So you don't always, when you're
doing the chugging, collapse
everything down.
You want to try and
see if there are some.
Because this is the amount
of work done at each step,
and you want to see if you
can see a pattern in that.
All right, so let's write
down our two recurrences
and sort of compare them.
So first we did Towers of
Hanoi, which was t of n
equals 2 t of n minus 1 plus 1.
And the answer was t of
n is tilde 2 to the n.
It was actually 2
to the n minus 1.
Then we did merge sort, which is
t of n equals 2 t of n over 2.
Difference there.
Plus n minus 1.
And now the answer was t of n.
What's the tilde for this one?
For merge sort.
n log n.
Yeah, I can forget the rest
if I know the tilde, n log n.
Wow.
These are pretty
different answers, right?
This one's nearly linear,
that one's exponential.
The recurrences
looks pretty similar,
in fact, this one
sort of looks worse
because I'm adding in n minus
1 every time instead of 1.
But the answer's a lot
better, a lot smaller.
What's the critical difference
between these recurrences?
Yeah?
AUDIENCE: You're dropping the
argument of t a lot faster.
You're cutting it
in half every time
instead of subtracting one.
TOM LEIGHTON: Right.
AUDIENCE: So it's going
to go away faster.
TOM LEIGHTON: That's right.
Here I've got two
problems of size one less.
Here I've got two
problems of half a size.
And that makes an enormous
difference in the answer.
Even though I'm
doing more work after
did the recursive problems,
much faster running
time or answer if these are
algorithms that I'm analyzing.
All right?
In fact, let's
take a look at what
happens if I didn't use that n
minus 1, I just put a 1 there.
Let's see what impact
that n minus 1 had.
So let's do the
following recurrence.
This one is going to be a little
bit different because I'm not
going to stick to powers of 2.
I'm going to get a little
messier, because that often
happens in reality.
And I'm going to
define s of 1 is 0.
s of n is s of the floor
function of n over 2
plus s of the ceiling
function of n over 2 plus 1.
Now, the ceiling function
means the smallest integer
that's at least that big.
So you're sort of rounding
up to the next integer.
And the floor function
means you're rounding down
to the next integer.
So it's the biggest integer
less than equal to n over 2.
All right.
Now, if we had just
powers of 2, this
would be much easier to write.
It would just be sn
equals 2 sn over 2 plus 1.
So it's analogous to what we're
doing up there where we're just
adding 1 instead of n minus 1.
But I'm being careful. now.
I'm going to make
this recurrence
work for all natural numbers.
Because sometimes
in reality, you're
dealing with non powers of 2.
All right, let's see
if we can solve this.
What's the first thing
to try to solve this?
What do you do?
What's the first step
when you get a recurrence?
You try to solve.
What method should you try?
Guess.
And to guess, well, you
got to plug some values in.
It's sort of hard to guess.
I mean, looking at that, I guess
nothing comes to mind other
than panic looking at that.
But if we plug in some values,
then maybe it won't be so bad.
So we've got s1 equals 0.
What's s2?
It's not 3, because this is 2
over 2 is 1, floor of 1 is 1.
s of 1 is 0.
Ceiling of 1 is 1
because it's an integer.
s of 1 is 0 plus 1.
So s of 2 is 1.
What's s of 3?
Yikes, all right.
3 over 2 is 1 and 1/2.
What's the floor of 1 and 1/2?
1.
s of 1 is 0.
That's gone.
What's the ceiling of 1 and 1/2?
2.
s of 2 is 1 plus 1.
What's the answer for s of 3?
2, because I've got 0, 1, and 1.
s of 4.
This one's easy
because it's even.
I get s of 2, which is 1 plus
s of two which is 1 plus 1.
So what's s of 4?
3.
You might try a guess.
What would you guess?
Yeah, n minus 1.
So we're going to guess
s of n equals n minus 1.
And that's not so bad, actually.
It's pretty simple.
Looking at that mess, you'd
never think of that right away.
But once you plug in some
values it's not so bad.
All right, let's verify it.
And we're going to verify
by strong induction.
The induction
hypothesis is our guess.
The base case is easy.
Because I've got s of 1 is 0.
And that equals 1 minus 1.
So that's good.
So let's do the induction step.
All right, so the
induction step.
We're going to assume that
p1 is true, p2 is true,
all the way up to pn, to
prove pn plus 1 is true.
For n better than or equal to 1.
And then we look at
the n plus first term.
That equals s of floor n plus
1 over 2 plus s of ceiling n
plus 1 over 2 plus 1.
All right, now I don't know what
these are exactly depending on
whether n is even a rod.
First let me plug in using
the induction hypothesis.
I know that this is n
plus 1 over 2 minus 1.
This is ceiling n
plus 1 over 2 minus 1.
And then I carry
the plus 1 down.
That's by the
induction hypothesis.
All right, now I don't
know exactly what
either of these values is
unless I know the parity of n.
But I do know the sum
of these two things.
The floor of something plus
the ceiling of that something
is twice the something.
All right, so this plus
that equals n plus 1.
Minus 1 minus 1
plus 1 is minus 1.
And so in fact, we
prove that s of n plus 1
equals n plus 1 minus 1.
So we verified the
induction hypothesis.
Yeah?
AUDIENCE: [INAUDIBLE].
Taking the n [INAUDIBLE].
TOM LEIGHTON: Let's
see, I'm not sure.
So are you asking why I
started sn plus 1 here?
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: Doesn't matter.
But the expression
is s of whatever
this is, the induction
hypothesis would be,
s of whatever's inside there
is what's inside there minus 1.
And so the induction hypothesis
says s of this is that minus 1.
Any other questions?
So let's write that
on our table up here.
So this is sn equals
s of floor n over 2
plus s of ceiling
n over 2 plus 1.
And the answer is, sn is tilde.
What's the tilde of?
n.
The answer is n minus 1.
So it's tilde of n.
Now, this is the same expression
is this if n's a power of 2.
I'd have 2 sn over 2 plus
1 instead of n minus 1.
And by going from n minus 1 to
1, I got rid of the log term.
But it's still not exponential.
And that's because I'm
cutting down the insides
here by a factor of 2 roughly.
All right, so all the
recurrences we've seen so far
have been pretty simple.
If guess and verify doesn't
work after seeing a few terms,
you can use plug and chug
and you can figure it out.
The good news is, that
happens a lot in practice.
The bad news is, sometimes
it doesn't and you
can get nasty recurrences.
So let me show you a nasty
one that actually was once
asked in 6046, I think, on a
quiz with disastrous results.
All right, so this one is t
of x equals 2 t of x over 2
plus 8/9 t of 3x
over 4 plus x squared
for x bigger and equal
to 1, and otherwise it's
0 when x is less than 1.
Now, this is nasty
for a lot of reasons.
First, it's not just a
find on the integers.
We can make an integer version
by putting floors or ceilings
here.
But the way you set it up now,
it's defined on real numbers.
You can, in theory, solve
it with plug and chug.
The nasty part is, you've got
a bunch of things going down
by powers of 2 and a bunch of
things going down by 3/4 here.
And so you have to keep
track of all the terms
that are decreasing, and that's
a nightmare plug and chug.
Because it's not just one
thing that's unfolding,
it's two things
that are unfolding.
So very painful.
Now, it used to be that
was the state of life.
20 years ago, I think we started
teaching this course sometime
around then, there
just was nothing else
to say other than life's
tough and sometimes you've
got to do these things
and it's a pain.
Just had to make the best of it.
And then in the mid-90s,
one day two students
at the American University
in Beirut came by my office.
They were named Akra and Bazzi.
And they'd actually
come to Boston
to escape the daily bombings.
This was when all heck was
breaking loose in Beirut.
It was a mess.
And they come into
my office and they
claim to have a simple
formula for solving
any recurrence like this.
In fact, you can even
add a 5 t of x over 7.
They said no problem.
Their simple formula
just will always work.
And the general form
that they claim to solve
was the following.
It's the general class of
divide and conquer recurrences.
So any recurrence you
get from an algorithm
where you break it into smaller
pieces by a constant factor,
they claim to solve.
So let me define this class.
It's a little painful.
So a divide and conquer
recurrence has the form.
There's a simple version
and the hard version.
The simple version is
it looks like that.
You got a bunch of terms.
Inside each term, you cut
it down by a constant factor
and then you add
something at the end.
That's the simple version
of what a divide and conquer
recurrence is.
And you can throw in floors and
ceilings and other junk too.
Now I'm going to give
you the formal version.
So it has the form t of x
equals some constant a1 times
t of a constant less than
b1 of x, plus some slop
you're allowed to have
like floors and ceilings
and who knows what else.
And then you get more of these.
You get a2, t of b2x plus
epsilon 2x and so forth.
And you can have
any number of these
you want up to a constant.
So maybe there's a sub k t
b sub kx plus epsilon kx.
Plus some function of x sitting
out here, like the x squared.
And this happens for x
bigger than some fixed value,
some constant.
Now, the constraints are,
well, the ai's are positive
constants.
The bi's are positive
and less than 1.
You gotta cut it down by a
constant factor to be a divide
and conquer recurrence.
That's critical.
k is fixed.
It's constant.
And these epsilon
functions can't be too big.
They can't be bigger than 0
of x over log squared of x.
Almost as big as x, but you
can't let them be too big
or you'll not be cutting things
down by a constant factor.
Now, you don't get
too hung up over this
because you don't get
too deep in the details.
But this is that most 0 of
x over log squared of x.
And finally, the g thing
has got to be polynomial.
It can't be an exponent.
You can't put 2 to the x here.
And the formal way
that's expressed
is the derivative of
g of x is less than x
to the c for some constant c.
That's it.
So this is a mouthful.
But like I said before,
anything that looks
like that, basically.
As long as you don't
throw in exponentials
or do something too
wild to it, it works.
So let's see which of our
guys over here are divide
and conquer recurrences.
How about this one?
Towers of Hanoi, is that a
divide and conquer recurrence?
Some yes, some no.
Who thinks no can
tell me why it's not?
Why isn't that?
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: That's right.
The recurrence
problem is n minus 1
and we've got to make it down
by a constant factor here.
The b1 is less than 1.
It can't be equal to 1.
And this is not big enough
to make up for the x over 2
term or something we're missing.
So whenever you have an
n minus 1 and minus 2,
that is not divide and conquer.
We'll discuss those
next time, on Thursday,
how to solve those.
So that's not
divide and conquer.
What about this one?
Is that divide and conquer here?
Yes.
You've got two problems
of half the size
and you're adding
a linear function.
It's not exponential.
It's polynomial.
That is divide and conquer.
What about this nasty guy?
Is that divide and conquer?
Yeah.
You've got basically
n over 2 here.
Two n over 2 terms now.
This is where that epsilon
thing comes in handy.
Because epsilon 1
of n, in this case,
is floor n over
2 minus n over 2.
And that's, at most, 1.
So maybe we're adding a half
here or something like that.
And we're allowed
to add anything up
to n over log
squared of n to fit.
So you can mess around with
the low order terms in here.
And adding 1 is fine as well.
So that qualifies.
Let's check that
this thing qualifies.
Let's check that.
Well, yeah.
I got 2 constant a1 t of 1/2 x.
That's OK.
8/9 t of 3/4 x.
That's OK.
My g function is a polynomial.
The derivative is 2x.
That's less than x to a
power or of x to a power.
So this fits.
Any questions about
the definition
of a divide and
conquer recurrence?
And you can see why
sort of they come up,
because whenever you're solving
a problem with recursive pieces
that are a constant
factor smaller,
you're going to have something
that looks like that.
OK, so getting back
to Akra and Bazzi.
B They show up in my office.
They claim to have solved
anything like this.
Now needless to say,
I'm a little skeptical.
I got to tell you, you're a
professor at MIT, especially
mathematics, and
people are always
coming by with proofs of
the Riemann hypothesis,
p equals np, p doesn't equal np.
And this was a classic problem.
Hundreds of
researchers had studied
recurrences for decades.
And not only, it
wasn't even believed
that there was
possibly a solution.
It's not like it's one of these
open questions where we all
believe p doesn't
equal np and it's just
a matter of proving it.
People didn't think a solution
existed that would be clean.
I mean, look at just
the definition's a mess.
And how could there possibly
be a clean solution to that?
So it's sort of
like the holy grail.
It doesn't exist, but
it's fun to think about,
that fantasy, maybe there's a
solution and you could find it.
And much less, these aren't
reputable researchers
in my office.
These are two
students from Beirut
named Akra and Bazzi
claiming they've figured out
this thing that all these
professors been teaching
this stuff in 6046 for
years said you've got to do
plug and chug.
Anyway, they'd been kicked out
of everybody else's office,
so I said, OK, let's do it.
Show me the solution.
And they did.
And I go, holy cow.
It's right.
It looks good.
Now, they had a
few details wrong,
but my goodness, the
answer was correct
and it's amazingly simple.
So now we teach it.
And in fact, you can
write the solution
down easier than you can
write the definition of divide
and conquer recurrence.
Really remarkable.
So the theorem by
Akra and Bazzi in '96.
Set p, real number, So that
the sum i equals 1 to k
of ai times bi to
the p equals 1.
So the ai's are
these things here.
The bi's are the fractions here.
All right, find a p
such that this sum is 1.
Then the answer is this.
Then t of x equals theta.
We're going to use our
asymptotic notation.
x to the p plus x to
the p times the integral
from 1 to x g of u.
G is that polynomial thing,
that thing at the end.
Over u to the p minus 1 du.
That's it.
This is really great
news for you guys,
especially if you
go on to algorithms,
because you don't
have to deal with plug
and chug on nasty things.
You just solve the answer.
Now, the proof is
by guess and verify.
And they made a
really good guess.
You wouldn't think of
looking at that, oh,
I'm going to guess this.
Probably not.
And there's no real numbers
to try to guess this.
It's asymptotic notation.
But it works.
You can verify it by induction.
It's a little painful
to verify by induction,
but you can do it.
We won't do it in class.
I did try one year,
and it was a disaster,
so we don't do it
in class anymore.
But let's see some examples
of how to apply this.
So let's start with merge sort,
which is the second one here.
Let's do that.
All right, so you
got an example here.
t of x equals 2 t of x over 2
plus n minus 1, or x minus 1, x
now.
And I'm not even going to worry
about powers of 2 anymore.
I don't care.
It's going to be fine.
All right, what
is p in this case?
I've only got one term.
I don't even need a sum from
that definition over there.
What's the value
of p for this guy?
So I've got a1 is 2.
b1 is 1/2.
Just one term.
What's the value of p such that
2 times 1/2 to the p equals 1?
1.
Doesn't get much simpler.
So 2 times 1/2 to the p
equals 1 implies p is 1.
All right, now I just plug
into the integral over there.
All right, let's do that.
So t of x equals theta.
Well, x to the p is x to
the 1 plus just take off
the 1, plus x times
the integral 1 to x.
What's the g function?
What's g of x in this case?
x minus 1.
So I take the integral of
g of u is just u minus 1
over u to the p
plus 1 p minus 1.
Did I write the formula right?
Is it plus 1 or minus 1 here?
Did I get it?
p plus 1.
Sorry, didn't write that right.
p plus 1 over here.
All right, so u to the p
plus 1 is u squared du.
This is theta of x plus x
integral 1 over u minus 1
over u squared.
The integral of 1 over
u is the log of u.
The integral of 1 over
u squared is 1 over u,
but I changed the sign there.
1 to x.
All right, so I get theta of
x plus x times log of x plus 1
over x minus log of 0.
That's log of 1 is nothing.
Minus 1.
And I'm doing thetas here so I
can forget the low order terms.
So I get x log x plus 1 minus x.
This is the only term
that survives, x log x.
And that's right.
t of n is in fact tilde n log n,
so in fact, it's theta n log n.
Now, in this case, going through
the integral and all that,
maybe that was harder than
just guessing and verifying.
And by the way, guess and verify
got it really the exact answer.
The nice thing is this works
for the nasty guys too.
In fact, let's figure
out the solution
to this thing, which I
guarantee you is a pain
to do without this method.
The first thing to
do is compute p.
So to do that, we need to
find a p such that 2 times 1/2
to the p plus 8/9 times
3/4 to the p equals 1.
I gotta find a p for
which that's true.
Any guesses what p works?
p equals 1 not going to do it.
p equals 2?
Let's try that.
I'd have 2 times 1/4
plus 8/9 times 9/16.
That looks pretty good.
I get 1/2 plus 1/2 equals 1.
So p equals 2 works.
All right, so let's
do the integral.
tx is theta of x to the p plus
x to the p times the integral.
All right, what is g
of x in this thing?
x squared.
So the integral
is 1 to x. g of u
is u squared divided
by u to the 1 plus p.
u to the 1 plus 2
is just u cubed.
This is theta x squared
plus x squared times, well,
the integral of 1 over
u is just log of you.
And that's just log of x.
So the answer is,
x square ln of x.
Done.
So that's pretty good
what these guys did.
In fact, that is
the correct answer.
So really easy to pug in.
Any questions on that?
Yeah.
AUDIENCE: [INAUDIBLE] or does
it give the actual number?
TOM LEIGHTON: It doesn't
give you the actual number.
It just gives you the
asymptotic growth.
All you know is that the limit
of tx over x squared ln of x
is less than infinity
and bigger than 0.
So it's growing as this.
You don't know if
it's 10x squared lnx
or a million squared lnx.
On the other hand,
for algorithms usually
you're forgetting the constant
factors anyway because they
depend on things
to do with the CPU
you're using and
stuff like that.
What you really
care about, often,
when you're studying algorithms
is the asymptotic growth, not
the actual value.
Yeah?
AUDIENCE: [INAUDIBLE]
log base e.
And then above I
see you using log,
which I assume is base two?
Does it matter?
TOM LEIGHTON: It does up there
because I've got a tilde.
And in the tilde, the
constant factor matters.
It doesn't matter
in here because log
base two and log base E are
within a constant factor,
namely log of 2, ln of 2
is the constant factor.
So logs don't
matter what the base
is once you're inside a theta
or an O or that kind of thing.
In a tilde, they matter because
the constant factor matters.
Yeah.
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: Yeah.
Because x squared
is small compared
to x squared ln of x because
it's all by the theta.
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: No.
You could put all sorts
of other stuff in here
and it doesn't matter
because it still equals
theta of x squared ln of x.
There's no hidden meanings by
having extra stuff in here.
That's important.
In this case, when you
put more stuff into theta,
doesn't mean a thing.
Any other questions?
Yeah.
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: Ah, good question.
Well, let's see.
It goes back to over here.
See, the b's are
between 0 and 1.
By making p big enough,
I drive the value down.
By making p be large and
negative, I drag the value up.
So there is some value of p.
But that's a good question.
Let's see what happens
when p is not so nice.
Because the examples I gave
you, in one case p was 1
and one p was 2, sort of nice.
Let's look at a bad case there.
Let's look at this recurrence.
3 t of x over 3 plus 4 t
of x over 4 plus x squared.
All right, so that's
the recurrence.
And first step is to compute p.
And this time it's
not going to be nice.
So I need to find a p such that
3 times 1/3 of the p plus 4
times 1/4 to the p equals 1.
And you might say, OK,
let's try p equals 1.
I get 1 plus 1 is 2.
That's too big.
So which way do I
have to go on p?
Do I need a bigger
p or a smaller p?
Bigger.
Because I got to get
from 2 down to 1.
So I know that p
is bigger than 1.
All right, so the
next thing to try
would be hoping the world
is nice, p equals 2.
Maybe I'll do that over here.
Let's try p equals 2.
So I get three times
1/9 plus 4 times 1/16
and that equals a 1/3 plus 1/4.
That's less than 1.
That didn't work.
What do I have to do now with p?
Is it bigger than
2 or less than 2?
Less than 2.
That's sort of a pain.
P is between one and two,
so it's not an integer.
Now, you can sort of do
divide and conquer here.
Next I'd try p is 3/2,
get my calculator out
and see which way it goes.
And you could sort
of home in on it.
But there's
something really nice
that happens with Akra Bazzi.
And that is if you
ever conclude that p
is less than your
exponent here, you
don't even have to compute it.
Let's see why.
Let's do an example.
This will happen a lot.
There's some p, but it's just
going to be a pain to evaluate.
But let's see what happens here.
So I'm just going
to try to solve
it, write the solution
anyway. t of x
equals theta x to the p plus
x to the p integral 1 to x.
What's g of x?
x squared.
So that's u squared
over u 1 plus p.
And I still don't
know what it is,
but I'm going to
give it a try here.
That's theta x to the
p plus x to the p.
Well, this is now the integral
of u to the 1 minus p du.
And p is bigger than 1.
I know that.
This is theta x to
the p plus x to the p.
Well, the integral here
is x to the 2 minus p.
Up to constants.
I don't even care about
dividing by 2 minus p,
because I got big theta.
Just put that p up here.
OK
Well, this equals theta x to
the p plus cancelled there,
x squared.
And what's the
answer going to be?
x squared, because
p is less than 2.
So this is a small order term.
So I didn't even
know what p was, just
that it was less than two.
And I got the answer.
And what do you know?
The answer was just this thing.
That's sort of nice and simple.
And in fact, that's a theorem.
Which will state, in general, if
g of x equals theta x to the t,
for some t bigger
and equal to 0.
And when you plug that
value in and take your sum.
Trying to compute the p value.
So I take ai times bi to the t.
And that comes out
to be less than 1
like it did here when I plugged
in p equals 2 and I computed it
and it was less than 1.
But if that happens in
general, then the answer
is theta g of x.
Plain and simple.
It just turns out to be the g of
x term up to constant factors.
So you don't even
have to compute it.
Just do that check on
the power here to see
is it smaller than that.
I won't prove that there.
It's actually not hard to do.
The proof is pretty much
what I did with this example,
so it's not too hard.
OK, any questions?
I want to show you
one more thing,
but just make sure that's OK.
And this is something never to
do with an asymptotic notation,
like we talked about last time.
When you're using Akra Bazzi and
when you're doing occurrences
with algorithm, you always
are using asymptotic notation.
Your answer is
theta of something.
And in fact, what will
start happening is
you'll do this stuff because the
constant factors don't matter.
Because you're wiping them
out in the theta notation,
what you'll see is
things like this.
Your recurrence will be set up
to be t of n is 2 t of n over 2
plus o of n or plus theta of n.
And then you'll
conclude that t of n
is theta of n log
n using Akra Bazzi
or whatever method you want.
Because it doesn't
really matter what
the constant is on g because
the constants disappear.
Doesn't matter.
And so people stop
worrying about it early on.
All right, now here's
the bad example.
Let me show you a false
proof like last time.
And this time I think it'll be
pretty easy to spot the flaw.
Theorem not.
If t of n equals 2 t of
n over 2 plus n minus 1,
this is the recurrence we had
for merge sort, and t of 1
equals 0, then I'm going to
prove that t of n is o event.
Now, that can't be right
because we've proved it's theta
then log n, tilde then log n.
And that is not o of n, right?
n log n grows faster than n.
But let's see the proof.
By strong induction.
The induction
hypothesis is going
to be what we're
trying to prove.
Base case.
n equals 1.
t of 1 is 0.
0 is surely o of 1.
The induction step.
We're going to assume p1, p2,
p3 up to pn minus 1 to prove pn.
And let's look at p of n.
So I look at tn is 2 t of
n over 2 plus n minus 1.
Induction hypothesis
on n over 2 says
this is 2 times o of n
over 2 plus n minus 1.
Twice o of n over 2 is
o of n plus n is o of n.
This is o of n.
And I'm done.
I just proved that
t of n is o of n.
What's the bad step?
What's the bug?
See the bug?
What is it?
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON:
Induction hypothesis.
What's wrong with that?
AUDIENCE: [INAUDIBLE].
TOM LEIGHTON: Right.
The end from the predicate
cannot be in the o of n.
Because remember, o of n means,
this statement here by itself
means the limit as n goes
to infinity of t of n over n
is less than infinity.
But this makes no sense if I
specified n in a predicate.
So what's the rule?
Same rule as last time.
Same bug.
What's the rule?
Never do asymptotic notation
in induction hypothesis.
Never mix big o and a predicate.
Never, ever.
It looks so nice to
do, but it is so wrong.
It just makes nonsense.
All right, very good.
So remember ice cream tonight,
recitation optional tomorrow.
But study session is there.
