Next, we're going to take a moment to 
describe in detail the scientific 
approach that we use in modern analysis 
of algorithms. 
so again, just a note about notation. 
in the theory of algorithms, when they're 
looking at upper bounds on worst case 
performance they use these notations big 
O, big omega, and big theta to try to 
capture the order of growth, and that's 
what they use for classifying algorithms. 
if g of N is big O of f of N, it means 
that the ratio of g of N to f of N is 
bounded from above as N goes to infinity. 
If it's omega, it means it's bounded from 
below. 
and if it's theta, it means that it's 
bounded from both above and below. 
so there's, so this one says, there's a 
constant such that g of N is less than 
that constant of f f of N. 
this one says, there's two constants 
that's it's in between. 
so that allows classification according 
to functional growth. 
as I mentioned mergesort is N log N and 
quicksort is N squared. 
so that's the, the notation that you 
often see throughout the literature 
describing the performance of algorithms. 
But as I mentioned, big O notation is 
dangerous. 
And I'll have more to say about that in 
just a minute. 
so, you can't, it's not scientific to use 
the big O notation to try to compare 
algorithms. 
you can't say, if you say the running 
time is big O of N to the c, that's not 
of any use for predicting performance. 
It's an upper bound on the worst case. 
the actual performance may be much better 
than the worst case. 
and it could be that the even the actual 
bound is less than what's given by the 
big O notation. 
It's fine for a first cut of classifying 
algorithms, but not useful for comparing. 
what we use instead is what's called the 
tilde notation. 
and so what we'll typically say is that 
the running time of an algorithm is 
tilde, a constant times say some function 
of N where N is the input size. 
That does provide an effective path for 
prediction performance. 
And I'll show some examples of that later 
on. 
So, we don't use the common big O, big 
theta omega notation very much except in 
a specific technical sense that I'll talk 
about later on when we talk about 
asymptotic approximations. 
So, [COUGH] big O notation is useful for 
a lot of reasons and it dates back a few 
centuries and, and we do use it in math. 
But it's a common error to think that big 
O notation is useful for predicting 
performance. 
and I just want to make sure to nip that 
problem in the bag right away. 
This is what often happens to me when I 
give talks around the world on this 
topic, typical exchange and say that Q 
and A for my talk, 
depending on how formal it is to 
somebody. 
I'll, I'll say, okay, big O notation is 
dangerous. 
You can't use it to predict performance. 
and somebody will ask or shout out but 
an algorithm that is big O of N log N 
surely beats one that's big O of N2. 
squared. 
And then, I trot out, say, the quicksort, 
mergesort example and say, well, not by 
the definition, big O is just an upper 
bound on the worst case. 
And they'll say, well, so use the theta 
notation which says that it's in between. 
and I say well, that maybe gets rid of 
the upper bound. 
But you're still typically bounding the 
worst case. 
is your input a worst case? 
and even with that logic, and with a 
compelling example like quicksort versus 
mergesort, 
usually what happens is the questioner 
whispers to one of his colleagues. 
Well, I'd still use the N log N 
algorithm, wouldn't you? 
[LAUGH] and actually such people usually 
don't program much. 
and shouldn't be recommending what 
practitioners do. 
but surely, we can do better than this. 
that's part of what analytic 
combinatorics is all about. 
There's another idea that's out there as 
well, and that's the concept of a 
galactic algorithm. 
and this, I found this on a friend's blog 
Dick Lipton, who said let's define a 
algorithm that will never be used as 
being galactic and why will it never be 
used? 
Because no one would ever notice any 
affect about this algorithm in this 
galaxy. 
because any savings is going to happen 
for input size so large that it couldn't, 
it couldn't happen in this galaxy. 
So, an example of galactic algorithm and 
this one is actually maybe planetary or 
something, it's actually close to the 
real world, 
is Chazelle's linear time triangulation 
algorithm. 
So, the problem is to find a way to 
triangulate a polygon in linear time. 
it was a surprising result and a 
theoretical tour de force to prove that 
it was possible to solve this problem in 
linear time. 
But the method is much too complicated 
for anyone to implement. 
And if anyone did implement it, the, the 
cost of the implementation would 
definitely exceed any savings until N is 
so hard, large that it would take another 
galaxy to deal with it. 
And 
[COUGH] this is you know, an interesting 
situation. 
I think one of the problems is that so 
many algorithms that are out there, that 
are being published in the literature, in 
this category after Lipton introduced the 
concept one of the contributors to his 
blog estimated that something like 75 to 
95% of the papers in the theoretical 
Computer Science conferences are 
galactic. 
and I think the problem is that 
practitioners aren't necessarily aware 
that they're galactic, and they maybe try 
to use them. 
when a relatively simple analysis would 
say there's no point in a practitioner 
taking a look at this the papers should 
have asterisks on them or something. 
So, I, I think it's okay for basic 
research to drive the agenda and there's 
nothing wrong with trying to find the 
algorithm with the best upper bound and 
worst case performance but we have to do 
something about the common error where 
people think that a galactic algorithm is 
actually useful in practice. 
There's a lot of denial out there. 
Now, here's another thing that often 
happens to me. 
This was an actual exchange with a very 
prominent theoretical computer scientist 
a few years ago. 
and he said in a talk, well the algorithm 
A, that actually is a pretty 
straightforward algorithm that's in 
widespread use, is a bad algorithm. 
Google and other internet providers 
should be interested in my new algorithm, 
algorithm B. 
[COUGH] 
and so, in the question and answer I 
said, well, what's the matter with 
algorithm A? 
and, and, and he responded, it's not 
optimal. it's running time has an extra 
log log N factor. 
and that's always a tip off for me. I 
say, well, but your algorithm is very 
complicated, it takes ten pages to 
describe. by the way we all know that log 
log N is less than six in this universe, 
that's two to the 64th. 
and, and so if N is to the 64th, it's 
less than six. 
And, and so, not only that it's just an 
upper bound. 
Not only that your algorithm is so 
complicated. 
It's certain to run ten to a hundred 
times faster in any conceivable real 
world situation. 
Why should Google care about algorithm B, 
as you said? 
and then the response was, well, I like 
algorithm B. 
I don't care about Google. 
and again, that's fine to do research for 
the intellectual challenge. 
just don't say that some practitioner 
should be interested in it. 
Surely we can do better than that. 
So, what I want to talk about 
specifically is the scientific approach, 
say, the modern rendition of what Knuth 
taught us that is used for many, many 
algorithms in the fourth edition of my 
algorithms book which is co-authored with 
Kevin Wayne. 
so, this is described in a lot of detail 
with examples in Section 1.4 of the book. 
Again, as Knuth said we start with a 
complete implementation that we can test. 
and then therefore we'll be able to make 
hypotheses about the performance of that 
implementation and test them. 
And then, what we're going to do is maybe 
try to avoid some of the detail in Knuth. 
And we're going to analyze the algorithm 
by trying to find an abstract operation 
that's in the inner loop. 
That is, it's get executed more times 
than any other operations. 
and then, we'll also need to develop some 
realistic model for the input to the 
program. 
That's still a sticking point. 
and then, we'll just analyze the 
frequency of execution of that one 
operation for input size N. 
and our hypothesis will be that the 
actual running time is proportional to a 
constant times that frequency. 
That's it. 
so the hypothesis might be wrong but we 
can test it. 
and we have the tilde operation and I'll 
show you specifically how we can go ahead 
and test it and actually the unknown 
constant is an annoying thing care 
around, carry around but not actually too 
bad because it does allow us to make 
specific mathematical calculations. 
So, 
what we'll do then is once we have the 
hypothesis, we're going to validate the 
hypothesis by, first of all, we want to 
generate some large inputs according to 
the model. 
So, for example, we'll look at sorting 
algorithms where the model is that the 
things are randomly ordered and distinct. 
And so, it's easy to write a program to 
generate large randomly ordered distinct 
files. 
and then, we can just run the program for 
large input to calculate A. 
And actually, we don't, we can even skip 
that step. 
I'll show you that in a minute. 
But we definitely can get A that way an 
estimate of A. 
And then that give us a model that we can 
use to predict performance for even 
larger inputs and check our hypothesis. 
that's the scientific approach to the 
analysis of algorithms. 
So really, then, later, what we need to 
do also is find an application that 
actually uses real world data in testing 
application context to validate the model 
as well. 
that's actually the hardest part that's 
often overlooked nowadays. 
and then, as in any application 
scientific method we'll refine and repeat 
and revise of the model and the algorithm 
as necessary or as we learn much about 
the problem. 
As I mentioned earlier, one of the great 
things that happens with this process is 
that often we learn things about the 
algorithm that allow us, enable us, to 
develop improved versions by doing this. 
And we have many, many examples of this 
for sorting and searching algorithms, and 
graph algorithms, and string processing 
and data compression and many, many other 
applications where we successfully apply 
this approach to develop good 
implementations and understand their 
performance. 
so that's now, what this course is going 
to be all about is this analysis of the 
frequency execution. 
That's where the math is. 
and so, but I want to give this full 
context so people can understand why 
we're going to this trouble. 
So as I mentioned, we don't use the, the 
big O notation, or omega or theta that 
much. 
In, 
instead that's for the theory of 
algorithms. 
Instead we're going to use the tilde 
notation and all the tilde notation means 
is that two functions g of N is tilde f 
of N if their ratio approaches one as N 
approaches infinity. And usually we have 
f of N is a constant times A standard 
function of N. 
so that's the notation. 
So we're just using tilde and that's 
good. 
that simplifies things a bit. 
and then, we're just left with these 
basic components. 
So, one of the basic components of 
algorithm analysis involves programming. 
So 
people who do analysis with algorithms 
need to be comfortable with implementing 
algorithms, running them and, and be able 
to at least count operations. 
so you need a good implementation. 
Now, maybe someone did, did the 
implementation but still you want to have 
experience with it on your own computer 
to test various hypotheses that you might 
have. 
then we got the mathematical challenge 
and that's again going to be most of our 
emphasis in this course where we develop 
a model, analyze the algorithm in the 
model. 
and what we're going to talk about in 
this course, is that really, we need to 
do the math. 
And that's the kind of math that I'm 
going to be talking about very soon in 
just a minute. 
and then, there's a scientific process of 
running the algorithm to solve a real 
problem and checking for agreement with 
the model. 
and so, you can't do that, you can't 
really compare two algorithms until 
you've done all the rest of this. 
so, it's very important to have that 
concept, 
context to understand why we're so 
interested in getting this math done 
right without excessive detail. 
Now there's definitely potential 
drawbacks still. 
And [COUGH] you, you know, 
the the theory of algorithms still it 
goes to places where we can't go. 
number one is, the model might not be, 
might not have a realistic model. 
And that's actually a challenge in every 
scientific discipline. 
Now, we do have a big advantage in 
Computer Science because we can randomize 
to make, make the model actually totally 
valid. 
And I'll show an example in just a 
minute. 
so, if we don't have a realistic model, 
maybe we can get a realistic model by 
randomizing. 
That's a very powerful concept that if 
you don't get, if you're doing science 
that involves going to the moon or 
killing monkeys or something. 
second thing is, the math might be too 
difficult. 
that's also a big challenge in any 
scientific discipline. 
statistical physics and many other areas. 
New developments in mathematics were 
involved in order to do the science. 
And that's certainly going to be it here. 
And really, that's what analytic 
combinatorics is. 
A calculus for analysis of algorithms is 
the true motivation for this course. 
and thirdly, it might be that the 
experiments are too difficult to really 
validate the model. 
and again that's not much of a point 
compared to any other scientific 
discipline. 
no problem for us to run an algorithm a 
million times. 
And we do it we do it a lot whereas, if 
you had to feed a million mice you might 
have some restrictions or or go to a 
million planets or whatever else. 
and, but it's a roadblock for a lot of 
people trying to study algorithms because 
actually they're trying to study galactic 
algorithms. 
It might be way too difficult to 
implement and if you can't implement it 
why, why try to analyze it? 
And it's fine for the intellectual 
challenge but again there are people out 
there that are thinking that the 
algorithms that you're developing are 
useful in practice and really they should 
be validated scientifically before the 
poor working programmer is faced with 
them. 
so that's an outline of our approach to 
the analysis of algorithms. 
