Welcome, this is going to be our first session.
So, in this we will be doing Introduction
to Optimization. We will follow this up with
a linear and non-linear regression, right.
So, in this we will be basically doing that
given a set of data points and a model; how
do we find out the model coefficients, right.
Then we will be discussing thisfive metaheuristic
techniques teaching; learning based optimization,
particle swarm optimization, differential
evolution, genetic algorithm, and artificial
bee colony optimization, right.
So, in genetic algorithm, we will be discussing
both binary and real coded g a. So, we will
not only be discussing this meta heuristic
techniques; we will also implement them in
MATLAB, right. So, we will be using MATLAB;once
you understand the techniques, you are free
to implement it on whichever programming language
you are comfortable with. So, the assignments
and quiz will notinvolve any codingexercise,
right.
So it is not necessary for you to know MATLAB
or learn MATLABto be able to do the assignments
orgive the exam; but we will be doingconsiderable
portion of this course in MATLAB, right. And
we will also introduce you to a couple of
other softwares which we will look into, right.
So, we will also be looking at constraint
handling. So, initially when we discuss these
problems, we will be,initially when we discuss
thismeta heuristic techniques we will be looking
at only at unconstraint problems; unconstrained
and bounded problems, right.
So, we will assume that there are no constraints
and then we will discuss these techniques.
Then we will let you, then we will discuss
how do we handle constraints; in this meta
heuristic techniques if the problem has constraints,
how do wehandle the constraints using this
meta heuristic techniques. So, broadly we
will be looking into penalty based and correction
approaches, right.
So once you know penalty based constraint
handling or correction approach for constraint
handling, you can use that with all of this
five techniques to solve any constrained optimization
problem, right. We will also be looking at
two mathematical programming techniques; linear
programming and mixed integer linear programming,
right. So, we are also bringing in this mathematical
programming techniques into this, so that
you can actuallyalso understand the limitations
of meta heuristic techniques, right.
So, given an optimization problem, there are
multiple ways to solve it, right. So, we should
be in a position to select the best possible
set of techniques that can help us to solve
the problem. Then we will be looking at a
production planning case study, right. So,
in this we will take one problem and we will
model it as a MILP, right. We will model and
solve it as an MILP, also we will model and
solve it using meta heuristic techniques.
So, the reason for doing this is, you will
be able to understand the strength and weakness
of both mathematical programming techniques
and meta heuristic techniques, right. So,
we will take a problem which you can consider
it to be a real life optimization problem;
and then we show how do we solve it with both
mathematical programming as well as meta heuristic
techniques, so that you can understand the
strength and weakness of both of them.
As part of this course we will be introducing
you to three differentstate of the optimization
solvers, right; one isoptimization toolbox
of MATLAB. We will be looking at IBM ILOG
CPLEX optimization studio, we will be looking
at general algebraic modeling system GAMS,
right and we will also look at NEOS optimization
solver, right.
So, both of this we will be using together,
right. So, optimization toolbox of MATLAB,
it has inbuilt mathematical programming techniques;
for linear programming, non-linear programming
and mixed integer linear programming. It has
inbuilt meta heuristic techniques which can
solve non-linear programming, for which mathematical
programming techniques are also available
and it can solve certain versions of mixed
integer non-linear programming.
Also we will be using MATLAB, right for implementing
the meta heuristic techniques that we will
discuss. IBM, ILOG, CPLEX optimization; the
full version is available as part of IBM academic
initiative, right. So, non academic users
can use either the demo or evaluation version,
right. So, IBM, ILOG, CPLEX optimization studio
can be used for solving linear programming,
mixed integer linear programming, quadratic
programming and mixed integer quadratic programming,
mixed integer quadratic constrained program
constrained programming as well as constrained
programming, all right.
So you can go to this link and you will have
to register right and then you will be able
to freely download that software, ok. So,
GAMS is primarily a modeling language, right.
So, very new merely state what is the problem
right; and then there are a set of solvers
which can be used to solve it, right. So,
the solvers which come with the free, with
the demo version of GAMSis limited right;
like it cannot solve problems of very large
dimension.
So, what we will do is, we will code the problem
in GAMS right and then we have this NEOS solver;
which can accept the GAMS files all right
and then it can solve the problem for us.
Once the problem is solved, we will be getting
the results; we can get the results over email
or it will even be displayed on your browser
where in you submit the when you submit the
problem, right.
So GAMS can be used for linear programming,
mixed integer linear programming, non-linear
programming and mixed integer non-linear programming,
right. And this NEOS optimization solver,
we will be using it only for GAMS; but if
you go and have a look atthis website, NEOS
server dot org, you will see that it accepts
input from a wide range of modeling platforms
not only from GAMS, right.
So, the reason for taking three different
solvers and not restricting to one solver
is, each of the solver has their own limitation,
right. So, for example, optimization toolbox
of MATLAB cannot solve mixed integer non-linear
programming when equality constraints are
involved, right. So, when equality constraints
are involved, it does not support integer
variables; there is no functionas of now that
is in 2019 which can solve a proper MINLP
problem.
The good thing aboutMATLAB isprogramming is
much easier in MATLAB, right. So that is why
we are going to use it to code the meta heuristic
techniques in MATLAB. This IBM, ILOG, CPLEX
optimization studio again cannot solve MINLP
problems, right; a generic MINLP problem it
cannot solve. But the good thing about this
particular software is it is full version
is available as part of IBM academic initiative
plus this constraint programming is available
which is not there in MATLAB, right.
Similarly the good thing about GAMS is no
matter what is the problem size, we will be
able to use it to model it and we can solve
it using the freely available NEOS optimization
solver. So, this can solve MINLP also right;
the drawback of GAMS is that the demo version
does not solve bigger larger problems. So,
we overcome that limitation with this NEOS
optimization solver, ok. So, each of the software
has their own set of benefits and drawbacks,
right.
So we are looking intothe best possible set
of tools, so that we can solve most of the
optimization problemsthat are encountered
in real life. In this session we will be looking
intowhat are the components of an optimization
problem and also how do we classify our optimization
problems and also into how we classify optimization
techniques.
Some of the common applications of optimization
are timetabling, right. So, for example,our
timetabling wherein we need to schedulethe
classes right, we need to decide which course
would be taught in which class at what time,
right. So, that is a classical application
of timetabling.
Another common application is site selection,
right. So, if you want to establish an industry,
you have lot of options right to establish
the industry. So, you need to decide where
you want to establish the industry. So, depending
upon the industry, a large number of factors
will play a role into how we decide the optimal
location of an industry. Optimization is also
widely used in production planning, controlling
and scheduling; it is used in tariff design
and these are some engineering applications
of optimization.
You can also look intovarious other applications,
right. So, if you go to this website, this
is a GAMS website, right. So, this is one
of the software that we will be learning as
part of this course, right. So, it has a collection
of problems right; depending upon the subject,
you can look into the problems depending upon
the subject. So, right here if we see it has
been used in macroeconomics, management science
and OR, finance,stochastic programming, microeconomics
chemical engineering, right.
So, this is just a very small subset of problems
which we are showing over here, right. So,
if you go into this; if you go to this website,
you will be able to see a large collection
of problems and you will be able to see that
it is used invariousareas, right. So all these
files are freely available; so you can even
sort them as per the type of problem right
and this briefly describes what is the problem.
So, these are some commercial success story
reported by mathworks. So, mathworks is the
company which owns MATLAB, right. So, here
we have shown four commercial applications.
If you are interested you can go and have
a look atthe individual story and see how
it has been applied in that particular industry.
These are some of the applications from IBM;
here we are only here we are restricting ourselves
withthe success stories of the three softwares
which we are going to study, but optimization
has been used in many commercial applications
apart from this.
So you can also get some resources from this
Google OR tools. So, Google has thisweb page,
right; so wherein they have given a lot of
OR tools which are freely available. In the
last three to four slides we just showed you
the commercial applications right, where the
tools which we are going to discuss as part
of this course have been used
So, now let us look into some of the interesting
optimization problems that are widely stated,
right. So, this traveling salesman problem;
a traveling person has to tour n cities. So,
in this case; so for example, let us considered
this case where in there are E cities. So,
there are five cities city A, B, C, D, E right
and there is a cost associated with each city.
So, if we travel from city A to city B the
cost is 5; if we travel from city A to city
E the cost is 2; if we travel from city A
to city D the cost is 3.
So if you are traveling from city E to city
D the cost is let us say 1; this is let us
say 4, this is let us say 8 and this is let
us say 9 and this let us say it is 2, right.
So, this is the cost of traveling in between
two cities. So, the task is, the traveling
salesman has to visit all the cities right
and come back to the city that started with
and each city is to be visited exactly once.
So, one of the solution can be from city A
to city B, city B to city E, city E to city
C and city C to city D and returned back to
city A; we know the costs associated with
each tour. So, A to B we know the cost is
5;, B to E the cost is 4; E to C the cost
is 2; C to D the cost is 9; D to A right D
to A the cost is 3, right. So, the total cost
is the summation of all this, right.
Similarly another tool can be, instead of
this person going to city B let us say he
decides to go to city D right, and then to
city E, then to city C, then to city B and
then he comes back to city A. So, again there
is a cost associated with each of this, right.
So, again we can calculate what is the total
cost, right. So, there is a different cost
over here, there is a different cost over
here; so depending upon the routethe person
takes, the cost is going to be different.
So, the task here is to find out this path;
that which path is to be chosen such that
the total costs required to tour all the cities
without repeating any of them and again come
back to the original starting point that cost
has to be minimum. So, this is a classical
problem,it is known as traveling salesman
problem, right. So, there are various ways
to solve this traveling salesman problem,
right.
So, another problem is now what is called
as knapsack problem. So, in this knapsack
problem we have a hiker right, who wants to
fillthe knapsack to a maximum value. So, let
us say there are 10 items, each item is associated
with the weight and it has a cost, right.
So, I have 10 items ; i denotes the index
w 1, w 2 all the way up to w 10 and then we
have c 1, c 2 all the way up to c 10. So,
c indicates the cost, right.
So, now the person wants to fill as many items
as possible in the bag, right and there is
a weight constraint; that the total weight
of the knapsack has to be less than a given
value, right. And for whatever value whatever
item he or she chooses to placein the knapsack,we
can calculate the total value that he is carrying.
Let us saythe we have c 1 is equal to 10,
c 2 equal to 5, c 3 is equal to 4, c 4 is
equal to 8; let us say he decides to carry
just c 1 and c 4.
So, the value that he is carrying is only
18; 10 plus 8, 18. And corresponding to the
first item there is also going to be a weight
and for the fourth item there is going to
be a weight. So, the weight that is carrying
going to carry is w 1 plus w 4, right. So,
that total weight has to be less than what
is prescribed, right. So, now, the task is
which of the items can be chosen, so that
the weight constraint is respected as well
as the person is able to carry the maximum
value with himself or herself. So that is
known as knapsack problem.
Both traveling salesman and knapsack problem
are optimization problem. There is something
called us feasibility problem. So, in feasibility
problem, we do not have an objective that
has to be minimized or maximized right; but
we have a set of constraints that need to
be satisfied, right. So, map coloring is one
such classical problem. So, here there are
six countries right; as shown over here there
are six countries Netherland, Belgium, Luxembourg,
Germany, Denmark, and France.
So, the task is to color each country such
that no two countries which are neighbors
are colored with the same color, right. So,
we have four sets of color blue, white, yellow,
green. So, each of the six city is to be colored
by one of these four colors right; and the
constraint is that no pair of neighboring
countries should have the same color, right.
So here if you see there is noobjective which
has to be minimized or maximized; but we have
a set of constraints that have to be satisfied.
So, the color of Denmark and Germany cannot
be the same; because they are neighboring.
The color of Germany Netherlands or Germany-Belgium,
Germany-Luxembourg cannot be same as they
have they are neighbors, right. Similarly
Netherlands cannot have the same color as
Belgium, Germany or Denmark, right.
So, this is a feasibility problem. So, here
we need to find a set of decision variables;
as long as it satisfies the constraints which
are given, it is sufficient, there is no objective
which is to be maximized or minimized. So,
feasibility problems are a subset of optimization
problem. So, if we solve an optimization problem;
the solution that which we get will be feasible,
right. But the reverse is not true; the solution
of a feasibility problem will be feasible
for an optimization problem, but it need not
be the best solution.
Another classicalfeasibility problem is soduko
problem; many of you would have solved the
sudoku problem. So, in the sudoku problem,
we have 81 squares; there are 9 rows and 9
columns, right; of 1 to 9 rows and 1 to 9
columns. So, there are a total of 81 cells,
right. So, each cell is to befilled withintegers
from 1 to 9 right; remember 0 is not allowed,1
right.
So, each row has to be filled with the numbers
any number from 1 to 9, such that no two numbers
in the same column or no two numbers in the
same row are identical, right. So, for example,
this is already filled with 9. So, I cannot
use 9 to fill any other empty cell in row
1, right.
Similarlythis column if you see,this particular
column it already has a 6 and 1. So the rest
of the cells which we have over here, can
take any value from 1 to 9 except 1 and 6,
right. So, that is the constraint that, no
two values in any row should be identical
and no two values in any column should be
identical, right. And there are these 9 squares.
So, here if you see this is the 1st square,
2nd square, 3rd, 4th, 5th, 6th, 7th, 8th and
9th, right.
So, each of the square is going to have 9
cells and the values in the 9 in these 9 cells
need to be unique. So, for example, this cell
if you see, right. So, there are 9 cells and
we have already used 3, 6, 9, 7, right. So,
the remaining values are 1, 2, 4, 5, 8, right
1, 2, 3, 4, 5, 6, 7 is already there 8, 9.
So these five boxes 1, 2, 3, 4, 5 have to
be filled with these values. Again the row
and column constraints need to be respected,
right. So, for example, 2 we cannot fill this
box with 2, because 2 is already there in
that row, right. So, this is sudoku problem;
there is no objective as such over here, there
are only constraints. So, a sudoko problem
can have more than one solution, right.
So, for example, for this problem there are
three solutions, right. So, all these three
solutions are unique right. So, consider these
two solutions here it is 1, 5; here it is
5, 1 and again here it is 5, 1 and here it
is 1, 5 right. So, these two solutions are
unique solutions, right. So, both of these
solutions are equally good for this feasibility
problem, right.
Similarly if we compare the solution 3, if
we call this a solution 3 and if we call this
a solution 2, right. So here if we see this
is also 5, 1, this is also 5, 1, this is also
1, 5, this is also 1, 5 right; but the change
happens here. So, here it is 3, 8, here it
is 8, 3 and this is 8 3 over here and this
is 3 8 over here, right. So, all these three
solutions s 1, s 2, s 3 are feasible solutions
for this sudoku problem right; however, any
feasibility problem can be converted into
an optimization problem, right.
Right now I can say that I want to fill this
box right, satisfying all the constraints
which we discussed so far and we want this
value to be maximum, right. So, let us say
if this box is represented since there are
81 cells right; let us say I this box is represented
by 9 comma 9 or x 9, 9, this is the name of
a variable right. So if I denote the value
filled; if I denote the cell as x 9, 9, right,
so it can take either the value of 1 over
here as shown over here, 5 as shown over here
or 5 as shown over here, right.
Then this solution is not optimal right because
we wanted to maximize this value right; where
as these two solutions are equally good for
this optimization problem. Remember first
we discussed feasibility problem; feasibility
problem we had to obey only the constraints,
right. So, all the solutions all the three
solutions are equally good; then we converted
into an optimization problem. In optimization
problem we said that, all the constraints
need to be satisfied; on top of that we said
that the last box right denoted by x 9, 9
should have the maximum possible value, right.
Once we say that, then s 1 is no longer the
optimal solution; because it has a value 1,
whereas solution 2 and solution 3 have a value
of 5. So we would not prefer solution 1, if
we had an objective maximize x 9, 9 ok; whereas
solution 2 and solution 3 are equally good
solution even for that optimization problem.
So, that is the difference between a feasibility
problem and an optimization problem. Now we
just saw that this problem can be converted
into an optimization problem, right.
So, here what we have are two approaches,
right. So, this is one approach one and this
is approach two; this is an mixed integer
linear programming formulation, right. So,
this is an optimization problem; here we can
either add a objective function right, that
one of the variable has to be maximum or minimum
or we can just say maximize 5, right. So;
obviously, this does not make sense because
this is a scalar it is going to be 5; but
these are constraints, right.
So, this problem can be mathematically formulated
into a mixed integer linear programming problem.
In fact, this is actually ILP, it is not even
mixed integer linear programming, it is just
integer linear programming; because all the
variables will take only integer values, right.
So, that is going to be one approach right;
wherein this whatever we discussed can be
mathematically translated into these equations,
right.
So, that is the modeling part; any optimization
problem is going to have a first a modeling
part right. Once we have the model, it is
only then we apply any optimization techniques
to solve. So, this is one approach; wherein
the sudoku problem is converted into a integer
linear programming and then solved.
This is another approach right called as constraint
programming. We will not be discussing constraint
programming as part of this course right;
but we thought since this is an introduction
lecture, we will at least let you know that
there are other techniques also, right.
So CP stands for constraint programming and
the sudoku problem is much easier to model
for constraint programming, right. So, you
do not need to worry about how did these equations
come as of now, right. When we are discussing
a mixed integer linear programming, we will
let you know how these equations were arrived
at, right. So, right now from the slide you
just need to understand that, there are feasibility
problems, there are optimization problem.
Any feasibility problem can be converted into
an optimization problem, right. So, if we
know how to solve an optimization problem,
we can solve feasibility problems, right.
So, now let us formally look into the optimization
and its components. So, optimizationas you
know,it is the selection of best choice right;
when we say best choice, it has to be based
on some criteria. If we are going to say select
the best choice that invariably means that,
we have a set of alternatives from which we
have to select the best choice, right.
So that decision is called as decision variables.
So, we need to decide something. So, in optimization
we will be calling it as a decision variable,
right. A objective function is a relation
of decision variables, which is what we want
to either minimize or maximize or in general
we say optimize. And we have constraintswhich
are restriction on the decision variables.
So, broadly we have three components; one
is decision variables, other is objective
function and the third one is constraint.
So, constraints limitwhat we can do right;objective
function is what we want to maximize or minimize;
and decision variables are those which we
can actually change, right. So, the point
is that, we need to find out the optimal value
of the decision variables, right. So, one
classical optimization problem that many of
us might have solved isselecting the cheapest
flight to travel from one city to another
city, right.
So in that case the decision variable is to
select a flight, right. So, these are the
decision variables, right. So, the decision
variable as such is which flight to select.
So, in this case if we see the minimum cost
is 6355, right. So, that is the optimal decision
with respect to minimizing the cost. So, the
nature of the decision variable objective
function and constraintshelps in the classification
of problems and in many times and at times
even it helps in the classification of techniques.
So, let us look into each of the componentin
a little bit more detail. The formulation
of an optimization problem starts with identifying
the decision variables. So, because we need
to knowwhat is that can be changed, so as
to improve the objective function. So, if
we are not allowed to alter anything, in that
case there is nothing to optimize.
So for example, if we tell that x has to take
a value of 2 right; then there is no choice
that has to be made for x. But if it is said
that x can take any value in the real domain,
then we have a set of alternatives for from
which we need to choose what is the best value
for x, right. So, this decision variablerelates
the objective function and constraints. So,
the decision variables can be continuous;
so for example, here if x is the decision
variable and if f of x is the objective function,
right.
So, x can take any value between 0 and 25.
So, in that case x is continuous; the lower
bound of x is 0 and upper bound of x is 25.
So, x is x can vary from 0 to 25. Our task
is to find out the value of x for which the
function is either minimum or maximum. At
times the variable can bediscontinuous; so
for example, if let us say this is 5, this
is 10, this is 20, and this is 30, right.
So, in this case this variable x 1can take
any value between 5 to 10 right and can take
any value between 20 to 30; but it cannot
take a value which is greater than 10 right
and we cannot take a value which is less than
20 right greater than 10 and. So, x greater
than 10 and x less than 20 is not allowed
right; whereas, x in between 5 and 10 is permissible
and x between 20 and 30 is a permissible,
right.
So, between 10 and 20 there is a discontinuity.
So, for integer variablesonly the integer
values are allowed. So, for example, one is
allowed, two is allowed,threes is allowed
for x 1, four is allowed, five is allowed;
but 1.1 is not allowed or 2.8 is not allowed
right, only the integers are allowed, right.
So, in this case x 1 can vary between 1 and
5, the only the integer values, right.
So, the decision variable can be continuous,
semi continuous or it can be integer or it
can even be a set. So, for example, we need
to decide which color is to be used to paint
a country on the map, right. So, there we
have options ofcolors, right; so red, blue,
green, yellow. So, the decision variable can
be set also. In most real life problem or
the decision variables are bounded; but mathematically
a decision variable can also be an unbounded
variable.
So, now, that we have done a decision variable.
Let us move onto the objective function right.
So, the objective function is the criteria
with respect to which the decision variables
are to be optimized. So our task is to determine
the values of the decision variable, so that
a function known as objective function is
either minimum or maximum, right.
So, every point in the decision variable space;
so for example, if we have a two variable
problem x 1 and x 2 right and let us say if
thisportion is the feasible region. Then every
point in the feasible region has a particular
scalar value called as objective function,
right. So, for example, let us say f is equal
tox 1 squared plus 3 x 2; if this is our objective
function, then for any value of x 1 and x
2 there is a unique value of f, right.
So, a point in the search space, so x 1 and
x 2 constitutes this search space; because
we are looking for the optimal values for
x 1 and x 2 in it is domain, right. So every
point over here has a corresponding scalar
value for it is objective function. So, now,
if we are given two solutions; let us say
this solution is over here it is corresponds
to over here and the solution corresponds
to over here and let us say this is3 and this
let us say this is 8, right.
So, if the optimization problem is to maximize
f, then this solution is better; and if the
optimization problem is to minimize f, then
this solution is better right. So, comparing
two solutions has nothing, but finding out
the minimum of those twoobjective function
values. So, similar to decision variables,the
objective function can be continuous or semi
continuous. So, here we have an example of
a continuous function, right.
So, as x varies fromlet us say point a to
point b, f of x is continuous. Over here we
have shown a discontinuous function; that
the objective function is defined from this
particular x to this particular x, but there
is no definition for the objective function
in this region, right. So, over here again
it is defined in this region; but there is
a discontinuity in between. So, the objective
function can be continuous or semi continuous.
So, for this course we will be most of the
time we will be either talking about a minimization
problem or a maximization problem. So, any
minimization problem can be converted to a
maximization problem or vice versa by multiplying
the objective function with a negative sign,
ok. So, for example, if the problem is to
maximize f of x; the solution for this problem
is the same as the solution for minimizing
minus f of x, right. So, the value of the
objective function might differ, but the value
of the solution x would be the same.
So, for example, you can look at this figure.
So, this is the curve that top curve is forf
of x right; this curve is for f of x and the
bottom curve is for minus f of x, right. So,
if you see that is an mirror image, right.
So, the minimum of this function f of x is
located over here; the value at which the
minimum occurs is x star, right. So, at x
star if you see,it is also the same point
at which the maximum of thisminus f of x occurs,
right. So, x star remains the same, right.
So, the objective function would be this,
if you are minimizing f of x. So, let us say
if you get 2 over here, over here you would
get minus 2 right so; but x star would be
the same. So, if we know that we are actually,
let us say we hadoptimization problem it said
a maximize f of x and we got a value of let
us say 2 and if we minimize it, minimize if
you do minus minimize; if you do minimize
negative of f of x we will get a solution
minus 2.
So, this can be brought back to this actual
value by just multiplying a with a negative
sign. There would not be any change at the
point at which the maxima or the minima occurs,
right. So, x star remains the same and that
is the task right; to find out the value of
x at which a function is either maximum or
minimum . So, the objective function just
like the decision variables can be bounded
or unbounded, right.
And there are some special problem called
as feasibility problems, in which there is
no objective function right; but there are
a set of constraintsthat need to be satisfied.
So, for example, sudoku problem is a example
for feasibility problem. We did briefly discuss
how this feasibility problem can be converted
into an optimization problem, right. So, in
feasibility problem we only have constraints;
in an optimization problem we may or may not
have restrictions on the decision variable.
Now that we have discussed decision variables
and objective function, let us move on to
constraints, right. So, broadly constraints
can be of two types; one isinequality constraints
and the other one is equality constraints.
So, inequality constraints usually arise when
we have resource constraints that something
has to be less than or equal to something
or something requires to be greater than or
equal to something , right.
So, in general they are denoted by g of x
less than 0; but it can be converted into
greater than 0,greater than equal to 0by just
multiplying with the negative sign, right.
So, if x 1 plus x 2 is less than equal to
10; if this is the requirement, then it is
the same as prescribing minus x 1 plus x 2
is greater than or equal to minus 10 not plus
10, but a minus 10 over here, right.
So, just like objective functionthe constraints
can also be converted from less than equal
to form to greater than equal to form or vice
versa greater than equal to form to less than
equal to form by multiplying with the negative
sign on both the sides of the equation. So,
equality constraints are usually considered
a very difficult to satisfy; in general they
are denoted by h of x is equal to 0, right.
So, here we have two examples that x 1 plus
x 2 is equal to 3 andover here x e to the
power minus x square minus y square should
be equal to phi. So, the task over here is
to come up for; if this is our constraint
the task here is to come up with the values
of x 1 and x 2 such that if we add both of
this number it should be exactly equal to
3, right. So, 3.00000 1 is not allowed and
2.99999 is also not allowed, right. So, that
is why these constraints are very difficult
to satisfy.
So, usually some tolerance is given, right.
So, this equality constraintusuallytolerance
is given to that. So, when we actually start
solving a problem, we will look into that.
So, based on this constraints; a solution
can be classified either as feasible solution
or an infeasible solution. So, if a solution
satisfies all the constraints in the problem,
it is said to be a feasible solution; and
if a solution does not satisfy even one constraint,
it is said to be an in feasible solution.
So, we may have a 100 constraints, a solution
may satisfy 99 constraints and maybe failing
on a 1 constraint; failing in the sense it
is not able to satisfythe requirement, even
in that case the solution is to be classified
as infeasible solution, right. And again constraints
can be classified as hard constraints and
soft constraints, right. So, hard constraints
are those that have to be satisfied, so that
a solution has to be accepted. Soft constraints
are those which can be allowed to relax to
some extent to accept the solution, right.
So, just let us look at couple of examples
to understand the feasibility of solution.
So, here we have a objective function, right.
So, the decision variables are x 1 and x 2,
right. So, now, we are supposed to find out
the values of x 1 and x 2 such that when those
values are plugged into this part that should
result in a value which is greater than or
equal to 300, right. So, if it does not happen
then the solution is said to be an infeasible
solution.
So, for example, consider the solution; x
1 is equal to 3 and x 2 is equal to 10 and
another solution x 1 is equal to 8 and x 2
is equal to 6. So, if we calculate the objective
function forboth of this, right. So, if I
plug 3 comma 10 in thisexpression right, we
will get a 7.05 and if we plug this 8 comma
6, we will get a value of 16.34. And if we
calculate this right hand side of this constraint
that pi x 1 square x 2 by 4; if we calculate
that part, it turns out to be 70.69for solution
1 and in solution 2 it turns out to be 301.59,
right.
So, if we look at the objective function f
1 and f 2, 7.05 is better than 16.34; because
our objective is to minimize. However, this
solutiondoes not satisfy the constraint, that
70.69 is not greater than equal to 300, right.
So, this solution is an infeasible solution,
right, it violates the constraint, it is an
infeasible solution; whereas, this solution
the solution is to it is a feasible solution,
since it satisfies the constraint, right.
So, solution 2 would be preferred to solution
1, right; despite the fact that f 1is better
than f 2, because the problem is minimizing.
So, a feasible solution is to be preferred
to an infeasible solution, right; no matter
how good is the objective function value for
an infeasible solution, it does not matter,
because it does not satisfy the constraintswhich
is a requirement.
So, now let us look atwhat is a bounded and
an unbounded problem, right. So, let us say
we have a function f which is 3 x 1 plus 2
x 2 right and we are supposed to minimize
that, right. So, and if there are no restrictions
of on x 1 and x 2, then x 1 and x 2 can take
a value of minus infinity and minus infinity
right and the problemthe objective function
value itself will be minus infinity.
So, this problem is known as an unbounded
problem. So, the minimum value of f sminus
infinity, right. So, now, if we put constraints
on the decision variables, so the decision
variables x 1 and x 2; here they wereunrestricted,
right. So, x 1 and x 2 could have taken any
values between minus infinity to plus infinity.
Now we have restricted that x 1 has to be
between 100 and 700 and x 2 has to be between
50 and 100, right.
So, we have this line this, this is x 1, this
is x 2. So, x 1 has to begreater than or equal
to 100, so we have this line right; and then
x 1 has to be less than equal to 700, so we
have this this line right. So, x 1 the feasible
region of x 1 is from here to here. Now, we
plot the feasible region of x 2. So, x 2 has
to be greater than or equal to 50. So, any
value over here is infeasible with respect
to x 2; with respect to x 1 it is feasible,
but with respect to x 2 it is infeasible.
Similarly x 2 has to be less than equal to
100, right. So, any value over here is infeasible
with respect tox 2, right. So, the feasible
region now is bounded. So, it is in this region
that we are supposed to locate the best solution
right. So, in this case it happens thatthis
objective function 3 x 1 plus 2 x 2 will have
a minimum valueof 400 at this point; any other
point if you calculateinside this entire feasible
region,you would see that it hasuh objective
function value which is actually greater than
400.
So, since we are looking at a minimization
problem; the least value that we can have
for this problem with these constraints is400,
right. So, now, let us add one more constraint;
that the values of x 1 and x 2 should satisfy
this constraint that x 1 minus 3 x 2 should
be greater than or equal to 400. So, to graphically
plot what we will plot is x 1 minus 3 x 2
is equal to 400; remember our constraint is
greater than or equal to 400, but we are plotting
this line x 1 minus 3 x 2 equal to 400, this
line can be plotted. This is a region right,
greater than equal to 400 is a region.
So, let us . So, in order to plot this line;
what we can do is we can take x 1 to be 0
right, if you take x 1 to be 0, right. So,
then minus 3 x 2 is equal to 400, which implies
x 2 is equal to 133.33. Similarly, so that
gives one point 0 comma 133.33 that is one
point. The second point is we will keep x
2 as 0 and calculate the value of x 1. So,
it will come turn out to be 0 comma 400. So,
now, we have two points, right. So, we can
draw this line over here which is shown, right.
So, that line passes through 0 comma 133.33
and 0 and 400 comma 0, right. So, that line
is plotted and so this feasible region which
we had over here, this rectangle has now beenreduced
to this particular region right, this shadedpart
which is shown here, right. So, that is becausein
this region x 1 minus 3 x 2 is greater than
400; whereas, in the region about this right,
it will not satisfy this constraint, x 1 minus
3 x 2 will be less than equal to 400. So,
you can try out some values.
So, for example, if you substitute this particular
value 100 comma 100 in this equation, right.
So, a 100 minus 3 times 100 rightthat will
be minus 200, which is not greater than 400.
So, this point is to it is not feasible. So,
similarlyall the points above this line are
infeasible right and over here we are restricted
by this line x 1 is 700 and over here we are
restricted by this line x 2 equal to 50. So,
thistriangle which we see herethat is the
feasible region. So, as you can see, as we
add a constraint the feasible regiondecreases,
right.
So, here there are two types of constraints;
one is redundant constraint and one is non
redundant constraint, right. So, a redundant
constraint does not help you to reduce the
feasible region and non redundant constraint
helps you to reduce the feasible region. So,
the optimalsolution for this problem isx 1
is equal to 550, x 2 is equal to 50. So, if
we plug these two values in this objective
function 3 x 1 plus 2 x 2; we will get a objective
function value of 1750, right.
So, when we do linear programming, this problem
is a linear programming problem; when we actually
do linear programming,you will see that how
did we arrive at this solution. Right now
what you need to understand is, what is a
feasible region and what is an infeasible
region. So, at times we can even have bounds
on the objective function though it is very
rare, but it is possible, right.
So, in that case if you see this is the line3
x 1 plus 2 x 2. So, just like we plotted x
1 minus 3 x 2 is equal to 400;we can also
plot a line 3 x 1 plus 2 x 2 is equal to 1800,
right. So, if we plot this line, it will turn
out to be this line, right. So, now, the feasibility
region is even reduced. So, here if you see
the star which is over here has actually moved
over, right. So, the optimal solution is 1800;
here we are limited by the objective function.
1750 is possible that solution is possible,
but since we have this additional constraint
that the fitness function has to be greater
than or equal to 1800, right the solution
actually is 1800, right. So, this solution
becomes infeasible, 1750 becomes infeasible;
because there is a constraint that the fitness
the objective function has to be greater than
or equal to 1800.
So, in this case we had bounds on the decision
variable. So, over here there is a small typo,
right. So, minus 3 x 2 is equal to 400. So,
if we are to calculate x 2 it has to be minus
400 by 3 and this has to be minus 133 by 3,
right. So, the line is still drawn correctly,
but just that we had a typo over there.
So, in the previous example we had seenthe
decision variables they were explicitly bounded;
however, it is not necessary that the decision
variables need to be explicitly bounded for
the problem to be a bounded problem. So, the
objective function is minimized 3 x 1 plus
2 x 2, the constraint is 2 x 1 plus x 2 greater
than 5. So, we do the same thing that we took
we take it as t x 1 plus x 2 is equal to 5.
And then take two values of x 1 calculate
the values of x 2 and we will be able to plot
this line, right. So, this line indicates
2 x 1 plus x 2 equal to 5. And the similar
to the calculation that we discussed in the
previous slide; this is this would be the
area of feasibility. Similarly we can also
plot these two lines x 1 plus x 2 is equal
to 4 and we take two values of x 1 determine
what is the value of x 2 and plot it, right.
So, this line is going to be a x 1 plus x
2 is equal to 4; and this line is going to
be x 1 minus x 2 equal to 2.
So, any region below this x 1 plus x 2 curvex
1 plus x 2 equal to four line is feasible
region. So, this is a feasible region, right.
So, this is also a feasible region . Similarly
for the other line x 1 minus x 2 is less than
equal to 2. So, this is the feasible region,
right. So, if we plot all the three all the
feasible regions of the constraint; we will
be left with only the shaded regions right.
Because let us sayany value which is feasible
for this x 1 plus x 2 equal to 4 need not
be feasible for 2 x 1 plus x 2greater than
equal to 5.
That is why some of the area which is feasible
forlet us say this constraint is not feasible
for this constraint, right. So, the common
area which is feasible for all the three constraints
is our feasible region. So, the optima has
to be located in this feasible region. So,
here if you see x 1 and x 2 can vary from
minus infinity to infinity right; there is
no explicit bound on x 1 and x 2, still the
problem is a bounded problem.
So, for us to declare a problem is feasible
or not; we need to actually look into the
nature of the constraints, right. Merely by
looking into the bounds of the variables,
it is not possible to say whether the problem
is bounded or unbounded for an arbitrary optimization
problem. So, this is a case wherein we show
a infeasible problem that the constraints
are conflicting.
So, over here also the, over here the objective
function is minimized 2 x 1 plus 3 x 2 and
the constraints are 4 x 1 plus x 2 is less
than 5, 3 x 2 is greater than equal to 4,
and x 1 minus x 2 is greater than or equal
to 3 and again x 1 and x 2 can vary from minus
infinity to plus infinity, right. So, if we
plot these three lines, again plotting the
lines is as we discussed previously we take
we convert this into a equality constraint
generate two points and since it is a linear
equation we draw straight lines, right.
In this case the feasible region for each
constraint is indicated by this dotted arrows,
right. So, for the constraint x 1 minus x
2 greater than 3, this is the feasible region,
below the constraint; for this curve 3 x 2
for this line 3 x 2 greater than or equal
to 4 right this is that line 3 x 2 equal to
4, right. So, basically what we are saying
is x 2 is equal to 1.33. So, x 2 has to be
greater than 1.33, so it is this region. In
this region x the value of x 2 is greater
than 1.33.
Similarly we plot the line x 1 minus x 2 equal
to 3, right. So, we this region is the infeasible,
this region is the feasible region, right.
So, now, if we see; there is no common area
for this three constraints, in this case we
call the problem as infeasible problem. So,
another easy way to understand is let us say
there is a constraint which says x has to
be greater than or equal to 2 and another
constraint which says x has to be less than
or equal to 1, right.
So, any value of x which is less than or equal
to 1 is feasible for this constraint; for
this constraint any value of x which is actually
greater than or equal to 2 is feasible, right.
So, if I have to satisfy both this constraints;
there is no single value of x which will satisfy
both of these equations, right. So, in this
case these two equations put together make
the problem infeasible.
Uh another term which will be commonly using
is contours. So, contours are lines which
have identical objective function value. So,
for example, here if we see if I if we take
x 1 is equal to 2 and x 2 is equal to 4; it
will give objective function value of 20.
If we take x 1 is equal to 4 x 2 is equal
to 2; we will get an objective function value
of 20. If you take x 1 is equal to 1 and fix
f to be 20 right; we can actually find out
what is the value of x 2.
So, similarly we can generate many points
and then we can plot. So, this is x 1, this
is x 2, right. So, all these values can be
plotted 2 comma 4, 4 comma 2, 1 comma whatever
we get over here, 3 comma something, 0 comma
something. So, we would get a curve like this.
So, this is called as contour. So, all the
points on this curve have an function value
of 20, right.
So, similarlywe can plot contours for 10,
we can plot contours for 5. So, similarly
we can plot counters for other values, right.
So, this is known as contour plot. So, this
shows ashow the objective function behaves
in the search space. So, the our search space
is x 1 is equal to minus 5 to 5 and x 2 is
equal to minus 5 to 5. So, in this region
how the objective function varies, right.
So, all the points in this curve we will have
an objective function value of 20; all the
points in on thiscontour, we will have a function
value of 5. So, very often we will be referring
to this contour plot. And then we have something
called as realizations, right. So, realizations
are those solutions which have the same objective
function value, but different decision variable
value. So, for example,if you remember the
slidein which the flight costs were given;
you may have two different flights which have
which which may have the same cost, right.
So, from the perspective of the objective
function, which is cost in our case the value
is same; but the decision variables can be
different, right. So, those are called as
realizations, right. So, for example, if we
have this problemmaximize f is equal to x
1 x 2; subject to these two constraints x
1 plus x 2 should be equal to 3 and x 1 minus
x 2 should be less than equal to 1 and bounded
by the constraint that x 1 and x 2 should
be greater than or equal to 0.
So, if we plot that. So, this is the line
x 1 plus x 2 equal to 3, right; and this is
the line x 1 minus x 2equal to 1, right. So,
since this is a less than equal to 1. So,
it this is the feasible region as indicated
by these dotted arrows, right. And the feasible
region is also this line; this x 1 plus x
2 equal to 3, because the points that are
lying on this straight line x 1 plus x 2 equal
to 3 only those points satisfy this constraint,
right.
So, the intersection point of these two constraint
is over here and remember this entire region
is feasible, right, This entire region isfeasible
because it has x 1 minus x 2 is less than
equal to 1. So, we have one more point over
here in addition to this, right. So, those
two points are x 1 is equal to 1, x 2 is equal
to 2 and x 1 is equal to 2 and x 2 equal to
1. So, at both these places the objective
function value is 2, right. So, these are
called us realizations right; same objective
function value, but differentdecision variables,
right.
In one case x 1 is in one case the solution
is 1 comma 2 and in the other case the solution
is 2 comma 1. So, this is an example again
for a realization, right. So, here we need
to design a can right; what we can vary is
the diameter and the height right, and the
diameter and height has to be between 0 and
31. So, the constraint is that the can should
have a volume of at least 300 ml, right.
So, the volume is given by pi d square h by
4. So, this has to be greater than equal to
300; because we have at least, right. So,
if it is more than 300 it is fine, but we
want at least 300 ml. So, that is why we have
this greater than equal to constraint and
the objective is to minimize the material
cost of the can, all right. So, that if you
see a see it denotes the cost and this will
correspond to the material cost of the can.
So, in this case if we takethe diameter as
8 centimeter and if we take the height as
10 centimeter; the objective function value
would be 22.87. And if you take the diameter
is 5 centimeter and the height as 19.9 centimeter;
the objective function value which is the
cost would still be 22.87 . So, the can either
look like this or it can look like this, right.
So, here the height is smallerthen the this
can; but in both cases the cost is 22.87,
right. So, depending upon other constraints;
one can choose either this solution or this
solution right without compromising on the
objective function value. So, that is why
it is good to know the realizations; not only
the optimal solution, but also many instances
in which the same optimal costs can be obtained.
Now, let us look at a monotonic function and
convex functions, right. So, we will be using
this term very often. So, monotonic functions
are functions which are either continuously
increasing or decreasing with respect to x.
So, for example, in this case if we increase
as we increase x, the value of let us say
this is f of x keeps increasing, right.
So, it has a monotonicnature to it; here in
the second plot if we in the second figure
if we see, if we keep increasing x, f of x
continuously decreases, right. So, this is
also monotonic, this is also monotonic; it
does not matter whether it is increasing or
decreasing as long as it is consistently increasing
or decreasing. Whereas, this third plot is
an example of a non-monotonic function, right.
So, here if we see as we increase x, f of
x is first decreasing till here; and as we
keep increasing x, f of x is increasing and
again as we keep increasing x, f of x is decreasing,
right.
So, here if you see from here to here, from
in this region the function is monotonically
decreasing; in this region the function is
monotonically increasing and again in this
region the function is monotonically decreasing.
When we have functions behaving like this,
those are known as non monotonic functions.
We will also be using this term convex functions
very often, right.
So, convex functions most of you would be
knowing convex function, right. So, convex
function if you take any two points in the
line, right. So, for example, if I take this
point and if I take this point and if I were
to draw a straight line connecting these two
points, right. So, the line would be above
the curve, right. So, if that happens that
is known as a convex function. So, that is
the geometrical interpretation of convex function.
Mathematically let us say if this is if I
do it two points x 1 and x 2, right. So, they
are their corresponding fitness function value
is the f of x 1 and f of x 2 right. So, we
have two points x 1, it is corresponding y
value or the function values f of x 1; we
have x 2 it is corresponding function values
f of x 2. And if we take any point right;
so let us say we take this point x 3, right.
So, x 3 is a linear combination of x 1 and
x 2, right. So, we can express x 3 is equal
to gamma x 1 plus 1 minus gamma x 2, right.
So, and again we can calculate f of x 3. So,
f of x 3 is nothing, but f of gamma x 1 plus
1 minus gamma x 2, right. So, that is what
is written over here. So, this is nothing,
but f of x 3, right. So, that would be less
than if that is less than gamma times f of
x 1 plus 1 minus gamma times f of x 2, then
the function is a convex function. So, as
you can see a convex functionwill have only
one point at which the gradient vanishes,
right. So, we will have only one optimal solution,
right.
So, the local optima itself will be the global
optima. So, whenever we have a convex function,
we it is sufficient if we find the stationary
point and for at that stationery point the
function is bound to be minimum, right. So,
it is not necessary to calculate the higher
derivatives.
And then we have unimodal functions and multimodal
functions, right. So, unimodal functions are
those for some value of m, if the function
is monotonically increasing for x less than
m and monotonically decreasing for x greater
than or equal to m, right. So, these are examples
of monotonically,these are examples of unimodal
function. So, if you see we basically have
one peak, right. So, till this pointthe function
is. So, this is our m.
So, till that point the function is actually
increasing after that it is decreasing, right.
So, we basically have this a one peak. So,
those are called as unimodal function, right.
Similarly what we have shown is over here
for maximization right. So, for minimization
it can be, it continuously decreases till
a particular value and then increases beyond
that value. So, this is also a unimodal function,
right. So, for the unimodal functionthe maximum
value of f of x is f of m, right. So, here
if we see the value of the function is maximum
at m, right.
So, that is a nice property about a unimodal
function, right and it has only one peak,
right. So, there are no other local maxima.
These are examples of multimodal functions,
right. So, for example, as x increases;here
it is increasing, then it is decreasing and
still if x is increasing this is starting
to increase, right. So, here if we see we
have multiple peaks over here and multiple
valleys over here, right. So, this plot chosen,
but one dimensional; so here the plot is between
x and f of x; here it is betweenx 1 x 2 and
f. So, it is a 3 D plot.
So, now, if we see traps are there, right.
So, there are lot of valleys, there are lot
of peaks; if our problem is to maximize is
to find out the peak that is that has the
best value, despite there being so many other
peaks. And if our problem is to minimize the
objective function, then our task is to find
out the values of x 1 and x 2 at which the
objective function has a lowest value. So,
that is unimodal and multimodal functions.
So, most real life optimization problems are
multimodal in nature.
So, optimal solutions can be said to be either
a local or global, right. So, what we are
discussing here which is with respect to a
minimization problem; but the same thing can
be extended to maximization problem. So, local
optima means the smallest function value in
it is neighborhood, right. So, for example,
if we take this point;in it is neighborhood
it has the least value right, same thing this
point it has the leastvalue in it is neighborhood,
right.
So, all those four points arelocal optima,
right. So, far a problemthere can be multiple
local optima; for global optima it is the
smallest function value in the entire feasible
region. So, the differences over here, so
local optima is the smallest function value
in it is neighborhood; whereas, global optimize
the smallest function value in the entire
feasible region, right.
So, here if we see out of these four points
1 2 3 and 4 right,all the four points are
local optima, right; because they are the
they have the least value in their neighborhood.
Whereas this particular pointis actually lowest
among the entire feasible region. So, if our
feasible region is from here to here for f
of x,for a uni for a single variable optimization
problem; then we can see that the best solution
is over here.
These are alsolocal optima, but this is the
global optima, right. So, if the function
is convex right; only one global optima exists
and there will be no local optima, right.
So, that is why we looked into a convex function,
just to understand what is a global optima.
So, if our function is convex in nature, then
if we are able to find an optima, we do not
need to worry about whether it is global optima
or local optima; because there is only one
optimal solution, right. If the function is
not convex or the function is multimodal;
then most algorithms fail to determine the
global optimal solution that is why non-linear
programming is still a open area of research.
To consolidate whatever we have seen so far;
so we had seen what are the components of
an optimization problem? We have decision
variable, objective function and constraints,
right. So, decision variable can be either
continuous or it can be semi continuous; it
can be integer or a set. Objective functioncan
be continuous or discontinuous. We saw how
we can convert a maximization problem into
a minimization problem or vice versa.
In constraints we saw basically there are
two types ofconstraints;one is in inequality
constraints and other that one is equality
constraints. So, inequality constraints are
of the form g of x is less than equal to 0.
If we happen to have a constraint which is
actually having let us say 3 x squared plus
5 y is greater than 10, greater than or equal
to 10; then we can multiply that equation
by a minus sign on both sides and the inequality
sign would get reversed, right. So, that is
what we saw as components of optimization
problem.
Then we looked into certain features about
a optimization problem; we saw what is boundary
region, what is a feasible solution, what
is an infeasible solution, what are realizations.
Then we looked into what are a unimodal functions
and multimodal function; we saw what is a
convex function. And then we also tried to
understandwhat is a local optima and the global
optima.
The global optimize what we are interested;
because we are interested in the value of
the objective function,which is the least
in the entirefeasible region. Now, let us
look into how do we classifyoptimization problems,
right. This is a typical optimizationproblem,
right.
So,let us say we have an objective function
which involves x which stand for the continuous
variable and y which stands for a binary variables
or integer variables. So, even if we have
sets that can be converted into a integer
variable, right. So, basically if we have
a objective function which involves both the
continuous variable and binary variable. Similar
the we have constraints gwhich involve the
continuous variable and binary variable, and
inequality constraints. So, this is inequality
constrains, this is equality constraint.
And then we have a bounds for each of the
variables. So, x belongs to the real domain
or we can have a bound constrains that a particular
value of x can be between let us say 5 and
10. Let us say if you have two variables x
1 is between x 1 has to be between 5 and 10
and x 2 has to be between let us say 50 and
70, right. So, those are the bound constraint.
Similarly for the integer variables or binary
variables we can have that y can y 1 can take
values1, 2, 3; whereas, y 2 another variable
can take values8, 9, let us say 10, 11, 12
and so on, right.
So, this is a generic representation of an
optimization problem. We have continuous variable,
we have discrete variable, we have inequality
constraints, we have equality constraints,
right. So, here if we seethese three are actually
functions; f of x comma y, g of x comma y
and h of x comma y are actually functions,
right. So, function we can classify as linear
or non-linear.
And decision variable we will just restrict
ourselves to whether it is continuous variable
or integer variable, right. So, we are as
of now we are ignoring for classifying this
problems, we are ignoring the semi continuous
variable. So, variables can be continuous
or discrete right or integer right; discreet
as stands for integer also. The objective
function can be linear or non-linear; the
constraints can be linear or non-linear. So,
depending upon this we have five classes of
problem, right.
Broadly five classes of problem; one is called
as linear programming. So, linear programming
as the name indicates right, the functions
are going to be linear, right. So, the objective
function is going to be linear, the constraints
are going to belinear, right. So, nonlinearity
is not allowed either in the objective function
or in the constraints and since it does not
mention anything about the nature of the decision
variables.
So, when we say linear programming, we are
not refer we are not explicitly specifying
what is the nature of the decision variable.
So, by default it means only continuous variable,
discrete variables are not allowed. So, a
linear programming is a problem in which the
objective function and the constraints are
linear and the variables are continuous; no
non-linearity is allowed in the constraints
or the objective function and the variables
cannot be discrete. So, that is linear programming.
So, similarly non-linear programming the variables
cannot be discreet;we are not explicitly specifying
anything about the nature of the decision
variable in the name, right, so the decision
variable have to be only continuous. The objective
function can be either linear or non-linear,
and the constraints can be either linear or
non-linear, right. So, it is possible thatsome
of the constraints are linear, some of the
constraints are non-linear; but if we have
even one single constraint which is a non-linear,
it falls under the category of non-linear
programming.
For integer linear programming,the name itself
specifies integer right so; that means, only
discreet variables are allowed continuous
variables are not allowed. And since it is
linear programming,objective function and
constraint should be only linear right; nonlinearity
is not allowedin integer linear programming
. Then we have mixed integer linear programming.
So, as the name indicates mixed integer right
so; that means, continuous variables are also
there discrete variables are also there linear
programming, right. So, non-linear objective
function is not allowed, non-linear constraint
is not allowed; the objective function has
to be linear and the constraint have to be
linear. In mixed integer non-linear programming
right,the variables some of the variables
can be continuous, some of the variables can
be integer; the objective function can be
either be linear or non-linear and the constraints
can either be linear or non-linear, right.
So, these are the five categories of problem.
Obviously, you can also have integer non-linear
programming; wherein the objective function
or the constraints are a non-linear in nature.
So, this gives the consolidated picture of
whatever we have discussed. Now that we have
classified problems;there are various ways
to classify optimization techniques, right.
For this course we will classify broadly into
three categories. One is mathematical programming
techniques, metaheuristic techniques and what
we call it as other techniques, right. So,
this is something that we are not going to
see as part of this course. So, in this course
we will be primarily focusing on metaheuristic
techniques, we will also touch up on a mathematical
programming techniques, right.
So, mathematical programming techniques as
based on geometrical properties of the problem.
So, if it isthe algorithm themselves are designedtaking
into account the nature of the problem. So,
for example, for linear programming problem,
we have something called as simplex algorithm.
So, that explicitly exploits the fact that,
the objective function and the constraints
are linear in nature, right.
Similarly non-linear programming make use
of the nature of the function to design the
algorithm. So, some of the non-linear programming
techniques are steepest decent, Newton's method,
Quasi Newton method; for integer programming
problems we have branch and bound and cutting
planes, right.
Interior point algorithm can be used for linear
programming as well as non-linear programming,
right. So, here for this course we will only
be looking at linear programming from the
perspective of mathematical programming techniques.
We will be looking into linear programming
only withsimplex algorithm. And for a mixed
integer linear programming , we will be looking
into a branch and bound.
There are various other mathematical programming
techniques for non-linear programming and
mixed integer non-linear programming, which
you would not be covering in this course,
right. From the perspective of metaheuristic
techniques, so these are nature inspired techniques,
right. So, we will be looking into genetic
algorithm, particles form optimization, differential
evolution, teaching learning based optimization,
artificial bee colony , optimization we will
be looking into this five techniques.
There are; obviously, lot of other techniques
which come up every year right, but we will
be restricting it tothese techniques. So these
techniques do not necessarily exploit the
structure of the problem. So, the way we solve
a linear programming problem using metaheuristic
technique is the same way we will be solving
a non-linear programming problem or a mixed
integer linear programming problem or a mixed
integer non-linear programming problem.
So, these techniques consider the problem
as a black box optimization problem; so we
will look into it in detailas an when westart
looking into metaheuristic techniques. So,
just for the sake of our knowledge, let us
also look into multi objective optimization,
right.
So, in single whatever we discussed so far
was with respect to single objective optimization.
In multi objective optimization we have more
than one objective, when we say more than
one objective; that means, the objectives
are conflicting in nature. Conflicting in
the sense like, if we try toimprovein one
objective, we end up deteriorating the other
objective.
And both the objectives are equally important,
right. So, that is when we have multi objective
optimization, right. So, classical example
fromProfessor Kalyanmoy debs book, right.
So, here you know someone wants to buy a car,
five optionsA, B, C, D, E , right. So, all
the cost of the cars are different and the
comfort that it provides is also different.
So, the cost of car A is somewhere around
1 k in some arbitrary monitory units, and
the comfort level that it provides is 30 percent
right; whereas, car Eis has a cost of a 100
k and the comfort level it provides provides
is 90 percent, right. So, if someone were
to say thattheir objective is to maximizelet
us say the comfort; then obviously, car E
is the best choice, because the solutions
A, B, C, D, E have a lower value ofthe comfort,
right.
So, that is easythat is single objective optimization.
Same way if someone were to say that minimizing
cost is their objective, then it is easy to
choose car A, because it has the lowest cost.
But if someone were to say that maximizing
comfort and minimizing costs is their objective,
then all these five solutions A, B, C, D,
E are equally good, right. So, for example,
if we take a if we comparecar Cand car E,
so car C has a lower cost right, but a lower
comfort.
So, C cannot beeliminated when compared to
car E that is because the cost is lower for
C and we want to minimize the costs; whereas
car E cannot be eliminated, because the comfort
is better than that for car C, right. So,
all these five points are equally good and
we have what is called as pareto solutions,
right or the set of non dominating solutions,
right. So, there is no solution which dominates
these solutions. So, non-dominatingsolutions.
So, similarly if we go back to that previous
example which we had that we want to select
a flight, right. And the objective is to have
the duration of the flight should be minimum
as well as the cost should be minimum that
both we want minimize f 1 and minimize f 2;
f 2 is price and f 1 is let us say duration.
So, for example, these two if we consider
6211, 6211; this takes 2 hours 15 minutes,
and this flight takes 2 hour 45 minutes.
So, this solution can be eliminated, because
we are not compromising on the cost right;
whereas, we have a betterduration, I mean
lower duration with this particular flight.
And if you compare 6411 and 6211; obviously,
this is better in cost; but this solution
has a lower duration, right, so between these
first two flights it is difficult to eliminateany
particular solution. Again this solution can
be eliminated this right, because we have
a solution which says 2 hours 45 minutes and
the cost is 6211; here the cost is also higher,
the duration is also higher.
So, this solution can be eliminated. So, here
if we see again the solutioncan be eliminated,
because we have a higher cost as well as the
duration is longer than these two. So, in
this case we have only two solutions for two
objectives; whereas, in this case in this
car problem even for two objectives, we hadfive
solutions, right.
So, it is not a in multi objective optimization
very often students have this misconception
that if there are two objectives, then there
are two solutions that is not the case, right.
With two objectives you can have even zero
solutions, you can have in finite solutions
in the Pareto front or you can have a discrete
Pareto front defined by a finite number of
points.
So, the search space in multi objective optimization
and single objective optimization. So, in
single objective optimization, let us say
we have a 3 variable problem x 1 and x 2,
x 3 and the shaded region let us say it is
the feasible region. Then any point in this
feasible region actually corresponds to a
scalar value in the objective function in
the objective function space right. So, this
is just a linear line, right. So, it is much
easier to say which solution is better.
So, for example, let us say one solution is
over here and another solution corresponds
to this particular value; if this is f 1 and
this is f 2 and if the objective is to maximize,
then f 2 is better than f 1. So it is easy
to make a make that call, because we are comparing
only two scalar values, all right. In multi
objective optimization every point in this
search space x 1, x 2, x 3 are the decision
variables. So, every point over here actually
corresponds to a pointin a; if we have two
objectives, then it corresponds to a point
in a two dimensional space, right.
So, this point is over here and this point
is over here, right. Here the search processes
little bit more complicated, because we intend
to get the set of non dominating solutions
or the Pareto solutions or the trade of solutions.
So, formally we can define a Pareto solution
as that solution S 1 is said to dominate solution
S 2. So, if you have two solutions S 1, S
2; S 1 is set to dominate S 2, if both the
following conditions are true. So the first
condition is that; S 1 is not worse than S
2 in it any of the objectives, right. So,
it is not definitely bad then S 2 in any of
the objectives; and that S 1 has to be strictly
better than S 2 in at least one of the objectives,
right. So, only then we can say S 1 is dominating
the solution S 2.
So, if we are given these points,again there
is a common misconception that for two objective
functions to be conflicting; one has to be
maximum, the other has to be minimum or it
has to be min max, right. So, that is not
correct; you can have two objective functions
which both need to be maximized and yet they
can beconflicting, right. So, this example
shows you that thing. So, for example, these
are the solutions; these are the solution
S 1, S 2, S 3, S 4, S 5.
What is shown over here is their objective
function values, right; these are not the
decision variable values. For some let us
say if it is a three variable problem;S 1
has let us say 2, 8, 9 ; S 2 has let us say
5, 7, 8. And then we havetwo objective function
f 1 is equal to let us say some relation between
x 1, x 2, x 3; and f 2 is a some relation
between x 1, x 2, x 3, right. So, if we plug
these decision variable values into these
objective function values, we will get f 1
and f 2 for each of the solutions.
So, that is what is shown over here; solution
S 1 has an objective function value of 9 and
an objective function value of 2, similarly
the other solutions how the respective objective
function values. If we compare all the solutions;
so let us say let us compare S 1 and S 2.
So, we want to maximize. So, between S 1 and
S 2 it is difficult to pick a solution, becauseS
1 is better than S 2, right in f 1; whereas
S 2 is better than f 1, right.
So, if we do with 3, the same thing that S
3 is better in f 1, S 1 is better than f 2;
because it has a value of 2. Over here if
we see S 4, completely dominates S 1; because
here we have 11 and 3 and we want to maximize
both of this. So, this S 1 is definitely not
a part of the Pareto solutions, because there
is this solution which will definitelyout
rank f 1. So, we would never choose S 1 if
we have S 4, all right. So, now, that we have
ruled out S 1, let us look at S 2, right.
So, between S 2 and S 3,we cannot make a call;
because S 3 is better than f 1, better in
f 1 and S 2 is better inf. Let us compare
with S 4, so S 2 does not lose out to S 4
also; because we have a 5 over here and the
3 over here, right. Between S 2 and S 5, again
S 2 does not lose out, right. So, S 2 is definitely
going to be the part of a Pareto front. So,
similarly you can do the other calculation,
right.
So, S 3 will not lose out to S 4,because of
this 12 being better right; but S 3 will lose
out to S 5, because we have a 16 which is
better than 12 and we have a 2 which is better
than 1. So, S 3 loses out to this. Now we
can compare S 4 and S 5, right. So, 11 and
16; 3 and 2 right, so obviously, S 4 does
not lose out to S 5 right and other comparisons
were made. So, these three solutions form
a part of non dominated points, right.
So for example, here if you see this solution
S 2; it had a poor objective function than
S 1 and S 3, right; because it is 8 and this
is 9 and 12. But still S 2 goes into the Pareto
point; S 1 and S 3 lose out, because this
S 2 has better value of f 2 which is why it
survives, right. So, suppose for example,
if I had to select S 1, then instead of selecting
S 1 I would as well select S 4; because it
is good in both the objectives, right.
So, this is an example which helps you to
establish that the nature of the function
is to be studied before declaring them to
be conflicting or not. So, for example, let
us have this function that f 1 is x cube;
f 2 is x squared and we want to maximize both
this function, right. So if the feasible region
is let us say the domain of x is between 0
and infinity right; then these two variables
then these two functions are not conflicting,
right.
I can ignore one of the objective function
and just solve with respect to the other objective
function; whatever is the optimal solution
I get that, would be the same for this one
also. But if we change the domain of the variables
from instead of from 0 to infinity; if we
do it from minus infinity to 0 right, then
they become conflicting, right.
So, one has to study the nature of the objective
function, the constraint, andthe domain of
the decision variable to determine whether
two objective functions are conflicting or
not right. So, this is just to give you the
difference between single objective and multi
objective optimization. In this course we
will be restricting ourselves to single objective
optimization, right.
So, let us say we have these four cases, right
. We have these two objective functions; f
1 and f 2, if both are to be minimized, right.
So, you have Pareto front would look something
like this. So, thishighlighted partis what
is the Pareto front, right. So, all the solutionsin
the direction of zero, zero would from the
Pareto front, right.
If you areobjective function f 1 is to be
minimized and f 2 is to be maximized, right;
so then we are looking at solutions which
are pointing in the direction of0 comma infinity.
So, here as you can see the Pareto front here
it was continuous, here it can be discreet,
right. So, these points are actually inferior
points you can work out; just like you can
randomly select one point over here and compare,
you will see that these highlighted points
actually dominate the other points, right.
So, if you have a max min, then we are looking
at; so this has to be f 1 right, then we are
looking at infinity comma 0, right points.
So, this direction is the Pareto front. So,
if the both the objectives are to be maximized
right; then we are looking atthe points which
are towardsinfinity comma infinity, right.
Just like we had realizations in single objective
optimization, we can have realization in multiple
objective optimization, right. So, let us
consider two solutions x 1 and x 2 and there
are some let us say there is some a function
f 1 and f 2 which is a relation of x 1, x
2, x 3 and this is also a relation of x 1,
x 2, x 3, right. So, here if we see if these
two solutions are different right; here it
means the x 1 is 5, x 2 is 2, x 3 is 1, right.
So over here it indicates x 1 is 4, x 2 is
3 and x 3 is 5, right.
So, in this case what happened, in this case
it can still happen depending upon what the
objective function which we have, both f 1
and f 2 values are same, right. So, this is
called this realization; these are not trade
off solutions, the objective function is exactly
identical both in x 1 and x 2, right. So,
these are not trade off solutions, but these
are realizations in multi objective optimization.
So, just like we had realizations in single
objective optimization, we can have realization
in multi objective optimization, right. So,
here are two points correspond to the same
value in the objective function space. So,
this is f 1 comma f 2 right. Now let us come
back to a single objective optimization; first
we will see how to find out the optima for
a single variable problem, right.
There is only one decision variable whose
value is to be found right; again we are not
considering the case wherein the constraints
are there. So, we have an unconstrained optimization
problem and single variable problem, right.
So, in that case the maxima or minima is located
at the stationary point. So, the stationary
point you would have come across in yourhigher
secondary education, right.
So, those can be determined by equating the
gradient of the function to 0. So, if it is
a single variable problem;if you have a function
f of x right and if you want to find out the
stationary point of this, then equateequate
the derivative to 0. So, that will give the
stationary point, right. Stationary point
only tells that either a minima or maxima
occurs at that point; you will have to check
the condition of the secondary derivative,
right.
So the second derivative has to be evaluated
at this stationary point. So, if the second
derivative is positive , then it is a minimum;
if the second derivative is negative, then
it is a maximum right; if the second derivative
is 0, then it is a saddle point. So, in for
a single variable we need to look at a higher
derivatives to decide. So, in terms of multi
variable problem, we will have what this Jacobian.
So, the Jacobian has to be equated to zero.
So, we will solve a multi variable problem.
So, we would better understand. And the second
derivative is nothing, but Hessian matrix;
again we will show you that. So, if the Hessian
matrix is positive definite, then it indicates
that the stationary point is minimum; if the
hessian is negative definite, thenit indicates
thatthe stationary point is maximum; and if
it is indefinite, then again it is a saddle
point. So it is not possible to decide whether
at that point the function has a minimum value
or a maximum value.
So, let us consider this single variable problem
f of x is equal x cubed plus 3 x squared minus
6 x, right. So, if you take the gradient of
this, that works out to be 3 x squared plus
6 x minus 6, right. So, we will have two stationary
points. So, now, this has to be equated to
0. So, this is a quadratic equation. So, if
we get, if we solve it we will get two stationary
points, right.
So, the solution of the equation x squared
plus b x plus c equal to 0 is x is equal to
minus b plus or minus root of b squared minus
4 a c by 2 a. So, if we applied this, then
we will get these two values, right. So, x
is equal to 0.73; so a 0.732 is a stationary
point. X is equal to minus 2.732 is a stationary
point, right.
So, these are stationary points. So,with this
information it is not possible to say whether
the function is minima or maxima, right. So,
we need to find out the second derivative,
right. So, second derivative for this is 6
x plus 6, right. Now the second derivative
has to be evaluated at each of the stationary
point 0.732 and minus 2.732, right. So, if
we do if we evaluate the second derivative
for 0.732 will turn out to be 10.392, right.
Since it is positive, it is a minimum right;
whereas the second point minus 2.732 since
it is negative, it turns out it will be a
maxima.
So, we have a minima at 0.732 and we have
a maxima at minus 2.732, right. So, this plot
showsthe variation of x with respect to f
of x, right. So, if we vary x from minus 5
to 5,this is how the function would look like.
And here if we see this value is 0.732 where
you can see it is actually a minima; and here
it will be the function will have a maxima
which is minus 2.732.
So, let us look into a multivariable function,
right. So, here f of x right isinvolvesx 1
as well as x 2 as well as x 3. So, we have
three decision variables. So,the Jacobian
is nothing, but the partial derivative of
f with respect to x 1; partial derivative
of f with respect to x 2 and the partial derivative
of f with respect to x 3. Since we have three
variables; if you have n variablesall the
partial derivatives are to be found, right.
So, here we these are the three equations,
right.
So if you differentiate it, you would getthese
three equations. So, this is our Jacobian
now, right. So, this has to be equated to
zero. So, this if you see it is simultaneous
linear equation, right. So, this can be written
as a 6 x 1 minus 2 x 2 minus 2 x 3 is equal
to 6, right. So, this equation can be written
as 6 x 1 minus 2 x 2 minus 2 x 3 is equal
to 6 and the rest of the two equation also
can be written.
And if we see, it will be a in the form of
linear equation which can be said a x equal
to b, right. So, if we solve this,we will
get the valueswe will get the stationary point.
So, in this case it happens to be 2, 1 and
2, right. So, this is a stationary point.
Again with this we cannot say whether it is
a minima or maxima, right.
So, we need to determine the second derivative
or the Hessian in this case. So, the Hessian
is given by dou x squared f by dou x 1 square
dou square f by dou x 1 dou x 2;all the way
up to dou square f by dou x 1 dou x 1 x. Similarly
we have dou x squared f dou x 2 x 1 dou x
square f dou x 2 square and so on, right.
So, in this case for the three variable problem,we
will have to determine these partial derivatives.
So, this is ah.
So, here we have dou f by dou x 1, dou f by
dou x 2 and dou f by dou x 3, right. So, if
we do dou x squared f by dou x 1 square. So,
this equation if we further differentiate
with respect to x 1, we will get only 6, right.
So dou xdou square f by dou x 1 dou x 2. So,
this equation if we differentiate with respect
to x 2, we will get minus 2 right. Similarly
this one, so dou x 2 by dou x 3. So, dou x
2 is this equation, right. So, we will get
only plus 2, right. So, if we plug that, we
will get this is our hessian matrix, right.
So, now, we will have to see whether it is
positive definite or negative definite.
So, positive definite; if the principal determinants,
so this determinant,this determinant and the
first 3 cross 3 determinant; if all of this
happened to be greater than zero, then it
is a positive definite matrix. So, in this
case it happensthat this is 6, right, so 6
into 4 minus 2 into minus 2 into minus 2,
right. So, that would work out to be 20. In
this case the all these three principle determinants
happen to be greater than zero, right.
So, that is whywe can display at this point
to be , we can declare Hessian to be positive
definite right and a stationary point which
we found is actually corresponding to a minima.
So, againthis example we will give it to you,you
can solve it, right. So, in this casewe have
a two variable problem x 1 and x 2, right.
So, if we do dou f by dou x 1, dou f by dou
x 2 that is our Jacobean if we equated to
zero, right. So, these two equationscan be
satisfied by these three points, these three
are our stationary points, right.
So for these three stationary points we need
to find out; whether the hessian ispositive
definite or negative definite, right. So,
Hessian again is dou square f by dou x 1 square,
dou square f by dou x 2 square, dou square
f by dou x 1 dou x 2, dou square f by dou
x 2 dou x 1. So, we will need this one and
you can again composethe hessian as shown
over here,right. So, if you compose the hessianyou
will get something similar to, we will get
this matrix right and then if you find out
the eigenvalues of thismatrix.
In the previous slide, in the previous problem
we only we sawthe principle we saw that the
principal determinants have to be positive
right, for a hessian matrix to be positive
definite. Or we can use the eigenvalue properties;
that if the eigenvalues of a matrix ispositive,
right. If all the eigenvalues of a matrix
is positive, then the matrix is positive definite.
If all the eigenvalues are negative, then
the matrix is negative definite; if some of
the eigenvalues are positive and some of the
values are negative eigenvalues are negative,
then the matrix is said to be indefinite,
right. So, in this case if you work outif
you find out the eigenvalues or even if you
calculate the determinant of this matrix;
you will see that,you will get a positive
which corresponds which tells that this is
a minima, this particular stationary point
is a minima.
So for the second point again if we plug in
this x 1 x 2 values into this Hessian matrix
right yeah; if you calculate the eigenvalues
right, you will see thatall the both the eigenvalues
are positive. So, again the session is a positive
definite. So, this point corresponds to a
minima right; whereas, the third point,here
if we calculate again plug this x 1 and x
2 into the hessian matrix and if we calculate
the eigenvalues it,one of the eigenvalues
negative and the other is positive.
So, this is indefinite. So, here it is a saddle
point. So, with this information whatever
we have we cannot say whether this is a minima
or a maxima. So, we will be using this conceptwhile
we are looking into regression right. So,
the first derivative has to be equated to
zero; we will get stationary point, at that
stationary point we need to evaluate the second
derivative, right. If the second derivative
is greater than zero for a single variable
problem, it corresponds to a minima; if the
second derivative is less than zero, it corresponds
to a maxima for a single variable problem.
For a multi variable problem, we need to equate
the Jacobian to zero, we will get the stationary
point and we need to evaluate the hessian
matrix, right. So, for the hessian matrix;if
it is positive definite, then the corresponding
stationary pointers minima; if it is negative
definite, the corresponding a stationary point
is maxima.
So, now let us look into the advantages and
disadvantages ofmathematical programming techniques.
So, the advantages of mathematical programming
technique is that, it has a guaranteed optimal
solution for well-structured problems. So,
if the problem is either linear programming,
mixed integer linear programming or quadratic
programming; there are algorithmswhich guarantee
that the solution that we will obtain at the
end of the procedure will be globally optimal,
all right.
So, again here in optimization you need to
remember, we have two things; one is that
we want the best solution that exists in the
search space, and the second thing is we want
to reach that solution as quickly as possible,
right. Given us a search space; obviously,
you canevaluate the objective function at
each and every point and see at which point
it is minima or maxima. But then in a search
space that can be infinite points, right.
So, that is why we do not do an exhaustive
search, we rely on optimization techniques,
right.
So, far these two category of problem, these
three category of problemsthere are algorithms
right, which guarantee the optimality of solution
right. And mathematical programming techniques
are usual are helpful in debottlenecking,
right. So, as in like it can provide reasonable
insight into the solution, as to what is stopping
us from having a better solutionthat can be
inferred from for some problemsusing mathematical
programming techniques, right.
It does not require multiple runswhich is
the case in metaheuristics techniques, right,
the same technique does for the same problem
will have to be run multiple times; because
it involves stochasticity, right. So, that
is not the case in mathematical programming
techniques . Again it requires lower computational
resources for certain classes of problems
and there are no parameters that are to beset
by trial and error, right. So, which is the
case in metaheuristic techniques, right.
So, those are the advantages of mathematical
programming techniques . The drawbacks include
that it has a rigid modeling framework, right.
So, all the constraints have to be expressed
in this form. For example, in sudoku problem
the constraint that we have is that, let us
say if we denote a sell by x 1 1 and it is
neighboring sellers x 1 2; then the constraint
that we have is actually x 1 1 is not should
not be equal to x 1 2, right.
But that constraint is not of this form that
is like this, right. So, x naught equal to
y if we have a problem wherein we have this
constraint that x should not be equal to y;
that has to be transformed into these such
constraints less than equal to or equal to,
only then mathematical programming techniques
can beused, right. So, that is a drawback
of mathematical programming techniques like.
So, they are not naturally amenable to multi
objective optimization problems. So, if you
want to solve a multi objective optimization
problem using mathematical programming technique;
then the same problem has to be solved multiple
times, right. And if it is a combinatorial
problem, as in like if we have a lot of integer
variables;as the size of the problem increases
right the computational time increases exponentially,
right.
So, it can become computationally intensive,
particularly for combinatorial problem. Usually
these techniques are designed to provide one
solution, right. Some of these drawbacks have
been overcome in the softwareslikein the software;
but in general these are the drawbacks of
mathematical programming techniques.
Let us now briefly look intothe four categories
of problem, right. So, linear programming
in linear programming problem; as we discussed
the objective function is linear and the variables
are continuous. So, for linear programming
problems it is guaranteed that the optima
always occurs at the vertex, right. Andlinear
programming problemsusing the simplex and
interior point method can be solved to global
optimality, right. So, that is an advantage
of linear programming problem .
So, this is a linear programming problem thatwe
have right. So, minimize z is equal to 4 x
1 minus x 2. So, our decision variables are
x 1 and x 2 ; the constraint is that x 1 and
x 2; 2 x 1 plus x 2 should be less than equal
to 8, right. So, again we draw this line 2
x 1 plus x 2 is equal to 8. So, we will get
this line;then we cananother constraint that
we have is x 2 less than equal to 5. So, that
leads to this particular linex 2 less than
x 2 equal to 5.
So, when we say x 2 less than 5, any value
below this is the feasible region, right.
So, 2 x 1 plus x 2 less than 8 is any value
less than I mean in this region is actually
a feasible solution with respect to this constraint
right; but with respect to this constraint
some of the feasible region is lost. Then
we have this constraint x 1 minus x 2 is less
than equal to 4. So, again we will draw the
line x 1 minus x 2 equal to 4, so we get this
line, right.
And then we have these variables that x 1
should and we have the bound constraint that,
x 1 should be greater than equal to 0, x 2
should be greater than equal to 0, right.
So, this lines we obtain, right. So, this
x 1 is equal to 0, so the region above it
and x 2 greater than 0, right, so this this
region, right. So, if we look at all the constraints.
So, this shaded region is actually the feasible
region, right. So, for linear programming,it
can be it has been theoretically established
that the optima always occurs at the vertex.
So, the optima is either here, here, here
or here of this four points. So, there is
no point searching at the interior point,
only these four pointshave the optimal; one
of these four points have the optimal solution.
This is our objective function line 4 x minus
x 2. So, if we equate 4 x minus 4 x 1 minus
x 2 as let us say minus 5 some value minus
5, then this is that line. So, throughout
this line the objective function value is
minus 5; throughout this particular line the
objective function value is 0, right. So,
these are known as Isocost lines, right. So,
throughout this line z 1 z is minus 5, right.
So, that is an Isocost line;throughout this
particular line z is equal to 0. In this case
the optima occurs at this particular point,
right. So, the other four points if you want
you can evaluate;you will see that ; the for
the other four points the objective function
value will work out to be 0 for this one,
16 for this one,again one for this point for
this vertex and minus 5, right. So since our
objective function is to minimizethis vertex
has the best possible value for the objective
function.
Coming to integer linear programming; so in
integer linear the programming the decision
variables arescalars and integers, like we
can have multiple decision variable, but all
of them are scalars right, but all of them
are scalars. So, the objective function and
constraints are supposed to be linear over
here. So, here we have a linear programming
problem, right. So, we want to minimize this
function 3 x 1 plus 2 x 2 that is our objective
function right; and the constraints are 4
x 1 plus 2 x 2 should be greater than equal
to 5; 2 x 2 is less than equal to 5; x 1 minus
x 2 should be less than equal to 2.
Again we convert all these constraints into
equality and we can plot all these lines,
right. So this is the feasible region right;
but because of this constraint that, x 1 comma
x 2 belongs to integer variables not real
variable. So, not all values of x 1 and x
2 are acceptable, only the integer values
of x 1 and x 2 are acceptable. So, that is
why we have a . So, because of thatthe feasible
points are only these six right; because other
points do not have integer values of x 1 or
x 2.
So, only these values are permissible values,
right. So, it might seem thatthe search spaces
smallerin case of integer variables; but integer
programming problems are difficult to solve,
because we do not have the property of continuity
of the decision variables. So, most integer
linear programming problemscan be solve to
global optimality given sufficient time, right.
But there are problems who sizes if the number
of decision variables is too large, then it
is difficult to solve in reasonable time,
right. But theoretically given sufficient
time these integer linear programming problems
can be solved to global optimality. So, the
common algorithms which are used forinteger
linear programming are branch and bound and
cutting planes.
So, this is an example for mixed integer linear
programming problems, right. In the previous
slide we saw that all the variables were integer;
over herethe objective function and constraints
are linear, at least one decision variable
should be integer, right and at least one
decision variable should be continuous that
is why we have thismixed, right. So, these
are the constraints, this is an example, right.
So, in this case all the three variables are
greater than or equal to 0; but x 3 can take
only integer value.
So, x 1 and x 2 can take real values, but
x 3 is an integer value. So, this is an example
of mixed integer linear programming problem.
Again mixed integer linear programming problem
can be solved to global optimality given sufficient
amount of time. So, mixed integer linear programming
problems are solved as a series of LP problems
and the algorithms that we commonly use are
branch and bound and cutting planes.
So, non-linear programming problems; in non-linear
programming problemseither the objective function
or at least one of the constraints are both
are non-linear, right. So, that the algorithms
which are commonly used for solving non-linear
programmings are; successive linear programming,
quadratic programming, successive quadratic
programming or generalized reduced gradient
method, right. There are many other algorithms,
we have this listed a few of them over here,
right.
So, here we have an objective function minimize
the f of x. We have m 1 constraints, m 1 inequality
constraints. So, j is equal to 1, j is equal
to 2, j is equal to 3, it can go all the way
up to m 1, right. And we havem 2 equality
constraints, right. So, we can have constraints
which is 3 x 1 squared plus x 2 is equal to
5 a 3 x 3 plusx 4is equal to 5. So, this is
a non-linear constraint, this is a linear
constraint, right. So, againinequalities are
allowed, equalities are allowed in non-linear
programming problem.
This is an example ofnon-linear programming
problem. In fact, this is a quadratic programming
problem, right. So, we have, we want to maximize
x 1 x 2 subject to the constraint that 2 x
1 plus 2 x 2 is less than equal to 16, right.
So, this is a linear constraint, the objective
function is quadratic right; and x 1 and x
2 should begreater than or equal to 0.
So, we can again draw this line. So any lineany
region below this is the feasible region.
So, the shaded part is the region. So, these
three lines show the value of the objective
function right so; but for this line since
the problem is to maximize even though z is
equal to 35 is a better value than z is equal
to 16, it is not in the feasible region, right.
Whereas, this z is equal to 16 is in the feasible
region, it touches this point, right. So,
that is the optimal solution in this case,
right.
So, again here we are not looking intohow
we are solving, how we are going to solve
a non-linear programming right; here we are
just introducing you to what is a non-linear
programming problem. So in a non-linear programming
problem global optimality is guaranteed only
if the problem is convex or it has some special
properties, right.
So, in a mixed integer non-linear programming
either the objective function or at least
one of the constraint is non-linear. Similar
to mixed integer linear programming there
is a either at least one a continuous variable
and one at least one discrete variable. So,
this is an example of an MINLP, right. So,
it involves two variables x 1 and x 2that,
both x 1 and x 2lie in the region 0 to 10;
but x 2 is also aninteger, right, so continuous
values for x 2 is notallowed.
So, mixed integer linear programming problems
can are usually solved as a series of NLP,
right. For an arbitrary mixed integer linear
programming, there are no algorithms which
can you guarantee that the solution which
has been found is globally optimal solution.
So that is why MIN, some of the algorithms
which are used to solve mixed integer linear
programming or outer approximation and branch
and bound.
So, the other class of techniques is metaheuristic
techniques, right. So, most of these techniques
arenature inspired; they do not require any
information about the physics of the problem.
So, it is more like a black box communication,
right. So, in this case the problem need not
be potion the conventionally equality or inequality
form, which was a drawback in the mathematical
programming techniques right; here we do not
have that requirement, right.
So, it is suitable to solve problems which
are multimodal,they have a large number of
decision variable and continuous , large number
of decision variable or constraints; and some
of the functions involved in the problem are
discontinuous. So, it does not give guarantee
on the optimality of the solution; but usually
it gives a satisfactorysolution especially
for problems which are difficult to be solved
by a conventional method.
Over here we buy conventional method, we need
the mathematical programming techniques and
it can also be used to solve black box optimization
problems, right. So, in black box optimization
problem you do not have a mathematical formulation;
but given a solution you have some kind of
mechanism wherein the quality of the solution
can be determined, right. So, those are black
box optimization problems.
So, for black box optimization problem, it
is not possible to use mathematical programming
techniques , most of the times we rely on
metaheuristic techniques. So, this is a Goldberg's
view. So, Goldberg was one of the pioneer
who worked in metaheuristic techniques, right.
So, these are the various types of problem
a unimodal problem, multi model. So, we have
seen, what is unitmodal problem, multi model
problem; combinatorial problems are those
which involve large number of integer variables,
right. So, this is problem type on the x axis;
over here we have efficiency of the technique,
right.
So if we employ some random scheme right,
it is going to perform poorly, right. So,
this is not a very high efficiency, it is
going to perform very poorly on all the problems,
right. Whereas, if we used specialized schemes;
so for example, linear programming simplex
method that is a specialized scheme. So, it
is going to work extremely well for unimodal
problems or that particular class ofproblems
right; but over ah, but otherwise it is going
to have a very poor performance, in fact performance
a poor than any random scheme.
So, by random scheme we mean you give an optimization
problem ; let us say we randomly decide thatthese
are the values of x 1, x 2 and x 3 the decision
variables and it happens to satisfy the constraints,
right. So, that is a random scheme right;whereas
this metaheuristic techniques are known as
robust scheme, right, so it does not have
a very high efficiency. Similar to a specialized
scheme, let us say for unimodal problems;
but across a wide range of problems it can
be used, right.
So, that is the advantage of metaheuristic
techniques. It does not give guaranteeeven
if for a special category of problem; but
overall it gives a satisfactory performance
for most of the optimization problems, right.
So, our objective in this course is to havea
knowledge of metaheuristic techniques as well
as some of the specialized scheme, right.
So, if we have a problem which actually falls
into let us say linear programming or mixed
integer linear programming; we would not choose
metaheuristic techniques, if we are aware
ofthe specialized schemes, right .
When I metaheuristic technique is designed,
it is usually tested on what is called as
benchmark functions, right. So for this benchmark
functions the optimize already known. So,
given any procedure if we want to save whether
the scheme is working or not, right. So, what
we will do is, we will test it on thisbenchmark
functions, right. So, since the optimize already
known for the benchmark functions; if the
scheme is able to find the optimal solutions,
then the scheme or the technique is said to
be a reasonably good technique, right.
So, some of the benchmark functions that we
will be referring in this course islisted
over here. So, the spear function; so sphere
function, the objective function is f is equal
to x 1 squared plus x 2 square let us say,
for a two variable problem x 1 and x 2. So,
this is how the objective function will behave
in the search spacebetween minus 10 and 10
, right for x 1 and x 2. So, this is the objective
function, right.
So, these are called a scalable function.
So, I can also have x 3 square, x 1 squared
plus x 2 plus x 3 square; this figure is then
no longer valid, but in comp, but the objective
function for sphere function is written like
this. So, these are called a scalable function,
we can make this objective function for two
variable problem, three variable problem,
hundred variable problem, thousand variable
problem, right.
So, that is what this d. So, this d determines
the number of decision variables. So, that
has to be fixed bythe user, right. So, by
fixing the d, we can test a metaheuristic
techniques whether they are able to determine
the optimal solution or not, right. So, this
is another function Rosenberg function, right.
So, the objective function here is as given
over here, right.
So here if we have a let us say two variable
problem, so the objective function is 100
x 2 minus x 1 the whole square plus x 1 minus
1 the whole square , right. So, this summation
does not come into picture if we have two
variable; but if we had three variable, then
it will be this right plus 100 x 3 minus x
2 the whole square plus x 3 minus 1 the whole
square x 2 minus 1 the whole square and so
on.
So, again by changing this d, we can actuallyhave
a objective function of multiple variables,
right. So, this is. So, here if you go into
this link, you will be able to see all these
functions right and MATLAB codes for all these
functions are available. So, if you want to
test a particular technique as we will do
in the later half a later part of the course;we
canuse this functions to testthe efficiency
of a particular techniques. So these are some
of the other commonly used benchmark functions,
right.
So, this shows the contour plots, right. So,
Rastrigin function you can have a look at,
it is objective functiongiven in the previous
slide. So, x 1 and x 2 this is how the search
space looks like; remember the contour plots
are lines of objective function which have
the same objective function value, right.
So, similarly over here for this function
if you see,all the points in this particular
on this particular curve have a similar objective
function, right.
So, the our task is to find outthe best value
of x 1 and x 2 in this region. So, as you
can see sometimes this regioncan be really
complex, right. So,a random scheme would not
be able to find out the optimal solution in
this complexregions, right. So the metaheuristic
techniques that we will discuss as part of
this course would be able to find out the
globally optimal solution even with this complex
functions.
These are some of the functions. So, over
here if we see this is the name of the function;the
actual function has given d denotes the how
many dimension, this is the range of the problem.
So, if you have let us say three variable
x 1, x 2, x 3; what is their domain? So, the
domain here is minus a 100 to 100 for all
the three variables . So, again in the later
half of the course we will testthe optimization
techniques that we study on these problems.
So, for most of this problem if you see the
optimize actually 0; the objective function
value is 0, except in a few cases. So, these
are the benchmark functions, right.
So, very oftenthe techniques are also tested
on engineering problems, right. So,right now
we just want you to be aware that these problems
exist, right. As and when required in the
course we will be using them, right. So, these
are some of the constraintproblems, right.
So, far weldedbeam, these are the equations.
So, these are already available. So, the task
for the technique is to find out the optimal
value such that whatever that function is
either minimum or maximum.
For metaheuristic techniques there are standardconference,
there are prestigious conferences. So, for
example, CEC, GECCO all these are names of
special, all these are names ofreputed conferences.
So, every yearwhen these conferences are conducted,
they have competition. So, they give out problems,not
the benchmark functions which you saw previously;there
are some drawbacks with the benchmark functions
which you have seen previously. We will discuss
it later, right.
But these every yearas part of these conferences;
thisbenchmark functions are given to the user.
So, a user is supposed to design techniques
and demonstrate its performance on these problems;
these are available for most of the programming
languages like C, MATLAB the functions are
already available, you can you do not need
to code the objective function as such,you
just need to evaluatewhatever technique you
have proposed on these functions.
So, this is some of the recent literature
right. So, metaheuristic techniques arehot
area of research, right. So, all these are
papers which are published in 2019-20, right.
So, all this is 2020. So, this is again a
new algorithm, new metaheuristic algorithms;
so bio inspired right, where the name of the
algorithm is manta ray foraging optimization,
this is social mimic optimization algorithm,
this is squirrel search algorithm, this is
based on squirrel search algorithm, right.
And then there are algorithms which are modifiedoften,
right. So, for example, bat algorithm already
exists. So, this paper proposes a hybridizingbat
algorithm with differential evolutions. So,
we will be looking into differential evolution
as part of this course, right. So, we will
be looking into teaching learning based optimization,
right. So, over here in this work they have
a hybridized with the neural network for engineering
design of optimization problem .
These are a couple of examples for newly proposed
multi objective optimization. So, for example,
here if you see it is a multi-objective artificial
sheep algorithm. So, this algorithm has been
probably designed based on the behavior ofsheep's,
right. Here again we have multi objective
firefly algorithm. So, you might know about
fireflies, right. So, depending upon their
behavior, this multi objective algorithm has
been designed.
So the point that I am trying to reinforce
is; once you learn this 4 or 5once we learned
few of the metaheuristic techniques, you will
be in a position to actually understand many
of this papers right and critically review
this work and also possibly propose your own
algorithm.
So, these are some of the classicaloptimization
books, right. So, this book particularlyhelps
in developing models, right. So, we will be
touching upon developing models in this course
ah; but greater details are available in this
book, Model Building in for mathematical programming.
This is a classical book by TAHA Operation
Researchthis is again a standard textbook
for a engineering optimization by S S RAo;this
one is by Reklaitis and coworkers, right.
So, this is Goldberg's book on a genetic algorithm;differential
evolution by Rainser Storm, right. And this
is another classical book by Kalyanmoy Deb,
right. And we also have thisoptimization of
chemical processes. So,since this course has
also been listed underchemical engineering;those
are few who are chemical engineers can actually
look into this book wherein you have examples
fromchemical engineering.
Againwhatever we are going to discuss as part
of this coursedoes not necessarily borrow
any concepts from any particular engineering
field right; whatever we are discussing is
going to be very generic, you can apply it
to your own area. With that we will be concluding
this session; in this session we have seen
components of optimization problem, classification
of optimization problems as well as classification
of optimization techniques.
Then we looked into multi objective optimization
right; in this course we are not going to
see how to solve a multi objective optimization
problem, but at least broadly you know the
difference between single objective optimization
and a multi objective optimization. We followed
this up with how to solve a single and multi-variable
optimization problem in the absence of constraints,
right. So, after that we looked into metaheuristic
techniques, right.
So, we just give you a brief introduction
to metaheuristic techniques and the major
portion of this course is going to be on a
metaheuristic techniques, right. So, the next
session is going to be on a linear regression,
right. So, the linear regression we have divided
into three sessions, right. So, first we will
be looking atlinear regression, simple linear
regression; then we will be extending that
to solve polynomial regression problem and
multi linear regression problem.
And we will also look into general linear
least squares, right. So, the final session
would be on how to use the inbuilt function
of MATLAB to solve linear and non-linear regression
problems. So, for those of you who are not
interested in learning MATLAB, you can skip
that particular session, right. So, the cuisine
assignment will not have anything from that
particularimplementation session, right. So,
with that we conclude the session.
Thank you .
