Lecture 2: Branch-and-Bound Method

(3 units)

Outline

I Linear program

I Simplex method

I Dual of linear programming

I Branch-and-bound method

I Software for integer programming

1 / 32

Linear Program

I Recall that the standard form of LP:

min cT x

s.t. Ax = b

x ≥ 0

where c ∈ Rn, A is an m × n matrix with full row rank,b ∈ Rm.

I A polyhedron is a set of the form {x ∈ Rn | Bx ≥ d} for somematrix B.

I Let P ∈ Rn be a given polyhedron. A vector x ∈ P is anextreme point of P if there does not exist y , z ∈ P, andλ ∈ (0, 1) such that x = λy + (1− λz).

2 / 32

Basic Solutions and Extreme Points

I Let S = {x ∈ Rn | Ax = b, x ≥ 0}, the feasible set of LP.Since A is full row rank, if the feaisble set is not empty, thenwe must m ≤ n. WLOG, we assume that m < n.

I Let A = (B,N), where B is an m ×m matrix with full rank,i.e., det(B) 6= 0. Then, B is called a basis.

I Let x =

(xB

xN

).We have BxB + NxN = b. Setting xN = 0 ,

we have xB = B−1b. x =

(B−1b

0

)is called a basic

solution. xB is called basic variables, xN is called nonbasicvariables.

I If the basic solution is also feasible, this is, B−1b ≥ 0, then

x =

(B−1b

0

)is called a basic feasible solution.

3 / 32

Basic Solutions and Extreme Points

I x ∈ S is an extreme point of S if and only if x is a basicfeasible solution.

I Two extreme points are adjacent if they differ in only onebasic variable.

I (Basic Theorem of LP) Consider the linear programmin{cT x | Ax = b, x ≥ 0}. If S has at least one extremepoint and there exists an optimal solution, then there existsan optimal solution that is an extreme point.

I Proof (representation of polyhedron)

I The feasible set of standard form linear program has at leastone extreme point.

I Therefore, we claim that the optimal value of a linear programis either −∞, or is attained an extreme point (basic feasiblesolution) of the feasible set.

4 / 32

A naive algorithm for linear programming

I Let min{cT x | Ax = b, x ≥ 0} be a bounded LP

I Enumerate all bases B ∈ {1, . . . , , n},(

mn

)= O(nm) many

I Compute associated basic solution x =

(B−1b

0

)

I Return the one which has largest objective function valueamong the feasible basic solutions

I Running time is O(nm ·m3)!!!

I Are there more efficient algorithms?

5 / 32

Simplex methodI Simplex method was invented by George Dantzig

(1914–2005) (father of linear programming)

I Suppose we have a basic feasible solution x =

(B−1b

0

),

A = (B,N).I Let x ∈ S be any feasible solution of the LP. Let

x =

(xB

xN

), c =

(cB

cN

). Then BxB + NxN = b and so

xB = B−1b − B−1NxN

cT x = cTB xB + cT

N xN

= cTB B−1b − cT

B B−1NxN + cTN xN

= cT xT + (cTN − cT

B B−1N)xN

I Let rN = cTN − cT

B B−1N (called reduced cost). If rN ≥ 0, thencT x ≥ cT xT and the current extreme point x is optimal

6 / 32

Simplex method

I Otherwise, there must exist an ri < 0, we can let currentnonbasic variable xi become a basic variable xi > 0 (enteringvariable).

I Suitably choosing basic variable to become a nonbasic variable(leaving variable), we can get a new basic feasible solutionwhose objective value is less than that of the current basicfeasible solution x .

I Geometrically, the above method (simplex method) movesfrom one extreme point to one of its adjacent extreme point.

I Since there are only a finite number of extreme points, themethod terminates finitely at an optimal solution or detectsthat the problem is infeasible or it is unbounded.

7 / 32

Geometry of Simplex Method

Figure: Simplex method

8 / 32

Simplex method

I Step 0: Compute an initial basis B and the basic feasible

solution: x =

(B−1b

0

).

I Step 1: If rN = cTN − cT

B B−1N ≥ 0, stop, x is an optimalsolution, otherwise go to Step 2.

I Step 2: Choose j satisfying cTj − cT

B B−1aj < 0, if

aj = B−1aj ≤ 0, stop, the LP is infinite; otherwise, go to Step3.

I Step 3. Compute the stepsize:

λ = min{ bi

aij| aij > 0} =

br

arj≥ 0.

Let x := x + λdj , where dj =

( −B−1aj

ej

). Go to Step 1.

9 / 32

Simplex Tableau

xB xN rhs

B N b

cTB cT

N 0=⇒

xB xN rhs

I B−1N B−1b

0 cTN − cT

B B−1N −cTB B−1b

10 / 32

Simplex method: An example

I Consider the following LP:

min − 7x1 − 2x2

s.t. − x1 + 2x2 + x3 = 4,

5x1 + x2 + x4 = 20,

2x1 + 2x2 − x5 = 7,

x ≥ 0.

I The initial tableau is

x1 x2 x3 x4 x5 rhs

−1 2 1 0 0 45 1 0 1 0 202 2 0 0 −1 7

−7 −2 0 0 0 0

11 / 32

I Choose the initial basis: B = (a1, a3, a4), basic variable:xB = (x1, x3, x4)

T . The simplex tableau is:

x1 x2 x3 x4 x5 rhs

0 3 1 0 −12 71

20 −4 0 1 21

2 212

1 1 0 0 −12 31

2

0 5 0 0 −72 241

2

I The basic feasible solution is:xB = (x1, x3, x4)

T = (312 , 71

2 , 212)T . Since −7

2 < 0, choose x5

as the entering variable, λ =2 1

2

2 12

= 1, so x4 is leaving variable,

the new basic variable is xB = (x1, x3, x5). The new tableau is

x1 x2 x3 x4 x5 rhs

0 115 1 1

5 0 80 −8

5 0 25 1 1

1 15 0 1

5 0 4

0 −53 0 7

5 0 28

12 / 32

I Choose x2 as the entering variable, computeλ = min{ 8

11/5 , 41/5} = 40

11 . x3 is the leaving variable, the newtableau is:

x1 x2 x3 x4 x5 rhs

0 1 511

111 0 40

110 0 8

11611 1 75

111 0 − 1

11211 0 36

11

0 0 311

1611 0 30 2

11

Since rN = ( 311 , 16

11) ≥ 0, the current basic feasible solutionx = (36

11 , 4011 , 0, 0, 75

11)T is the optimal solution with optimalvalue: −30 2

11 .

13 / 32

Dual theory: motivation

I Consider the following minimization problem with a singleequality constraint:

(P) min f (x)

s.t. h(x) = 0,

x ∈ X ,

where h(x) = 0 is a hard constraint and x ∈ X is a “easy”constraint.

I How could we solve this problem? The idea is to relax thehard constraint h(x) = 0 and solve an easier problem.

I This can be done by incorporating the constraint into theobjective function. A price is associated if the constraint isviolated.

14 / 32

I Given a Lagrangian multiplier λ (price), the Lagrangian dualfunction defined by

d(λ) = minx∈X

L(x , λ) := f (x) + λh(x)

gives a lower bound for (P):

d(λ) ≤ f (x), ∀ feasible solution of(P).

I The best lower bound is found by solving

(D) maxλ

d(λ).

I (D) is called the Lagrangian dual problem of (P).

15 / 32

Dual of Linear Program

I Consider the standard LP:

(LP) min cT x

s.t. Ax = b, x ≥ 0,

I Associate multipliers λ ∈ Rm to Ax = b, the dual function is

d(λ) = minx≥0

{cT x + λT (b − Ax)} = bTλ + minx≥0

(c − ATλ)T x

=

{bTλ, if ATλ ≤ c−∞, otherwise

I So, the dual problem is

(LD) max bTλ

s.t. ATλ ≤ c .

16 / 32

Inequality ConstraintsI Consider the standard LP:

(LP) min cT x

s.t. Ax ≥ b, x ≥ 0,

I Adding slack variables s ≥ 0, we get

(A,−I )

(xs

)= b.

I This leads to dual constraints

(A,−I )Tλ ≤(

c0

).

I Hence, the dual of LP is:

(LD) max bTλ

s.t. ATλ ≤ c

λ ≥ 0. 17 / 32

Primal and Dual Forms

We can dualize general LPs as follows:

(LP)

∣∣∣∣∣∣∣∣

min cT x

s.t. Ax≥≤=

b, x≥≤

free0⇔(LD)

∣∣∣∣∣∣∣∣

max bT y

s.t. AT y≤≥=

c , y≥≤

free0

Primal (Minimize) Dual (Maximize)

≥ bi ⇔ ≥ 0Constraints ≤ bi ⇔ ≤ 0 Variables

= bi ⇔ free

≥ 0 ⇔ ≤ cj

Variables ≤ 0 ⇔ ≥ cj Constraintsfree ⇔ = cj

18 / 32

Branch-and-Bound Method

I Branch-and-bound strategy:I Solve the linear relaxation of the problem. If the solution is

integer, then we are done. Otherwise create two newsubproblems by branching on a fractional variable.

I A node (subproblem) is not active when any of the followingoccurs:(1)The node is being branched on;(2) The solution is integral;(3) The subproblem is infeasible;(4) You can fathom the subproblem by a bounding argument.

I Choose an active node and branch on a fractional variable.Repeat until there are no active subproblems.

19 / 32

Branch-and-bound search tree:

20 / 32

I Example. Consider the following 0-1 knapsack problem:

max 8x1 + 11x2 + 6x3 + 4x4

s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 14

x ∈ {0, 1}4.

I The linear relaxation solution is x = (1, 1, 0.5, 0) with a valueof 22. The solution is not integral.

I Choose x3 to branch. The next two subproblems will havex3 = 0 and x3 = 1 respectively.

21 / 32

I The search tree is:

I The linear relaxation solutions to the two subproblems areI x3 = 0: x = (1, 1, 0, 0.667), objective value=21.65;I x3 = 1: x = (1, 0.714, 1, 0), objective value=21.85.

I At this point we know that the optimal value is no more than21.85 (we actually know it is less than or equal to 21. but westill do not have any feasible integer solution. So, we will takea subproblem and branch on one of its variables.

22 / 32

I Choose the node with x3 = 1 and branch on x2. The branchtree is:

I The linear relaxation solutions areI x3 = 1, x2 = 0: x = (1, 1, 1, 1), objective value=18;I x3 = 1: x2 = 1: x = (0.6, 1, 1, 0), objective value=21.8.

23 / 32

I Choose node with x3 = 1, x2 = 0 and branch on x1. Thesearch tree is

I The linear relaxation solutions areI x3 = 1, x2 = 0, x1 = 0: x = (0, 1, 1, 1), objective value=21;I x3 = 1: x2 = 1: x1 = 1: infeasible.

I The optimal solution is x = (0, 1, 1, 1).

24 / 32

Solving 0-1 Knapsack by B&B

I 0-1 Knapsack Problem

(01KP) max{cT x | aT x ≤ b, x ∈ {0, 1}n}.

I To solve the LP relaxation of (01KP), you need only begreedy! Assume that the variables have been ordered suchthat

c1/a1 ≥ c2/a2 ≥ · · · ≥ cn/an. (1)

Let s be the maximum index k such that

k∑

j=1

aj ≤ b. (2)

25 / 32

I Theorem (Dantzig (1957)): The optimal solution to thecontinuous relaxation of (01KP) is

wj = 1, j = 1, . . . , s,

wj = 0, j = s + 2, . . . ,N,

ws+1 = (b −s∑

j=1

aj)/as+1.

I If cj , j = 1, . . . ,N, are positive integers, then an upper boundof the optimal value of (LKP) is given by

UB =s∑

j=1

cj + b(b −s∑

j=1

aj)cs+1/as+1c, (3)

26 / 32

How to Branch?

I We want to divide the current problem into two or moresubproblems that are easier than the original. A commonlyused branching method:

xi ≤ bx∗i c, xi ≥ dx∗i e,where x∗i is a fractional variable.

I Which variable to branch? A commonly used branching rule:Branch the most fractional variable.

I We would like to choose the branching that minimizes thesum of the solution times of all the created subproblems.

I How do we know how long it will take to solve eachsubproblem?Answer: We don’t.Idea: Try to predict the difficulty of a subproblem.

I A good branching rule: The value of the linear programmingrelaxation changes a lot!

27 / 32

Which Node to Select?

I An important choice in branch and bound is the strategy forselecting the next subproblem to be processed.

I Goals: (1) Minimizing overall solution time. (2) Finding agood feasible solution quickly.

I Some commonly used search strategies:

I Best FirstI Depth-FirstI Hybrid StrategiesI Best Estimate

28 / 32

The Best First Approach

I One way to minimize overall solution time is to try tominimize the size of the search tree. We can achieve this bychoosing the subproblem with the best bound (lowest lowerbound if we are minimizing).

I Drawbacks of Best FirstI Doesn’t necessarily find feasible solutions quickly since feasible

solutions are “more likely” to be found deep in the treeI Node setup costs are high. The linear program being solved

may change quite a bit from one node evaluation to the nextI Memory usage is high. It can require a lot of memory to store

the candidate list, since the tree can grow “broad”

29 / 32

The Depth First Approach

I The depth first approach is to always choose the deepest nodeto process next. Just dive until you prune, then back up andgo the other way

I This avoids most of the problems with best first: The numberof candidate nodes is minimized (saving memory). The nodeset-up costs are minimized

I LPs change very little from one iteration to the next. Feasiblesolutions are usually found quickly

I Drawback: If the initial lower bound is not very good, then wemay end up processing lots of non-critical nodes.

I Hybrid Strategies: Go depth-first until you find a feasiblesolution, then do best-first search

30 / 32

The nodes expanded in depth-first branch-and-bound search:

31 / 32

Software for integer programming

I Matlab code: bintprog for solve binary integer linearprogramming problems.

I Commercial optimization software CPLEX and Gurobi forsolving mixed integer linear and quadratic programmingproblems.

I Two simple ways of calling CPLEX MIP solver:I Optimization Programming Language (OPL) in CPLEX

Optimization Studio.I CPLEX Matlab interface.

I Examples using Matlab and CPLEX

32 / 32