ch8.1-8.3 linear programming and game theorywit.kaist.ac.kr/linear algebra/ch8.1-8.3_linear...

Chap 8. Linear Programming and Game Theory

KAIST wit Lab

2012. 07. 20

유인철

KAIST wit lab

I. Linear Inequalities

KAIST wit lab

Introduction

Linear Programming (from wikipedia)Mathematical method for determining a way to achieve the best

outcome (such as maximum profit or lowest cost) in a given mathematical model for some list of requirements

A technique for the optimization of a linear objective function, subject to some constraints

Objective of this chapter To see the geometric meaning of linear inequalities To find the optimal soloution

Hyperplane and Halfspace Hyperplane : Halfspace :

3

KAIST wit lab

The Feasible Set and the Cost Function

Feasible set the intersection of halfspaces composition of the solutions to

• a family of linear inequalities like Example)

2 4, 0, 0 Fundamental constraint to linear programming

KAIST wit lab

Problem in linear programming Find the point that

• lies in the feasible set• minimizes or maximize the cost

Example : in the previous figure Cost : 2 3Minimum cost : 2 ∗ 3 ∗ 6, ∗ 0 ∗ 2

• Vector (0,2) is feasible and optimal The optimal vector occurs at a corner of the feasible set

Possible categories 1. The feasible set is empty2. The cost function is unbounded on the feasible set3. The cost reaches its minimum (or maximum) on the feasible set

5

The point is feasible The point is optimal

Cost increases

Cost decreases

KAIST wit lab

Slack Variables

Slack Variable To change the inequality to an equation

• Result : equation + nonnegativity constraints Primal problem :

• Constraints : Example : in the previous example

• Inequality : 2 4

• Slack variable : 2 4• Nonnegativity constraint : 0

6

0

KAIST wit lab

The Diet Problem and Its Dual

Diet problem in linear programming 2 sources of protein :

• A pound of peanut butter : a unit of protein• A steak : 2 unit of protein

Diet : • at least 4 units are required• Contains

– x pounds of peanut butter– y steaks

Cost :• A pound of peanut butter : $2• A steak : $3

7

Constraints :2 4, 0, 0

the cost of whole diet :2 3

∴ optimal diet for minimum cost = 2 steaks ( ∗ 0, ∗ 2)

KAIST wit lab

Duality theoremThe minimum in the given “primal problem” equals the maximum in its dual. Every linear program has dual. Example : Dual problem of previous one

8

2, 2 3 0

optimal price : ∗ $1.50 maximum revenue :4 ∗ $6

KAIST wit lab

Typical Applications

1. Production Planning : General motors

Also, the average car must get 18 miles per gallon

What is the maximum profit in 8 hours (480 minutes) ?

9

Chevrolet Buick Cadillac

profit $200 $300 $500

miles per gallon 20 17 14

Assemble duration 1 minute 2 minutes 3 minutes

KAIST wit lab

2. Portfolio Selection

We can buy amounts , , not exceeding a total of $100,000 No more than $20,000can be invested in junk bonds The portfolio’s average quality must be no lower than municipals

What is the maximum interest?

10

federal bonds municipals junk bonds

interest 5% 6% 9%

KAIST wit lab

Problem Set 8.1

Cost : .

Cost : 3• 2 0,2 12 4,0

Cost : • 2 0,2 4 4,0

11

4

2

1

KAIST wit lab

II. The Simplex Method

KAIST wit lab

Introduction

Linear Programming with n unknowns 0m constraints

Minimum problem

Optimal ∗

occurs at the point where the planes first touch the feasible set Compute ∗: Find all the corners of the feasible set

Simplex method• A systematic way to solve linear programs

13

feasible vectors meet conditions

KAIST wit lab

The Geometry : Movement Along Edges

Corner in linear algebra The meeting point of different planes

Possible intersection point of linear programming Review the feasible set : the intersection space of

14

given by 1 equation

0

equations (by slack var.)+

fundamental constraints

equations ( , ⋯ , )+

fundamental constraints

equations, but only equations are needed

Subspace of

KAIST wit lab

How equations are chosen for intersection point? • constraints : 2 6, 2 6, 0, 0

15

2 62 6 ,

0, 00, 0

P : intersection of 0&2 6

⇔ 0

∴ equations 0,⋯ , 0can be chosen out of equations

The number of possible intersection points : C !! !

KAIST wit lab

The intersection point is a genuine corner ifIt also satisfies the remaining inequality constraints.Otherwise, it is a complete fake.

Ex) in the previous figure

16

P : 0& 0 6, 6

∴

J

J : 0& 0 3, 3

∴

KAIST wit lab

Key idea of simplex methods Go from corner to corner along the edges of the feasible set

Edge of the feasible set If one of the intersecting planes is removed, remaining 1

equations form an edge that comes out of the corner

17

P : 0& 0 6, 6

↑ :

↓ :

KAIST wit lab

Newly defined linear programming by slack variable Slack variable : ⇔ Equality constraints and nonnegativity

18

Minimize , subject to , 0

Newly defined

Minimize , subject to , 0

KAIST wit lab

Example 1.

19

KAIST wit lab

The Simplex Algorithm

Free variables vs. Basic variables in constraints Free variables : n components Basic variables (Pivot variables) : m components∵ : ⇔ equations ⇔ pivots

Basic feasible solutions of A solution is basic when of its components are zero,and it is feasible when it satisfies 0 Setting the free variables to zero, basic solution is a genuine

corner if its nonzero components are positive

Phase of simplex method Phase I : find one basic feasible solution Phase II : move step by step to the optimal ∗

20

KAIST wit lab

Which corner do we go to next? 1 basic variables remain basic Only 1 basic variables will become free (⇔ become zero),

while other 1 basic variables will stay positive

Terminology Entering variable :

the free variable which will become basic Leaving variable :

the basic variable which will become free

Ex) in the previous exampleIf P moves to Q, ∶ Entering variable, : Leaving variable

21

KAIST wit lab

Example 2

22

1) 0, 0, 0 8, 9 : genuine

7 3 0

There is possibility that cost can be minimized

2) As is increased, : entering variable

2 83 9

0 40 3 : leaving variable

3) At this point, 7 3 9

KAIST wit lab

Quick Way

Observation of Example 2

The right sides divided by the coefficients of the entering variableRatio : 4, 3

23

0, 0 2 83 9

hits zero first

KAIST wit lab

The next step of Example 2 The current step ends at the new corner 2,0,0,0,3

• Free variable : , ,• Basic variable : ,

Pivot by substituting 9

Repeat Phase II• ∗ 0,0,0, ⁄ , 3 ⁄ 3

24

Leaving variable

Entering variable

KAIST wit lab

The Tableau

Tableau (or large matrix) One way to organize the simplex step into matrix

Applying row operation to tableau Renumbering if necessary

• , ⋯ , ∶ Tableau at corner

25

Cost : Constraints :

∶ , ∶

: 1 , : 1

: 1, : 1

KAIST wit lab

Ex) For Example 2cost : 0 0 7 1 3constraints : 2 4 1

8 2 16 20 3 8

9

26

2 4 18 2 10 0 7

6 2 80 3 91 3 0

KAIST wit lab

Reduced tableauThe basic variables will stand alone when elimination multiplies by

Ex)

27

2 4 18 2 10 0 7

6 2 80 3 91 3 0

1 0114

0 1314

0 0 7

614

414

1014

2414

514

2314

1 3 0

KAIST wit lab

Fully reduced : R=rref(T)Subtract times the top block row from bottom row

Ex)

28

12

2 1

1 0114

0 1314

0 0 7

614

414

1014

2414

514

2314

1 3 0

KAIST wit lab

Meaning of matrix R

Cost :Constraints : At corner 0 : &

29

0

Stopping test (optimality condition)r : coefficients of

KAIST wit lab

Example 3. Figure 8.3MinimizeSubject to 2 6, 2 6

<solve problem using tableau>

30

Slack variable : 2 6 2 6

1 22 1

1 00 1

66, 1 10 0

KAIST wit lab

1) Find initial feasible point Let basis variables(free variable) be zero

31

2 6 2 6 0, 0 6, 6 : fake

Find other feasible corner as initial point

0, 0 6, 6 : genuine

We choose this point as initial one

KAIST wit lab

2) Renumbering Take basic variable(=non‐zeros) firstly

32

∶ →

1 22 1

1 00 1

66, 1 10 0

This means that problem is changed as below

1 20 1

1 02 1

66 , 0 11 0

KAIST wit lab

3) Make tableau and take row operation

33

100

211

121

010

660 , ,

100

011

321

210

660

100

010

321

211

666

6 at point66 ,

00

KAIST wit lab

4) Stopping test

1 1

34

There is possibility that cost can be minimized

100

010

321

211

666

0 0 1 1 →

Let be variable which becomes basic (=non‐zero)

⟺ 3

KAIST wit lab

5) Find the variable which becomes free(=zero)Let : a column of

: the index of smallest ratio 1,⋯ ,

Then, becomes free variable (=zero)

35

100

010

321

211

666

∵

Ratio : 2 1 3 2 1 : becomes zero

KAIST wit lab

6) Find next solution Tableau is reconstructed by changing column and

36

321

010

100

211

666 , ,

101

010

1/32/30

2/31/31

226

100

010

1/32/31/3

2/31/31/3

224

KAIST wit lab

7) Stopping test again

37

100

010

1/32/31/3

2/31/31/3

224

Since 0, this point is optimal.

4

at the point22 ,

00

KAIST wit lab

The Organization of a Simplex Step

Reduced simplex method For computational purposes, only , , are used.

38

1) 0

2)

100

010

321

211

666

321

010

100

211

666

KAIST wit lab

reduce the computation since has the form as below

39

KAIST wit lab

Karmarkar’s Method

2012. 07. 26.

Wonho Kang

40

KAIST wit lab


Definition An iterative algorithm that, given an initial point and parameter

, generates a sequence , , … , which are the solutions of linear program (LP)

Assumption1. LP has a strictly feasible point, and the set of optimal point is

bounded2. LP has a special “canonical” form:

minimize zsubject to 0, ∑ 1, 0

3. The value of the objective at the optimum is known and is equal to zero, ∗ 0

41

KAIST wit lab


Minimize Subject to 0

Minimize ′ ′ , 0 , , 1 Subject to 0 , z 0

Minimize ′ , … , , Subject to 0 y 0

42

⋯ 1 , 1,⋯ ,

1⋯ 1 , 1

KAIST wit lab


Minimize ′ Subject to 0 1 y 0

43

KAIST wit lab


Canonical formminimize zsubject to 0, ∑ 1, 0

Denote nullspace of Ω ∈ | 0

Define simplex∆ ∈ | 1, 0 , 1,… , 1

Denote center of the simplex ∆

,… , , ∈ ∆

44

KAIST wit lab


Rewritten canonical formminimize zsubject to ∈ Ω ∩ ∆

Note that the constraint set (or feasible set) Ω ∩ ∆ can be represented as

Ω ∩ ∆ ∈ | 0, 1, 0Ω ∩ ∆ ∈ | 0

1 , 0

Ω ∩ ∆ ∈ | 01 , 0

45

KAIST wit lab


Procedure1. Initialize: Set ≔ 0; / ;

2. Update: Set Ψ whereΨ is an update map

3. Check stopping criterion: If the condition 2 is satisfied,

then stop where is a given termination parameter;

4. Iterate: Set ≔ 1, go to 2

46

KAIST wit lab


First update step in the algorithm1. Compute the orthogonal projector onto the nullspace of

2. Compute the normalized orthogonal projection of onto the nullspace of

3. Compute the steepest‐descent direction vector, 1/ 1

4. Compute using

where is the prespecified step size, ∈ 0,1Karmarkar recommands a value of ¼ in his original paper

47

KAIST wit lab


General update step1. Compute the matrices

,⋱

,

and

is, in general, not at the center of the simplex, so whose diagonal entries are the components of the vector is used to transform this point to the center

2. Compute the orthogonal projector onto the nullspace of

48

KAIST wit lab


General update step3. Compute the normalized orthogonal projection of onto the

nullspace of

4. Compute the steepest‐descent direction vector

5. Compute using

6. Compute by applying the inverse transformation

49

KAIST wit lab


ExampleMinimize 3 3

subject to 2 3 0 01 1

, , 0

3 3 1 Ω | 0 2 3 1 Δ | 1, 0 1 1 1 3

50

KAIST wit lab


Solution1. Initialize:2. Set ≔ 0;3. 1 1 14. 35. / 1/3 1/3 1/3

6. | 0, 1, 0

7. | 01 , 0

8. | 01 , 0

51

//

//

KAIST wit lab


2. Update: Set Ψ whereΨ is an update map1) Compute the orthogonal projector onto the nullspace of 2 3 3 13 2 3 14 1 1 1

5 2 3 11 1 1

6 | 00

7

816 4 204 1 520 5 25

52

KAIST wit lab


2. Update: Set Ψ whereΨ is an update map2) Compute the normalized orthogonal projection of onto the

nullspace of 3 3 3 1

416 4 204 1 520 5 25

5415

53

KAIST wit lab


2. Update: Set Ψ whereΨ is an update map3) Compute the steepest‐descent direction vector4 1 1 15 3

6 1/ 1 1/ 67 is the radius of the largest sphere8) inscribed in the set9 | 1, 0

54

KAIST wit lab


2. Update: Set Ψ whereΨ is an update map3) Compute the steepest‐descent direction vector

4415

5 1/ 6

6415

55

KAIST wit lab


2. Update: Set Ψ whereΨ is an update map4) Compute

5 1/3 1/3 1/3

6415

70.08140.27030.6483

, 1

3) Note:4 makes be guaranteed5) to lie in the constraint set6 | 0, 1, 07 ∩ |8) (a strictly interior point of the set)

56

KAIST wit lab


3. Check stopping criterion:

4. If the condition 2 is satisfied, then stop where is a

given termination parameter5. 3 3 16. 1/3 1/3 1/37. 0.0814 0.2703 0.6483 cf) ∗ 0 1/4 3/4

8. 0.243 2

9. if 2.04, then stop, or

4. Iterate: Set ≔ 1, go to 2

57

KAIST wit lab


58

-0.5

00.5

11.5

22.5

3

-0.50

0.51

1.52

2.53

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

cc0d0

x(0)

x(1)

KAIST wit lab


0.0810.2700.648

0.0810.270

0.648

1/3 1/3 1/3

Minimize Minimize ′ Subject to 0 Subject to ′ 0 1 ′ 1 0 ′ 0

59

KAIST wit lab


0.1627 0.8110 0.64831 1 1

0.8017 0.2668 0.5349

0.9273 0.1089 0.2184

0.0060 0.4422 0.5517

0.0010 0.2503 0.7487

60

KAIST wit lab


61

-0.5

00.5

11.5

22.5

3

-0.50

0.51

1.52

2.53

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

cc0d0

x(0)

x(1)

KAIST wit lab


62

-0.5

00.5

11.5

22.5

3

-0.50

0.51

1.52

2.53

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

cc1

d1

x'(1)

x'(2)

x(1)x(2)

KAIST wit lab

III. The Dual Problem

KAIST wit lab

The Dual ProblemMinimizeSubject to , 0

Ex) In Section 8.1,

64

MaximizeSubject to , 0

∶ n 1feasible set ∈

y ∶ 1feasible set ∈

Dual

Minimize 2 3 Subject to 1 2 4

Dual

Maximize 4 Subject to 1 2 2 3

KAIST wit lab

Duality theoremWhen both problems have feasible vectors,

the minimum cost c ∗ the maximum income ∗

Weak duality If x and y are feasible in the primal and dual problems, then

(proof)x and y are feasible and

y

65

KAIST wit lab

Reverse of duality theorem If the vectors x and y are feasible and ,

then x and y are optimal

Ex)Minimize 4subject to 2 6, 5 3 7

66

Dual

Maximize 6 7subject to 2 5 1, 3 4

2 15 3 , 6

7 , 1 4

: 3 30

∶ 3 1/20

KAIST wit lab

Complementary slackness condition Relationship between the primal and dual Assume (P) means primal problem and have a optimal solution ∗

and (D) means dual problem and have a optimal solution ∗, then

(proof)

67

1. If ∗ in (P), then ∗ 02. If ∗ in (D), then ∗ 0

∗ c ∗∗ ∗ ∗ ∗ ∗ c ∗

∗ should be always satisfied.However, from this equation, if ∗ 0, ∗

∴ ∗ is ∗ 0

KAIST wit lab

The proof of Duality

68

From simplex method, , by reordering

, ∗

In dual problem,if y is feasible

∴ ⟹ ∗ ⟺ ∗ 0

If ∗ , ∗ 0 optimal

∴ ∗ ∗

ch8.1-8.3 linear programming and game theorywit.kaist.ac.kr/linear algebra/ch8.1-8.3_linear...

Documents