optimization problems

Optimization Problems

虞台文大同大學資工所智慧型多媒體研究室

ContentIntroductionDefinitionsLocal and Global OptimaConvex Sets and FunctionsConvex Programming

Problems

Optimization Problems

Introduction

大同大學資工所智慧型多媒體研究室

General Nonlinear Programming Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

objective function

constraints

Local Minima vs. Global Minima

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

objective function

constraints

local minimum

global minimum

Convex Programming Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

objective function

constraints

f (x)

gi (x)

hj (x)

convex

concave

linear

Local optimality Global optimality

Linear Programming Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

objective function

constraints

f (x)

gi (x)

hj (x)

linear

linear

linear

Local optimality Global optimality

a special case of convex programming problems

Linear Programming Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

objective function

constraints

f (x)

gi (x)

hj (x)

linear

linear

linear

Local optimality Global optimality

Integer Programming Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx Z

objective function

constraints

f (x)

gi (x)

hj (x)

linear

linear

linear

The Hierarchy of Optimization Problems

NonlinearPrograms

ConvexPrograms

LinearPrograms

(Polynomial) IntegerPrograms(NP-Hard)

Flowand

Matching

Optimization Problems

General Nonlinear Programming Problems

Convex Programming Problems

Linear Programming Problems

Integer Linear Programming Problems

Optimization Techniques

General Nonlinear Programming Problems

Convex Programming Problems

Linear Programming Problems

Integer Linear Programming Problems

ContinuousVariables

DiscreteVariables

ContinuousOptimization

CombinatorialOptimization

Optimization Problems

Definitions

大同大學資工所智慧型多媒體研究室

Optimization Problems

( )f xminimize

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

( )f xminimize

Optimization Problems

( ) 0 1, ,ig x i m subject to

( ) 0 1, ,jh x j p nx R

Define the set of feasible points

F

Minimize cost c: FR1

Definition:Instance of an Optimization Problem

(F, c) F: the domain of feasible points

c: F R1 cost function

Goal: To find f F such that

c( f ) c(g) for all gF.

A global optimum

Definition:Optimization Problem

A set of instances of an optimization problem, e.g.– Traveling Salesman Problem (TSP)– Minimal Spanning Tree (MST)– Shortest Path (SP)– Linear Programming (LP)

Traveling Salesman Problem (TSP)

Traveling Salesman Problem (TSP)

Instance of the TSP – Given n cities and an n n distance matrix [dij], t

he problem is to find a Hamiltonian cycle with minimal total length.

on F n all cyclic permutations objects

( )1

n

j jj

c d

1 2 3 4 5 6 7 8

2 5 3 6 1 8 4 7

e.g.,

Minimal Spanning Tree (MST)

Minimal Spanning Tree (MST)

Instance of the MST – Given an integer n > 0 and an n n symmetric distance m

atrix [dij], the problem is to find a spanning tree on n vertices that has minimum total length of its edge.

( , ) {1,2, , }VF E V n all spanning trees with

( , )

: ( , ) iji j E

c V E d

Linear Programming (LP)

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

Linear Programming (LP)

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

1

2

n

c

cc

c

11 12 1

21 22 2

1 2

n

n

m m mn

a a a

a a aA

a a a

1

2

m

b

bb

b

1

2

n

x

xx

x

minimize

Subject to

c x

Ax b0x

Linear Programming (LP)

, , 0nx x R AF x b x

:c x c x

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

minimize

Subject to

c x

Ax b0x

Example:Linear Programming (LP)

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

1 2 34 2 3x x x

1 2 3

1 2 3

2

, , 0

x x x

x x x

4 2 3c

1 1 1A 2b

minimize

Subject to

Example:Linear Programming (LP)

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

minimize 1 1 2 2 n nc x c x c x

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

n n

n n

m m mn n m

a x a x a x b

a x a x a x b

a x a x a x b

1 2, , , 0nx x x

Subject to

1 2 34 2 3x x x

1 2 3

1 2 3

2

, , 0

x x x

x x x

minimize

Subject to

x1

x2

x3

v1

v2

v3

c(v1) = 8

c(v2) = 4

c(v3) = 6

The optimum

The optimal point is at one of the vertices.

Example:Minimal Spanning Tree (3 Nodes)

1 2 34 2 3x x x

1 2 3 2x x x

minimize

Subject to

c1=4

c3=3

c2=2

1 2 3, , {0,1}x x x

x1{0, 1}

x2{0, 1}

x3{0, 1}

Integer Programming

x1

x2

x3

Example:Minimal Spanning Tree (3 Nodes)

1 2 34 2 3x x x

1 2 3 2x x x

minimize

Subject to

c1=4

c3=3

c2=2

x1{0, 1}

x2{0, 1}

x3{0, 1}

Linear Programming

x1

x2

x3

1 2 3, , 0x x x 1 2 3, , 1x x x

Some integer programs can be transformed into linear programs.

Optimization Problems

Local and Global Optima

大同大學資工所智慧型多媒體研究室

Neighborhoods

Given an optimization problem with instance

(F, c),

a neighborhood is a mapping

defined for each instance.

: 2FN F

For combinatorial optimization, the choice of N is critical.

TSP (2-Change)

f F gN2(f )

2 ( ) N f g g F g and can be obtained as above

TSP (k-Change)

( )

.k

g F gN f g

k f

and can be obtained

by changing edges of

MST

f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.

( ) N f g g F g and can be obtained as above

LP

minimize

Subject to

c x

Ax b0x

( ) , 0, N x y Ay b y y x and

Local Optima

Given(F, c)

N

an instance of an optimization problem

neighborhood

f F is called locally optimum with respect to N (or simply

locally optimum whenever N is understood by context) if

c(f ) c(g) for all gN(f ).

0 1 F

c

small

Local Optima

F = [0, 1] R1

( ) , 0, N f x x F y x f and

C

B

A Local minimum

Local minimum

Global minimum

Decent Algorithm

f = initial feasible solution

While Improve(f ) do

f = any element in Improve(f )

return f

Improve( ) ( ) ( ) ( )f s s N f c s c f and

Decent algorithm is usually stuck at a

local minimum unless the neighborhood N

is exact.

Exactness of Neighborhood

Neighborhood N is said to be exact if it makes

Local minimum Global Minimum

Exactness of Neighborhood

0 1 F

c

F = [0, 1] R1

( ) , 0, N f x x F y x f and

C

B

A Local minimum

Local minimum

Global minimum

N is exact if 1.

TSP

N2: not exact

Nn: exact

f F gN2(f )

MST N is exact

f F gN(f )1. Adding an edge to form a cycle.2. Deleting any edge on the cycle.

( ) N f g g F g and can be obtained as above

Optimization Problems

Convex Sets and Functions

大同大學資工所智慧型多媒體研究室

Convex Combination

x, y Rn

0 1 z = x +(1)y

A convex combination of x, y.

A strict convex combination of x, y if 0, 1.

Convex Sets

S Rn

z = x +(1)y

is convex if it contains all convex combinations of pairs x, y S.

convex nonconvex

0 1

Convex Sets

S Rn

z = x +(1)y

is convex if it contains all convex combinations of pairs x, y S.

n = 1

S is convex iff S is an interval.

0 1

Convex Sets

Fact: The intersection of any number of convex sets is convex.

c

Convex Functions

x yx +(1)y

c(x)

c(y)c(x) + (1)c(y)

c(x +(1)y)

S Rn a convex set

c:S R a convex function if

c(x +(1)y) c(x) + (1)c(y), 0 1

Every linear function is convex.

LemmaS

c(x)

t

a convex set

a convex function on S

a real number

( ) ,tS c x x Stx

is convex.

Pf) Let x, y St x +(1)y S

c(x +(1)y) c(x) + (1)c(y)

t + (1)t

= t

x +(1)y St

Level Contours

c = 1

c = 2

c = 3

c = 4

c = 5

Concave Functions

S Rn a convex set

c:S R a concave function if

c is a convex

Every linear function is concave as well as convex.

Optimization Problems

Convex Programming Problems

大同大學資工所智慧型多媒體研究室

Theorem

(F, c) an instance of optimization problem

a convex set

a convex function

Define ( )N x y y F x y and

( )N x is exact for every > 0.

• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point.

Theorem

(F, c) an instance of optimization probleman instance of optimization problem

a convex set

a convex function

Defi ne ( )N x y y F x y and

( )N x is exact f or every > 0.

(F, c) an instance of optimization probleman instance of optimization problem

a convex set

a convex function

Defi ne ( )N x y y F x y and

( )N x is exact f or every > 0.

Pf)

xF

( )N x

yNext, we now want to show that c(y) c(x).

• Let x be a local minimum w.r.t. N for any fixed > 0.• Let yF be any other feasible point. <<1 such that• Since c is convex, we have

• Therefore,

Theorem

(F, c) an instance of optimization probleman instance of optimization problem

a convex set

a convex function

Defi ne ( )N x y y F x y and

( )N x is exact f or every > 0.

(F, c) an instance of optimization probleman instance of optimization problem

a convex set

a convex function

Defi ne ( )N x y y F x y and

( )N x is exact f or every > 0.

Pf)

xF

( )N x

yz

(1 ) ( ).x y xz z N and

( ) ( (1 ) )c c x yz

( ) (1 ) ( )c x c y

( ) ( )( )

1

zc c xc y

( ) ( )

1

c x c x

( )c x

( ) ( )zc c x

Convex Programming Problems

(F, c)

Defined by ( ) 0, 1, ,ig x i m

: nig R R

Convex function

an instance of optimization problem

Important property:

Local minimum Global Minimum

Concave functions

Convexity of Feasible Set

(F, c)

Defined by ( ) 0, 1, ,ig x i m

: nig R R

Convex function

an instance of optimization problem

Important property:

Local minimum Global Minimum

Concave functions

( ) : ig x convex

( ) : ig x concave

( ) 0 : ig x convex

( ) 0 : ig x convex

: iF convex

1

: m

ii

F F

convex

Convex Programming Problems

(F, c)

Defined by ( ) 0, 1, ,ig x i m

: nig R R

Convex function

an instance of optimization problem

Important property:

Local minimum Global Minimum

Concave functionsConvex

Theorem

In a convex programming problem, every

point locally optimal with respect to the

Euclidean distance neighborhood N is also

global optimal.