seam carving for content-aware image resizing

Seam Carving for Content-aware Image Resizing

資訊碩一 10077034 蔡勇儀Date ： 2012/01/03

@LAB 603

Introduction Basic Theory Application & Implementation

◦ Aspect Ratio Change◦ Retargeting with Optimal Seams-Order◦ Enlarging◦ Content Amplification◦ Seam Carving in the gradient domain◦ Object Removal

Multi-size Images Limitation Conclusions and Future Work

Outline

Muti-Media and Embedding System(e.g. Cell Phone) grow fast ， Resize or Multi-Size scaling are more important than past.

Standard image scaling is not sufficient since it is oblivious to the image content and typically can be applied only uniformly.

For improve the problem, many researcher prove some good idea.

Introduction(1/2)

The following is main methods for scaling ：◦ Corp (figure2(b))◦ Column or Row removal (figure2(c))◦ Pixel energy removal (figure2(e))◦ Optimal Pixel energy removal (figure2(f))◦ Object detection ◦ Seam Craving (figure2(d))

We can found the seam method have better result!

Introduction(2/2)

Introduction Basic Theory Application & Implementation

◦ Aspect Ratio Change◦ Retargeting with Optimal Seams-Order◦ Enlarging◦ Content Amplification◦ Seam Carving in the gradient domain◦ Object Removal

Multi-size Images Limitation Conclusions and Future Work

Outline

Step1 – Find seam◦ Find a path which have the minimum energy sum

from image top to bottom.

Step2 – Remove the Min. seam◦ When found all seam, select the Min. seam

remove.

Step3 – Repeat above step until get the demand size

Basic Theory

Give an energy function

Define Seam

Define the Seam Cost

Basic Theory – Step1

Find the minimum seam

Remove S* form image and lnstead of neighbors

Basic Theory – Step2

Repeat above step until get the demand size

Basic Theory – Step3

What’s energy function is the best?◦ e1

◦ Entropy 9*9 Windows add to e1

◦ Segmentation ( add to e1)◦ Histogram of Gradients

11*11cell around a pixel, 8-bins

Basic Theory – Energy Function

Basic Theory – Energy Function They all accommodate a similar range for

resizing.

We found either e1 or eHoG to work quite well.

Introduction Basic Theory Application & Implementation

◦ Aspect Ratio Change◦ Retargeting with Optimal Seams-Order◦ Enlarging◦ Content Amplification◦ Seam Carving in the gradient domain◦ Object Removal

Multi-size Images Limitation Conclusions and Future Work

Outline

Only one axis adjust

A picture size n*m n*m’ or n’*m n >= n’ m >= m’

Remove n-n’ or m-m’ seams

Enlarge at other page

Aspect Ratio Change

What’s the optimal order for remove seams? Column or Row or Other? How could decide?

Using dynamic programming

◦ where k = r+c, c = (m−m’), r = (n−n’)◦ αi is used as a parameter

that determine if at step i we remove a horizontal or vertical seam:

Retarget with Optimal Seam-Order(1/3)

Define transport map T◦ T(r,c)=min(T(r-1,c)+E(sy(In-r+1*m-c)), T(r,c-1)+E(sx(In-r*m-c+1)) )◦ where In-r*m-c

denotes an image of size (n−r)×(m−c), ◦ E(sx(I)) and E(sy(I))

are the cost of the respective seam removal operation.

Build the 1 bit map for record the direction

Retarget with Optimal Seam-Order(2/3)

Retarget with Optimal Seam-Order(3/3)

When m’ > m or n’ > n, we should insert seams to the picture.

Find the smallest energy seam for copy and insert, repeat until equal the demand scale.

But…

Enlarge(1/3)

Every time found the same seam, so we should decide all seams which need copy at first.

If m’ > m then we need insert (m’-m) seams. Find them and copy it for insertation.

Enlarge(2/3)

Enlarge(3/3)

The origin picture

Scalar Seam

Using same scalar enlarge then use seams-carving for recover to the origin size.

Content Amplification

If energy funciton use the gradient, then color show at remove place will be more nature after seam carving.

Seam Carving in the gradient domain

User mark the part which want to remove.

Decrease the energy on the part is removed.

Insert seams for keeping origin size.

Object remove

Introduction Basic Theory Application & Implementation

◦ Aspect Ratio Change◦ Retargeting with Optimal Seams-Order◦ Enlarging◦ Content Amplification◦ Seam Carving in the gradient domain◦ Object Removal

Multi-size Images Limitation Conclusions and Future Work

Outline

User want find the optimal picture scalar for their demand, so we need the real time opreation.

But the picture’s size 400*500 to 100*100 in

about 2.2 seconds, it is too long to real time.

How could do for real time?

Multi-size Images(1/3)

Make the index map for seams before user operation.

Build the horizontal & vertical index map (H&V)

But there will a big problem for operation that is H & V will be collided.

The sample solution is decide one just do one direction and then other direction need degenerate the index and redo the select seams operation

Multi-size Images(2/3)

Multi-size Images(3/3)

Introduction Basic Theory Application & Implementation

◦ Aspect Ratio Change◦ Retargeting with Optimal Seams-Order◦ Enlarging◦ Content Amplification◦ Seam Carving in the gradient domain◦ Object Removal

Multi-size Images Limitation Conclusions and Future Work

Outline

this method ◦ does not work automatically on all images.◦ can be corrected by adding higher level cues, either manual

or automatic. Figure 14, Figure 15 Other times,

◦ not even high level information can solve the problem. two major factors that limit this seam carving approach.

◦ The first is the amount of content in an image. If the image is too condensed, it does not contain ‘less important’ areas, then any type of content-aware resizing strategy will not succeed.

◦ The second type of limitation is the layout of the image content. In certain types of images, albeit not being condensed,the content is laid

out in a manner that prevents the seams to bypass important

Limitation(1/2)

Limitation(2/2)

Introduction Basic Theory Application & Implementation

◦ Aspect Ratio Change◦ Retargeting with Optimal Seams-Order◦ Enlarging◦ Content Amplification◦ Seam Carving in the gradient domain◦ Object Removal

Multi-size Images Limitation Conclusions and Future Work

Outline

to extend this approach to other domains, ◦ the first of which would be resizing of video. ◦ Since there are cases

when scaling can achieve better results for resizing, would like to investigate the possibility

to combine the two approaches, Specifically to define more robust multi-size images.

◦ would also like to find a better way to combine horizontal and vertical seams in multi-size

images.

Conclusions and Future Work

Q&A

Thank for your listening