NCKU CSIE

Visualization & Layout for Image LibrariesBaback Moghaddam, Qi Tian

IEEE Int’l Conf. on CVPR 2001

Speaker: 蘇琬婷

NCKU CSIE

Outline

• System Introduction

• Visualization and Layout Optimization

• Context and User Modeling

• Discussion

NCKU CSIE

System Introduction-PDH

• Personal Digital Historian (PDH)

• Interface Design :

• Polar coordinate visual layout

• circular display area

• touch sensitive table surface

• top projection table with a whiteboard as the table

surface

NCKU CSIE

PDH Table

NCKU CSIE

4W’s: The Organization Principal

Who What Where When

NCKU CSIE

NCKU CSIE

Content-based Visualization

• Contend-based Image Retrieval(CBIR)

• Images would be indexed by their visual contents

• Feature(content) extraction

• Visualization

• Traditional interfaces

• PCA Splats

• Display optimization

NCKU CSIE

Traditional Systems

• Visualization

• Simple 1-D list

• Sorted by decreasing similarity to the query

• Drawback

• Relevant images can appear at separate and distant

locations in the list

• Improvement

• 2-D display technique

NCKU CSIE

Top 20 Retrieved Images

• Ranked top to bottom and left to right

NCKU CSIE

PCA Splats

• Principal component analysis(PCA)

• project the images from the high-dimensional feature sp

ace to the 2-D screen

• 37 visual features(color, texture, structure)

• on the basis of the first two principal components norm

alized by the respective eigenvalues

• The maximum distance preservation from the original h

igh-dimensional feature space to 2-D space

NCKU CSIE

Display Optimization

• The drawback of PCA splat

• images are partially or totally overlapped

• Optimization

• Minimizing overlap (decreasing the overlap of the

images)

• Minimizing deviation (deviating as little as possible

from their initial PCA splat positions)

• Minimizing the total cost

NCKU CSIE

Cost Function

F(p) : cost function of the overall overlap

G(p) :cost function of the overall deviation from the

initial image positions

S : scaling factor and S = (N-1)/2

N : the number of images

λ: weight and λ 0≧

pGSpFJ

NCKU CSIE

Minimizing Overlap

ri : image size is represented by its radius ,i = 1,…,N

(xi, yi) : image center coordinates

u : measure of overlapping

σf : curvature-controlling factor

range of F(p):

(N-1)+(N-2)+…+1 = N(N-1)/2

22

1 1

00

01

2

jijiji

u

N

i

N

ij

yyxxrru

u

uepf

pfpF

f

i

j

NCKU CSIE

2

2

1

2

1

oii

oii

v

N

i

yyxxv

epg

pgpG

g

Minimizing Deviation

： the optimized and initial cent

er coordinates of the ith image, respectively

v : measure of deviation

σg : curvature-controlling factor

range of G(p) : N

range of F(p) : N(N-1)/2

∴S = (N-1)/2

oi

oiii yxyx ,,,

pGSpFJ

NCKU CSIE

Optimized PCA Splat

NCKU CSIE

Context and User Modeling

• Image content and “meaning” is ultimately based on semantics• user’s notion of content : high-level concept

• visual features : low-level concept

NCKU CSIE

NCKU CSIE

Context and User Modeling

• User modeling or “context awareness”

• constantly be aware of and adapting to the changing

concepts and preferences of the users

• learn from a user-generated layout

• a novel feature weight estimation scheme : α-estimation

• α: weighting vector for feature (color, texture, structure)

• α = (αc, αt, αs)T

• αc,t,s : the weight for color, texture, structure

• αc + αt + αs = 1

NCKU CSIE

Estimation of Feature Weights

Xc, t, s : Lc, t, s × N matrix where the ith column is the color, text

ure, structure feature vector of the ith image, i = 1,…,N

Lc, t, s : the lengths of color, texture, structure features

dij : the distance Euclidean-based between the ith image and t

he jth image

• minimizing with an Lp norm (with p = 2)

• non-negative least squares solutions

2

1 1 1 11

N

i

N

j

L

k

L

k

pkjs

kis

ps

pkjt

kit

pt

L

k

pkjc

kic

pc

pij

t sc

XXXXXXdJ

NCKU CSIE

an example of a user-guided layout

αc = 0.3792

αt = 0.5269

αs = 0.1002

NCKU CSIE

PCA splat on larger set of images

estimated weight

randomly generated weight

NCKU CSIE

User Modeling for Automatic Layout

user-guided layout

computer layout

NCKU CSIE

Future Work

• Having the system learn the feature weights from

various sample layouts provided by the user

• Incorporate visual features with semantic labels

for both retrieval and layout

• Incorporation of relevance feedback

• Automatic “summarization” and display of large

image collections