INTRODUCTION

Heesoo Myeong and Kyoung Mu LeeDepartment of ECE, ASRI, Seoul National University, Seoul, Korea

http://cv.snu.ac.kr

Tensor-based High-order Semantic Relation Transfer for Semantic Scene Segmentation

Goal

PROPOSED METHOD Results on Jain et al. Dataset

Quantitative Results on Standard Datasets

Jain et al. dataset (Jain et al., ECCV10): 250 training images, 100 test images, 19 labels SIFT Flow dataset (Liu et al., CVPR09): 2,488 training images, 200 test images, 33 labels Polo dataset (Zhang et al., CVPR11): 80 training images, 237 test images, 6 labels

Table 1: Per-pixel classification rates and (average per-class rates)

1. For a test image, retrieve similar training images using global features

2. Apply semantic relation transfer algorithm to transfer third-order semantic relation from each training image to the test image

3. Integrate the high-order score into MRF optimization framework and ob-tain semantic scene segmentation

EXPERIMENTS

Inference Use fully connected third-order Markov random field (MRF)

model:Key idea

Our Contributions The use of high-order semantic relations for semantic segmentation

A novel tensor-based representation of high-order semantic relations

A quadratic objective function for learning the semantic tensor and an effi-cient approximate algorithm

Semantic scene segmentation: identifying and segmenting all objects in a scene

Among superpixels ( is the number of total superpixels, is the number of object classes), third-order semantic relation is defined as a number of semantic tensors

The variable indicates confidence score of how likely the region triplet would be labeled as , respectively

To describe observed third-order semantic relation within the retrieved im-age, we define another number of semantic tensors

where denotes the ground truth class of region

Now, the semantic relation transfer problem is reformulated as the problem of estimating the magnitude of confidence scores for all superpixel triplets and for all object class triplets based on

We have presented a novel approach to learn high-order semantic relations of regions in a nonparametric manner

We develop a novel semantic tensor representation of the high-order se-mantic relations

We cast the high-order semantic relation transfer problem as a quadratic objective function of semantic tensors and propose an efficient approximate algorithm

𝒥 (c )=∑𝑖

𝐸𝐷 (𝑐𝑖 )+¿∑𝑖 , 𝑗

𝐸𝑃 (𝑐 𝑖 ,𝑐 𝑗)+∑𝑖 , 𝑗

𝐸𝐻 (𝑐 𝑖 ,𝑐 𝑗 ,𝑐𝑘) ,¿where represents data term, represents pairwise term, and is confidence score by the semantic relation transfer algorithm

Exploiting high-order(mostly third-order) semantic relation

sky

tree

building car

road

Test image Semantic scene segmenta-tion

build-ing

car

sky

road

sky

per-son

car

sky

tree

High-order relations in the training dataset

Previous works & Limitations Conventional context models mainly focus on learning pairwise relation-

ships between objects

Pairwise relations are not enough to represent high-level contextual knowl-edge within images

skytree

build-ing

car

roadroad

car

sky

per-son

tree

tree

build-ing

Query im-age

Integrated high-order relationSemantic segmenta-tion

Retrieved im-ages

…

Groundtruth

sky

road

building

tree

car

side-walkAnnotations of

retrieved im-ages

Predicted top scored high-order

relation

tree

sky

road

building

sky

car

tree

per-son

road

Results on LMO Dataset

Results on Polo Dataset

Overview

Semantic Relation Transfer Algorithm

[ ]ijk ijkx X

1 if ( ) , ( ) , ( ) , ( , , )

otherwis,

0 e

i j k i j k

ijk

retrievedG s c G s c G s c s s s Sy

skysky

road

We separately deal with the semantic relations transfer problem with re-spect to

The quadratic objective function with respect to as

where is the triplet-wise similarity between two region triplets and

111 112{ , ,..., }KKKX X X X

111 112{ , ,..., }KKKY Y Y Y

Y

Problem Statement

Objective FunctionY

2 2,

, , , , , , ,

1( ) ( ) ( ) ,

2

N N

ijk lmn ijk lmn ijk ijki j k l m n i j k

F w x x x y X

Pairwise semantic relation

car

build-ing

car

build-ing

Y

grassgrass

grass grassgrass grass

horsehorse

horsehorse

horse horse

horsehorsegrass grass

personperson

personpersonpersonperson

QueryGround truth

Pro-posed

QueryGround truth

Pro-posed

building

build-ing

sky

skyskyskysky

buildingbuilding

building

side-walk

side-walk

moun-tain

moun-tain

window

doorwindow

doorbuild-

ing

sky

window

carcar

QueryGround truth

Pro-posed

QueryGround truth

Pro-posed

skytree

building carroad

sky

road

build-ing

tree

car

side-walk

build-ing

build-ing

tree

car

tree

roadroad

build-ing

building

carroad

bison

mountain

grass grass

bison

tree

QueryGround truth

Pro-posed

QueryGround truth

Pro-posed

CONCLUSION