University of Science and Technology of China

Random Walks on Graphs to

Model Saliency in Images

作者： Viswanath Gopalakrishnan， Yiqun Hu， Deepu Rajan

讲解人：曹梦霏

2009年11月12日星期四

University of Science and Technology of China2

作者相关信息Deepu Rajan

Assistant Professor, School of Computer Engineering NTU

Research InterestsVisual attention detectionMultimedia signal processingHuman action recognitionImage and video editing

Homepagehttp://www.ntu.edu.sg/home/asdrajan/

Viswanath Gopalakrishnan(stu.)Visual attention detection

Doc.Yiqun Hu(graduated staff) http://www3.ntu.edu.sg/home5/y030070/

University of Science and Technology of China3

文章信息• 文章来源

IEEE Computer Society Conference

• 发表时间

Miami, FL, USA，June 20-June 25.

• 引用V. Gopalakrishnan, Yiqun Hu, D. Rajan, "Random walks ongraphs to model saliency in images," cvpr, pp.1698-1705,2009 IEEE Conference on Computer Vision and PatternRecognition, 2009

University of Science and Technology of China4

背景介绍

最大的目标是模拟人类视觉： Brain system

Vision system

显著性区域检测：用于自动聚焦感兴趣区域或者修割重要区域突出目标确定位置用于目标识别削弱背景遮挡的影响，增强图片检索系统

University of Science and Technology of China5

Original AbstractWe formulate the problem of salient region detection in images as Markov random walks performed on images represented as graphs.

the global properties of the image are extracted from the random walk on a complete graph,

the local properties are extracted from a k-regular graph.

The most salient node is selected as the one which is globally most isolated but falls on a compact object. The equilibrium hitting times of the ergodic Markov chain holds the key for identifying the most salient node.

The background nodes which are farthest from the most salient node are also identified based on the hitting times calculated from the random walk.

Finally, a seeded salient region identification mechanism is developed to identify the salient parts of the image. The robustness of the proposed algorithm is objectively demonstrated with experiments carried out on a large image database annotated with ‘ground-truth’ salient regions.

University of Science and Technology of China62009/11/

15

文章摘要

•完全图的随机游走全局性质

•K正则图的随机游

走局部性质

•全局分离 紧致物体

•正态历经马氏链的访问时间显著节点

•离显著节点远

•游走的访问时间背景节点

Seeded显

著性区域

University of Science and Technology of China7

前人方法

对显著性区域不同的理解，有不同的方法：自下至上的方法，利用color和orientation，与周边差别imitate the property of the neurons in human cortex

信息论的角度，显著性就是差异性，自信息熵大

傅立叶变换，幅度差商包络、相位谱

University of Science and Technology of China8

相关的基本思想

将在图像图上的随机游走的性质应用于显著性，前人的相关方法有：

Costa等人提出的，通过对在均衡状态下各个节点的访问频率考察得到显著性区域的标识。

问题：该方法仅用两张合成图片实验，没有好的实验结果支撑；

J.Harel等人提出的，使用图中边来表征两节点的差异度，越不同，边强度越低。因此，最不常（frequently）被访问的节点是图中差异性越大的节点。

问题：背景更大的对比度，有更大的显著性。

University of Science and Technology of China9

本文的基本思想

考虑节点全局的isolation和局部的compact；

创造了鲁棒的特征描述符来分别建立全连通图，k-正则图，其中，全连通图考察全局性，后者考察局部性；

利用上两图的随机游走性质，获得显著性节点和背景节点；

最后，利用这两种节点提取图像的显著性区域。

University of Science and Technology of China

10

本文方法（一）

数学准备：一个有N个状态的马氏链可由N*N的转移矩阵P指定，其中pij是从i到j的转

移概率，并能算出初始分布；各态历经的马氏链是指从任意状态出发经有限步可到任意状态的马氏链；

正则马氏链是指转移矩阵的某次幂全为正。本文的模型是各态历经的，不一定是正则的。

均衡分布1*N的向量π有： （1）。矩阵W为将π堆积而来的N*N的矩阵，是P的幂极限。马氏链的基本矩阵Z： （2）。定义为t=0时刻从状态i返回i的时间期望。

定义为t=0时刻从状态i到j的时间期望hitting time。是从均衡状态π到i的期望时间。关系式：

（3）。

University of Science and Technology of China11

本文方法（二） 图呈现

Graph representation：We represent the image as a graph G(V;E), where V is the set of vertices or nodes and E is the set of edges.

其中点是8×8的patches，边由特征决定。特征提取：

颜色，图像使用YCbCr域的表示，选取Cb、Cr两个值方向，计算五个尺度的方向直方图的熵，得到五个值

得到每个patch的1×7的特征描述符x，计算边权；

（5）其中实验中sigma归一化。

University of Science and Technology of China12

本文方法（三）图呈现

完全图计算：

相似度矩阵

然后就可以利用转移矩阵和（1）~（3）算出所有参数

University of Science and Technology of China13

本文方法（四）图呈现

K-正则图计算：N(i)是i点的k领域

然后就可以利用转移矩阵和（1）~（3）算出所有参数

University of Science and Technology of China14

本文方法（五）节点选取

Node selection：显著性节点

When a node is globally a pop-out node, what it essentially means is that it is isolated from the other nodes so that a random walker takes more time to reach such a node.On the other hand, if a node is to lie on a compact object, a random walker should take less time to reach it on a k-regular graph. We now elaborate on these concepts.

University of Science and Technology of China15

本文方法（六）节点选取

Node selection：背景节点

背景是全局的，只用全连通图.The most important feature of a background node is obviously the less saliency of the node as calculated from equation(11).Moreover, the background nodes have the property that it is at a large distance from the most salient node, Ns.

The background in an image is, more often than not, inhomogeneous. Additional condition of maximizing the distance to all background nodes identified so far.

University of Science and Technology of China16

本文方法（七）显著性区域提取

Seeded Salient region extraction：

A particular node k is regarded as part of the salient region：

如果它到显著性点的hitting time比到所有背景节点的hitting time都短。

University of Science and Technology of China17

本文方法（七）显著性区域提取

Seeded Salient region extraction：关于全局还是局部的两点考虑：

in a global random walk it may turn out that a node that is far from a salient node in the spatial domain, but close to it in the feature domain (as indicated by the edge weights) may be erroneously classified as belonging to the salient region.On the other hand, a local random walk may treat a background region that is spatially close to a salient node as part of the salient object, since the random walk is restricted to a smaller area. Hence,

因此，采用：

University of Science and Technology of China18

实验结果

数据库：图片来自：T. Liu, J. Sun, N. Zheng, X. Tang, and H. Y.Shum. Learning to detect a salient object. In CVPR, 2007.

Ground truth来自:人工标定显著性区域

数值结果： 5000 images, an accuracy of 89.6%落入用户标定 直接得到2-bit位图，没有阈值的问题 利用precision, recall, F-measure度量

University of Science and Technology of China19

实验结果

数值结果：

University of Science and Technology of China20

实验结果

图例结果：

University of Science and Technology of China21

实验结果

图例结果：

University of Science and Technology of China22

实验结果

图例结果：

University of Science and Technology of China23

实验结果

图例结果：

the similarity of features on the salient object with the background which affectsthe random walk

Depth problem

University of Science and Technology of China24

总结文章贡献提出了解决通过随机游走提取显著性区域的算法。

同时利用局部和全局信息，随着特征性能提高能够增强本身算法

以后的工作解决全局和局部的比例问题

继续研究复杂背景下的检测