object tracking review - ku ittc · robust object tracking and adaptive detection ... • object...

ROBUST OBJECT TRACKING AND ADAPTIVE DETECTION FOR AUTO NAVIGATION OF UNMANNED AERIAL VEHICLE

Master’s Thesis DefenseSoumyaroop Nandi

Electrical Engineering and Computer Science DepartmentUniversity of Kansas

Advisor: Dr. Richard WangCommittee Members: Dr. James Rowland, Dr. James Stiles

9th Jan, 2017

1

2

What is Object Detection ?

What is Object Tracking?

OVERVIEW

3

Background Motivation and Goals Contributions Literature Review

• Object Detection• Object Tracking

System Overview• MBD Detection• Post Processing• KCF Tracking• Adaptive Redetection• Algorithm 1: Proposed Method

Experiments and Results• Quantitative Evaluation• Qualitative Evaluation

Limitations Conclusions and Future Work

OUTLINE

4

BACKGROUND

[1] Source: Amazon, “Amazon prime air”, https://www.youtube.com/watch?v=98BIu9dpwHU, 2013.[2] Source: Google, “ Google Project Wing”, https://www.youtube.com/watch?v=cRTNvWcx9Oo, 2014[3] Source: Dailymail.com, “DHL Parcelcopter”, http://www.dailymail.co.uk/video/sciencetech/video-1122742/DHL-parcel-copter-makes-test-delivery-flights-Germany.html, 2016

Amazon Prime Air[𝟏𝟏] Google Project Wing[𝟐𝟐] DHL Parcelcopter[𝟑𝟑]

5

BACKGROUNDMajor Tasks:

1. Image Registration

3. Tracking the Detected Object

2. Object Detection

6

Challenges:

BACKGROUND

Background Clutter Presence of Noise Occlusion Fast Camera Motion

Varying Object velocity Rapid Background Change Appearance Variation Illumination Variation

7

• Till date no smart visual system is developed, helping automated UAV

navigation flawlessly.

• Real-time (at least 5 frames per second), autonomous, sense and

avoidance navigation system.

• Transponder based collision avoidance system fails in presence of non-

cooperative rigid objects or other mobile objects.

• Vision-based methods:

Robust to electromagnetic interference

Compact and low power consumption

• Tracking with forward looking camera.

MOTIVATION AND GOALS

8

Scale adaptive bounding box around the object in real time for

all the frames.

Runs automatically from the first frame till the last frame.

Closed loop system with long-term, error free tracking.

No computationally expensive supervised training.

Adaptive redetection technique proposed

Highest speed and best performance when compared to 6 other

state-of-the-art trackers in 14 challenging datasets.

CONTRIBUTIONS

9

Salient Object Detection: Top-Down Methods

Supervised training performed in previously detected frames. Prior knowledge about object’s approximate location helps reduced

search area. Examples are [5], [6], [19].

Bottom-Up Methods Compare low level image features (like color, text, shape, contrast,

gradient, Spatio-temporal) between background and salient regions. Fails in complex images. Examples are [7], [8], [20], [21], [22].

Proposal Window generating Methods Do not provide 1 bounding box, but multiple ranked proposals. Examples are [25], [26].

Deep Learning/ Neural Network based Methods Generate more accurate result, but require huge training datasets Examples are [20], [22], [36].

LITERATURE REVIEW

10

Object Tracking: Generative Approaches

Takes into account only the positive image patches from thepreviously tracked frames to train the classifier.

Examples are [5], [6], [11], [12]. Discriminative Approaches

Takes into account both positive and negative samples from thepreviously tracked frames to train the classifier.

Examples are [3], [13], [14]. Ng et al. [15] has mathematically shown that discriminative asymptotic

error is lower than generative asymptotic error.

LITERATURE REVIEW

TRACKING APPROACH FEATURES/LIMITATIONS

Detection and Tracking [16] Not real-time, manual initialization, supervised training on huge dataset required.

Saliency-based Tracking [17] Not real-time, automatic initialization enabled, supervised training required.

Multiple Camera Tracking [18] Real-time, camera calibration required calibration, inefficient without GPS data.

11

Correlation Filter based Object Tracking:LITERATURE REVIEW

TRACKER FEATURES/LIMITATIONS

MOSSE filter[28] First real-time correlation filter based tracker trained on grayscale images in Fourier Domain.

CSK [29] Kernel based correlation filter was introduced to MOSSE filter [28].

CN [32] Used color features in CSK filter [29]. The speed was reduced by many folds.

KCF [3] Introduced HOG features (31 bins added linearly in Fourier domain) and Gaussian kernels forcorrelation filtering. Biggest limitation is that it is scale invariant. Fails in long-term tracking.

SAMF [2] Adaptive to scale variation. Used HOG feature templates in each frame. Drastic speed reduction

DSST [33] Integrated multi-scale features along with HOG features of KCF [3] to deal with scale variation.Drastic speed reduction. Lost the purpose of fast tracking based on correlation filter.

STC [46] Used Spatio-temporal features in KCF [3] for context learning. Improved performance than KCF.But found tracking failure in challenging datasets. Speed reduced slightly too.

Deep Learning/Neural Network based Trackers [37], [38] use deep learning for better accuracy. But cannot be used in real-time application yet.

12

SYSTEM OVERVIEWThe Proposed Tracker:

Fig 5.1: Illustration of our approach: (a) input frames, (b) auto initialization and KRLS training, (c)redetection,(from left to right) saliency map generation, binary image thresholding with postprocessing and bounding box drawn around redetected object

13

Segmentation done by image properties like appearance contrast, rarity, regionuniformity, spatial compactness, uniqueness, color cue, spectral cue, edge cue,super-pixel cue of pixels.

Mathematically, these characteristics can be represented as distance functions[44].

Let P be the set of all integers (i,j). The function f from P x P into some non-negative integers is termed as:

a) Positive definite, if f(x,y) = 0, if and only if x = y.

b) Symmetric, if f(x,y) = f(y,x), for all x,y in P.

c) Triangular, if f(x,z) ≤ f(x,y) + f(y,z), for all x, y, z in P.

If f satisfies all the three conditions (a-c), then f can be termed as a distance functionor a pseudo-metric.

In an image, distance function computed as Euclidean distance or some variantof Dijkstra’s algorithm.

MBD DETECTION

14

Image Processing and Graph Theory Equivalence

Consider a graph G = {V,E}, G is a connected graph consisting of finite set of vertices V and finite set of edges E.

In image processing, V represent image intensity, color, contrast or other image characteristics.

A path (π), in a connected graph G = {V,E}, can be represented with a set of edges E, π = π(0),π(1), … … . ,π(n) , for all {π(i), π(i+1)} ∈ E and i ∈ {1,2,3,…n}.

Multiple paths between two same image pixels.

Cost function or weight λ(π) ≥ 0 is assigned to each path π.

Minimum distance function 𝑓𝑓𝝀𝝀 = min {π(i)}, i ∈ {1,2,3,…n}. For every path π = π(0),π(1), … … . ,π(n) ,

a) λ(π) = λ( π(n),π(n − 1), … … . ,π(0) )

b) λ(π) ≤ λ( π(0), … ,π(i) ) + λ( π(i), … ,π(n) ) for all 0 ≤ i ≤ n

MBD DETECTION

Seed Set

Shortest Path

15

Cost function λ(π) can either produce edge weight map or vertex weight map.

Geodesic Distance function [7] generate edge weight map with small-weight-accumulation problem.

MBD function 𝑓𝑓𝝀𝝀 [8,9,10] generate vertex weight map.

𝑓𝑓𝝀𝝀 = minπ i ∈ Ia,b

(λ+(π) − λ−(π))

= minπ i ∈ Ia,b

(max 𝒊𝒊=𝟎𝟎,𝟏𝟏,…𝒏𝒏{λ(π(i))} − min𝒊𝒊=𝟎𝟎,𝟏𝟏,…𝒏𝒏

{λ(π(i))})

Ia,b represents all the possible paths from pixels a to b.

Cost function λ(π) remains constant (until higher intensity than previous seed is encountered) unlikeother distance functions.

MBD cannot be computed using Dijkstra’s algorithm. Exact MBD [9] Approximate MBD [10] Raster scanned MBD [8].

Order of complexity is O(mnlogn). m is the number of distinct intensities in all the pixels and n is the total number of pixels in the image.

MBD DETECTION

16

Raster Scanned MBD:

Raster Scanning Inverse Raster Scanning The path of the raster and inverse raster scans consist of two parts –

Path from background seed to the adjacent pixel - π𝒚𝒚 and Path from adjacent pixel to the chosen pixel – π{𝒚𝒚,𝒙𝒙} Let the entire path cost function be represented as λ(π) = λ(π𝒚𝒚) + λ(π{𝒚𝒚,𝒙𝒙}), then the distance function for a

single scan is given by:𝒇𝒇𝝀𝝀 = 𝒎𝒎𝒎𝒎𝒙𝒙

𝐢𝐢=𝟎𝟎,𝟏𝟏,…𝒏𝒏{𝛌𝛌(𝛑𝛑(𝐢𝐢))} − 𝒎𝒎𝒊𝒊𝒏𝒏

𝐢𝐢=𝟎𝟎,𝟏𝟏,…𝒏𝒏{𝛌𝛌(𝛑𝛑(𝐢𝐢))}

= max {A(y),I(x)} – min{B(y),I(x)} (3.4) A(y) and B(y) are the highest and lowest pixel values on the entire path π from background seed to chosen

pixel in a single scan.

MBD DETECTION

17

Raster Scanned MBD: Let 𝑓𝑓𝑰𝑰 be the final MBD map. 𝑓𝑓𝑰𝑰 is modified in each scan by choosing the minimum between previous

𝑓𝑓𝝀𝝀 and current 𝑓𝑓𝑰𝑰 as:𝑓𝑓𝑰𝑰 = min (current 𝑓𝑓𝑰𝑰, previous 𝑓𝑓𝝀𝝀) (3.5)

A(y) and B(y) are modified after each raster and inverse raster scan, as the path changes every timeand this in turn modifies 𝑓𝑓𝝀𝝀 and 𝑓𝑓𝑰𝑰.

Each iteration stop when 𝑓𝑓𝝀𝝀 becomes equal to 𝑓𝑓𝑰𝑰.

MBD DETECTION

Fig 3.4: Original Image pixel Fig 3.5: Initial 𝒇𝒇𝑰𝑰, the MBD map Fig 3.6: Initial Highest pixel Fig 3.7: Initial Lowest pixel intensityintensity distribution intensity of the path A(y) intensity of the path B(y)

18

Otsu’s Method: Let H be the histogram of saliency map s_m. Intensity levels L ∈ [0, l-1], ni be the number of

pixels with intensity i ∈ L

H = ∑𝑖𝑖=0𝑙𝑙−1 𝑛𝑛𝑖𝑖 Let Hn be the normalized histogram of s_m, such that for every threshold value t, t ∈ L, we divide

the normalized histogram into two groups – C1 ∈ Hni, i ∈ [0,t] and C2 ∈ Hn

i, i ∈ [t+1,l-1]

P1 = P(C1) = ∑𝑖𝑖=0𝑡𝑡 𝐇𝐇𝒏𝒏

𝒊𝒊 P2 = P(C2) = ∑𝑖𝑖=𝑡𝑡+1𝑙𝑙−1 𝐇𝐇𝒏𝒏

𝒊𝒊 = 1 – P1

m1 = ∑𝑖𝑖=0𝑡𝑡 𝑖𝑖.𝑃𝑃(i/𝐂𝐂1) = ∑𝑖𝑖=0

𝑡𝑡 𝑃𝑃(𝑪𝑪1/i).P(𝑖𝑖)𝑃𝑃(𝑪𝑪1)

= 1𝐏𝐏𝟏𝟏∑𝑖𝑖=0𝑡𝑡 𝑖𝑖.𝐇𝐇𝒏𝒏

𝒊𝒊 m2 = 1𝐏𝐏𝟏𝟏∑𝑖𝑖=𝑡𝑡+1𝑙𝑙−1 𝑖𝑖.𝐇𝐇𝒏𝒏

𝒊𝒊

m1 and m2 are the mean intensities of all the pixels in group1 (C1) and group2 (C2) respectively.

POST PROCESSING

19

Otsu’s Method: Let mg be the global intensity mean and mt be the mean intensity up to t.

𝜎𝜎𝑏𝑏2 = P1.(m1 – mg)2 + P2.(m2 – mg)2 = 𝑚𝑚𝑔𝑔.𝐏𝐏𝟏𝟏−𝑚𝑚𝑡𝑡

𝐏𝐏𝟏𝟏.(1− 𝐏𝐏𝟏𝟏) 𝜎𝜎𝑏𝑏2 is the inter- class variance and the optimal threshold topt for the saliency map s_m is given

by:𝜎𝜎𝑏𝑏2(topt) = max

0<𝑡𝑡<𝑙𝑙−1𝜎𝜎𝑏𝑏2(𝑡𝑡)

Result obtained by thresholding the saliency map s_m with 𝜎𝜎𝑏𝑏2 (topt) produces a binary image.

POST PROCESSING

Fig 3.8 (a) Saliency Map generated by Fast MBD detector with airplane_016 dataset [30] (b) Bounding box around original image frame from airplane_016 dataset [30] after detection

20

KCF TRACKINGDense Sampling: Discriminative methods use positive samples from foreground and negative samples from background.

Background is generally huge, with large negative samples. Most discriminative trackers randomly chooseless number of negative samples to reduce complexity.

KCF works with correlation filter in Fourier domain. Huge number of positive and negative samples can beused for training the classifier without increasing complexity.

The classifier is trained with a Regularized Least Squares (RLS) Let X and Y be 2 sets of random variables such that there are n sets of statistically independent and

identically distributed training samples T = {(x1,y1), (x2,y2),……, (xn,y𝑛𝑛)}. We define a binary classifier B(x) to distinguish positive and negative labels from the training sample set T.

B(x) is trained with RLS as:RLS = min

𝐁𝐁∑i=1

n 𝑮𝑮 (yi,𝐁𝐁(xi)) + λ ||B||2 (4.1) G((yi,B(xi)) is a loss function, B is the regularization trade off and λ is a regularization parameter used to

control the regularization and makes sure that there is no overfitting. The loss function chosen in RLS is G(y,B(x)) = (y - B(x))2. Eigen decomposition is performed to express B as:

B = (XTX + λI)-1 Xty (4.2) I is an identity matrix and XT is the transpose of matrix X. In the complex plane, XT becomes XH.

B = (XHX + λI)-1 Xhy (4.3)

21

KCF TRACKINGCirculant Matrix: Base samples are represented in a Circulant matrix. Let x = (x1,x2,……,xn) be a vector consisting of n input

samples. Circulant matrix M(x) obtained from x is given by:

M(x) =

x1 x2 x3 ⋯ x𝑛𝑛x𝑛𝑛 x1 x2 … x𝑛𝑛−1

x𝑛𝑛−1 x𝑛𝑛 x1 … x𝑛𝑛−2⋮ . . ⋱ ⋮

x2 x3 x4 ⋯ x1

Δ =

0 0 0 ⋯ 11 0 0 … 00 1 0 … 0⋮ . . ⋱ ⋮0 0 0 ⋯ 1 0

Δ is a permutation matrix. M(x) can be obtained from Δ.xT

Dense samples are created by cyclically shifting base samples. We train the classifier with input sample vector x and test label vector y.

M(x).y = F-1{F*(x) ⊙F(y)} (4.6) M(x).y actually computes the convolution between vectors x and y. Sums, products and inverses are also Circulant – so we do not have to store all of them in the memory. The matrix inverse operation is calculated efficiently in the Fourier domain using Eigen decomposition.

B = diagonal{(x*⊙ y) /(x* ⊙ x + λ)} (4.11) B is the binary classifier in frequency domain, x is the training vector in frequency domain, y is the testing

vector sample in the frequency domain and λ is the regularization parameter defined in equation (4.3). Complexity of taking DFT is linear O(nlogn). Divisions and multiplications are element-wise with a complexity

of O(n). Standard method of computing RLS has a complexity of O(n3).

22

KCF TRACKINGNon-linear Regression: Computes regression function (expressed as kernels) with multiple outputs instead of binary classification. Non-linear classifiers using power spectrum are difficult to kernelize. Kernel trick [40] solves the problem. Maps input vector x to higher dimensional non-linear feature space H(x). Kernel function k measures similarity between two input vectors 𝐱𝐱𝟏𝟏 and 𝐱𝐱𝟐𝟐 with a mapping in real-plane.

k: X x X →ℝ (x1,x2) → k(x1,x2) Simplest is inner product kernel function k(x): 𝑥𝑥1, 𝑥𝑥2 = ∑𝑖𝑖,𝑗𝑗=1

𝑛𝑛 [𝑥𝑥𝑖𝑖] [𝑥𝑥𝑗𝑗], (4.13) A distance function f(y) segregates labels y in the feature space. Distance between f(y) and y determines the classification. This distance can be equal for multiple kernels. So

weight αi is assigned representing importance of a particular kernel. In constrained optimization problem, this weight αi is made equivalent to a Lagrange’s multiplier αi>0 and

the distance function can be derived from a Lagrangian. The distance function f(y), when represented in terms of inner product kernel function k in the higher

dimension feature space H(x) with respect to RLS binary classifier B looks like:f(y) = BT y = ∑𝑖𝑖=1

𝑛𝑛 𝛼𝛼𝑖𝑖𝑘𝑘(𝒚𝒚, 𝑥𝑥𝑖𝑖) (4.17)

This distance function f(y) is the new kernelized classifier. A kernel matrix K (nxn) stores all the innerproducts between all the sample pairs.

Kij = k(𝑥𝑥𝑖𝑖,𝑥𝑥𝑗𝑗) (4.18)

23

KCF TRACKINGNon-linear Regression: KRLS Training We can use the kernel matrix K, testing sample y and the regularization parameter λ to evaluate the dual

space variable or the kernel weight α from equations (4.3), (4.17) and (4.18) as:α = (K + λI)-1y (4.19)

Henrique et al. [3] states that a kernel matrix is Circulant if the kernel function satisfies:k(x1,x2) = k(Δ x1, Δ x2) (4.20)

Δ is the Permutation matrix. The kernel matrix in (4.19) is evaluated in the Fourier domain using a correlation operation and is known as

the kernel correlation. The kernel correlation between two vectors x1 and x2 is represented as:𝑘𝑘𝑖𝑖𝐱𝐱𝟏𝟏𝐱𝐱𝟐𝟐 = k(x2, Δi-1 x1) (4.21)

Kernel autocorrelation vector of the input sample vector x can be evaluated and stored in a kernel vector like(4.21) as:

𝑘𝑘𝑖𝑖𝐱𝐱𝐱𝐱 = k(x, Δi-1 x) (4.23)

The kernel weight vector α can be evaluated from input sample kernel autocorrelation by simplifyingequation (4.19) and taking an inverse Fourier transform as:

α = F-1 (y (𝑘𝑘𝐱𝐱𝐱𝐱 + λ)-1) (4.24) Equation (4.24) represents KRLS weight vector used to train the non-linear classifier. Conventional kernel computation has complexity O(n4). Equation (4.24) has a complexity of O(n2logn).

24

KCF TRACKINGNon-linear Regression: KRLS Detection After training the classifier, we detect object pixels from the testing sample. The kernelized classifier f(y) defined in equation (4.17) classifies all the elements in testing vector y to

determine them as foreground or background. Let By be the new classifier Kernel matrix for non-linear regression, similar to kernel matrix K defined in

equation (4.18). The cyclic shifts between base input training sample vector x and the base output testing sample vector y

generates all the possible training and testing sample vectors - used to develop By. Let 𝑘𝑘𝑖𝑖,𝑗𝑗

𝐱𝐱𝒚𝒚 is the kernel correlation between the base input training sample vector x and the base outputtesting sample vector y. Δ is the permutation matrix.

𝑘𝑘𝑖𝑖,𝑗𝑗𝐱𝐱𝒚𝒚 = k(Δj-1y, Δi-1 x) (4.25)

Let 𝑘𝑘𝐱𝐱𝒚𝒚 be the kernel correlation vector comprising of all the elements 𝑘𝑘𝑖𝑖,𝑗𝑗𝐱𝐱𝒚𝒚.

The Circulant matrix M(𝑘𝑘𝐱𝐱𝒚𝒚) of 𝑘𝑘𝐱𝐱𝒚𝒚 is used to evaluate the classifier kernel matrix By as:By = M(𝑘𝑘𝐱𝐱𝒚𝒚) (4.26)

The kernel classifier f(y) of equation (4.17) can be termed as the KRLS classifier and defined as:f(y) = (By)T α (4.27)

We can evaluate KRLS filter f(y) vector in the Fourier domain as:f(y) = F-1( 𝑘𝑘𝐱𝐱𝐱𝐱 ⊙ α ) (4.28)

25

KCF TRACKINGNon-linear Regression: KRLS Detection Radial basis kernel function following (4.20) like Gaussian kernel k(x1, x2) is used.

k(x1, x2) = exp( - 1𝜎𝜎2 ||x1 - x2||2 ), σ2 is the constant variance

Gaussian kernel correlation vector 𝑘𝑘𝐱𝐱𝟏𝟏𝐱𝐱𝟐𝟐 is evaluated in the Fourier domain as:

𝑘𝑘𝐱𝐱𝟏𝟏𝐱𝐱𝟐𝟐 = exp(- 1𝜎𝜎2(||x1||2 + ||x2||2 - 2F-1 (x1

* ⊙ x2))) (4.36)

Image Pre Processing: Fourier transforms are periodic. But in correlation filtering with images, left edge touch right edge and top

edge touch bottom edge, making them non-periodic.

Cosine window is applied on each pixel xij. Boundary pixel intensity becomes zero and center pixels becomes

highest magnitude pixels.

xij = (xijraw – 0.5) sin (πi/n) sin (πj/n), ∀(i,j) = 0,1,2,…….,n-1 (4.37)

26

ADAPTIVE REDETECTION Evaluating the saliency map in each image frame of a video sequence or evaluating the saliency map over the

entire image is computationally very expensive. The redetection of the object in our proposed tracker is directly related to the correlation response of the

KRLS filter. It has been seen experimentally that as the tracker starts failing to lose the object, its correlation response

starts going down. Let γ be the correlation filter response threshold that we will check for redetection. Let S be the adaptive search area for redetection whenever correlation filter response goes below the

threshold γ . Let height and width be the dimension of the last successful bounding box.

Case 1: If 0.4 ≤ γ1 < 0.5, height1 = 1.5* height, width1 = 1.5* width andS1 = height1* width1 = 2.25*S

Case 2: If 0.3 ≤ γ2 < 0.4, height2 = 2* height, width2 = 2* width andS2 = height2* width2 = 4*S

Case 3: If γ3 < 0.3, S3 = S i.e. searched over entire image This adaptive search area S is again fed as input to the fast MBD detector for redetection. The saliency map generation with post processing is provided as the initialization for the KRLS filter. Redetection is performed iteratively whenever the object is partially or absolutely outside the bounding box.

27

ALGORITHM 1: PROPOSED TRACKERInput: Image frames I1,I2,…..,In from the video sequence,where n is total number of frames, Gaussian kernel withvariance var, regularization parameter λ, correlationfilter response threshold γ.Output: Bounding box around target object in each ofoutput image frames O1,O2,…..,On of the video sequence.

function Oi = main(Ii, var, λ, γ)Ii = rgb2gray(Ii); // Converted to grayscale images_m1 = saliency_map(I1); // saliency map of 1st frameO1 = s_m1;Oi = correlation_response(s_m1, Ii, var, λ, γ);return;

end

29

ALGORITHM 1: PROPOSED TRACKER

31

ALGORITHM 1: PROPOSED TRACKER

32

EXPERIMENTATION AND RESULTS The results were presented in IEEE 28th International Conference on

Tools with Artificial Intelligence (ICTAI) at San Jose, 2016

System Information: Tested in Intel (R) Xeon (R) W3520 2.67 GHz CPU, 6 GB RAM OpenCV v.3.2.14, Visual Studio Code (C++) and MATLAB 2015b

Datasets: Aircraft (620 frames), big_2 (382 frames), airplane_001 (200 frames),airplane_004 (200 frames), airplane_005 (200 frames), airplane_006(200 frames), airplane_007 (200 frames), airplane_011 (300 frames), airplane_012 (300 frames), airplane_013 (300 frames), airplane_015(300 frames), airplane_016 (300 frames) from [43], youtube_dataset_2(475 frames) and youtube_dataset_3 (301 frames)

Competing Trackers: STC [46], CT [12], CN [32], DSST [33], SAMF [2], KCF [3] and Ours [47]

33

EXPERIMENTATION AND RESULTSGround Truth Annotation: Manually labelled pixel coordinates of a rectangular bounding

box x and y coordinates of top-leftmost corner, height and width

noted Bounding box must cover all the object pixels and minimum

background pixels 4,278 frames from 14 datasets were labelled

Deciding Factors: Execution Speed Bounding box around entire object in all frames

34

QUANTITATIVE EVALUATIONComparison Metric : Center Location Error (CLE) [30] - Average Euclidean distance between

the centermost pixel of the bounding box given by tracker and that byground truth. Lower CLE better tracker.

Precision Rate (PR) [30] – Percentage of frames where CLE is lowerthan a threshold distance (20 pixels chosen).

Success Rate (SR) [30] – Evaluates overlapping of bounding box.Overlap Rate=

𝑎𝑎𝑡𝑡∩𝑎𝑎𝑔𝑔𝑎𝑎𝑡𝑡∪𝑎𝑎𝑔𝑔

, Tracker successful if Overlap Rate>θ, θ=0.5 (chosen)

at Area of tracker’s bounding box, ag Area of tracker’s bounding box, θ Threshold

SR is the percentage of frames for which Overlap Rate > θ.

35

QUANTITATIVE EVALUATIONComparison Metric :

Trackers are sensitive to initialization.

One Pass Evaluation (OPE) [30] – Tracker ran from first frame to lastframe to compute all the metrics (CLE, PR, SR).

Temporal Robustness Evaluation (TRE) [30] – Compute OPE for asegment of video and average all the metric (CLE, PR, SR) from eachsegment.

36

QUANTITATIVE EVALUATIONTABLE I

QUANTITATIVE ANALYSIS OF PROPOSED AND SIX COMPETING TRACKERS ON 14DATASETS. THE BEST AND THE SECOND BEST RESULTS ARE HIGHLIGHTED WITHBOLD-FACE AND UNDERLINE-POINT-STYLES RESPECTIVELY.

OPE evaluated on 4,278 frames for each tracker (Total 29,946 frames). Each dataset divided into 20 segments for TRE evaluation and so 85,560 frames

for each tracker (Total 5,98,920 frames) .

OURS CT STC CN DSST SAMF KCF

Average Precision Rate (OPE) 0.83 0.15 0.49 0.44 0.46 0.48 0.44

Average Success Rate (OPE) 0.62 0.19 0.41 0.38 0.41 0.41 0.39

Average Precision Rate (TRE) 0.78 0.29 0.54 0.45 0.54 0.49 0.48

Average Success Rate (TRE) 0.56 0.31 0.41 0.39 0.46 0.43 0.42

CLE (in pixels) 9.02 232.72 31.97 84.85 54.92 45.71 91.88

Average Speed (fps) 108.94 22.11 32.87 30.04 7.09 5.53 65.28

37

QUANTITATIVE EVALUATIONOne Pass Evaluation (OPE): Precision Rate

TrackerDatasets OURS CT STC CN DSST SAMF KCF

Aircraft 0.85 0.18 0.60 0.16 0.17 0.60 0.51airplane_001 0.92 0.02 0.38 0.20 0.26 0.43 0.12airplane_004 0.79 0.25 0.42 0.44 0.50 0.21 0.37airplane_005 0.81 0.09 0.36 0.33 0.32 0.39 0.27airplane_006 0.92 0.22 0.65 0.54 0.54 0.75 0.53airplane_007 0.76 0.08 0.46 0.61 0.36 0.27 0.37airplane_011 0.90 0.27 0.31 0.43 0.28 0.31 0.25airplane_012 0.74 0.02 0.83 0.15 0.88 0.99 0.81airplane_013 0.89 0.02 0.26 0.32 0.32 0.19 0.12airplane_015 0.82 0.35 0.49 0.73 0.58 0.57 0.79airplane_016 0.83 0.18 0.65 0.76 0.73 0.52 0.45

big_2 0.89 0.31 0.85 0.82 0.91 0.94 0.85youtube_dataset_2 0.81 0.04 0.66 0.65 0.56 0.54 0.46

youtube_dataset_3 0.69 0.06 0.12 0.07 0.07 0.05 0.26

38

QUANTITATIVE EVALUATIONOne Pass Evaluation (OPE): Success Rate




youtube_dataset_3 0.62 0.10 0.16 0.24 0.22 0.06 0.29

39

QUANTITATIVE EVALUATIONOne Pass Evaluation (OPE): Precision Plot

40

QUANTITATIVE EVALUATIONOne Pass Evaluation (OPE): Success Plot

41

QUANTITATIVE EVALUATIONTemporal Robustness Evaluation (TRE): Precision Rate




youtube_dataset_3 0.73 0.13 0.39 0.22 0.29 0.24 0.34

42

QUANTITATIVE EVALUATIONTemporal Robustness Evaluation (TRE): Success Rate




youtube_dataset_3 0.61 0.17 0.25 0.27 0.33 0.28 0.31

43

QUANTITATIVE EVALUATIONTemporal Robustness Evaluation (TRE): Precision Plot

44

QUANTITATIVE EVALUATIONTemporal Robustness Evaluation (TRE): Success Plot

45

QUANTITATIVE EVALUATIONSpeed Comparison:



big_2 160.82 33.84 37.76 15.02 3.79 7.57 68.54youtube_dataset_2 83.23 18.76 32.80 25.91 6.12 9.07 124.93youtube_dataset_3 72.72 18.58 16.57 7.16 1.60 3.19 22.47

mean 108.94 22.11 32.87 30.04 7.09 5.53 65.28

46

QUALITATIVE EVALUATIONScale Variation: youtube_dataset_3 dataset

Ours

CT

STC

CN

DSST

SAMF

KCF

47

QUALITATIVE EVALUATIONPartial Occlusion: airplane_005 dataset [43]

Ours

CT

STC

CN

DSST

SAMF

KCF

48

QUALITATIVE EVALUATIONFast Motion: airplane_001 dataset [43]

Ours

CT

STC

CN

DSST

SAMF

KCF

49

QUALITATIVE EVALUATIONIllumination Variation: airplane_006 dataset [43]

Ours

CT

STC

CN

DSST

SAMF

KCF

50

QUALITATIVE EVALUATIONIn-plane Rotation: Aircraft dataset [43]

Ours

CT

STC

CN

DSST

SAMF

KCF

51

QUALITATIVE EVALUATIONOut-of-plane Rotation: big_2 dataset [43]

Ours

CT

STC

CN

DSST

SAMF

KCF

52

LIMITATIONS1. Failure with multiple objects

2. Bad result with MBD detection failure

53

CONCLUSIONS AND FUTURE WORK• We have defined a new computer vision problem of object detection

and tracking to guide an UAV in its navigation.

• Extensive research was performed to track down the most relevant

approaches.

• Proposed a method for tracking objects from a forward looking camera

and examined thoroughly with challenging datasets.

• Six other state-of-the-art trackers were extensively tested and

compared (both qualitatively and quantitatively).

• Obtained best result for most of the datasets and second best for few

of them in terms of speed and accuracy.

54

CONCLUSIONS AND FUTURE WORKOther Applications:

Collision Avoidance in Aero planes Automatic Navigation in Water Fire Detection

Automatic Traffic Surveillance Object Recognition and Classification Face Detection

55

REFERENCES1. A. Smeulders, D. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah, “Visual tracking: an experimental survey,” TPAMI, 2013.2. Y. Li and J. Zhu, “A scale adaptive kernel correlation filter tracker with feature integration,” in European Conference on Computer Vision. Springer,

2014, pp. 254–265.3. J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE Transactions on Pattern Analysis

and Machine Intelligence, vol. 37, no. 3, pp. 583–596, 2015.4. L. Itti and C. Koch, “Computational modelling of visual attention,” Nature reviews neuroscience, vol. 2, no. 3, pp. 194–203, 2001.5. H. Cholakkal, D. Rajan, and J. Johnson, “Top-down saliency with locality-constrained contextual sparse coding.” BMVC, 2015.6. J. Yang and M.-H. Yang, “Top-down visual saliency via joint crf and dictionary learning,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE

Conference on. IEEE, 2012, pp. 2296–2303.7. Y. Wei, F. Wen, W. Zhu, and J. Sun, “Geodesic saliency using background priors,” in European Conference on Computer Vision. Springer, 2012, pp. 29–

42.8. J. Zhang, S. Sclaroff, Z. Lin, X. Shen, B. Price, and R. Mech, “Minimum barrier salient object detection at 80 fps,” in Proceedings of the IEEE

International Conference on Computer Vision, 2015, pp. 1404–1412.9. K. C. Ciesielski, R. Strand, F. Malmberg, and P. K. Saha, “Efficient algorithm for finding the exact minimum barrier distance,” Computer Vision and

Image Understanding, vol. 123, pp. 53–64, 2014.10. R. Strand, K. C. Ciesielski, F. Malmberg, and P. K. Saha, “The minimum barrier distance,” Computer Vision and Image Understanding, vol. 117, no. 4,

pp. 429–437, 2013.11. B. Babenko, M.-H. Yang, and S. Belongie, “Visual tracking with online multiple instance learning,” in Computer Vision and Pattern Recognition, 2009.

CVPR 2009. IEEE Conference on. IEEE, 2009, pp. 983–990.12. K. Zhang, L. Zhang, and M.-H. Yang, “Real-time compressive tracking,” in European Conference on Computer Vision. Springer, 2012, pp. 864–877.13. S. Hare et al., "Struck: Structured Output Tracking with Kernels," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 10, pp.

2096-2109, Oct. 1 2016.doi: 10.1109/TPAMI.2015.2509974

14. Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,” IEEE transactions on pattern analysis and machine intelligence, vol. 34, no. 7, pp.1409–1422, 2012.

15. A. Jordan, “On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes,” Advances in neural informationprocessing systems, vol. 14, p. 841, 2002.

16. M. Andriluka, S. Roth, and B. Schiele, “People-tracking-by-detection and people-detection-by-tracking,” in Computer Vision and Pattern Recognition,2008. CVPR 2008. IEEE Conference on. IEEE, 2008, pp. 1–8.

56

REFERENCES17. V. Mahadevan and N. Vasconcelos, “Saliency-based discriminant tracking,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE

Conference on. IEEE, 2009, pp. 1007–1013.18. A. Nussberger, H. Grabner, and L. Van Gool, “Robust aerial object tracking in high dynamic flight maneuvers,” ISPRS Annals of the Photogrammetry,

Remote Sensing and Spatial Information Sciences, vol. 2, no. 1, p. 1, 2015.19. A. Kocak, K. Cizmeciler, A. Erdem, and E. Erdem, “Top down saliency estimation via superpixel-based discriminative dictionaries.” in BMVC, 2014.20. S. He, R. W. Lau, W. Liu, Z. Huang, and Q. Yang, “Supercnn: A superpixelwise convolutional neural network for salient object detection,” International

Journal of Computer Vision, vol. 115, no. 3, pp. 330–344, 2015.21. P. Wang, J. Wang, G. Zeng, J. Feng, H. Zha, and S. Li, “Salient object detection for searched web images via global saliency,” in Computer Vision and

Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 3194–3201.22. R. Zhao, W. Ouyang, H. Li, and X. Wang, “Saliency detection by multi-context deep learning,” in Proceedings of the IEEE Conference on Computer

Vision and Pattern Recognition, 2015, pp. 1265–1274.23. Y. Sui, G. Wang, Y. Tang, and L. Zhang, “Tracking completion,” arXiv preprint arXiv:1608.08171, 2016.24. W. Zhu, S. Liang, Y. Wei, and J. Sun, “Saliency optimization from robust background detection,” in Proceedings of the IEEE conference on computer

vision and pattern recognition, 2014, pp. 2814–2821.25. J. Feng, Y. Wei, L. Tao, C. Zhang, and J. Sun, “Salient object detection by composition,” in 2011 International Conference on Computer Vision. IEEE,

2011, pp. 1028–1035.26. P. Siva, C. Russell, T. Xiang, and L. Agapito, “Looking beyond the image: Unsupervised learning for object saliency and detection,” in Proceedings of

the IEEE conference on computer vision and pattern recognition, 2013, pp. 3238–3245.27. A. Borji, M.-M. Cheng, H. Jiang, and J. Li, “Salient object detection: A survey,” arXiv preprint arXiv:1411.5878, 2014.28. D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” in Computer Vision and Pattern

Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 2544–2550.29. J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “Exploiting the circulant structure of tracking-by-detection with kernels,” in European conference

on computer vision. Springer, 2012, pp. 702–715.30. Y. Wu, J. Lim, and M.-H. Yang, “Online object tracking: A benchmark,” in Proceedings of the IEEE conference on computer vision and pattern

recognition, 2013, pp. 2411–2418.31. Y. Sui, Z. Zhang, G. Wang, Y. Tang, and L. Zhang, “Real-time visual tracking: Promoting the robustness of correlation filter learning,” arXiv preprint

arXiv:1608.08173, 2016.32. M. Danelljan, F. Shahbaz Khan, M. Felsberg, and J. Van de Weijer, “Adaptive color attributes for real-time visual tracking,” in Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition, 2014, pp. 1090–1097.

57

REFERENCES33. M. Danelljan, G. H¨ager, F. Khan, and M. Felsberg, “Accurate scale estimation for robust visual tracking,” in British Machine Vision Conference,

Nottingham, September 1-5, 2014. BMVA Press, 2014.34. T. Liu, G. Wang, and Q. Yang, “Real-time part-based visual tracking via adaptive correlation filters,” in Proceedings of the IEEE Conference on Computer

Vision and Pattern Recognition, 2015, pp. 4902–4912.35. C. Ma, X. Yang, C. Zhang, and M.-H. Yang, “Long-term correlation tracking,” in Proceedings of the IEEE Conference on Computer Vision and Pattern

Recognition, 2015, pp. 5388–5396.36. S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural

information processing systems, 2015, pp. 91–99.37. C. Ma, J.-B. Huang, X. Yang, and M.-H. Yang, “Hierarchical convolutional features for visual tracking,” in Proceedings of the IEEE International

Conference on Computer Vision, 2015, pp. 3074–3082.38. H. Li, Y. Li, and F. Porikli, “Deeptrack: Learning discriminative feature representations online for robust visual tracking,” IEEE Transactions on Image

Processing, vol. 25, no. 4, pp. 1834–1848, 2016.39. R. M. Gray, Toeplitz and circulant matrices: A review. now publishers inc, 2006.40. B. Scholkopf and A. J. Smola, Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.41. R. Rifkin, G. Yeo, and T. Poggio, “Regularized least-squares classification,” Nato Science Series Sub Series III Computer and Systems Sciences, vol. 190,

pp. 131–154, 2003.42. N. Otsu, “A threshold selection method from gray-level histograms,” Automatica, vol. 11, no. 285-296, pp. 23–27, 1975.43. A. Li, M. Lin, Y. Wu, M. Yang, and S. Yan, “NUS-PRO: A New Visual Tracking Challenge,” IEEE Transactions on Pattern Analysis and Machine

Intelligence, vol. 38, no. 2, pp. 335–349, 2016.44. A. Rosenfeld, J. L. Pfaltz, Distance functions on digital pictures, Pattern Recognition 1 (1968) 33-61.45. Y. Zhai, M. Shah (2006), Visual attention detection in video sequences using spatiotemporal cues, MM '06 Proceedings of the 14th ACM international

conference on Multimedia. Pages 815-82446. Zhang, Kaihua, "Fast visual tracking via dense spatio-temporal context learning." in European Conference on Computer Vision. Springer International

Publishing, 2014.47. Bharati, S., Nandi, S., Wu, Y., Sui, Y., Wang, G. (2016), “Fast and Robust Object Tracking with Adaptive Detection”, The 28th IEEE International

Conference on Tools with Artificial Intelligence (ICTAI), 2016.

58

ACKNOWLEDGEMENTS Dr. Richard Wang, EECS Department, University of Kansas.

Dr. James Rowland, EECS Department, University of Kansas

Dr. James Stiles, EECS Department, University of Kansas.

My sincere respect is shown to all the faculty and staff members of EECS

Department, University of Kansas.

I want to express my gratitude for all the students and researchers at the

Computer Vision group, University of Kansas.

I want to thank all my friends in Lawrence for their support in this journey.

59

Thank you

object tracking review - ku ittc · robust object tracking and adaptive detection ... • object...

Documents