![Page 1: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/1.jpg)
Edge-Adaptive Image Interpolation with
Contour Stencils
Pascal Getreuer
Dec 27, 2010
![Page 2: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/2.jpg)
TV along Curves
Let u be an image. For C a smooth simple curve, define
‖u‖TV(C) =
∫ T
0
∣∣ ∂∂t
u(γ(t)
)∣∣ dt, γ : [0,T ]→ C .
Strategy: Find approximate con-tours of u by finding curves C suchthat ‖u‖TV(C) is small.
γ′(t)
C
![Page 3: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/3.jpg)
Contour Stencils
A contour stencil is a function S : Z2 × Z2 → R describingedges between pixels, and TV is estimated as
(S ? [u])(k) :=∑
m,n∈Z2
S(m, n) |uk+m − uk+n| ≈ ‖u‖TV(C)
where S describes edges that approximate C .+n1
+n2
(0, 0)(S ? [u])(i , j) =
(|ui,j−1−ui+1,j |
+ |ui−1,j−1 − ui,j | + |ui,j − ui+1,j+1|+ |ui−1,j−ui,j+1|
).
![Page 4: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/4.jpg)
Contour Stencils
Estimate the contours locally by finding a stencil with low TV,
S?(k) = arg minS∈Σ
(S ? [u])(k)
where Σ is a set of candidate stencils.
Σ =
![Page 5: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/5.jpg)
Contour Stencils
Input Contour Stencils Sobel
For each pixel, S?(k) is determined to estimate the localcontour orientation.
![Page 6: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/6.jpg)
Why TV?
Total variation is invariant under diffeomorphisms on space.Consider the change of variables
t = ϕ(s)
dt = ϕ′(s) ds
u(t) u(ϕ(s)
)
and suppose that ϕ′(s) > 0, then∫ T
0
|u′(t)| dt =
∫ ϕ−1(T )
ϕ−1(0)
∣∣u′(ϕ(s))∣∣ϕ′(s) ds
=
∫ ϕ−1(T )
ϕ−1(0)
∣∣ ∂∂s
u(ϕ(s)
)∣∣ ds.
![Page 7: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/7.jpg)
Interpolation Problem
Given discrete image v and point spread function h(x , y), findfunction u(x , y) such that
vi ,j = (h ∗ u)(i , j) for all i , j .
Input v Interpolation u
![Page 8: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/8.jpg)
Edge Directed Interpolation
Theorem (Edge Directed Interpolation)
Consider approximating u(x) by
u(x) = (1− λ)u(a) + λu(b)a
b
x
C1
C2
and let C = C1 ∪ C2 be a curve passing through a, x, and b.Then the approximation error is bounded by
|u(x)− u(x)| ≤ max|1− λ| , |λ|
‖u‖TV(C) .
Choosing the stencil with the smallest TV minimizes theestimated interpolation error:
|u(x)− u(x)| ≤ ‖u‖TV(C) ≈1|S?|(S
??[u])(k) = minS∈Σ
1S (S?[u])(k).
![Page 9: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/9.jpg)
Contour Stencil Windowed Zooming
Local reconstructions:
uk(x) = vk +∑n∈N
cn ϕnS?(k)(x − n), θ
k +N
k
if S =
k +N
k
if S =
where
vk : kth pixel of input image
N : neighborhood
ϕnS?(k) : function oriented with the
best-fitting stencil S?(k)
cn : coefficients such that(h ∗ uk)(m) = vk+m for m ∈ N
![Page 10: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/10.jpg)
Contour Stencil Windowed Zooming
Combine local reconstructions with overlapping windows
u(x) =∑k∈Z2
w(x − k)uk(x − k),
where w satisfies∑k w(x − k) ≡ 1 s.t. method reproduces constants
w(k) = 0 for k 6∈ N s.t. ↓(h ∗ u) ≈ v
w compact support for computational efficiency
Example: Cubic B-spline
w(x) = B(x1)B(x2)
B(t) =(1− |t|+ 1
6|t|3 − 1
3
∣∣1− |t|∣∣3)+
![Page 11: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/11.jpg)
Original Image (332×300) Input Image (83×75)
Estimated Orientations Proposed Interpolation (PSNR 25.97, 0.125 s)
![Page 12: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/12.jpg)
Cubic B-spline (PSNR 25.92, 0.011 s) Fourier (PSNR 25.34, 0.062 s)
TV Minimization (PSNR 25.73, 0.784 s) Proposed Interpolation (PSNR 25.97, 0.125 s)
![Page 13: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/13.jpg)
AQua-2 (PSNR 24.72, 0.016 s) Fractal Zooming (PSNR 24.65)
Roussos (PSNR 25.87, 2.518 s) Proposed Interpolation (PSNR 25.97, 0.125 s)
![Page 14: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/14.jpg)
Zooming Comparison
Average PSNR on the Kodak Image Suite
Zoom Factor 2× 3× 4×AQua-2 25.06 22.48 21.35Fractal Zooming 29.00 27.20 25.25Contour Stencils 29.87 27.77 25.93Roussos 30.56 27.97 26.19
Computation Time (s) vs. Output Image Size
Image Size 128× 128 256× 256 512× 512AQua-2 0.0048 0.017 0.068Contour Stencils 0.025 0.088 0.34Roussos 0.23 2.22 8.64
![Page 15: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/15.jpg)
Analysis of Contour Stencils
CC
C ≈ C =⇒ ‖u‖TV(C) ≈ ‖u‖TV(C)
Curve Perturbation
Let C and C be smooth curves parameterized byγ : [0,T ]→ C and γ : [0,T ]→ C . Then if u is twicecontinuously differentiable,∣∣‖u‖TV(C) − ‖u‖TV(C)
∣∣≤∥∥|∇u|
∥∥∞
∥∥|γ′ − γ′|∥∥1
+ ‖∇2u‖∞( |C |+|C |
2+ 1
4
∥∥|γ′ − γ′|∥∥1
)∥∥|γ − γ|∥∥∞.
![Page 16: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/16.jpg)
Analysis of Contour Stencils
u ∈ C 2 =⇒ discrete TV is first-order accurate
TV Discretization
Suppose u ∈ C 2[0,T ] and 0 = t0 < t1 < · · · < tN = T , anddefine hi = ti − ti−1. Then
‖u‖TV−13T h2
max
havg‖u′′‖∞ ≤
N∑i=1
|u(ti)− u(ti−1)| ≤ ‖u‖TV .
![Page 17: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/17.jpg)
Analysis of Contour Stencils
Let S?2(k) denote the second best-fitting stencil and definethe separation sepv (k) between the first and second best,
S?2(k) := arg minS∈Σ\S?(k)
(S ? [v ])(k),
sepv (k) :=(S?2(k) ? [v ]
)(k)−
(S?(k) ? [v ]
)(k).
Stability of the Best-Fitting Stencil
Suppose that for two images v and v
sepv (k) > 2M ‖v − v‖2 ,
M = maxS∈Σ
[∑m∈Z
(∑n∈Z
∣∣S(m, n) + S(n,m)∣∣)2]1/2
.
Then they have the same best-fitting stencil at k .
![Page 18: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/18.jpg)
Contour Stencil Design
Let f 1, . . . , f J, f j : Z2 → R, be a set of image features.
Example: f j(x) = x1 sin π8j − x2 cos π
8j , j = 0, . . . , 7
We want to design stencils S1, . . . ,SJ that distinguishbetween these features.
![Page 19: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/19.jpg)
Contour Stencil Design
We want stencil S j to be the best-fitting stencil on f j ,
Want: 1|S j |(S
j ? [f j ])(0) < 1|S i |(S
i ? [f j ])(0) for all i 6= j .
Ignoring the 1|S| normalizations, this condition becomes
(S j ? [f j ])(0) < (S i ? [f j ])(0)
=⇒((S j − S i) ? [f j ]
)(0) < 0.
We can try to satisfy this condition by minimizing
minS1,...,SJ
J∑i=1
J∑j=1
((S j − S i) ? [f j ]
)(0) + γ
J∑j=1
‖S j‖1
s.t. 0 ≤ S j(m, n) ≤ 1
![Page 20: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/20.jpg)
Contour Stencil Design
minS1,...,SJ
J∑i=1
J∑j=1
((S j − S i) ? [f j ]
)(0) + γ
J∑j=1
‖S j‖1
s.t. 0 ≤ S j(m, n) ≤ 1
The minimization has closed-form solution
S j(m, n) =
1 if |f j
m − f jn | < 1
J
∑Ji=1|f i
m − f in | −
γJ,
0 if |f jm − f j
n | > 1J
∑Ji=1|f i
m − f in | −
γJ.
![Page 21: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/21.jpg)
Corner-Shaped Stencils
Example: Corner-shaped stencils designed from the features
f j(x) = maxx1 cos π4j − x2 sin π
4j ,
x1 sin π4j + x2 cos π
4j.
![Page 22: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/22.jpg)
3D Stencils
In d dimensions, a stencil S : Zd × Zd → R is applied at voxelk ∈ Zd as
(S ? [u])(k) :=∑
m,n∈Zd
S(m, n) |uk+m − uk+n| .
Small TV detects isosurfaces. Some 3D stencils:
![Page 23: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/23.jpg)
3D Stencils
Example: Stencils applied to ui ,j ,k =√
i2 + j2 + k2
The results are visualized by assigning a color to the region ofspace having a particular best-fitting stencil.
![Page 24: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/24.jpg)
3D Stencils
Example: Stencils applied to an MRI brain volume
![Page 25: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/25.jpg)
Demosaicing
Original Mosaiced Demosaiced
Bayer Grid
![Page 26: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/26.jpg)
Demosaicing Stencils
Centered on a green pixel:
Axially-oriented Diagonally-oriented π8
-oriented
Centered on a red or blue pixel:
Axially-oriented Diagonally-oriented π8
-oriented
![Page 27: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/27.jpg)
Demosaicing Stencils
Stencil orientation estimation on mosaiced data:
![Page 28: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/28.jpg)
Demosaicing Stencils
Stencil orientation estimation on mosaiced data:
![Page 29: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/29.jpg)
Preliminary Demosaicing Method
Let f be the given mosaiced image. We consider demosaicingby the minimization of
arg minu
∑k∈Y ,Cb,Cr
∑n∈Ω
∑m∈N (n)
wm,n
∣∣u(k)m − u(k)
n
∣∣+
λ
2
∑k∈R,G ,B
∑n∈Ω(k)
(fn − u(k)n )2
where
wm,n : weights choosen according the best-fitting stencils
N (n) : neighbors of pixel n
λ : fidelity parameter
Ω(k) : subset of Ω where kth channel is given
![Page 30: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/30.jpg)
Exact
Demosaiced
Exact
Demosaiced
Exact
Demosaiced
![Page 31: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/31.jpg)
Exact
Bilinear
Malvar et al.
Hamilton-Adams
Gunturk et al.
Li
Zhang-Wu
Proposed
![Page 32: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/32.jpg)
Exact
Bilinear
Malvar et al.
Hamilton-Adams
Gunturk et al.
Li
Zhang-Wu
Proposed
![Page 33: Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original Image (332 300) Input Image (83 75) Estimated Orientations Proposed Interpolation (PSNR](https://reader034.vdocuments.pub/reader034/viewer/2022042415/5f2fc225c757ce5532725399/html5/thumbnails/33.jpg)
Exact
Bilinear
Malvar et al.
Hamilton-Adams
Gunturk et al.
Li
Zhang-Wu
Proposed