Accelerating Spatially Varying Gaussian Filters

Jongmin Baek and David E. JacobsStanford University

Motivation

Input

GaussianFilter

SpatiallyVaryingGaussianFilter

1) Accelerating Spatially Varying Gaussian Filters

2) Accelerating Spatially Varying Gaussian Filters

3) Accelerating Spatially Varying Gaussian Filters

4) Applications

Roadmap

Gaussian Filters

Given pairs as input,

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅ exp (− ∥𝑝𝑖−𝑝 𝑗∥2

2 σ2 )

Position Value

Gaussian Filters

Each output value …

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅ exp (− ∥𝑝𝑖−𝑝 𝑗∥2

2 σ2 )

Gaussian Filters

… is a weighted sum of input values …

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅ exp (− ∥𝑝𝑖−𝑝 𝑗∥2

2 σ2 )

Gaussian Filters

… whose weight is a Gaussian …

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅ exp (− ∥𝑝𝑖−𝑝 𝑗∥2

2 σ2 )

Gaussian Filters

… in the space of the associated positions.

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅ exp (− ∥𝑝𝑖−𝑝 𝑗∥2

2 σ2 )

Gaussian Blur

Gaussian Filters: Uses

𝒑 𝒊=(𝒙 𝒊 , 𝒚 𝒊 ) ,𝒗 𝒊=(𝒓 𝒊 ,𝒈𝒊 ,𝒃𝒊)

Bilateral Filter

Gaussian Filters: Uses

𝒑 𝒊=(𝒙 𝒊 , 𝒚 𝒊 ,𝒓 𝒊 ,𝒈𝒊 ,𝒃𝒊 ) ,𝒗 𝒊=(𝒓 𝒊 ,𝒈𝒊 ,𝒃𝒊)

Non-local Means Filter

Gaussian Filters: Uses

𝒑 𝒊=(𝒙 𝒊 , 𝒚 𝒊 , )𝒗 𝒊=(𝒓 𝒊 ,𝒈𝒊 ,𝒃𝒊)

Applications Denoising images and meshes Data fusion and upsampling Abstraction / Stylization Tone-mapping ...

Gaussian Filters: Summary

Previous work on fast Gaussian Filters Bilateral Grid (Chen, Paris, Durand; 2007) Gaussian KD-Tree (Adams et al.; 2009) Permutohedral Lattice (Adams, Baek, Davis; 2010)

Summary of Previous Implementations:

A separable blur flanked by resampling operations. Exploit the separability of the Gaussian kernel.

Gaussian Filters: Implementations

Spatially Varying Gaussian Filters

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅N (𝑝𝑖−𝑝 𝑗 ;𝜎

2 𝐼 )

𝑣ˆ

𝑖=∑𝑗𝑣 𝑗⋅N (𝑝𝑖−𝑝 𝑗 ;𝐾 𝑖 )

Spatially varying covariance matrix

Spatially Invariant

Trilateral Filter (Choudhury and Tumblin, 2003)

Tilt the kernel of a bilateral filter along the image gradient.

“Piecewise linear”instead of“Piecewise constant”model.

Spatial Variance in Previous Work

Spatially Varying Gaussian Filters: Tradeoff

Benefits: Can adapt the kernel spatially. Better filtering performance.

Cost: No longer separable. No existing acceleration

schemes.

Input Bilateral-filtered Trilateral-filtered

Problem: Spatially varying (thus non-separable) Gaussian filter

Existing Tool: Fast algorithms for spatially invariant Gaussian filters

Solution: Re-formulate the problem to fit the tool. Need to obey the “piecewise-constant” assumption

Acceleration

Naïve Approach (Toy Example)I LOST THE GAME

Input Signal

Desired Kernel1 1 12 3 4

filtered w/ 1filtered w/ 2filtered w/ 3filtered w/ 4

1 1 12

3Output Signal4

In practice, the # of kernels can be very large.

Challenge #1

Pixel Location x

Desired Kernel K(x)

Range ofKernels needed

Sample a few kernels and interpolate.

Solution #1

Desired Kernel K(x)

Sampledkernels

Interpolate result!

Pixel Location x

K1

K2

K3

Interpolation needs an extra assumption to work:

The covariance matrix Ʃi is either piecewise-constant, or smoothly varying.

Kernel is spatially varying, but locally spatially invariant.

Assumptions

Runtime scales with the # of sampled kernels.

Challenge #2

Desired Kernel K(x)

Filter only some regions of the image with each kernel.

(“support”)

Pixel Location x

Sampledkernels

K1

K2

K3

In this example, x needs to be in the support of K1 & K2.

Defining the Support

Desired Kernel K(x)

Pixel Location x

K1

K2

K3

Dilating the Support

Desired Kernel K(x)

Pixel Location x

K1

K2

K3

Algorithm

1) Identify kernels to sample.2) For each kernel, compute the support needed.3) Dilate each support.4) Filter each dilated support with its kernel.5) Interpolate from the filtered results.

Algorithm

1) Identify kernels to sample.2) For each kernel, compute the support needed.3) Dilate each support.4) Filter each dilated support with its kernel.5) Interpolate from the filtered results.

K1

K2

K3

Algorithm

1) Identify kernels to sample.2) For each kernel, compute the support needed.3) Dilate each support.4) Filter each dilated support with its kernel.5) Interpolate from the filtered results.

K1

K2

K3

Algorithm

1) Identify kernels to sample.2) For each kernel, compute the support needed.3) Dilate each support.4) Filter each dilated support with its kernel.5) Interpolate from the filtered results.

K1

K2

K3

Algorithm

1) Identify kernels to sample.2) For each kernel, compute the support needed.3) Dilate each support.4) Filter each dilated support with its kernel.5) Interpolate from the filtered results.

K1

K2

K3

Algorithm

1) Identify kernels to sample.2) For each kernel, compute the support needed.3) Dilate each support.4) Filter each dilated support with its kernel.5) Interpolate from the filtered results.

K1

K2

K3

Applications

HDR Tone-mapping

Joint Range Data Upsampling

Application #1: HDR Tone-mapping

Input HDR

Detail

BaseFil

ter Output

Attenuate

Tone-mapping Example

Bilateral Filter Kernel Sampling

Application #2: Joint Range Data Upsampling

Range Finder Data

Sparse Unstructured Noisy

Scene Image

Output

Filter

Synthetic Example

Scene Image Ground Truth Depth

Synthetic Example

Scene Image Simulated Sensor Data

Synthetic Example : Result

Kernel SamplingBilateral Filter

Synthetic Example : Relative Error

Bilateral Filter Kernel Sampling

2.41% Mean Relative Error 0.95% Mean Relative Error

Real-World Example

Scene Image Range Finder Data

*Dataset courtesy of Jennifer Dolson, Stanford University

Real-World Example: Result

Input

Bilateral

Naive

KernelSampling

Performance

Kernel Sampling

Choudhury and Tumblin (2003) Naïve

Tonemap1

5.10 s 41.54 s 312.70 s

Tonemap2

6.30 s 88.08 s 528.99 sKernel

SamplingKernel Sampling

(No segmentation)Depth1 3.71 s 57.90 sDepth2 9.18 s 131.68 s

1. A generalization of Gaussian filters• Spatially varying kernels• Lose the piecewise-constant assumption.

2. Acceleration via Kernel Sampling• Filter only necessary pixels (and their support)

and interpolate.

3. Applications

Conclusion