dvpa: digital video processing & applications course...

DVPA: Digital Video Processing &

Applications

Course Introduction

郑喆坤西安电子科技大学电子工程学院

zhengzk@xidian.edu.cn

2015.09.17

Digital Video Processing & Analysis

Instructor: Cheolkon Jung

Office: #217, #100 Building

Email: zhengzk@xidian.edu.cn

Webpage: http://web.xidian.edu.cn/zhengzk

http://see.xidian.edu.cn/media

Lecture time:

Monday AM 8:30-10:00 (Every week);

Lecture place: J3-08, Teaching Building

Webpage http://web.xidian.edu.cn/zhengzk

Digital Video Processing & Analysis

Office:

北校区主楼III区309

Email:

zhengzk@xidian.edu.cn

Media Lab

Advanced media

Ultra high definition (UHD) video

High dynamic range (HDR) video

Surveillance video

Social media & Big data

Stereoscopic 3D video

Main research topics

Multimedia annotation, retrieval, and security

Image restoration & Super-resolution

Computational photography

Perceptual video coding and H.265

Stereoscopic 3D video processing

Topics include, but are not limited to them

Research In Progress

Research In Progress

1. Stereoscopic 3D Media

13

2D to 3D conversion 3DTV

Development of core technologies for stereoscopic 3D media:

1) Conversion of existing 2D into exciting 3D

2) Free viewpoint synthesis

3) High-quality stereoscopic view generation

4) Perceptual stereoscopic video coding

5) Real-time rendering using GPU

6) Visual fatigue prediction and reduction

Research In Progress

14

Various displays from small size to large size

: Image/video super-resolution based on machine learning

Different kinds of displays with various size such as TV, PC monitor, mobile phone

Resolution of images and videos is fixed.

No high frequency information while playing small size images in large size display

2. Super-resolution

3. Image Restoration

15

: Image restoration which preserves image details and reduces noise/artifacts

Most imaging devices use CCD/CMOS sensors.

Captured images are compressed for efficient transmission and storing.

Sensor noise and compression artifacts inevitably occur and degrade image quality.

Sensor noiseImaging Devices Compression

Artifact

Research In Progress

4. Semantic Concept Detection

16

Manual annotation for huge archive

is labor-intensive and time-consuming

Labeled samples are too small

Research In Progress

Exponentially Increasing Personal Contents

How to retrieve our favorite contents?

Semi-supervised

learning based

semantic concept

detection

5. Computational Photography

17

Research In Progress

Tone mapping

Mobile imaging devices with comparable computing power (iPhone 5: 1.5GHz CPU)

1) High dynamic range imaging by tone mapping and multi-exposure fusion

2) Specularity removal in color images

3) Interactive image segmentation and background change

4) High-quality motion deblurring

5) Power constrained image contrast enhancement

Background change

Outline

3

24

2 Why digital?

What is DVPA?

Why DVPA Is Required?

25 Applications of DVPA

26 Course overview

21 About Videos

18

Videos

Videos

Videos

3D Videos

3D Videos

Rich information from visual data.

Moving pictures: videos

Movies in Theaters

Broadcast contents: newscast, sportscast, entertainment and

education contents, etc.

Personal videos obtained by video camera, webcam, mobile cam,

etc.

CCTV surveillance videos

About Videos

Difference Between Images and Videos

Images

Shape

Color

Texture

Videos

Shape

Color

Texture

Motion

Depth in 3D videos

Motion

Depth

Human 3D: perceive depth information from two slightly

different views by two eyes.

Depth Perception

Depth

Human 3D: perceive depth information from two slightly

different views by two eyes.

Depth Perception

About Videos

Source of videos:

Captured with an analog or digital TV/movie camera

Time-based sequences: radar, satellite, or aerial surveillance

vehicle

Video units:

Why Digital?

Exactness

Perfect reproduction without degradation

Perfect duplication of processing result

Convenient & powerful computer-aided processing

Can perform rather sophisticated processing through hardware

or Software

Even kids can do it!

Easy storage and transmission

1 CD can store 1 video or hundreds of family photos!

Paperless transmission of high quality photos through network

within seconds

Films Sensors

What is DVPA?

DVPA deals with computational frameworks

To extract useful video information

To restore original video information

To compress huge amounts of video data

To represent (structure) raw unstructured videos

To assess video quality (video quality assessment)

DVPA:

What is DVPA?

Digital input: imaging devices (video cam, digital cam, web cam)

Low level processing: encoding (MPEG, JPEG, etc), video

processing & analysis (filtering, enhancement, etc)

Transmission/storage: channel coding (transmission error

minimization)

Remote video processing: decoding, video processing &

analysis

Digital output: displays (TV, mobile devices, etc)

Why DVPA is required?

Facilitate image storage and transmission: Efficiently store an image in a digital camera

Send images to other persons effectively

Image/video compression, motion estimation

Prepare for display or printing: Adjust image size

Video enhancement, video super-resolution, image half-toning

Enhance and restore images: Remove noise and artifacts from transmission or compression

Video denoising, compression artifact reduction

Extract useful information from images: Extract or track objects including faces, texts, and so on

Object segmentation & detection & recognition & tracking, behavior detection & understanding, video segmentation & representation & indexing & annotation & retrieval

Why DVPA is required?

Why DVPA is required?

Increasing Personal Contents Storage Size Explosion: 1 Terabyte (2007)

Multiple Channels: +300ch (2007)

Personal Contents Explosion

Difficult to Retrieve Want to Preview the ContentsNot Enough Time

to Consume

Why DVPA is required?

Popular mobile imaging devices: Smart phone, video camera, personal media player (PMP), etc

Storage size expansion: HDD, CD, DVD, flash memory, etc

Digital videos are increasing exponentially: Social, commercial, industry and military applications

Entertainment, education, medicine, databases, security, etc.

High-speed computing environments in most digital devices

Accordingly, DVPA has been practically applied to several video applications.

Why DVPA is required?

Video compression

Video enhancement

Video surveillance

Video content analysis

Video indexing & retrieval

Object detection & recognition

Motion estimation

Research Fields

Stereo video processing

Applications of DVPA

Video compression: Lossless & lossy compression

MPEG, H.264, AVC, M-JPEG, Microsoft VC-9, etc

Video enhancement: Video super-resolution

Video denoising & compression artifact reduction

Video surveillance: Video object detection, recognition, and tracking

Motion estimation and segmentation

Action recognition

Video content analysis: Video segmentation & indexing & browsing

Video summarization & retrieval

Stereo video processing: Depth estimation & 2D-to-3D conversion

Depth-image-based-rendering

Free viewpoint view synthesis

Image Superresolution

Optical Engineering 2013

Image Superresolution

(a) Original image, (b) Nearest neighbor, (c) Bicubic, (d) TOG 2008,

(e) TIP 2010, (f) TPAMI 2010, (g) Pro-I (coupled dictionary learning), (h) Pro-II (KSVD)

GPU-Accelerated Image Super-resolution

JRTIP 2015

De-convolution based image super-resolution on GPU and Multicore Platform

Gradient-consistency-anisotropic-regularization prior (Frequency domain)

Parallel Computing

IEEE ICME 2014; SPIC 2014

42

Face Hallucination

More natural-looking-facial images are required for

high-quality image communications

Resolution of images is fixed.

No high frequency information.

Learning based super-

resolution method

Low-Resolution

High-Resolution

Face Hallucination

High-Quality Image Communications

IEEE SPL 2011

43

Face Hallucination

More natural looking facial images!!

LR Proposed HR

Image Deblocking

<Learned Dictionary>

Block Diagram

Signal Processing: Image Communication 2012

ScienceDirect Top25 Hottest Article

Image Deblocking

Video Deblocking

Video Deblocking

Interactive Image Segmentation

Interactive Image Segmentation

Original Image Interactive Markers Segmentation Result

Pattern Recognition 2014

Step 1.

Interactive ROI

extraction (part 1).

Step 2.

Initial image ranking

(Part 2).

Step 3.

Relevance feedback

and global

discriminative

learning by ACSP

(Part 4).

Step 4.

Image reranking for

retrieval (Part 2).

Neurocomputing 2015

Interactive Image Retrieval

Interactive Image Retrieval

Neurocomputing 2015

52

Specularity Removal

Original Image Diffuse Reflection Specular Reflection

Optical Engineering 2013

53

Multi-exposure Fusion

Different exposure images Fusion result

Over

Normal

Under

Optical Engineering 2013

Sports Video Indexing

<Overview of the proposed method>

IEEE TBC 2009

Sports Video Indexing

<Defined exciting events>

Recognized scores

Depth Estimation

Multimedia Systems 2015

57

2D-to-3D Conversion

Proposed Framework

IEEE TMM 2014

2D-to-3D Conversion

IEEE TMM 2014

2D-to-3D Conversion

60

Depth-Image-Based Rendering

Holes Proposed MethodIEEE ICME 2014

Depth-Image-Based Rendering

Stereo-Plus-Depth Imaging System

Stereo+Depth Imaging System

Xidian & SKKU

Joint Research

Superpixel Matching Based Depth Propagation

IEEE ICIP 2015

Block Matching Proposed Ground truth

Outline

3

24

2 Why digital?

What is DVPA?

Why DVPA Is Required?

25 Applications of DVPA

26 Course overview

21 About Videos

Course Overview

Main topics: Some Basics on Digital Videos

Video Modeling

Motion Estimation

Video Compression

Video Content Analysis

Stereo and 3D Videos

Text book:

Main: 视频信号处理与通信 (Video processing and communications by Wang et al., 清华大学出版社)

Side: Journal or Conference Papers

Course Overview

Methods Lecturer: provide high-quality lecture

• Provide lectures on some basic theories such as main topics of DVPA

• Guide attendees to achieve successful study results

• Evaluate proposal/presentation/demonstration/technical report

Attendees: deal with one of the practical DVPA issues• Select one issue among the recommended candidates by the

lecturer

• Make a proposal and study it after the lecturer confirmation

• Present and demonstrate the study results

• Submit the technical report & source code (C or C++)

All of the lectures will be provided in English

Proposal/presentation/technical report should be provided in English

Course Overview

Schedule (total 32 hours):

Course introduction (2 hours)

Basics of analog and digital video (6 hours)

Motion estimation and analysis (4 hours)

Video filtering and enhancement (6 hours)

Video compression & standards (6 hours)

Applications (8 hours)

Evaluation:

Attendance (10)

Student project (60+Bonus)

Final examination (20)

Homework (10)

Course Overview

Criteria:

Difficulty

Execution (completion)

Presentation (attitude, technical writing)

If there is novelty in the technical report, bonus points will be

given.