dvpa: digital video processing & applications course...
TRANSCRIPT
DVPA: Digital Video Processing &
Applications
Course Introduction
郑喆坤西安电子科技大学电子工程学院
2015.09.17
Digital Video Processing & Analysis
Instructor: Cheolkon Jung
Office: #217, #100 Building
Email: [email protected]
Webpage: http://web.xidian.edu.cn/zhengzk
http://see.xidian.edu.cn/media
Lecture time:
Monday AM 8:30-10:00 (Every week);
Lecture place: J3-08, Teaching Building
Webpage http://web.xidian.edu.cn/zhengzk
Digital Video Processing & Analysis
Office:
北校区主楼III区309
Email:
Media Lab
Advanced media
Ultra high definition (UHD) video
High dynamic range (HDR) video
Surveillance video
Social media & Big data
Stereoscopic 3D video
Main research topics
Multimedia annotation, retrieval, and security
Image restoration & Super-resolution
Computational photography
Perceptual video coding and H.265
Stereoscopic 3D video processing
Topics include, but are not limited to them
Research In Progress
Research In Progress
1. Stereoscopic 3D Media
13
2D to 3D conversion 3DTV
Development of core technologies for stereoscopic 3D media:
1) Conversion of existing 2D into exciting 3D
2) Free viewpoint synthesis
3) High-quality stereoscopic view generation
4) Perceptual stereoscopic video coding
5) Real-time rendering using GPU
6) Visual fatigue prediction and reduction
Research In Progress
14
Various displays from small size to large size
: Image/video super-resolution based on machine learning
Different kinds of displays with various size such as TV, PC monitor, mobile phone
Resolution of images and videos is fixed.
No high frequency information while playing small size images in large size display
2. Super-resolution
3. Image Restoration
15
: Image restoration which preserves image details and reduces noise/artifacts
Most imaging devices use CCD/CMOS sensors.
Captured images are compressed for efficient transmission and storing.
Sensor noise and compression artifacts inevitably occur and degrade image quality.
Sensor noiseImaging Devices Compression
Artifact
Research In Progress
4. Semantic Concept Detection
16
Manual annotation for huge archive
is labor-intensive and time-consuming
Labeled samples are too small
Research In Progress
Exponentially Increasing Personal Contents
How to retrieve our favorite contents?
Semi-supervised
learning based
semantic concept
detection
5. Computational Photography
17
Research In Progress
Tone mapping
Mobile imaging devices with comparable computing power (iPhone 5: 1.5GHz CPU)
1) High dynamic range imaging by tone mapping and multi-exposure fusion
2) Specularity removal in color images
3) Interactive image segmentation and background change
4) High-quality motion deblurring
5) Power constrained image contrast enhancement
Background change
Outline
3
24
2 Why digital?
What is DVPA?
Why DVPA Is Required?
25 Applications of DVPA
26 Course overview
21 About Videos
18
Videos
Videos
Videos
3D Videos
3D Videos
Rich information from visual data.
Moving pictures: videos
Movies in Theaters
Broadcast contents: newscast, sportscast, entertainment and
education contents, etc.
Personal videos obtained by video camera, webcam, mobile cam,
etc.
CCTV surveillance videos
About Videos
Difference Between Images and Videos
Images
Shape
Color
Texture
Videos
Shape
Color
Texture
Motion
Depth in 3D videos
Motion
Depth
Human 3D: perceive depth information from two slightly
different views by two eyes.
Depth Perception
Depth
Human 3D: perceive depth information from two slightly
different views by two eyes.
Depth Perception
About Videos
Source of videos:
Captured with an analog or digital TV/movie camera
Time-based sequences: radar, satellite, or aerial surveillance
vehicle
Video units:
Why Digital?
Exactness
Perfect reproduction without degradation
Perfect duplication of processing result
Convenient & powerful computer-aided processing
Can perform rather sophisticated processing through hardware
or Software
Even kids can do it!
Easy storage and transmission
1 CD can store 1 video or hundreds of family photos!
Paperless transmission of high quality photos through network
within seconds
Films Sensors
What is DVPA?
DVPA deals with computational frameworks
To extract useful video information
To restore original video information
To compress huge amounts of video data
To represent (structure) raw unstructured videos
To assess video quality (video quality assessment)
DVPA:
What is DVPA?
Digital input: imaging devices (video cam, digital cam, web cam)
Low level processing: encoding (MPEG, JPEG, etc), video
processing & analysis (filtering, enhancement, etc)
Transmission/storage: channel coding (transmission error
minimization)
Remote video processing: decoding, video processing &
analysis
Digital output: displays (TV, mobile devices, etc)
Why DVPA is required?
Facilitate image storage and transmission: Efficiently store an image in a digital camera
Send images to other persons effectively
Image/video compression, motion estimation
Prepare for display or printing: Adjust image size
Video enhancement, video super-resolution, image half-toning
Enhance and restore images: Remove noise and artifacts from transmission or compression
Video denoising, compression artifact reduction
Extract useful information from images: Extract or track objects including faces, texts, and so on
Object segmentation & detection & recognition & tracking, behavior detection & understanding, video segmentation & representation & indexing & annotation & retrieval
Why DVPA is required?
Why DVPA is required?
Increasing Personal Contents Storage Size Explosion: 1 Terabyte (2007)
Multiple Channels: +300ch (2007)
Personal Contents Explosion
Difficult to Retrieve Want to Preview the ContentsNot Enough Time
to Consume
Why DVPA is required?
Popular mobile imaging devices: Smart phone, video camera, personal media player (PMP), etc
Storage size expansion: HDD, CD, DVD, flash memory, etc
Digital videos are increasing exponentially: Social, commercial, industry and military applications
Entertainment, education, medicine, databases, security, etc.
High-speed computing environments in most digital devices
Accordingly, DVPA has been practically applied to several video applications.
Why DVPA is required?
Video compression
Video enhancement
Video surveillance
Video content analysis
Video indexing & retrieval
Object detection & recognition
Motion estimation
Research Fields
Stereo video processing
Applications of DVPA
Video compression: Lossless & lossy compression
MPEG, H.264, AVC, M-JPEG, Microsoft VC-9, etc
Video enhancement: Video super-resolution
Video denoising & compression artifact reduction
Video surveillance: Video object detection, recognition, and tracking
Motion estimation and segmentation
Action recognition
Video content analysis: Video segmentation & indexing & browsing
Video summarization & retrieval
Stereo video processing: Depth estimation & 2D-to-3D conversion
Depth-image-based-rendering
Free viewpoint view synthesis
Image Superresolution
Optical Engineering 2013
Image Superresolution
(a) Original image, (b) Nearest neighbor, (c) Bicubic, (d) TOG 2008,
(e) TIP 2010, (f) TPAMI 2010, (g) Pro-I (coupled dictionary learning), (h) Pro-II (KSVD)
GPU-Accelerated Image Super-resolution
JRTIP 2015
De-convolution based image super-resolution on GPU and Multicore Platform
Gradient-consistency-anisotropic-regularization prior (Frequency domain)
Parallel Computing
IEEE ICME 2014; SPIC 2014
42
Face Hallucination
More natural-looking-facial images are required for
high-quality image communications
Resolution of images is fixed.
No high frequency information.
Learning based super-
resolution method
Low-Resolution
High-Resolution
Face Hallucination
High-Quality Image Communications
IEEE SPL 2011
43
Face Hallucination
More natural looking facial images!!
LR Proposed HR
Image Deblocking
<Learned Dictionary>
Block Diagram
Signal Processing: Image Communication 2012
ScienceDirect Top25 Hottest Article
Image Deblocking
Video Deblocking
Video Deblocking
Interactive Image Segmentation
Interactive Image Segmentation
Original Image Interactive Markers Segmentation Result
Pattern Recognition 2014
Step 1.
Interactive ROI
extraction (part 1).
Step 2.
Initial image ranking
(Part 2).
Step 3.
Relevance feedback
and global
discriminative
learning by ACSP
(Part 4).
Step 4.
Image reranking for
retrieval (Part 2).
Neurocomputing 2015
Interactive Image Retrieval
Interactive Image Retrieval
Neurocomputing 2015
52
Specularity Removal
Original Image Diffuse Reflection Specular Reflection
Optical Engineering 2013
53
Multi-exposure Fusion
Different exposure images Fusion result
Over
Normal
Under
Optical Engineering 2013
Sports Video Indexing
<Overview of the proposed method>
IEEE TBC 2009
Sports Video Indexing
<Defined exciting events>
Recognized scores
Depth Estimation
Multimedia Systems 2015
57
2D-to-3D Conversion
Proposed Framework
IEEE TMM 2014
2D-to-3D Conversion
IEEE TMM 2014
2D-to-3D Conversion
60
Depth-Image-Based Rendering
Holes Proposed MethodIEEE ICME 2014
Depth-Image-Based Rendering
Stereo-Plus-Depth Imaging System
Stereo+Depth Imaging System
Xidian & SKKU
Joint Research
Superpixel Matching Based Depth Propagation
IEEE ICIP 2015
Block Matching Proposed Ground truth
Outline
3
24
2 Why digital?
What is DVPA?
Why DVPA Is Required?
25 Applications of DVPA
26 Course overview
21 About Videos
Course Overview
Main topics: Some Basics on Digital Videos
Video Modeling
Motion Estimation
Video Compression
Video Content Analysis
Stereo and 3D Videos
Text book:
Main: 视频信号处理与通信 (Video processing and communications by Wang et al., 清华大学出版社)
Side: Journal or Conference Papers
Course Overview
Methods Lecturer: provide high-quality lecture
• Provide lectures on some basic theories such as main topics of DVPA
• Guide attendees to achieve successful study results
• Evaluate proposal/presentation/demonstration/technical report
Attendees: deal with one of the practical DVPA issues• Select one issue among the recommended candidates by the
lecturer
• Make a proposal and study it after the lecturer confirmation
• Present and demonstrate the study results
• Submit the technical report & source code (C or C++)
All of the lectures will be provided in English
Proposal/presentation/technical report should be provided in English
Course Overview
Schedule (total 32 hours):
Course introduction (2 hours)
Basics of analog and digital video (6 hours)
Motion estimation and analysis (4 hours)
Video filtering and enhancement (6 hours)
Video compression & standards (6 hours)
Applications (8 hours)
Evaluation:
Attendance (10)
Student project (60+Bonus)
Final examination (20)
Homework (10)
Course Overview
Criteria:
Difficulty
Execution (completion)
Presentation (attitude, technical writing)
If there is novelty in the technical report, bonus points will be
given.
