Visual Computing Theory and

Engineering Applications

Li Song(宋利)

Bingbing Ni (倪兵兵)

http://medialab.sjtu.edu.cn

Shanghai Jiao tong University

Welcome all of you!

What is visual computing?

.

Design algorithm or system that can interpret or

processing images/video

1. Interpret: Patter

Recognition, Image Search,

Video Mining; Data ->

Semantic

2. Processing:

Filtering/Transform/Analysis;

Signal -> Signal or Feature

3. “Visual” and “Computing” ：

Real problem need both

Why study visual computing?

.

Images and movies are everywhere

Our 80% information come from them

Fast-growing collection of useful applications

3D Contents Creation : Avatar

Smart Surveillance : Object Video

Movie and TV post processing : IMAX

Greater understanding of human vision

how does object recognition work?

Let us see some

interesting images firstly!

Parallel Lines?

Crossing Lines?

Circle?

Which bigger?

Which longer?

Static or moving picture?

Do you see waving?

What do you see?

What do you see?

Impossible things?

What are related branches?

There is still not a general theory for vision, problem driven solution now…

What this course is about?

3D Reconstruction View Marr’s vision theory

Edge extraction ( and segmentation )

Geometry( and Motion, Depth)

Patter Recognition View Invariance

Detection

Tracking

Recognition

Computational Neuroscience View Human vision system

Neural Computing

Two free reference textbooks

• R. Szeliski, "Computer

Vision: Algorithms and

Applications", 2010.

• S. Prince, "Computer vision:

models, learning and inference",

2012.

Marr’s Vision Theory

3D Reconstruction

Stereo reconstruction 3D from multiple views

It can only handle simple objects in real world

Edge detection itself is difficult without

information of high layer

The function of Human Vision is more than 3D

reconstruction.

It is found that visual neurons work in the ways of

both bottom-up and top-down.

Challenge of Reconstruction

Detection

Tracking

CAMSGPF PF

Recognition

Search by example

Recognition

Video Index and retrieval

Recognition

Face detection

Recognition

Event detection

It must be problem dependent

no common method

Prior knowledge is definitely needed:

Related Context: digging multiple cues

Need many samples: learning from samples

Useful low level and mid level will be helpful:

Get important feature will make things easy

Challenge of PR Viewpoint

Neuroscience: final solution?

Deep Learning-A light of hope?

What is course objective?

Introducing the State-Of-The-Art research

For most of us, build your work on the

shoulders of giants/genius

Making the students capable of implementing the

algorithms introduced by newly published papers.

The skill of writing scientific papers(Publications),

and presenting your work (PPT)

The skill and capability to do research as a team

How to score your performance?

Show up at class (30%)

Sign each time

Paper Reading Report (30%)

PPT

Final project: (40%)

Review paper from selected topic

Tips: how to learn?

Technology

Engineering

Methodology

Concept & Model

Algorithm & Framework

System & Application

Science

Philosophy

Tips

Teaching materials will include

PPT, and recommend papers

Videos/PPTs from internet

Highly recommendation

Reading more after class, what you get from

class if just beginning

Writing report when reading

Programming is key for your success

Just because we are engineers…

Tips

How to treat Publication?

Very important for: summer interns, job

hunting, get well known, etc.

But！

“An article about computational

science in a scientific publication is

not the scholarship itself, it is merely

advertising of the scholarship. The

actual scholarship is the complete

software environment and the

complete set of instructions which

generated the figures.”

Tips

• Resources

▪ Journals：TPAMI, IJCV; NC, IJML; TIP, TMM; PR…

▪ Conf.：ICCV, CVPR, ECCV; NIPS, ICML; ACCV,

MM, ICIP, ICPR, CVGIP; SIGGRAPH...

• Homepages ▪ Google search：computer vision

• Computer vision homepage(CMU) • Cvonline (online turtorial) • Computer vision Research Groups

(CMU/MIT/BERKELY/Standford/UIUC…)

• Big guys ▪ Just search according to their name and affiliate

Tips

Open Courses/Lectures -http://www.coursera.org/course/

-http://videolectures.net

-…

Tips

Ongoing courses http://vision.stanford.edu/teaching/cs231n/index.html

Tips

Programming Open sources

Collect…

Assemble…

Integrate…

More information cab be found at:

http://medialab.sjtu.edu.cn/teaching/CV/CV.htm