introduction to my research
DESCRIPTION
Personal Homepage http://www.hci.iis.u-tokyo.ac.jp/~kylo/TRANSCRIPT
Research Review
Kuo-Yen Lo 羅国彦
ロ コウエン
2013.4.18Sato Laboratory, University of Tokyo
Short Curriculum Vitae
• Personal Homepagehttp://www.hci.iis.u-tokyo.ac.jp/~kylo/
• Period of past yearsUniversity (2006 – 2010)Internship (2010 – 2011)Research Assistant (2012 – 2013)
• Two main research topic: 1. 2D-to-3D conversion2. Photo Aesthetics.
• Wrap up
Short Curriculum Vitae
• Language: English (TOEIC 935), JLPT(N1), Chinese(Native)
• Programming: C/C++, Matlab, Java(android)Technique: SIFT/SURF/HOG, K-means, GMM, kNN, SVM, PCA/LDA/ITML, bad-of-visual word, bilateral filter.
• Have traveled to: USA, Korea.Want to travel to: China, Thailand, Spain
• Why Japan-- historical and cultural connection-- camera companies and electronics maker here-- founded by Panasonic Scholarship
Visual Cues Low-levelMathematics
Machine LearningPsychology
Computer Vision
Overview
2006NTU
2007UPenn
2008OpenCV
2009RoboticsContest
GenderRecognition
Contest
2012ResearchAssistant
ICPR2012
ACCV_w2012
>>2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
National Taiwan University
2006NTU
2007UPenn
2008OpenCV
>>
People in Vision• Yi-Ping Hung(MM12, CVPR11, CHI11, UIST11)• Yung-Yu Chuang(CVPR12*3)• C.J. Lin (Libsvm)• H.T. Lin(ICML12, NIPS12, CVPR11
KDD12 Champion)• Winston H. Hsu(MM12*6)
33,0001928 B.C.
1.6World rank 80
Summer School in UPenn
2006NTU
2007UPenn
2008OpenCV
>>Summer Language ProgramUniversity of Pennsylvania
Join the Lab
2006NTU
2007UPenn
2008OpenCV
>>Biophotonics and Bioimaging Laboratory
Prof. Ta-Te LinOpenGLOpenCV
BorlandC++ Builder
Robotics Contest @ ASABE 2009
2009RoboticsContest
GenderRecognition
Contest
>>• Problem:
Detecting and Positioning the circular obstacle• Technique:
Graphical simulation(OpenGL), Sensor• Material:
Boe-Bot Toolkit, IR sensor, Ultrasonic sensor, Zigbee wireless communication
Robotics Contest @ ASABE 2009
2009RoboticsContest
GenderRecognition
Contest
>>
Please Visit the following link for viewing the video:
http://youtu.be/8EjON8Y2OJ0
Gender Recognition Contest
2009RoboticsContest
GenderRecognition
Contest
>>• Problem:
Recognize the gender with single face image• Technique:
Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check
Rotate the image 35 degreeto detect all possible tilt face
Gender Recognition Contest
2009RoboticsContest
GenderRecognition
Contest
>>• Problem:
Recognize the gender with single face image• Technique:
Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check
Use Eye detectorto wrap the face tountilt view.
Gender Recognition Contest
2009RoboticsContest
GenderRecognition
Contest
>>• Problem:
Recognize the gender with single face image• Technique:
Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check
Utilize Skin color model, Eye and Mouth detector to filter the false-positive result from the V-J face detector.
Gender Recognition Contest
2009RoboticsContest
GenderRecognition
Contest
>>• Problem:
Recognize the gender with single face image• Technique:
Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check
• Performance1.2 second per 480*320 image
• Result~85% face detection accuracy~75% gender recognition accuracyWin 3rd place among 20 teams (Taiwan and China). Bonus 60,0000yen.
Fish Recognition
2010ISMAB
NTUGraduate
Internship@ TV corp.
>>• Problem:
Tuna species recognition for fishery conservation and management
• Task:Detection and Classification
Bigeye
YellowfinAlbacore
3 spices are considered2011Navy
Fish Recognition>>
• Problem:Tuna species recognition for fishery conservation and management
• Task:Detection and Classification
Fish Image are capturedin certain lighting condition with measurement plate.Body part is smooth,makes it reflect light well.
72%
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Fish Recognition>>
• Problem:Tuna species recognition for fishery conservation and management
• Task:Detection and Classification
B Y AB 89 10 5Y 9 86 3A 10 9 81
84%
34% 72% 52% 58%
Confusion Matrix
(Head)(Abdomen)(Tail fin) (Tail)
Discriminate part!
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Yeh, Graduation!>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
2D-to-3D conversion>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
• Problem:Generating 3D videofrom 2D content.
• Inspiration:3D information isrecovered by depthcues
Captured View + Depth
2D-to-3D conversion>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Reality Comfort
• Accurate depth map• Correct depth order• Real-time processing
• Clear boundary• Temporal smoothness• Visual impression
How people perceive depth?>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
1. Low-level cue 2. Scene Recognition
2D-to-3D conversion>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Video frame + motion estimation[ICCE 2009]
Approaches1. Depth map by motion2. Depth map by saliency 3. Depth map by prior
information fusion
2D-to-3D conversion>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy Video frame + Saliency map [SDA 2010]
Approaches1. Depth map by motion2. Depth map by saliency 3. Depth map by prior
information fusion
Introduction to Bilateral Filter>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Bilateral Filter [Tomasi, ICCV98]:
f(x) h(x)
“Bi” lateral = Spatial term + Range term
Introduction to Bilateral Filter>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Bilateral Filter [Tomasi, ICCV98]:
f(x) h(x)
“Bi” lateral = Spatial term + Range term
Introduction to Bilateral Filter>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Bilateral Filter [Tomasi, ICCV98]:
f(x) h(x)
“Bi” lateral = Spatial term + Range term
Application of Bilateral Filter>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
“Bi” lateral = Spatial term + Range termSmooth Target Edge-Preserving Result
And this one?
2D-to-3D conversion>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Prior fusion [Siggraph 2009]1. Decide Geometric perspective2. Integrate Image and Depth map by Bilateral filter
One-year in Navy>>
2010ISMAB
NTUGraduate
Internship@ TV corp.
2011Navy
Academia Sinica
2012ResearchAssistant
ICPR2012
ACCV2012
>>Institute of Information Science(Central Research Academy)
People in Vision• Chu-Song Chen (CVPR12, CVPR11*2)• Mark H. Liao (MM12*2, MM11*2)• Y.-C. Frank Wang (ECCV12, CVPR12)• Yen-Yu Lin (CVPR13, MM12, TPAMI11)
Prof. Chen
PHOTO AESTHETICS CLASSIFICATIONPredicting the visual appealing quality of photos
good?>>
jcar@DPChallenge
good?>>
Mnet @ DPChallenge
Which one is better?>>
Voted by online photo community>>
Average: 5.088 votes
Average: 7.292 votes
Reason?>>
Reason?>>
BoundaryAlternating repetition(Texture)
Contrast Levels of scale
Roughness
Strong centers
Positive space
Local symmetries
The Void
Not-separateness
Good shape
GradientsEchoes
Simplicity and Inner Calm
Deep interlock and ambiguity
Color
Composition
HarmoniumRichness
Application>>
Image Search & Management Photo evaluation system
Embedded Camera system Media analysis
Photo Aesthetics
2012ResearchAssistant
ICPR2012
ACCV2012
>>• Problem:
Recognition the appealing quality ofphoto by computational approaches.
• Technique:Image analysis, Pattern recognition,Crowdsourcing, Psychology, Photography
• Application
ICPR 2012
2012ResearchAssistant
ICPR2012
ACCV2012
>>As a Pattern Recognition Problem…Comparison of feature 1. Edge distribution, Color histogram,
Hue, Saturation.. [Ke, CVPR06]2. SIFT + BOV [Marchesotti , ICCV11]3. Composition layout (Edge + HSV),
Color palette, contrast.. [Proposed]Result
Item Speed on PC Accuracy
CVPR06 0.2s 81%
ICCV11 4s 85%
Proposed 0.16s 84%
[ Photo aesthetics assessment with efficiency ]
Extraction of Color Information
Extract N Dominant colors
(we set N=5)K-Nearest Neighbor
(K=20)
List of Palettes
Dictionary
HQ Palettes Dictionary
LQ Palettes Dictionary
Palettes of Photo
Retrieved by Frequency
Retrieved by Kmeans(Cluster Center)
Proposed (Weighted Kmeans)
Finding the Dominant Colors
Video Demo
2012ResearchAssistant
ICPR2012
ACCV2012
>>[ Intelligent Photographing Interface with On-Device Aesthetic Quality Assessment ]
Please Visit the following link for viewing the video:
http://youtu.be/o8mKuTfO6ao
Discussion 1 Device : On-line assistive camera system
• Contextual Information (Viewing angle)
camera < human• Feedback
from analysis to advice• Human behavior
What do people take?How do people take?
• Computation
Server-based v.s. Device• Market and Needs
Discussion 2 Algorithm: photo aesthetic value assessment
• Definition of photo aesthetics
Expert v.s. Volkswagen• Labeling process
Individual bias and variance.Absolute or Relative evaluationEffect of Labeling order
• Quantify photo aesthetic
Modeling, the Personalization
Thanks for your attention!
10 ratings5.00/7 average
5 ratings4.90/7 average