msc in rsip jie zhang s0567061 10/02/06
DESCRIPTION
Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando Fujitsu Laboratories Ltd., Atsugi, 243-0197 Japan ). MSc in RSIP Jie Zhang s0567061 10/02/06. Background. Why Lip reading? - PowerPoint PPT PresentationTRANSCRIPT
Advanced Image ProcessingStudent Seminar:
Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Morito
shi AndoFujitsu Laboratories Ltd., Atsugi, 243-0197 Japan )
MSc in RSIP
Jie Zhang
s0567061
10/02/06
Background Why Lip reading?
Human computer interaction Automatically recognize speech contents by
processing lip movement images Potential application for disable facilities
What is the principle? Dealing with frame images sampled from video Lip Feature extraction Lip Feature recognition Various methodology
System overview
Lip colour extraction algorithm Colour characteristics of lip fe
ature Roughly mouth detection with
colour algorithm Precise mouth position detecti
on Eigen image Eigen template Matching algorithm
Eigenwaveform recognition Mouth status Dictionary data Thresholding
Training Lip images
Test Lip images
Colour extraction
Eigenvector template build
Colour extraction
Eigentemplate Detection
Eigenwaveform
recognition
Colour extraction algorithm
Colour components of face featureStrongly brightness dependence !
Colour system HSV or RGB Evenly luminance Brightness normalization
Lip region colour characteristics Three classes: lip, skin, tee
th Looking for the separation
of spectrum componentsFigure 1
Colour extraction algorithm
Figure 2: Brightness dependent face feature colour distribution of normalised RGB image.
Well distinguished in R &G!
Colour extraction algorithm
Lip Extraction with colour distribution Threshold function of R an
d G to separate skin and mouth
Label and Extract largest bright area to appropriate size as it is deem to be mouth
Remove teeth and lips with second R and G threshold function
Position determination with four key points of the lips
Figure 3
Lip edge is the combination of teeth and mouth cavity.
Eigentemplate method Aim
----- precisely detect the location of lip! Creation of eigen image
Use appropriate colour extracted image (trimmed image)
PCA + one dimension vector for a single trimmed image
Form image series matrix with x vectors
Eigentemplate method
Calculate eigenvector
Convert eigenvectors to eigenimage
Eigentemplate method
Figure 4
Eigentemplate method
Eigentemplate Trimmed test image
Recover eigenimage to template image
Similarity calculationpkp yb
ky
Searching and trimming the image, compare trimmed image will template. Largest similarity gives the exact location.
Eigentemplate method
Figure 5
Figure 6
Eigenwaveform recognition
Brief introduction to preprocessing A threshold method to detect mouth states and henc
e recognize particular mouth shape Define mouth state: i.e. open, wide open, close, tight
closed… Aspect ratio: distance b between upper and lower ed
ge points over distance a between left and right edge points
Projection vector components: project the image of mouth and lip into eigen space through time scale.
Thresholding
Eigenwaveform recognition
Dictionary matching
Figure 7
Small difference sw recognize the test utterance as the same as the template utterance
Summary & Analysis
Lip reading system Colour extraction method roughly classify lip and other face feat
ure Eigentemplate method precisely detect the location Eigenwaveform algorithm recognize the utterance
Analysis Widely used image processing technique Hard to get high precision Difficult for language different, culture different, appearance diff
erence user Potential problems of algorithm: brightness dependence; overla
p between lip & skin; similar words; uncommon mouth shape; various speak speed and so on.
Need more improvement!
That’s all!Thank you very much!
Question?