Download - Nhom02 Chuong05 SVM BaoCao
-
1
TRNG I HC KHOA HC T NHIN
CAO HC CNG NGH THNG TIN KHA 22
MY HC
BO CO:
SUPPORT VECTOR MACHINE GVHD:
TS. Trn Thi Sn
HVTH:
12 11 027 Nguyn Thanh Hng
12 11 011 Bi Th Danh
12 11 075 V Quang Trng
12 11 069 Bnh Tr Thnh
12 11 024 Phm Minh Hong
- TP. HCM 2/2013 -
-
2
Mc lc
1 Gii thiu ............................................................................................................... 3
2 Support Vector Classifier - SVC ............................................................................ 3
2.1 Phn lp nh phn vi SVC ............................................................................. 3
2.2 Vn d liu khng phn tch tuyn tnh ..................................................... 6
2.2.1 Soft margin ............................................................................................... 7
2.2.2 Th thut Kernel ....................................................................................... 9
2.3 Cc phng php hun luyn SVC ............................................................... 13
2.3.1 Phn on (Chunking) ............................................................................ 13
2.3.2 Phng php ca Osuna ......................................................................... 14
2.3.3 SMO Sequential minimal optimization ............................................... 14
2.4 Cc hng pht trin ..................................................................................... 15
2.4.1 Hiu qu tnh ton .................................................................................. 15
2.4.2 La chn kernel ...................................................................................... 15
2.4.3 Phn tch tng qut ................................................................................. 16
2.4.4 Hc SVM c cu trc ............................................................................. 17
3 Support Vector Regressor SVR ........................................................................ 18
3.1 Gii thiu bi ton hi quy ............................................................................ 18
3.2 Hi quy vi SVR ........................................................................................... 21
3.3 Support Vector Regression ..................................................................... 27
4 Ph lc .................................................................................................................. 30
5 Ti liu tham kho................................................................................................ 30
-
3
1 GII THIU
Support Vector Machine (SVM) l phng php mnh v chnh xc nht trong s cc
thut ton ni bt lnh vc khai thc d liu. SVM bao gm hai ni dung chnh l:
support vector classifier (SVC), b phn lp da theo vector h tr, v support vector
regressor (SVR), b hi quy da theo vector h tr. c pht trin u tin bi Vapnik
vo nhng nm 1990, SVM c nn tng l thuyt c xy dng trn nn mng l thuyt
xc sut thng k. N yu cu s lng mu hun luyn khng nhiu v thng khng
nhy cm vi s chiu ca d liu. Trong nhng thp nin qua, SVM pht trin nhanh
chng c v l thuyt ln thc nghim.
Trong cc phn tip theo sau y, nhm s trnh by chi tit v Support Vector
Classifier v Support Vector Regressor.
2 SUPPORT VECTOR CLASSIFIER - SVC
2.1 PHN LP NH PHN VI SVC
Xt mt v d ca bi ton phn lp nh hnh v, ta phi tm mt ng thng
sao cho bn tri n ton l cc im , bn phi n ton l cc im xanh. Bi ton m
dng ng thng phn chia ny c gi l phn lp tuyn tnh (linear
classification).
Hnh 1: Minh ha phn lp tuyn tnh
Hm tuyn tnh phn bit hai lp nh sau:
( ) (1)
Trong :
l vector trng s hay vector chun ca siu phng phn cch, T l k hiu chuyn v.
l lch
Lu rng, nu khng gian l 2 chiu th ng phn cch l ng thng, nhng
trong khng gian a chiu th gi l siu phng.
-
4
Tp d liu u vo gm N mu input vector {x1, x2,...,xn}, vi cc gi tr nhn
tng ng l {t1,,tn} trong * +. Gi s tp d liu c th phn tch tuyn tnh hon ton, ngha l cc mu u c phn ng lp bi ng phn cch. Khi ,
gi tr tham s w v b theo (1) lun tn ti v tha ( ) cho nhng im c nhn v ( ) cho nhng im c , v th m ( ) cho mi im d liu hun luyn.
tm ng phn cch, SVC thng qua khi nim gi l l, ng bin (margin).
L l khong cch nh nht gia im d liu gn nht n mt im bt k trn ng
phn cch, xem hnh HNH 2.
Hnh 2: Minh ha margin (l)
Theo SVC, ng phn cch tt nht l ng c margin ln nht. iu ny c ngha
l tn ti rt nhiu ng phn cch xoay theo cc phng khc nhau, v khi phng
php s chn ra ng phn cch m c margin ln nht.
Hnh 3: Minh ha ng phn cch ti u
Khong cch t im d liu n ng phn cch nh sau:
| ( )|
(2)
Khng mt tnh tng qut, Vapnik xp x bi ton thnh:
-
5
{
(3)
Cc im d liu lm cho du = xy ra trong biu thc trn c gi l cc vector
h tr (support vector). Chng cng chnh l cc im d liu gn ng phn cch ti
u nht. Theo , khong cch t cc support vector n mt phn cch ti u s l:
( )
{
(4)
Khi , l phn cch gia hai lp l
(5)
tm c ng phn cch ti u, SVC c gng cc i theo w v b:
( )
(6)
iu ny tng ng vi:
( )
(7)
y c xem l bi ton c s (primal problem). gii quyt bi ton ny, ngi
ta dng phng php nhn t Lagrange (Lagrange multiplier). Hm Lagrange tng ng
cho (7) l:
( )
, (
) -
(8)
Ly o hm L theo hai bin w v b, ta c
{
( )
( )
(9)
Suy ra:
-
6
{
(10)
Th vo hm Lagrange, thu c:
( )
(11)
iu kin b sung Karush-Kuhn-Tucker (KKT) l:
, ( ) - (12)
Theo , ch nhng support vector (xi, yi) mi c i tng ng khc khng, nhng
im d liu cn li c i bng 0. Support vector chnh l ci m ta quan tm trong qu
trnh hun luyn ca SVM. Vic phn lp cho mt im d liu mi s ch ph thuc vo
cc support vector.
Bi ton kp (dual problem) (11) l lp bi ton ti u quy hoch bc 2 li (convex
quadratic programming optimization) tiu biu. Trong nhiu trng hp, n c th t ti
u ton cc khi p dng cc thut ton ti u ph hp, v d SMO (sequential minimal
optimization). Chi tit SMO s c trnh by phn sau.
Sau khi tm c cc nhn t Lagrange ti u i th chng ta c th tnh w v b ti u
theo cng thc bn di. Lu vi b th ch cn ly mt vector h tr dng (tc t = +1)
l c, nhng m bo tnh n nh ca b, chng ta c th tnh bng cch ly gi tr
trung bnh da trn cc support vector.
(13)
2.2 VN D LIU KHNG PHN TCH TUYN TNH
Vic yu cu d liu phi phn tch tuyn tnh hon ton l nghim ngt v khng
ph hp vi cc bi ton thc t, c bit l cc trng hp phn lp phi tuyn phc tp.
Trong khi , cc mu khng phn tch tuyn tnh hon ton dn n vic khng th gii
quyt cc bi ton ti u tm w v b tng ng. gii quyt vn ny, c hai cch
tip cn chnh:
-
7
Soft-margin Th thut Kernel.
2.2.1 SOFT MARGIN
V nhiu l do, do bn cht hoc do sai st trong qu trnh thu thp d liu, tn ti mt
s im thuc lp ny ln ln vo lp kia, iu ny s lm ph v s phn tch tuyn
tnh. Nu ta c tnh phn tch hon ton s lm cho m hnh d on qu khp. chng
li s qu khp, ngi ta m rng SVC n chp nhn mt vi im phn lp sai. K
thut ny gi l soft margin.
Hnh 4: Minh ha trng hp d liu nhiu;
lm iu ny, mt bin (gi l slack variable) i c thm vo biu thc cn ti
u nhm cho php m hnh phn lp thc hin phn lp sai mc chp nhn c:
( )
(14)
Tham s C dng cn bng gia phc tp tnh ton v s lng im khng th
phn tch. N c gi l tham s chun ha (regularization parameter). Gi tr C c
th c lng nh thc nghim hoc phn tch d liu.
Cc bin i c thm cho tng im d liu, cho bit s sai lch khi phn lp vi
thc t. C th:
cho nhng im nm trn l hoc pha trong ca l.
( ) cho nhng im cn li.
-
8
Theo , nhng im nm trn ng phn cch ( ) s c v nhng im phn lp sai s c . Theo Lagrange ta vit li:
( )
* ( ) +
(15)
Trong * + v * + l cc nhn t Lagrange.
Cc iu kin KKT cn tha l:
( )
( ( ) )
vi i = 1,,n
Ly o hm (15) theo w, b v { }:
Th tt c vo (15) ta c:
( )
(16)
Suy ra cng thc cho bi ton kp vi soft margin nh sau:
-
9
( )
(17)
iu kin KKT tng ng l:
, ( ) -
(18)
Nh trc , tp cc im c khng c ng gp g cho vic d on im d liu mi. Nhng im cn li to thnh cc support vector. Nhng im c v theo (18) tha:
( )
(19)
Nu th , suy ra . l nhng im nm trn l.
Nhng im c th , c th l nhng im phn lp ng nm gia l v ng phn cch nu hoc c th l phn lp sai nu
xc nh tham s b, chng ta dng nhng support vector m , tng ng vi . Ln na, m bo tnh n nh ca b ta nn tnh theo trung bnh.
2.2.2 TH THUT KERNEL
Theo nh l Cover v s phn tch mu, mt bi ton phn lp mu phc tp m
chuyn sang khng gian c s chiu cao bng php chuyn i phi tuyn th c kh nng
phn tch tuyn tnh cao hn khi chuyn sang khng gian c s chiu thp. Nh vy, ta
c th gii quyt vn khng phn tch tuyn tnh bng cch thc hin php chuyn i
phi tuyn d liu u vo sang khng gian c s chiu cao hn (thm ch l v cng).
Tuy nhin, s chuyn i nh vy s lm tng phc tp tnh ton v xc nh s chiu
ph hp l vn khng d dng. Th thut kernel c a ra gii quyt vn ny.
Mu cht ca th thut kernel l xc nh hm kernel ph hp tnh c tch v hng
ca mu d liu sau khi thc hin chuyn i m khng cn phi quan tm s chiu l
bao nhiu.
-
10
Hnh 5. Bin i khng gian d liu sang khng gian c trng.
Gi : X H l php bin i phi tuyn t khng gian u vo m chiu X vo khng
gian c trng H m cc mu c th phn tch tuyn tnh. Khi ng phn cch
ti u c nh ngha nh sau:
( )
(20)
Khng mt tnh tng qut, gn b = 0 v cng thc n gin thnh:
( )
(21)
Lm tng t nh phn trn th vector trng s ti u l trong khng gian c trng mi s l:
( )
(22)
Theo siu mt phng ti u trong khng gian c trng mi l:
( ) ( )
(23)
Trong ( ) ( ) l tch v hng ca hai vector (x) v (xi). T y, chng ta c th p dng hm kernel tch v hng.
nh ngha (Kernel tch v hng): Kernel l mt hm K(x, x), sao cho vi mi x,
x thuc X, X l tp con ca khng gian m chiu Rm tha iu kin sau:
( ) ( ) ( ) (24) Trong l php bin i khng gian u vo X sang khng gian c trng H.
Theo , hm xc nh siu phng ti u s thnh:
-
11
( )
(25)
u im ca kernel l c th xy dng ng phn cch ti u m khng phi quan
tm chi tit n dng hm ca php bin i . Nh vy, hm kernel lm cho thut ton
khng nhy cm vi s chiu, trnh c cc tnh ton phc tp khi tnh tch v hng
cng nh thit k b phn lp. nh l Mercer ch ra cc thuc tnh m hm kernel
K(x,x) cn phi c.
nh l Mercer: Cho K(x, x) l mt hm i xng lin tc c nh ngha trn min
gi tr ng v tng t cho x. Hm K(x, x) l kernel nu c th m rng theo dy nh sau:
( ) ( ) ( )
(26)
Trong , i l h s dng vi mi i. s m rng ni trn l hp l v hi t th
cn iu kin sau:
( ) ( ) ( )
(27)
ng vi mi (.) m
( )
(28)
Din gii nh l Mercer: c im hu ch nht cn ch khi xy dng kernel l bt
k mt tp con hu hn ngu nhin trong khng gian u vo X th ma trn xy dng
tng ng vi hm kernel K(x, x) l ma trn i xng v bn xc nh, cn gi l ma
trn Gram.
( ( ))
K l ma trn bn xc nh nu tt c cc tr ring ca n l khng m. Mt s bi ton,
rng buc iu kin ma trn l xc nh dng, ngha l cc tr ring phi ln hn 0
m bo rng bi ton s hi t v gii php l duy nht.
Theo rng buc ni trn th vic chn hm kernel vn kh phng khong, Cc loi
hm kernel c th s dng nh:
Tuyn tnh:
-
12
( ) (29)
a thc:
( ) ( ) (30)
Gaussian (radial basis function):
( ) (
)
(31)
Vic iu chnh cc tham s trong hm cho ph hp vi bi ton l mt cng vic kh
vt v. Hm a thc v radial basis function l hai hm c s dng ph bin nht. Tuy
nhin, chn c hm ph hp chng ta nn da theo thng tin m ta mun rt trch ra
t d liu. Chng hn,
Hm a thc cho php m hnh ha s lin kt ca cc c trng (tng theo bc ca a thc)
Radial basis function cho php m hnh ha nhng c trng c th phn tch d liu phn b theo hnh trn (hoc siu cu)
Hin nay, c mt s nghin cu h tr hm kernel mt cch t ng, chng hn nh
nghin cu ca Tom Howley v Michael MaddenError! Reference source not found..
So snh th thut kernel vi soft margin c th thy hai hng tip cn theo nhng
cch khc hn nhau.
Margin mm Th thut kernel
Thm rng buc ni lng v cho php li
Khng hiu qu khi m d liu khng th phn tch tuyn tnh nhiu, chng hn phn lp phi tuyn, v li phn lp nhm qu cao
Chuyn d liu sang khng gian c chiu cao hn mt cch khng tng mnh nh hm kernel lm cho bi ton c th phn tch tuyn tnh.
Khng th m bo rng bi ton c th phn tch tuyn tnh tuyt i trn tt c d liu, c bit l nhng bi ton phc tp.
Da trn u v khuyt im ca hai phng php, chng ta c th kt hp tng
chnh xc. Khi , bi ton ti u ha c rng buc s c dng nh sau:
( )
( )
(32)
-
13
Theo phng php nhn t Lagrange, ta thu c hm phn lp ti u nh sau:
( ) ( )
(33)
Trong , b* c th tnh bi cc vector h tr dng,
( )
(34)
2.3 CC PHNG PHP HUN LUYN SVC
Trng tm hun luyn SVC l hun luyn cc vector h tr t d liu u vo. Nh
cp trn, cc nhn t Lagrange cho php chng ta xc nh c cc vector h tr.
Tm cc nhn t Lagrange ti u cho tng im d liu thuc dng bi ton quy hoch
bc 2 (quadratic programming), vit tt l QP. Kch thc d liu hun luyn rt ln, lm
cho bi ton QP trong SVM tr nn kh gii vi cc k thut gii QP chun. C th l ma
trn lu tr d liu cn thit chy thut ton c kch thc bng bnh phng s lng
mu, vi 4000 mu hun luyn th ma trn b nh 128MB l khng .
T kh khn , nhiu tc gi xut cc phng php hun luyn SVC vi kch
thc d liu ln. Trong phn ny, chng ti cp 3 phng php tiu biu theo th t
thi gian ra i.
2.3.1 PHN ON (CHUNKING)
Phng php phn on c xut bi Vapnik, da trn thc t l kt qu thu c
ca bi ton QP trn ma trn ban u v trn ma trn b i cc dng v ct ca d liu
c nhn t Lagrange bng 0 l nh nhau.
Theo , tc gi chia bi ton QP ban u thnh cc bi ton QP nh hn, vi mc
tiu l xc nh tt c cc nhn t Lagrange khc khng v loi b cc nhn t Lagrange
bng 0. mi bc lp, phng php gii bi ton QP gm cc mu sau:
Cc mu tng ng vi nhn t Lagrange khc 0 bc lp trc.
M mu t nht vi phm iu kin KKT, vi M l gi tr cho trc. Nu c t
hn M th ly ht cc mu vi phm.
-
14
Mi bi ton con QP c khi to theo kt qu ca bi ton con trc . bc
cui cng, tt c cc nhn t Lagrange khc 0 s c xc nh v bi ton c gii.
Phng php phn on gim kch thc ca bi ton xung cn xp x bnh phng
cc mu hun luyn c nhn t Lagrange khc 0. Tuy nhin, kch thc ma trn vn cn
ln v khng th lu trong b nh.
2.3.2 PHNG PHP CA OSUNA
Nm 1997, Osuna v ng tc gi chng minh thnh cng nh l lin quan n
cc thut ton QP. nh l ny ni rng, bi ton QP ln c th chia nh thnh tp cc bi
ton QP nh hn. Ch khi t nht mt mu vi phm iu kin KKT c thm vo tp
mu cho bi ton con trc , mi bc s gim hm mc tiu v duy tr kh nng tha
mn tt c cc rng buc. Do , mt dy cc bi ton con QP lun lun thm t nht mt
mu vi phm s m bo hi t. Thut ton phn on trn cng tun th cc iu kin
ca nh l nn n hi t
Osuna v cc ng s xut gi cho kch thc ca ma trn lun c nh trong tt
c cc bi ton con. iu ny ng ngha vi vic s lng mu thm vo v xa i l
bng nhau. S dng ma trn c kch thc c nh cho php hun luyn trn tp d liu
ln. Trn l thuyt, Osuna ngh thm vo mt mu v ly ra mt mu mi bc. Tuy
nhin, trong thc t tng tnh hiu qu cc nh nghin cu thao tc trn nhiu mu,
vic chn mu da trn heuristic.
Tuy nhin, cc nghin cu vn s dng phng php s hc gii cc bi ton QP
con. Nhng phng php s hc th rt kh thu c kt qu ng do sai s.
2.3.3 SMO SEQUENTIAL MINIMAL OPTIMIZATION
SMO l phng php n gin c th gii quyt nhanh bi ton QP trong SVM m
khng yu cu b nh lu tr thm cng nh khng s dng phng php s hc mi
bc.
SMO cng phn r thnh cc bi ton nh hn, nhng khng ging cc phng php
trc, n chn bi ton ti u nh nht c th mi bc. C th hn, l bi ton ti u
ch lin quan n hai nhn t Lagrange. mi bc, SMO chn ra hai nhn t Lagrange
ti u ha ng thi, v cp nht SVC phn nh cc thay i.
Mt u im ca SMO l v ch lm vic vi hai nhn t Lagrange nn c th s dng
phng php gii tch, trnh c vn sai s ca phng php s hc. Ngoi ra, v
khng yu cu thm b nh nn SMO ph hp vi mi bi ton hun luyn SVC ln trn
my tnh c nhn hoc workstation.
-
15
SMO gm 2 thnh phn: thut ton phn tch tm gi tr ca nhn t Lagrange v
mt heuristic chn nhn t ti u mi bc.
Hnh 6. Cc thnh phn ca SMO.
Chi tit cc thnh phn c th tham kho trong ti liu Error! Reference source not
found..
2.4 CC HNG PHT TRIN
Trong thp k trc, SVM c pht trin nhanh c v l thuyt ln thc hnh.
Hin nay, nhiu nghin cu vn tip tc v vn ny. Trong phn ny, chng ta s lit
k mt s hng nghin cu chnh ang c thc hin v nhiu vn nghin cu m.
2.4.1 HIU QU TNH TON
Mt trong nhng hn ch ca SVM trc y l phc tp tnh ton cao trong giai
on hun luyn, dn ti khng th p dng thut ton i vi CSDL ln. Mt cch tip
cn mi l chia vn ti u ha thnh chui cc vn nh hn, trong mi vn
ch lin quan ti mt cp gi tr c chn k lng vic ti u ha c th c thc
hin mt cch hiu qu. Tin trnh lp li cho ti khi cc vn ti u ha c phn tch
v gii quyt thnh cng.
Mt cch tip cn gn y l coi vn hc SVM nh l vic tm xp x nh nht ca
tp ng. Cc mu ny mt khi a vo khng gian N chiu s biu din mt tp im
c s dng xy dng mt xp x nh nht. Gii quyt cc vn hc SVM trn cc
tp ct li ny c th to ra cc li gii xp x tt vi tc tt. V d nh my vector li
[18] c th hc SVM cho hng triu d liu trong mt giy.
2.4.2 LA CHN KERNEL
Trong SVM kernel, vic la chn hm kernel phi p ng nh l Mercer
(http://en.wikipedia.org/wiki/Mercer's_theorem). Bi vy, cc hm kernel ph bin bao
-
16
gm 3 loi: sigmod, a thc v radial basis function. iu ny s gii hn kh nng p
dng kernel trick. Gn y, Pekalska v cng s cung cp mt cch xem xt mi l
thit k mt hm kernel da trn i snh quan h xp x tng qut. Hm kernel mi
khng cn phi p ng cc iu kin ca Mercer cng khng b gii hn ch trong khng
gian c trng v cho kt qu phn lp tt hn kt qu thc nghim kernel Mercer thng
thng. Tuy nhin, c s l thuyt ca kernel tng qut ca Pekalska cn thc hin cc
nghin cu xa hn trong tng lai.
Hn na, mt cch tip cn ph bin khc l hc nhiu kernel. Thng qua vic kt ni
cc kernel, h thng c th cho kt qu tt hn. iu ny cng tng t vic s dng
ng b cc kernel. Bng vic thit lp cc hm mc tiu ring, s la chn cc tham s
kernel tt hn c th c thc hin cho php ha trn kernel.
2.4.3 PHN TCH TNG QUT
Chng ta quen vi vic s dng kch thc VC c lng gii hn li tng qut
ca cc my kernel. Tuy nhin, gii hn i hi phc tp phi c nh, khng ph
thuc vo d liu hun luyn v dn ti khng th thc hin phn lp hiu qu. gii
quyt vn ny, phc tp ca Rademacher c gii thiu nh mt gii php thay
th nh gi phc tp ca b phn lp thay cho kch thc VC truyn thng, kh
nng phn lp ph hp vi d liu ngu nhin. N c nh ngha nh sau:
nh ngha 5.1 ( phc tp Rademacher). i vi mu S = {x1, x2, , xn} c
to ra bi phn b D trn tp d liu X v lp hm gi tr thc F vi min X, phc tp
Rademacher thc nghim ca F l gi tr ngu nhin
1 2
1
2 ( ) [ sup | ( ) ||x , x , ... , x ]n
n i i nf F i
R F E f xn
Trong , = {1, 2, , n} l cc gi tr {1} gi tr ca bin ngu nhin
(Rademacher). phc tp Rademacher ca F l:
1
2( ) [R (F)]=E [ sup | ( )|]n
n S n S i if F i
R F E f xn
Phn sup bn trong biu thc tnh k vng l o tng quan tt nht c th tm
c gia hm cc lp v nhn ngu nhin. Hn na, trong cc my kernel chng ta c
th t c mt gii hn trn cho phc tp Rademacher:
nh l 5.2 Phn tch phc tp. Nu k: X x X -> R l mt kernel v S = {x1, x2,
, xn} l mt mu cc im t X, th phc tp Rademacher thc nghim ca b phn
lp FB tha mn:
-
17
1
2 2 ( ) ( , ) ( )n
n B i i
i
B BR F k x x tr K
n n
Trong B l gii hn ca cc trng s w trong b phn lp.
Mt iu ng ch l phc tp Rademacher ch bao gm ma trn kernel tng
ng c xc nh thng qua d liu hun luyn c th. N kh thi hn so vi s dng
kch thc VC truyn thng ti u phc tp b phn lp.
2.4.4 HC SVM C CU TRC
Gn y, mt s thut ton quan tm ti thng tin cu trc c pht trin nhiu hn
SVM truyn thng. H cung cp mt quan im mi khi thit k mt b phn lp, ni m
mt b phn lp c th nhy cm vi cu trc phn b ca d liu. Cc thut ton ny
thng c 2 cch tip cn chnh. Cch tip cn th nht l hc nhiu fold. Gi s d liu
c chia thnh cc fold nh trong khng gian u vo v thut ton in hnh nht l
my h tr vector Laplacian (LapSVM). u tin, chng ta c th xy dng LapSVM
thng qua th Laplacian trong mi lp. Sau , chng ta a cu trc nhiu fold ca
d liu vi cc ma trn Laplacian tng ng vo cc framework truyn thng ca SVM
nh ton hng b sung.
Cch tip cn th hai l bng cch tm hiu cc thut ton gom cm bng cch gi s
rng d liu s cha mt s cm da vo thng tin phn b ban u. Gi s ny dng
nh tng qut hn so vi trng hp hc nhiu fold, iu ny dn ti xut hin mt s
margin machine ln ph bin. Cch tip cn gn y c bit ti l margin machine ln
c cu trc (SLMM). SLMM p dng cc k thut gom cm nm bt thng tin cu
trc trong cc lp. Sau , n s dng khong cch Mahalanobis nh mt o khong
cch t cc mu ti cc mt siu phng quyt nh thay th cho khong cc Euclidean
truyn thng, gii thiu thng tin cu trc lin quan theo mt quy nh no . Mt s
my margin ln ni ting nh SVM MPM (minimax probability machine), M4 (maxi-min
margin machine) c xem nh l cc trng hp c bit ca SLMM. Thc nghim cho
thy SLMM cho kt qu phn lp tt hn. Tuy nhin, khi vn ti u ha ca SLMM
c thnh lp cng thc nh SOCP (second order cone programming) hn l QP trong
SVM th SLMM c ch ph tnh ton cao hn nhiu so vi SVM truyn thng khi hun
luyn. Hn na, n khng tng qut i vi cc quy m d liu v trong trng hp
nhiu lp. Do , my h tr vector c cu trc mi (SSVM) c pht trin khm
ph framework c in ca SVM hn nhiu so vi cc hn ch trong SLMM. Vn ti
u ha tng ng c th vn c gii quyt bi QP nh trong SVM v gi cho li gii
khng ch tha tht m cn kh nng m rng. Hn na, SSVM c ch ra ni chung
l tt hn v mt l thuyt v thc nghim hn so vi SVM v SLMM.
-
18
3 SUPPORT VECTOR REGRESSOR SVR
3.1 GII THIU BI TON HI QUY
Gi s rng chng ta a ra mt tp hun luyn bao gm N = 10 phn t (x1, , xn),
quan st gi tr u vo ca x cng vi cc gi tr mc tiu tng ng (t1, , tn).
ng mu xanh l cy c to ra t hm f(x) = sin(2x) v 10 im mu xanh
(blue) c to ra t hm sin cng thm 1 sai s nht nh (iu ny nhm m phng
trong thc t khi thu thp d liu s c mt sai s nht nh). Mc tiu l ta cn tm 1
ng hi quy gn nh tng t vi ng sin t tp d liu m khng h bit g v
ng mu xanh l cy.
Nhn xt: iu ny bn cht l mt vn kh khn khi chng ta phi khi qut ha
mt d liu hu hn. Hn na d liu quan st b hng vi nhiu. V vy cho mt gi tr x
khng chc chn l thu c gi tr t chnh xc.
Chng ta kho st phng trnh ng cong bc M nh sau:
Trong : M l bc ca a thc, xj l th hin ca x, wj l h s chung biu th bng
cc vector. Lu , mc d hm a thc y(x,w) l mt hm phi tuyn theo x v tuyn tnh
theo w.
Cc gi tr ca cc h s s c xc nh bng cch a vo cc a thc trn tp d
liu hun luyn. iu ny c th c thc hin bng cch gim thiu chc nng li gia
cc hm y(x,w). Mt cch n gin chn la hm li c s dng rng ri l hm li
c cho bi tng ca bnh phng sai gia cc tin on y(xn,w) cho mi im d liu
xn v tng ng gi tr mc tiu tn, v vy chng ta c hm li ti thiu:
-
19
Gi tr 1/2 c thm vo nhm thun tin cho vic tnh ton sau ny cho sau ny.
Tip theo chng ta s cp n vic la chn hm li nh th no l tt nht.
Chng ta c th gii quyt vn qu khp bng cch chn gi tr ca w hm E(w)
nh nht c th. Bi v hm li l mt hm bc 2 theo tham s w, cc dn xut ca n i
vi cc h s s l tuyn tnh trong cc phn t ca w, v v s ti thiu ca hm li c
1 gii php duy nht, th hin bi w*, c th tm thy trong cng thc cui cng. Kt qu
a thc c mang li bi hm y(x, w*).
Nhn xt: Vi M=0, chng ta nhn thy rng ng hi quy khng i. M=1, a thc
cho kt qu km khp vo cc d liu. M=3, a thc cho kt qu khp tt nht. V cui
cng vi M=9, a thc cho kt qu qu khp trong hun luyn d liu.
Cho mi ln chn M, sau chng ta c th nh gi cc gi tr cn li ca E(w*)
c mang li bi E(w) cho vic hun luyn d liu.
i khi n l thun tin hn s dng hm li ERMS (RMS=root-mean-square):
Trong phn chia bi N cho php chng ta so snh kch c cc b d liu khc nhau
mt cch bnh ng, v cn bc m bo rng ERMS c o trn cng mt t l (v trong
cng mt n v) nh l cc bin mc tiu t.
-
20
Sau khi p dng hm li ERMS ta c th training v test nh sau:
Ta nhn thy gi tr M nm trong khong 3
-
21
Tuy nhin, nu chng ta s dng qu ln th kt qu s km ph hp. Ta c bng
kim tra gi tr (w*)
Sau khi p dng hnh pht li mi vi ta c th training v test nh sau:
3.2 HI QUY VI SVR
tng c bn ca SVR l nh x khng gian u vo (m nu ta p dng trc tip
hi qui tuyn tnh th khng hiu qu) sang mt khng gian c trng nhiu chiu m
, ta c th p dng c hi qui tuyn tnh.
c im ca SVR l cho ta mt gii php tha (sparse solution); ngha l xy
dng c hm hi qui, ta khng cn phi s dng ht tt c cc im d liu trong b
-
22
hun luyn. Nhng im c ng gp vo vic xy dng hm hi qui c gi l nhng
Support Vector.
Hm hi qui cn tm c dng:
( ) ( ) (35) Trong , l vc t trng s, T l k hiu chuyn v, l hng s,
l vc t u vo, ( ) l vc t c trng, lm hm nh x t khng gian u vo sang khng gian c trng.
Nh vy, mc tiu ca vic hun luyn SVR l tm ra c w v b.
Cho tp hun luyn {(x1, t1), (x2, t2), , (xN, tN)} . Vi bi ton hi qui n
gin, tm w v b ta phi ti thiu ha hm li chun ha:
* +
(36)
Vi l hng s chun ha.
c c mt gii php tha, ta s thay hm li trn bng hm li -insensitive. c
im ca hm li ny l nu tr tuyt i ca s sai khc gia gi tr d on y(x) v gi
tr ch nh hn (vi > 0) th n coi nh li bng 0.
( ( ) ) { | ( ) |
| ( ) | (37)
Hnh 7: Minh ha hm li thng thng v hm li -insensitive
Hnh trn so snh gia hm li bc 2 thng thng (ng cong mu xanh) v hm
li -insensitive (ng gp khc mu .) Nh ta c th thy khi s sai khc gia gi tr
d on v gi tr ch nm trong [-, ] th hm li -insensitive c gi tr bng 0.
Nh vy by gi, ta phi ti thiu ha hm li chun ha sau:
( ( ) )
(38)
-
23
Vi ( ) ( ) , C l hng s chun ha ging nh nhng c nhn
vi hm li thay v vi .
cho php mt s im nm ngoi ng , ta s a thm cc bin lng (slack
variable) vo. i vi mi im d liu xn, ta cn hai bin lng v , trong
ng vi im m ( ) (nm ngoi v pha trn ng) v ng vi im m ( ) (nm ngoi v pha di ng.) Hnh di minh ha cho cc bin lng ny.
Hnh 8: Minh ha SVR vi cc bin lng
iu kin mt im ch nm trong ng l: vi yn = y(xn). Vi vic s dng cc bin lng, ta cho php cc cc im ch nm ngoi ng (ng vi
cc bin lng > 0) v nh th th iu kin by gi s l:
(39)
(40) Nh vy, ta c hm li cho SVR:
( )
(41)
Mc tiu ca ta l ti thiu ha hm li ny vi cc rng buc:
Gi y l vn ti u ha A.
C ngay hm Lagrange:
-
24
( )
( )
( )
( )
(42)
Vi ( ) ( )
Ly o hm theo w, b, , v cho bng 0, ta c:
( ) ( )
(43)
( )
(44)
(45)
(46)
Dng 4 kt qu ny th vo (42), ta s loi b c w, b, , , , :
( )
( )( ) ( )
( )
( )
(47)
Vi k l hm nhn: ( ) ( ) ( ). Bt k mt hm no tha iu kin Mercer th u c th c dng lm hm nhn. Hm nhn c s dng ph bin nht l
hm Gaussian:
( ) ( ) (48)
Nh vy, ta chuyn t vn ti u ha A sang vn ti u ha tng ng B:
Ti a ha (47) vi cc rng buc:
-
25
( )
Li ch chnh ca vic chuyn i t vn ti u ha A sang vn ti u ha B l
vn ti u ha B c s dng hm nhn. iu ny s gip cho vic tnh ton trong
khng gian nhiu chiu tr nn rt hiu qu.
Th (43) vo (35), ta c s d on cho mt mu mi x:
( ) ( )
( ) (49)
Theo iu kin KKT (KarushKuhnTucker), c ngay:
( ) (50)
( ) (51) ( ) (52) ( ) (53)
T y, ta c th rt c nhng thng tin quan trng nh sau:
Nu th : im nm bin trn ca ng ( ) hoc nm ngoi v pha trn ca ng ( )
Nu th : im nm bin di ca ng ( )
hoc nm ngoi v pha di ca ng ( )
v khng th cng dng v nu vy th ta c: v
, cng li ta s thy ngay v tri lun dng, trong khi v phi bng 0: v l!
Nhng im Support Vector l nhng im ng gp vo hm d on (49), ngha
l nhng im c hoc : nhng im nm trn bin ng hoc nm ngoi ng.
Nhng im nm trong ng s c v do khng ng gp g vo qu trnh d on.
Thy ngay d tnh c b bng cch xt mt im xn c . T (52) ta c . T (50) ta c . Kt hp vi (49) c ngay:
( )
( ) (54)
Ta cng s c kt qu tng t nu xt im c .
vng chc hn, ta nn ly trung bnh ca tt c cc gi tr ca b li.
-
26
Vi SVR s dng hm li -insensitive v hm nhn Gaussian ta c 3 tham s cn
phi xc nh: h s C, tham s ca hm nhn Gaussian v rng ca ng .
C 3 tham s ny u nh hng n chnh xc d on ca m hnh v cn phi
chn la k cng (do ngi dng truyn vo trc khi hun luyn, c th dng cc thut
ton nh GA, GridSearch tm cc tham s ny).
Nu C qu ln th s u tin vo phn li hun luyn, dn n m hnh phc
tp, d b qu khp. Cn nu C qu nh th li u tin vo phn phc tp m
hnh, dn n m hnh qu n gin, gim chnh xc d on.
ngha ca cng tng t C. Nu qu ln th c t vect h tr, lm cho m
hnh qu n gin. Ngc li, nu qu nh th c nhiu vect h tr, dn n m
hnh phc tp, d b qu khp.
Tham s phn nh mi tng quan gia cc vc t h tr nn cng nh hng n chnh xc d on ca m hnh.
Mt vi nhn xt gia SVC v SVR:
SVC c l c nh bng 1, vi mc ch tch lp, tc l y cc im d liu ra 2
pha ng phn cch, iu ny th hin r phn iu kin ca (7).
SVR c l do ngi dng truyn vo, vi mc ch hi quy, tc l ht cc im d
liu vo trong ng , iu ny th hin r phn iu kin ca (41).
Vy v sao vi SVC l bng 1, nhng SVR phi la chn ? Do mc tiu ca SVC
l tm ng phn cch nhng vi tiu ch phi c l ln nht, ta s thy c rng
khi ta c ng phn cch c th, th vic xc nh l l d dng. Do l l im
gn nht n ng phn cch, im ny l xc nh c khi c ng phn cch
c th nn n gin ta cho l bng 1 (iu ny lm c do ch cn scale w v b
trong cng thc khong cch m khng nh hng tnh cht vn ).
Nhng vi SVR l l cha c xc nh bi mt im c bit no c, v rng
ca l c nh hng n cht lng ca hm hi qui nn cn xem nh tham s
truyn vo.
Khi dng hm Kernel tng s chiu c phi cng ln cng tt? iu ny l khng
m bo, c th ta c th xt v d mc 3.1, khi m qu ln d dn n qu khp.
Do thng thng khi tng s chiu d liu ta khng nn tng thm qu s
chiu ban u.
-
27
3.3 SUPPORT VECTOR REGRESSION
Chnh sa t -SVR vi:
c xc nh trong qu trnh tnh ton thay v chn t u.
v khng m.
i vi -SVR ta cn cc tiu ha hm.
v-SVR b sung thm s hng v, vo hm cc tiu ha
Lp phng trnh Lagrange:
Tnh o hm tng ng vi cc bin ta c:
Gii h 4 phng trnh trn ta s dn n bi ton ti u ha hm
Vi cc rng buc i km l
-
28
v cn trn ca t l li (fraction of errors)
v cn di ca t l SV(fraction of SVs)
v xp x bng 2 t l trn
v-SVR i tm li gii vi cc bin , w,b trong khi -SVR i tm li gii vi cho
trc v cc bin l w, b.
S dng SVR th nhng thay i nh cc b gi tr ch (target value) ca nhng
im nm ngoi ng ng khng lm nh hng n kt qu xy dng ng hi quy.
xc nh c khng ph thuc vo s mu d liu
So snh v nhn xt:
-
29
Vi 2 mc nhiu khc nhau (0 v 1) th phng php v-SVR lun hiu chnh c
( 0 v 1.19 ) tt v ph hp vi tp d liu.
v SVR : gim khi v tng, v >1 th = 0. Do nn chn v: 0 v 1
Nu v < 1 th thng > 0, vn c th bng 0 nu d liu khng nhiu.
Chn v khc nhau ch lm s ln nhy gi tr ca khc nhau, cui khng i.
v SVR : tng khi s nhiu tng
M hnh tham s trong v-SVR
c s dng trong trng hp c nhiu ph thuc vo mu d liu x
Gi l tp 2p hm trong khng gian u vo. Vi , ta cc
tiu ha hm:
Tha rng buc:
-
30
Tnh ton tng t nh cc phn trn duy ch mt s rng buc thay i nh sau:
4 PH LC
L thuyt nhn t Lagrange
Vn cc i hm f(x) tha iu kin ( ) s c vit li di dng ti u ca hm Lagrange nh sau:
( ) ( ) ( )
Trong x v phi tha iu kin Karush-Kuhn-Tucker (KKT) nh sau:
( )
( )
Nu l cc tiu hm f(x) th hm Lagrange s l
( ) ( ) ( )
5 TI LIU THAM KHO
[1] Bernhard Schlkopf, Alexander J. Smola, Learning with Kernels: Support Vector
Machines, Regularization, Optimization, and Beyond, The MIT Press, 2001, pp.
251-277
[2] Bishop C. M., Pattern Recognition and Machine Learning, Springer, 2006
[3] John C. Platt, Sequential Minimal Optimization, A Fast Algorithm for Training
SVM, 1998
[4] Tom Howley, Michael Madden, The Genetic Kernel Support Vector Machine,
Kluwer Academic, 2005
[5] Xindong Wu, Vipin Kumar, The Top Ten Algorithms in Data Mining, Chapman
and Hall/CRC, 2009, pp. 37-59