chương 1 - tổng quan về search engine

Chng I : TNG QUAN V H THNG SEARCH ENGINE1. Khi nim chung v b cng c Search Engine

1.1. Khi nim v Search EngineSearch Engine l mt cng c thc hin vic tm kim v sp xp thng tin gip ngi dung tm kim thng tin cn thit trn kho d liu khng l Internet . Cc Search Engine phi p ng c hai yu cu c bn : Thng tin m cc Search Engine tm c phi tha mn yu cu ngi dung v thi gian tm kim phi mc c th chp nhn c . Ngi s dng lun mong mun cc Search Engine nhng thng tin cn thit khng tht r rng nhng li yu cu kt qu tr v cc Search Engine l cc thng tin m mnh mong mun mt cch chnh xc nhanh chng v c cu trc hp l . Do , cc Search Engine ngy cng pht trin hn, cng hon thin hn p ng cc yu cu ngi dung . Cc Search Engine c nh gi theo cc chun sau : Hiu qu trong vic nh v v phn loi cc ti liu nh dng web Phm vi tm kim Cp nht c nhng thay i trn Internet kp thi Giao din thn thin vi ngi dung Kt qu tm kim c ngha v tin dng Giao din h thng ph hp vi ngi dung truy vn

1.2. Cu trc c bn ca Search EngineTrong thc t , cc Search Engine hin nay s dng kin trc c 2 thnh phn chnh : L giao din vi ngi dung gm giao din truy vn v b x l truy vn L b tm kim d liu web gm : b trnh duyt v b to ch mcCc thnh phn ny c th c thit lp tp trung mt ni hay phn tn ring l nhiu ni v c tn gi khc nhau nhng u c hu ht cc Search Engine.Kin trc c bn ca Search Engine :

Hnh : Kin trc ca mt Search EngineTy theo thit k ca tng Search Engine m vic t chc c s d liu ca chng c phc tp khc nhau v c chia thnh cc thnh phn khc nhau nhn chung mi Search Engine u c c s d liu nh sau : CSDL v cc URL CSDL v cc ti liu trc v sau khi chuyn i CSDL v d liu c to ch mc CSDL phc v cho cc mc ch c bit nh lp lch thu thp URL , cc t khng c to ch mc cc t ng ngha . Do cc Search Engine lm vic vi Internet nn c rt nhiu ngi c th truy xut cng lc v ng thi thc hin cc cng vic khc nhau. Th nn CSDL ca cc Search Engine cng nh cc chng trnh truy vn CSDL phi c thit k sao cho c th p ng c yu cu ny . Ngi ta s dng c ch phn chia d liu thnh cc phn nh tng tc truy vn . Cc t to ch mc c phn phi vo cc kho cha d liu khc nhau v cc kho ny c nh a ch nhm gip vi truy vn nhanh hn. Trong lc Search Engine thc hin vic tm kim , cng mt thi im c th c nhiu kho cha khc nhau truy vn .

2. Cc b phn cu thnh Search Engine2.1. B tm kim d liu web

2.1.1. B thu thp thng tin Robot ( Hay Crawler ) Robot l mt chng trnh t ng duyt qua cc cu trc siu lin kt thu thp ti liu & mt cch quy n nhn v tt c ti liu c lin kt vi ti liu ny. Robot c bit n di nhiu tn gi khc nhau : spider, web wanderer hoc web worm, Nhng tn gi ny i khi gy nhm ln, nh t spider, wanderer lm ngi ta ngh rng robot t n di chuyn v t worm lm ngi ta lin tng n virus. V bn cht robot ch l mt chng trnh duyt v thu thp thng tin t cc site theo ng giao thc web. Nhng trnh duyt thng thng khng c xem l robot do thiu tnh ch ng, chng ch duyt web khi c s tc ng ca con ngi.2.1.2. B lp ch mc Index H thng lp ch mc hay cn gi l h thng phn tch v x l d liu, thc hin vic phn tch, trch chn nhng thng tin cn thit (thng l cc t n , t ghp , cm t quan trng) t nhng d liu m robot thu thp c v t chc thnh c s d liu ring c th tm kim trn mt cch nhanh chng, hiu qu. H thng ch mc l danh sch cc t kho, ch r cc t kho no xut hin trang no, a ch no.2.1.3. B tm kim thng tin Search EngineSearch engine l cm t dng ch ton b h thng bao gm b thu thp thng tin, b lp ch mc & b tm kim thng tin. Cc b ny hot ng lin tc t lc khi ng h thng, chng ph thuc ln nhau v mt d liu nhng c lp vi nhau v mt hot ng. Search engine tng tc vi user thng qua giao din web, c nhim v tip nhn & tr v nhng ti liu tho yu cu ca user. Ni nm na, tm kim t l tm kim cc trang m nhng t trong cu truy vn (query) xut hin nhiu nht, ngoi tr stopword (cc t qu thng dng nh mo t a, an, the,). Mt t cng xut hin nhiu trong mt trang th trang cng c chn tr v cho ngi dng. V mt trang cha tt c cc t trong cu truy vn th tt hn l mt trang khng cha mt hoc mt s t. Ngy nay, hu ht cc search engine u h tr chc nng tm c bn v nng cao, tm t n, t ghp, cm t, danh t ring, hay gii hn phm vi tm kim nh trn mc, tiu , on vn bn gii thiu v trang web,.. Ngoi chin lc tm chnh xc theo t kho, cc search engine cn c gng hiu ngha thc s ca cu hi thng qua nhng cu ch do ngi dng cung cp. iu ny c th hin qua chc nng sa li chnh t, tm c nhng hnh thc bin i khc nhau ca mt t. V d : search engine s tm nhng t nh speaker, speaking, spoke khi ngi dng nhp vo t speak.2.2. Giao din vi ngi s dngSearch Engine l mt cng c tm kim , nn cc Search Engine cng khng th thiu phn giao din ngi dung. Do tnh c th ca Search Engine l lm vic trn Internet v lng thng tin trn Internet a dng v phong ph nn phn giao din ca Search Engine vi ngi dung phi c thit k thch hp . Giao din vi ngi dng khng ch lm vic tt vi cc module ca Search Engine m cn phi d s dng v t hiu qu trc quan cao . Phn ln ngi s dng tm kim thng tin trn Internet thng qua cc trnh duyt nn phn giao din nhn yu cu v giao din tr kt qu v thng thit k theo khun dng file *.html .C nhiu k thut c s dng nhn cc cu truy vn t ngi s dng .Chng c chia lm hai nhm chnh : Nhm c tng tc vi ngi s dng v nhm khng tng tc. Vi nhm c tng tc : Search Engine a ra cc cu hi n gin ngi s dng tr li . Thng thng ngi s dng tr li bng cch chn mt trong cc cu tr li nh sn nh : tm kim c phn bit ch hoa, ch thng khng ? Vi nhm khng tng tc : ngi s dng ch cn a thng tin v ra lnh tm kim .C nm k thut ch yu c s dng tr kt qu tm kim cho ngi s dng l : Hin th kt qu tm thy dng hyperlink : khi ngi s dng chn mt trong cc kt qu th trnh duyt s chuyn n a ch URL . Nu dng k thut ny th thng tin v kt qu s ngho nn v kh cho ngi s dng la chn nu c nhiu kt qu tr v . Hin th kt qu tr v ni dung mt hay nhiu dng vn bn tm thy trong ti liu : do kch thc hn ch ca mn hnh nn mi mn hnh thng hin th 10 kt qu . Ngi s dng phi dng cc thanh cun c cc trang tip theo . Hin th s lng kt qu tm thy : thng tin tr v ca cc Search Engine l s lng kt qu tm thy gip ngi s dng quyt nh c nn thu hp hay m rng phm vi tm kim hay khng ? Hin th mc tng t ca ti liu tm thy so vi yu cu tm kim : cho trc quan , tng t khng nhng c th hin qua nhng con s t l phn trm m cn th hin qua cc hnh nh i km . Hin th t kha tm thy trong ti liu : ngi s dng quyt nh c phi l ti liu mnh cn khng ? K thut ny rt hu dng trong trng hp a cc t kha khng chnh xc .Mt ni dung khng th thiu c trong giao din vi ngi s dng ca cc Search Engine l mc tr gip s dng . Do mi Search Engine c cu trc khc nhau nn vic h tr ngi s dng l cn thit , gip ngi s dng dng Search Engine , bit cch nhp vo chnh xc thng tin cn tm kim , gii hn hay m rng phm vi tm kim . Vic tr gip ny cung cp , hng dn ngi s dng dng cc tnh nng nng cao ca Search Engine .

3. C ch hot ng ca Search Engine

ng vi hai thnh phn chnh ca Search Engine l b Crawler v b Indexer, c ch hot ng c chia thnh hai hot ng c bn : th nht l hot ng thu nhp v phn loi thng tin . Th hai l hot ng to ch mc thng tin . u tin ngi qun tr cung cp mt hay nhiu a ch URL cc Web Crawler tm n my ch cha cc a ch ny ly trang web tng ng . Thng tin v cc URL mi cha trong trang web c lu tr vo c s d liu URL tm ra cc trang web mi . Nhng thng tin m ngi dng c th tm kim c chuyn sang phn Indexer . Ngoi cc URL trch lc t cc trang web, Search Engine cn cho php ngi qun tr cp nht thng tin v cc URL mi b sung vo c s d liu v cc URL .

Hnh : Xc nh cc URL mi t cc trang web c tng nng sut truy cp , cc Search Engine c th c nhiu Crawler hot ng theo c ch a lung. Nh k thut ny m nhiu a ch , nhiu site khc nhau c th truy cp ng thi theo chiu su . C ch truy cp theo chiu su : Web Crawler xc nh a ch ca mt site v truy cp ht cc trang ca site ny cho n khi khng th truy cp c na , sau mi sang site khc. C ch ny thng cho php cp nht v nhng lnh vc ca site m n truy cp. Tc truy cp ph thuc vo tc ng truyn v kh nng p ng ca site . C ch truy cp theo chiu rng : Web Crawler truy cp tt c cc trang Web c a ch trong trang hin thi . Cc a ch ny ph thuc vo nhiu site khc nhau .C ch ny cho php cc Search Engine c th truy cp thng tin b thay i mt cch nhanh chng, tri rng trn nhiu lnh vc, tc truy cp n nh.Cc ti liu thu c t cc Crawler s qua mt s bc x l nh chuyn i khun dng , lc b nhng thng tin d tha, phn loi ni dung . Chng c b Indexer to ch mc . D liu to ch mc c dng cho vic tm kim ti liu nhanh chng v chnh xc .Hnh v sau y m t c ch hot ng ca Search Engine :

Hnh : C ch hot ng ca Search Engine

4. Phn loi cc Search EngineHin nay c nhiu Search Engine c dng trn Internet . Mi Search Engine dng cc k thut tm kim khc nhau .Shi nanhi chia ra lm su loi cng ngh khc nhau l : hyperlink expiration , information retrieval , metsearches, sql approaches , content base multimedia searches v cc loi khc . y Search Engine chia lm hai loi chnh theo phng php tm kim l tm kim thng thng v siu tm kim .

4.1. Tm kim thng thngCc loi Search Engine phc v yu cu tm kim theo ng tng nguyn thy ca n tc l : thu nhp ti liu v phn loi , to ch mc. Chng gm hai loi : Search Engine s dng th mc ch . Search Engine phn loi v to ch mc t ng .

4.1.1. Cc Search Engine s dng th mc ch Cc th mc ch l mt hnh thc phn loi cc site trn Internet theo ch v theo cp hn ch . Cp ch thp thng l mt chuyn ngnh hay lnh vc rng . Cc ch cp cao hn l ch hp ca ch trn n . Cp ch cui cng s ch n ti liu theo kin ch quan ca ngi phn loi . Vic phn loi ny cho php ngi s dng chn mt ch truy cp v khi cc trang web hin th loi ch m ngi s dng chn . Ngi s dng tm kim thng tin m mnh cn bng cch chn mt ch no , sau i su vo ch n cp cao hn thu hp phm vi tm kim cho n khi thy ch ca mnh v cc ta liu trong ch . Do c to bi con ngi nn th mc ch thng chnh xc hn v c c s d liu nh hn c s d liu ca cc Search Engine s dng cng ngh t ng phn loi.4.1.2. Cc Search Engine phn loi v to ch mc ti liu t ngSau khi thu thp cc ti liu t cc trang web khc , da vo cc t kha tm thy trong ti liu v khung dng ti liu , cc Search Engine s phn loi ti liu v to ch mc mt cch t ng . Khi c mt yu cu tm kim , Search Engine s dng cng ngh tm kim trn file to ch mc xc nh ti liu c lin quan nht ti yu cu tr kt qu cho ngi s dng . Do c phn loi v to ch mc t ng nn thi gian tm kim trn cc Search Engine ny thng nhanh hn nhng kt qu tr v thng c chnh xc khng cao bng cch phn loi theo ch .

4.2. Siu tm kim Cng mt yu cu tm kim , nhng ta nhn thy rng cc Search Engine khc nhau tr v cc kt qu khc nhau , v cc thng tin , ti liu m mi Search Engine thu thp c khng hon ton ging nhau v c ch tm kim trn cc ti liu c nh ch mc ca chng cng khc nhau . Mi mt Search Engine c mt chin lc v tiu chun tm kim khc nhau . V vy tm kim c nhiu ti liu cn thit , chng ta phi tm kim trn nhiu Search Engine khc nhau . Khi chng ta phi mt nhiu thi gian tm hiu cch s dng Search Engine . khc phc cc nhc im trn ca cc Search Engine thng thng cc Metearch c to ra Metasearch cng l mt cng c tm kim trn Internet . Chng lm nhim v tm kim thng tin trn chnh cc Search Engine v trn cc Webn Directory ti cng mt thi im tm ra cc ti liu ph hp nht t tt cc cc ti liu tm c . Thc cht ca Metasearch l mt cng c chuyn i cu truy vn ca ngi s dng sang khun dng ph hp cho mi Search Engine chng c th tm kim c, sau tch hp cc kt qu tr v t cc Search Engine . 5. Mt s vn lin quan ti Search Engine5.1. Nhng tr ngi ca vic tm kim thng tin trn InternetC hai vn c t ra cho vic tm kim trn web l : d liu ca web v ngi dng s dng Search Engine. D liu web c nhng kh khn : Phn b d liu : d liu c phn b rng khp th gii . N c t cc ni khc nhau , trn mi h iu hnh v khng c mt nh dng chung . Tnh khng n nh ca d liu cao : cc ti liu c th c them vo hoc xa b mt cch d dng. Lu tr d liu : d liu tng theo hm m gy tr ngi cho vic lu tr S d tha v khng c cu trc d liu : cc thng tin a ln trang web khng nht thit dng siu vn bn , c rt nhiu kiu d liu vi cu trc khc nhau, nn khng th xem chng c cu trc chung . tin cy ca d liu : d liu cc trang web c th ng, c th sai , qu hn hoc ni dung ngho nn, cc ni dung ny khng m bo bi cc nh pht hnh. D liu khng ng nht : cc ti liu trn trang web c vit di nhiu nh dng khc nhau : video, hnh nh, vn bn ,. S tng tc gia ngi s dng v Search Engine Nhn thng tin truy vn d liu t d liu ngi s dng : ngi s dng nhp thng tin cn tm kim , cc Search Engine cn phi x l cc thng tin m ngi s dng yu cu tr v cc thng tin tm c lin quan n cu lnh truy vn . Tr v kt qu tm c : Search Engine cn phi tr v cc thng tin tm c trong mt khun dng thch hp ngi dng c th xc inh c ti liu mong mun. Cc Search Engine tt cn c th c nh gi v sp xp kt qu theo mt vi ch tiu no .

5.2. H thng tm kimCc Search Engine truyn thng tiu biu dng cc tp tin ch mc c xy dng trc c lu tr trn sever . Cc tp tin ch mc thc s ny lm cho qu trnh tm kim ca cc Search Engine rt hiu qu trong phn ln cc trang web, nhng kt qu tr v khng m bo rng vn cn c gi tr bi v cc trang web thng thay i ni dung sau khi c to ch mc . T ra i khi nim tm kim trn web ng . Trong cch tip cn ny , cc Search Engine khng phi thu thp ch mc trc . Thay vo , khi ngi s dng yu cu truy vn th h thng lt nhanh qua cc trang web v pht hin ra cc thng tin lin h cc lin kt .Hnh sau ch ra kin trc ca Dynamic Search Engine n gm c : Phn giao din , Spider v Cassifier

Hnh : Cu hnh ca Dynamic Search EngineGiao din giao tip vi ngi s dng , cn Crawler giao tip vi cc trang web . Crawler c iu khin bi Classifier . Classifier s nh gi v c lng cc a ch URL m s c truy cp hoc khng c truy cp da trn tn sut xut hin t trong mi ti liu v da trn cu trc lin kt thu c bi Crawler . Mt khi ngi s dng ch r s kt qu cn thit thu nhp hoc thi gian gii hn ca ngi dng ht th Crawler s ngng vic truy cp trang web v tr kt qu v cho ngi s dng. Tng qut , Dynamic Search Engine gi s rng truy vn ngi dng l truy vn tm kim theo t kha trng vi mt URL gc . URL gc c g l root page. Khi bt u tm kim , Dynamic Search Engine truy cp cc ta liu cng vi ng dn lin kt ca trang root page , c gng trnh nhng trang web khng lin quan ti yu cu ngi s dng v em nhng d liu c lin quan n truy vn tm kim. ngha bn cht trong tm kim ng l ng dn lin kt c s u tin, bt u t root page v truy vn c yu cu . mi bc nhng trang cha c truy cp c u tin cao nht c truy cp bi Crawler v c phn tch bi Classifier. Classifier chu trch nhim cho vic nghin cu th nhn ti liu c lin quan t cc trang cung cp bi ngi dng v xem xt trn cc trang thu thp c quyt nh hng m rng lin kt . Nu mt trang c tm thy c lin quan th nhng lin kt trang s c nh gi l c u tien cao v s c chn vo danh sch hng i cc URL s c truy cp .

5.3. Cc cu lnh truy vn Cc cu lnh truy vn i vi Search Engine c hiu l cc thng tin v yu cu tm kim c ngi s dng a vo . Cu trc ca Search Engine, c bit l phn Indexer c quan h cht ch vi vic qun l cc cu lnh truy vn. Mi mt m hnh Search Engine thng thch hp vi mt kiu truy vn xc nh .

5.3.1. X l cc cu lnh truy vnX l cc cu lnh truy vn l ch cch chuyn i cc yu cu ngi s dng sang khun dng m Search Engine c th hiu c. Mc u tin lin quan n khun dng cc thng tin thng l cc t m ngi s dng a vo . l cc cu hi hoc danh sch cc t nhp vo vi lin kt , rng buc khc nhau. Mc tip theo l cc Search Engine cn phi chuyn nhng t ny thnh cc th (Token). Mc ny kim tra khun dng chnh t . Cui cng mc 3 l dng cc th ny tm ti liu .

Hnh : Mc x l cc cu lnh truy vn

5.3.2. Truy vn s dng ton t BooleanCc cu lnh truy vn s dng ton t Boolean l cc cu lnh s dng ton t AND,OR,NOT lin kt cc t . Nu ngi s dng mun tm kim cc ti liu c cha c hai t search v string th cu lnh nhp vo s l search OR string. Truy vn s dng ton t Boolean cho phsp m rng hay thu hp phm vi tm kim mt csc d dng . im yu cu chng l khng phi ngi s dng no cng c th s dng thnh tho ton t Boolean mc d rt n gin .

5.3.3. Truy vn s dng ngn ng t nhinTruy vn s dng ngn ng t nhin l cc truy vn m ngi s dng a ra yu cu thng qua cc cu hi hay tr li cc cu hi m Search Engine a ra. x l cu lnh truy vn s dng ngn ng t nhin , Search Engine phi phn tch cu lnh , xc inh cc t no trong truy vn c d liu c to ch mc , t no khng cn thit tm kim. Kiu truy vn ny thun tin cho ngi s dng nhng kh cho my tnh x l v n kh xc nh c ngha ca t trong mt ng cnh c th .

5.3.4. Truy vn s dng t inTruy vn s dng t in l kiu truy vn m ngi s dng chn cc t truy vn t mt tp cc t c xc ng trc bi Search Engine . Kiu truy vn ny thng c s dng tm kim mt chuyn ngnh hp vi cc thut ng ca ngnh . u im ca kiu ny l bc x li u tin c l t ng cho ngi s dng.

5.3.5. Truy vn khng chnh xcTruy vn khng chnh xc l kiu truy vn m cc cu lnh truy vn c cc t kha c hiu theo ngha ny , ngha khc, t ny ,t khc. Cc t trong cu lnh truy vn c tch phn gc t , kim tra li chnh t, so snh vi cc t c to ch mc tm kim cc t , cc ti liu ph hp .

5.3.6. Truy vn theo lnh vc xc sut - thng kTruy vn theo lnh vc xc sut - thng k lin quan n cc thc Search Engine xc nh mc lin quan ca ti liu vi cu lnh truy vn . Truy vn ny s dng mt hm thng k nh tnh s t xut hin trong ti liu, tm cc t ng ngha,.. xc nh kt qu tr v cho ngi truy vn .