Εύρεση ∆ιαχείριση Πληροφορίαςστον...
TRANSCRIPT
1
Εύρεση & ∆ιαχείρισηΠληροφορίας στονΠαγκόσµιο Ιστό
∆ιδάσκων –∆ηµήτριος Κατσαρός, Ph.D.
@ Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύωνΠανεπιστήµιο Θεσσαλίας
∆ιάλεξη 7η: 17/04/2007
2
Web characteristics
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 3
Search use …(iProspect Survey, 4/04, http://www.iprospect.com/premiumPDFs/iProspectSurveyComplete.pdf)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 4
Without search engines the web wouldn’t scale
1. No incentive in creating content unless it can be easily found –other finding methods haven’t kept pace (taxonomies, bookmarks, etc)
2. The web is both a technology artifact and a social environment • “The Web has become the “new normal” in the American way of
life; those who don’t go online constitute an ever-shrinking minority.” – [Pew Foundation report, January 2005]
3. Search engines make aggregation of interest possible: • Create incentives for very specialized niche players
• Economical – specialized stores, providers, etc • Social – narrow interests, specialized communities, etc
4. The acceptance of search interaction makes “unlimited selection”stores possible:• Amazon, Netflix, etc
5. Search turned out to be the best mechanism for advertising on the web, a $15+ B industry. • Growing very fast but entire US advertising industry $250B –
huge room to grow• Sponsored search marketing is about $10B
5
Classical IR vs. Web IR
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 6
Basic assumptions of Classical Information Retrieval
• Corpus: Fixed document collection• Goal: Retrieve documents with information
content that is relevant to user’s information need
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 7
Classic IR Goal
•Classic relevance• For each query Q and stored document D in a given corpus
assume there exists relevance Score(Q, D)• Score is average over users U and contexts C
• Optimize Score(Q, D) as opposed to Score(Q, D, U, C)• That is, usually:
• Context ignored• Individuals ignored• Corpus predetermined
Bad assumptionsin the web context
8
Web IR
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 9
The coarse-level dynamics
Content creators Content aggregators
Feeds
Crawls
Content consumers
Adv
ertis
emen
tEd
itoria
l
Subs
crip
tion
Tran
sact
ion
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 10
Brief (non-technical) history• Early keyword-based engines
• Altavista, Excite, Infoseek, Inktomi, ca. 1995-1997• Paid placement ranking: Goto.com (morphed into
Overture.com → Yahoo!)• Your search ranking depended on how much you paid• Auction for keywords: casino was expensive!
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 11
Brief (non-technical) history• 1998+: Link-based ranking pioneered by Google
• Blew away all early engines save Inktomi• Great user experience in search of a business model• Meanwhile Goto/Overture’s annual revenues were
nearing $1 billion• Result: Google added paid-placement “ads” to the
side, independent of search results• Yahoo follows suit, acquiring Overture (for paid
placement) and Inktomi (for search)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 12
Algorithmic results.
Ads
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 13
Ads vs. search results• Google has maintained that ads
(based on vendors bidding for keywords) do not affect vendors’rankings in search results
Web Results 1 - 10 of about 7,310,000 for miele. (0.12 seconds)
Miele, Inc -- Anything else is a compromise At the heart of your home, Appliances by Miele. ... USA. to miele.com. Residential Appliances. Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System ... www.miele.com/ - 20k - Cached - Similar pages
Miele Welcome to Miele, the home of the very best appliances and kitchens in the world. www.miele.co.uk/ - 3k - Cached - Similar pages
Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten ... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter www.zu-tisch.de. Miele weltweit ...ein Leben lang. ... Wählen Sie die Miele Vertretung Ihres Landes. www.miele.de/ - 10k - Cached - Similar pages
Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE ... www.miele.at/ - 3k - Cached - Similar pages
Sponsored Links
CG Appliance Express Discount Appliances (650) 756-3931Same Day Certified Installation www.cgappliance.com San Francisco-Oakland-San Jose, CA Miele Vacuum Cleaners Miele Vacuums- Complete SelectionFree Shipping! www.vacuums.com Miele Vacuum Cleaners Miele-Free Air shipping! All models. Helpful advice. www.best-vacuum.com
Search =miele
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 14
Ads vs. search results
• Other vendors (Yahoo, MSN) have made similar statements from time to time• Any of them can change anytime
• We will focus primarily on search results independent of paid placement ads• Although the latter is a fascinating technical
subject in itself
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 15
Web search basics
The Web
Ad indexes
Web Results 1 - 10 of about 7,310,000 for miele. (0.12 seconds)
Miele, Inc -- Anything else is a compromise At the heart of your home, Appliances by Miele. ... USA. to miele.com. Residential Appliances. Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System ... www.miele.com/ - 20k - Cached - Similar pages
Miele Welcome to Miele, the home of the very best appliances and kitchens in the world. www.miele.co.uk/ - 3k - Cached - Similar pages
Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten ... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter www.zu-tisch.de. Miele weltweit ...ein Leben lang. ... Wählen Sie die Miele Vertretung Ihres Landes. www.miele.de/ - 10k - Cached - Similar pages
Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE ... www.miele.at/ - 3k - Cached - Similar pages
Sponsored Links
CG Appliance Express Discount Appliances (650) 756-3931 Same Day Certified Installation www.cgappliance.com San Francisco-Oakland-San Jose, CA Miele Vacuum Cleaners Miele Vacuums- Complete Selection Free Shipping! www.vacuums.com Miele Vacuum Cleaners Miele-Free Air shipping! All models. Helpful advice. www.best-vacuum.com
Web spider
Indexer
Indexes
Search
User
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 16
User Needs
• Need [Brod02, RL04]• Informational – want to learn about something (~40% / 65%)
• Navigational – want to go to that page (~25% / 15%)
• Transactional – want to do something (web-mediated) (~35% / 20%)• Access a service• Downloads
• Shop• Gray areas
• Find a good hub• Exploratory search “see what’s there”
Low hemoglobin
United Airlines
Seattle weatherMars surface images
Canon S410
Car rental Brasil
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 17
Web search users• Make ill defined
queries• Short
• AV 2001: 2.54 terms avg, 80% < 3 words)
• AV 1998: 2.35 terms avg, 88% < 3 words [Silv98]
• Imprecise terms• Sub-optimal syntax (most
queries without operator)• Low effort
• Wide variance in• Needs• Expectations• Knowledge• Bandwidth
• Specific behavior• 85% look over one
result screen only (mostly above the fold)
• 78% of queries are not modified (one query/session)
• Follow links –“the scent of
information” ...
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 18
Query Distribution
Power law: few popular broad queries, many rare specific queries
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 19
How far do people look for results?
(Source: iprospect.com WhitePaper_2006_SearchEngineUserBehavior.pdf)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 20
True example*
Corpus
TASK
Info Need
Query
Verbal form
Results
SEARCHENGINE
QueryRefinement
Noisy building fan in courtyard
Info about EPA regulations
What are the EPA rules about noise pollution
EPA sound pollution
Mis-conception
Mis-translation
Mis-formulation
PolysemySynonimy
* To Google or to GOTO, Business Week Online, September 28, 2001
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 21
Users’ empirical evaluation of results
• Quality of pages varies widely• Relevance is not enough• Other desirable qualities (non IR!!)
• Content: Trustworthy, new info, non-duplicates, well maintained,• Web readability: display correctly & fast• No annoyances: pop-ups, etc
• Precision vs. recall• On the web, recall seldom matters
• What matters• Precision at 1? Precision above the fold?• Comprehensiveness – must be able to deal with obscure queries
• Recall matters when the number of matches is very small• User perceptions may be unscientific, but are
significant over a large aggregate
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 22
Users’ empirical evaluation of engines• Relevance and validity of results• UI – Simple, no clutter, error tolerant• Trust – Results are objective• Coverage of topics for poly-semic queries• Pre/Post process tools provided
• Mitigate user errors (auto spell check, syntax errors,…)• Explicit: Search within results, more like this, refine ...• Anticipative: related searches
• Deal with idiosyncrasies• Web specific vocabulary
• Impact on stemming, spell-check, etc• Web addresses typed in the search box• …
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 23
Loyalty to a given search engine(iProspect Survey, 4/04)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 24
The Web corpus
• No design/co-ordination• Distributed content creation, linking,
democratization of publishing• Content includes truth, lies, obsolete
information, contradictions …• Unstructured (text, html, …), semi-
structured (XML, annotated photos), structured (Databases)…
• Scale much larger than previous text corpora … but corporate records are catching up.
• Growth – slowed down from initial “volume doubling every few months”but still expanding
• Content can be dynamically generatedThe Web
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 25
The Web: Dynamic content
• A page without a static html version• E.g., current status of flight AA129• Current availability of rooms at a hotel
• Usually, assembled at the time of a request from a browser• Typically, URL has a ‘?’ character in it
Application server
Browser
AA129
Back-enddatabases
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 26
Dynamic content
• Most dynamic content is ignored by web spiders• Many reasons including malicious spider traps
• Some dynamic content (news stories from subscriptions) are sometimes delivered as dynamic content• Application-specific spidering
• Spiders commonly view web pages just as Lynx (a text browser) would
• Note: even “static” pages are typically assembled on the fly (e.g., headers are common)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 27
The web: size
• What is being measured?• Number of hosts• Number of (static) html pages
• Volume of data
• Number of hosts – netcraft survey• http://news.netcraft.com/archives/web_server_survey.html• Monthly report on how many web hosts & servers are out there
• Number of pages – numerous estimates (will discuss later)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 28
Netcraft Web Server Surveyhttp://news.netcraft.com/archives/web_server_survey.html
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 29
The web: evolution
• All of these numbers keep changing• Relatively few scientific studies of the
evolution of the web [Fetterly & al, 2003]• http://research.microsoft.com/research/sv/sv-pubs/p97-
fetterly/p97-fetterly.pdf
• Sometimes possible to extrapolate from small samples (fractal models) [Dill & al, 2001]• http://www.vldb.org/conf/2001/P069.pdf
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 30
Rate of change• [Cho00] 720K pages from 270 popular sites sampled
daily from Feb 17 – Jun 14, 1999 • Any changes: 40% weekly, 23% daily
• [Fett02] Massive study 151M pages checked over few months• Significant changed -- 7% weekly• Small changes – 25% weekly
• [Ntul04] 154 large sites re-crawled from scratch weekly• 8% new pages/week • 8% die• 5% new content• 25% new links/week
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 31
Static pages: rate of change• Fetterly et al. study (2002): several views of data, 150 million pages
over 11 weekly crawls• Bucketed into 85 groups by extent of change
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 32
Other characteristics
• Significant duplication• Syntactic – 30%-40% (near) duplicates [Brod97, Shiv99b, etc.]• Semantic – ???
• High linkage • More than 8 links/page in the average
• Complex graph topology• Not a small world; bow-tie structure [Brod00]
• Spam• Billions of pages
33
Spam
Search Engine Optimization
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 34
The trouble with paid placement…
• It costs money. What’s the alternative?• Search Engine Optimization:
• “Tuning” your web page to rank highly in the search results for select keywords
• Alternative to paying for placement• Thus, intrinsically a marketing function
• Performed by companies, webmasters and consultants (“Search engine optimizers”) for their clients
• Some perfectly legitimate, some very shady
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 35
Simplest forms• First generation engines relied heavily on tf/idf
• The top-ranked pages for the query maui resort were the ones containing the most maui’s and resort’s
• SEOs responded with dense repetitions of chosen terms• e.g., maui resort maui resort maui resort• Often, the repetitions would be in the same color as the
background of the web page• Repeated terms got indexed by crawlers• But not visible to humans on browsers
Pure word density cannot be trusted as an IR signal
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 36
Variants of keyword stuffing• Misleading meta-tags, excessive repetition• Hidden text with colors, style sheet tricks,
etc.
Meta-Tags = “… London hotels, hotel, holiday inn, hilton, discount, booking, reservation, sex, mp3, britney spears, viagra, …”
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 37
Search engine optimization (Spam)
• Motives• Commercial, political, religious, lobbies• Promotion funded by advertising budget
• Operators• Contractors (Search Engine Optimizers) for lobbies, companies• Web masters• Hosting services
• Forums• E.g., Web master world ( www.webmasterworld.com )
• Search engine specific tricks • Discussions about academic papers ☺
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 38
Cloaking• Serve fake content to search engine spider• DNS cloaking: Switch IP address. Impersonate
Is this a SearchEngine spider?
Y
N
SPAM
RealDocCloaking
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 39
The spam industry
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 40
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 41
More spam techniques
• Doorway pages• Pages optimized for a single keyword that re-direct to the
real target page• Link spamming
• Mutual admiration societies, hidden links, awards – more on these later
• Domain flooding: numerous domains that point or re-direct to a target page
• Robots• Fake query stream – rank checking programs
• “Curve-fit” ranking programs of search engines• Millions of submissions via Add-Url
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 42
The war against spam
• Quality signals - Prefer authoritative pages based on:
• Votes from authors (linkage signals)
• Votes from users (usage signals)• Policing of URL
submissions• Anti robot test
• Limits on meta-keywords• Robust link analysis
• Ignore statistically implausible linkage (or text)
• Use link analysis to detect spammers (guilt by association)
• Spam recognition by machine learning
• Training set based on known spam
• Family friendly filters• Linguistic analysis, general
classification techniques, etc.• For images: flesh tone
detectors, source text analysis, etc.
• Editorial intervention• Blacklists• Top queries audited• Complaints addressed• Suspect pattern detection
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 43
More on spam• Web search engines have policies on SEO practices
they tolerate/block• http://help.yahoo.com/help/us/ysearch/index.html• http://www.google.com/intl/en/webmasters/
• Adversarial IR: the unending (technical) battle between SEO’s and web search engines
• Research http://airweb.cse.lehigh.edu/
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 44
Answering “the need behind the query”
• Semantic analysis• Query language determination
• Auto filtering • Different ranking (if query in Japanese do not return English)
• Hard & soft (partial) matches• Personalities (triggered on names)• Cities (travel info, maps)• Medical info (triggered on names and/or results)• Stock quotes, news (triggered on stock symbol)• Company info• Etc.
• Natural Language reformulation• Integration of Search and Text Analysis
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 45
The spatial context -- geo-search• Two aspects• Geo-coding -- encode geographic coordinates to make search
effective• Geo-parsing -- the process of identifying geographic context.
• Geo-coding• Geometrical hierarchy (squares)• Natural hierarchy (country, state, county, city, zip-codes, etc)• Geo-parsing• Pages (infer from phone nos, zip, etc). About 10% can be parsed.• Queries (use dictionary of place names) • Users
• Explicit (tell me your location -- used by NL, registration, from ISP)• From IP data
• Mobile phones • In its infancy, many issues (display size, privacy, etc)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 46
Yahoo!: britney spears
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 47
Ask Jeeves: las vegas
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 48
Yahoo!: salvador hotels
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 49
Yahoo shortcuts• Various types of queries that are “understood”
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 50
Google andrei broder new york
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 51
Answering “the need behind the query”: Context
• Context determination • spatial (user location/target location)• query stream (previous queries)• personal (user profile) • explicit (user choice of a vertical search, ) • implicit (use Google from France, use google.fr)
• Context use• Result restriction
• Kill inappropriate results• Ranking modulation
• Use a “rough” generic ranking, but personalize later
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 52
Google: dentists bronx
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 53
Yahoo!: dentists (bronx)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 54
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 55
Query expansion
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 56
Context transfer
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 57
No transfer
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 58
Context transfer
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 59
Transfer from search results
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 60
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 61
Resources• IIR Chapter 19
62
Web characteristics II:Web size measurementNear-duplicate detection
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 63
Today’s topics
• Estimating web size and search engine index size
• Near-duplicate document detection
64
Size of the web
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 65
What is the size of the web ?• Issues
• The web is really infinite • Dynamic content, e.g., calendar • Soft 404: www.yahoo.com/<anything> is a valid page
• Static web contains syntactic duplication, mostly due to mirroring (~30%)
• Some servers are seldom connected• Who cares?
• Media, and consequently the user• Engine design• Engine crawl policy. Impact on recall.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 66
What can we attempt to measure?•The relative sizes of search engines • The notion of a page being indexed is still reasonably well
defined.• Already there are problems
• Document extension: e.g. engines index pages not yet crawled, by indexing anchortext.
• Document restriction: All engines restrict what is indexed (firstn words, only relevant words, etc.)
•The coverage of a search engine relative to another particular crawling process.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 67
New definition?
• (IQ is whatever the IQ tests measure.)• The statically indexable web is whatever
search engines index.• Different engines have different preferences
• max url depth, max count/host, anti-spam rules, priority rules, etc.
• Different engines index different things under the same URL:• frames, meta-keywords, document restrictions, document
extensions, ...
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 68
Statistical methods• Random queries
• Random searches
• Random IP addresses
• Random walks
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 69
A ∩ B = (1/2) * Size A
A ∩ B = (1/6) * Size B
(1/2)*Size A = (1/6)*Size B
∴ Size A / Size B =
(1/6)/(1/2) = 1/3
Sample URLs randomly from A
Check if contained in B
and vice versa
A ∩ B
Each test involves: (i) Sampling (ii) Checking
Relative Size from Overlap [Bharat & Broder, 98]
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 70
Sampling URLs
• Ideal strategy: Generate a random URL and check for containment in each index.
• Problem: Random URLs are hard to find! Enough to generate a random URL contained in a given Engine.
Key lesson from this lecture.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 71
Random URLs from random queries [Bharat & B, 98]
• Generate random query: how?• Lexicon: 400,000+ words from a crawl of Yahoo!• Conjunctive Queries: w1 and w2
e.g., vocalists AND rsi
• Get 100 result URLs from the source engine• Choose a random URL as the candidate to
check for presence in other engines.• This distribution induces a probability weight
W(p) for each page. • Conjecture:
W(SE1) / W(SE2) ~ |SE1| / |SE2|
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 72
Query Based Checking
• Strong Query to check for a document D:• Download document. Get list of words. • Use 8 low frequency words as AND query
• Check if D is present in result set.• Problems:
• Near duplicates• Frames• Redirects• Engine time-outs• Might be better to use e.g. 5 distinct conjunctive
queries of 6 words each.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 73
Computing Relative Sizes and Total Coverage [BB98]
• Six pair-wise overlapsfah* a - fha* h = ε1fai* a - fia* i = ε2fae* a - fea* e = ε3fhi* h - fih* i = ε4fhe* h - feh* e = ε5fei* e - fie* i = ε6
• Arbitrarily, let a = 1.
• We have 6 equations and 3 unknowns.
• Solve for e, h and i to minimize Σ εi2
• Compute engine overlaps.
• Re-normalize so that the total joint coverage is 100%
a = AltaVista, e = Excite, h = HotBot, i = Infoseek
fxy = fraction of x in y
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 74
Advantages & disadvantages
• Statistically sound under the induced weight.• Biases induced by random query
• Query Bias: Favors content-rich pages in the language(s) of the lexicon
• Ranking Bias: Solution: Use conjunctive queries & fetch all• Checking Bias: Duplicates, impoverished pages omitted• Document or query restriction bias: engine might not deal
properly with 8 words conjunctive query• Malicious Bias: Sabotage by engine• Operational Problems: Time-outs, failures, engine
inconsistencies, index modification.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 75
Random searches• Choose random searches extracted from a local
log [Lawrence & Giles 97] or build “random searches” [Notess]• Use only queries with small results sets. • Count normalized URLs in result sets.• Use ratio statistics
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 76
Advantages & disadvantages
• Advantage• Might be a better reflection of the human perception of
coverage
• Issues• Samples are correlated with source of log• Duplicates• Technical statistical problems (must have non-zero
results, ratio average, use harmonic mean?)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 77
Random searches [Lawr98, Lawr99]
• 575 & 1050 queries from the NEC RI employee logs• 6 Engines in 1998, 11 in 1999• Implementation:
• Restricted to queries with < 600 results in total• Counted URLs from each engine after verifying query match• Computed size ratio & overlap for individual queries • Estimated index size ratio & overlap by averaging over all
queries
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 78
• adaptive access control • neighborhood preservation
topographic • hamiltonian structures • right linear grammar • pulse width modulation
neural • unbalanced prior probabilities • ranked assignment method • internet explorer favourites
importing • karvel thornber• zili liu
Queries from Lawrence and Giles study
• softmax activation function • bose multidimensional system
theory • gamma mlp• dvi2pdf • john oliensis• rieke spikes exploring neural • video watermarking • counterpropagation network • fat shattering dimension • abelson amorphous computing
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 79
Random IP addresses [Lawrence & Giles ‘99]• Generate random IP addresses• Find a web server at the given address
• If there’s one• Collect all pages from server.• Method first used by O’Neill, McClain, & Lavoie,
“A Methodology for Sampling the World Wide Web”, 1997. http://digitalarchive.oclc.org/da/ViewObject.jsp?objid=0000003447
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 80
Random IP addresses [ONei97, Lawr99]
• HTTP requests to random IP addresses • Ignored: empty or authorization required or excluded• [Lawr99] Estimated 2.8 million IP addresses running
crawlable web servers (16 million total) from observing 2500 servers.
• OCLC using IP sampling found 8.7 M hosts in 2001• Netcraft [Netc02] accessed 37.2 million hosts in July 2002
• [Lawr99] exhaustively crawled 2500 servers and extrapolated• Estimated size of the web to be 800 million• Estimated use of metadata descriptors:
• Meta tags (keywords, description) in 34% of home pages, Dublin core metadata in 0.3%
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 81
Advantages & disadvantages
• Advantages• Clean statistics• Independent of crawling strategies
• Disadvantages• Doesn’t deal with duplication • Many hosts might share one IP, or not accept requests• No guarantee all pages are linked to root page.
• Eg: employee pages • Power law for # pages/hosts generates bias towards sites
with few pages.• But bias can be accurately quantified IF underlying
distribution understood• Potentially influenced by spamming (multiple IP’s for
same server to avoid IP block)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 82
Random walks[Henzinger et al WWW9]
• View the Web as a directed graph• Build a random walk on this graph
• Includes various “jump” rules back to visited sites• Does not get stuck in spider traps!• Can follow all links!
• Converges to a stationary distribution• Must assume graph is finite and independent of the walk. • Conditions are not satisfied (cookie crumbs, flooding)• Time to convergence not really known
• Sample from stationary distribution of walk• Use the “strong query” method to check coverage by SE
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 83
Dependence on seed list• How well connected is the graph? [Broder et al.,
WWW9]
•
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 84
Advantages & disadvantages• Advantages
• “Statistically clean” method at least in theory!• Could work even for infinite web (assuming convergence)
under certain metrics.• Disadvantages
• List of seeds is a problem.• Practical approximation might not be valid.• Non-uniform distribution
• Subject to link spamming
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 85
Conclusions• No sampling solution is perfect. • Lots of new ideas ...• ....but the problem is getting harder• Quantitative studies are fascinating and a good
research problem
86
Duplicate detection
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 87
Duplicate documents
• The web is full of duplicated content• Strict duplicate detection = exact match
• Not as common• But many, many cases of near duplicates
• E.g., Last modified date the only difference
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 88
Duplicate/Near-Duplicate Detection
• Duplication: Exact match can be detected with fingerprints
• Near-Duplication: Approximate match• Overview
• Compute syntactic similarity with an edit-distance measure• Use similarity threshold to detect near-duplicates
• E.g., Similarity > 80% => Documents are “near duplicates”• Not transitive though sometimes used transitively
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 89
Computing Similarity• Features:
• Segments of a document (natural or artificial breakpoints)• Shingles (Word N-Grams)• a rose is a rose is a rose →
a_rose_is_arose_is_a_rose
is_a_rose_is • Similarity Measure between two docs (= sets of shingles)
• Set intersection [Brod98](Specifically, Size_of_Intersection / Size_of_Union )
Jaccard measure
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 90
Shingles + Set Intersection
• Computing exact set intersection of shingles between all pairs of documents is expensive/intractable• Approximate using a cleverly chosen subset of shingles from
each (a sketch)• Estimate (size_of_intersection / size_of_union)based on a short sketch
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 91
Sketch of a document
• Create a “sketch vector” (of size ~200) for each document• Documents that share ≥ t (say 80%)
corresponding vector elements are near duplicates
• For doc D, sketchD[ i ] is as follows:• Let f map all shingles in the universe to 0..2m (e.g., f =
fingerprinting)• Let πi be a random permutation on 0..2m
• Pick MIN {πi(f(s))} over all shingles s in D
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 92
Computing Sketch[i] for Doc1
Document 1
264
264
264
264
Start with 64-bit f(shingles)
Permute on the number line
with πi
Pick the min value
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 93
Test if Doc1.Sketch[i] = Doc2.Sketch[i]
Document 1 Document 2
264
264
264
264
264
264
264
264
Are these equal?
Test for 200 random permutations: π1, π2,… π200
A B
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 94
However…
Document 1 Document 2
264
264
264
264
264
264
264
264
A = B iff the shingle with the MIN value in the union of Doc1 and Doc2 is common to both (I.e., lies in the intersection)
This happens with probability:Size_of_intersection / Size_of_union
BA
Why?
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 95
Set Similarity• Set Similarity (Jaccard measure)
• View sets as columns of a matrix; one row for each element in the universe. aij = 1 indicates presence of item i in set j
• Example
ji
jiji CC
CC)C,Jaccard(C
U
I=
C1 C2
0 11 0
1 1 Jaccard(C1,C2) = 2/5 = 0.40 01 10 1
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 96
Key Observation• For columns Ci, Cj, four types of rows
Ci Cj
A 1 1B 1 0C 0 1D 0 0
• Overload notation: A = # of rows of type A• Claim
CBAA)C,Jaccard(C ji ++
=
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 97
“Min” Hashing• Randomly permute rows• Hash h(Ci) = index of first row with 1 in column Ci• Surprising Property
• Why?• Both are A/(A+B+C)• Look down columns Ci, Cj until first non-Type-D row• h(Ci) = h(Cj) type A row
[ ] ( )jiji C,CJaccard )h(C)h(C P ==
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 98
Min-Hash sketches• Pick P random row permutations • MinHash sketch
SketchD = list of P indexes of first rows with 1 in column C
• Similarity of signatures • Let sim[sketch(Ci),sketch(Cj)] = fraction of permutations
where MinHash values agree • Observe E[sim(sig(Ci),sig(Cj))] = Jaccard(Ci,Cj)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 99
Example
C1 C2 C3R1 1 0 1R2 0 1 1R3 1 0 0R4 1 0 1R5 0 1 0
SignaturesS1 S2 S3
Perm 1 = (12345) 1 2 1Perm 2 = (54321) 4 5 4Perm 3 = (34512) 3 5 4
Similarities1-2 1-3 2-3
Col-Col 0.00 0.50 0.25Sig-Sig 0.00 0.67 0.00
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 100
Implementation Trick• Permuting rows even once is prohibitive• Row Hashing
• Pick P hash functions hk: {1,…,n} {1,…,O(n)}• Ordering under hk gives random row permutation
• One-pass Implementation• For each Ci and hk, keep “slot” for min-hash value• Initialize all slot(Ci,hk) to infinity• Scan rows in arbitrary order looking for 1’s
• Suppose row Rj has 1 in column Ci• For each hk,
• if hk(j) < slot(Ci,hk), then slot(Ci,hk) hk(j)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 101
Example
C1 C2R1 1 0R2 0 1R3 1 1R4 1 0R5 0 1
h(x) = x mod 5g(x) = 2x+1 mod 5
h(1) = 1 1 -g(1) = 3 3 -
h(2) = 2 1 2g(2) = 0 3 0
h(3) = 3 1 2g(3) = 2 2 0
h(4) = 4 1 2g(4) = 4 2 0
h(5) = 0 1 0g(5) = 1 2 0
C1 slots C2 slots
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 102
Comparing Signatures• Signature Matrix S
• Rows = Hash Functions• Columns = Columns• Entries = Signatures
• Compute – Pair-wise similarity of signature columns
• Problem• MinHash fits column signatures in memory• But comparing signature-pairs takes too much time
• Technique to limit candidate pairs?• Locality Sensitive Hashing (LSH)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 103
Resources
• IIR 19• See also
• Phelps & Wilensky. Robust Hyperlinks & Locations, 2002
• Ziv Bar-Yossef and Maxim Gurevich. Random Sampling from a Search Engine’s Index, WWW 2006.
• Broder et al. Estimating corpus size via queries. CIKM 2006.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 104
More resources• Related papers:
• [Bar Yossef & al, VLDB 2000], [Rusmevichientong & al, 2001], [Bar Yossef & al, 2003]
105
Crawling and web indexes
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 106
Today’s lecture• Crawling• Connectivity servers
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 107
Basic crawler operation• Begin with known “seed” pages• Fetch and parse them
•Extract URLs they point to•Place the extracted URLs on a
queue• Fetch each URL on the queue and
repeat
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 108
Crawling picture
Web
URLs crawledand parsed
URLs frontier
Unseen Web
Seedpages
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 109
Simple picture – complications• Web crawling isn’t feasible with one machine
• All of the above steps distributed• Even non-malicious pages pose challenges
• Latency/bandwidth to remote servers vary• Webmasters’ stipulations
• How “deep” should you crawl a site’s URL hierarchy?• Site mirrors and duplicate pages
• Malicious pages• Spam pages • Spider traps – incl dynamically generated
• Politeness – don’t hit a server too often
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 110
What any crawler must do
• Be Polite: Respect implicit and explicit politeness considerations for a website• Only crawl pages you’re allowed to• Respect robots.txt (more on this shortly)
• Be Robust: Be immune to spider traps and other malicious behavior from web servers
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 111
What any crawler should do
• Be capable of distributed operation: designed to run on multiple distributed machines
• Be scalable: designed to increase the crawl rate by adding more machines
• Performance/efficiency: permit full use of available processing and network resources
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 112
What any crawler should do
• Fetch pages of “higher quality” first• Continuous operation: Continue
fetching fresh copies of a previously fetched page
• Extensible: Adapt to new data formats, protocols
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 113
Updated crawling picture
URLs crawledand parsed
Unseen Web
SeedPages
URL frontier
Crawling thread
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 114
URL frontier
• Can include multiple pages from the same host
• Must avoid trying to fetch them all at the same time
• Must try to keep all crawling threads busy
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 115
Explicit and implicit politeness
• Explicit politeness: specifications from webmasters on what portions of site can be crawled• robots.txt
• Implicit politeness: even with no specification, avoid hitting any site too often
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 116
Robots.txt
• Protocol for giving spiders (“robots”) limited access to a website, originally from 1994• www.robotstxt.org/wc/norobots.html
• Website announces its request on what can(not) be crawled• For a URL, create a file URL/robots.txt• This file specifies access restrictions
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 117
Robots.txt example• No robot should visit any URL starting with
"/yoursite/temp/", except the robot called “searchengine":
User-agent: *Disallow: /yoursite/temp/
User-agent: searchengine
Disallow:
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 118
Processing steps in crawling• Pick a URL from the frontier• Fetch the document at the URL• Parse the URL
• Extract links from it to other docs (URLs)• Check if URL has content already seen
• If not, add to indexes• For each extracted URL
• Ensure it passes certain URL filter tests• Check if it is already in the frontier (duplicate URL
elimination)
E.g., only crawl .edu, obey robots.txt, etc.
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 119
Basic crawl architecture
WWW
Fetch
DNS
Parse
Contentseen?
URLfilter
DupURLelim
DocFP’s
URLset
URL Frontier
robotsfilters
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 120
DNS (Domain Name Server)• A lookup service on the internet
• Given a URL, retrieve its IP address• Service provided by a distributed set of servers – thus,
lookup latencies can be high (even seconds)• Common OS implementations of DNS lookup are
blocking: only one outstanding request at a time• Solutions
• DNS caching• Batch DNS resolver – collects requests and sends them
out together
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 121
Parsing: URL normalization• When a fetched document is parsed, some of
the extracted links are relative URLs• E.g., at
http://en.wikipedia.org/wiki/Main_Pagewe have a relative link to
/wiki/Wikipedia:General_disclaimer which is the same as the absolute URLhttp://en.wikipedia.org/wiki/Wikipedia:General_disclaimer
• During parsing, must normalize (expand) such relative URLs
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 122
Content seen?
• Duplication is widespread on the web
• If the page just fetched is already in the index, do not further process it
• This is verified using document fingerprints or shingles
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 123
Filters and robots.txt• Filters – regular expressions for URL’s
to be crawled/not• Once a robots.txt file is fetched from a
site, need not fetch it repeatedly• Doing so burns bandwidth, hits web
server• Cache robots.txt files
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 124
Duplicate URL elimination
• For a non-continuous (one-shot) crawl, test to see if an extracted+filtered URL has already been passed to the frontier
• For a continuous crawl – see details of frontier implementation
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 125
Distributing the crawler
• Run multiple crawl threads, under different processes – potentially at different nodes• Geographically distributed nodes
• Partition hosts being crawled into nodes• Hash used for partition
• How do these nodes communicate?
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 126
Communication between nodes• The output of the URL filter at each node is sent
to the Duplicate URL Eliminator at all nodes
WWW
Fetch
DNS
ParseContentseen?
URLfilter
DupURLelim
DocFP’s
URLset
URL Frontier
robotsfilters
Hostsplitter
Tootherhosts
Fromotherhosts
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 127
URL frontier: two main considerations
• Politeness: do not hit a web server too frequently
• Freshness: crawl some pages more often than others• E.g., pages (such as News sites) whose content
changes oftenThese goals may conflict each other.(E.g., simple priority queue fails – many
links out of a page go to its own site, creating a burst of accesses to that site.)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 128
Politeness – challenges
• Even if we restrict only one thread to fetch from a host, can hit it repeatedly
• Common heuristic: insert time gap between successive requests to a host that is >> time for most recent fetch from that host
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 129
URL frontier: Mercator scheme
Prioritizer
Biased front queue selectorBack queue router
Back queue selector
K front queues
B back queuesSingle host on each
URLs
Crawl thread requesting URL
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 130
Mercator URL frontier
• URL’s flow in from the top into the frontier• Front queues manage prioritization• Back queues enforce politeness• Each queue is FIFO
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 131
Front queues
Prioritizer
1 K
Biased front queue selectorBack queue router
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 132
Front queues
• Prioritizer assigns to URL an integer priority between 1 and K• Appends URL to corresponding queue
• Heuristics for assigning priority• Refresh rate sampled from previous crawls• Application-specific (e.g., “crawl news sites
more often”)
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 133
Biased front queue selector
• When a back queue requests a URL (in a sequence to be described): picks a front queue from which to pull a URL
• This choice can be round robin biased to queues of higher priority, or some more sophisticated variant• Can be randomized
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 134
Back queues
Biased front queue selectorBack queue router
Back queue selector
1 B
Heap
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 135
Back queue invariants
• Each back queue is kept non-empty while the crawl is in progress
• Each back queue only contains URLs from a single host• Maintain a table from hosts to back queues
B
1
3…
Back queueHost name
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 136
Back queue heap
• One entry for each back queue• The entry is the earliest time te at which
the host corresponding to the back queue can be hit again
• This earliest time is determined from• Last access to that host• Any time buffer heuristic we choose
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 137
Back queue processing• A crawler thread seeking a URL to crawl:• Extracts the root of the heap• Fetches URL at head of corresponding back queue
q (look up from table)• Checks if queue q is now empty – if so, pulls a
URL v from front queues• If there’s already a back queue for v’s host, append v to q
and pull another URL from front queues, repeat• Else add v to q
• When q is non-empty, create heap entry for it
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 138
Number of back queues B
• Keep all threads busy while respecting politeness
• Mercator recommendation: three times as many back queues as crawler threads
139
Connectivity servers
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 140
Connectivity Server[CS1: Bhar98b, CS2 & 3: Rand01]
• Support for fast queries on the web graph• Which URLs point to a given URL?• Which URLs does a given URL point to?
Stores mappings in memory from• URL to outlinks, URL to inlinks
• Applications• Crawl control• Web graph analysis
• Connectivity, crawl optimization• Link analysis
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 141
Most recent published work
• Boldi and Vigna• http://www2004.org/proceedings/docs/1p595.pdf
• Webgraph – set of algorithms and a java implementation
• Fundamental goal – maintain node adjacency lists in memory• For this, compressing the adjacency lists is the
critical component
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 142
Adjacency lists
• The set of neighbors of a node• Assume each URL represented by an
integer• E.g., for a 4 billion page web, need 32 bits
per node• Naively, this demands 64 bits to represent
each hyperlink
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 143
Adjaceny list compression
• Properties exploited in compression:• Similarity (between lists)• Locality (many links from a page go to
“nearby” pages)• Use gap encodings in sorted lists• Distribution of gap values
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 144
Storage
• Boldi/Vigna get down to an average of ~3 bits/link• (URL to URL edge)• For a 118M node web graph
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 145
Main ideas of Boldi/Vigna• Consider lexicographically ordered list of all URLs,
e.g., • www.stanford.edu/alchemy• www.stanford.edu/biology• www.stanford.edu/biology/plant• www.stanford.edu/biology/plant/copyright• www.stanford.edu/biology/plant/people• www.stanford.edu/chemistry
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 146
Boldi/Vigna• Each of these URLs has an adjacency list• Main thesis: because of templates, the adjacency
list of a node is similar to one of the 7 preceding URLs in the lexicographic ordering
• Express adjacency list in terms of one of these• E.g., consider these adjacency lists
• 1, 2, 4, 8, 16, 32, 64• 1, 4, 9, 16, 25, 36, 49, 64• 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144• 1, 4, 8, 16, 25, 36, 49, 64
Encode as (-2), remove 9, add 8
Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 147
Resources• IIR Chapter 20• http://research.microsoft.com/~najork/mercator.pdf• www.robotstxt.org/wc/norobots.html• http://www.www2004.org/proceedings/docs/1p595.pdf