รหัสวิชา 273383 การทำเหมืองข้อมูล เว็ป...
DESCRIPTION
รหัสวิชา 273383 การทำเหมืองข้อมูล เว็ป (Web Mining). สาขาเทคโนโลยีสารสนเทศ คณะเทคโนโลยีสารสนเทศและการสื่อสาร. แนะนำวิชา. คำอธิบายรายวิชา (Course Description) - PowerPoint PPT PresentationTRANSCRIPT
รหั�สวิ�ชา 273383การทำ�าเหัมื�องข้�อมื�ลเวิ�ป
(Web Mining)
สาขาเทคโนโลยี�สารสนเทศคณะเทคโนโลยี�สารสนเทศและการส��อสาร
ค�าอธิ�บายีรายีวิ�ชา (Course Description)
หล�กการเบ��องต้ นการท�าเหมื�องข อมื"ล สถาปั%ต้ยีกรรมื ของเวิ&ปั สถาปั%ต้ยีกรรมืของโปัรแกรมืค นหา การท�า
เหมื�องโครงสร างเวิ&บและการวิ�เคราะห'เช��อมืโยีง เท คน�คครอวิ'ล�ง การค นหาและการท�าดรรชน� การท�า
เหมื�องเวิ&บคอนเท&น การวิ�เคราะห'แฟ้*มืลงบ�นท+กเข า ออกบนเซิ�ร'ฟ้เวิอร' และการปัระยี-กต้'ใช
Web mining concept, Web architecture, Searching program architecture, Web structure mining, Link analysis, crawling technique, Indexing and searching, web mining content, Server logs, and implementation
Web Mining2
แนะน�าวิ�ชา
ดร.ส-ขชาต้ร� ปัระสมืส-ข (Sukchatri PRASOMSUK, Ph.D.) PhD. (Computational Linguistics), INALCO, Paris, FR M.Eng.(IT), & Grad.Dip. In Applied IS, RMIT, Melbourne, AU B.Sc.(คณิ�ตศาสตร�), มื.รามืค�าแหัง เคยทำ�างานทำ� :
DATA SOLVE Co.,Ltd. & Central Trading Co.,Ltd กรมืวิ�ทำยาศาสตร�บร�การ กระทำรวิงวิ�ทำย�ฯ มื.หัอการค�า และ มื.แมื%ฟ้'าหัลวิง
Contact : [email protected] FaceBook : Sukchatri PSK 0804509105
Notes/Slides Download: http://www.ict.up.ac.th/skchatri/
Web Mining3
แนะน�าอาจารย�และน�ส�ต
เกณฑ์'การวิ�ดผลและปัระเมื�นผล (Evaluation criteria) เกณฑ์'การวิ�ดผล 1.1. ภาคบรรยีายี รวิมื 70% - การสอบกลางภาค/ปลายภาค 35% + 35% 1.2. ภาคทดลอง/ภาคปัฏิ�บ�ต้�การ รวิมื 30%
การเข้�าเร�ยน 10 %
จ�ตพิ�ส�ยและงานทำ�มือบหัมืาย 10 %
LAB 20 %
รวิมืท��งหมืด 100 % เกณฑ์'การต้�ดเกรด
ใช�เกณิฑ์�การต�ดเกรดแบบอ�งเกณิฑ์�/อ�งกล.%มื อ�งเกณิฑ์� < 50 คะแนน ได�เกรด F
Web Mining4
การประเมื�นผลการเร�ยน
ระบบการจ�ดเก�บและการส�บค�นสารสนเทำศด�วิยคอมืพิ�วิเตอร�, พิ�ทำ�กษ์�การพิ�มืพิ�, พิ�มืพิ�คร�2งทำ� 2, ดร.ศ.ภช�ย ต�2งวิงศ�ศานต�, 2553
An Introduction to Information Retrieval, Online edition (c)2009 Cambridge UP, Draft of April 1, 2009
Data Mining, Practical Machine Learning Tools and Techniques, Third Edition, Ian H. Witten Eibe Frank Mark A. Hall, Elsevier, 2011
Web Mining and Social Networking, Techniques and Applications, Series Editor: Yanchun Zhang, Victoria University, Australia, Springer, 2011
Web mining : applications and techniques / Anthony Scime, Editor, State University of New York College at Brockport, USA, Idea Group Inc., 2005
Web Mining5
เอกสารปัระกอบการสอน(Teaching Materials / References)
การต�ดต%อได�ทำ�2งทำาง e-mail หัร�อ โทำรศ�พิทำ� หัร�อเข้�าพิบทำ� หั�องพิ�ก
E-mail : แจ�ง ช� อ- นามืสก.ล รหั�ส เร� องทำ� จะ ต�ดต%อ ค�าข้อบค.ณิป4ดทำ�าย
Mobile Phone : แจ�ง ช� อ- นามืสก.ล น�ส�ตวิ�ชา เร� องทำ� จะต�ดต%อ ค�าข้อบค.ณิเมื� อพิ�ดธุ.ระเสร�จ
การเข้�าพิบทำ� หั�องพิ�กอาจารย� : แจ�งการเข้�าพิบโดย การน�ดทำาง e-mail หัร�อ โทำรมืาน�ด วิ�นและเวิลา
ก%อนการเข้�าพิบเตร�ยมืเร� องข้องตนเองใหั�พิร�อมื การเตร�ยมืและล�าด�บค�าถามืใหั�พิร�อมื พิ�ดภาษ์าไทำย
ใหั�ช�ดเจนWeb Mining6
มืารยาทำในการต�ดต%ออาจารย�
ป4ดมื�อถ�อ อยี4า- ค.ยก�บเพิ� อน ส%งเส�ยงด�ง ค.ยโทำรศ�พิทำ� เล%นมื�อถ�อ หัร�อคอมืพิ�วิเตอร� หัร�อ อาการทำ� ไร�มืารยาทำในหั�องบรรยายเช%น
…นอนหัล�บ เตร�ยมืต�วิเสมือ พิร�อมืทำ� จะตอบค�าถามื หัร�อถามื (โดยการยกมื�อ) ไมื%ก�นน�2าหัร�ออาหัารข้ณิะมื�บรรยาย ไมื%ล.กออกไปจากหั�อง โดยไร�มืารยาทำ (อาจารย�มื�เวิลาพิ�กใหั�เป7นระ
ยะๆ) ข้ณิะเพิ� อนหัร�ออาจารย�ก�าล�งบรรยายหัร�อพิร�เซนต� ควิรใหั�เกร�ยต�
อาจารย�หัร�อเพิ� อน โดยการต�2งใจฟ้:งอย%างมื�มืารยาทำ ต�2งใจเร�ยนด�วิยควิามืสนใจ (อาจารย�จะด�พิฤต�กรรมืข้องน�ส�ต
ตลอดเพิ� อหั�กคะแนน) เพิราะเวิลาเร�ยนเป7นเวิลาทำ� มื�ค%าอย%างส�ง
Web Mining7
มืารยาทำในหั�องเร�ยนข้ณิะบรรยายหัร�อเร�ยน
ข้ณิะเร�ยน เมื� ออาจารย�ถามื ตอบได�เสมือ ไมื%ไปถามืเพิ� อนข้�างๆ มื�ควิามืประพิฤต�ในหั�องเร�ยนด�ตลอด เข้�าเร�ยนสมื� าเสมือ ควิามื
ต�2งใจด� ส%งงานทำ� มือบหัมืายอย%างสมื� าเสมือ จ�าค�าศ�พิทำ�ทำ� เป7นภาษ์าอ�งกฤษ์ประกอบด�วิย เพิ� อประโยชน�ในการ
สอบ เวิลาสอบ การทำ�าข้�อสอบ : กรณิ�เป7นข้�อเข้�ยนหัร�ออ�ตน�ย (เป7น
เทำคน�คในการตอบค�าถามื ใช�ได�ก�บทำ.กวิ�ชา ทำ� วิโลก) เข้�ยนหัร�ออธุ�บายใหั�ได�ใจควิามืมืากทำ� ส.ด (ถ�ามื�ศ�พิทำ�เทำคน�คเป7นภาษ์า
อ�งกฤษ์ต�องเข้�ยนก�าก�บมืาด�วิย) ยกต�วิอย%าง (ถ�ามื�) เข้�ยนภาพิหัร�อแผนผ�งประกอบ (ถ�ามื�)
Web Mining8
วิ�ธุ�และเทำคน�คในการทำ�าและร�กษ์าคะแนน
An introduction to Web Mining
Source : Bettina Berendt, K.U. Leuven, Belgium, : www.berendt.de
Review/Present by Dr.Sukchatri Prasomsuk,
IT. ICT, University of Phayao, ThailandWeb Mining9
Web mining - is the application of data mining techniques to discover patterns from the Web. According to analysis targets, web mining can be divided into three different types, which are Web usage mining, Web content mining and Web structure mining.
Web Mining10
Definition
Web usage mining Web structure mining Web content mining
Web Mining11
Main topic of web mining
Web Mining12
Web mining structure
Web Mining13
Web mining structure
Using of Web Mining
Web Mining is the use of the data mining techniques to automatically discover and extract information from web documents/services
Discovering useful information from the World-Wide Web and its usage patterns
Using data mining techniques to make the web more useful and more profitable (for some) and to increase the efficiency of our interaction with the web
Web Mining14
Web Mining
Data Mining Techniques Association rules Sequential patterns Classification Clustering Outlier discovery
Applications to the Web E-commerce Information retrieval (search) Network management
Web Mining15
Web Mining
The WWW is huge, widely distributed, global information service centre for Information services: news, advertisements,
consumer information, financial management, education, government, e-commerce, etc.
Hyper-link information Access and usage information
WWW provides rich sources of data for data mining
Web Mining16
Why Mine the Web? Enormous wealth of information on Web
Financial information (e.g. stock quotes) Book/CD/Video stores (e.g. Amazon) Restaurant information (e.g. Zagats) Car prices (e.g. Carpoint)
Lots of data on user access patterns Web logs contain sequence of URLs accessed by
users
Possible to mine interesting nuggets of information People who ski also travel frequently to Europe Tech stocks have corrections in the summer and rally
from November until FebruaryWeb Mining17
Why is Web Mining Different?
The Web is a huge collection of documents except for Hyper-link information Access and usage information
The Web is very dynamic New pages are constantly being generated
Challenge: Develop new Web mining algorithms and adapt traditional data mining algorithms to Exploit hyper-links and access patterns Be incremental Web Mining18
Web Mining Applications
E-commerce (Infrastructure) Generate user profiles Targetted advertizing Fraud Similar image retrieval
Information retrieval (Search) on the Web Automated generation of topic hierarchies Web knowledge bases Extraction of schema for XML documents
Network Management Performance management Fault management
Web Mining19
Problems with Web Search Today
Today’s search engines are plagued by problems: the abundance problem (99% of info of no
interest to 99% of people) limited coverage of the Web (internet sources
hidden behind search interfaces)Largest crawlers cover < 18% of all web pages
limited query interface based on keyword-oriented search
limited customization to individual users
Web Mining20
Problems with Web Search Today
Today’s search engines are plagued by problems: Web is highly dynamic
Lot of pages added, removed, and updated every day
Very high dimensionality
Web Mining21
Improve Search By Adding Structure to the Web
Use Web directories (or topic hierarchies) Provide a hierarchical classification of documents (e.g., Yahoo!)
Searches performed in the context of a topic restricts the search to only a subset of web pages related to the topic
Recreation Science Business News
Yahoo home page
SportsTravel Companies Finance Jobs
Web Mining22
Network Management Objective: To deliver content to users quickly
and reliably Traffic management Fault management
Service Provider NetworkRouter
Server
Web Mining23
Why is Traffic Management Important?
While annual bandwidth demand is increasing ten-fold on average, annual bandwidth supply is rising only by a factor of three
Result is frequent congestion at servers and on network links
during a major event (e.g., princess diana’s death), an overwhelming number of user requests can result in millions of redundant copies of data flowing back and forth across the world
Olympic sites during the games NASA sites close to launch and landing of shuttles
Web Mining24
Traffic Management
Key Ideas
Dynamically replicate/cache content at multiple sites within the network and closer to the user
Multiple paths between any pair of sites
Route user requests to server closest to the user or least loaded server
Use path with least congested network links
Akamai, Inktomi
Web Mining25
Traffic Management Need to mine network and Web traffic to determine
What content to replicate? Which servers should store replicas? Which server to route a user request?
What path to use to route packets?
Network Design issues Where to place servers? Where to place routers? Which routers should be connected by links?
One can use association rules, sequential pattern mining algorithms to cache/prefetch replicas at server
Web Mining26
Web Mining Issues
Size Grows at about 1 million pages a day Google indexes 9 billion documents Number of web sites
Netcraft survey says 72 million sites(http://news.netcraft.com/archives/web_server_survey.html)
Diverse types of data Images Text Audio/video XML HTML
Web Mining27
Number of Active Sites
Total Sites Across All Domains August 1995 - October 2007Web Mining28
Systems Issues Web data sets can be very large
Tens to hundreds of terabytes Cannot mine on a single server!
Need large farms of servers How to organize hardware/software
to mine multi-terabye data sets Without breaking the bank!
Web Mining29
Different Data Formats Structured Data Unstructured Data OLE DB (Object Linking and Embedding,
Database) offers some solutions!
Web Mining30
Web Data
Web pages Intra-page structures Inter-page structures Usage data Supplemental data
Profiles ข้�อมื�ลรายละเอ�ยด Registration information ข้�อมื�ลการลง
ทำะเบ�ยน Cookies
Web Mining31
Web Usage Mining
Pages contain information Links are ‘roads’ How do people navigate the
Internet Web Usage Mining (clickstream
analysis) Information on navigation paths
available in log files Logs can be mined from a client or
a server perspectiveWeb Mining32
Website Usage Analysis
Why analyze Website usage? Knowledge about how visitors use Website could
Provide guidelines to web site reorganization; Help prevent disorientation
Help designers place important information where the visitors look for it
Pre-fetching and caching web pages การด<งข้�อมื�ลล%วิงหัน�าและแคชหัน�าเวิ�บ
Provide adaptive Website (Personalization) ใหั�เวิ�บไซต�ทำ� ปร�บแต%งได�
Questions which could be answered What are the differences in usage and access patterns among
users? What user behaviors change over time? How usage patterns change with quality of service (slow/fast)? What is the distribution of network traffic over time?
Web Mining33
Website Usage Analysis
Analog – Web Log File AnalyserGives basic statistics such as
• number of hits• average hits per time period• what are the popular pages in your site• who is visiting your site• what keywords are users searching for to get to you• what is being downloaded
http://www.analog.cx/
Web Mining34
Web Usage Mining Process
Web Mining35
Web Mining Outline
Goal: Examine the use of data mining on the World Wide Web
Web Content Mining Web Structure Mining Web Usage Mining
Web Mining36
Web Mining Taxonomy
Modified from [zai01]
Web Mining37
Web Content Mining Examine the contents of web pages as well as
result of web searching Can be thought of as extending the work
performed by basic search engines Search engines have crawlers to search the
web and gather information, indexing techniques to store the information, and query processing support to provide information to the users
Web Content Mining is: the process of extracting knowledge from web contents
Web Mining38
Semi-structured Data
Content is, in general, semi-structured Example:
Title Author Publication_Date Length Category Abstract Content
Web Mining39
Structuring Textual Data
Many methods designed to analyze structured data
If we can represent documents by a set of attributes we will be able to use existing data mining methods
How to represent a document? Vector based representation
(referred to as “bag of words” as it is invariant to permutations)
Use statistics to add a numerical dimension to unstructured text
Web Mining40
Document Representation
A document representation aims to capture what the document is about
One possible approach: Each entry describes a document Attribute describe whether or not a term appears in
the document
Web Mining41
Document Representation
Another approach:• Each entry describes a document• Attributes represent the frequency in which a term appears in the
document
Web Mining42
Document Representation
• Stop Word removal: Many words are not informative and thus
irrelevant for document representationthe, and, a, an, is, of, that, …
• Stemming: reducing words to their root form (Reduce dimensionality)
A document may contain several occurrences of words like fish, fishes, fisher, and fishers. But would not be retrieved by a query with the keyword “fishing”Different words share the same word stem and should be represented with its stem, instead of the actual word “Fish”
Web Mining43
Web structure mining is the process of using graph theory to analyze the node and connection structure of a web site. According to the type of web structural data, web structure mining can be divided a into two kinds:
1. Extracting patterns from hyperlinks in the web: a hyperlink is a structural component that connects the web page to a different location.
2. Mining the document structure: analysis of the tree-like structure of page structures to describe HTML or XML tag usage.
Web Mining44
Web Structure Mining
Web usage mining is the process of extracting useful information from server logs e.g. users' history.
Web usage mining is the process of finding out what users are looking for on the Internet.
Web Server Data: The user logs are collected by the Web server.
Application Server Data: Commercial application servers have significant features to enable e-commerce applications to be built on top of them with little effort.
Application Level Data: New kinds of events can be defined in an application, and logging can be turned on for them thus generating histories of these specially defined events.
Web Mining45
Web Usage Mining
The start point:Key notions of Information Retrieval Representation, storage, organization of, and access to
information items Focus is on the user information need
User information need example:Find all docs containing information on college tennis teams which: (1) are maintained by a USA university and (2) participate in the NCAA tournament.
Information retrieval information about a subject or topic semantics is frequently loose small errors are tolerated
IR system: interpret contents of information items generate a ranking which reflects relevance notion of relevance is most important
Retrieval
Browsing
Database
[from Berthier Ribeiro-Neto’s slides for the Baeza/Ribeiro-Neto IR book]
Web Mining46
IR and KD
Information Retrieval (IR)
Knowledge Discovery * (KD)
*(better term for data mining)
Web Mining47
IR and KD: Different ways of utilizing databases (DBs) IR: „retrieving the information from a DB that matches a user‘s information
need“
query (formal statement of information need) object (an entity which stores information in a database)
KD: „finding new knowledge about the real-world entities described in a DB“
data/information (sometimes plus query) patterns („knowledge“)
Web Mining48
IR and KD: confluences
Conceptually: IR can be seen as a classification of objects
to the classes „relevant to the user‘s query“ / „not relevant to the user‘s query“
(and classification is a typical KD task) KD needs to extract the information from
objects like documents, in order to find new knowledge
(and information extraction is a typical IR task)
Pragmatically: e.g. overlaps of topics and techniques in
papers at SIGIR, SIGKDDWeb Mining49
Web Mining Knowledge discovery
(aka Data mining):“the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data.” 1
Web Mining: the application of data mining techniques on the content, (hyperlink) structure, and usage of Web resources. Web mining areas:
Web content mining
Web structure mining
Web usage mining
1 Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., & Uthurusamy, R. (Eds.) (1996). Advances in Knowledge Discovery and Data Mining. Boston, MA: AAAI/MIT Press
Navigation, queries, content access & creation
Web Mining50
What‘s different about Web mining (different from data mining in general)?
The data and the necessary data preparation steps
To some extent, the applicable techniques
Web Mining51
Other application areas of knowledge discovery
Customer relationship management Finance Banking / Credit Scoring and Insurance Healthcare Bioinformatics (e.g., genomics) Impact of genetically modified
organisms Texts (on- or offline) ...
Web Mining52
The process part of knowledge discovery
CRISP-DM • CRoss Industry Standard Process for Data Mining• a data mining process model that describes commonly used
approaches that expert data miners use to tackle problems.Web Mining53
The structural/algorithmic part of knowledge discovery (“modelling“ in CRISP-DM): Patterns, data mining tasks, methods (examples)
Global patterns Description
Clustering K-means, EM, hierarchical clustering, ...
Hidden Markov Models Link patterns (e.g., ciation analysis à la Google)
Prediction Classification
Bayes techniques, Decision trees, Support Vector Machines, ...
Regression Time series analysis
Local patterns Frequent itemsets, sequences, subgraphs
» A priori and methods derived from it Association rules Cliques (“Web Communities“)
Web Mining54
Recall: from clustering to ontology learning
Web Mining55
http://www.cs.washington.edu/research/textrunner/
Web Mining56
http://quest.sandbox.yahoo.net
Web Mining57
New material: usageOther examples
E-commerce questions How do people utilize (or not) service
options? Which advertising campaign brings the
most Visitors Customers ? , Google Analytics
E-commerce / information systems questions
What do queries tell us about which content we should inform about?
Search-engine questions How can click-through behaviour aid
relevance assessments re-ranking (“learning to rank“) query recommendation
Personalization (based on explicit or implicit features – e.g. gender prediction)
Web Mining58
Example: Google Analytics Advertising ROI
Visualize the Conversion Funnel
Cross Channel and Multimedia Tracking, Benchmarking
Customized Reporting (define your own metrics)
Web Mining59
Who is this?(Sample from a search-query log)
Web Mining60
Result(a 1-identified person)
[M. Barbaro and T. Zeller. A face is exposed for AOL Searcher No. 4417749. New York Times, 9 August 2006]
Web Mining61
Is this the same person?
Web Mining62
Thanks ! – Questions ?
Web Mining64
Further reading : An excellent textbook introduction
Web Mining65