presenter: lung-hao lee ( 李龍豪 ) january 7, 2010@room 309

Presenter: Lung-Hao Lee (李龍豪 )January 7, 2010@Room 309

Introduction Calculating Page Similarity Finding Similar Pages

◦ Click Data Model (CDM)◦ Query Constraint (QC) algorithm

Experimental Results Discussion Conclusion

2

Large labor cost of annotating the data

The aggregated click data across many users over time provides valuable information

Leveraging click logs to argument training data by propagating class labels to unlabeled similar documents

3

“Two pages that tend to be clicked by the same user queries tend to be topically similar”

4

A B

“How to tie a tie”

“How to tie a tie”

“How to tie a neck tie knots ”

“How to tie a neck tie knots ”

“Tying a tie”“Tying a tie”

Label as “Positive” (class “How-to”)

Unknown Label“Positive” ?

A page is represented as a node in the similar graph

Normalize all the URLse.g. the following 4 URLs are treated as the same(1)“http://www.acm.org”(2)“www.acm.org”(3)“www.acm.org/”(4)“http://www.acm.org/”

5

Each URL is represented as a vector of queries that users issued and clicked through to the page

6

Pantel & Lin (2002)

Compute the similarity between two pages using the cosine similarity of their respective feature vector

sim (p1,p2) > sim (p1,p3) sim (p1,p2) > sim(p2,p3)Because p1 and p2 share more common queries than p3

7

What’s a “seed set” ?A set of some labeled data

Two algorithms for seed set expansion◦ Click Data Model (CDM)◦ Query Constraints (QC) algorithm

8

Two phases◦ Updating score phase◦ Filtering phase

Input◦ S1 (positive set) ◦ S2 (negative set)◦ G (click graph)

Output◦ E1 (positive)◦ E2 (negative)

Thresholds◦ 0.1<T1<0.6

◦ 0.6<T2<1.2 9

Additional Module that checks whether the common queries between two nodes have certain term patterns

10

Reduce the amount of human annotation effort by leveraging the click data

Build an expansion model with labeled training data and use it to select next round of training data

11

Click Data◦ During December 2008 from Yahoo! Search

engine◦ Only the top 10 URLs are considered◦ URLs with less than 10 clicks are excluded

Tree classification tasks◦ How-to◦ Adult◦ review

12

Training sets◦ 10,000 manually labeled positive and negative examples◦ For “review” classifier, queries such as “digital camera

reviews” or “baby swing reviews”◦ For “How-to” classifier, queries such as “how to clean

uggs” or “best way to loose weight”

Testing sets

13

Classifier◦ Gradient Boosting Decision Tree (GBDT)

Features◦ Textual, Link, URL, HTML, Other features

Metrics◦ Area Under the ROC Curve (AUC) (Fawcett, 2003)◦ F score◦ Accuracy

14

The big improvement of CDM is observed with a model using 5000 labeled data as a seed set (+1.07% in F-score, +0.81 in Accuracy and +0.25% in AUC)

15

Reduce the manual labor by 50%

QC (exclude pages that do not have “review” in query terms) is useful when labeled data is small

16

With 1000 and 2000 human labeled data, CDM performs worse than the baseline

QC (exclude pages that do not have “How-to” in query terms)

17

Baseline: Type A

CDM: Type C

18

From “How-to” Classifier Seed 1Seed 2 (human label from Expnd1)

Expand2

19

A random sample of 50 positive and 50 negative example from “how-to” classifier

Positive class has 82.3% precision whereas negative class has 83.6% precision

20

Is the proposed method always useful for web page classification ?

How can we improve the quality of automatically labeled data from unlabeled data ?

21

Present a method for improve webpage classification by leveraging click data to augment training data

Argument manually labeled data by modeling the similarity between pages in a click graph

22

Thank you very much Questions & Answers

23