predicting information seeker satisfaction in community question answering

Predicting information seeker satisfaction incommunity question answering

SIGIR 2008

Yandong Liu, Jiang Bian, Eugene Agichtein from Emory & Georgia Tech University

CQA -Community Question Answering

問答社群近來成為一個尋找資訊的有效方法

IS ASKER IN QA

COMMUNITY SATISFIED?

IS ASKER IN QA

COMMUNITY SATISFIED?發問者是否得到滿意的回答主宰著問答社群的發展或沒落

Asker use case

Lifecycle of a question in CQA

Asker use case

Lifecycle of a question in CQA

DefinitionAn asker in a CQAis considered satisfied iff:

The asker personally has closed the question, selected the best answer, and provided a rating of at least 3 stars for best answer quality.

DefinitionThe Asker Satisfaction Problem:

Given a question submitted by an asker is CQA, predict whether the user will be satisfied with the answers contributed by the community.

Features

Question Question-Answer Relationship Asker User History Answerer User History Category Features Textual Features

Formally a two-class classification problembutprimarily focus on the satisfied class.

Human judgment

Amazon’s paid rater service Mechanical Turk

Datasets

•Random 5000 from top 10000•Skewed distribution•Satisfaction varies by category

Setting

Methods Human Heuristic Baseline

Evaluation Metrics Precision Recall F1 Accuracy

ASP ASP_SVM ASP_RandomForest ASP_C4.5 ASP_Boosting ASP_NaiveBayes

Result

Selected features

Analysis

Online vs. Offline

Analysis

Feature Ablation

Analysis

Textual Features

Analysis

Past Experience

Conclusion

First to quantify and predict asker satisfaction

Shown importance of asker history is this

Our system outperform human assessors