Measuring the Quality of Online

ServiceJin Young Kim

Senior Applied ScientistMicrosoft Web Search and AI

About Jin Young Kim• Data Scientist at Microsoft

• Quantified Self Enthusiast(10 years of happiness tracking)

• Author of ‘Hello, Data Science’(#1 Bestseller in Korea)

Data is the ingredients for all these issues• Data for training and evaluating ML models

• Data for discovering the defect and issues

• Data for monitoring the health of existing service

• Data for measuring the value of new service

Issues in Online Service Development• Planning• How to set business objective & plan?

• Implementation• How to train and improve ML models?

• Evaluation• How much are users satisfied with the service?

Plan

ExecuteEvaluate

How can we collect data for these purposes?

Case Study: Data Collection for Restaurants• Customer Behavior• Facial expression• Quantity of leftovers• Pace of dining

Only limited type of data is available, possibly with lots

of noise

Case Study: Data Collection for Restaurants• Panel Survey• Satisfaction for Food• Satisfaction for Service• Satisfaction for Environment

Survey can provide insights into customer satisfaction, but with

some caveat

Data Collection for Online Service• User Behavior• Various ‘signals’ from behavioral data• Limited type of data is available, with lots of noise• Needs substantial user base required

• Panel Survey• Hire a group of panels, or use crowdsourcing• Collect feedback for all aspects of service quality• Cost of hiring and maintaining panel

Data Collection for Online Service (2)• Direct User Feedback• Request real-time feedback from customers• Typically low response rate, with potential nuisance• Widely used for personalized services (i.e., recommendation)

Panel Survey User Behavior

User Feedback

How does major online service companies collect data for measurement?

Search Engine: Google / Bing• Early stage: panel-based survey• Late stage: user behavior-based experiments• Source: Google

How to evaluate the quality of this SERP?

Social Network: Facebook• Before: use only user behavior• Nowadays: user behavior + panel survey + user feedback• Source: Slate / Quora

We could expose contents users are actually satisfied instead of click-baits by using panel survey and user feedback in addition to signals from user behavior

- Julie Zhuo, Product Design VP at Facebook

User feedback for Facebook News Feed

Recommendation System: Netflix• Combine user feedback and behavior for measurement• Source: Netflix

Movie Recommendations from Netflix

Algorithm A

Algorithm B

Can you tell if algorithm A vs. B is better?Even the users

themselves can’t!!!

Movie Recommendations from Netflix (2)

Results below are more relevant, but users engage more with the above

So, how should I collect data for my service?

• What signals can we extract out of user behavior?• Are there incentives for users to provide feedback?

Service Characteristics

• Do you already have substantial volume of active users?• Can a panel evaluate user experience as a substitute?

Feasibility of Collection

• Do you have marketing budget for building a user base, or for a panel survey?

Cost of Collection

How to evaluate the quality of this SERP?

Evaluation based on user behavior• Which result did users click?• Is click the only measure of satisfaction?

• How long did a user stay on a result?• Is longer dwell-time already better?

• Do users perform search repeatedly?• Does loyalty mean satisfaction?

User behavior is an important clue, but a noisy one.

How can you design a panel survey for SERP evaluation?

How would you evaluate the search results for query ‘crowdsourcing’? Bad Good Excellent Perfect

Q: Who do you think so?

Alternative: Evaluating a Webpage How would you evaluate the search results for query ‘crowdsourcing’? Bad Good Excellent Perfect

Q: Who do you think so?

Alternative: Side-by-Side SERP EvaluationQ: How would you compare two results? Left much better Left slightly better About the same Right slightly better Right much better

Q: Why do you think so?

Conclusions

Summary…• As a first step in data science, plan on collecting high-quality data

• Combine various data collection methods depending on the characteristics and lifecycle of service

• It takes a lot of consideration to get the panel survey done right

For more information…• What you need to know about data even if you’re not a Data Scientist

• SIGIR’2015 Tutorial on Offline Search Evaluation

• Offline Evaluation for Information Retrieval Foundation and Trend in IR Journal (To Appear)