Am Introduction to Recommendation Systems

Hasan Davulcu

CIPS, ASU

Recommendation Systems

U: C X S RC: profile: age, gender, incomeS: title, director, actor, year, genre

Recommendations:

Recommender Systems

Content Based

Limitations Too Similar ! New user problem

User - Collaborative Methods U(C,S) is estimated

based on utilities U(Cj,S) by those users Cj who are “similar” to user C.

Limitations New Item Problem Sparsity

Amazon’s Item-to-Item

Comparing Human Recommenders to Online

Systems

Rashmi Sinha & Kirsten Swearingen

SIMS, UC Berkeley

Recommender Systems are technological proxy for a social process

Which one should I read?

Recommendations from friends

Recommendations from Online

Systems

I know what you’ll read next summer (Amazon, Barnes&Noble)

what movies you should watch… (Reel, RatingZone, Amazon)

what music you should listen to… (CDNow, Mubu, Gigabeat)

what websites you should visit (Alexa)

what jokes you will like (Jester)

& who you should date (Yenta)

Method Philosophy: Testing & Analysis as part of the Iterative Design Process

Design

Evaluate

Analyze

Slide adapted from James Landay

Use both quantitative & qualitative methods

Generate Design Recommendations

Taking a closer look at the Recommendation Process

Input User incurs cost in using system:

Time, Effort, Privacy Issues

Receives Recommendation

Cost in reviewing recommendations

Benefit only if recommended item appeals

Judges if he/she will sample recommendation

Amazon’s Recommendation Process

Input: One artist/author name

Output: List of Recommendations Explore / Refine Recommendations

Search usingRecommendations

Book Recommendation Site: Sleeper

Input: Ratings of 10 books for all users Use of continuous Rating Bar

(System designed by Ken Goldberg)

Output: List of items with brief information about each item

Degree of confidence in prediction

Sleeper: Output

What convinces a user to sample the recommendation

Judging recommendations: What is a good recommendation from the

user’s perspective?

Trust in a Recommender System: What factors lead to trust in a system?

System Transparency: Do users need to know why an item was

recommended?

Study of RS has focused mostly on Collaborative Filtering Algorithms

Social Recommendations

Collaborative Filtering Algorithms

Output (Recommendations)

Input from user

Beyond “Algorithms Only” : An HCI Perspective on Recommender Systems

Comparing the Social Recommendation Process to Online Recommender Systems

Understanding the factors that go into an effective recommendation (by studying users interaction with 6 online RS)

The Human vs. Recommenders Death Match

Book Systems

Amazon Books

Sleeper

Rating Zone

Movie Systems

Amazon Movies

Reel

Movie Critic

Method

For each of 3 online systems: Registered at site Rated items Reviewed and evaluated recommendation set Completed questionnaire

Also reviewed and evaluated sets of recommendations from 3 friends each

19 participants, age:18 to 34 years

Results

Defining Types of Recommendations

Good Recs. (Precision)•% items user felt interested in

Useful Recs.•Subset of Good Recs.•User felt interested in and had not read / viewed yet

USEFULNot yet read/viewed

GOOD: User likes

0

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Amazon(15)

Sleeper(10)

RatingZone (8)

Friends(9)

Amazon(15)

Reel(5-10)

MovieCritic (20)

Friends(9)

Books Movies

% Good Recommendations

% Useful Recommendations

Comparing Human Recommenders to RS: “Good” and “Useful” Recommendations

RS AverageAve. Std. Error (x) No. of Recommendations

However users like online RS

This result was supported by post test interviews.

Do you prefer recommendations from friends or online systems?

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

Books Movies

Num

ber

pref

erri

ng s

yste

m

System

Friends

Why systems over friends? “Suggested a number of things I hadn’t heard of,

interesting matches.”

“It was like going to Cody’s—looking at that table up front for new and interesting books.”

“Systems can pull from a large database—no one person knows about all the movies I might like.”

Items users had “Heard of” before

Friends recommended mostly “old” previously experienced items

% Heard of

0

10

20

30

40

50

60

70

Amazon Sleeper RatingZone Friends Amazon MovieCritic Reel Friends

MoviesBooks

What systems did users prefer?

Sleeper and Amazon books average highest ratings Split opinions on Reel, MovieCritic

Would you use system again?

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Amazon Sleeper RatingZone Amazon Reel MovieCritic

Av

era

ge

Rat

ing

No

Yes

MoviesBooks

Why did some systems…

Provide useful recommendations but leave users unsatisfied? RatingZone, MovieCritic & Reel

Possible Reasons

Previously Enjoyed Items are important: We term these Trust-Generating Items

Adequate Item Description & Ease of Use are important Missing from List: Time to Receive Recommendations &

No. of Items to Rate not important!

What predicts overall usefulness of a System?

0

0.1

0.2

0.3

0.4

0.5

0.6

Good Rec. Useful Rec. TrustGenerating

Rec.

AdequateItem

Description

Ease ofUse

Co

rre

lati

on

All correlations are significant at .05

USEFULNot yet read/viewed

TRUST-GENERATINGPreviously read/viewed

A Question of Trust…

GOOD: User likes Post Test Interviews showed that users “trust” systems if they have already sampled some recommendations

•Positive Experiences lead to “trust

•Negative Experiences with Recommended Items lead to mistrust of system

Difference between Amazon and Sleeper highlights the fact that there are different kinds of good Recommender Systems

A Question of Trust …Number of Trust Generating Items

.00

1.00

2.00

3.00

4.00

5.00

6.00

Amazon Sleeper RatingZone Amazon Reel MovieCritic

Num

ber

of It

ems

Books Movies

Adequate Item Description: The RatingZone Story

0 % of Version 1 and 60% of Version 2 users found item description adequate

% Useful For Both Versions of RatingZOne

0

5

10

15

20

25

30

35

40

45

Version 1: Without Description Version 2: With Description

% U

sefu

l R

ecs

.

An adequate item description, and links to other sources about item was a crucial factor in users being convinced by a recommendation.

System Transparency Why was this item recommended?

Do users understand why an item was recommended

Users mentioned this factor in post test interviews

Effect of System Transparency on Recommendation

0

10

20

30

40

50

60

System Reasoning wasTransparent

System Reasoning NotTransparent

% G

oo

d R

eco

mm

en

da

tio

ns

Discussion & Design Recommendations

Design Recommendations: Justification

Justify your Recommendations

Adequate Item Information: Providing enough detail about item for user to make choice

System Transparency: Generate (at least some) recommendations which are clearly linked to the rated items

Explanation: Provide an Explanation, why the item was recommended.

Community Ratings: Provide link to ratings / reviews by other users. If possible, present numerical summary of ratings.

Design Recommendations:Accuracy vs. Less Input

Don’t sacrifice accuracy for the sake of generating quick recommendations. Users don’t mind rating more items to receive quality recommendations.

A possible way to achieve this: have multilevel recommendations. Users can initially use the system by providing one rating, and are offered subsequent opportunities to refine recommendation

One needs a happy medium between too little input (leading to low accuracy) and too much input (leading to user impatience)

Design Recommendations: New Unexpected Items

Users like Rec. Systems as they provide information about new, unexpected items.

List of recommended items should include new items which the user might not find out in any other way.

List could also include some unexpected items (e.g., from other topics / genres) which the user might not have thought of themselves.

Design Recommendations: New Unexpected Items

Users (especially first time users) need to develop trust in the system.

Trust in system is enhanced by the presence of items that the user has already enjoyed.

Generating some very popular (which have probably been experienced previously) in the initial recommendation set might be one way to achieve this.

Design Recommendations: Trust Generating Items

Systems need to provide a mix of different kinds of items to cater to different users:

Trust Generating Items: A few very popular ones, which the system has high confidence in

Unexpected Items: Some unexpected items, whose purpose is to allow users to broaden horizons.

Transparent Items: At least some items for which the user can see the clear link between the items he /she rated and the recommendation.

New Items: Some items which are new.

Design Recommendations: Mix of Items

Question: Should these be presented as a sorted list / unsorted list/ different categories of recommendations?

Allow users to provide ratings on a continuous scale. One of the reasons users liked Sleeper was

because it allowed them to rate on a continuous scale. Users did not like binary scales.

Design Recommendations: Continuous Scales for Input

Limitations of Study Simulated first-time visit, did not allow system to

learn user preferences over time

Source of recommendations known to subjects—might have biased towards friends

Fairly homogenous group of subjects, no novice users

Future Plans: Second Generation Music Recommender Systems

•Have evolved beyond previous systems

•Use a variety of sophisticated algorithms to map users preferences over music domain

•Require a lot more input from the user

•Users can sample recommendations during the study!

MusicBudha (Mubu.com): Exploring Genres

Mubu.com: Exploring Jazz Styles

Mubu.com: Rating Samples

Mubu.com: Recommendations as Audio Samples

Compare systems, friends and experts Anonymize the source of recommendation

The Turing Test for Music Recommender Systems

Study Design

Goal: Compare recommendations by online RS, Experts (who have same information as RS) & Friends

Music Experts

Friends

Online RS

So far we have heard what this study tells us about Recommender Systems

But what (if anything) does it have to say about human nature?

In conclusion…

Recommender Systems tantalize us with the idea that we are not as unique and unpredictable as we think we are.

Study results show that Recommender Systems do not know us better than our friends!

But …

11 out of 19 users preferred Recommender Systems over Friends Recommendations!

Ultimately, we all want to be tables in a database!

Email: sinha@sims.berkeley.edu

Web address: http://sims.berkeley.edu/~sinha