evaluation of cross-domain news article recommendations
Post on 15-Jan-2015
853 Views
Preview:
DESCRIPTION
TRANSCRIPT
Competence Center Information Retrieval & Machine Learning
10. April 2023
UMAP‘13 Doctoral Consortium
Evaluation of Cross-Domain News Article Recommendations
Benjamin Kille
210. April 2023
Agenda
► Problem description► Challenges in News Article Recommendation
Sparsity Dynamic item collection Evaluation
► Research Questions► Data outline► Preliminary results► Conclusions► Next steps
310. April 2023
Problem description
► Information overload amount of on-line accessible news articles increases limited user perception limited time capacity
► Solution: Recommender System filtering news articles with respect to relevance/utility
► Special challenges for news recommender systems Sparsity Dynamics
► General challenges for recommender systems Evaluation strategy
410. April 2023
Problem formalization
► Set of domains each referring to a publisher► Each domain comprises a set of users , along with a set of items ► We observe preferences as users interact with items ► Let denote the set of items the target user interacted with, and
denote the set of users who interacted with a specific item► We define the recommendation task as
Rank the set of items according to estimated relevance/utility
510. April 2023
Sparsity
► Cold-Start Problem ( and/or )Cacheda, F. et al., 2011. Comparison of collaborative filtering algorithms. ACM Transactions on the Web, 5(1), pp.1–33.► Providing preferences better recommendations (trade-off)Cremonesi, P., Milano, P. & Turrin, R., 2012. User Effort vs . Accuracy in Rating-based Elicitation. In 6th ACM Conferene on Recommender Systems. pp. 27–34.► News websites usually avoid log-in requirements to attract
larger user segments Incomplete user profiles Inconsistent user profiles
610. April 2023
Dynamics
► News dynamic contentBillsus, D. & Pazzani, M.J., 2007. Adaptive News Access. In P. Brusilovsky, A. Kobsa, & W. Nejdl, eds. The Adaptive Web. Springer, pp. 550–570.► Unlike music or movies rarely re-consumed► For instance: Deutsche Presse Agentur (DPA)
750 messages 220k words 1,5k images
http://www.dpa.de/Zahlen-Fakten.152.0.html
710. April 2023
Evaluation
► Strategy on-line: A/B testing (user-centric) off-line: data set (data-centric)
► Numerous facets utility relevance novelty serendipity …
► Dependending on the model formulation preference prediction (requires numerical preference data) item ranking
810. April 2023
Evaluation (cont‘d)
► on-line
Dispatcher
● recommendation request click
●
●
●
●
●
●
CTR = # clicks# requests
910. April 2023
Evaluation (cont‘d)
► off-line (replay)
Dispatcher
● recommendation request click
● ● ● ● ●●
CTR = # clicks# requests
[𝑖1 ,𝑖2 , 𝑖3 ] [𝑖1 ,𝑖2 , 𝑖3 ]
? ?
Li, L. et al., 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining - WSDM ’11. p. 297.
1010. April 2023
Cross-domain setting
D1
D2
U
U
I
ID1
D2
U
U
I
I
D1
D2
U
U
I
I
D1
D2
U
U
I
I
no o
verla
p
Use
r ove
rlap
Item
ove
rlap
full
over
lap
Cremonesi, P., Tripodi, A. & Turrin, R., 2011. Cross-Domain Recommender Systems. In 2011 IEEE 11th International Conference on Data Mining Workshops. IEEE, pp. 496–503.
1110. April 2023
Research Questions
► How can other publishers' user interactions contribute to
decrease sparsity for the target publisher?
► What characteristics must recommender algorithms exhibit to
successfully cope with dynamically changing item collections?
► How to evaluate cross-domain recommender systems with
dynamically changing item collections? How do standard
evaluation metrics compare to the observed clicks?
1210. April 2023
Data outline
► > 1-2M impressions by 12 publishers (general news, local news, finance, information technology, sports, etc.) on a daily basis
► user features such as browser ISP OS device
► news article features such as title text URL Image
► http://www.dai-labor.de/en/irml/epen/
►Real interactions with actual users!
1310. April 2023
Preliminary results
► Sparsity► Histogram of the relative frequency of user interactions
1410. April 2023
Preliminary results (cont‘d)
► Dynamics
1510. April 2023
Preliminary results (cont‘d)
► Popularity
1610. April 2023
Conclusions
► News recommender systems must handle enormous sparsity► Dynamic item collections
Continuously, news items enter the system Continuously, items stop to be relevant/read
► We observe a serious popularity bias► We suppose that considering user preferences on additional
domains will decrease sparsity ► We suppose that identifying general preference pattern will
allow us to deal with item collection dynamics
1710. April 2023
Next steps
► Implementation of existing cross-domain recommender algorithms
► Evaluating recommender algorithms with respect to CTR novelty diversity
► Investigate UI effects► Analyze applicability of context-sensitive recommendations► User/Item clustering to speed-up computation time
1810. April 2023
Thank you for the attention!
Questions???
1910. April 2023
Announcement: NRS 2013
► International News Recommender Systems Workshop and Challenge► In conjunction with ACM RecSys 2013IMPORTANT DATES
July 21, 2013 paper submission deadline July 1, 2013 data set release August 15, 2013 on-line challenge kick-off
HIGHLIGHTS Access to a real recommender system Real-time requirements Big Data Cross-domain Implicit feedback
Website: https://sites.google.com/site/newsrec2013/homeTwitter: @NRSws2013
Competence Center Information Retrieval &Machine Learning
20
www.dai-labor.de
FonFax
+49 (0) 30 / 314 – 74+49 (0) 30 / 314 – 74 003
DAI-Labor
Technische Universität BerlinFakultät IV – Elektrontechnik & Informatik
Sekretariat TEL 14Ernst-Reuter-Platz 710587 Berlin, Deutschland
10. April 2023
Benjamin KilleResearcher / PhD student
benjamin.kille@dai-labor.de74128
top related