recomendação de conteúdo para redes sociais educativas

60
Content Recommenda,on Based on Data Mining in Adap,ve Social Networks Marcel Pinheiro Caraciolo [email protected] Orientador: Germano Crispim [email protected]

Upload: marcel-caraciolo

Post on 06-May-2015

3.656 views

Category:

Technology


2 download

DESCRIPTION

Palestra ministrada no X Seminário de Engenharia da Computação (SEC) Na Escola Politécnica de Pernambuco, em Recife Pernambuco em 21.10.2011

TRANSCRIPT

Page 1: Recomendação de Conteúdo para Redes Sociais Educativas

Content  Recommenda,on  Based  on  Data  Mining  in  Adap,ve  

Social  NetworksMarcel  Pinheiro  Caraciolo

[email protected]

Orientador:  Germano  Crispim [email protected]

Page 2: Recomendação de Conteúdo para Redes Sociais Educativas

Who is Marcel ?

2

Marcel Pinheiro Caraciolo - @marcelcaracioloSergipano, porém Recifense.

Cientista Chefe da Orygens (http://orygens.com)

Mestrando em Ciência da Computação no CIN/UFPE na área de mineração de dados

Membro e Moderador da Celúla de Usuários Python de Pernambuco (PUG-PE)

Minhas áreas de interesse: Computação móvel e Computação inteligente

Meus blogs: http://www.mobideia.com (sobre Mobilidade desde 2006)

http://aimotion.blogspot.com (sobre I.A. desde 2009)

Jovem Aprendiz ainda nas artes pythonicas.... (desde 2007)

Page 3: Recomendação de Conteúdo para Redes Sociais Educativas

The  Problem

Provide features that can enhance online social learning environment

One social software in special are recommender systems

Several approaches have been applied to web-based educational systems [1] [2]

But only a few recommender systems use data mining and explanations in the recommendations

Page 4: Recomendação de Conteúdo para Redes Sociais Educativas

Objec,vesDesign a methodology for social recommender systems that incorporate different knowledge sources

It can be structured such as messages, personal profile, timeline, questions , answersIt can be non-structured study plans, disciplines, classes watched, relationships

ommending friends, study groups and educational resources(books, video classes, etc.) to students in a social envi-ronment. We are currently developing a recommender en-gine framework that may incorporate di!erent recommenda-tion techniques for general recommendation called Crab [15].This framework can be used in several domains, includingthe educational context which is to suggest online learn-ing activities to students, based on their preferences, knowl-edge of other students with similar interests. The system isalready deployed in a brazilian educational social networkcalled AtePassar, which connects students, helping them toget approved in the public exams [9]. The innovative aspectof our approach is the use of data mining on social networksas also the incorporation of explanations at the given rec-ommendation which will bring the student more confidenceon the recommendations and better learning understanding.Only a few other recommender systems use data mining andexplanations in the recommendations [5].

This paper is arranged as follows: first we describe the re-lated background about recommender engines and their usein educational social learning. Then, we describe the recom-mender framework that we are developing, integrated intothe Atepassar social network. Finally, we will describe ourcurrent results that we have performed, our expected resultsbased on the experiments that we will carry out, conclusionsand future works.

2. RECOMMENDER SYSTEMSRecommender systems are applications that o!er personal-ized suggestions and provide to users information for facil-itating the decision process by incorporating the user char-acteristiscs [10]. These systems implicitly assume what theuser wishes by using appropriate recommendation algorithms,converting in item recommendations. Although recommen-dation approaches were first proposed and applied in e-commerceweb sites to sugest products, there are also several worksabout the application of recommender systems in E-learning[11], [6].

The recommendation algorithms can be classified into threedi!erent categories: rule based filtering systems, contentfiltering-based systems and collaborative filtering systems [12].The rule-based systems rely on a generated set of rules thatare used to recommend items to users. Content-based fil-tering systems recommend items according to the similaritybetween the item features and the content description inthe user profile. Collaborative filtering systems can be de-scribed as techniques that match the rating of a current userfor items with those of similar tasted users (neighborhood)in order to produce recommendations, for items not yet seen.

In our work we have developed a recommender system frame-work to support machine learning researchers and develop-ers in order to build, evaluate and deploy customized rec-ommender approaches. This system is called Crab and itis open-source available for the community [15]. For testingour recommender engine, we have integrated into the socialnetwork AtePassar, which enables the suggestion of classes,friends and study data groups for the resgistered users. Thesystem also integrates explanations, which can enhance thequality of the recommendations produced.

!"#$%&"'

(%)*++%&$%,''-&./&%'

01'01'01'

(%)*++%&$23*&4'56786-'

599786-'

!"#$%&":4'7*.'

8&;*,+23*&'9/<"%,/&.'

=%)>&/?#%4'

Figure 1: Architecture proposal for the recom-mender system

3. CONTENTRECOMMENDATION INED-UCATIONAL SOCIAL NETWORKS

3.1 ObjectivesThe overall objective of this research project is to designa methodology for social recommender systems that incor-porate di!erent knowledge sources from the interaction be-tween students within a social network site. Those sourcescould be structured or not. In the first case, we consider thestudent’s information stored in personal profile and at histimeline, a variation of a microblog where students can postcomments, share links by simple text messages. They aredescribed at our recommender engine with feature vectors,that can be easily queried with standard query languages.In the second case, we consider the additional informationprovided from the students that are generated by interact-ing with a web site. Examples of this interaction include therelationships with other users (friends and followers), studygroups that they participate, classes that the student haswatched and study plans created with disciplines and sub-jects. To extract knowledge from this type of data reposito-ries we use social-filtering algorithms.

At our current architecture illustrated at the Figure 1, weintegrate the mentioned sources of knowledge by using datamining techniques in order to exploit the appropriate fil-tering algorithm for each available source. Additional ob-jectives will be derive from an empirical study, that will beperformed, such as the tuning parameters of each recommen-dation algorithm and evaluate new strategies for aggregatingresults of the various algorithms.

Our recommender engine at AtePassar uses a hybrid rec-ommendation approach which analyzes the user interactionand produces daily recommendations with several compo-nents available from the social network. Moreover, we alsoprovide an improved explanation of the recommendation tothe user. Generally, in the currently available systems, theuser only receives the recommended item together with anoverall score indicating the appropriateness of such recom-mendation [13]. We improved this explanation by provid-ing detailed explanations, increasing the recommendationacceptance rate. The educational social networks are pow-erful source of knowledge and provide many features thatcan serve the learning sector in di!erent ways. In distancelearning and virtual universities , the recommendation can

It uses several components of the social network

Page 5: Recomendação de Conteúdo para Redes Sociais Educativas

Objec,vesDesign an improved explanation of the recommendation to the user

Generally in the currently available systems, the recommendation only comes with an overall score

Increase the recommendation acceptance rate in order to enhance the student experience in social networks

You have A,B,C friends in common

You have X, Y, Z followers in common

You have P,Q, R study groups in common.

You both live at the same state Pernambuco.

You are both interested in portuguese, maths and english.

Your friends P,X and Y follows him.

You have X, Y, Z concursos in common

You answered questions about X, Y, Z. You answered questions about X, Y, Z.

Your friend asked this question.

Because you have difficulty in portuguese, maths and english.Because it is a most searched course

Page 6: Recomendação de Conteúdo para Redes Sociais Educativas

What recommender systems really do?

1. Predict how much you would like a certain product or service

2. Suggest a list of N items ranked based on your interests

3. Suggest a list of N users ranked for a product/service

4. Explain to you why those recommendations were given

5. Adjust the prediction and the recommendation based on your feedback and from others.

Page 7: Recomendação de Conteúdo para Redes Sociais Educativas

Mee,ng  Recommenda,on  SystemsContent  Based  Filtering

MathsPortuguese

Similar

Text Interpretation Biology

Marcel

likesrecommend

Items

Users

Page 8: Recomendação de Conteúdo para Redes Sociais Educativas

Mee,ng  Recommenda,on  SystemsCollabora,ve  Based  Filtering

LiteraturePortuguese

Similar

Maths Physics

Marcel

like

recommends

Items

Rafael Amanda Users

like

Page 9: Recomendação de Conteúdo para Redes Sociais Educativas

Our  Approach  -­‐  Hybrid  Recommender  System

Mee,ng  Recommenda,on  Systems

!"#$%&$'()#% !"#$%*'+,-)%./0#$-+1'/%

%%

!"#$%#$&'()%*&+,-$%.,#/&%

2$,#/3"%456575689%:'+-1'/;%%<#+,=#%

*+>')-$">,?;%%@$-3A-0#3%%

%%

!"--(0".(12%&'()%*&+,-$%.,#/&%

!"#"$%&&'%()*&+,-(.'&/,-0&+,-(.'&%12%&'303#2,&('",'&2,"&34&

%

B#0-%<#+'CC#/3#$%%% %&-$-C#0#$"%%

<#+'CC#/3-1'/"%

Figure 2: Meta Recommender Components Interac-tion

be highly beneficial given that students do not meet phys-ically. It may result in their becoming more socially con-nected, thereby enhancing their social learning environmentand student experience.

3.2 The MethodologyIn order to design our recommender engine for an adaptativelearning environment, we investigated the user behavior in aeducational social network, in our scenario the AtePassar so-cial network and incorporated our own ideas into the systemdesign. We have concluded that the knowledge data sharedbetween students in the learning context are quite di!erent:likeness, text messages, social graphs, etc. For this reason,we have adopted an meta recommender system architecture[14].

A meta recommender approach provides users with cus-tomized control over the generation of a single recommen-dation list, generated from a aggregation of rich data. Thispersonalized control can be implemented analyzing the na-ture of a specific data source. In our approach, for instance,if the item to be recommended is a new friend, where forprivacy concerns, in term of attributes, only the relation-ships (list of friends) are available, then the system reliesmore on collaborative filtering. In other hand, if the item tobe recommended has a rich structured description such as acourse, then the system tends to use more the content-basedapproach. The final recommendation score is computed byaggregating the results of both recommenders. The Figure2 illustrates an overview of our approach.

For new users at the social network, which can su!er fromthe cold start problem, that is a common problem in rec-ommender systems for new users that don’t have historicalrecords in the system, we provide popular recommendationswhich were accepted from another users registered in thenetwork. We hypothesize that popular items recommendedfor the new users would help the users to even interact morewithin the social network, as the system learns from his ini-tial interests.

At this moment, we are still studying di!erent approaches in

Figure 3: AtePassar Recommender System Inter-face

how to compute the final score for the recommendation. Thecurrent approach takes either the final scores of each recom-mender as input to a weighted average function, where theweights can be derived for implicit or explicit user prefer-ences for a certain recommender.

3.3 The Atepassar based Mining and Recom-mender System

We have developed a recommendation framework called Crab,which is a recommender framework that aims to provide arich set of components from which you can build a personal-ized recommender system from a set of algorithms. It is writ-ten in Python, which is a popular programming languagedesigned for scability, flexibility and performance, makinguse of scientific optimized packages in order provide e"cientand easy-to-use solutions in several contexts [15].

We have integrated this engine with easy-to-use interface forstudents into the popular brazilian social network AtePas-sar, an educational virtual learning environment with morethan 70.000 students registered interested at studying forthe public examinations in order to get a civil job. TheFigure 3 presents a screenshot of our recommender systemat AtePassar. Each recommendation comes with a explana-tion, allowing the student to better understand the reason ofthe given suggestion by the system. Another functionalityis that the user can accept or refuse the recommendation,and in this processs the feedback and result obtained can bedirectly applied to future recommendations.

3.4 The Current ResultsIn the current recommender system, we are providing friends,study groups and products (e.g. video classes) to the activeusers at AtePassar. The recommender engine is runningsince January 2011 and recommended more than 100.000items to over than 60.000 users. We are currently devel-oping new features by recommending another componentspresent in the social network such as studyplans, disciplinesand questions.

Meta recommender system architectureCustomized control over the generation of

a recommendation list

It adapts in accordance to the structure of the data

Recommendation of Friends ?

More weight to Collaborative Filtering

Recommendation of Courses ?

More weight to Content Filtering

Cold Start: Mitigate using Popular Recommendations accepted from another users

Feedback and temporal slicing:

Learn from users and select limited results by time

Page 10: Recomendação de Conteúdo para Redes Sociais Educativas

Methodology  and  Current  Results

!"#$%&$'()#% !"#$%*'+,-)%./0#$-+1'/%

%%

!"#$%#$&'()%*&+,-$%.,#/&%

2$,#/3"%456575689%:'+-1'/;%%<#+,=#%

*+>')-$">,?;%%@$-3A-0#3%%

%%

!"--(0".(12%&'()%*&+,-$%.,#/&%

!"#"$%&&'%()*&+,-(.'&/,-0&+,-(.'&%12%&'303#2,&('",'&2,"&34&

%

B#0-%<#+'CC#/3#$%%% %&-$-C#0#$"%%

<#+'CC#/3-1'/"%

Figure 2: Meta Recommender Components Interac-tion

be highly beneficial given that students do not meet phys-ically. It may result in their becoming more socially con-nected, thereby enhancing their social learning environmentand student experience.

3.2 The MethodologyIn order to design our recommender engine for an adaptativelearning environment, we investigated the user behavior in aeducational social network, in our scenario the AtePassar so-cial network and incorporated our own ideas into the systemdesign. We have concluded that the knowledge data sharedbetween students in the learning context are quite di!erent:likeness, text messages, social graphs, etc. For this reason,we have adopted an meta recommender system architecture[14].

A meta recommender approach provides users with cus-tomized control over the generation of a single recommen-dation list, generated from a aggregation of rich data. Thispersonalized control can be implemented analyzing the na-ture of a specific data source. In our approach, for instance,if the item to be recommended is a new friend, where forprivacy concerns, in term of attributes, only the relation-ships (list of friends) are available, then the system reliesmore on collaborative filtering. In other hand, if the item tobe recommended has a rich structured description such as acourse, then the system tends to use more the content-basedapproach. The final recommendation score is computed byaggregating the results of both recommenders. The Figure2 illustrates an overview of our approach.

For new users at the social network, which can su!er fromthe cold start problem, that is a common problem in rec-ommender systems for new users that don’t have historicalrecords in the system, we provide popular recommendationswhich were accepted from another users registered in thenetwork. We hypothesize that popular items recommendedfor the new users would help the users to even interact morewithin the social network, as the system learns from his ini-tial interests.

At this moment, we are still studying di!erent approaches in

Figure 3: AtePassar Recommender System Inter-face

how to compute the final score for the recommendation. Thecurrent approach takes either the final scores of each recom-mender as input to a weighted average function, where theweights can be derived for implicit or explicit user prefer-ences for a certain recommender.

3.3 The Atepassar based Mining and Recom-mender System

We have developed a recommendation framework called Crab,which is a recommender framework that aims to provide arich set of components from which you can build a personal-ized recommender system from a set of algorithms. It is writ-ten in Python, which is a popular programming languagedesigned for scability, flexibility and performance, makinguse of scientific optimized packages in order provide e"cientand easy-to-use solutions in several contexts [15].

We have integrated this engine with easy-to-use interface forstudents into the popular brazilian social network AtePas-sar, an educational virtual learning environment with morethan 70.000 students registered interested at studying forthe public examinations in order to get a civil job. TheFigure 3 presents a screenshot of our recommender systemat AtePassar. Each recommendation comes with a explana-tion, allowing the student to better understand the reason ofthe given suggestion by the system. Another functionalityis that the user can accept or refuse the recommendation,and in this processs the feedback and result obtained can bedirectly applied to future recommendations.

3.4 The Current ResultsIn the current recommender system, we are providing friends,study groups and products (e.g. video classes) to the activeusers at AtePassar. The recommender engine is runningsince January 2011 and recommended more than 100.000items to over than 60.000 users. We are currently devel-oping new features by recommending another componentspresent in the social network such as studyplans, disciplinesand questions.

Integrated this engine with the popular brazilian social network AtéPassar

More than 70.000 students registered studying for the public examinations

Recommend StudyGroups, Friends, Video Classes, Questions and Concursos

More than 70.000 items available for recommend

Written in Python using a open-source framework Crab

Framework available for building recommender systems (My contribution)

It is running since January 2011

In March 2011 , questionnaire was performed.

23%

77%

Liked Not Liked

Page 11: Recomendação de Conteúdo para Redes Sociais Educativas
Page 12: Recomendação de Conteúdo para Redes Sociais Educativas

Expected  ResultsImprove the Learning Process

Analyze how the recommenders can increase the learning process in online educational social networks.

Explore the Hidden KnowledgeThere are several knowledge sources in a educational social network.

All those sources as basis for discovery novel content and learning resources

Better Recommendation UnderstandingThe explanations can be quite helpful to better understanding of the given resource

Better learning interaction and simplification of the human/computer interaction

Mining all types of source in social networks in forms of recommendations

Page 13: Recomendação de Conteúdo para Redes Sociais Educativas

Conclusions

Proposal of a personalized recommender system that incorporate different types of source of knowledge

It can be applied in adaptive social networks

Bring machine learning to web educational systems

Help the students to find an optimal path to learning resources

Help the researchers and designers how to design

the social network to achieve those tasks efficiently

Page 14: Recomendação de Conteúdo para Redes Sociais Educativas

Contribu)ons

Page 15: Recomendação de Conteúdo para Redes Sociais Educativas

CrabA Python Framework for Building

Recommendation Engines

Marcel Caraciolo@marcelcaraciolo

Bruno Melo@brunomelo

Ricardo Caspirro@ricardocaspirro

Page 16: Recomendação de Conteúdo para Redes Sociais Educativas

What is Crab ?

A python framework for building recommendation engines

A Scikit module for collaborative, content and hybrid filtering

Mahout Alternative for Python Developers :D

Open-Source under the BSD license

https://github.com/muricoca/crab

Page 17: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

Collaborative Filtering algorithms

Evaluation of the Recommender Algorithms

User-Based, Item-Based and Slope One

Precision, Recall, F1-Score, RMSE

Precision-Recall Charts

Page 18: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

Page 19: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab>>>#load the dataset

Page 20: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab>>>#load the dataset

>>> from crab.datasets import load_sample_movies

Page 21: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab>>>#load the dataset

>>> from crab.datasets import load_sample_movies

>>> data = load_sample_movies()

Page 22: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab>>>#load the dataset

>>> from crab.datasets import load_sample_movies

>>> data = load_sample_movies()

>>> data

Page 23: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

{'DESCR': 'sample_movies data set was collected by the book called \nProgramming the Collective Intelligence by Toby Segaran \n\nNotes\n----- \nThis data set consists of\n\t* n ratings with (1-5) from n users to n movies.', 'data': {1: {1: 3.0, 2: 4.0, 3: 3.5, 4: 5.0, 5: 3.0},  2: {1: 3.0, 2: 4.0, 3: 2.0, 4: 3.0, 5: 3.0, 6: 2.0},  3: {2: 3.5, 3: 2.5, 4: 4.0, 5: 4.5, 6: 3.0},  4: {1: 2.5, 2: 3.5, 3: 2.5, 4: 3.5, 5: 3.0, 6: 3.0},  5: {2: 4.5, 3: 1.0, 4: 4.0},  6: {1: 3.0, 2: 3.5, 3: 3.5, 4: 5.0, 5: 3.0, 6: 1.5},  7: {1: 2.5, 2: 3.0, 4: 3.5, 5: 4.0}}, 'item_ids': {1: 'Lady in the Water',  2: 'Snakes on a Planet',  3: 'You, Me and Dupree',  4: 'Superman Returns',  5: 'The Night Listener',  6: 'Just My Luck'}, 'user_ids': {1: 'Jack Matthews',  2: 'Mick LaSalle',  3: 'Claudia Puig',  4: 'Lisa Rose',  5: 'Toby',  6: 'Gene Seymour',  7: 'Michael Phillips'}}

>>>#load the dataset

>>> from crab.datasets import load_sample_movies

>>> data = load_sample_movies()

>>> data

Page 24: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

Page 25: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> from crab.models import MatrixPreferenceDataModel

Page 26: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> from crab.models import MatrixPreferenceDataModel

>>> m = MatrixPreferenceDataModel(data.data)

Page 27: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> print mMatrixPreferenceDataModel (7 by 6)         1 2 3 4 5 ...1 3.000000 4.000000 3.500000 5.000000 3.0000002 3.000000 4.000000 2.000000 3.000000 3.0000003 --- 3.500000 2.500000 4.000000 4.5000004 2.500000 3.500000 2.500000 3.500000 3.0000005 --- 4.500000 1.000000 4.000000 ---6 3.000000 3.500000 3.500000 5.000000 3.0000007 2.500000 3.000000 --- 3.500000 4.000000

>>> from crab.models import MatrixPreferenceDataModel

>>> m = MatrixPreferenceDataModel(data.data)

Page 28: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

Page 29: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> #import pairwise distance

Page 30: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

Page 31: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

>>> #import similarity

Page 32: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

>>> #import similarity>>> from crab.similarities import UserSimilarity

Page 33: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

>>> #import similarity>>> from crab.similarities import UserSimilarity

>>> similarity = UserSimilarity(m, euclidean_distances)

Page 34: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

>>> #import similarity>>> from crab.similarities import UserSimilarity

>>> similarity = UserSimilarity(m, euclidean_distances)

>>> similarity[1]

Page 35: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

[(1, 1.0), (6, 0.66666666666666663), (4, 0.34054242658316669), (3, 0.32037724101704074), (7, 0.32037724101704074), (2, 0.2857142857142857), (5, 0.2674788903885893)]

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

>>> #import similarity>>> from crab.similarities import UserSimilarity

>>> similarity = UserSimilarity(m, euclidean_distances)

>>> similarity[1]

Page 36: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

[(1, 1.0), (6, 0.66666666666666663), (4, 0.34054242658316669), (3, 0.32037724101704074), (7, 0.32037724101704074), (2, 0.2857142857142857), (5, 0.2674788903885893)]

>>> #import pairwise distance

>>> from crab.metrics.pairwise import euclidean_distances

>>> #import similarity>>> from crab.similarities import UserSimilarity

>>> similarity = UserSimilarity(m, euclidean_distances)

>>> similarity[1]

MatrixPreferenceDataModel (7 by 6)         1 2 3 4 5 ...1 3.000000 4.000000 3.500000 5.000000 3.0000002 3.000000 4.000000 2.000000 3.000000 3.0000003 --- 3.500000 2.500000 4.000000 4.5000004 2.500000 3.500000 2.500000 3.500000 3.0000005 --- 4.500000 1.000000 4.000000 ---6 3.000000 3.500000 3.500000 5.000000 3.0000007 2.500000 3.000000 --- 3.500000 4.000000

Page 37: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

Page 38: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> from crab.recommenders.knn import UserBasedRecommender

Page 39: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> from crab.recommenders.knn import UserBasedRecommender

>>> recsys = UserBasedRecommender(model=m, similarity=similarity, capper=True,with_preference=True)

Page 40: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> from crab.recommenders.knn import UserBasedRecommender

>>> recsys = UserBasedRecommender(model=m, similarity=similarity, capper=True,with_preference=True)

>>> recsys.recommend(5)array([[ 5. , 3.45712869],       [ 1. , 2.78857832],       [ 6. , 2.38193068]])

Page 41: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> recsys.recommended_because(user_id=5,item_id=1)array([[ 2. , 3. ],       [ 1. , 3. ],       [ 6. , 3. ],       [ 7. , 2.5],       [ 4. , 2.5]])

>>> from crab.recommenders.knn import UserBasedRecommender

>>> recsys = UserBasedRecommender(model=m, similarity=similarity, capper=True,with_preference=True)

>>> recsys.recommend(5)array([[ 5. , 3.45712869],       [ 1. , 2.78857832],       [ 6. , 2.38193068]])

Page 42: Recomendação de Conteúdo para Redes Sociais Educativas

The current Crab

>>> recsys.recommended_because(user_id=5,item_id=1)array([[ 2. , 3. ],       [ 1. , 3. ],       [ 6. , 3. ],       [ 7. , 2.5],       [ 4. , 2.5]])

>>> from crab.recommenders.knn import UserBasedRecommender

>>> recsys = UserBasedRecommender(model=m, similarity=similarity, capper=True,with_preference=True)

>>> recsys.recommend(5)array([[ 5. , 3.45712869],       [ 1. , 2.78857832],       [ 6. , 2.38193068]])

MatrixPreferenceDataModel (7 by 6)         1 2 3 4 5 ...1 3.000000 4.000000 3.500000 5.000000 3.0000002 3.000000 4.000000 2.000000 3.000000 3.0000003 --- 3.500000 2.500000 4.000000 4.5000004 2.500000 3.500000 2.500000 3.500000 3.0000005 --- 4.500000 1.000000 4.000000 ---6 3.000000 3.500000 3.500000 5.000000 3.0000007 2.500000 3.000000 --- 3.500000 4.000000

Page 43: Recomendação de Conteúdo para Redes Sociais Educativas

Distributed Computing with mrJobhttps://github.com/Yelp/mrjob

Page 44: Recomendação de Conteúdo para Redes Sociais Educativas

Distributed Computing with mrJobhttps://github.com/Yelp/mrjob

It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or local (for testing)

Page 45: Recomendação de Conteúdo para Redes Sociais Educativas

Distributed Computing with mrJobhttps://github.com/Yelp/mrjob

It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or local (for testing)

Page 46: Recomendação de Conteúdo para Redes Sociais Educativas

Distributed Computing with mrJobhttps://github.com/Yelp/mrjob

"""The classic MapReduce job: count the frequency of words."""from mrjob.job import MRJobimport re

WORD_RE = re.compile(r"[\w']+")

class MRWordFreqCount(MRJob):

    def mapper(self, _, line):        for word in WORD_RE.findall(line):            yield (word.lower(), 1)

    def reducer(self, word, counts):        yield (word, sum(counts))

if __name__ == '__main__':    MRWordFreqCount.run()

It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or local (for testing)

Page 47: Recomendação de Conteúdo para Redes Sociais Educativas

Studies with Sparse MatricesReal datasets come with lots of empty values

Apontador Reviews Dataset

http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html

Solutions:

scipy.sparse package

Sharding operations

Matrix Factorization techniques (SVD)

Page 48: Recomendação de Conteúdo para Redes Sociais Educativas

Studies with Sparse MatricesReal datasets come with lots of empty values

Apontador Reviews Dataset

http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html

Solutions:

scipy.sparse package

Sharding operations

Matrix Factorization techniques (SVD)

Crab implements a Matrix Factorization with Expectation

Maximization algorithm

Page 49: Recomendação de Conteúdo para Redes Sociais Educativas

Studies with Sparse MatricesReal datasets come with lots of empty values

Apontador Reviews Dataset

http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html

Solutions:

scipy.sparse package

Sharding operations

Matrix Factorization techniques (SVD)

Crab implements a Matrix Factorization with Expectation

Maximization algorithmscikits.crab.svd package

Page 50: Recomendação de Conteúdo para Redes Sociais Educativas

Why migrate ?

Old Crab running only using Pure Python

Recommendations demand heavy maths calculations and lots of processing

Compatible with Numpy and Scipy libraries

High Standard and popular scientific libraries optimized for scientific calculations in Python

Scikits projects are amazing! Active Communities, Scientific Conferences and updated projects (e.g. scikit-learn)

Turn the Crab framework visible for the community Join the scientific researchers and machine learning developers around the Globe coding with

Python to help us in this project

Be Fast and Furious

Page 51: Recomendação de Conteúdo para Redes Sociais Educativas

How are we working ?

Sprints, Online Discussions and Issues

https://github.com/muricoca/crab/wiki/UpcomingEvents

Page 52: Recomendação de Conteúdo para Redes Sociais Educativas

How are we working ?

Our Project’s Home Page

http://muricoca.github.com/crab

Page 53: Recomendação de Conteúdo para Redes Sociais Educativas

Future Releases

Planned Release 0.1Collaborative Filtering Algorithms working, sample datasets to load and test

Planned Release 0.11Evaluation of Recommendation Algorithms and Database Models support

Planned Release 0.12Recommendation as Services with REST APIs

....

Page 54: Recomendação de Conteúdo para Redes Sociais Educativas

Join us!

1. Read our Wiki Pagehttps://github.com/muricoca/crab/wiki/Developer-Resources

2. Check out our current sprints and open issueshttps://github.com/muricoca/crab/issues

3. Forks, Pull Requests mandatory

4. Join us at irc.freenode.net #muricoca or at our discussion list

http://groups.google.com/group/scikit-crab

Page 55: Recomendação de Conteúdo para Redes Sociais Educativas

Recommended books

SatnamAlag, Collective Intelligence in Action, Manning Publications, 2009

Toby Segaran, Programming Collective Intelligence, O'Reilly, 2007

Sites such as TechCrunch and ReadWriteWeb

Page 56: Recomendação de Conteúdo para Redes Sociais Educativas

Recommended Conferences- ACM RecSys.

–ICWSM: Weblogand Social Media

–WebKDD: Web Knowledge Discovery and Data Mining

–WWW: The original WWW conference

–SIGIR: Information Retrieval

–ACM KDD: Knowledge Discovery and Data Mining

–ICML: Machine Learning

Page 57: Recomendação de Conteúdo para Redes Sociais Educativas

ReferencesIn March 2011 we also performed a questionnaire with thefeedback of the users about the current recommender en-gine. From 83 users who filled out, 67 said that liked thesuggestions provided by the recommender. This broughtus confidence to keep developing and ehancing the qualityof the recommendations, pushing it into more personalizedcontent discovery.

3.5 The Expected Results3.5.1 Improve the Learning ProccessThose collective intelligence tools enable social networkingto take place. Our idea is to analyze how the recommen-dations can increase the learning process by providing tousers new learning resources to maintain the interaction ofstudents in online educational networks. We believe thatmining those knowledge rich data provided in online educa-tional networks in forms of recommendations will improvethe student learning experience.

3.5.2 Explore the Hidden KnowledgeWe hypothesize that there are several knowledge sources ina educational social network. For instance, the hours of theday where the user is logged in or even the location of theuser. All this information could be used to improve therecommendation process. Explore this hidden informationwould serve as basis for discovery of novel content and ex-plore more the learning resource materials, one of the goalsof the e-learning system.

3.5.3 Better Recommendation UnderstandingThe explanations can be quite helpful for users to betterunderstand if the given resource is (or is not ) suitable forthem. Those explanations can be beneficial in how to for-mulate a better learning interaction and hence to simply andshorten the human/computer interaction.

4. CONCLUSIONSIn this paper, we presents a personalized recommender sys-tem that incorporate di!erent types of sources of knowledgein a adaptative educational social network by using a meta-recommender approach. We believe that this technique willbring more personalized and better recommendations for thestudents.

Although we have integrated the system in the AtePassar, itcan, in principle, also be used in other web-based educationalsystems. More research need to be conducted in the areaof the recommendation systems in social learning and howthey can influence the student satisfaction, motivation andlearning process.

Bringing machine learning and artificial intelligence to edu-cational social networks is one of the main contributions ofthis work, which is resulting on a new generation of web-educational systems. The task of recommendation in web-based education can be considered as special type of col-laborative learning, due to the fact that they help studentwith similar interests to find an optimal path through thelearning material and facilitate the social network design tobe e!ectively be used to achieve these targets. We providea video demonstrating the recommender system in action atAtePassar, which can be accessible at œ[16].

5. REFERENCES[1] Brusilovsky, P., Peylo, C.: Adaptive and Intelligent

Web-based Educational Systems. International Journalof Artificial Intelligence in Education. 13, 2003, 156 -169.

[2] Conole, G. & Culver, J.: The design of Cloudworks:Applying social networking practice to foster theexchange of learning and teaching ideas and designs In:Computers & Education , Vol. 54 , Nr. 3 Elsevier, 2010, S. 679-692 .

[3] Gogoulou, A., Gouli, E., Grigoriadou, M., Samarakou,M., and Chinou, D.: A Web-based Educational SettingSupporting Individualized Learning, CollaborativeLearning and Assessment. In Proceedings ofEducational Technology & Society. 2007, 242-256.

[4] Schafer, J.B.: The application of data-mining torecommender systems. In J. Wang (Ed.), Encyclopediaof data warehousing and mining. Hershey, PA. IdeaGroup 2005 44 - 48.

[5] Farzan, R., Brusilovsky, P.: Social Navigation Supportin a Course Recommendation System. In proceedings of4th International Conference on Adaptive Hypermediaand Adaptive Web-based Systems. Dublin, 2006.91-100.

[6] Ksristofic, A.: Recommender System for AdaptiveHypermedia Applications. In Proceeding of Informaticsand Information Technology Student ResearchConference, Bratislava, 2005. 229-234.

[7] Romero, C., Ventura, S.: Educational Data Mining: aSurvey from 1995 to 2005. Expert Systems withApplications. Elsevier 1:33, 2007. 135-146.

[8] Romero, C., Ventura, S.: Data mining in e-learning.Wit Press, 2006.

[9] Atepassar. Available at: http://atepassar.com[10] J. B. Schafer, J. A. Konstan, and J. Riedl.:

E-commerce recommendation applications. Data Miningand Knowledge Discovery, 2001.

[11] Shen, L.P., Shen, R.M.: Learning ContentRecommendation Service Based-on Simple SequencingSpecification. In Proceedings of Advanced in Web-basedLearning 2004 363-

[12] M. Montaner, B. L’opez, and J. Rosa.: A taxonomy ofrecommender agents on the internet. In ArtificialIntelligence Review, volume 19, pages 285-330. KluwerAcademic, 2003.

[13] J. Herlocker, J. Konstan, and J. Riedl.: Explainingcollaborative filtering recommendations. In Inproceedings of ACM 2000 Conference on ComputerSupported Cooperative Work, 2000.

[14] B. Schafer, J. Konstan, and J. Riedl.:Meta-recommendation systems: user-controlledintegration of diverse recommendations. In Proceedingsof the 11th conference on Information and knowledgemanagement, pages 43-51. ACM Press, 2002.

[15] Crab. Available at: http://muricoca.github.com/crab[16] AtePassar Video. Available at:

http://www.youtube.com/watch?v=gzn7I1EJmb4

In March 2011 we also performed a questionnaire with thefeedback of the users about the current recommender en-gine. From 83 users who filled out, 67 said that liked thesuggestions provided by the recommender. This broughtus confidence to keep developing and ehancing the qualityof the recommendations, pushing it into more personalizedcontent discovery.

3.5 The Expected Results3.5.1 Improve the Learning ProccessThose collective intelligence tools enable social networkingto take place. Our idea is to analyze how the recommen-dations can increase the learning process by providing tousers new learning resources to maintain the interaction ofstudents in online educational networks. We believe thatmining those knowledge rich data provided in online educa-tional networks in forms of recommendations will improvethe student learning experience.

3.5.2 Explore the Hidden KnowledgeWe hypothesize that there are several knowledge sources ina educational social network. For instance, the hours of theday where the user is logged in or even the location of theuser. All this information could be used to improve therecommendation process. Explore this hidden informationwould serve as basis for discovery of novel content and ex-plore more the learning resource materials, one of the goalsof the e-learning system.

3.5.3 Better Recommendation UnderstandingThe explanations can be quite helpful for users to betterunderstand if the given resource is (or is not ) suitable forthem. Those explanations can be beneficial in how to for-mulate a better learning interaction and hence to simply andshorten the human/computer interaction.

4. CONCLUSIONSIn this paper, we presents a personalized recommender sys-tem that incorporate di!erent types of sources of knowledgein a adaptative educational social network by using a meta-recommender approach. We believe that this technique willbring more personalized and better recommendations for thestudents.

Although we have integrated the system in the AtePassar, itcan, in principle, also be used in other web-based educationalsystems. More research need to be conducted in the areaof the recommendation systems in social learning and howthey can influence the student satisfaction, motivation andlearning process.

Bringing machine learning and artificial intelligence to edu-cational social networks is one of the main contributions ofthis work, which is resulting on a new generation of web-educational systems. The task of recommendation in web-based education can be considered as special type of col-laborative learning, due to the fact that they help studentwith similar interests to find an optimal path through thelearning material and facilitate the social network design tobe e!ectively be used to achieve these targets. We providea video demonstrating the recommender system in action atAtePassar, which can be accessible at œ[16].

5. REFERENCES[1] Brusilovsky, P., Peylo, C.: Adaptive and Intelligent

Web-based Educational Systems. International Journalof Artificial Intelligence in Education. 13, 2003, 156 -169.

[2] Conole, G. & Culver, J.: The design of Cloudworks:Applying social networking practice to foster theexchange of learning and teaching ideas and designs In:Computers & Education , Vol. 54 , Nr. 3 Elsevier, 2010, S. 679-692 .

[3] Gogoulou, A., Gouli, E., Grigoriadou, M., Samarakou,M., and Chinou, D.: A Web-based Educational SettingSupporting Individualized Learning, CollaborativeLearning and Assessment. In Proceedings ofEducational Technology & Society. 2007, 242-256.

[4] Schafer, J.B.: The application of data-mining torecommender systems. In J. Wang (Ed.), Encyclopediaof data warehousing and mining. Hershey, PA. IdeaGroup 2005 44 - 48.

[5] Farzan, R., Brusilovsky, P.: Social Navigation Supportin a Course Recommendation System. In proceedings of4th International Conference on Adaptive Hypermediaand Adaptive Web-based Systems. Dublin, 2006.91-100.

[6] Ksristofic, A.: Recommender System for AdaptiveHypermedia Applications. In Proceeding of Informaticsand Information Technology Student ResearchConference, Bratislava, 2005. 229-234.

[7] Romero, C., Ventura, S.: Educational Data Mining: aSurvey from 1995 to 2005. Expert Systems withApplications. Elsevier 1:33, 2007. 135-146.

[8] Romero, C., Ventura, S.: Data mining in e-learning.Wit Press, 2006.

[9] Atepassar. Available at: http://atepassar.com[10] J. B. Schafer, J. A. Konstan, and J. Riedl.:

E-commerce recommendation applications. Data Miningand Knowledge Discovery, 2001.

[11] Shen, L.P., Shen, R.M.: Learning ContentRecommendation Service Based-on Simple SequencingSpecification. In Proceedings of Advanced in Web-basedLearning 2004 363-

[12] M. Montaner, B. L’opez, and J. Rosa.: A taxonomy ofrecommender agents on the internet. In ArtificialIntelligence Review, volume 19, pages 285-330. KluwerAcademic, 2003.

[13] J. Herlocker, J. Konstan, and J. Riedl.: Explainingcollaborative filtering recommendations. In Inproceedings of ACM 2000 Conference on ComputerSupported Cooperative Work, 2000.

[14] B. Schafer, J. Konstan, and J. Riedl.:Meta-recommendation systems: user-controlledintegration of diverse recommendations. In Proceedingsof the 11th conference on Information and knowledgemanagement, pages 43-51. ACM Press, 2002.

[15] Crab. Available at: http://muricoca.github.com/crab[16] AtePassar Video. Available at:

http://www.youtube.com/watch?v=gzn7I1EJmb4

Page 58: Recomendação de Conteúdo para Redes Sociais Educativas

AtéPassar  -­‐  hOp://atepassar.comData  available  for  researching  -­‐  [email protected]

Page 59: Recomendação de Conteúdo para Redes Sociais Educativas

Thanks !!

Source: Hunch.com

Where will you be ?

Page 60: Recomendação de Conteúdo para Redes Sociais Educativas

Content  Recommenda,on  Based  on  Data  Mining  in  Adap,ve  

Social  NetworksMarcel  Pinheiro  [email protected]