Learning analytics and Moodle: So much we could measure, but what do we want to measure?

Associate Professor Michael Sankey, EdDDirector, Learning Environments and Media

SAF

Embarking on a project that involves: Establishing common codebase across all our 3

Moodle environments fully aligned with Mahara Extending the functionality of eAssessment within

Moodle (replacing EASE, CMA, EMS) Establishing a suite repositories in Equella Create new digital rights management workflow Enhance discoverability Establish learning analytics across L&T systems Align help resources to new regime and Provide PD

Learning analytics for our systems

Which systems? What tools? How big do we want the data? Is it just our USQ systems? What do we want to know? How do we want to use this data? Who gets involved? Who makes the decisions? I’ll come back to these questions at the end But first some background…

Siemens, G. 2013. Structure and logic of analytics. Available from http://www.learninganalytics.net/

Three levels within the institution

Learning Analytics

Educational Data mining

Educational

Analytics

All focus on the learner to some degree, either as an individual or in context to the institution

Educational Analytics

Academic and Learning Analytics

http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education

Siemens, G. 2013. Structure and logic of analytics. Available from http://www.learninganalytics.net/

Developing an analytics framework

Identify Tier 1 data

Data that exists (technical

analytical)

Form Tier 2 data Issues

intelligence

Create Tier 3 data

Context intelligence

Adapted from Terenzini, 2013; Padró & Frederiks, 2013

SBMI and Peoplesoft• AUSSE/UES• Grades• Graduation rate• Persistence• Retention• Student demographics• Student satisfaction data• Transfer rates

L&T Systems & RightNow• Co-curricular student

engagement activities data• Course interactions• Systems data• Learning Centre data• Other student learning

support activities data

Institutional emphasis for data collection & analysis:Customer service (transactional) or Student development

National policy preference

Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.

http://www.solaresearch.org/

Where did it originate

SoLAR exists to ensure that there is an expansive, transformative vision for what analytics might mean for the future of learning and to promote a very critical discourse that is non-partisan, and grounded as far as possible in practice-based research. SoLAR is a non-profit organization. Incorporation is currently underway.

Scope

At what level do we pitch? LMS data analytics

Easier to implement Limited data so not the whole picture Logs in Moodle are good, but not comprehensive

The Learning ecosystem analytics Complex – needs an open standards model and

potentially access to external repositories Much more holistic picture In our case, Mahara, Equella, BB Collaborate,

EASE, Library, lecture capture, etc

LMS is one of the primary providers of the data, since it preserves digital footprints of student interactions which can be mined for patterns of learning behaviour and teaching practice, and this allows for benchmarking and the monitoring of institutional quality initiatives.

Predictive analysis indicates that some students are higher risk than others; for those who are first in family or from a low socio-economic background, the risk of failure increases. There is a question as to what constitutes quality learning. Analytics is a key player in this field, given that it provides a vast amount of data and techniques for its analysis. Learners and their context are vitally important in this discussion.

How big is the data?

Typically in our Moodle we generate between 50-100 million log records per year

What is the Aim Finding out we have a problem before it fully

manifests If we accept this it gives us a framework to

consider our options

An perspective on big data OUA

As with other fields the key questions to ask are: what do you want to know; why do you want to know it and what are you going to do next?

Analytics makes sound educational and financial sense, it increases retention and encourages students to enrol again.

The biggest factor in student’s retention is intent of purpose; why are they doing what they want to do? There are things that can be done to aid them to achieve their intent of purpose.

Previous education is the single biggest predictor of success. So the question becomes, what supports can we put in place for those without this. E.g.: Invigilated exams in a student’s first unit decreases their chance of

success. This raises learning design issues for our introductory units. Other data shows that older students and female students are more likely

to succeed in their first Course. Coaching and contact are also predictors of retention and success, along

with preparatory units.

QUT study – Wendy Harper

Overall, the conclusions she drew were: The key predictor of success in a unit is GPA. The number of hits and days visiting a QUT Blackboard unit

site also predicts unit success. Students who are likely to fail a unit often do not engage early

enough with their online environment. Students who fail a unit often have alternating high peaks of

engagement and total disengagement. Factors such as gender, international or domestic enrolment,

and age make very little difference to student behaviour in online units.

‘Narrowly failing’ students often perform a much greater amount of online activities in the unit they are struggling in.

‘Narrowly failing’ students often show high engagement around early assessment pieces, but this drops off as the semester progresses.

“Learning and knowledge creation is often distributed across multiple media and sites in networked environments. Traces of such activity may be fragmented across multiple logs and may not match analytic needs. As a result, the coherence of distributed interaction and emergent phenomena are analytically cloaked”

Suthers, Rosen, 2011

Athabasca's approach

Siemens, G. 2013. Structure and logic of analytics. Available from http://www.learninganalytics.net/

Ethics

The ethical professional Respecting the rights of students Stepping in to

provide pastoral support advise about risks of failure advise about increasing chances of success

Research ethics Risk minimisation Needed for publication Issues with:

accessing ‘databanks’ anonymity

Privacy Issues

The Greater Good vs Big Brother Teacher:

“It’s unethical not to tell a student they are at risk of failing”

Student: “I don’t want you to be looking over my shoulder. I

can make my own choices about my study.” Reports to staff vs dashboards for students

Usefulness of analytics

What is the question to which analytics is the answer? Don’t just buy a product

Learning analytics are just indicators of behaviour They don’t explain behaviour

A single source of analytic data is probably insufficient Combine data into a data warehouse

Time to look at some different options

The Engagement Analytics block

http://docs.moodle.org/22/en/report/analytics/index It provides information about student progress against a range of

indicators. It provides feedback on the level of "engagement" of a student. “Engagement" refers to activities which have been identified by current research to have an impact on student success in an online course.

The plugin was developed as part of a NetSpot Innovation Fund project by Monash University (Dr Phillip Dawson), with code by NetSpot developers (Ashley Holman & Adam Olley).

It is a block that teachers can add to their Moodle course that will provide them with a quick graphical snapshot of which students are at risk.

It is important to note that the purpose of the plugin is to provide teachers with information only, it does not automatically take any action based on the indicators e.g. NO email or notification is sent to students automatically.

If desired the teacher would follow up on the information themselves, based on what they know about the student and their other communications.

GISMO

It is a visualization tool for Moodle that obtains tracking data, transforms the data into a form convenient for processing, and generates graphical representations that can be explored and manipulated by course instructors to examine social, cognitive, and behavioral aspects of distance students.

It can be included in any Moodle course as side block. Since it is aimed to help instructors, this block will be visible only to users who have the instructor role (students don't see it).

Each time the Moodle cron jobs runs, GISMO fetches students' data from Moodle logs, and performs some statistical calculations. The lifetime of GISMO data corresponds to the length of time of your Moodle logs.

It has Accesses overview

A graph reporting the student's accesses to the course. Accesses to the course

A graph reporting accesses for each student in a timeline. Accesses overview on resources

A graph reporting the number of accesses made by the students to the resources of the course

Assignments overview A graph reporting the submission of assignments. Color is mapped to the grade

assigned by the teacher. Quizzes overview

A graph reporting the submission of quizzes. Color is mapped to the grade. Resources accesses overview

A graph reporting an overview of the number of accesses to resources of the course.

Resources accessed by a particular student A graph reporting an overview of the student's accesses to resources on a timeline.

Students' accesses to resources A graph reporting, for each student, the number of accesses to resources of the

course.

A graph reporting the student's accesses to the course.

A graph reporting accesses for each student in a timeline.

A graph reporting the number of accesses made by the students to the resources of the course

A graph reporting the submission of assignments. Color is mapped to the grade assigned by the teacher.

A graph reporting an overview of the number of accesses to resources of the course.

Macquarie, UNSW and Netspot are working on a new student centric analytics tool for Moodle

SNAPP

http://www.snappvis.org/

SNAPP

The Social Networks Adapting Pedagogical Practice (SNAPP) tool performs real-time social network analysis and visualization of discussion forum activity within popular commercial and open source Learning Management Systems (LMS).

It essentially serves as a diagnostic instrument, allowing teaching staff to evaluate student behavioral patterns against learning activity design objectives and intervene as required a timely manner.

Dawson, S., Macfadyen, L., Lockyer, L., & Mazzochi-Jones, D. (2011). Using Social Network Metrics to Assess the Effectiveness of Broad-Based Admission Practices. Australasian Journal of Educational Technology, 27(1), 16-27. Also available from: http://www.snappvis.org/?page_id=4

Info from Shane Dawson (UniSA) Social interaction is one of the most important

of student behaviours and predictors of success. Student networks are the “single most potent source

of influence.” The tool provides a visualisation of social

networking. Different patterns are available to the individual, and mechanics which allow the data to be manipulated for different purposes.

It demonstrates that with students, like responds to like; they form self-regulating structures. It may be possible to manipulate group structures so that high-

performing students can assist low-performing ones. It may also be possible to direct teachers’ time to areas of need.

Dawson, S., Macfadyen, L., Lockyer, L., & Mazzochi-Jones, D. (2011). Using Social Network Metrics to Assess the Effectiveness of Broad-Based Admission Practices. Australasian Journal of Educational Technology, 27(1), 16-27.

Learning Analytics for Understanding

Low 10% student located in network

Students with a grade >75% < 90%

Social Network Analysis

Available from http://grsshopper.downes.ca/about.htm

Available from http://grsshopper.downes.ca/about.htm

Refining the signals from the Twitter feed

http://mashe.hawksey.info/2012/11/cfhe12-analysis-summary-of-twitter-activity/

BIM

David Jones FoE BIM (BAM into Moodle). BAM = Blog Aggregation Management. BIM is a Moodle module that supports an activity where:

Each student registers an individual external web feed. The feed might be generated by a blog, twitter or any other tool that produces a web feed. It's the student's choice what they use.

Each student uses that external feed to respond to a set of questions. Currently, those questions usually encourage the student in reflecting on their learning, often in the form of a reflective journal.

There is no need to have a set of questions. it maintains a copy of each students web feed, and attempts to

allocate student posts to the questions. it allows different teachers to track, manage and mark posts for

different groups of students. Allows a coordinating teacher to allocate teaching staff to different

groups, track their marking progress and all student activity. Student results can be sent to the Moodle gradebook.

Jones, D. 2013. BIM – Feed Aggregation. Available from http://davidtjones.wordpress.com/research/bam-blog-aggregation-management/

ACODE prepared a literature review containing 165 categorised references in an Endnote library

Learning analytics for our systems

The big Q Your big Answer

Which systems?

What tools?

How big do we want the data?

Is it just our USQ systems?

What do we want to know?

How do we want to use this data?

Who gets involved?

Who makes the decisions?