stem 2012 automatically assessing free-form diagrams in e-assessment systems pete thomas kevin waugh...
TRANSCRIPT
STEM 2012
Automatically assessing free-form diagrams in E-assessment systems
Pete Thomas
Kevin Waugh and Neil Smith
Open University, UK
STEM 2012
Aim
• To extend the capabilities of e-assessment systems (online tests) to include questions that require students to draw free-form diagrams in answers.
• In particular:– Minimal constraints on what might be drawn–Not multiple choice– Support for a variety of diagram types– Automatic marking – Moodle VLE (via OpenMark)
STEM 2012
Example
Student draws diagramin response to question
STEM 2012
What is a diagram?• Free-form
–Not free-hand–Uses a drawing tool
• Specialised for the domain by the teacher• As unconstrained as the teacher wishes
• Graph based–Shows relationships between objects
STEM 2012
Biological Flow Diagram
Illustrates the flow of hormones between glands.
STEM 2012
Entity-Relationship Diagram
Shows the relationships (links) between entities (boxes)
STEM 2012
Use case diagram (UML)
Shows the actions that an actor Book Borrower takes part in when using a library
STEM 2012
Spray Diagram
Illustrates how UK farms have diversified in recent times.
STEM 2012
Spatial Relationships
It is the relative positions of boxes that determines whether a relationship exists. Adult and Child are inside Person – inside indicates subtype relationship..
STEM 2012
Sequences
time
The messages, represented by arrows, are sequenced in time.
STEM 2012
Question in VLE test
This is what the user sees in their browser
STEM 2012
Run the Drawing Tool
Up pops the drawing tool – an applet
STEM 2012
Automatic marker provides feedback
Feedback from the
automatic marker
STEM 2012
Get Answer
STEM 2012
Automatic Marking
• The student’s diagram is compared with the model solution(s) to see how well the objects and the relationships match.
• Similarity measures of how well each element on the student diagram matches with an element on the model solution.– A mixture of NLP and heuristics
• A marking scheme is applied.– Designed by the question setter from a template
STEM 2012
Does it work? – Experiment 1394 diagrams from a real exam - marked out of 7.
Difference between auto and human markers
0 0.5 1.0 1.5 2.0 3.5
Number 389 4 1 0 0 0
% 98.7 1.0 0.25 0 0 0
Cumulative % 98.7 99.7 100 100 100 100
Main corpus
Diagrams only have entity types and relationships
Published in 2011
STEM 2012
Comparison Human – Auto(Distribution of marks)
STEM 2012
Support Tools
• An authoring tool for teacher to design the question, the model solution(s) and the mark scheme.
• A specification tool to define a type of diagram. – Used to specialise both the drawing tool and the
automatic marker
• A marking tool for marking sets of diagrams off-line.
STEM 2012
Authoring Tool for Questions,Model Solutions and Marking Schemes
Question(text)
Model Solution(diagram)
STEM 2012
Specification Tool
A graphical user interface for defining a new type of diagram
STEM 2012
Summary
• A new question type for online tests• Free-form diagrams (graph based)• Automatic marking • Implemented in Moodle• Set of support tools
STEM 2012
Thank You
• We always want more examples of types of diagram. Send me your suggestions.
• There is a website where you can try this technology out. Please contact me for details.