effective? use of vital (in engineering) tim bullough, engineering with thanks to: john marsland,...

Effective? use of VITAL (in Engineering)

Tim Bullough, Engineering

with thanks to:John Marsland, Electrical Engineering and ElectronicsMark Johnson and Richard Dodds, Engineering

University of Liverpool L&T ConferenceJune 06

QUESTION

What Faculty are you from?

A. Arts B. SES C. ScienceD. Medicine/Vets E. Engineering F. Teaching Support (CLL…) G. Other

Press your response, then press SEND

1/1

Blackboard (VITAL) is a VLE (virtual learning environment)

Module information Learning resources• weblinks• interactive self-study• lecture handouts/notes

Communications• discussion forums• file exchange• online chat

Online assessment• tests and surveys• formative and summative• feedback to students• automated marking and analysis• student evaluation

Module management• student registration• student tracking• resource control• group-work

In theory a VLE should:

• allow individual and self-paced learning

• direct students to lots of data/information

• encourage student-student interaction (peer learning)

• allow testing of student understanding

• provide guidance when errors or misconceptions are found

• be cheaper and less time-consuming than conventional teaching

Still not clear to e-learning designers or teaching staff:

what students expect from e-learning

how to motivate students to learn using the technologies

how to bring in human interaction.

how to blend with conventional teaching

QUESTION

Do you use VITAL for teaching?

A. Yes (a lot)

B. Yes (moderate)

C. Yes (very occasionally)

D. Not at all

E. don’t teach


1/4

QUESTION

What does VITAL stand for?

A. Very Irritating Test for fAiling Lecturers

B. Virtual Interactive Teaching at Liverpool

C. Version One To Aid Learners

D. Vital Initiation Tool for Archiving Lectures

E. Don’t Know


2/4

QUESTION

What software does VITAL use?

A. WebCT

B. Whiteboard

C. Blackboard

D. LUSID

E. Access


3/4

QUESTION

What topic are you most interested in?

A. Formative assessment

B. PBL

C. Group/teamwork

D. Student feedback/evaluation

E. All of the above


4/4

(Very) short history of VITAL/Blackboard at Liverpool• WebCT trail in 02/03• Launched VITAL(Blackboard) in May 03• Started properly in 03/04 session

Peak monthly usage:

Session Active users (unique) Modules03/04 9572 (Mar 04) 351204/05 12453 (Nov 05) 418105/06 14329 (Jan 06) 5006 (24.5million hits)

Staff using VITAL:

BioSci Chem Engg Geog Health Law Sociol Mangt VetMar 04: 25 20 23 23 - 25 22 41 21Mar 05: 48 28 32 21 47 20 24 44 27Mar 06: 63 45 54 32 60 26 26 58 48

VLEs: A National PictureUCISA/JISC national survey 2005 into VLEs in HE:

• VLEs in ~95% of institutions

• Blackboard (43%) and WebCT (37%) dominate

• Subject take-up (pre-92 HE): Business/Law (85%)

English/History/Psychol/Sociol (72%)

Languages (70%)……..

Medicine/Engineering/Phys.Sci (52%)

• 46% of respondents do more than just post lecture notes and weblinks

Barriers to development of e-learning:

1. lack of time

2. lack of money

3. lack of support staff

4. lack of academic staff development

5. lack of incentives/career development

Very little concern about technical issues, student demand,

institutional strategy etc.

VLEs: A National PictureUCISA/JISC national survey 2005 into VLEs in HE:

Engineering in HE:

Engineering students…. in Yr 1:

• Don’t know how to self-learn

• Don’t know how to access/evaluate information

• Not comfortable mixing maths and physics

• Have no idea at all about how well they are learning at University - but they really want to know!

Engineering programmes:

• High direct contact hours (20 lectures/week + labs)

• Lecture-based modules compete with 100%-coursework modules for the attention of students

A little bit about Engineering…….

Industry skills/issues….& ManagementTeamwork & Problem solving

Applied Maths/Physics topics Engineering Design

+

Effective? use of VITAL (in Engineering):

1. Formative assessment

2. Developing a Student Case-Study: Sustainable Development

3. PBL and group-work

4. Collecting student Feedback & Evaluation

5. Other things: recorded lectures, e-lectures, summative assessment

QUESTION

Which quotation do you think best describes “Formative assessment” …?

A. A range of …. assessment procedures ….. to modify and enhance learning and understanding.

B. A process to recognise and respond to student learning in order to enhance that learning, during the learning.

C. The provision of information as feedback to modify the teaching and learning activities.

D. Providing the capacity for students to monitor the quality of their own work during its production, so they can improve it.

E. Any assessment designed to promote further improvement of student attainment

Press one response, then press SEND1/2

Press response, then press SEND

QUESTION

Have you used the VITAL assessment tool in your teaching?

A. Yes (auto marked, formative) B. Yes (auto marked, summative) C. Yes (manually marked, text responses), D. No (but have been considering it),

E. No (can’t see it is relevant to my teaching)

F. Don’t teach

2/2


(me and John Marsland)

Feedback-assisted learning and self-diagnostic:

• scores do not contribute (much) to

grades

• smaller, more frequent assessments

• detailed (immediate) feedback to

every student

• tutor and students can monitor

progress

• students can take the tests as often as

they wish to assist learning and revision

What is formative assessment?

Benefits (in theory)

• anywhere anytime

• students work at their own pace

• enables frequent small tests

• provides rapid feedback

• can include multimedia etc

• analysis of results automated

• eliminates marking burden/errors

• scales well for large groups

Limitations

• investment of time & effort

• cannot test “higher order”

knowledge

• requires access to networked PC

• staff need to learn new skills

• for summative assessments:

reliability, security and validation

Computer-aided assessment

Assessment in Blackboard (VITAL)

Types of questions:

True/False (easy, but of limited use)Multiple Choice (good for simple recall of facts, and calculations)Multiple Answer (tests recall and relationships)Ordering (tests recall of sequences, but difficult to think of questions)Matching (same as mcq. Can be used for feedback)Fill in the Blank (facts and spellings, but need to be careful to include all variants)Calculated formula (question includes an equation with different random variables for each student)Hot Spot (student has to point to specific area on image)Essay/Short Answer (text box - needs manual marking)

• Include figures, attached documents, web-links, videos, etc…..• Randomise questions (from pool?)?

Formative assessment using VITAL:what’s been tried in Engineering?

• Both taught traditional Yr 1 engineering lecture-modules, with one “class-test” or paper-based “home-work” problem sheet around wk8

• Major coursework marking burden for 100-300 students in Yr1 classes

• More importantly, wanted to improve student engagement with taught material during the semester, not just in the week before the exam.

• + Give new Yr 1 students a better idea of their understanding?

BUT

How much time & effort to set-up, mark and maintain?

How do you get students to do the assessments?

Do you provide academic support to any students who have problems?

(Me and John Marsland)

Year 1 Structural Materials (265 students)

2003/4: • ~100 mcq’s put into VITAL as “Practice Questions” But only the

best students used practice questions!• Single mid-semester class-test, and mcq exam no better than before

2004/5 onwards: • Replaced paper class-test with four VITAL assessments (5% each).

Tests 1 and 2: knowledge/understanding of technical information.

Tests 3 & 4 include some real engineering problems. • Retained Practice Questions, but linked explicitly to the formative tests

Pre-2003/4: • No VITAL. Single mid-semester paper class-test, and mcq exam.

(me)

Create content area for the assessments

encourage peer discussion

~2 weeks to complete test

Attempt ALL questions (negative marking?)?

Scaling?

Save and resume?

Ask for feedback

Give clear instructions

Delay score-return by 1 week

Types of questions: factual knowledge…..

decide a point value for each question

Effective questions?• clear, succinct questions and

responses• don’t give away the answer (to

other questions)• avoid ‘negative’ questions?• are the wrong responses

plausible (esp for calculations)?

avoid negative questions?!

Types of questions: simple numerical questions….

Effective questions?

Are the wrong responses plausible

Include images and links to websites….

Language subtleties and over-complicated questions can be impossible for non-native English speakers:

• Optional and instantly marked

• >120 questions

• don’t count

towards module

mark

• marked instantly

• ~50% of practice questions are "directed related" to test and/or exam questions

Practice Questions…. to replace direct tutor support

Attempt rate for Practice Questions >60-70%

Practice Questions (cont.)

Detailed worked solutions to numerical questions provided after test marked

tutorial support (cont.)

MARKING automatic and IMMEDIATE when test submitted: returns

“score only”, “score and correct answers”, “score and feedback”,

UNLESS……

Include a (0-point)

short-answer text

feedback

question: delays

disclosure of score

and provides

feedback

Release marks

manually later on…

Submission rate for formative VITAL tests 90-95%

Missing marks are often students who have left the Univ, NOT non-submissions

Gradebook: Alphabetic “Grade List” shows all students’ scores for all tests:

individual student's responses

Summary of all responses to each question in Assessment Attempt Details …..

Assessment Attempt Details: provides feedback about level of whole class understanding:

2005/6 VITAL usage (MATS112)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

05-Sep-05 25-Oct-05 14-Dec-05 02-Feb-06

Date

Hit

s e

ac

h d

ay

ExamStart of sem

Test 1&2

Test 3

Test 4

Did it work?

• Good test submission (95% in 05/06) and Practice Question usage (~70%) • Most usage just before test deadlines:

• Regular feedback comments VERY informative: discussed in class • Response data indicates topics they are having problems with

…..Probably added about 5% to average exam performance

03/04: 1 class-test + optional VITAL practice questions

Exam 41.5%

04/05: 4 VITAL tests + linked Practice Questions

Exam 49.5%

Exactly the same exam!

Problems:

• Need to be very careful with language, esp for Yr1 sem1 OS students

• “work-around” to temporarily withhold marks on test submission

• (VITAL Equation Editor not very good for Engg)

Did it work?

EFFORT: 2-3 weeks to set up questions, and ~2 days each year to maintain

Now:

• 8 x weekly VITAL mcq-type 10-question assessments

• Each test worth 2-3%

• Best to allow each test to be taken at any time over 3 week period

• Students told to work alone, but working together ok

• Return “score only” instantly on submission of test. Correct responses available 3 weeks after deadline. No other feedback/help given/needed.

Year 1 Digital Electronics 250 students Year 1 Audio Electronics 50 students

Year 1 Electronic Circuits and Devices 80 students(John Marsland)

2003/4: • One in-term paper-based homework assignment - lots of marking. • Relatively poor exam performance: little student engagement with lecture material.

Problems: • Late/incorrect registrations in year 1 sem 1.• (Some Equation Editor problems).

Did it Work?

• Excellent completion rate: ~90% submission

• VERY positive student feedback

• Interaction with students probably greater than if no VITAL for large

classes: students view it as active participation

• Excellent for tracking student retention/drop-out in Yr1 sem1

EFFORT: about 1 week/module to set up questions, ~1 day/session to maintain

Year 1 Digital Electronics 250 students Year 1 Audio Electronics 50 students

Year 1 Electronic Circuits and Devices 80 students(John Marsland)

“Engineering students are fundamentally lazy”. Do the formative assessments really improve student performance, or is it the threat of failure in any assessment that motivates and rewards them so they attend more lectures and labs, and do a bit more private study?

…. a few final comments on using Blackboard (VITAL) for (formative) assessment:

• are there educational benefits? [yes]• are there any other advantages (financial, time)? [yes]• are there problems? [yes]• does it integrate with the rest of the module? [yes]• how much additional effort is required...

• to create the online assessments and surveys? [quite a lot]• to manage their use? [a little]


(Richard Dodds and me)

QUESTION

In what year did the first definition of sustainable development appear?“to ensure that it meets the needs of the present without compromising the ability of future generations to meet their own needs.” (The Brundtland Report)

A. 1887

B. 1922

C. 1964

D. 1976

E. 1987


1/3

QUESTION

Which one of the following is not one of the "Five Principles" of sustainable development (UK Govt 2005 Strategy, Securing the future)

A. Living within environmental limits

B. Achieving a sustainable economy

C. Promoting good governance

D. Using sound science responsibly

E. Ensuring a strong, healthy and just

society

F. Minimising pollution and waste


2/3

QUESTION

Have you ever used the VITAL

Assignment Tool and/or Word

templates?

A. Only the Assignment Tool

B. Only Word templates

C. Both Assignment Tools and Word

templates

D. Neither (but I know what they are)

E. Neither (don’t know what they are)


3/3

Teaching of Sustainable Development in Design to all Year 1 Engineers (250 students)

The Challenge for 2005/6 (I took over as coordinator)

• Retain individual assessment

• Wanted on-going individual formative feedback to improve quality of final document

• Marking load limit set at ~10 mins/student, with just one pg student and 1 academic staff.

• Individual paper-based “structured essay”, based on four lectures.• Only about 75% submission (typically 4-10 pages).• VERY disappointing report (both technical content and style).• VERY heavy marking load (6 academic staff).

Pre-2005/6

Approach taken

• VITAL Assignment Tool and WORD Template (drop-down

menus and limited text areas standard format)

• Word template issued via VITAL in wk1 and resubmitted part-

filled using Assignment Tool in 3 weekly stages, before FINAL

version submitted.

• Nominal marks (5%, 3% or 0) at each stage

• Optional feedback comments - provided for all at penultimate

version

• Electronic + hard-copy retained for marking FINAL version

Teaching of Sustainable Development in Design to all Year 1 Engineers (250 students)

Submit part-completed case-study each week……..

word template downloaded by student

part-completed word template doc uploaded as attached file

part-completed word doc for marking

mark

feedback

Assignment grading tool opened from gradebook…..

final doc submitted after 4 weeks

Advantages of process

• 80-90% submission at each stage: easy and quick for pg to mark

• Feedback comments possible (but takes time!). Used for penultimate version or when student wildly off-track.

• Much higher quality FINAL case-study (technically and structurally).

• Download all submitted work as a record (and for safekeeping).

• Weekly mark in VITAL (+comments) useful feedback to students: but also discuss general issues in subsequent lecture.

Suggested Improvements/problems

• Not all students use feedback comments before doing current version?

• Easy to accidentally delete all submitted documents!

• Chinese character filenames

3. PBL and group-work in VITAL

QUESTION

Which of these is NOT one of the 7 stages in a typical Problem Based Learning exercise?

A. Make sure everyone understands the problem precisely

B. Establish what members of the group already know about the problem

C. Formulate the questions which need answering to enable the problem to be solved

D. Go away and gather all the knowledgeE. Check to see if the problem can be solved, and

if not, reformulate new questionsF. Reflect upon the experienceG. Identify which students have not been pulling

their weight, and shoot them.


1/3

QUESTION

What is the main problem with PBL from a student's viewpoint?

A. They don't think they are learning anything

unless they hear it in a lecture/seminar

B. They don't like group-work because they

have to rely on other students

C. They find it difficult to manage their own time

D. They don't like group-work because it may

not be fairly assessed

E. They don't know how to critically evaluate

information

F. Something else

G. Don't know enough about PBL to comment


2/3

QUESTION

Have you used "Groups" and/or run a PBL activity in VITAL?

A. Just "Groups"

B. Just PBL

C. Both Groups and PBL

D. Neither, but do use group-work

and/or PBL in teaching

E. Neither


3/3

Year 2 Materials Design PBL

AIMS:

• Evaluate and solve a real engineering/science/design “problem”, with real current industrial relevance: 21st Century Autobody Steel

Each group acts as a team of (materials) engineers in the steel industry charged by their company’s Executive Board with producing a response to a very specific “threat” of aluminium replacing steel in the automotive marketplace (….. “The Problem”).

• Mimic group-working industrial practice and procedures, as far as possible.

• Make use of “industrial” specialists, input and visits

• Design The Problem to utilise excellent internet learning resources produced by MATTER

PBL Process: Groups of ~6. One 50-minute facilitated formal group "committee" meeting each week (chairperson, minutes secretary etc.)

VITAL: Becomes the Virtual Steel Company’s intranet.• Database of company resource documents• Agenda and Minutes (incl Actions) for each meeting - keeps tutor

aware of progress. • Limited access (3 questions/week) to “virtual consultants” via the

discussion board - technical support and steering groups.

2001/2: standard lectures and a group report

2002/3: PBL. Use WebCT for group activities (Agenda and Minutes of meetings etc) and tutor interactions

2003/4 onwards: PBL. Use VITAL for group activities, tutor support via "virtual consultancy", and test of technical understanding. Specific assessment of group's effective use of VLE.

Company resource database

Each group has its own site

technical support each week

Agenda, Minutes, web-resources, docs….

Effective use of the VLE is 20% of assessment!

Bainite group site

• 3 questions per week allowed• real expert used• insist on professional approach

Virtual Consultancy

Individual assessment in VITAL utilised to steer students to web-resources and technical issues….

Positive outcomes

• VITAL provides excellent framework for PBL tracking

• Virtual “consultants” provide tutorial support

• Always positive comments from facilitators and students

Minor problems

• Some group functionality limited (no tests in group sites,

and can’t rename some group activities).

• Intra-group peer-moderation too cumbersome in VITAL -

used paper.


• informally at module level• formally at Year/Programme level

QUESTION

Does your Department have a Dept-wide general VITAL site for all students?

A. Yes

B. No

C. Don't know

D. Not in a Dept


1/2

QUESTION

Does your Department use VITAL to obtain student feedback?

A. Yes, at module level

B. Yes, at programme/year level

C. Yes, regularly for various issues

D. Not aware

E. Not associated with a Dept.


2/2


• informally at module level: use the formative assessments

• formally at Year/Programme level

Use a (0-point) short-answer feedback question at the end of every formative assessment:

Does it work?

140 out of 250 submitted assessments included some comments

Can’t always do much about the issues - spend 15 minutes in a lecture discussing the issues


• informally at module level: use the formative assessments

• annually at Year/Programme level: trial this session

1026 users

Used our Department of Engineering VITAL site:

Set up an Annual Programme Review content area:

Divide into MSc(Eng)/MRes pgms & 6 BEng/BSc/MEng pgm groups:

Each ug pgm group divided into 3-5 years, and BEng/MEng:

Use same Annual Pgm Review SURVEY in each content area….

Identical 10 questions for every pgm/year 43 identical surveys:

all responses anonymous (no cross-correlation)

1. Question Refering to the modules you have studied this session, please tell us which module(s), or specific parts of modules, you feel have been taught particularly well, and give your reasons. (Your programme structure, giving module codes and titles, can be viewed via the link below. Please give both the module code(s) and title(s) if possible) Programme Structures UG 05/06 2. Question Refering to the modules you have studied this session, please tell us which module(s), or specific part of modules, you feel have been taught poorly, and give your reasons and suggestions for improvement. (Your programme structure, giving module codes and titles, can be viewed via the link below. Please give both the module code(s) and title(s) if possible) Programme Structures UG 05/06 3. Question Please give an overall rating of the quality of teaching you have experienced during the current session (1=excellent, 5=very poor): 4. Question What are your views on the quality of the lecture theatres and teaching rooms? Please specifically name any lecture theatres that you wish to mention. 5. Question What are your views on the Department's Student Support Office in the Derby building 6. Question What are your views on the general quality of the information you are given during the year (such as timetables, emails about changes to lectures, student handbooks, VITAL announcements, careers/vacation work information etc)? 7. Question What things have helped you most with your learning this session (examples could include...library resources, PC access at University/home, use of VITAL, access to teaching staff, personal development planning etc)? 8. Question What things have made learning difficult during this session? (examples....lack of background knowledge, having to work, poor accommodation, illness, social/sport commitments etc, and the examples in the previous question)? 9. Question Please give an overall rating of how much you have enjoyed your studies during the current session (1=very enjoyable, 5=not at all enjoyable): 10. Question What actions would you suggest the Programme Director or Department should consider in order to improve the current year of your degree programme for future students?

Annual Programme Review Questions….

http://www.liv.ac.uk/engdept/downloads/ughandbook05_06/Programme%20structures_1.doc

http://www.liv.ac.uk/engdept/downloads/ughandbook05_06/Programme%20structures_1.doc

Mainly text responses:

Rating-questions for year-on-year, or pgm-to-pgm comparison:

Access survey feedback comments via the gradebook:

1026 students

43 surveys

What sort of information do you get?

• Trial in 05/06: 3 emails over 4 weeks either side of Easter sem2

• ~20% response rate from Yr 1, decreasing to ~5% in Yr 4

• Need someone to process comments

• Need a strategy in place to deal with all the information

Did it work?

What do you do with the information?

Year 1 Aerospace feedbackYear 1 Aerospace feedback……

C:\Documents and Settings\Tim Bullough\Desktop\vital.liv.ac.uk-webapps-assessment-stats-stats.mdi

Potential advantages: • Learn at one's own pace, any time and place• Repeat a lecture for revision• Optimised delivery every time• Save time spent delivering lecture• Potential for highly flexible timetable and on-line courses

Potential disadvantages:• Lack of life/spontaneity in e-lectures. • No opportunity to ask questions - vital to have tutorials/problem

classes.• Lecturers could be redundant! …or more staff time spent interacting

with students in tutorials etc.

e-lectures: ppt slides with voice-over commentary

(Mark Johnson)

e-lectures: ppt slides with voice-over commentary

• Used NO live lectures at all - but additional tutorials/problem classes - for past 2-3 years for Year 4 CFD module (25-30 students) and Year 1 Thermo module (about 150 students).

• Use highly-animated powerpoint slides, with synchronised voice track (prepared using the powerpoint software)

• Takes about a day (8-10 hours) to convert a one hour lecture (i.e. payback only after 8-10 years!)

• about 50% preferred e-lectures and about 50% live-lectures: no indifference… either loved them or hated them!

• exam performance in 2005 and 2006 was as good as, or better than, before

• Sept resit exam performance even better

Did it work?

The Process:

Conclusions



3. PBL and group-work



QUESTION

How many VITAL hits in Jan 06?

A. 2 450

B. 24 500

C. 245 000

D. 2.45 million

E. 24.5 million

F. 245 million


1/2

QUESTION

How would you rate this presentation/session?

A. A complete waste of my time

B. Useful and interesting in small

parts

C. Mostly useful and interesting

D. Exactly what I came for

E. Would rather not say


2/2