MERLOT: The Peer Review of Digital Scholarship

Professor Cathy Owens Swift

Georgia Southern University

Professor Susan M. Moncada

Indiana State University

Professor Theresa B. Flaherty

James Madison University

AACSB (April 26, 2003)

Presentation

• What is MERLOT? – Swift

• The Peer Review Process – Moncada

• Benefits of Submitting Online Learning Materials to MERLOT & Peer Reviewing – Flaherty

• Benefits to You (Dean) - Swift• Questions or Comments

What is MERLOT?

Cathy Owens Swift

Georgia Southern University

Common Issues

- Quality Control- Dissemination- Sustainability

• Institutions invest in developing instructional software without provisions for:

• Lack of quality, interactive, web-based learning materials

• Faculty often work in isolation while developing online courses

Evaluation of Digital Scholarship

• Chronicle of Higher Education (6/2/2000) “Counting Digital Scholarship & Teaching in Faculty Evaluations.” – Need peer review of teaching innovations

• Syllabus, (1/2001) “Faculty Rewards in Digital Instructional Environments”

• Carnegie Teaching Academy:– Scholarship of teaching definition includes “peer review”

• Physics Model of Scholarship of Teaching:– Discussions on “review and evaluation by acknowledged

national experts”

• Faculty should remain in control of the teaching/learning process.

• Peer-reviews will contribute to expanded use and effectiveness of digital learning materials.

Assumptions

• Sharing of materials will maximize everyone’s investments.

• Faculty need mechanisms to document teaching and learning contributions.

Vision & Mission

• MERLOT’S VISION is to be the place where faculty from around the world will share teaching-learning materials and pedagogy.

• MERLOT’S MISSION is to improve the effectiveness of teaching & learning by expanding the quantity and quality of peer-reviewed online learning materials that can be easily incorporated into faculty designed courses.

Aspects of MERLOT• MERLOT is a COOPERATIVE

• Institutional Partners• Organizational Affiliates• Individual Members

• MERLOT is a SET OF PROCESSES• Building, organizing, reviewing, and developing applications of

online teaching-learning materials • Building and sustaining online academic communities.

• MERLOT is SOFTWARE• A user-centered, searchable database of online learning

materials, pedagogical support, and people.

MERLOT follows the model of peer review of scholarship

IndividualReview #1

IndividualReview #2

CompositeReview

MERLOT Alliance Partners

•National Science Foundation (NSF) Grant • $400,000• Online peer review training module

• NLII of Educause• AAHE – American Association of Higher Education•Education.au Limited•IMS Global Learning Consortium, Inc•Health Education Assets Library•NEEDS – Digital Library for Engineering•SMETE.org

Discipline Communities

Business

Biology

Chemistry

Engineering

Health Sciences

History

Info Tech

Music

Math

Physics

Psych

Teacher Ed

Teachingw/Tech

World Languages

Parties Involved in MERLOT

• Administration Team (12)• System Partners (22)• Campus Partners (2)• Discipline Editorial Boards (14)

– Editors/Co-Editors (25)– Editorial Review Board Members (150)

• Peer Reviewers (100+)

• Others – Authors, Submitters, Users

Institutional Partner Commitments

• Faculty Reviewers– 6-8 faculty to serve as experts

• Faculty Development & Academic Technology Personnel

• Travel– Training, presentations, meetings

• Project Director (10+%)– Planning, coordinating, budgeting, etc.

• Participation Fee: $25K

MERLOT: The Peer Review Process

Susan M. Moncada

Indiana State University

Editorial Boards

Support development of discipline communities (14)

Editorial Review Board Members (150)– Editors/Co-Editors (25)– Associate and Assistant Editors– Peer Reviewers (100+)

Support development of discipline communities (14)

Editorial Review Board Members (150)– Editors/Co-Editors (25)– Associate and Assistant Editors– Peer Reviewers (100+)

Board Qualifications

Expertise in scholarship of their field Excellence in teaching Experience in using technology in

teaching and learning Connections to professional

organizations Experience in conducting peer reviews

of online learning resources

Expertise in scholarship of their field Excellence in teaching Experience in using technology in

teaching and learning Connections to professional

organizations Experience in conducting peer reviews

of online learning resources

Board Responsibilities

Expand and manage the collection Implement the peer review process Post peer reviews Recruit and train peer reviewers Education and outreach to the

community of educators

Expand and manage the collection Implement the peer review process Post peer reviews Recruit and train peer reviewers Education and outreach to the

community of educators

MERLOT’s Business Discipline

• Accounting• Business Law• Economics• E-commerce• Finance• General

• Accounting• Business Law• Economics• E-commerce• Finance• General

• Information Systems

• International Business

• Management• Marketing

• Information Systems

• International Business

• Management• Marketing

TYPES OF MODULES

• Simulations

• Tutorials

• Animations

• Drills & Practice

• Simulations

• Tutorials

• Animations

• Drills & Practice • Quiz/Tests

• Lecture/Presentations

• Collections

• Reference materials

• Quiz/Tests

• Lecture/Presentations

• Collections

• Reference materials

Profile of the MERLOT Learning Module

Title, author and affiliation Peer Review Link User Comments Link Type of learning material Location (URL for the module) Subject classification Description Submitter Audience

Title, author and affiliation Peer Review Link User Comments Link Type of learning material Location (URL for the module) Subject classification Description Submitter Audience

Editor’s Evaluation Process

• Stage 1:– Cursory review to identify worthy modules– Post triage comments and triage value online

• Stage 2:– Editor assigns “worthy” materials to reviewers– Reviewers apply MERLOT standards to write

reviews.

• Stage 1:– Cursory review to identify worthy modules– Post triage comments and triage value online

• Stage 2:– Editor assigns “worthy” materials to reviewers– Reviewers apply MERLOT standards to write

reviews.

MERLOT follows the model of peer review of scholarship

IndividualReview #1

IndividualReview #2

CompositeReview

Editor’s Evaluation Process

Stage 3:Editor sends review to author for feedback and

permission to post– Authors can elect to modify materials and

request review be modified– Authors can request module be pulled from the

repository– Authors can request 2 letters from MERLOT

summarizing peer review process and report to 2 people of their choice.

Stage 4:Peer review is posted

Stage 3:Editor sends review to author for feedback and

permission to post– Authors can elect to modify materials and

request review be modified– Authors can request module be pulled from the

repository– Authors can request 2 letters from MERLOT

summarizing peer review process and report to 2 people of their choice.

Stage 4:Peer review is posted

Standard Evaluation Criteria (Strengths & Concerns)

1. Quality of Content

1. Quality of Content

2. PotentialEffectiveness

2. PotentialEffectiveness

3. Ease ofUse

3. Ease ofUse

1. Quality of Content

Current and relevant Accurate information Clear and concise Informed by scholarship Completely demonstrates concepts Flexibility Integrates/summarizes concept well

Current and relevant Accurate information Clear and concise Informed by scholarship Completely demonstrates concepts Flexibility Integrates/summarizes concept well

2. Potential Effectiveness

• Specifies learning objectives• Identifies prerequisite knowledge• Is very efficient • Reinforces concepts progressively• Builds on prior concepts• Demonstrates relationships between

concepts

• Specifies learning objectives• Identifies prerequisite knowledge• Is very efficient • Reinforces concepts progressively• Builds on prior concepts• Demonstrates relationships between

concepts

3. Ease of Use

• Is easy to use• Has clear instructions• Is engaging• Has visual appeal• Is Interactive• Uses effective navigation techniques• All elements work as intended

• Is easy to use• Has clear instructions• Is engaging• Has visual appeal• Is Interactive• Uses effective navigation techniques• All elements work as intended

Star Rating System

Excellent all around

Very good w/few minor concerns (4.0-4.9)

Meets/exceeds standards with some significant concerns (3.0-3.9)

Standards not met, some limited value (2.0-2.9)

Not worth using at all (1.0-1.9)

Standards for Scholarly Work*

• Endeavors require high level of discipline expertise

• Breaks new ground and is innovative• Is of significance• Can be replicated or elaborated upon• Can be documented• Has the potential to be peer reviewed * (Merton, 1973)

• Endeavors require high level of discipline expertise

• Breaks new ground and is innovative• Is of significance• Can be replicated or elaborated upon• Can be documented• Has the potential to be peer reviewed * (Merton, 1973)

Benefits of Submitting Online Learning Materials to MERLOT & Peer Reviewing

Theresa B. Flaherty

James Madison University

Who contributes materials to MERLOT?

• MERLOT materials are added by people who have joined MERLOT (MERLOT members). – Anyone may join MERLOT, and there is no

cost or other obligation.

• Materials may be added by:– the people who created them (author), or – any member who finds a great resource to

share with others (submitter).

How to Contribute Modules to MERLOT as an Author

Step 1: Develop your Module

Step 2: Fill out a shortform to contributeyour material. Can send a note to Editorto request peer review.

The Peer Review of this Module

Avoids “reinventing the wheel”

Documents Teaching Efforts

Provides Networking

Opportunities

Search for members by discipline

Peer Reviews are developed through online

workspace

The peer review can be completed in different segments

After completion of the review, the Peer Reviewer is recognized for his/her contribution by a letter from the Editor. This letter can be sent to two other people as well.

To become a Peer Reviewer, an individual must:

• be an instructor at an institution of higher learning

• demonstrate expertise in the discipline• be recognized for excellence in teaching• have experience using technology in

teaching, and • have participated in the activities of the

discipline.

Benefits of MERLOT(Dean’s Perspective)

• Qualitative & Quantitative Dimensions to Teaching Efforts– Peer Review is Available for General Public

• Assist in Evaluation of Faculty Efforts (P&T)• Encourage Faculty to Develop Digital Materials

that benefit multiple disciplines• Encourage Faculty to Provide Service to their

Disciplines as Peer Reviewer– Training Provided by MERLOT

• Encourage Faculty to Use Digital Materials

Your Participation in MERLOT • System Partner• Campus Partner• Inform Faculty of Opportunities

– Author– Submitter– User– Peer Reviewer

• Conduct Faculty Training• Recognize MERLOT Contributions of Faculty• Join MERLOT

Questions or Comments?