caveon webinar lessons learned at atp and nces
DESCRIPTION
Late February presented two auspicious testing industry events: ATP's “Innovations in Testing” Conference, and the US Department of Education’s National Center for Education Statistics' “Testing Integrity Symposium.”Join Caveon leaders John Fremer and Steve Addicott as they share the Top Ten Security Lessons Learned from the two events.TRANSCRIPT
Upcoming Caveon Events
• NCME in Vancouver, April 12-16– NATD Breakfast Plenary, Dr. John Fremer, April 13
• CCSSO TILSA SCASS at National Harbor, MD• Conference for the Statistical Detection of Potential Test
Fraud, University of Kansas, May 23-24• National Conference on Student Assessment in
Minneapolis, June 26-29• International Testing Commission Conference in
Amsterdam, July 3-5• National Collegiate Testing Association in Minneapolis,
August 1-4
www.caveon.com
Caveon Online• Caveon Security Insights Blog
– http://www.caveon.com/blog/
• Twitter– @Caveon
• LinkedIn– Caveon Company Page
– Caveon Test Security Group• Please contribute!
• Facebook– Will you be our “friend?”
– “Like” us!
www.caveon.com
Caveon Webinar Series
“Lessons Learned at ATP and NCES”
Steve Addicott Dr. John FremerVice President PresidentCaveon Caveon Consulting Services
March 28, 2012
Agenda
• ATP 2012 Security Lessons Learned– Steve Addicott
• NCES 2012 Security Lessons Learned– Dr. John Fremer
• Q&A
About Caveon….
• Our mission: To help protect the tests, programs, and reputations of our clients, some of the most important test programs and test publishers in the world– Certification & Licensure– State DOEs & School Districts– DOD– International
• Entering our 9th year of operation• Provide practical, measureable security
enhancements
www.caveon.com
Test Security Highlights from ATP
Theme:
• A dozen sessions focused on security– Too many at the same time!
• Caveon sessions with our clients– Conducting investigations– Managing the media in cheating scandals – How to implement Data Forensics– Embedded verification tests
• Security Committee breakfast – Standing room only, despite the 7am start time!
Key Industry Veterans Focusing on Security
• Liberty Munson, Microsoft
• Chad Buckendahl, Alpine Testing
• Mark Poole, Pearson VUE
• Cathy Donath, Donath Group
• Ray Nicosia,
ETS
• Ashok Sarathy, GMAT
• Steven Barkley, CFP
• Daniel Eyob, GMAT
• Nikki Eatchel, Questar
Conducting Investigations
Ben Mannes, Director of Test Security, ABIM
Lessons Learned• Was a security breach (due to a possible vulnerability in your
policies or infrastructure) identified during the incident?
• If so, what can be done to change policies, procedures, or technologies to fix it?
• A cost/benefit analysis should be done to address the vulnerability identified by the incident.
• Are there other ramifications to the organization from the incident?
Gimme Shelter: Managing the Media Storm of a Security Breach
Jim Vasalek, Sr Director of Public Affairs, LSAC
NON-Traditional MediaBlogs, bulletin boards, chat rooms, file-sharing sites, and social
mediaNo professional code of ethicsNo editorsNo fact checkersNo obligation to get your side of the story
Can keep a story alive well after traditional media has lostinterest, even well after it actually has died
May be more influential with your key audiences than traditionalmedia
Implementing Data Forensics
Aimee Hobby Rhodes, Director of Exam Security, CFA Institute
Collect All the Evidence You Can!!• Use multiple statistical analyses
– Similarity Analysis, Erasure Analysis, Prior score analysis (gains/losses), identical response analysis, etc
• Suspected Cheating/Incident Reports• Seating Chart/Floor plans• Candidate Connections• Anything else you can find!
Implementing Data Forensics
Jennifer Semko, Partner, Baker & McKenzie
Your Candidate Agreement is foundational!
• Are candidates on notice that – sharing items is a breach?– studying from recalled items is improper?– Data Forensics may be used?
• Did you reserve the right to:– Invalidate scores? Suspend access to examination? Take
other action?
• What are the grounds for action? Is there a “catch all”?
• Do you:– regularly review your agreement language?– have uniform security procedures and policies in place?
Embedded Verification Test
Liz Burns, Senior Manager of Technical Certifications Programs, Juniper Networks
Key Benefits of New Security Tactic, EVT• Legally defensible “proof” for enforcement actions
– Does not require “intent”• Enforces against unqualified candidate• Works on broad range of content theft• Fairly inexpensive to do• Provides intelligence on vulnerability of your exams• Can do “results hold” for proactive enforcement
– Changes definition of “PASS” to reaching the cut score AND the fraud detection threshold
Social Media
• Large emphasis placed on Social Media at the conference. – twitter hash tags, QR codes, or LinkedIn links
• Many were tweeting and blogging about the conference during live sessions– Getting the word out instantaneously.
• Get on the train….NOW!– be a part of the online conversation– how information is being shared.
Security Committee Breakfast
• Standing room only…really– Early morning didn’t deter anyone– Despite myriad “celebrations” the evening before
• Lots of important work– Security Survey– Enforcement– Lobbying– Standards
• All volunteers– Pat yourself on the back!!
JOHN FREMER – CAVEON TEST SECURITYSTATE ASSESSMENTS IN TRANSITION
JOHN FREMER – CAVEON TEST SECURITYSTATE ASSESSMENTS IN TRANSITION
Test SecurityTest Security
StormStorm
The PerfectThe Perfect
SECRETARY DUNCAN
• “States, districts, schools, and testing companies should have sensible procedures in place to ensure tests accurately reflect student learning.”
• Quoted in recent Atlanta Journal Constitution article– “Cheating our children: Suspicious school
test scores across the nation.”
ETS – CARSWELL WHITEHEAD
• All High Stakes Tests are Subject to Attempts to Gain an Unfair Advantage
• Must Build in Processes to Detect Irregularities– Before– During– After
GREG CIZEK – UNC, CHAPEL HILLPREVENTION OF IRREGULARITIES IN
ACADEMIC TESTING:
• Test Providers / Contractors
* Clear definition of cheating* Clear, educator-referenced materials* Web-based qualification utility, database?* Less corruptible formats* CBT/CAT delivery
DAVID FOSTER – KRYTERIONSEVEN DETECTION PRINCIPLES
1. Focus…Concentrate on high-risk threats first
2. Adapt…Match detection methods to the threat
3. Backup…Use layers of detection methods
4. Predict…Watch out for new threats
5. Filter…Rule out other explanations
6. Evaluate…Use detection to evaluate security
7. Plan…Set up to succeed
BRIAN JACOBS - UNIV OF MICHIGANDETECTING IRREGULARITES
• Benefits of statistical analysis– Relatively low cost– Covers the entire population– Measure of the “extent” of the irregularity – e.g.,
likelihood it would have occurred by chance– Systemic patterns as well as individual cases of concern
• Limits of statistical analysis– Like preliminary screens for rare diseases, statistical
analyses to detect cheating can have a high rate of false positives and false negatives
• Critical to complement statistical analyses with other methods
– Can never identify the individual responsible for the manipulation
JAMES LIEBMAN – COLUMBIA UNIVERSITY LAW SCHOOL
• Keep an open line for reports of infractions
• Teachers as allies
• Duty to report immediately
• Multiple locations (principal, monitors, local test office, state test office, “special investigations”)
• Allow anonymity; offer confidentiality
SCOTT NORTON – LA DEPT OF EDAreas for Improvement at State Level
• More state oversight is needed for district-led investigations.
• Standardization across states may be needed for established procedures such as erasure analysis.
• Better information is needed about other statistical analysis procedures for detecting suspect patterns of responses, unusual gains or losses, etc.
STEVE FERRARA – PEARSON
• [Speaking primarily as a former Maryland SAD]
• School Personnel Not Well Equipped to Handle Investigations
• Don’t have Training
• Teaching Draws on Nurturing and Supporting Skills
John Fremer – Caveon Test Security
Ten Recommendations Moving Forward
John Fremer – Caveon Test Security
Ten Recommendations Moving Forward
1. Acknowledge the seriousness of security issues
2. Expect cheating and plan to be proactive
3. Use multiple detection methods and forensic statistics
4. Minimize testing windows
5. Strengthen the chain of custody
Ten Recommendations Moving Forward…Cont. Ten Recommendations Moving Forward…Cont.
6. Increase the emphasis on security training
7. Allocate adequate resources for test security
8. Pilot techniques for detection of cheating
9. Continue to learn from others
10. Monitor new advances in anomaly detection and prevention (e.g. “Epidemiological Model”)
RESOURCES
• CCSSO/ATP – Operational Best Practices• ATP Security Committee• Caveon Test Security
– Blog– Webinars
• (In development) NCME Guidelines• (In development) TILSA Guidebook for State
Assessment Directors on Data Forensics
Continue The Conversation
- Follow us on twitter @caveon for updates and events
- Check out our blog…www.caveon.com/blog/
- LinkedIn Group – Join our Caveon Test Security group to ask questions and join discussions about test security topics
- Slides of this and past Webinars are available at www.caveon.com/resources/webinars
Thank you!
Steve Addicott Dr. John Fremer
Vice President President
Caveon, LLC Caveon Consulting Services