cerias talk on testing and evaluation
DESCRIPTION
This slidedeck highlights the recent tech talk for CERIAS.TRANSCRIPT
BIOMETRICS – TESTING
AND EVALUATION
• Traditional biometric testing
• Algorithm testing
• Well established metrics
• Well understood testing methodologies
• Operational testing
• Harder to do
• Access to environments
• Test methodologies dependent in some cases on the test
BIOMETRIC TESTING
•Essentially trying to understand how a system
performs
• Maybe more fundamentally – who or what is
causing the errors?
TESTING AND EVALUATION
PROBABLY NOT MISSING,
BUT HARD TO DO
• There have been several papers on the contribution of individual error on performance
• What causes these errors?
• There are some papers that examine meta-data and the contribution of variables (age) and examines the training of algorithms
GAPS IN BIOMETRIC TESTING
•Training
• How do users get accustomed to devices?
• Can they remember how to use them?
• How do we provide good training to the users that has a consistent message?
GAPS IN BIOMETRIC TESTING
• Accessibility
• How many people know people who have problems with interacting with a biometric systems?
• How do we deal with accessibility and usability issues?
• Hearing and sight issues
GAPS IN BIOMETRIC TESTING
• Human Factors
• Testing and evaluating biometric systems by looking at how the users interact with the system
• Are performance results different in an operational environment than collect in a lab?
• Are these performance results due to the environment?
GAPS IN BIOMETRIC TESTING
• Is the error always subject centric?
• The role of the device?
• The role of the operator?
GAPS IN BIOMETRIC TESTING
FILLING IN THOSE GAPSA WORK IN PROGRESS
• Devices are:
• Now more complex
• Variety of form factors, deployments, applications
• In the hands of more customers
• They become more demanding
• Technology development cycle is short
WORK IN PROGRESS
• What are biometrics?
• Why should we test?
• Complexities of testing today
• Examples of testing
• Operational / Scenario / Hybrid
• Our approach to testing
• Upcoming tests
TODAY – AN UPDATE TO TESTING AND
EVALUATION
•How do we identify people?
•As we move around, the need to identify
individuals increases
INITIAL CONCEPTS
•Biometric – of or having to do with biometrics
•Biometrics – automated recognition of
individuals based on their behavioral and
biological characteristics
TWO MAIN DEFINITIONS
•Biometrics appear in a number of different
environments:
• Large scale government programs
• Individual identification on a laptop
INITIAL CONCEPTS
•Border control – US VISIT and Global Entry
•Apple fingerprint sensors
TWO EXAMPLES
•How do we test and evaluate these products to
make sure that we have good performance?
TWO VERY DIFFERENT SCENARIOS
• International Center for Biometric Research, and its predecessors have been testing and evaluating biometric products for over 14 years.
• The devices have changed over time, but the needs of testing and evaluation haven’t
• Companies want to test and evaluate products to prove that they will work in the marketplace
ICBR HISTORY
•Anyone know what this
is?
BIOMETRIC DEVICES AND
MODALITIES
• Iris camera
•Now what year was this
launched?
BIOMETRIC DEVICES AND
MODALITIES
• Iris camera
•2001
BIOMETRIC DEVICES AND
MODALITIES
• We have tested mobile iris, eye vein, voice, signature, face, palm, and two-factor authentications, all since 2001
• However, today, testing and evaluation is a lot more complex
• Biometrics has a huge market opportunity, but to deploy successfully, we need to test
SO MOBILE ISNT NEW!
TESTING AND EVALUATION
– THINGS TO CONSIDER
• What is the purpose of the test?
• To get an idea of the performance of the biometric
• To get an idea of the usability
• To calculate the throughput of an ABC gate
• To understand the differences of whether there is a difference in one group or another
• Take the example of the phone versus the border gate – what are we interested in?
WHAT ARE WE EVALUATING
•Understanding the problem statement is key
for a good test protocol
•Tests are expensive and time consuming, and
we want to be able to collect the data
efficiently, accurately, and within budget
WHAT ARE WE EVALUATING
• Discuss with the clients the different aspects of evaluation that you can do
• Such as straight performance through to understanding the issues relating to performance
• Sample size – the complexity of the test will drive the sample size
WHAT ARE WE EVALUATING
•Scenario
•Technology
•Operational
DIFFERENT TYPES OF TESTS
•The matching algorithms are then tested on a
sequestered dataset
TECHNOLOGY EVALUATIONS
•Takes place in a testing facility
•Matchers are installed in an “office
environment” and biometric devices are tested
SCENARIO EVALUATIONS
•Volunteers are recruited
•Group uses the system to collect over time
•This creates a set of databases that can be
used for technology evaluations later on
SCENARIO EVALUATIONS
• From Mansfield and Wayman:
• Provide a framework for developing and fully describing test protocols
• Help avoid systematic bias due to incorrect data collection or analytic procedures in evaluation
• To help testers achieve the best possible estimate of field performance while expanding the minimum effort in conducting their evaluation
• To improve the understanding of the limits of applicability of test results and test methods
SCENARIO EVALUATIONS
•This type of evaluation is used to determine
the performance of a biometric system in a
real-world environment
OPERATIONAL EVALUATION
• A combination of technology and scenario testing
• We want to solve these additional problems of what are these biometric system errors, and how can we fix them
• So we do:
• test protocol development, human subject testing, many different modalities
• Usability testing
• Surveys
• Focus groups
OUR TESTING
USABILITY TESTING
HUMAN BIOMETRIC SENSOR
INTERACTION
HBSI MODEL
•ABC gates are important, but these are
complicated from the perspective of biometric
testing
HBSI MODEL ADVANCES
•Token- the passport
•People traveling- potential incorrect passport
•Throughput issues- how to measure
CHALLENGES
TOKEN HBSI MODEL
Token HBSI Model
TOKEN HBSI MODEL
TOKEN HBSI MODEL
TOKEN HBSI MODEL
• HBSI and Border Gates
• Building a border gate in the center to test various technologies, including iris, fingerprint, documents
• Throughput research
• Usability using the Kinect
• Novel performance metrics such as the Stability Score Index
TESTING AND EVALUATION RESEARCH
• Mobile devices are important as well, and we continue to work in this area:
• Wild testing
• Illumination / Noise / Controlled environments
• Voice, face, signature, palm, and multi-factor authentication
TESTING AND EVALUATION RESEARCH
QUESTIONS