evaluating models of human performance: safety-critical ... · national aeronautics and space...
TRANSCRIPT
National Aeronautics and Space Administration
www.nasa.gov
Evaluating Models of Human Performance:
Safety-Critical Applications
Michael Feary HSS Technical Lead
HFES Annual Meeting May 24, 2012
https://ntrs.nasa.gov/search.jsp?R=20130013550 2018-06-03T17:19:47+00:00Z
Human Performance Modeling in Safety-‐Cri7cal, Mixed-‐Ini7a7ve Domains
• Increased use of automa7on emphasizes need for evalua7on of aAen7on
• Focusing on evalua7ons of acceptability as well as op7mality
• Need to account for distribu7ons of behavior
• Tasks as the linkage for complex system evalua7on
Cogni7ve Modeling for Safety Cri7cal, Mixed-‐Ini7a7ve Applica7ons
• Suppor7ng Tradeoff Decisions -‐ Social Penetra7on: Characterizing the problem space
• Focus on evalua7ng the correct ques7ons
-‐ Metricity: Avoiding the Illusion of precision • Applicability, Uncertainty
-‐ Robustness: Defining strategic varia7on
-‐ Data Applicability: Iden7fying the measurement needs
Limita7ons: Bridging the Granularity Gap
• New architectures – Bounded Op7mality – focus on the
problems the human has – Strategy Genera7on – focus on how the
goals could be achieved, strategic varia7on
(Brumby, Howes, Salvucci, 2007)
Insights obtained from FAA discussions Toward useful(?) modeling outputs and metrics
The “Anchor and Expand” Approach
In some cases, data may beavailable from a limitedsubject population on alimited task set in a limited(set of) evaluationenvironment(s).
Such empirical human dataprovide an anchor for theanalysis, from which we canexpand—throughmodeling—to new subjectpopulations, new tasks, andnew evaluation scenarios. 0 10 20 30 40 50 60
0.0
0.5
1.0
1.5
2.0
Multi−tasking Strategy Space Analysis
Device Task Duration (s)
Late
ral D
evia
tion
(m)
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!! !
!
!
!
!
!
!
!
!
! !
!
!
!!
!
!!
!
! !
!
!
!
!
!
!
! !
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!!!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
! !
!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!! !
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!!
! !
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
! !
!
!
!
!
!
!
!
!!
! !
!
!
!
!!
!
!! !
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
! !!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!!
!
! !
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
! !
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
! !
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!
! !
!
!
!
!
!
!!
!
!
Wheel crosses lane boundary
!FASTEST
!SAFEST
!
HUMAN DATA
Lewis (University of Michigan) Michigan/NASA Modeling Task 20 July 2011 12 / 24
Limita7ons: Bridging the Granularity Gap • New Tools for Domain Experts
– CogTool
• New empirical measures – Focus on aAen7on – eye tracking – fnirs images
– EEG
CogTool (John et al., 2004) SanLab, Gray
Portable fNIRS
Thank You
Taxonomy