Neo4j in the Future of Interaction Design
A novel approach to gesture recognition integrating Neo4j with the Leap Motion
This Talk
! Introduction ! Interaction Design
! The Tyranny of Finger-‐On-‐Glass ! The Leap Motion
! Promises and Limitations ! Gesture Recognition
! Current State-‐of-‐the-‐Art ! Building a New Strategy for the Leap
! Conclusions
Who am I?
! Education ! Work
My Collaborators
The Leap Motion
http://youtu.be/3b4w749Tud8
A Brief History of Interaction Design
Basic Technology and Indirect Mappings
Higher Layers of Abstraction
Fingers On Glass
Why is This Bad?
Enter The Leap Motion
Here are Some Live Demos
There Are Even Simple Gestures Included
But Something is Rotten in Denmark
! Complex Motions are infeasible ! Self-‐Obfuscation is a huge problem ! Interface is surprisingly exhausting ! Drivers are proprietary and imperfect ! Bounding box is small ! Data is fundamentally inconsistent
The Real Faceoff
Vs.
Developers?
Gesture Recognition
Problems with Classical Approaches to Gestures
! Geared towards easily benchmarked, previously studied problems.
! Primarily Developed by narrowly-‐defined industry applications
Hidden Markov Models
Problems With HMMs
! State depends only on current state, intuitive hand gestures are inherently hysteretic. ! Depends on discrete gesture-‐identification, no sense of “variations on a theme” ! Storage space exponentiates when faced with inconsistent data-‐streams ! NOT built for the Leap
Size?
! Minimum 6 DoF per finger + 7 for the palm ! 2 hands, even assuming only two modes of motion:
1.9 * 1022
Motion as a Graph
Pros
! Basic mathematics is close enough to that of HMMs that much of the established infrastructure can be leveraged ! Path similarity doesn’t rely on consistent data streams and allows for regression testing ! Database can easily be trimmed to reduce size concerns
Cons
! The Leap is very fast, and sub graph comparisons are computationally intensive ! Lots of data that isn’t hugely useful to us. ! Continuous data ends up being very sensitive to slight perturbations in paths ! A few orders of magnitude down, but just a few
Karger’s Algorithm
Is That Really a Big Difference Though?
! Syncs up well with our natural perception of gestures ! Reduction of almost 7 full orders of magnitude for comprehensive gesture coverage ! Diffs from node epicenters are more robust and improve regression results ! Greatly reduces number of calls made to REST API
Preliminary Results
! Constrained digit recognition benchmarked at 93.4% ! Maximum latency for immersion is ~120 ms ! Learning rates for gesture based interface is about 40% faster than for gesture-‐free interfaces ! Partnership with zSpace ! Continued mentoring from SolidWorks and Belmont Labs founder, Scott Harris.
Probing the Future of Human-‐Interface Design
What’s Coming Next?
