mole: motion leaks through smartwatch sensors

MoLe: Motion Leaks through Smartwatch

SensorsB00902069 傅蕎

2

Demo video• https://www.youtube.com/watch?v=scZEHExzems

&feature=youtu.be

3

MoLe• Motion sensor leaks input words• Accelerometer & gyroscope on Samsung watch

4

Related work• Inferring keystroke on keyboard• Acoustic• Input timing• Electromagnetic

• Information from smart sensors• Typing on virtual keyboard• Keystroke cause vibration

5

Specialty• No need training user• No brand-new signal processing or learning algorithm• No right hand data, with watch on left hand• Must dictionary word• Must not typing error• Must “correct” typing way• Output top-K possible words• Not support whole sentence, difficulty in “space bar”• Can support different keyboard

6

System overview

7

Watch data• Watch & phone camera & keyboard log• CPC(character point cloud)

8

Keystroke detector - timing• Z axis• Error occurs• Moving from one key to another• Subtle motion, e.g. “asdf”

9

Keystroke detector - location• Origin as “F” key• Gravity removal• Watch’s axis → absolute• Kalman smoothing

10

Cloud fitting• UPC(unlabeled point cloud)• Reference template

11

Bayesian inference• Output ranked list of possible word• Conditional probability• # of keystrokes• Consecutive character, e.g. “er”, “sa”• Watch displacement• Character transition, e.g. “va” VS “ra”• Keystroke interval

12

Evaluation• 8 subjects familiar with English-typing• Variable factor• Word length• Bayesian model• Sampling rate• Keyboard

13

Question?

14

Thanks!:D