comp 4026 lecture 6 wearable computing

LECTURE 6: WEARABLE COMPUTING

COMP 4026 – Advanced HCI Semester 5 - 2016

Mark Billinghurst University of South Australia

September 1st 2016

Major changes in computing

Living Heads Up vs. Heads Down

What is a Wearable Computer ? ▪  A computer that is: ▪  Portable while operational ▪  Enables hands-free/hands-limited use ▪  Able to get the user’s attention ▪  Is always on, acting on behalf of the user ▪  Able to sense the user’s current context

Rhodes, B. J. (1997). The wearable remembrance agent: A system for augmented memory. Personal Technologies, 1(4), 218-224.

Wearable Computing ▪  Computer on the body that is: ▪  Always on ▪  Always accessible ▪  Always connected

▪  Other attributes ▪  Augmenting user actions ▪  Aware of user and surroundings

The Ideal Wearable ▪  Persists and Provides Constant Access:

Designed for everyday and continuous user over a lifetime.

▪  Senses and Models Context: Models the users environment, mental state, it’s own state.

▪  Augments and Mediates: Information support for the user in both the physical and virtual realities.

▪  Interacts Seamlessly: Adapts its input and output modalities to those most appropriate at the time.

Starner, T. E. (1999). Wearable computing and contextual awareness (Doctoral dissertation, Massachusetts Institute of Technology).

History of Wearables ▪  1960-90: Early Exploration ▪  Custom build devices

▪  1990 - 2000: Academic, Military Research ▪  MIT, CMU, Georgia Tech, EPFL, etc ▪  1997: ISWC conference starts

▪  1995 – 2005+: First Commercial Uses ▪  Niche industry applications, Military

▪  2010 - : Second Wave of Wearables ▪  Consumer applications, Head Worn

Thorp and Shannon (1961)

• Wearable timing device for roulette prediction •  Audio feedback, four button input

Ed Thorp

Keith Taft (1972)

• Wearable computer for blackjack card counting •  Toe input, LED in Glasses for feedback

Belt computer Shoe Input

Glasses Display

Steve Mann (1980s - )

http://wearcomp.org/

MIT Wearable Computing (1996)

Enabling Technologies (1989+) ▪  Private Eye Display (Reflection Technologies)

▪  720 x 280 dipslay ▪  Red LED ▪  Vibrating mirror

▪  Twiddler (Handykey) ▪  Chording keypad ▪  Mouse emulation

MIT Tin Lizzy (1993) ▪  General Purpose Wearable ▪  Doug Platt, Thad Starner ▪  150 MHz Pentium CPU ▪  32-64 Mb RAM ▪  6 Gb hard disk ▪  VGA display ▪  2 PCMCIA slots ▪  Cellular modem

http://www.media.mit.edu/wearables/lizzy/lizzy/index.html

Early Wearable Computing

US Military Wearables (1989- ) ▪  Early experimentation

▪  386 computer, VGA display ▪  GPS, mapping software

▪  Land Warrior (1991-) ▪  Integrated wearable system ▪  Camera, colour display, radio ▪  Navigation, reports, photos

Zieniewicz, M. J., Johnson, D. C., Wong, C., & Flatt, J. D. (2002). The evolution of army wearable computers. IEEE Pervasive Computing, 1(4), 30-40.

Wearables at CMU (1991–2000) ▪  Industry focused wearables ▪  Maintenance, repair

▪  Custom designed interface ▪  Dial/button input

▪  Rapid prototyping approach ▪  Industrial designed, ergonomic

http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html

Early Commercial Systems

▪  Xybernaut (1996 - 2007) ▪  Belt worn, HMD, 200 MHz

▪  ViA (1996 – 2001) ▪  Belt worn, Audio Interface ▪  700 MHz Crusoe

•  Symbol (1998 – 2006) •  Wrist worn computer •  Finger scanner

Prototype Applications ▪  Remembrance Agent ▪  Rhodes (97)

▪  Augmented Reality ▪  Feiner (97), Thomas (98)

▪  Remote Collaboration ▪  Garner (97), Kraut (96)

•  Maintenance •  Feiner (93), Caudell (92)

▪  Factory Work ▪  Thompson (97)

Mobile AR: Touring Machine (1997) ▪  University of Columbia ▪  Feiner, MacIntyre, Höllerer, Webster

▪  Combines ▪  See through head mounted display ▪  GPS tracking ▪  Orientation sensor ▪  Backpack PC (custom) ▪  Tablet input

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.

MARS View ▪  Virtual tags overlaid on the real world ▪  “Information in place”

Backpack/Wearable Systems 1997 Backpack Wearables

▪  Feiner’s Touring Machine ▪  AR Quake (Thomas) ▪  Tinmith (Piekarski) ▪  MCAR (Reitmayr) ▪  Bulky, HMD based

Piekarski, W., & Thomas, B. (2002). ARQuake: the outdoor augmented reality gaming system. Communications of the ACM, 45(1), 36-38.

2008: Location Aware Phones

Nokia Navigator Motorola Droid

2009 - Layar (www.layar.com)

•  Location based data – GPS + compass location – Map + camera view

•  AR Layers on real world – Customized data – Audio, 3D, 2D content

•  Easy authoring •  Android, iPhone

Watches

Second Gen. Systems

• Recon (2010 - ) • Head worn displays for sports • Ski goggle display •  Investment from Intel (2013)

• Google (2011 - ) • Google Glass • Consumer focus

Demo Video

•  https://www.youtube.com/watch?v=u24cbjqiVfE

Google Glass (2011 - )

• Hardware • CPU TI OMAP 4430 – 1 Ghz • 16 GB SanDisk Flash,1 GB Ram • 570mAh Battery

• Input • 5 mp camera, 720p recording, microphone • GPS, InvenSense MPU-9150 inertial sensor

• Output • Bone conducting speaker • 640x360 micro-projector display

View Through Google Glass

Always available peripheral information display Combining computing, communications and content capture

dsfh

Smart Watches

• Eg Motorola Moto 360, Apple Watch • Measure heart rate, movement • Provide notifications, runs apps, messages, etc

Moto 360 Smart Watch

https://www.youtube.com/watch?v=Q0prQniEZTA

Activity Monitor

• Fitbit, Microsoft Band • Measures footsteps, heart rate, calories, sleep

Device Ecosystem

Usage Matterns

Number of Devices Shipped

Summary Wearables are a new class of computing

Intimate, persistent, aware, accessible, connected Evolution over 50 year history

Backpack to head worn Custom developed to consumer ready device

Enables new applications Collaboration, memory, AR, industry, etc

Many head worn wearables are coming Android based, sensor package, micro-display

TECHNOLOGY

Wearable Attributes

▪  fafds

Some Key Aspects

▪  Display Technologies ▪  Input Devices ▪  Interaction Metaphors ▪  Perceptual Factors ▪  Ergonomics ▪  Cognitive Aspects

Types of Head Mounted Displays

Occluded See-thru

Multiplexed

Multiplexed Displays ▪  Above or below line of sight ▪  Strengths ▪  User has unobstructed view of real world ▪  Simple optics/cheap

▪  Weaknesses ▪  Direct information overlay difficult

•  Display/camera offset from eyeline ▪  Wide FOV difficult

Vuzix M-100

▪  Monocular multiplexed display ($1000) ■ 852 x 480 LCD display, 15 deg. FOV ■ 5 MP camera, HD video ■ GPS, gyro, accelerometer

Display Types

▪  Curved Mirror ▪  off-axis projection ▪  curved mirrors in front of eye ▪  high distortion, small eye-box

▪  Waveguide ▪  use internal reflection ▪  unobstructed view of world ▪  large eye-box

See-through thin displays

▪  Waveguide techniques for thin see-through displays ▪  Wider FOV, enable AR applications ▪  Social acceptability

Opinvent Ora

Waveguide Methods

See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable

Holographic Hologram diffracts light Limited FOV Colour bleeding

Diffractive Slanted gratings Total internal reflection Costly, small FOV

Waveguide Methods

See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable

Clear-Vu Reflective Several reflective elements Thinner light guide Large FOV, eye-box

Reflective Simple reflective elements Lower cost Size is function of FOV

Input Options ▪  Physical Devices ▪  Keyboard ▪  Pointer ▪  Stylus

▪  Natural Input ▪  Speech ▪  Gesture

▪  Other ▪  Physiological

Twiddler Input

▪  Chording or multi-tap input ▪  Possible to achieve 40 - 60 wpm after 30+ hours ▪  Chording input about 50% faster than multi-tap ▪  cf 20 wpm on T9, or 60+ wpm for QWERTY

Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April). Twiddler typing: One-handed chording text entry for mobile phones. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 671-678). ACM.

Virtual Keyboards

▪  In air text input ▪  Virtual QWERTY keyboard up to 20 wpm

-  On real keyboard around 45-60+ wpm

▪  Word Gesture up to 28 wpm -  On tablet/phone Word Gesture up to 47 wpm

▪  Handwriting around 20-30 wpm

A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014)

Unobtrusive Input Devices

▪  GestureWrist ▪  Capacitive sensing ▪  Change signal depending on hand shape

Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE.

Unobtrusive Input Devices

▪  GesturePad ▪  Capacitive multilayered touchpads ▪  Supports interactive clothing

Interaction on the Go

▪  Fitt’s law still applies while interacting on the go ▪  Eg: Tapping while walking reduces speed by > 35% ▪  Increased errors while walking

Lin, M., Goldman, R., Price, K. J., Sears, A., & Jacko, J. (2007). How do people tap when walking? An empirical investigation of nomadic data entry.International Journal of Human-Computer Studies, 65(9), 759-769.

Where to put Wearables?

▪  Places for unobtrusive wearable technology

Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October). Design for wearability. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 116-122). IEEE.

Where to Place Trackpad?

▪  User study 25 people different postures ▪  Front of thigh most preferred, torso/upper arm worst

Thomas, Bruce, et al. "Determination of placement of a body-attached mouse as a pointing input device for wearable computers." 2012 16th International Symposium on Wearable Computers. IEEE Computer Society, 1999.

Where do users want Wearables?

29% on clothing 28% on wrist 12% on Glasses

RAPID PROTOTYPING FOR GLASS

How do you Design for this?

Typical Development Steps

▪  Sketching ▪  Storyboards ▪  UI Mockups ▪  Interaction Flows ▪  Video Prototypes ▪  Interactive Prototypes ▪  Final Native Application

Increased Fidelity & Interactivity

Sketched Interfaces

▪  Sketch + Powerpoint/Photoshop/Illustrator

GlassSim – http://glasssim.com/

▪  Simulate the view through Google Glass ▪  Multiple card templates

GlassSim Card Builder ▪  Use HTML for card details ▪  Multiple templates ▪  Change background ▪  Own image ▪  Camera view

GlassSim Samples

Glass UI Templates

▪  Google Glass Photoshop Templates ▪  http://glass-ui.com/ ▪  http://dsky9.com/glassfaq/the-google-glass-psd-template/

Glass Application Storyboard

• http://dsky9.com/glassfaq/google-glass-storyboard-template-download/

ToolKit for Designers

▪  Vectoform Google Glass Toolkit for Designers ▪  http://blog.vectorform.com/2013/09/16/google-glass-

toolkit-for-designers-2/

▪  Sample cards, app flows, icons, etc

Glassware Flow Designer

•  Visual design tool for Glass interaction flows •  Collaboratively design apps using common patterns and layouts. •  https://glassware-flow-designer.appspot.com/

Glass Application Flow

▪ Series of still photos in a movie format. ▪ Demonstrates the experience of the product ▪ Discover where concept needs fleshing out. ▪ Communicate experience and interface ▪ You can use whatever tools, from Flash to iMovie.

Video Sketching

See https://vine.co/v/bgIaLHIpFTB

Example: Video Sketch of Vine UI

Interactive Wireframing ▪  Developing interactive interfaces/wireframes ▪  Transitions, user feedback, interface design

▪  Web based tools ▪  UXpin - http://www.uxpin.com/ ▪  proto.io - http://www.proto.io/

▪  Native tools ▪  Justinmind - http://www.justinmind.com/ ▪  Axure - http://www.axure.com/

UXpin - www.uxpin.com

▪  Web based wireframing tool ▪  Mobile/Desktop applications ▪  Glass templates, run in browser

https://www.youtube.com/watch?v=0XtS5YP8HcM

UXpin Demo

http://www.youtube.com/watch?v=0XtS5YP8HcM

Viewing on Glass

• Use Android Design Preview • Push desktop onto Android screen •  https://github.com/romannurik/AndroidDesignPreview

Android Design Preview Demo

•  https://www.youtube.com/watch?v=WvQrP1szEzg

Processing and Glass ▪  One of the easiest ways to build rich

interactive wearable applications ▪  focus on interactivity, not coding

▪  Collects all sensor input ▪  camera, accelerometer, touch

▪  Can build native Android .apk files ▪  Side load onto Glass

Hello World //called initially at the start of the Processing sketch!void setup() {! size(640, 360);! background(0);!} !!//called every frame to draw output!void draw() {! background(0);! //draw a white text string showing Hello World! fill(255);! text("Hello World", 50, 50);!}!

Demo

Hello World Image PImage img; // Create an image variable!!void setup() {! size(640, 360);! //load the ok glass home screen image! img = loadImage("okGlass.jpg"); // Load the image into the program !}!!void draw() {! // Displays the image at its actual size at point (0,0)! image(img, 0, 0);!}!

Demo

Touch Pad Input •  Tap recognized as DPAD input

!void keyPressed() {!!if (key == CODED){!! !if (keyCode == DPAD) {!

!// Do something ..!•  Java code to capture rich motion events

•  import android.view.MotionEvent;!

Motion Event //Glass Touch Events - reads from touch pad!public boolean dispatchGenericMotionEvent(MotionEvent event) {! float x = event.getX(); // get x/y coords ! float y = event.getY();! int action = event.getActionMasked(); // get code for action! ! switch (action) { // let us know which action code shows up! !case MotionEvent.ACTION_DOWN:! ! !touchEvent = "DOWN";! ! !fingerTouch = 1;! !break; ! !case MotionEvent.ACTION_MOVE:! ! !touchEvent = "MOVE";! ! !xpos = myScreenWidth-x*touchPadScaleX;! ! !ypos = y*touchPadScaleY;! !break;!

Demo

Sensors • Ketai Library for Processing

•  https://code.google.com/p/ketai/

•  Support all phone sensors •  GPS, Compass, Light, Camera, etc

•  Include Ketai Library •  import ketai.sensors.*;!•  KetaiSensor sensor;!

Using Sensors • Setup in Setup( ) function

•  sensor = new KetaiSensor(this);!•  sensor.start();!•  sensor.list();

• Event based sensor reading void onAccelerometerEvent(…)!{! accelerometer.set(x, y, z);!}!

Sensor Demo

Sensors ▪  Ketai Library for Processing ▪  https://code.google.com/p/ketai/

▪  Support all phone sensors ▪  GPS, Compass, Light, Camera, etc

▪  Include Ketai Library ▪  import ketai.sensors.*; ▪  KetaiSensor sensor;

Using the Camera ▪  Import camera library

▪  import ketai.camera.*; ▪  KetaiCamera cam;

▪  Setup in Setup( ) function ▪  cam = new KetaiCamera(this, 640, 480, 15);

▪  Draw camera image void draw() { //draw the camera image image(cam, width/2, height/2); }

Camera Demo

RESOURCES

Online Wearables Exhibit

Online at http://wcc.gatech.edu/exhibition

Glass Resources ▪  Main Developer Website ▪  https://developers.google.com/glass/

▪  Glass Apps Developer Site ▪  http://glass-apps.org/glass-developer

▪  Google Design Guidelines Site ▪  https://developers.google.com/glass/design/

index?utm_source=tuicool ▪  Google Glass Emulator ▪  http://glass-apps.org/google-glass-emulator

Other Resources ▪  AR for Glass Website ▪  http://www.arforglass.org/

▪  Vandrico Database of wearable devices ▪  http://vandrico.com/database

Books ▪  Programming Google Glass ▪  Eric Redmond

▪  Rapid Android Development: Build Rich, Sensor-Based Applications with Processing ▪  Daniel Sauter

www.empathiccomputing.org

@marknb00

mark.billinghurst@unisa.edu.au