426 lecture 8: mobile augmented reality
DESCRIPTION
COSC 426 graduate class on Mobile Augmented Reality taught at the University of Canterbury by Mark Billinghurst, Sept. 12th 2012.TRANSCRIPT
COSC 426: Augmented Reality
Mark Billinghurst
Sept 12th 2012
Lecture 8: Mobile Augmented Reality
1983 – Star Wars
1999 - HIT Lab US
CPU: 300 Mhz HDD; 9GB RAM: 512 mb Camera: VGA 30fps Graphics: 500K poly/sec
1998: SGI O2 2008: Nokia N95
CPU: 332 Mhz HDD; 8GB RAM: 128 mb Camera: VGA 30 fps Graphics: 2m poly/sec
Mobile Phone AR Mobile Phones
camera processor display
AR on Mobile Phones Simple graphics Optimized computer vision Collaborative Interaction
2005: Collaborative AR
AR Tennis Shared AR content Two user game Audio + haptic feedback Bluetooth networking
Mobile AR History
Evolution of Mobile AR
Wearable AR
Handheld AR Displays
Camera phone
1995 1997 2001 2003 2004
Camera phone - Self contained AR
Wearable Computers
PDAs -Thin client AR
PDAs -Self contained AR
Camera phone - Thin client AR
Handheld Displays Tethered Applications
Fitzmaurice Chameleon (1994) Rekimoto’s Transvision (1995) Tethered LCD PC Processing and Tracking
Handheld AR Display - Tethered
1995, 1996 Handheld AR ARPad, Cameleon Rekimoto’s NaviCam, Transvision Tethered LCD PC Processing and Tracking
NaviCam (Rekimoto, 1995)
Information is registered to real-world context Hand held AR displays
Interaction Manipulation of a window
into information space
Applications Context-aware information displays
AR Pad (Mogilev 2002) Handheld AR Display
LCD screen Camera SpaceOrb 3 DOF controller Peripheral awareness Viewpoint awareness
Mobile AR: Touring Machine (1997) University of Columbia
Feiner, MacIntyre, Höllerer, Webster
Combines See through head mounted display GPS tracking Orientation sensor Backpack PC (custom) Tablet input
MARS View
Virtual tags overlaid on the real world “Information in place”
Backpack/Wearable AR
1997 Backpack AR Feiner’s Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr) Bulky, HMD based
PCI 3D Graphics Board
Hard Drive
Serial
Ports
CPU
PC104 Sound Card
PC104 PCMCIA
GPS Antenna
Tracker Controller
DC to DC Converter
Battery
Wearable Computer
GPS RTK correction
Radio
Example self-built working solution with PCI-based 3D graphics
Columbia Touring Machine
Mobile AR - Hardware
1997 Philip Kahn invents camera phone 1999 First commercial camera phone
Sharp J-SH04
Millions of Camera Phones
Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge) PDA used as I/O device Wireless connection to workstation Room-scale ultrasonic tracking (Bat)
2001 AR-PDA (C Lab) PDA thin graphics client Remote image processing www.ar-pda.com
2003 ARphone (Univ. of Sydney) Transfer images via Bluetooth (slow – 30 sec/image) Remote processing – AR Server
Mobile Phone AR – Thin Client
Early Phone Computer Vision Apps 2003 – Mozzies Game - Best mobile game Optical motion flow detecting phone orientation Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH) Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)
Computer Vision on Mobile Phone Cameras and Phone CPU sufficient for computer vision
applications Pattern Recognition (Static Processing)
QR Code Shotcode (http://www.shotcode.com/)
Motion Flow (2D Image Processing) GestureTek
- http://www.gesturetekmobile.com/
TinyMotion
3D Pose Calculation Augmented Reality
Handheld AR – Self Contained 2003 PDA-based AR
ARToolKit port to PDA Studierstube ported to PDA AR Kanji Educational App. Mr Virtuoso AR character Wagner’s Invisible Train
- Collaborative AR
Mobile Phone AR – Self Contained 2004 Mobile Phone AR
Moehring, Bimber Henrysson (ARToolKit) Camera, processor, display together
AR Advertising
Txt message to download AR application (200K) See virtual content popping out of real paper advert Tested May 2007 by Saatchi and Saatchi
2008 - Location Aware Phones
Nokia Navigator Motorola Droid
Real World Information Overlay Tag real world locations
GPS + Compass input Overlay graphics data on live video
Applications Travel guide, Advertising, etc
Eg: Mobilizy Wikitude (www.mobilizy.com) Android based, Public API released
Other companies Layar, AcrossAir, Tochnidot, RobotVision, etc
Layar – www.layar.com
HIT Lab NZ Android AR Platform Architectural Application Loads 3D models
a OBJ/MTL format
Positions content in space GPS, compass
Intuitive user interface toolkit to modify the model
Connects to back end model database
Architecture
Android application Database server
Postgres
Web Interface
Add models
Web application java and php server
1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quake
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian
History of Handheld and Mobile AR
2012 State of the Art Handheld Hardware available
PDA, mobile phones, external cameras Sensors: GPS, accelerometer, compass Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia Graphics: OpenGL ES Authoring: Layar, Wikitude, Metaio Creator
What is needed: High level authoring tools Content development tools Novel interaction techniques User evaluation and usability
Mobile AR Companies Mobile AR
GPS + compass
Many Companies Layar Wikitude Acrossair PressLite Yelp Robot vision Etc..
$2 million USD in 2010 $732 million USD in 2014
Handset Manufacturers Qualcomm
$100 million USD investment
Nokia 25+ people in NRC
Samsung Exploring the space
Apple 586 AR Applications on App Store
Google Google goggles/Android AR Applications
Qualcomm
Acquired Imagination October 2010 - Releases free Android AR SDK Computer vision tracking - marker, markerless Integrated with Unity 3D renderer http://developer.qualcomm.com/ar
Rock-em Sock-em
Shared AR Demo Markerless tracking
Apple
iPhone 4 SDK supports direct camera access Launches AR theme on App Store
Mobile AR Technology
Technology Components Software Platform
Eg studierStube platform
Developer Tools iOS, Android
Tracking Technology Computer vision, sensor based
Mobile Graphics OpenGL ES
Interaction Methods Handheld Interaction
What are the challenges? Many platforms (operating systems) Slow CPUs (low clock rates, often no FPU)
Difficulties with tracking Difficulties with visualizations that require a lot of data processing
Small and slow memory access No or weak hardware 3D acceleration
Difficulties with graphics
Poor cameras Bad under low light, noise, etc
What makes a device interesting for AR?
Open and easy to program Good camera Fast CPU, FPU is a plus Good H/W 3D support Large installed basis
Easy access to devices
GPS, accelerometer, compass Enough memory/storage
Typical Smart Phone Hardware CPU
300-800+ Mhz
GPU None, or Power VR Chip (OpenGL ES1.0/2.0)
Input Touch screen, keyboard, keypad
Sensors GPU, compass, accelerometer, camera (1.3-5mb+)
Networking Bluetooth, Wifi, 3G
Screen 320x240 up to 800x480
HTC Desire Android Fast CPU (1GHz) Smaller screen
3.7”, 800x480
Multi-touch Hardware 3D GPS, compass and
accelerometer
iPhone 4 Apple iOS 4.0 Faster CPU (1.2GHz) High screen resolution
3.5”, 960x640
(Finally) camera API Multi-touch Hardware 3D GPS, compass, accelerometer
and gyroscope
Hardware Sensors Camera (resolution, fps)
Maker based/markerless tracking Video overlap
GPS (resolution, update rate) Outdoor location
Compass Indoor/outdoor orientation
Accelerometer Motion sensing, relative tilt
Studierstube ES Framework
Typical AR application framework
Developed at TU Graz
Hardware
OS/Low Level API
Programming Libraries
End User Application
The Studierstube ES framework
Tracking
Platform
Graphics
Content
User Interface - Application
Mobile Development Environments Not like developing for desktops
Wide range of different OS Limited CPU, low memory, poor graphics, no floating point
Popular Mobile OS iPhone (Objective C) Android (Java, Native NDK wrapper) Windows Mobile (C#, C++, Visual Studio)
Other Blackberry, Linux
Programming iPhone Harsh restrictions from Apple
Apps have to go through the apps store
Xcode IDE for development Nice development tools
Objective-C Required for application development Can call into C/C++ code
Limited camera API except in iOS 4 Can overlay on live video No image processing
Android Hardware creators
HTC, LG, Samsung, Motorola
Widely available phone Different form factors – tablets, phones, PC, etc
Multiple versions and fragmentation Android 1.0, 1.6 Android 2.0, 2.1
Free Tools Eclipse Development App Integrator
Mobile Graphics
Computer Graphics on Mobile Phones Small screen, limited input options Limited support for accelerated graphics
Most phones have no GPU
Mobile Graphics Libraries OpenGL ES (1.0, 2.0)
- Cross platform, subset of OpenGL - C/C++ low level library
Java M3G - Mobile 3D graphics API for J2ME platform - Object importer, scene graph library - Support from all major phone manufacturers
OpenGL ES Small-footprint subset of OpenGL
OpenGL is too large for embedded devices!
Powerful, low-level API, full functionality for 3D games Can do almost everything OpenGL can Available on all key platforms Software and hardware implementations available
Fully extensible Extensions like in OpenGL
OpenGL ES SLIDE 61
OpenGL ES on mobile devices
OpenGL ES History Khronos Group – founded 2000
Goal: create royalty free open multimedia standards Create OpenML specification Consortium of > 120 companies
OpenGL ES working group – formed 2002 20-30 companies Create API for 3D graphics on embedded devices Cleaned up and trimmed down version of OpenGL OpenGL ES 1.0 released 2004, 1.1 in 2005, 2.0 spec in 2007
Versions Two major tracks
Not compatible, parallel rather than competitive
OpenGL ES 1.x Fixed function pipeline Suitable for software implementations All 1.x are backwards compatible
OpenGL ES 2.x Vertex and pixel shaders using GLSL ES All 2.x will be backwards compatible
Fixed Function (1.x)
h"p://www.khronos.org/opengles/2_X/
Programmable (2.x)
h"p://www.khronos.org/opengles/2_X/
OpenGL ES 1.x vs 2.0
Hardware supporting OpenGL ES 2.x Qualcomm SnapDragon Texas Instruments OMAP3 NVIDIA Tegra (APX2500) Samsung S3C6410 POWERVR SGX
Now becoming available in phones! Typically come with OpenGL ES 1.x drivers too
The emulation on GLES 2.0 hardware can be slow!
Tracking
Mobile Augmented Reality’s goal
Create an affordable, massively multi-user, widespread platform
© Tinmith, U. of South Australia
Tracking Requirements for AR on Phones Fast and efficient Form factor: light and robust Track simultaneously
a large number of objects by a large number of users
Requires little or no … Device modification Manual calibration (targeting non-technical users) Instrumentation of the physical environment
Low costs
How to not do it…
Ka-Ping Yee: Peephole Displays, CHI’03
Tracking on mobile phones Vision-based tracking
Marker-based tracking Model-based natural feature tracking Natural feature tracking in unknown environments
Sensor tracking GPS, inertial compass, gyroscope
Backpack-based
Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)
Laptop, HMD Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking Hand tracking w/ fiducial markers
Tablet PC / UMPC-based
Schall et al., 2006 Hybrid tracking on UMPC
Camera fiducial marker tracking When no marker in view inertial sensor + UWB tracking
PDA-based 1.
BatPortal (Newman et al., 2001) PDA as thin client (rendering &
tracking on server + VNC) Ultrasonic tracking
SHEEP (MacWilliams et al., 2003)
Tracking by ART (external IR cameras + retroreflective target)
Projection-based AR environ.
Signpost on PDA (Wagner & Schmalstieg, 2003)
First “truly” handheld AR platform: PDA + camera Standalone, self-contained AR system Optimized fiducial marker tracking library
PDA-based 2.
History of non-AR Tracking on Phones (1)
AR-PDA (Gausemeier et al., 2003)
Model-based tracking PDA = thin client
tracking off-loaded to server Not real-time
Kick Real (Paelke et al., 2004)
Edge detection of real foot + collision detection w/ virtual ball
2D tracking and limited interaction (tailored to game)
History of non-AR Tracking on Phones (2)
Mosquito Hunt (Siemens, 2003) Marble Revolution (BitSide, 2004) Pingis (VTT, 2006) Game control w/ optical flow techniques
TinyMotion (Wang et al., 2006) GUI control & input on cell phones
w/ image differencing & block correlation
History of AR Tracking on Phones (1)
2003 ARToolKit on PDA Wagner et at.
2004 3D Marker on Phone Möhring et al.
2005 ARToolKit on Symbian Henrysson et al.
History of AR Tracking on Phones (2)
2005 Visual Codes Rohs et at.
2008 Advanced Marker Tracking Wagner et al.
2008 Natural Feature Tracking Wagner et al.
What can we do on today‘s mobile phones?
Typical specs 600+ MHz ~5MB of available RAM 160x120 - 320x240 at 15-30 Hz camera
Possible to do Marker tracking in 5-15ms Natural feature tracking in 20-50ms
Other Mobile Sensors Orientation
Compass
Relative movement/rotation Accelerometer, gyroscope
Context Audio, light sensor, proximity
Location GPS, A-GPS, Wifi positioning, Cell tower triangulation
Handheld AR Interfaces
Handheld HCI
Consider your user Follow good HCI principles Adapt HCI guidelines for handhelds Design to device constraints Rapid prototyping User evaluation
Sample Handheld AR Interfaces Clean Large Video View Large Icons Text Overlay
Handheld Display vs Fixed Display
Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasma
Users performed (1) navigation task, (2) selection task Moving handheld display provided greater perceived FOV, higher degree of
presence, faster completion time
J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
Search Task Completion Time
FOV, Presence and Immersion
Outdoor AR: Limited Field of View
Possible solutions Overview + Detail spatial separation; two views
Focus + Context merges both views into one view
Zooming temporal separation
TU Graz – HIT Lab NZ - collaboration Zooming panorama Zooming Map
Zooming Views
Zooming AR interfaces
Context Compass Zooming Panorama Zooming Map
Interface Types Compass (C) Compass + Zooming Panorama (CP) Compass + Zooming Map (CM) Compass, Zooming Panorama, Zooming Map (CPM)
Twitter 360
www.twitter-360.com iPhone application See geo-located tweets in real world Twitter.com supports geo tagging
Wikitude – www.mobilizy.com
Blah
Blah
Blah Blah Blah
Blah Blah
Blah
Blah Blah Blah
Blah Blah
Blah
Blah Blah
Blah
Blah Blah
Blah
Blah Blah
Information Filtering
Information Filtering Information Filtering (Julier et al. ’00)
• Remove clutter by goal- and distance based filtering • User’s task is route finding: Sniper and relevant buildings are displayed; objects, which are determined to be unnecessary, removed
HandHeld AR Wearable AR
Output: Display
Input
Input & Output
HMD vs Handheld AR Interface
Handheld Interface Metaphors Tangible AR Lens Viewing
Look through screen into AR scene Interact with screen to interact with AR
content - Eg Invisible Train
Tangible AR Lens Manipulation Select AR object and attach to device Use the motion of the device as input
- Eg AR Lego
Translation Study Conditions
A: Object fixed to the phone (one handed) B: Button and keypad input C: Object fixed to the phone (bimanual) - one hand for rotating tracking pattern
Results – Translation • 9 subjects – within subject design • Timing
• Tangible fastest • twice as fast as keypad
• Survey • Tangible easiest (Q1) • Keypad most accurate (Q2) • Tangible quickest (Q3) • Tangible most enjoyable (Q4)
• Ranking • Tangible favored
A B C
Rank 1.44 2.56 2.0
Conditions A: Arcball B: Keypad input for rotation about
the object axis C: Object fixed to the phone (one handed)
D: Object fixed to the phone (bimanual)
Rotation Study
• Timing • Keypad(B) and Arcball(A) fastest
• No significant survey differences
A B C D Rank 3.0 2.3 2.4 2.2
Results – Rotation
TAT Augmented ID
Design Guidelines
Apply handheld HCI guidelines for on-screen content - large buttons, little text input, etc
Design physical + virtual interface elements Pick appropriate interface metaphor
- “handheld lens” approach using handheld motion - Tangible AR for AR overlay
Build prototypes Continuously evaluate application
Mobile AR Browsing
Junaio - www.junaio.com
QR Code Launch
Glue Tracking - Markerless
Search for “instant tracker”
Junaio Interface
Interface
List View, Map View, AR View
Architecture
Back-end Servers
Data Flow
Search.php <?xml version=\"1.0\" encoding=\"UTF-8\"?> <results>
<poi id=\"1\" interactionfeedback=\"none\"> <name><![CDATA[[Hotel Hello World]]]></name> <description><![CDATA[[This is a beautiful, family hotel and restaurant, just around the
corner. Special Dinner and Rooms available.]]]></description>
<l>37.776685,-122.422771,0</l> <mime-type>text/plain</mime-type>
<icon>http://dev.junaio.com/publisherDownload/tutorial/icon_map.png</icon> <thumbnail>http://dev.junaio.com/publisherDownload/tutorial/thumb.jpg</thumbnail> <phone>555/1234567</phone> <homepage> http://www.hotelaroundthecorner.com </homepage> </poi>
</results>
Creating a New Channel
AR Outcome
Mime Tag Types
<mime-type>text/plain</mime-type> Changing mime tag changes media loaded
AR Browsers Commercial outdoor AR applications
Junaio, Layar, Wikitude, etc
All have their own language specifications Wikitude – ARML Junaio - XML
Need for common standard Based on existing standards for geo-located content etc Support for dynamic/interactive content Easier to author mobile AR applications Easy to render on AR browsers
BirdsView - http://www.birdsview.de/
Location Based CMS Add content, publish to Layar or Junaio
BirdsView on Junaio
BirdsView on Junaio
Limitations BirdsView Branding
Their logo, not yours
Limited POI Content Images, Text, URL
Public Channel Your content + everyone else's
Hoppala Augmentation
http://www.hoppala-agency.com/ Rich media overlay
Hoppala in Junaio
Metaio Creator
Drag and drop Junaio authoring
BuildAR – buildar.com
3DOn
Onsite Visualization Single Building GPS + Compass input Overlay graphics data on live video/Photos
3D On Interface