Vivek Kumar

Quick Video

• Before we begin …

• Are our devices location aware ?• In global respect only• GPS, Radio navigation, WLAN• What about local arena ?• Tango comes into play• Uses computer vision to enable mobile devices, such as smartphones

and tablets, to detect their position relative to the world around them without using GPS or other external signals.

Location Awareness

Introduction

• Development being done at Google under the aegis of ATAP - Advanced Technology and Projects wing.

• Johnny Chung Lee is the technical leader for the project.• Working with universities, research labs, and industrial partners

spanning nine countries around the world to build on the last decade of research in robotics and computer vision, concentrating that technology into a unique mobile device.

Collaborators

3D Motion and Depth Sensing

• Devices contain customized hardware and software designed to track the full 3D motion of the device, while simultaneously creating a map of the environment.

• About 2,50,000 3D measurements per second.• Sets of localised data combined to get a 3D path.• Depth sensing is achieved with special sensors.

Motivation

• Provides mobile devices a human-scale understanding of space and motion.

• Smartphones, Tablets will be able to SEE.• Physical world as a UI to computers.• Visual SLAM : Simultaneous Localization and Mapping.• Phones that don’t limit their boundaries to touchscreen.

Tango Developer Kit

Specs

Specs contd ...

• 7’’ Screen• NVIDIA Tegra K1 processor• 4 GB RAM• 128 GB Storage• Integrated Depth Sensing• Motion Tracking Camera with fish eyed lens• WiFi, BTLE, 4G LTE• Available only for developers

Specs contd ...

• Echo locator sensors, gyroscope.• Uses centralised sensors to figure out where the device you’re

holding is within a known environment.• Present tech allows us to put so much compute power and so many

sensors in a device that it can figure out it’s own environment.• Different algorithms are the basis of 3D rendering and modelling in

real time.• APIs available in C, C++, Java and Unity game engine.

Core Functionalities

• Motion Tracking

• Area Learning

• Depth Perception

Motion Tracking

• Project Tango’s core functionality is measuring movement through space and understanding the area moved through.

• The Project Tango APIs provide the position and orientation of the user's device in full six degrees of freedom, referred to as its pose .

6 degrees of freedom

Area Learning

• Project Tango allows devices to use visual cues to navigate and understand the world around them.

• Visual features of the area it is moving through is recognized when it sees those features again. These features can be saved in an Area Description File (ADF) to use again later.

Possible Use Cases

• Device's coordinate frame can be intentionally aligned with a pre-existing coordinate frame so that content in a game or app always appears in the same physical location.

• Another advantage is for multi-player experiences, where two users can localize to the same coordinate frame, shared through a cloud service. This allows multiple people to interact in the same physical space where all of their relative positions are known.

• Location-aware shopping experience.

Depth Perception

• Project Tango devices are equipped with integrated 3D sensors that measure the distance from a device to objects in the real world.

• The Project Tango APIs provide a function to get depth data in the form of a point cloud. This format gives (x, y, z) coordinates for as many points in the scene as are possible to calculate.

Use Cases

• A game might want to detect when the user is approaching a wall or other object in the area and have that be part of the gameplay.

• Combining depth perception with motion tracking, distances can be measured between points in an area that aren't in the same frame.

• Exact elevation data can be used in apps, games to embed intentional effects.• Point cloud can be filtered to get customized effects.

Applications

• Mapping the physical world• Indoor mapping - home, malls, workplace.• Visual annotation - Add direction overlays, see the plan for a play your team is

executing on the helmet screen, highlight "dangerous"(weaving, too fast, whatnot) drivers on the car dashboard.

• Integrate sensor data to extend human perception - Add an IR overlay. Sample sound across the room, do a volumetric display of noise levels. "see" the strength of your WiFi signal.

Applications contd ...

• Image post processing• You have a 3d map of an area, plus pictures of all textures - rearrange to your

hearts content - Interior Design.• 3D context aware mobile applications - travelling, meeting.• Gamification of local environment.• Role Playing Games can be played in local environment.

Applications contd ...

• Physical training manager - immediate feedback.• Relief & Rescue - 3D plan of house.• Application already developed :• Sensopia's Magic Plan - Interior Mapping• Developer Limbic's - Zombie Gunship IR game• 3D Printing

Applications Contd ...

• NASA being already using at ISS.• Only limited to imagination.

References

● Google ATAP - https://www.google.com/atap/project-tango/● Resource-Aware Large-Scale Cooperative 3D Mapping from Multiple Cell

Phones● Chao X. Guo, Ryan C. DuToit, Kourosh Sartipi, Georgios Georgiou, Ruipeng

Li, John O’Leary, Esha D. Nerurkar, Joel A. Hesch and Stergios I. Roumeliotis● Google unveils 'Project Tango' 3D smartphone platform

http://phys.org/news/2014-02-google-unveils-tango-3d-smartphone.html● Hollister, S. Google's Project Tango Is Now Sized For Smartphones.

http://www.gizmodo.com.au/2015/06/

References contd ..

● hands-on-googles-project-tango-is-now-sized-for-smartphones/ hands-on ● http://www.eweek.com/innovation/google-working-on-tablets-with-enhanced-visi

on-capabilities.html● CHISEL: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device

using Spatially-Hashed Signed Distance Fields● Matthew Klingensmith Carnegie Mellon Robotics Institute Google Project Tango

A Convenient 3D Modeling Device ● Deval Keralia, Khyati K.Vyas and Khushali Deulkar Bell, L. Google Project

Tango and Intel RealSense smartphone