cognitive assistant for the blind

Cognitive Assistant for the Blind�Chieko Asakawa,

IBM Fellow,

IBM Research

Kris Kitani,

Robotics Institute,

Carnegie Mellon University

1

© 2016 IBM Corporation

18th century, the mathematician Euler authored a leading textbook on mathematics after he became blind.

Captions for TV programs were invented for deaf people, but are now widely used in sports bars and to help immigrants, etc.

Based on Self-‐‑‒standing wheelchair.

2011, voice dialog technologies for smartphones

TelephoneKeyboard

Voice recognition

Voice dialogTV Captions

Voice synthesis and OCR

Modern Mathematics

Automatic vehicles

Segway

“Blind driving challenges”

Early use of keyboards was a to help people with hand-‐‑‒and-‐‑‒motor disabilities in writing

First consumer product was a reading assistant for the blind in the 1970s.

1876, Alexander Graham Bell invented the phone after teaching hearing impaired people.

Smart Machines

2

17th century, Dom Perignon. Champagne

Accessibility Needs Ignite Innovation

1960s Talking Typewriter

1960s Talking Typewriter

1975 1403 Braille Printer

1984 Talking 3270 Terminal

1990 VoiceType™

2008 Social Accessibility

1998 ViaVoice®

2000 Accessibility Center

1997 Home Page Reader

1988 ScreenReader/DOS

2004 aDesigner

2007 Eclipse Accessibility Tools Framework

2009 Rich Internet Accessibility Technologies

1914 First IBMer with disabilities

2012 Brazil Accessible Vocational Training Project

2011 Senior Workforce Project

1975 1403

Braille Printer

IBM History of Accessibility

3 © 2016 IBM Corporation

Progress in Information Accessibility

Exponential increase of accessible information (Conceptual chart)

1980 1990 2000 2010

Paper Braille

Digital Braille

Voice Web access

4

Am

ount

of a

cces

sibl

e in

form

atio

n


5

Walk by myself Finding discount posters Finding restaurants Shopping

Many things are difficult…

Challenges of Real World Accessibility


Chika

6

Chika from “Esper the Light Speed”


Natural Language

Processing

Infer from Knowledge

Predict the Future

Natural Interaction

Learning Senses

Watson 1.0, 2.0, …

Apply Watson Modeling

and Simulation

Neural Chip

SyNAPSE Image

Processing Speech

Recognition Technology

Traffic Simulation

Dialog System

Question and Answering System

7

Challenges of “Cognitive Computer”


8

Demonstration Video


Localization Radio-wave

Vision Dead reckoning

Recognition Objects People

Environment

Knowledge Personal

Social media Encyclopedic

Interaction Speech Gesture

Sonification

Building Blocks of Cognitive Assistant

9 © 2016 IBM Corporation

Understanding the Functionality of the Environment

Nick Rhinehart and Kris Kitani, CVPR 2016


When we observe a scene…

… we know how we can act in that environment




Walk

Walk




Sit Sit

Sit




Open

Open Open


Can we teach a computer to

understand scene functionality by

observing people?


Setup:

One wearable camera

Detect actions using computer vision

Learn the function of an environment

21

Information Accessibility Progress (revisited)

1980 1990 2000 2010

Paper Braille

Digital Braille

Voice Web access

Am

ount

of a

cces

sibl

e in

form

atio

n

Exponential increase of accessible information (Conceptual chart)

2020 2030

2016


“Real-world” information access