2013 april meetup asia - enabling augmented reality

© Copyright Khronos Group 2013 |

Enabling Augmented Reality 增强现实的实现 Beijing China, April 2013 北京中国， 2013年4月

Erik Noreke Technology Visionary

Vice President Business Development, The Khronos Group


Topics 主题 • Components of Augmented Reality

• Khronos standard APIs for Augmented Reality

• Vision and sensor processing for Augmented Reality

• Example Use of Khronos APIs in AR

• 增强现实的组成

• 增强现实中的Khronos标准API

• 增强现实的视觉和传感处理

• 增强现实中Khronos API使用案例


Main Components of AR 增强现实的主要组成部分

Sensor Input 传感输入

Rendering 渲染

Scene Analysis and

Computation 场景分析和计算

Data Access, Analysis and Cloud Computing 数据获取，分析和云计算

External Data 外部数据


Data Comms


Local Device 当地设备

Software

Silicon


Visual-based Augmented Reality 视觉为基础的增强现实

Camera video stream sent to the compositor

3D Augmentation Rendering 3D增强渲染

3D augmentations composited with video stream

Camera Tracking 相机跟踪

GPS, accelerometer,

gyro and camera images

used to track the camera’s location and

orientation

Camera-to-scene transform locks the 3D rendering to the real world

摄像机视频流发送至合成器

3D 增大与视频流合成 GPS，加速计，回

转仪和相机图片用于跟踪相机位置和

方位


Khronos Standards

15 Active Standards 15个现行标准

2000 2003 2004 2005 2006 2008 2009 2011

Advanced Audio

Parallel Computing

Cross platform desktop 3D

Plugin-free 3D Web Content

Unified Sensor and Input Processing

Embedded and Mobile 3D

嵌入和移动3D

Camera, Images and Streaming Media

照相机、图片、流媒体

Vector 2D

矢量 2D

3D Digital Asset Exchange format

3D 数字资产交换格式

高级音频

跨平台桌面3D

并行计算

无插件3D网页内容

统一传感器和输入处理

Web Compute

网页计算

Hardware accelerated vision functions

硬件加速视觉功能


API Standard Evolution API标准的演进

New API technology first

evolves on high-end platforms

全新API技术首先进入高端平台

Mobile is the new platform for apps innovation. Mobile

APIs unlock hardware and conserve battery life

手机是应用创新的新平台。移动AP

I解脱了硬件和电池消耗的束缚

Apps embrace mobility’s unique strengths and need complex, interoperating APIs with rich

sensory inputs e.g. Augmented Reality

各种应用涵盖了移动设备的特殊性能并

需要复合功能的实现，于是带来了拥有丰富传感输入的内部操作API，如

增强现实

Diverse platforms – mobile, TV, embedded – means HTML5 will become increasingly important

as a universal app platform

各种平台– 手机、电视、嵌入式HTML

5将越来越重要

DESKTOP 桌面

MOBILE 移动

INTEROP, VISION AND SENSORS

交互操作，视觉和传感

WEB 网页


Mobile Innovation Hot Spots 移动创新热点 • New platform capabilities being driven by SILICON and APIs

• 芯片和API所带动的新平台性能

Console-Class 3D Performance, Quality, Controllers and TV connectivity 遥控类3D 性能，质量，遥控器和电视连接

HTML5 Web Apps that can be

discovered on the Net and run on any platform

网页应用可以在网页上开发并在任何平台上运行

Vision - Camera as sensor Computational Photography

Gesture Processing Augmented Reality 视觉 – 摄像机传感器

计算性相片姿势处理增强现实

Sensor Fusion Devices become ‘magically’ context aware – location, usage, position 传感器融合设备成为“神奇的”文本感应 – 地点，使用，位置

•  Media and Image Streaming •  Heterogeneous Parallel Processing

•  媒体和图像流 •  异构并行处理


How Many Sensors are in a Smartphone? 智能手机中有多少传感器？

• Light 灯光 • Proximity 邻近效应 • 2 cameras 2个照相机 • 3 microphones (ultrasound) 3个麦克风（超声波） • Touch 触摸 • Position 位置 -  GPS -  WiFi (fingerprint) -  Cellular (tri-lateration) -  NFC, Bluetooth (beacons)

• Accelerometer 加速计 • Magnetometer 磁力计 • Gyroscope 回转仪 • Pressure 压力 • Temperature 温度 • Humidity 湿度

8

19!


Market Demand for Sensor Fusion API 传感融合API的市场需求

Innovative use of growing sensor diversity

不断增长的传感多样性创新使用

PORTABLE apps need to be isolated from sensor details 可移植应用需要从传感细节中分

离出来

Application developers do not wish to be Sensor Fusion

experts 应用开发者不想成为传感融合专家

Effective use of multiple interoperating

sensors in one app 一个应用内多重互操作传

感的高效使用

Do NOT force the application developer to access individual sensors (unlike almost all other sensor APIs)

High-level API enables sensor vendors

to drive and deliver competitive sensor fusion innovation

不强迫应用开发者使用单独传感（这点与其他大多数传感API是不同的）

高水平API可以让传感用户带动并提供具有

竞争力的传感融合创新


Portable Access to Sensor Fusion 传感融合的便携式获取

Advanced Sensors Everywhere RGB and depth cameras, multi-axis

motion/position, touch and gestures, microphones, wireless controllers, haptics

keyboards, mice, track pads 高级传感无处不在

RGB和深度相机，多轴动作/位置，触摸和姿势，麦克风，无线遥控，触觉键盘，

mice，track pad

Apps Need Sophisticated Access to Sensor Data

Without coding to specific sensor hardware

应用程序需要复杂的传感数据接入无需编码到具体的传感硬件

Apps request semantic sensor information StreamInput defines possible requests, e.g.

“Provide Skeleton Position” “Am I in an elevator?” 应用程序需要语义的传感信息

StreamInput定义了可能的需求，如“提供骨骼位置”，“我是⼀一个电梯吗？”

Processing graph provides sensor data stream Utilizes optimized, smart, sensor middleware Apps can gain ‘magical’ situational awareness

处理图形提供传感数据流使用优化的，只能传感中间件

应用程序可以拥有“神奇的”感应能力

Universal Timestamps

Standardized Node Intercommunication

Input Device

Input Device

Input Device

Filter Node

Filter Node

App Filter Node

Spec expected in 2013 预计2013年推出规范


OpenVX •  Vision Hardware Acceleration Layer -  Enables hardware vendors to implement accelerated

imaging and vision algorithms -  For use by high-level libraries or apps

•  Focus on enabling real-time vision -  On mobile and embedded systems

•  Diversity of efficient implementations -  From programmable processors, through GPUS to dedicated

hardware pipelines

• 视觉硬件加速层

-  使硬件供应商可以执行加速成像和视觉算法 -  用于高水平库和应用程序

• 关注实现实时视觉

-  在移动和嵌入式系统上

• 高效执行的多样化

-  从可编程处理器，通过GPUS到专门的硬件管线

Open source sample implementation? 开源实例执行？

Hardware vendor implementations 硬件供应商执行

OpenCV open source library 开源库

Other higher-level CV libraries

其他更高水平CV库

Application 应用

Dedicated hardware can help make vision processing performant and low-power enough for pervasive ‘always-on’ use

专門的硬件可以帮助是视觉处理实现性能，同时低能耗足够满足长期的持续使用

OpenVX does not duplicate OpenCV functionality – JUST

provides essential acceleration OpenVX不是复制OpenCV的功能 –

只是提供必要的加速


OpenVX Participants and Timeline 参与和计划

• Aiming for provisional specification in 1H 2013 -  Final specification 2H13

• Itseez is working group chair

• 计划在2013年上半年推出建议版规范 - 最终版规范在2013年下半年推出

• Itseez是工作组主席


OpenVX Execution Flow 执行流 •  OpenVX Graph for efficient execution -  Each Node can be implemented in software or

accelerated hardware

•  EGL provides data and event interop – with streaming -  BUT use of other Khronos APIs are not

mandated

•  VXU Utility Library provides efficient access to single nodes -  Open source implementation

OpenVX Node

OpenVX Node

OpenVX Node

OpenVX Node

Camera Control & Image Processing 相机控制和图片处理 UI and Display 用户界面和显示

Compute Processing 计算处理

Other Outputs 其他输出

Compute Processing 计算处理

Other Inputs 其他输入

•  OpenVX Graph 用于高效的执行 -  每个Node都可以在软件或加速的硬件上执行

•  EGL 提供数据和事件交互操作 – 通过流 -  但是，其他Khronos的API不受命令

•  VXU 使用库提供高效的独立node获取 -  开源执行


EGLStream – Streaming Images 流图片

•  EGL – originally embedded version of WGL -  Abstraction layer to window systems and memory buffers

•  Role has expanded to provide API interop – data and events -  EGLImages – single buffers that can be passed between APIs -  EGLStreams – provides stream of images – with AV sync -  Cross process EGLStreams – Producer and Consumer can be in different processes for performance or

security – e.g. browser compositing process

•  Android SurfaceTexture is a Java wrapper around EGLStreams -  Captures video decode or camera preview to OpenGL ES texture

•  EGL – 来源于嵌入式版本的WGL -  窗口系统和内存缓冲的缩写层

•  功能扩展为提供API交互操作-数据和事件 -  EGLImages – 单独的缓冲可以在API间传递 -  EGLStreams – 提供图片流 – 通过AV同步 -  跨流程EGLStreams – 制造商和消费者可以在不同的功能或安全流程中 – 如浏览器编写流程

•  安卓SurfaceTexture是一个在EGLStreams周边的Java包装 -  捕捉视频解码或相机预览到OpenGL ES纹理


EGLStream – Streaming Images 流图片

Camera File URL Etc. 相机文件

MEDIA PLAYER Object

媒体播放器目标 GL_TEXTURE_EXTERNAL

OpenMAX AL Media Player is the EGLStream “Producer” and controls

production of frames. OpenMAX AL媒体播放器是

EGLStream“生产者”和产品框架控制

OpenGL ES GL_TEXTURE_EXTERNAL is the EGLStream “Consumer” and

converts video format into RGB OpenGL ES texture

OpenGL ES GL_TEXTURE_EXTERNAL 是EGLStream“消费者”，并将视频格式转换成RGB OpenGL ES纹理

? EGLStream

EGLStreams enables and hides details of video frame transport. Enables multiple buffering modes for different uses cases

e.g.: FIFO and explicit latch/release EGLStreams实现并隐藏了视频框架传输细节。实现了不用使用情况的多缓冲模式，如FIFO和

显式锁定/释放


Example Use of Khronos APIs in AR 增强现实中Khronos API的使用

Camera Control Preprocess and generate

video streams 相机控制流程并生成视频流

3D Rendering and Video Composition

On GPU GPU上的3D渲染和视频编写

Audio Rendering 音频渲染

Application on CPUs

and GPUs 在CPU和GPU上的应用

Positional and GPS Sensor Data 位置和GPS传感数据

Computer Vision/Tracking & Computational Photography 计算机视觉/跟踪&计算拍摄

Position and Tracking Semantics

位置和跟踪语义

Synchronization and sensor

fusion 同步和传感融合

Positional Sensors 位置传感

Camera 相机 EGLStream

Image stream to GPU

EGLStream图片流到GPU

Tracked features 跟踪功能


AR on the Web? 网页上的增强现实 •  Leverage native API investments into the Web -  Faster API development and deployment -  Familiar foundation reduces developer learning curve

•  Will this enable web based AR apps? -  Access to necessary hardware acceleration is there!

• 本地API到网页上的投资

-  比API开发和执行的更快 -  对基础的熟悉降低了开发者的学习曲线

• 这将实现网页基础的增强现实应用程序吗？

-  必要的硬件加速已经推出！

Native APIs shipping or working group underway

JavaScript API shipping or working group underway

WebVX? Vision

Processing 视觉处理

WebMAX? Camera control

and video processing

相机控制视频处理

Possible future JavaScript APIs

Device and Sensor APIs

Device Orientation

Working Groups

设备和传感API设备主导工作组

Native

JavaScript Canvas WebSL?

Easy to use JavaScript

Audio 使用简便

本地API已推出或工作组正在进行中

JavaScript API已推出或工作组正在进行中

未来可能的 JavaScript API


Practical AR 现实的增强现实

AC Delco # 19181880 $27.79

Add Part Info

1995 GMC K2500 7.4L – Error code 25


In Summary 总结 • APIs are key to enable AR applications on advanced hardware – APIs developed on

high-end hardware are now enabling AR on mobile devices

• APIs no longer exist alone – they interoperate and provide input AND output processing to form a complete platform for augmented reality

• Khronos is driving open standards for hardware acceleration Enabling portable AR applications in mobile devices!

•  API是在高级硬件上实现增强现实应用的关键 – 在高端硬件上开发的API现在已经实现了移动

设备增强现实

•  API不再单独存在了 – 其交互操作并提供输入和输出处理，以为增强现实搭建一个完整平台

• Khronos带动着硬件加速开放标准，以实现移动设备上的可移植增强现实应用程序！

Connecting Software to Silicon