2013 april meetup asia - multimedia stack overview

© Copyright Khronos Group, 2013 -

Multimedia Stack Overview 多媒体堆栈概述

Erik Noreke Chair of OpenSL ES Working Group

Chair of OpenMAX AL Working Group Peruser of OpenMAX IL


Today’s Consumer Requirements 今日消费者的需求 •  Rich media applications and UI

-  Consumer decisions are driven by the multimedia experience -  Drives smart phone market

•  Enhanced user experience -  Games -  Theater-like audio for video playback -  Streaming media services -  Mobile TV

•  Availability of third party applications -  Consumers expected to purchase $35 billion of mobile applications by 2014

•  多媒体应用程序和用户界面

-  多媒体体验带动消费者决定 -  推动智能手机市场

•  增强的用户体验

-  游戏

-  视频播放拥有影院般的音响效果 -  流媒体服务

-  移动电视

•  第三方应用程序的实现

-  预计到2014年消费者对移动应用程序的购买力将达到350亿美元


Application Developer Requirements 应用程序开发者的需求

• Use case driven design -  For ease of use

• Comprehensive feature set -  Including advanced audio and

streaming media

• Consistent APIs -  Same APIs for both hardware

and software implementations

•  Full, predictable functionality -  No need to guess what's

available or how it is going to work

• Portable -  No need to port application for

different platforms

• 使用案例带动设计发展 -  简化使用

• 综合参数设置 -  包括高级音频和流媒体

• API的一致性 -  同时可应用于硬件和软件的

同一个

• 完整可预测的性能 -  无需猜测什么是可以实现的

或者应该如何使用

• 可移植性 -  不同平台间无需使用接口应

用程序


System Integration Requirements 系统整合的需求 •  Function driven design -  For ease of integration

•  Comprehensive feature set -  Including advanced audio and

streaming media

•  Consistent APIs -  Same APIs regardless if using a

software implementation or hardware acceleration

•  Full, predictable functionality -  No need to guess about

available functionality or how it is going to work

•  Ease of Integration -  Expected set of components

reduces integration time and costs

• 功能带动设计 -  简化整合工作

• 综合参数设置

-  包括高级音频和流媒体

• API一致性 -  相同的API同时可以使用在软件执行和硬件加速上

• 完整的可预测的性能 -  无需猜测什么是可以实现

的或者应该如何使用

• 简化整合 -  未来会有整套零部件以降

低整合时间和成本


Multimedia Stack – Overview 多媒体堆栈概述 Applications 应用程序

Mul$media System Access 多媒体系统入口

HW Integration / Implementation Layer 硬件整合/执行层

System Hardware 系统硬件

Multimedia Control Layer 多媒体控制层

Application Access 应用程序入口

Ease of use 简化使用

Ease of integration 简化整合

Choice of HW / SW 硬件/软件选择

HW Accelera*on 硬件加速

Abstraction level 抽象水平

Use case driven design 使用案例带动设计

Use cases 使用案例

Implementation of Multimedia functions in either software or hardware 软件和硬件中多媒体功能执行

Function driven design with access regardless of implementation 通过各种执行的实现，功能带动设计

Creates abstraction layer for application developer 为应用开发者创建抽象层













Use cases 使用案例 Application developers

应用程序开发者

Operating Systems 操作系统

Device Manufacturers 驱动制造商


Why Create Khronos Multimedia APIs? 为什么创建Khronos多媒体API？

Khronos APIs

• Conformance Tests

• Portable

•  IP Protection

• Well defined layers of implementation

• Predictable set of functionality

Open Source Framework

•  Free to change

•  Every implementation unique

• GPL license

• Crosses implementation layers

•  Functionality dependent on implementation

Khronos APIs

• 符合性测试

• 可移植性

•  IP 保护

• 专业定义的执行层

• 可预测的整套性能

开源框架

• 自由更换

• 每个执行都是不同的

• GPL 授权

• 跨执行层

• 性能独立于执行


Streams – The Core of Multimedia 流 – 多媒体的核心 •  Streams are content delivery

channels -  Video, Audio, Text data flow -  Additional metadata

•  Controlling multimedia is about manipulating the streams -  Play rate, effects, source and destinations

•  The streams are sent through various components -  Readers, parsers and decoders (codecs) -  Effects, schedulers and renderers -  Splitters and mixers

•  Combined with HW clock synch and system controls this forms the core of the multimedia stack

•  与硬件时钟同步和系统控制结合，形成了多媒体堆栈的核心流是内容传递的渠道

-  视频、音频、文本数据流

-  其他meta数据

•  控制多媒体就是对流的控制

-  播放速率、效果、来源、目的

•  流是通过各种零部件传输的

-  阅读器、解析器、解码器（编码器）

-  效果、调度程序、渲染器

-  分离器、混合器


OpenMAX IL – Integration Layer 整合层 Applications 应用程序













OpenMAX IL – Managing Stream Flow 管理流 Component Networks •  Enables arbitrary multimedia pipelines by

plugging blocks together -  Componentized architecture abstracts multimedia

functionality block interfaces

•  Wide variety of building blocks for imaging, video and audio functions -  Encode, decode, apply an effect, capture, render,

split, mix, etc

•  Enables blocks from different sources to work together -  Blocks can be implemented in software or

hardware

Portable & Re-usable streaming media building blocks 可移植可再用的流媒体搭建块

组件网络 • 通过一起插入块，实现任意多媒体管线

- 组件化结构缩写多媒体性能块界面

• 拥有成像、视频和音频功能的各种建筑模块

- 编码、解码、效果应用、捕捉、渲染、

分离、混合等等

• 从不同来源共同实现块

- 块可以在软件和硬件内执行


...pBuffernOffset = 9nAllocLen = 96nFilledLen = 21nFlags inc 0x40...

4 bytes

nSize = 28

nVersion

nPortIndex

eType != OMX_ExtraDataNone

nDataSize = 6

OMX_OTHER_EXTRADATATYPE

padding

data

offset padding

extra data

nSize = 24

nVersion

nPortIndex

eType = OMX_ExtraDataNone

nDataSize = 0

data

OMX_OTHER_EXTRADATATYPE

OMX_BUFFERHEADERTYPE

padding

unused

padding

•  Component abstracts a single actor – -  Decoder, effect or renderer

•  Standardized ports allows chaining of arbitrary components -  Including port tunneling for

compatible components

•  Commands are sent to the component

•  Events are received from the component

•  Ports have calls and callbacks

OpenMAX IL Component 组件 • 组件缩写成单独动作- -  解码器、效果或渲染器

• 标准化接口实现任意组件的

链接 -  包括兼容组件的接口隧道技术

• 命令被发送到组件

• 从组件接收事件

• 接口有调用和回调

OMX IL Component 组件

Input ports 输入接口 • Callbacks 回调

Output ports 输出接口 • Calls 调用 • Callbacks 回调

Commands 命令

Events 事件


OpenMAX IL Example Graph 示例图 •  Standardized component interfaces

enable flexible media graphs

• Components are chained into pipelines as needed for each stream

•  Includes multi-stream synchronization

RTSP (RTCP + RTP)

Streaming

Server Control

Clock for AV Sync

Audio Decoder

Audio Renderer

Video Scheduler

Video Renderer

Speakers

Display

AAC Audio

MPEG4/ H.264 Video

Time Data

Video Decoder

Decompressed Video

OpenMAX IL Component Interfaces Enables components to be flexibly connected in any graph topology

Example: MPEG-4 video synchronized with AAC audio decode

Audio Scheduler

Plug-in

E.g. send dance beats to 3D application for visualization

• 标准化组件界面实现灵活的媒体图形

• 根据每个流的需求，组件被链接到不同的管线

• 包括多流同步


OpenSL ES OpenMAX AL

– Application Access 应用入口 Applications 应用程序













OpenSL ES OpenMAX AL

•  Hardware and low level access is powerful but complex -  More power than most application developers

require

•  Most application developers just want to playback and record media -  Specifying where the content comes from -  Specifying where the content should be

rendered to -  Manipulate a few playback controls -  Have simple configurability

•  OpenMAX AL and OpenSL ES provide… -  Simple high-level multimedia APIs for playback

and recording use cases

•  …With Cross-platform portability -  No need to rewrite an application for every

platform

– Controlling Stream Flow 流控制

•  硬件和低水平入口很强大但是很复杂 -  比大多数开发者所要求的更高效

•  大多数应用开发者仅仅需要回放和记录媒体 -  内容具体来自哪 -  内容具体应该被渲染到哪 -  进行一些回放控制 -  简单的可配置性

•  OpenMAX AL和OpenSL ES提供… -  简单的高水平多媒体API，用于回放和记录

使用案例 •  凭借跨平台可移植性 - 无需为每个平台重写应用程序


OpenMAX AL Focused on Streaming Media – Not Media Streams 关注流媒体 – 不是媒体流

•  Media playback and recording -  Playback and recording of everything from video files to radio to camera

input

•  Full range of video effects and controls – including playback rate, post processing, and image manipulation -  Apply advanced effects to captured images and video before sending to

friends

•  Complete metadata support -  Read, Edit, and Set the metadata for most types of content

• 媒体回放和记录

- 回放和记录从视频文件、音频到照相机输入获得的所有内容

• 完整的视频效果和控制 – 包括回放速率、后处理、图片管理

- 在与朋友分享之前，给捕捉图像和视频应用高级效果

• 完整的meta数据支持

- 大多数类型内容的Meta数据阅读、编辑、设置


OpenSL ES – Powerful 3D Audio 强大的3D音频

•  Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization -  Experience rich, enhanced sound from locations other than the

handset, even moving, for the ultimate multimedia experience

•  Full 3D audio functionality enhances any gaming experience -  Perfect companion to OpenGL ES

• Designed to take full advantage of system capabilities

• 各种效果和控制，包括高级3D效果，例如多普勒和虚拟

- 丰富体验、从地点加强音效，不仅仅是手持设备、甚至移动音效，用于多

媒体体验

• 完整3D音频功能加强各种游戏体验

- OpenGL ES的完美匹配

• 为了充分利用系统性能而设计


OpenSL ES and OpenMAX AL •  Working groups collaborated to define the common API functionality

•  Independent, Compatible, Consistent and Distinct

• 工作组合作共同定义共同的API性能

• 独立、可移植、一致的、相区别的

3D Audio

Audio Effects

Advanced MIDI

Audio Playback

Audio Recording

Basic MIDI

Video playback

Camera

Video recording

Image capture & display Buffer

queues

Radio and RDS


The OpenSL ES / OpenMAX AL Object 目标 • Object represents a logical grouping of functionality -  Acts on a stream or set of streams -  Player, recorder, mixer

• Object has a source and a destination -  Acts on a stream or set of streams as they flow through

• Objects have control interfaces -  Grouped methods and callbacks -  Play, Record, Seek

• 目标代表一个逻辑功能群组

- 作为一个流或一组流

- 播放器、记录器、混合器

• 目标有一个源和目的地

- 当其流过时作为一个流或一组流

• 目标有控制界面

- 播放、记录、搜寻

Object

Source

Destination

Interfaces

Dataflow


OpenMAX AL & OpenSL ES – Play Audio Shared Use Case 播放音频分享使用案例


OpenMAX AL & OpenSL ES – Play Video Shared Use Case


Multimedia Stack 多媒体堆栈– Khronos APIs Applications 应用程序













Stream Control vs. Stream Management - Audio playback 流控制和流管理 – 音频回放

File Reader

Clock

Audio Decoder

Audio Sink

Audio Mixer

Audio Effect

OR

Player URI Data Source

Output Mix


Multimedia Stack 多媒体堆栈– Khronos APIs Applications 应用程序













Example: 3D Playback of two audio files 示例：2个音频文件的3D回放


Example: Video Camera 示例：视频摄像机


Multimedia – Managing Streams 多媒体 – 管理流 •  OpenMAX IL manages the pipeline -  Controls where and how the stream flows

•  OpenMAX AL and OpenSL ES manage the flow -  Controls when and how fast the stream flows

•  Together they separate the domains of the multimedia stack -  A distinct integration layer -  Use case driven application access layer

•  To form the basis for a complete multimedia stack -  Without limiting implementation or innovation

•  OpenMAX IL管理管线 - 控制流的位置和方式

•  OpenMAX AL和OpenSL ES 管理流

- 控制流的位置和速度

• 通过分离多媒体堆栈的各个域

- 一个明确的整合层 - 使用案例带动应用入口层

• 从最基础的部分到完整的多媒体堆栈

- 对执行或创作完全没有限制


Thank You 谢谢

2013 april meetup asia - multimedia stack overview

Technology