yapi.js introduction (mopcon 2016 version)
TRANSCRIPT
ADAPTIVE MEDIA PLAYER BASED ON HTML5
陳建⾠ jessechen
yapi.js
OTT VOD SERVICE台⽇韓中超⼤⽚庫
⼤數據精準推薦
離線播放
投放⾄電視
PLAYBACK EXPERIENCEON WEB
WITH HTML5
影⽚yapi .js
（台）
WHAT IS BEING PLAYEDmedia
analog-to-digital adapter
digital raw data
PREPARE MEDIA1. Make it digital
analog signal
H.264 / H.265Encode
PREPARE MEDIA2. Encode video file
Raw data .mp4 / .ts file
Easier for storage and transference
server holds media
VINTAGE WAY
Player get it and play
single file
super slow
server holds media
EFFICIENT WAY
Player get part of video and
start play
streaming(progressive download)
Keep requesting rest part of video during
playback
inflexible
PREPARE MEDIA3. Fit adaptive need
.mp4 / .ts file
more storage needed but much flexible
different bitrate
+
+
fragments
manifestfile
MANIFEST TELLS- adaptions: video, audio or subtitle
- available bitrates
- how file is fragmented
- encryption
ADAPTIVE STREAMING TYPES
SS Smooth Streaming
HLS HTTP Live Streaming
DASH Dynamic Adaptive
Streaming through HTTP
.mp4 / .ts file different bitrate
+
+
fragments
manifestfileRaw data
imaging there are thousands of media needs to handle
CONTENT OF KKTV IS PREPARED WITH
MASShttps://github.com/KKBOX/mass
https://goo.gl/6TCmWqintroduction by Jinkuen:
download
adaptive streamingclient
yapi.js
media
manifestfile
adaptivealgorithm
fragments
adaptive algorithm decides which quality to download
• network status
• how long does buffered duration left
algorithm considering…
adaptive
starting lag
buffering times / duration
overall bitrate
a/v sync
decode
decrypt
‘playback’client
yapi.js
media
manifestfile
fragments
decrypt
encryption + decryption
H.264 / H.265Encode
ENCRYPTION
Raw data .mp4 / .ts file
encryption implemented
here
CLIENT SIDE DECRYPTION
• Client get a license instead of a key
• A blackbox/sandbox get the key by processing license
• That blackbox/sandbox provide key for decryption and directly output to display
• This solution called DRM(Digital Right Management)
DRM ON BROWSER
• For browser, that blackbox called CDM (Content Decrypt Module)
• Each browser support different DRM
context - “a blackbox or sandbox get the key by processing license”
DRM ON BROWSER
Widevine FairplayPlayready PrimetimeWidevine
INTRODUCE EME
• Even though browser support its own DRM, W3C defines a EME spec, in order to expose same api to use
• prefixed api was implemented on earlier version of chrome(smart tv)
Encrypted Media Extension
YAPI’S PROTECTION LOGIC
• implement protection on client side through EME
• give what DRM server needs to retrieve license
• deal with different browsers (versions)
EMECDM
attempting to play encrypted media
get ‘challenge’
DRM license server
request with challenge
get license
yapi
provide license for CDM to decrypt content
EMECDM
PROTECTION LOGIC FLOW
a/v sync
decode
html5 media element + MSE
MEDIA ELEMENT
var videoNode = document.createElement(‘video’);
videoNode.src = VIDEO_URL;
With html5 media element, you can play single video source easily
single source multiple source
+
+
MSE
“ MSE (Media Source Extension) extends HTMLMediaElement to allow JavaScript to generate media streams for playback.
Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams. “
media source extension
var video = document.createElement(‘video’);
video.src = VIDEO_URL
MEDIASOURCE IS A ‘SOURCE’
set ‘src’ attribute of video element to an url pointed to media source
new window.MediaSource();window.URL.createObjectURL(ms);
SOURCE BUFFER
sourceBufferVideo = ms.addSourceBuffer(VIDEO_CODEC); sourceBufferAudio = ms.addSourceBuffer(AUDIO_CODEC);
// get stream buffer via network
sourceBufferVideo.appendBuffer(buffer);
// sourcebuffer provides buffer info after append complete
BUFFER INFO
var buffered = sourceBuffer.buffered; buffered.length; // how many discontinuous buffered time range
buffered.start(0); // first buffer start time
buffered.end(0); // first buffer end time
get buffer information from buffered attribute
MSE EXTENDS MEDIA ELEMENT
• MSE focus on providing stream buffer to media element
• playback behavior still hold by media elemente.g play, pause, seek
a/v sync
decode
‘playback’client
yapi.js
media
manifestfile
fragments
decrypt
streaming
adaptive
protection / EME
mediaElement / MSE
client
yapi.js
BETWEEN MODULES1. hold reference
problems
• dependency
streaming adaptive
BETWEEN MODULES2. a parent module
problems
• defining ‘parent’
• multiple parent modules?
streaming
adaptive
parent
parent？
BETWEEN MODULES3. event system
• modules are parallel
• semantic event name indicates current ‘state’ of player
streaming
adaptive
protection / EME
mediaElement / MSE
eventBus
streaming
adaptive
MSECtrl
eventBus
eventBus
eventBus
eventBus
function downloadFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() { // notify FRAGMENT_DOWNLOADED }
function onAppended() { // notify FRAGMENT_APPENDED }
register
eventBus.mapHandler(‘FRAGMENT_DOWNLOADED’, )adaptive. calculateQuality
eventBus.mapHandler(‘FRAGMENT_DOWNLOADED’, )MSECtrl. appendToSourceBuffer
eventBus.mapHandler(‘FRAGMENT_APPENDED’, )streaming. downloadFragment
{ }
FRAGMENT_DOWNLOADED: [ , ],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
eventBus
function downloadNextFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() { eventBus.dispatch(‘FRAGMENT_DOWNLOADED’) }
function onAppended() { // notify FRAGMENT_APPENDED }
dispatch event
eventBus
{ }
FRAGMENT_DOWNLOADED: [ , ],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
invoke callbacks
function downloadNextFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() { eventBus.dispatch(‘FRAGMENT_DOWNLOADED’) }
function onAppended() { eventBus.dispatch(‘FRAGMENT_APPENDED’) }
dispatch event
eventBus
{ }
FRAGMENT_DOWNLOADED: [ , ],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
invoke callbacks
works with app level
yapi.js
app
ui
client
api
works with app level
yapi.js
app
ui
client
events
api
MEDIA EVENTS
http://www.w3.org/2010/05/video/mediaevents.html
Dispatched by HTML5 media element, notifying playback process
YAPI PROXY/PATCH EVENTS
Media Element event event with same name
yapi.js
USEFUL MEDIA EVENTS
loadstart: Indicate loading media begins, this fires when setting src attribute
loadedmetadata: while element has basic info of playback, e.g duration
timeupdate: while current time of playback is ticking
seeking/seeked: while conducting seek
ended: playback ends
play/playing: playback resume from other status to playing
ADDITIONAL EVENTSloadedmanifest: after manifest is loaded/parsed
bitratechanged: when bitrate is changed
enableabr: when adaptive activation changed
buffering: when playback pending
cuechanged: webvtt subtitle cue changed (in and out)
yapi.load(MANIFEST_URL); // exposed api
yapi.addEventListener(‘playing’, onPlaying);
function onPlaying() { // app logic // or ui reaction, e.g showing playing status ui }
// onPlaying invoked when playing
app yapi ui
LESSONS
1. test case helps
2. handle errors
3. boss wants ‘numbers’
streaming
adaptive
protection / EME
mediaElement / MSE
client
yapi.js
statisticplaying smoothly is not
enough
you need to tell how good it is
mse spec: https://goo.gl/L3HPKP media event sample: https://goo.gl/BsrH18MDN media element: https://goo.gl/JsojcN HTML MediaElement doc: https://goo.gl/Y3lbO0dash.js: https://goo.gl/xAm2Ve
THANK YOUKKTV is hiring
web / android / iOS / backend developers