이동영 / LG전자 dongyoung.lee@lge.com

신개념 N스크린 웹 앱 프레임워크 'PARS'

1. 멀티스크린 웹 앱 아키텍처의 발전 방향

2. PARS 데모

3. PARS 구현 방법 및 이슈

4. Summary

CONTENTS

1. 멀티스크린 웹 앱 아키텍처의 발전 방향

Multiscreen Applications

A user has multiple devices. And they tend to use them together simultaneously.

Manual coordination or ad-hoc methods.

“Orchestrated” user experience. Browsing a picture gallery on a smartphone, displaying large pictures on a TV.

Watching TV, with a program guide on a tablet.

Watching a lecture on a TV, answering exercise questions on a tablet.

A map on a TV, with a list of points-of-interest on a smartphone.

Multiplayer games, with TV as a shared screen.

A cooperating set of programs running on a set of devices.

Approach 1. Fixed Services

An application uses fixed services on other devices

Role of each device is fixed.

E.g., play a video, display a picture.

UI is fixed, and may be inconsistent across devices.

UPnP, DLNA, AirPlay

Platform features

Discovery

APIs

Approach 2. Cooperating Applications

A set of cooperating applications (a dedicated app for each device)

Role of each device is programmable, but still pre-defined.

E.g., display a public view (in a game), display a personal view.

Unified UI across devices.

A user has to pre-install and launch apps on all devices.

Early multiscreen applications.

Platform features

Usually none (generic networking, etc.)

Approach 2.5 Cooperating Apps with Launch

A set of cooperating applications (a dedicated app for each device) Role of each device is programmable, but still pre-defined.

Unified UI across devices.

The platform launches (and sometimes installs) an app on another device.

A user doesn’t have to launch an app on every device.

LG Connect SDK, Samsung Multiscreen SDK, DIAL

Fragmented

Platform features Discovery, Launching

Installing, Messaging, Time synchronization, etc.

Multi-platform / protocol support

Not only for LG TV

Open source

Connect SDK

http://connectsdk.com/

Approach 3. A Distributed Application

A distributed application (a single application) Roles of devices can be re-organized during runtime.

A part of an app can migrate from a device to another.

No installation (of apps).

PARS, COLTRAM, and other works from KAIST, NTT, etc.

Platform features Discovery, Launch

Migration / Replication / Re-organization (during runtime), State

synchronization

Messaging

Roles of devices can be re-organized during runtime.

Parts of an app migrate or are replicated across devices.

A cross platform is required.

Composable UI is desirable.

UI adaptation is needed.

No installation.

Sandbox security model allows executing alien code without installation.

A Distributed “Web” Application

2. PARS 데모

Demo 1. Map (Single User)

Demo 2. Gallery (Multi User)

3. PARS 구현 방법 및 이슈

Architecture

A PARS app consists of components

<div>, JS variables and methods

Component migration / replication

State synchronization

Component management UI

Device discovery

Messaging

A PARS app consists of components

<div>, JavaScript variables and methods (public functions)

All access to another component must be via remote function (method) calls.

Directly accessing another component is prohibited (DOM, JS objects).

Application layer

Application layer

<div PARS = “componentA”> This is component A. <p id=“A”> Default 0</p> </div> <div PARS = “componentB”> This is component B. <button type=“button” onclick=“componentB.myFunction()”> Click to change </button> </div> <script> componentA.count = 0; componentA.setText = function(text) { componentA.count++; document.getElementById(“A”).innerHTML = text + “ “ + componentA.count; } componentB.myFunction = function() { PARS.invoke(componentA, setText, “Changed”); } </script>

How to keep the state of a component synchronized with those of other components on other devices?

A component interacts with another component only via remote function calls.

For components on other devices, relay remote function calls.

Open Issues

Call order, timing, network error, etc.

Local API calls

Framework Layer: State Synchronization

Take a snapshot, send it, and restore it.

HTML: innerHTML

JavaScript: toString (JSON)

Open Issues

Canvas: toDataURL

CSS: CSSOM

Cookie, local storage, …

Hooks before and after migration / replication

Framework Layer: Component Migration

Framework Layer: Replication

How to keep the states of replicated components in sync? Replicate remote function calls.

Alternatives Mutation observer (HTML)

Object.observe (JavaScript)

Multi-User Case Some components may need to have different states according to the user.

E.g., the viewer and local pictures components in the gallery demo

Framework Layer: Network Adaptation

To accommodate any underlying network protocols that provides discovery and messaging.

discovery()

registerMessageHandler()

send

Device discovery

Messaging

Remote execution (then we don’t need a daemon)

Time synchronization (esp. for synchronized media playback)

Current implementation is based on node.js.

Each device runs a node service, which discovers other devices and relay

messages.

The framework connects to the local node service using socket.io.

Platform Layer: Requirements

Webinos (http://webinos.org/) We use only discovery and messaging features.

Based on node.js.

ParsNet Each device runs a simple node service, which discovers other devices and relay

messages.

The framework connects to the local node service using Web Socket.

Alternatives Discovery and messaging via a cloud service.

Platform Layer: Implementations

Interoperability

Abstract APIs to support various underlying technologies, including existing devices.

Standard protocols for interoperability across vendors.

Security and privacy vs. usability

UA or web app?

Platform Layer: Standards

WebRTC Network Service Discovery

DIAL Presentation API

Discovery

P2P messaging

Remote execution

Time synchronization

W3C Second Screen CG WG

Presentation API

/* controller.html */ <script> function playVideo() { if (!navigator.presentation.displayAvailable) return; var p = navigator.presentation.requestShow('player.html'); p.then(function (display) { display.postMessage('http://example.org/video.mp4', '/'); }, function (error) { console.log('Request to show failed: ' + error.msg); } ); } playVideo(); </script

/* player.html */ <video> <script> var v = document.querySelector('video'); window.onmessage = function (event) { v.src = event.data; v.play(); }; </script>

4. Summary

A distributed web app as a solution for multiscreen UX.

You can make your own “PARS”.

E.g., on top of Connect SDK.

Stay tuned for the Presentation API.

Summary

Q&A

THANK YOU