Kohta Kiyokawa, Shinya Yamamoto †, Naoki Shibata ‡,

Keiichi Yasumoto and Minoru Ito

　　　　　　 Nara Institute of Science and Technology, † Tokyo University of Science Yamaguchi, ‡ Shiga University

　

February 23, MMSys2010

• Propose framework “UbiREMOTE” for remotely controlling appliances with 3D virtual space I/F– Intuitive operation for anyone – Intuitive monitoring of target spaces where appliances exist– Smooth operation on mobile terminals

• Implement and evaluate the framework– Runs smoothly on mobile terminals– Has better usability than text-based

remote controller

Overview

2

Outline

3

1. Background and Goals

2. UbiREMOTE Framework

3. Proposed Techniques

4. Evaluation

5. Conclusions and Future Works

Background Several frameworks for controlling appliances

through network are available UPnP, DLNA, OSGi, ECHONET, etc

These frameworks enable remote control of appliances Examples

warm/cool the living room before arriving home check if he/she switched off an appliance (e.g.,

heater) and turn it off remotely

4

5

• Several existing services– E.g., “JIG multi remocon” by jig.jp– Operate multiple appliances

through a cell phone– User I/F is based on figures/icons

and texts

• Drawbacks– Usability rapidly reduces when operating many similar

kind of appliances at different places (rooms)• Difficult to find target appliance from many

figures/texts on screen• UI is different from dedicated remote controllers

Existing services and drawbacks

© jig.jp co.ltd http://remocon.jig.jp/

Our goals

• Remote controller service that allows users to remotely control networked appliances in their home by– Intuitive operation from a remote place– Intuitive monitoring of target spaces

• Current states of appliances (on/off, …etc)• Conditions (temperature, humidity, …etc) of rooms

– Smooth operation by mobile terminals

6

Outline

7

1. Background and Goals

2. UbiREMOTE Framework

3. Proposed Techniques

4. Evaluation

5. Conclusions and Future Works

Smooth operation on mobile terminals

Intuitive monitoring of target spaces

8

Challenges and ApproachesIntuitive operation

Reproduce building with many networked appliances (target space) in 3D virtual space Imitate the real remote controller for each appliance by

using control window Show states of

appliances by visual effects

9

Challenges and ApproachesIntuitive operation

There are too many 3D objects in buildingPoor 3D capability on mobile deviceWe introduce lightweight 3D rendering

mechanism

Smooth operation on mobile terminals

Reproduce building with many networked appliances (target space) in 3D virtual space

10

Challenges and ApproachesIntuitive operation

Gather data by sensors deployed in each roomDetection/reflection of appliance positions

Lightweight 3D rendering mechanism

Smooth operation on mobile terminals

Intuitive monitoring of target spaces

Reproduce building with many networked appliances (target space) in 3D virtual space

11

Challenges and ApproachesIntuitive operation

12

Intuitive UI based on 3D virtual space

Appliance (lamp) in real space is turned on

Monitor real space conditionsMove viewpoint

25.0℃

Select target appliance (lamp), and turn it on

Target environment

13

Home server

Remote controller terminal

Sensors

Control

•Reflect information in virtual space

•Allow intuitive operations for appliances

Appliances

User

Wireless communication

3D Virtual Space (UI)

Operation

Synchronization

Target Real Space

Communication

Outline

14

1. Background and Goals

2. UbiREMOTE Framework

3. Proposed Techniques– Lightweight (LW) 3D Rendering– Reflecting Real Space Info into Virtual Space

4. Evaluation

5. Conclusions and Future Works

Why we need LW 3D rendering?• Motivation

– Render 3D objects on mobile terminals– However, there are so many 3D objects in home, and

mobile device’s 3D rendering capability is not sufficient

• Requirements– Smooth viewpoint movement– Quick response when operating an appliance– Sufficient quality of rendered image

15

16

Straightforward approach• Thin Client Method

– A powerful server renders 3D image, captures a still image from a specified viewpoint, and sends the image to a client

• Drawback– Large operation delay (response time) is produced

4. Drawing image2. Renderingand capturingstill image

3. Sending captured image

1. Operation

Server Client

Some time lag produced

17

Tradeoff for 3D rendering

short

Client renders all objects

Client renders all objects

Thin C

lient Method

Thin C

lient Method

Response time

3D renderingheavy light

long

• Our approach– Propose compromise between two methods– Utilize a fact that users are likely to operate

appliances near viewpoint

18

• Surround viewpoint with a cube, where 6 still images are captured for virtual space and put on its faces

• Client renders only objects inside the cube

Proposed LW 3D rendering technique

These objects are drawn as background images (rendered by server)

These objects inside the cube are rendered by client

Viewpoint

VP

Cube

Updating cube when viewpoint moves

Set a sphere of radius R with the same center as cube

While viewpoint (VP) stays within the sphere

No update of cube When VP goes out of sphere

Prefetch BG images for

new cube, and replace

current cube by new oneR

BG image

20

Message sequence for cube update

viewpoint movement

Viewpointmovement

ServerClient

New center pos for cube

BG images for new cube

User going out of sphere

Rendering BG images

Updating cube

Updating cube

User going out of sphere RendereingBG images

Outline

21

1. Background and Goals

2. UbiREMOTE Framework

3. Proposed Techniques– Lightweight (LW) 3D Rendering– Reflecting Real Space Info into Virtual Space

4. Evaluation

5. Conclusions and Future Works

22

• Difference between virtual and real spaces causes the following situations

• A user remotely operated a heater in living room (on the screen), but it actually had been moved to children room

• Approach to avoid above situations– Show some real space information in virtual

space• Temperature, humidity, etc• Position of appliances / furniture

Why need to reflect real space info?

23

Reflecting state change in real space into virtual space

Home server

Wireless communication

Communication

Virtual space

living roomTemp:25CHum:60 ％

Sensors at each room detect temperature change, movement of appliance/ furniture, etc

Sensors at each room detect temperature change, movement of appliance/ furniture, etc

Deploy sensors at each room, and update information when there is a big change in sensor values

Detection/reflection of appliance positions

• Attach to each appliance a sensor node equipped with– Acceleration sensors for detecting movement– ZigBee communication capability for estimation of

new position

• Use existing ZigBee based localization techniques to estimate new position of appliance

24

Outline

25

1. Background and Goals

2. UbiREMOTE Framework

3. Proposed Techniques

4. Evaluation

5. Conclusions and Future Works

26

Experimental evaluation

• Implemented UbiREMOTE prototype on tablet PC • Evaluated performance

– Smoothness : Frame rates and response time– Usability : Questionnaire for User satisfaction in remote operation

TVElectric

Fan

Ceiling Light

Stand Light

CD Player

Floor plan and position of viewpoint

Controller of viewpoint

Controller foran appliance

27

Exp1 : Rendering speed

will achieve practical rendering speed on mobile terminal

Environment : Tablet PC

# objects (# objects inside cube)

# polygonsFrame rate (fps)

Render all objects

20 200000 1

Proposed 20 (1) 108 276

Proposed 20 (2) 2100 262

Proposed 20 (3) 4100 250

OS: Windows XP tablet PC editionCPU: Pentium M 1.2GHzMemory: 512MBGraphics Chip: Mobile Intel ®915GM

Exp2 : Response time (system configuration)

28

Mobile Terminal（ Tablet PC ）

Home Server

UPnPTCP/IP

3G cellular

network /WiFi

Electric Fan

Light

Heater

infrared Remote Controller

Exp2 : Response time (Result)

Reasonable response time for practical use 29

UbiREMOTE

appliances (a)WLAN (b)CellularElectric

fanON 0.94 3.22

OFF 0.92 3.14Light ON 0.73 2.98

OFF 0.61 3.13Heater ON 0.93 3.18

OFF 0.84 3.30average 0.828 3.141

30

Exp3:Usability evaluation by questionnaire

• 6 testees (graduate students)– operated 4 appliances by Remote controller that has

text-based interface and UbiREMOTE, – Return questionnaires.

ComparedText-based

remote controller

• It is easy to monitor appliances/conditions with UbiREMOTE31

サンプル数：24

サンプル数：24

■ Very Good■ Good■ Not good■ Bad

Q1: Is operation comfortable?

Q2:Can monitor conditions of appliances?

Text-based

Exp3 : Result of usability evaluation by questionnaire(1)

6 testees×4 appliances= 24samples

32

• Q3 : Did you feel something wrong in UbiREMOTE?

Exp3 : Result of usability evaluation by questionnaire(2)

Proposed LW 3D rendering does not spoiluser feeling and comfort in operation

3333

Video Demonstration

34

Conclusion and Future work• Conclusion

– UbiREMOTE framework• Intuitive and unified GUI for remotely controlling appliances

– Techniques to realize UbiREMOTE• LW 3D rendering based on cubic virtual space• Automatic reflection of real space information

– Performance evaluation• Achieved great improvement of 3D rendering performance• Usability was much better than Text-based interface

• Future work– Implementation on iPhone and Android– Usability evaluation by more detailed scenario

35