1

智慧型家庭網路之技術與應用

Professor Yau-Hwang Kuo

DirectorCenter for Research of E-life Digital Technology

(CREDIT)

National Cheng Kung University

Tainan, Taiwan

Outline Introduction Structure of Smart Home Network Realization of Device & Network Layers Agent-based Platform Affective HCI Integrated Perception Cognition Layer Smart Home Services Conclusion

Trend of Digital Home

House_n (MIT) 、 Aware Home (Geogria Tech.) 、 Interactive Workspace (Stanford Univ.) 、MavHome (UTA) 。

Digital Home Working Group: HP, Intel, IBM,... ECHONET: Energy Conversation and Homec

are Network. CELF: Consumer Electronic Linux Forum. OSGi: Open Service Gateway Initiative Easy Living: Microsoft

Scenarios of Digital Life

1. smart digital housekeeper.2. ubiquitous digital nursing agent.3. affective digital tutor.4. ubiquitous home security monitor.5. ubiquitous home content service.6. universal cyber circles.7. ubiquitous universal messaging service.8. personal knowledge warehouse/navigation.9. nomadic personal digital secretary.10. secure traffic navigator.

Microsoft’s View for Digital Home Solution

Total connectivityNo more islands of functionality

Personalized experiencesCustomized entertainment,

communications, and controlUbiquitous access

Your PCs, devices, and content,securely accessible everywhere

Microsoft’s View for Digital Home Solution

Technology “by invitation only”, not imposed

Highly personal and personalized spaceVirtually random, unmanaged “build out”Complex mix of products and services

Issues of Digital Home

人機互動能否人性化？robustness 、 adaptability 、 multi-modal collaboration 人性化互動特質。

感官、認知、情緒、協調、合作實現人性化互動的技術要素。

ubiquitous multi-modal affective human-machine interaction 數位家庭的人性化互動需求。

Issues of Digital Home (cont.)

人際互動能否得到提昇擴大？去空間限制、去時間限制、去工具限制、去安全限制。

家電間的協力合作能力能否得到提昇 ?connectivity among appliances 、 autonomous collaboration of appliances 、 interoperability of appliances 。

Issues of Digital Home (cont.)

人在數位生活空間的自由度是否得到提昇？可移動性、可轉移性、可調整性。ubiquitous integration home network 、location-awareness 、 universal access 、multi-modal human-machine interaction

Issues of Digital Home (cont.)

人在數位生活空間的便利度是否得到提昇？生活機能完整性、設備與網路無縫結合度、生活機能可獲性 (availability) 、用戶干預度、操作易度、穩私與安全等。

人在數位生活空間所獲得的生活輔助機能能否得到提昇 ?

Smart home network is necessary!

Goals:Infrastructure & applications

Create a new life space supported by a smart home service network and attached digital appliances.

Develop e-services over the smart home network and digital appliances to realize a new life style.

Develop a service modeling and execution environment over the smart home network to realize various e-services.

Goals:technologies

Develop nomadic HCI technologySpeech, vision, physiology, sensors.

Develop affective HCI technologyDevelop agent-based home service net

work middleware.Develop embedded platform & SoC for

smart appliances.

Applications (health care, entertainment, surveillance, etc.)Application Layer

Speech Vision (face) Physiology SmellPerception Layer

Home Network (802.11, Bluetooth, HomePlug) + Mobile Internet (SIP +3G)Network Layer

Layered Structure of Smart Home Service Network

Service Model Execution Platform (script translation, scheduling, QoS)

Integrated Perception

Device LayerNetworked Physiology

& EnvironmentMonitoring Appliances

Networked Microphones;

Cameras;Speakers

Home Comm. Gateway;Home Perception Server;

Home Media Center

WirelessA/V Streaming

Appliances

Mobile Agent PlatformAgent Layer

Emotion / Semantics / Behavior / Intention Understanding

Corpus of Knowledge(Ontology)

InferenceEngine

Natural LanguageProcessing

(text, spoken)

Cognition / Affection Layer

Vision (gesture)

Application Scripts for Various Living Support Functions

Service Layer

Speech Vision

Perception Layer

Home Network (802.11 + Home Plug)

NetworkLayer

ContentDelivery

Content & Context Information Collection

Device Layer

電子丫環阿桂

硬體設施

Device BridgeAdaptation

Layer

Semantics/Behavior/Intention/Emotion/Context Understanding

Corpus of Knowledge(Ontology)

Human-MachineInteraction Engine

Natural LanguageProcessing

Cognition/Affection

Layer

Location Sensing

BackboneAccess Networks (FTTH + 3G)

資訊伺服器阿文

硬體設施

屋控伺服器阿金

硬體設施

家電設備

感應器通訊伺服器

阿銘硬體設施

CO Routers

Internet/WWW Servers(阿美)

IPv4/IPv6

HTTP/RTP/RTSP (streaming) + SCP/RTCP/UPnP/SOAP (control) + RDP (UI remoting) + SIP (messaging)

Device and Media ManagementOSGi Gateway

Remote AccessManagementProtocol Bridge Media Adaptation

Physiology Sensing Environment Sensing Text

ContentRetrieval

Telephony/Messaging

ActivityScheduling

ServiceDiscovery

ApplianceControl

OnlineTrading

ActivityPrediction

AwarenessHandling

ServiceAggregation

UI Adaptation

Agent-based Task Scheduling/Dispatch/Migration

Web

SERVICE

PLATFORM

阿桂的工作環境：Layered Structure of Smart Home Network

Device & Network Layers: types of digital appliances

Client-type devices 802.11g-based multifunctional audio/voice adaptor 802.11g/MPEG-4-based multifunctional video adaptor 802.11g/MPEG-4-based smart IP camera Bluetooth-based ECG device

Gateway-type devices Multimedia communication gateway

Server-type devices House control server Human-machine interaction server Content server Application server

Device & Network Layers: relationship among server appliances

用戶端設備

FTTH/3G/WiMAX

通訊伺服器

CO

Internet/WWW

Telephony

house control & housekeeping devices

data store

A/V devices

WiFi/Home Plug

WiFi/Home Plug

主

人 應用伺服器

屋控伺服器

WiFi/Home Plug

WiFi/Home Plug

內容伺服器

Scheduling Algorithm XMLXML

Service Agent

Task agent

Task agent

Task agent

ASI

BIS

PMS

LKN

FEA

ASI_1_1

ASI_1_2

BIS_1_1

BIS_2_1

PMS_1_2PMS_1_1

LKN_1_1

LKN_2_1

FEA_1_1FEA_2_1

Service Server

Register

SDHScenarioServer DBDB

User Request

LocationServer

Script Script

XMLXML

What to do ?

How to do ?

Where to do ?

CommonAPI

Architecture of agent platform

Agent-based Runtime Environment Execution environment: IBM Aglets system Common API

Event_Trigger

getDataFromSub(Int subsystemId,Int destSubsystemId

String[][] function_name, parameters)

Start_Service_Agent (Int subsystemId,Int deviceLocation, String text)

Start_Service_Agent (Int subsystemId,Int deviceLocation, File file)

getSDHStatus(void)

getSubList (void)

getFunctionList (void)

getSubList (Int subsystemId )

getScenarios (void)

getUserLoc (Int userId)

Event_Trigger

getDataFromSub(Int subsystemId,Int destSubsystemId

String[][] function_name, parameters)

Start_Service_Agent (Int subsystemId,Int deviceLocation, String text)

Start_Service_Agent (Int subsystemId,Int deviceLocation, File file)

getSDHStatus(void)

getSubList (void)

getFunctionList (void)

getSubList (Int subsystemId )

getScenarios (void)

getUserLoc (Int userId)

Adaptive Service Provider:architecture

Scheduling Algorithm

Scheduling Algorithm

User request

(data, args)

XMLXML

Service Agent

Service Agent

Task agent

Task agent

Task agent

Task agent

Task agent

Task agent

ASI

BIS

PMS

LKN

FEA

ASI_1_1

ASI_1_2

BIS_1_1

BIS_2_1

PMS_1_2PMS_1_1

LKN_1_1

LKN_2_1

FEA_1_1FEA_2_1

Service Server

Service Server

Register

Service Agent

Service AgentTask

List

Scheduling Algorithm

Scheduling Algorithm

User request

(data, args)

XMLXML

Service Agent

Service Agent

Task agent

Task agent

Task agent

Task agent

Task agent

Task agent

ASI

BIS

PMS

LKN

FEA

ASI_1_1

ASI_1_2

BIS_1_1

BIS_2_1

PMS_1_2PMS_1_1

LKN_1_1

LKN_2_1

FEA_1_1FEA_2_1

Service Server

Service Server

Register

Service Agent

Service AgentTask

List

Adaptive Service Provider:functionalities

FunctionalitiesRegistry mechanism for subsystem, device and

functionalitiesService provider for user requestsLoad balanced service scheduling algorithm

according to system resources Agent cooperation mechanism

Adaptive Service Provider:components

Service server Subsystem and devices functionalities registration Service portal for usersMonitoring each subsystem and device

Service agents Provide service for each user requestService composition Task assignment and task agent dispatch according

to predefined XML-based scenarios

Adaptive Service Provider:components (cont.)

Task agent Execute each functionality on each subsystemCommon API

Service scheduling algorithmProvide a task list for service agent according to

registry and pre-defined scenarios in databaseA Petri net based & load balanced scheduling

algorithm for adaptive service path in each subsystem and device

Agent-based Middleware: mobility management

Location detectionDevice-followed type: mobile IP; signal ana

lysisDevice-free type: speech interaction; vision

monitoring.Seamless handoff and transcoding for u

biquitous service following Roaming path tracking and prediction

Agent-based Middleware: appliance collaboration management

Collaboration among homogeneous appliances: data fusion, task migration.

Collaboration among heterogeneous appliances: multi-modal HCI.

Scheduling, concurrency control & synchronization of collaborative tasks.

Self-organization for service deployment

Agent-based Middleware: interoperability management

Device bridgeProtocol bridgeTranscryptionTranscodingContent translation & adaptation

Agent-based Middleware: remote access management

Remote service deploymentremote service accessremote service managementauto-configurationservice re-directionservice aggregationUI remoting

Agent-based Middleware: other management functions

load management:Client-server load partitionServer load sharing Load scheduling of appliance farm

availability managementFault toleranceJust-in-time activation of appliances

service quality management

Affective Speech Conversation

EmotionEmotion

Dialog Dialog SystemSystem

TextText TextTextSpeechSpeech SpeechSpeechASRASR SynthesisSynthesis

EmotionEmotion

Emotional Speech Synthesis

TextAnalysis

Sad

Neutral

Angry

EmotionalSpeech Database

DatabaseSelection

SyntacticAnalysis

UnitSelection

SpeechSmoothing

TextInput

EmotionSelection

User’s Action

HappySpeech

Segmentation

Behavior Understanding by Vision

High-Level behavior understanding from videosState Machine

Human Activity RecognitionTwo-Stage recognition process

Accident/Abnormal behavior detectionContext & domain knowledge Combination

System Architecture

ImageImageImage

Video Stream

Segmentation &Tracking

Background(Update)

Tracking

Foregrounddetection

Feature Extraction

PostureRecognition

MotionEstimation

HistoryMap

Activity Recognition

PosturesAnalysis

MotionsAnalysis

SizeAnalysis

Accident Detection

Violent Motions

Lying & Static

Normal Detection

State Transition

ContextCombination

Abnormal Detection

Daily lifeinformation

Temporalinformation

Method – Activity Recognition

Activity RecognitionLevel 1 - postures

Posture SequenceLevel 2 – motion/history

History Map Matching

Method – Behavior understanding

BehaviorNormal behavior

State Machine Activity + Contexts

Abnormal behaviorNormal behavior + domain knowledge

AccidentUnreasonable activity + domain knowledge

Facial Expression Analysis

Face Acquisition

Acquisition Segmentation

Facial Feature Extraction

Deformation Extraction

Motion Extraction

Representation

Facial ExpressionClassification

RecognitionKey frameSelection

YCbCr

Color

space

Eye Region

Mouth Region

RegionOf

Interest

EyePoints

MouthPoints

Displacement

Vectors

Fuzzy

Neural

NetworkInvariantMoments

OpticalFlow

Key Frame

Image Sequence

Results

Integrated Perception:fuzzification of reference perceptual models

Manipulate all kinds of perception in a uniform process to ease the perceptual integration.

Due to high vagueness of perception, fuzzy logic based approach is a good choice to establish the reference models of perception.

The reference models which fuzzify perceptual attributes and perceptual decision subspaces will be embedded into the integrated perception model.

FL-based Acoustic Reference Model for Emotion Recognition

feature extraction

AAU1

model

SVM clustering

for emotion 1

SVM clustering

for emotion 2

SVM clustering

for emotion V

…

speech corpus

AAU2 model

AAUS model

… …

fuzzification of acoustic features (AFs) and construction of acoustic action unit

s (AAUs)

FL-based Acoustic Reference Model for Emotion Recognition

(cont.) Adopt SVM clustering approach in the subspace of

each emotion type to gather the clusters of acoustic training patterns.

Inspect all produced SVM clusters in the whole feature space and merge the highly overlapped clusters.

Each cluster is modeled as an AAU represented with its fuzzy cluster center where each feature is a fuzzy set whose membership function is determined by the least-square curve fitting approach on the feature values of training samples included in the cluster.

FL-based Acoustic Reference Model for Emotion Recognition

(cont.) The mapping between AAUs and emotion typ

es is dependent on the SVM clustering result of each emotion type.

Each emotion type is associated with a set of clusters of acoustic samples. The weight of each cluster is determined by the ratio of the number of samples it contains with respect to the total amount of samples of the same emotion.

FL-based Facial Reference Model for Emotion Recognition

graphical head model

morphological process to simulate

AUs

FACS AUs identification process

feature points (FPs) extraction process

fuzzy logic based reference model for

FACS

correspondence

Membership grade

Membership grade

FP1 value

FP2 value

FAU1 FAU2

FAUi FAUj FAUk

FL-based Facial Reference Model for Emotion Recognition (cont.)

Intend to construct a computational reference model for FACS action units based on the measurable features of facial expression.

An approach similar to the construction of acoustic reference model is adopted.

The training samples are generated from a generic head model with necessary morphological manipulation.

FL-based Facial Reference Model for Emotion Recognition (cont.)

The membership functions will be determined by the least-square curve fitting approach according to the sample patterns produced from the morphological process.

Each AU may just represent a partial facial expression and relate to more than one emotion.

FAU1 FAU2 FAUK AAU1 AAU2 AAUS

FP1FPn AF1 AFm

emotion type layer

representative concept layer

scaled feature layer

primaryfeature layer

Face Features Expression Acoustic Features

Fuzzy Neural Network for Integrated Emotion Recognition

Fear Anger Surprise Fear Anger Surprise

Fuzzy group decision process

{< total ordering of emotion types>, group level of agreement}

Fuzzy Neural Network for Integrated Emotion Recognition (cont.)

All kinds of perceptual information are fused by the FNN model to realize emotion recognition.

Each appliance will have an instance of the corresponding FNN to join the emotion recognition job.

A two-layered (emotion type & concept layers) BP learning algorithm is adopted by using the training samples in constructing reference models. The fuzzy group decision process does not join the learning.

Scaling input value to [0,1] in the second layer is realized by the membership function of the corresponding fuzzy set.

Fuzzy Neural Network for Integrated Emotion Recognition (cont.)

The links between AUs and scaled features are not fully connected.

The FAU/AAU nodes realize normalized weighted sum for the membership grades of input features weighted by their respective link strength.

Each emotion type node determines output value by the normalized weighted sum of its inputs from the representative concept layer.

Cognition Layer:understanding and response

Understand the semantics of multi-modal expression.

Classify and recognize the intention/

need/emotion of semantic expression.Summarize the semantics of multi-

modal expression according to classified result.

Cognition Layer:understanding and response (cont.)

Predict the user behavior sequence according to the classified result.

Schedule the response sequence according to the prediction result.

Determine the instant response.

Stimulus Perception Cognition

spokenlanguage

gesture

faceexpression

physiologicalsignals

text

SpeechProcessing

VisionProcessing

SignalProcessing

VideoProcessing

SpeechProcessingApplication

Control

Response

SemanticFeature

Extraction Conceptualization

Event Detector(Neural Network- based Approach)

EmotionRecognition

EmotionEpisode

Discovery

OntologyPersonalEvent /

Emotion Log

ContextualRules

StimulusSemanticSummaryExtraction

User BehaviorPrediction

(Episode-based Approach)

ResponseScheduling

InstantResponse

Determination

StimulusResponseTemplates

EventSequence

Case base

EmotionSequenceCase base

SemanticExpression

EmotionAttributes

Features Concepts

Events

SemanticSummary

Emotion Episodes PredictionResult

Response

Emotion Types

ResponseRoadmap

Smart Home Services

nomadic content services health care by integrated perception smart home surveillance smart e-mail and calendar arrangement

Conclusion

Life style of human being will be heavily affected by ICT, but the technological gap is still big.

Ubiquitous HCI and OCI technologies will be important to realize digital life style.

Cognitive computing and affective computing are important to improve the effectiveness of HCI technology.

Description of Context-Aware Middleware

UserProfile

AdmissionControl

PersonalAgent

ContextReasoning

ContextAggregator

ResourceManagement

WrapperWrapper

ServiceDevice

ServiceAgent

Context-Aware Middleware Architecture

OSGi Platform

UPnPBundle

ContextReasoning

Bundle

ResourceManagement

Bundle

WrapperBundle

LocationDetection

SpeechRecognition

PostureRecognition

Identification

Context Aggregator and

Ontology Reasoning

Interface

Bundle RepositoryJADE

Agent Platform(Service Scenario)

JAVA Virtual Machine

Operation System