引領創新科技 建設智慧澳門
智慧城市生態圈的AI應用技術賈維嘉,講座教授、實驗室副主任
Weijia Jia, Chair Professor
Deputy Director, State Key Lab of IoT for Smart City
University of Macau 1
引領創新科技 建設智慧澳門
➢ 智能搜索➢ 語義理解➢ 知識圖譜
➢ 對象關聯➢ 實體識別➢ 霧式管理
➢ 全效感知➢ 時空覆蓋➢ 三元協同
Feed
back
Co
ntro
l
Ap
p D
ata
2
Info/Data & Knowledge Flows
IoT/sensor networksApplications
候選子圖 Top-k結果圖知識圖譜用戶查詢圖
應用層
中間層
感知層
引領創新科技 建設智慧澳門
我們要做什麼?智慧城市物聯網生態圈!
引領創新科技 建設智慧澳門
1. Background
2. AI at Bottom: Sensors/IoT
3. AI at Middle: Fog/Edge Networks
4. AI at Upper: Knowledge Graph
5. Discussions: AI S&P
Contents
3
引領創新科技 建設智慧澳門
o 2018年7月獲國家科技
部批准
o 2018年10月8日揭牌
背景
4引領創新科技 建設智慧澳門
引領創新科技 建設智慧澳門 5
智慧城市物聯網國家重點實驗室
引領創新科技 建設智慧澳門
1. 全球已啟動或在建的智慧城市達1000多個
• 正在以20%以上的速度增加
• 歐盟在2009年提出《歐洲2020戰略》
• 日本提出 “智慧日本”
• 新加坡推動政府服務的電子化
• 荷蘭提出 Amsmarterdam 城市能源地圖
2. 國內截止2015年底
• 85%以上的城市都在進行智慧城市建設
• 智慧城市試點已接近300個
• 規劃對智慧城市的投資總規模將逾5000億元
智慧城市國內外研究現狀
6引領創新科技 建設智慧澳門
引領創新科技 建設智慧澳門
澳門的挑戰與機遇
7引領創新科技 建設智慧澳門
安全防災
能源配置 生態影響
交通擁塞
引領創新科技 建設智慧澳門
State Key Lab@University of Macau
Fundamental Research & Key
Technology 基础研究/关键技术
为支撑
Talent Training人才培养为平台
City Development 城市发展为主线
IOTSC 智慧城市物联网国家重点实验室
Applications应用支持为特征
8
引領創新科技 建設智慧澳門
Research focuses
Smart Sensing 感
知技术
Data Analysis and Control
数据分析与控制技术
Advance Networking &
Fog Computing网络存储,传输与雾
计算
Platformof AI/Big
Data
Smart Grid
能源物联网
Smart
Transportation
基于物联网的智能交通
Public Safety Monitoring
and Disaster Prevention
城市公共安全监控和灾害防治
Big Data &
Intelligent Tech
大数据与智能技术
Intelligent Sensing
and Net Comm.
智能傳感與網路通信
9
引領創新科技 建設智慧澳門
1. Background
2. Bottom: Sensors/IoT
3. Middle: Fog/Edge Networks
4. Upper: Knowledge Graph
5. Discussions: AI S&P
Contents
10
引領創新科技 建設智慧澳門
➢ 智能搜索➢ 語義理解➢ 知識圖譜
➢ 對象關聯➢ 實體識別➢ 霧式管理
➢ 全效感知➢ 時空覆蓋➢ 三元協同
Feed
back
Co
ntro
l
Ap
p D
ata
11
Info/Data & Knowledge Flows
IoT/sensor networksApplications
候選子圖 Top-k結果圖知識圖譜用戶查詢圖
應用層
中間層
感知層
引領創新科技 建設智慧澳門
智慧城市物聯網生態圈!
Our IoT devices
1. High frequency
radio, low power
consumption, wall
penetration
2. Real-time video
push and alarm to
user's mobile
phone
3. WeChat alarm and
personal service
inquiry and report
12
Optimal patterns achieving 2D-coverage and k-connectivity: k = [1,…, 6]
Rs – Sensing RadiusRc – Connection Radius
2D Optimal Connectivity/Coverage
13
引領創新科技 建設智慧澳門
1. Background
2. Bottom: Sensors/IoT
3. Middle: Fog/Edge Networks
4. Upper: Knowledge Graph
5. Discussions: AI S&P
Contents
14
引領創新科技 建設智慧澳門
Key Tech: Fog/Edge Computing
PublicCloud A
Cloud/Servers
Fog/edge networks
App Environments
IoT & SensorNetworks
Feed
back
Co
ntro
l
EnterpriseCloud B
PrivateCloud C
Ap
p D
ata
IoT/sensors
Info/Data Flows
Bottom layer
Middle layer
Upper layer
How the data/tasks are handled by the
fog nodes?15
引領創新科技 建設智慧澳門
Issues with Fog/Edge
• Resource allocations
• Power/energy awareness
• Real-time QoS
• Mobility service support
• Offloading
• Edge-to-edge data transfer
• Edge-to-edge AI services …
Container Migrations
Migrating data/tasks➔Allocating resources from fog node 𝐹1to 𝐹3 through containers ➔ QoS ↑ costs↓ .
Migration Modeling and Learning Algorithms for Containers in Fog Computing, IEEE Transactions on Service
Computing (TSC), 2018 17
• Containers are much lighter than VMs
• Compared with VM, application data and runtime libs are migrated without migrating the guest OS.
Containers: Why not VM?
18
引領創新科技 建設智慧澳門
Delay
• Network delay:
• Computation delay:
• Total delay:
引領創新科技 建設智慧澳門
Parameter Attentions
• Resource utilization:
• Total power consumption:
• Container migration cost:
引領創新科技 建設智慧澳門
Nightmare: Dimensional Curse
• Delay of Ci at Fj
• Matrix of delays:
• Overall state space:
• To minimize• Delay 𝑑𝑡𝑜𝑡𝑎𝑙• Power 𝑝𝑡𝑜𝑡𝑎𝑙• Migration cost 𝑚𝑡𝑜𝑡𝑎𝑙 subject to resource
requirements:min𝐶 = 𝜔1𝑑𝑡𝑜𝑡𝑎𝑙 +𝜔2𝑝𝑡𝑜𝑡𝑎𝑙 +𝑚𝑡𝑜𝑡𝑎𝑙
𝑠. 𝑡. 0 ≤ σ1 𝐶𝑗.𝑙 𝑡 =𝐹𝑖
𝐶𝑗.𝑎 𝑡
𝐹𝑖.𝑐≤ 1 𝑖 = 1,2, … ,𝑚
• Difficulties: • Bin-pack (NP-hard) • High dimensional state
Objectives
22
• RL Loop:
• State 𝑆𝜏, Action 𝐴𝜏, and Reward 𝑅𝜏:• 𝑆𝜏 = 𝑋 𝜏 , 𝐶. 𝑙 𝑡0 + 𝜏 𝑇 , 𝐶. 𝑎 𝑡0 + 𝜏 𝑇 ∈ 𝑺.• 𝐴𝜏 = 𝐶. 𝑙 𝑡0 + 𝜏 𝑇 ∈ 𝑨.
• 𝑅𝜏 = − 𝜔1𝑑𝑡𝑜𝑡𝑎𝑙 𝜏 + 𝜔2𝑝𝑡𝑜𝑡𝑎𝑙 𝜏 + 𝑚𝑡𝑜𝑡𝑎𝑙 𝜏
• 𝑄 𝑣𝑎𝑙𝑢𝑒: 𝑄(𝑆𝜏−1, 𝐴𝜏−1)
Reinforcement Learning (RL)
23
引領創新科技 建設智慧澳門
1. Background
2. Bottom: Sensors/IoT
3. Middle: Fog/Edge Networks
4. Upper: Knowledge Graph – The key
5. Discussions: AI S&P
Contents
24
引領創新科技 建設智慧澳門
AI framework
PublicCloud A
Cloud/Servers
Fog/edge networks
App Environments
IoT & SensorNetworks
Feed
back
Co
ntro
l
EnterpriseCloud B
PrivateCloud C
Ap
p D
ata
IoT/sensors
Info/Data Flows
Bottom layer
Middle layer
Upper layerAI Framework
25
引領創新科技 建設智慧澳門
• 重點研究知識圖譜與數據相結合的學習與表示技術算法
• 探討城市大數據的多源獲取,提出多源數據的融合技術,利用遷移學習和聯邦學習對城市大數據進行挖掘和分析
• 提出將結構化信息和數據結合的聯合模型,以刻畫知識圖譜內外實體的有效聯繫
26
城市大數據與智能技術
引領創新科技 建設智慧澳門
27
研發具國際領先的智慧城市物聯網共性基礎科學及平臺方案;人-物&物-物之間的多關係提取與智能鏈接。
城市大數據與物聯網智能
代表性成果:
• 物聯網邊緣動態傳算:IEEE TPDS 2019,TSC2018
- 長尾任務動態調度
- 容器智能遷移
- 強化學習以降低解空間維度
• 語意實體多關係提取:AAAI 2019, ACL2019, EMNLP 2018, NAACL2019
- 實體多關係並行提取
- 優化膠囊網絡及设计新型對抗網路進行關係精煉提取
• 多反饋網絡實體信息抽象集成, IJCAI 2019,ACM Trans Data Mining/Discovery 2019
- 多反饋網絡建模- 關係網絡對演模型
引領創新科技 建設智慧澳門
1. Extracting Entities (E)
2. Extracting Relations (R)
3. Predicating E-R in KG
Knowledge Graph (KG) --Fundamental Challenges
28
引領創新科技 建設智慧澳門
What is Knowledge Graph?• Freebase, Concept Graph, NELL, WordNet…
• Triplet: (head entity--A, relation--r, tail entity--B)
• ”The height of Liu is 174cm” ➔ (Liu, height, 174cm)
引領創新科技 建設智慧澳門
Entity Triple Extractions
• Entity linking forming KG• entity = node• relation = edge• fact = triple (A, r, B)
• Typical knowledge graphs• WordNet, Freebase, Nell• DBpedia, Wikidata• …
App Example: Time-Sync CommentsVideo-Tagging
• Time-sync comments are crowdsourced & interactive
• Content involving current video, feelings of users or replies to other comments
• Obtain more accurate and specific tags
(1) Semantic relevance
(2) Real-time
(3) Inter-dependence
(4) Noise
31
Mining Tags as Relations
• 1. Graph Construction
• 2. Topic Partitioning & Radius
• 3. Weight Distribution and Tag Extraction
ICME, 2017, ICME 2019
Outcomes: {User A, “Messi”, User B} { Video V, “Messi”, [0:18, 0:24]}
32
引領創新科技 建設智慧澳門
1. Extracting Entities (E)
2. Extracting Relations (R)
3. Predicating E-R in KG
Fundamental challenges
33
引領創新科技 建設智慧澳門
Two recent works on extracting relations from
Natural Language:
• Single Relation Extraction in Sentence
• – EMNLP-18
• Multiple Overlapping Relation Extractions
• – AAAI-19
2. Extracting Relations
34
引領創新科技 建設智慧澳門
Problem and Challenges
• Multi-labeled Relation Extraction
ID Instances Relations
S1[Arthur Lee], the leader of Love, died on
Thursday in [Memphis].person/place_death
person/place_lived
S2[Arthur Lee] was born in [Memphis],
Tennessee, and lived there until 1952.person/place_birth
person/place_lived
S3
[Abraham Lincoln] was an American
statesman and lawyer who served as the
16th President of the [United States].
person/president_of
person/nationality
Overlapping
Relations
Discrete
Features
引領創新科技 建設智慧澳門
Extracting Overlapping Features
• Capsule Network
Capsule Network inImage Recognition
Capsule Network inRelation Extraction (RE)
Examples for Capsule Network in image recognition come from Aurelien Geron’s lecture.https://youtu.be/pPN8d0E3900
[Arthur Lee] was born in [Memphis], Tennessee, and lived there until 1952.
place_birth
place_lived
引領創新科技 建設智慧澳門
Attentive Capsule Network
• Attention-based Routing Algorithm
• Sliding-margin loss
引領創新科技 建設智慧澳門
Experiments• Experiments on two previous benchmarks (NYT-10 and
SemEval-2010 Task 8)
引領創新科技 建設智慧澳門
1. Extracting Entities (E)
2. Extracting Relations (R)
3. Predicating E-R in KG
Fundamental challenges
39
引領創新科技 建設智慧澳門
KGC– Predict Entity/Relations
• Triples in KGs.• (Isaac Newton, _birth_in, Lincolnshire)
• How to predict the missing part: (A,r, ?), (A, ?, B), (?,r, B)
John was bornin ?
John was born in Miami
KnowledgeGraph
VectorSpace
引領創新科技 建設智慧澳門
Translation-based Models
• Representation learning: Similar entities arerepresented as similar vectors
• Typical: TransE (Translation Embedding):
𝐴, 𝑟, 𝐵 ➔ Ԧ𝐴 + Ԧ𝑟 = 𝐵
• Optimization: minimizing Ԧ𝐴 + Ԧ𝑟 − 𝐵• E.g.: TransH, TransR, TransG, DKRL, TKRL,
SSP, …
TransT: Type-based Multiple Embedding Representations for Knowledge Graph Completion, Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2017.
引領創新科技 建設智慧澳門
Contributions: Multiple Embeddings
• Assign one vector for each semantics
𝒑𝒕𝒓𝒖𝒆 𝑨, 𝒓, 𝑩 = σ𝑖=1𝑛𝐴 σ𝑗=1
𝑛𝐵 𝑤𝐴,𝑖𝑤𝐵,𝑗𝑝𝑡𝑟𝑢𝑒 𝑣𝐴,𝑖 , 𝑣𝑟 , 𝑣𝐵,𝑗
𝑝𝑡𝑟𝑢𝑒is defined as
𝑝𝑡𝑟𝑢𝑒 𝑣𝐴,𝑖 , 𝑣𝑟 , 𝑣𝐵,𝑗 = 𝑒− 𝑣𝐴,𝑖+𝑣𝑟−𝑣𝐵,𝑗
引領創新科技 建設智慧澳門
Relation Prediction Classifications
• Predicting the missing relation when given two entities. FB15K is the benchmark dataset.
• Triple classifications predict if a given triple is correct. FB15K is the benchmark dataset of this task.
引領創新科技 建設智慧澳門
1. Introduction
2. Bottom: Sensors/IoT
3. Middle : Fog computing
4. Upper: Knowledge Graph
5. Discussions: AI S&P
Contents
44
From attacker’s view: What can attackers do in AI?
From AI’s view: How can we prevent attacks from AI?
ATTACKER
AI
privacysecurity
DATAMODEL
Security & Privacy
45
DeepFakes
46
SpamTracer: Manual Fake Review Detection for O2O Commercial Platforms by Using Geolocation Features, Inscrypt 2018.
Machine learning for detectingfake reviewers
47
Bae H, Jang J, Jung D, et al. Security and Privacy Issues in Deep Learning, ACM Computing Surveys 2018
Potential Threats in AI
48
引領創新科技 建設智慧澳門
➢ 智能搜索➢ 語義理解➢ 知識圖譜
➢ 對象關聯➢ 實體識別➢ 霧式管理
➢ 全效感知➢ 時空覆蓋➢ 三元協同
S&
P F
eed
back
Co
ntro
l
S&
P A
pp
Data
49
S&P Info/Data & KnowledgeFlows
S&P 5G & IoT/sensor networksApplications
候選子圖 Top-k結果圖知識圖譜用戶查詢圖
應用層
中間層
感知層
引領創新科技 建設智慧澳門
智慧城市物聯網生態圈
引領創新科技 建設智慧澳門
MOST China; FDCT Macau, SAR; NSFC
University of Macau
Shanghai Jiaotong University
Acknowledgements
50
引領創新科技 建設智慧澳門
謝謝!Thanks!
51