P.1

企業微服務平台建置與維運經驗分享Enterprise Microservice Platform and Operation Experience Sharing

謝逸凡 Ivan Hsieh

P.2 2

Agenda

• Microservice Architecture

• How to break a Monolith into Microservices

• Microservice Operation Monitor

Developer

Code Build Deploy

Monitor

Traditional

Container Architecture

Serverless( FaaS / BaaS )

Microservice Cloud Native Architecture

Focus on Service Objective

• 擴展企業規模• 支援進入新市場• 支援既有市場創新• 短時間交付新功能

服務目標 架構原則

• 標準 REST• 封裝舊系統• 消除整合資料庫• 發佈整合模型• 獨立小服務• 快速部屬• 減少客製

設計與交付實務

• 降低慣性 加快上版速度

• 提高可用度，容量可以快速調整

• 服務使用重組• 資料快速整合處

理產生應用

Characteristics of a Microservice Architecture

容易部屬 ，自動化• 一份程式代碼 多環境部屬• 環境配置檔案與程式代碼分開隔離• DevOps CICD • 獨立部屬• 開發環境與正式環境相同

技術的異質性• 專注服務• 組織調整，去中心化• 持續創新

韌性 高可用度• 失敗隔離 ,隔艙• 隱藏內部細節• 數據隔離，資料一致性• 一個或多個無狀態服務一起運行

擴展• 持久化資料都在外部，快速容量

擴展• 說明依賴關係並隔離管理依賴• 非同步架構

服務建構開發&運行• 服務重組• 後端服務當作附加資源• 透過網路API調用後端服務• 漸進式設計• Log / 管理監控機制• 快速重啟容器特性

Microservice Architecture

6

Client App

WebApp

API G

atew

ay

Identity micro service

Catalog micro service RDB

Cache

Ordering micro service

RDB

Marketing micro service

NoSQL

Even

t Bus

Microservice Build

服務切割越細可擴展性越強，彈性越好，複雜度也越高

• 外 鬆耦合Loosely Couple• 內 高度聚合• 微服務邊界與介面 Boundary Context / Explicit Interface • 微服務間共用資料模型• 微服務隱藏細節• 不夠瞭解服務過早分解• 需要找到平衡點與自己的技術管理能力對齊• 不要想要㇐步到位

3 Tier

UI

BusinessLogic

Database

UI

Service

Database

Service Service

ESB (Business Logic)

UI

API

Service Oriented Micro Service

MS

Database

MS

Database

MS

Database

MS

Database

MS

Database

Application Modernization

One Size fit all tight coupling

Fine Grained components Loosely coupling

SuperFine service Loosely decoupled

AP

Load Balancer

AP AP

Load Balancer

AP AP AP

Service1

Load Balancer

Service2

Service3Service1

Load Balancer

Service2

Service3

Microservice Application Stateless

Service2

Break a Monolith into Microservices

Service1

Service

Service3

Monolith Service

Service 4Service2

Service

Service3Monolith Service

Service 4Service2

API

Cache

Break a Monolith into Microservices - Database

Database Shading

service A

Read/Write splitting

service E

Master

Slave Slave

Portal2

service A

Portal2

service B

Portal2

service C

Write

Read

CDN

1~100

200~300

Data Table FK

Break a Monolith into Microservices - Database

資料庫依賴問題• 打破FK • 公用靜態資料• 共用靜態資料庫• 共用資料表

Data Integrity Check

Break a Monolith into Microservices - Database

Share Table

Share Data

Read Only Copy to each service Property fileRead/Write Build a data service

Spilt Table

Legacy Transaction Boundary

Database Two Phase Commit

Java XA

Database Two Phase Commit

DB Link DB Function

Microservice A

Event Bus (Push / Subscribe)

Microservice B

Microservice C

Microservice Transaction Boundary

API / Services

Compensating Service

bklog

Data Eventual Consistency (CAP)

• 稍後再試• 終止交易• 分散交易

Microservice Reporting & DW

Microservice

DatabaseService

QueueData Service

PlatformEvent Data Pump

ETL

Cloud Native

AP

Database

Legacy System

Web

ODS/DW

ETL ETL

Service Application

Platform

Application ModernizationData Service API

Data Mark BI

StorageClone

Microservice Operation

• Testing Policy • Deployment

• DevOps CI CD• Service Mesh

• Architecture • Deployment Enhancement• Error Handle

• Log System

Microservice Architecture

18

Client App

WebApp

API G

atew

ay

Identity micro service

Catalog micro service RDB

Cache

Ordering micro service

RDB

Marketing micro service

NoSQL

Even

t Bus

Microservice Testing

Identity micro service

Catalog micro service RDB

Cache

Ordering micro service

RDB

Marketing micro service

NoSQL

Build UnitTesting

ServiceTestingCode

Build UnitTesting

ServiceTestingCode

Build UnitTesting

ServiceTestingCode

Build UnitTesting

ServiceTestingCode

E2E Testing

• 端到端測試複雜• Who is owner• 相互干擾

Microservice Testing – Load Test

VU (Virtual User) 同時執行客戶數TPS (Transaction Per Second) 每秒可執行的交易RT (Response Time) 响應時間

VU (Virtual User)

TPS

RT

CPU

Best Loading Max Loading

• 每次都要執行壓測 ?• 單元壓測• 服務壓測• E2E 壓測

K8S Prod Cluster

SIT Image Registry

Code Change/Commit

V1Image

V2 Image

Code V1

Trigger build code

& test

Code V2

Push service image

sync image

Deploy request

Prod Image Registry

V1Image

V2 Image

K8S SIT Cluster

Service V1

Service V1

Exchange Pool

V1Image

V2 Image

sync image

Release event

A10 Deploy request

Pull image& deploy

Service V2

Service V2

Service V1

Service V1

Service V2

Service V2

Service V0

Service V0

Pull image& deploy

DevOps CI/CD - Docker 21

Virtualization Container Container Orchestration Tools

Microservice Platform

微服務間通信的基礎設施層，以Slidecar模式與應用配合運行，負責服務之間的網路調用、路由轉發、負載均衡、限流、熔斷和監控。

Application

Core Functionality

Sidecar

Peripheral tasks such as：•Platform abstraction•Proxy to remove service• Logging•Configuration

Service Mesh Concept

Application

Sidecar

Service Mesh Concept

Istio Service Mesh Architecture

Policy & Telemetry(Mixer)

TLS Certs(Citadel)

Control Plane

Sidecar (Envoy)

App A

Pod

Service A

Sidecar (Envoy)

App B

Pod

Service B

Sidecar (Envoy)

App C

Pod

Service C

Kubernetes API

Secret Service Account

Secret Secret Secret

mTLS mTLS

Config Data(Pilot)

Data Plane

jaeger

Service Mesh Deployment

Service / Ingress

V1 V2

Original Traffic

Mirrored Traffic

Service / Ingress

V1 V2

Shadow Bule /Green

Service Mesh Canary Deployment

Service / Ingress

Canary

Traffic split

V1

95% 5%

Service / Ingress

Canary

Contain Base

V1

User agent : iPhone User agent : Android

Service Mesh Error Handle

Retry

Service / Ingress

Timeout 3S Delay 3S

Return Error

• Return Error • Retry times• Delay

Service Mesh Error Handle

Circuit Break

Service / Ingress

Time out or Error

Hit ThresholdReturn errorBreak Service

Service / Ingress

Retry

Hit ThresholdReconnect Service

E2E Monitor

Business Transaction

Hybrid IT Infrastructure

Application

StorageWeb Database

AP ServerNetwork Server

Call CenterStore

eService

CRM

ERP

Ordering Billing

Use service E2E point of view to create a monitor

AIOps Monitor

Time 0 12 24

CPU100

50

0

Fix Threshold

Vibrable Threshold (+- 10%)

Add resource Alert & check root cause

Threshold Actual

Monitor Log Center – Service Trace

Web

API

Service Trace

Business

啟用服務 開戶

A1 A2

A11 A6

B1 B2

B5 B7

C1

C5

D1

D2

S1 S3 S6 S9

A2 B2 C1 D1 B1 A1 B1 D1 B1 C1

A服務 S1

A2 B2 C1 D1 B1

A6 B7 C5 D2 B5

Log Center

Web

Trace ID

Span ID

Parent ID

Host IP

App name

Service name

End Time

Elapsed time

Return Code

OPID

Client IP

Session ID

API

Trace ID

Span ID

Parent ID

Host IP

App name

Service name

Start Time

Elapsed time

Return Code

Chanel code

Protocol name

APP

Trace ID

Span ID

Parent ID

Host IP

App name

Service name

Start Time

Elapsed time

Return Code

DB

Trace ID

Span ID

Parent ID

Host IP

App name

Service name

Start Time

Elapsed time

Return Code

DB Name

SQL

Log Center

Span ID : S001

Trace ID : T001Parent ID

Span ID : S002

Trace ID : T001

Parent ID : S001

Span ID : S003

Trace ID : T001

Parent ID : S002

Span ID : S004

Trace ID : T001

Parent ID : S002

Span ID : S005

Trace ID : T001

Parent ID : S003

End Time : 14:20:13:20Elapsed time : 20

14:20:13:2020

T001

Start Time : 14:20:13:01Elapsed time : 7

7 14:20:13:0914:20:13:01

8

614:20:15:10 14:20:13:19

4

S001

S002S003

S004 S005

14:20:13:00

jaeger

35

What’s Next

►Monitor & Log continue enhancement ►Automation for Microservice environment►Use more Visualization tool to support operation

►cloud-native application on Service ,Data ,Container infrastructure

►Focus on service►Not all in Microservice