萬事萬物皆是 log - 系統架構也來點科普

萬事萬物皆是 LOG

Poga @ COSCUP 2016.08.20

LOGIf you want maintainable, understandable software system

什麼是 LOG

LOG 的性質

● 按照時間出現

● 出現過的就不會改變（append-only）

● 目的：記錄（某個時間）發生了（某件事）○ 常常用來作為 debug 推理的依據

Things based on LOG

● Database● Version Control● Distributed System and Distributed Consensus● Synchronization● Replication● Messaging● UI Framework● ...And almost EVERYTHING

聽完就理解了軟體架構的真理（誤）

POGASoftware Developer, g0v.tw, KKTIX(前)

資料庫

ID Username Balance

231 poga 100

7212 moon 12314

834 et 927

... ... ...Database

● 大量讀寫資料，長期儲存

● 確保資料正確性（relation, validation, schema, ACID）

● 支援多種高效查詢（SQL, index）

如果你從沒用過資料庫...

寫入資料時當機？

如何避免寫入資料時當機導致資料消失？

寫入：ID Username Balance

231 poga 100

...2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}...

真正寫入： Validation, Update Index… etc.

...2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}...

時間 發生了什麼事

WRITE-AHEAD LOG

LOG

● Fast and simple

RECAP

資料庫

ID Username Balance

231 poga 100

7212 moon 12314

834 et 927

... ... ...Database

● 單一 Database 擴充性有其極限

● 多台 Database，分散負載○ 需要確保多台 Database 之間資料同步

Replication

確保多台資料庫間資料同步？

寫入一台資料庫

寫入：ID Username Balance

231 poga 100

...2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}...

真正寫入： Validation, Update Index… etc.

一次寫入多台資料庫

寫入：ID Username Balance

231 poga 100

...2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}...

真正寫入： Validation, Update Index… etc.

...2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}...

真正寫入： Validation, Update Index… etc.

DB1 DB2

LOG-SHIPPING

LOG

● Fast and simple● Can be used to share “current state”

RECAP

微服務Microservice

Monolith v.s. Microservice

Monolith

● 所有邏輯集中在單一系統中，共

享儲存運算資源

● 容易開發

● 不易擴充

Microservice

● 將邏輯按照 Domain 切割，有

獨立的儲存與運算資源

● 容易擴充，團隊間分責明確

● 門檻較高：不易維護，需要進階

系統維護能力

微服務間怎麼互相溝通？

Event Stream

EVENT STREAM = LOG

LOG

● Fast and simple● Can be used to share “current state”● LOG are easy to parse/understand, Universal Interface

RECAP

前端架構Flux/Redux

● Many, many states

● Constantly changing state○ User Interaction, Animation, Network

● Hard to reproduce bug and debug

UI State

前端介面狀態多變，如何維護？

Flux/Redux

● Unidirectional Data Flow○ Append only○ Handle action one by one, reasonable

● Immutable State○ Produce new state based on previous state and action

● Everything else can be stateless(pure)

UNIDIRECTIONAL = LOG

LOG

● Fast and simple● Can be used to share “current state”● Text are easy to parse/understand, Universal Interface● Append-only, easy to reason, everything else can be

stateless

RECAP

區塊鏈Blockchain

● 2016.08.20 13:31:23, A 給 B 100 bitcoin

● State: A 跟 B 身上剩下多少 bitcoin

● Double Spend: ○ A 把 bitcoin 交給 B 後，不能再把同樣的 bitcoin 交給 C

Transaction

如何讓全世界都認知交易的成立？a.k.a. Avoid Double-Spending

● 如果○ 所有參與者都知道/可以驗證所有人的交易記錄

○ 既有的交易紀錄無法被竄改

○ 便沒有偽造空間

● Blockchain○ 將所有的交易記錄用 block 封裝，每個 block 替之前的交易記錄加密驗證

○ 想要偽造一筆交易記錄 = 必須偽造整個 blockchain 上的交易記錄

○ 所有人都基於相同的 blockchain 驗證交易

Distributed Consensus

LOG AGREEMENT = CONSENSUSRaft: A Consensus Algorithm for Replicated Logs

Apache Mesos - The Mesos Replicated Log (PAXOS)

LOG

● Fast and simple● Can be used to share “current state”● Text are easy to parse/understand, Universal Interface● Append-only, Immutable, easy to reason, everything else

can be stateless● Replicate log = consensus

RECAP

大數據

Data Pipeline

● > PB● Data coming from everywhere

○ User Input○ Business Data○ Tracking○ External Data

● ETL(Extract-Transform-Load)● Analysis, Machine Learning… etc

BIG Data

如何處理雜亂、無序、大量的資料？

Data Pipeline

APPEND ONLY LOG = HIGH SCALABILITY

LOG

● Fast and simple● Can be used to share “current state”● Text are easy to parse/understand, Universal Interface● Append-only, Immutable, easy to reason, everything else

can be stateless● Replicate log = consensus● High Scalability

RECAP

LOG = DETERMINISM

相同的過程 = 相同的結果DETERMINISM

Make your system deterministic

● 容易理解

● 容易DEBUG● 容易同步

● 容易儲存

● 容易擴展

LOG = DETERMINISM = 穩定的系統

設計系統架構時，先試著用 LOG 表示你的系統

Thanks!