e db: e t p in b f t e - hpts
TRANSCRIPT
![Page 1: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/1.jpg)
EXPODB: EFFICIENT TRANSACTION PROCESSING IN BYZANTINE FAULTTOLERANT ENVIRONMENTS
Suyash Gupta, Jelle Hellings, Thamir Qadah, Sajjad Rahnama, Mohammad Sadoghi
�1
Mohammad Sadoghi Department of Computer Science
Exploratory Systems Lab
18th International Workshop onTransaction High Performance Transaction Systems (HPTS)
November 3-6, 2019
![Page 2: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/2.jpg)
ExpoLab Members
Mohammad Sadoghi(Principal Investigator)
Suyash Gupta, PhD(Blockchain)
Thamir Qadah, PhD(Coordination-free Concurrency)
Masoud Hemmatpour, PhD
(RDMA KV-Stores)
Domenic Cianfichi, MSc
(Blockchain)
Shreenath Iyer, MSc
(Blockchain)
Robert He, MSc
(Coordination-free Concurrency)
Patrick Liao, BSc
(Blockchain)
Sajjad Rahnama, PhD(Blockchain)
Jelle Hellings, PostDoc(Blockchain)
![Page 3: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/3.jpg)
�3
SQL Analytics
FPGA Acceleration: FQP (Flexible Query Processor) [VLDB’10, ICDE’12, VLDB’13, ICDE’15, SIGMOD Record’15, ICDE’16, USENIX ATC’16, ICDCS’17, ICDE'18, TKDE'19]
Journey…
![Page 4: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/4.jpg)
�4
SQL Analytics
SQL Transactions
High-dimensional Indexing: (e.g., BE-Tree, BE-topK) [SIGMOD’11, ICDE’12, TODS’13, ICDCS’13, ICDE’14, ICDCS’17, Middleware'17]
Journey…
![Page 5: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/5.jpg)
�5
SQL Analytics
SQL Transactions
Concurrency Control Protocols: (e.g., 2VCC, QueCC - Best Paper Award) [VLDB’13, VLDB’14, VLDBJ’16, Middleware’16, TDKE’15, SIGMOD’15, ICDE’16, Middleware'18]
Journey…Co
ncur
renc
y Pr
otoc
ols
![Page 6: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/6.jpg)
�6
SQL Analytics
SQL Transactions
Concurrency Control Protocols: (e.g., 2VCC, QueCC - Best Paper Award) [VLDB’13, VLDB’14, VLDBJ’16, Middleware’16, TDKE’15, SIGMOD’15, ICDE’16, Middleware'18]
QueCC: Queue-Oriented Planning and Execution Architecture
Journey…Co
ncur
renc
y Pr
otoc
ols
![Page 7: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/7.jpg)
�7
SQL Analytics
SQL Transactions
HTAP Column-store: L-Store (Lineage-based Data Store) [VLDB’12, ICDE’14, ICDCS’16, EDBT'18, 34 filed US patents]
Graphs on SQL: (e.g., GRFusion) [SIGMOD’18, EDBT’18]
HTAP: Unifying Storage Layer (Lineage-based Storage Architecture)
Journey…Co
ncur
renc
y Pr
otoc
ols
![Page 8: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/8.jpg)
Journey…
�8
SQL Analytics
SQL Transactions
Agreement Protocols: (e.g., EasyCommit, Q-Store) [EDBT’18, DAPD’19]
HTAP: Unifying Storage Layer (Lineage-based Storage Architecture)
SQL Transactions
Agreement Protocols
Conc
urre
ncy
Prot
ocol
s
![Page 9: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/9.jpg)
�9
Journey…
SQL Analytics
SQL Transactions
Consensus Protocols: (e.g., ResilientDB, Blockplane, Blocklite) [SC'19, ICDE'19, DISC'19 (2x), arXiv'19 (6x)]
Agreement Protocols
HTAP: Unifying Storage Layer (Lineage-based Storage Architecture)
Resilient Consensus Protocols
Conc
urre
ncy
Prot
ocol
s
![Page 10: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/10.jpg)
�10
Transaction Processing on Modern Hardware. Synthesis Lectures on Data Management, Morgan & Claypool Publishers 2019
Fault-Tolerant Distributed Transactions on Blockchain. Synthesis Lectures on Data Management, Morgan & Claypool Publishers, to appear 2020
Books
![Page 11: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/11.jpg)
�11
BooksTransaction Processing on Modern Hardware.
Synthesis Lectures on Data Management, Morgan & Claypool Publishers 2019
Fault-Tolerant Distributed Transactions on Blockchain. Synthesis Lectures on Data Management, Morgan & Claypool Publishers, to appear 2020
PressAdvancements TV With Ted Danson - CNBC, Yahoo! Finance, Market Insider,
CoinDesk, Crypto Media, Davis Enterprise, Times Union, WBOC TV/Radio
![Page 12: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/12.jpg)
�12
Application Layer / Testbed ( YCSB, SYCSB, TPC-C Benchmarks)
Crypto Toolkit
Enable/Disable Secure Transactions
Block Creator(Distributed Ledger)
Concurrency Control Protocols(e.g., 2PL, QueCC, 2VCC, DORA, MVCC, Timestamp, H-Store, NoWait, Silo, Foedus, MOCC, TicToc, Cicada)
Consensus Protocols(e.g., GeoBFT, MBF,T PoE, PBFT, RBFT, Zyzzyva,
Steward, HotStuff, Bitcoin-NG, PoW)
L-Store: Lineage-based Storage Architecture
IndexesData
Transaction Manager
Parallel, Pipelined Runtime
Message/IO Queues
LoggingCommit Protocols:
(e.g., Q-Store, 2PC, 3PC, Calvin, EasyCommit)
ExpoDB Architecture
LevelDBSQLite
PlasticCoinJusteaze
![Page 13: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/13.jpg)
ResilientDB
�13
Coming Soon…
![Page 14: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/14.jpg)
Quantifiable Resiliency (Graduate Student Experiments)
�14
![Page 15: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/15.jpg)
�15
Aloha Lake, Desolation Wilderness 15 Miles Long
2,500 Feet Elevation Gain (8,700 Feet at Summit)
![Page 16: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/16.jpg)
�16
Tomales Point Trail, Point Reyes National Seashore 9.4 Miles Long
1,579 Feet Elevation Gain
![Page 17: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/17.jpg)
Non-Quantifiable Resiliency
�17
![Page 18: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/18.jpg)
�18
Proof-of-Execution: Reaching Consensus Through Fault-Tolerant Speculation [arXiv’19]
PoE scales beyond 32 replicas, in presence of failures, outperforms PBFT up to 40%
Fault-tolerant Proof-of-Execution Protocol
Out-of-Order message processing to reduce replica idleness Speculative Execution with revertible/divergent replicas & eager/irrevertible client commit
![Page 19: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/19.jpg)
�19
MultiBFT: Scaling Blockchain Databases Through Parallel Resilient Consensus Paradigm [arXiv’19]
Brief Announcement: Revisiting Consensus Protocols through Wait-free Parallelization. DISC 2019
A wait-free meta-protocol… Designate multiple replicas as Primaries!
Run multiple parallel consensuses on each replica independently
Fault-tolerant MultiBFT Protocol
![Page 20: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/20.jpg)
�20
MultiBFT: Scaling Blockchain Databases Through Parallel Resilient Consensus Paradigm
Brief Announcement: Revisiting Consensus Protocols through Wait-free Parallelization. DISC 2019
Throughput up to 300,000 txns/s (with failures)
Throughput up to 350,000 txns/s (without failures)
![Page 21: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/21.jpg)
�21
GeoBFT: Global Scale Resilient Blockchain Fabric [arXiv’19]
A meta-protocol, locally running any BFT in parallel and independently Global ordering provably requires only linear communication
Provably sufficient for primary to send a certificate to at most f+1 replicas, malicious primary is detectable and replaceable
Client Request Pre-prepare Prepare Commit Response
CLUSTER L
CLIENT PRIMARY
REPLICA
REPLICA
REPLICA
CLUSTER J
CLIENT PRIMARY
REPLICA
REPLICA
REPLICA
Certify Order
Fault-tolerant GeoBFT Protocol
![Page 22: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/22.jpg)
�22Brief Announcement: The Fault-Tolerant Cluster-sending Problem. DISC 2019
GeoBFT scales a permissioned blockchain up to 60 replicas globally.
GeoBFT: Global Scale Resilient Blockchain Fabric [arXiv’19]
ResilientDB easily scales across 6 countries in 4 continents due to GeoBFT protocol.
![Page 23: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/23.jpg)
�23
The Fault-Tolerant Cluster-Sending Problem
Brief Announcement: The Fault-Tolerant Cluster-sending Problem. DISC 2019
the problem of sending a message from one Byzantine cluster to another Byzantine cluster in a reliable manner,
establishing lower bounds on the complexity of this problem under crash failures and Byzantine failures
(linear in the size of clusters)
![Page 24: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/24.jpg)
�24
Permissioned Blockchain Through the Looking Glass: Architectural and Implementation Lessons Learned [arXiv’19]
Single-threaded Monolithic Design Out-of-ordering Consensus Communication
De-coupled Ordering and Execution Off-Chain Memory Management
Expensive Cryptographic Practices (DS vs. MAC)
Multi-Threaded Deep Pipeline
Client Requests
Prepare& Commit
Input
Network
Messagefrom
Clientsand
Replicas
NetworkBatch Creation
Worker
Checkpoint
Execute
Output Messageto
Replicasor
Clients
Can a well-crafted system based on a classical BFT protocol outperform a modern protocol?
ResilientDB
![Page 25: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/25.jpg)
Mount Tallac, Lake Tahoe 12.1 Miles Long
3,931 Feet Elevation Gain (9,738 Feet at Summit)
![Page 26: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/26.jpg)
THANK YOU https://expolab.org
2019.middleware-conference.orgCOMING TO UC DAVIS IN DECEMBER 2019
ACM MIDDLEWARE 2019
![Page 27: E DB: E T P IN B F T E - HPTS](https://reader030.vdocuments.pub/reader030/viewer/2022012508/618581ab46786d4f606e4ced/html5/thumbnails/27.jpg)
FOR COMPLETE REFERENCES