Upload File

ceph & rocksdb · rocksdb group commit . thread scalibility 0 10,000 20,000 30,000 40,000...

  • Home
  • Documents
  • Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB
22
Ceph & RocksDB 변일수(Cloud Storage팀)

Upload: others

Post on 21-May-2020

17 views

Category:

Documents


1 download

Report

TRANSCRIPT

Page 1: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Ceph & RocksDB

변일수(Cloud Storage팀)

Page 2: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Ceph Basics

Page 3: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Placement Group

myobject mypool

PG#1 PG#2 PG#3

hash(myobject) = 4 % 3(# of PGs) = 1 ← Target PG

Page 4: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

CRUSH

mypool

PG#1 PG#2 PG#3

OSD#1 OSD#3 OSD#12

Page 5: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Recovery

mypool

PG#1 PG#2 PG#3

OSD#1 OSD#3 OSD#12

Page 6: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

OSD

ObjectStore

Replication, ???, …

OSD

Peering, Heartbeat,

…

FileStore

BlueStore

https://www.scan.co.uk/products/4tb-toshiba-mg04aca400e-enterprise-hard-drive-35-hdd-sata-iii-6gb-s-7200rpm-128mb-cache-oem

Page 7: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

ObjectStore

https://ceph.com/community/new-luminous-bluestore/

Page 8: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

OSD Transaction

Page 9: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Consistency is enforced here!

CRUSH

Page 10: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

SSD

op_w

qkv_committing

pipe

out

_q

Request

Ack

write

flusth

ROCKSDBsync transaction

Shard

finish

er q

ueue

• To maintain ordering within each PG, ordering within each shard should be guaranteed.

BlueStore Transaction

Page 11: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

• Metadata is stored in RocksDB.• After storing metadata atomically, data is available to users.

Request

SSTFile

LogfileMemtable

Transaction Log

Flush

JoinBatchGroup (leader)

PreprocessWrite

WriteToWAL

MarkLogsSynced

ExitAsBatchGroupLeader

Write to memtable

#1 Thread #2 Thread

JoinBatchGroupAwaitState

#3 Thread

JoinBatchGroupAwaitState

PreprocessWrite

WriteToWAL

leader

LaunchParallelFollower

MarkLogsSyncedfollower

ExitAsBatchGroupFollower

CompleteParallelWorker

Concurrent write to memtable

Group commit

RocksDB Group Commit

Page 12: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Thread Scalibility

0

10,000

20,000

30,000

40,000

50,000

60,000

1 shard 10 shard

IOPS

Shard Scalability

WAL disableWAL

PUT

PUT

PUT

PUT

WAL

RocksDB

Page 13: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

RadosGW

Page 14: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

• RadosGW is an application of RADOS

RadosGW

OSD Mon Mgr

Rados

RadosGW CephFS

Page 15: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

• All atomic operations depen on RocksDB

RadosGW Transaction

Prepare Index

Write Data

Complete Index

Put Object

RADOS

Index Object Data Object

k vk vk v

k vk vk v

RocksDB

Page 16: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

SSD

op_w

qkv_committing

pipe

out

_q

Request

Ack

write

flusth

ROCKSDBsync transaction

Shard

finish

er q

ueue

bstore_shard_finishers= true

• To maintain ordering within each PG, ordering within each shard should be guaranteed.

BlueStore Transaction

Page 17: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Performance Issue

Page 18: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Tail Latency

Page 19: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

Performance Metrics

Page 20: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

• "SILK: Preventing Latency Spikes in Log-Structured Merge Key-Value Stores" (ATC'19)

RocksDB Compaction Overhead

Page 21: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

• Ceph highly depends on RocksDB• Strong consistency of Ceph is implemented using RocksDB transactions

• The performance of ceph also depends on RocksDB• Especially for Small IO

• But RocksDB has some performance Issues• Flushing WAL• Compaction

• [email protected]

Conclusions

Page 22: Ceph & RocksDB · RocksDB Group Commit . Thread Scalibility 0 10,000 20,000 30,000 40,000 50,000 60,000 1 shard 10 shard S Shard Scalability WALdisableWAL PUT PUT PUT PUT WAL RocksDB

THANK YOU


KANAL 5 - Kostrena · 2019. 7. 5. · put ŽUKNICA put put put put CRIKVENICA Jadranska magistrala RIJEKA cesta VRH MARTINŠĆICE put MARGITIĆEVO put SV.LUCIJA SV. BARBARA put DUKINO

In de put, uit de put', zelf depressiviteit overwinnen

Disclaimer - Seoul National Universitys-space.snu.ac.kr/bitstream/10371/161528/1/000000157310.pdf · 활용하여 RocksDB 시스템의 전체 성능은 유지하면서 DRAM 사용량을

ISUSOV KRIŽNI PUT I KRIŽNI PUT NEROĐENIH

4.A. - grad-vinkovci.hr PPUG... · Vojni kanal Put =DJUHE +ä '* 7RYDUQLN Put Duga ulica Kneza Branimira Put %RULQDþND %RULQDþND Put Put Blato Ulica dudova Put %XQMHYDþND XOLFD

N KUČINA ŠODIĆI - KostrenaŽLAČ UMEJČIĆ CRIKVENICA Jadranska magistrala RIJEKA CRIKVENICA Jadranska magistrala RIJEKA cesta ŠODIĆI ŽURKOVO put SV. LUCIJA put put put put put

语音交互中的 自然语言处理技术cips-upload.bj.bcebos.com/ccks2017/upload/CCKS交流-语音交互中 … · lucene l1 scorer l2 scorer qc srv 2 shard 0 shard 0 shard 0

prema TS KOTLOVNICA · zemljani put zemljani put zemljani put zemljani put zemljani put zemljani put zemljani put-cesta-SPLIT zemljani put V V V V V V V V V V V V V V V V V V V V

Быстрый и простой способ шардирования MySQL с помощью CUBRID SHARD - 2013 RIT++, Москва

JUGOISTOČNA EVROPA: PUT U EU ILI PUT ZA NIGDEngofractal.org/wp-content/uploads/2018/10/JI-Evropa-Put...• “Put u EU” – put u održiv, energetski efikasan sistem, kojim zemlje

AUTOKAMP - Kostrena · 2019. 7. 5. · put ŽUKNICA put put put put CRIKVENICA Jadranska magistrala RIJEKA cesta VRH MARTINŠĆICE put MARGITIĆEVO put SV.LUCIJA SV. BARBARA put DUKINO

Bela Hamvaš - Put predaka i put bogova

©stéphane vandenplas TRIOPOLYCORDES€¦ · Shard de Elliot CaRTeR guitare solo-Luimen de Elliot CaRTeR Trompette, trombone, vibraphone, mandoline, guitare et harpe Shard, oeuvre

1.A. Korištenje i namjena površina · 2018-08-07 · Crnopalj put put Preko put put put put put put put Figurica 129,*5$' FHVWD 60,/ý,û NOVIGRAD - cesta - POSEDARJE %UDYDULü

Presentación para slider shard

Isusov put je put kršćana

MLIJEČNI PUT

Pag - cesta - Košljun SLUŽBENA OBALNA CRTA ......put put put put put put put put put Pag - cesta - Košljun more more more more more more more 82.0 82.0 81.5 105.0 102.5 100.0 97.5

Osho Moj Put Put Belih Oblaka

jezgra naselja 1000 m - Grad Zadar GP Petrcane[2].pdf · TS TS TS TS TS TS Sv. Bartol Sv. Ivana i Pavla Vir - cesta - Zadar put put Vir - cesta - Zadar put put cesta put put put cesta

Aces Shard

Shard-Query, an MPP database for the cloud using the LAMP stack

4.2. Način gradnje · 2018-08-07 · Crnopalj put put Preko put put put put put put put Figurica 129,*5$' FHVWD 60,/ý,û NOVIGRAD - cesta - POSEDARJE %UDYDULü put Ladina 5.0m 5.0m

JAVNA AUKCIJA - PRVI PUT- DRUGI PUT I DIREKTNA PRODAJA

Flink技术栈及其适用场景3. rocksDB: kv store rocksDB issue-1988 performance problem . windows

GRADA ZADRA - IZMJENE I DOPUNEkrsevan.grad-zadar.hr/ArhCPlanovi/PPUG/III IiD PPUGZ (GGZ 2-16)/PDF/4.3... · PUNTA SKALA put put put put put put put put put put put put put put put

2.3. Telekomunikacijska i energetska mreža · 2018-08-07 · Crnopalj put put Preko put put put put put put put Figurica 129,*5$' FHVWD 60,/ý,û NOVIGRAD - cesta - POSEDARJE %UDYDULü

Performance analysis of Ceph Block Device Service (RBD)bos.itdks.com/762149bc9b2e4e1f8bdc68ee73e4f092.pdfdelay rocksdb compaction 151 210 0 50 100 150 200 250. IOPS(K) Delay RocksDB

Osho - Moj Put, Put Belih Oblaka

1.7. GRAFIČKA DOKUMENTACIJA...1.7. GRAFIČKA DOKUMENTACIJA 3262 3259 3260 put 4949 put 4950 put 4953 put 4954 put 4947 reka Krivaja 4716 3264 3362 3363 1883 1886 1887 put 4960 SEPJ

Bela Hamvas - Put Predaka i Put Bogova

Shard of Lights (for Harp Solo)

Millmyen Put Mirnogo Voina Put Mirolyubivogo Voina

2.2. Vodoopskrbna mreža i odvodnja · 2018-08-07 · Crnopalj put put Preko put put put put put put put Figurica 129,*5$' FHVWD 60,/ý,û NOVIGRAD - cesta - POSEDARJE %UDYDULü put

Put KOVA^KA DOLINA - Žabljakzabljak.me/docs/1398234222-02 - Izvod iz PUP.pdf · N Put @abqak - Mojkovac Put Put 3XW ÿÿDEOMDN 1DGJRUD Put K q e { t i n a Put VC S Rz Put Kqe{tina