cистема распределенного, масштабируемого и...

Kirill Korotaev

VP of Advanced Research, Parallels

Система распределенного,

масштабируемого и высоконадежного

хранения данных для виртуальных

машин (и не только)HighLoad++, 22-23 октября 2012

Prehistory

Virtualization & needs

3

1. VM migration 2. HA on failures

SAN storage

4

Redundancy

High availability

Monitoring

Self healing

Reliability

Fiber channel

Performance

How to get best of 2

worlds?

5

local

SAN

100$ 100,000$

Focus on needed capabilities is a key to success

Idea of maximum simplification

From required capability to consistency

7

Consistency - “some well defined and understandable order” of operations.

Required for real file systems (NTFS, etx3/4,…) cause they use and rely on

these properties of real disks.

A AB B

Journal Metadata

• Immediate/Strict consistency

• Sequential consistency (all observers see the same order)

• Eventual consistency (no time guarantees, most object storages)

• Weak consistency (only synchronization primitives define order)

We want grow on demand also…

• Grow on demand means ability to allocate more then locally

available (>HDD or no free space)

• Requires split of data into pieces

• Probability(any of N computer failure) is increasing with N

• So redundancy/replication and auto recovery is a MUST

• But recovery can be done in parallel (~1TB in 10mins)

• Good news is that MTTF ~1/T2, where T is recovery time.

• So let’s split all objects to chunks (64Mb)

• Replicate chunks, not objects

• Spread chunks over the cluster to reduce recovery T

• Need to take into account topology of cluster

8

More ideas

9

Simplifications:

• No need for POSIX FS

• Optimize for big objects only

• Can assume infrequent/slow metadata updates

Major assumption:

• Shit happens: servers die, even DC can crash

Why consistency is not easy? Replication…

10

Simplest case: object is fully stored on a single node

Next step: object is replicated, i.e. multiple instances present

Object

v1

v1

v1

v2

v2

v1 Stale data can be read!

Server2

Server3

t

Server1

DC

cra

sh

c1v2

c2v2

Why consistency is not easy? Data striping…

11

Splitting data into chunks leads to similar issues as replication:

Server1

c1v1

File

c2v1

c1v1

c2v1

c1v2

c2v2

c1v1 c2v1

twrite

actual state is

c1v2+c2v2,

but all combinations

civj can be found

Note: c1v1 or/and c2v1 should never be read!

=

Server2

Server3 c1v1

Server4

Server5

Server6

c2v1

DC

cra

sh

write

So why consistency is not easy?

12

Data versioning is crucial and should be attributed

to data, plus heavily checked on operations!

Problems with versions:

1. Tracking versions requires transactions support and sync operations.

SLOW!

2. Can’t store versions near data only! Node can not decide alone whether

it’s uptodate or not.

Solution:

1. Let’s update version only when some server can’t complete write! And

leave it constant on successful data modifications.

2. To solve #2 let’s store versions on metadata servers (MDS) – authority

which tracks full cluster state. It should be reliable and HA.

Design

Overall design

14

MDS

CS

Clients

…

…

Meta Data Server (MDS)

• Metadata database

• Chunks and versions tracker

• HA

Ethernet

CS CS CS CS CS CS

Chunk Server (CS)

• Stores data (chunks)

• Chunk management

• READ/WRITE

Magic (MetaData) Server HA

• Single point of failure => need HA…

• Ouch, but we don’t want replicated DB, MySQL or Oracle…

It’s a nightmare to manage. Not scalable.

• Database adds noticeable latency per chunk access

15

We have to create our own replicated DB for MDS

Local

journal

Meta Data Server database

16

Local

journal

Full state in memory

Ideas from GoogleFS:

• ~128 byte per chunk

• 64Mb RAM describe ~32 Terabytes

chunks

Commit deltas

• Journal stores modification records

• It’s growing, so need compacting

method

• To compact in background, need

memory snapshotting

• Journal and state can be synced to

outdated nodes

Meta Data Server database compacting

17

Local

journal

Full state in memory

New

journal

Snap-

shot

New

journal

Snap-

shot

PAXOS algorithm

The Part-Time Parliament, Leslie Lamport:

• The Island of Paxos, parliament government

• Legislators are traders, travel a lot and often not present in

chamber

• They used non-reliable messengers for notifications and

communication

• Decree is adopted using ballots, need majority of votes

• Legislators are each having it’s own journal of records

• Consistency of decrees is required

• Add/removal of legislators is needed

• Progress is needed

18

Performance

Chunk server performance tricks

20

Write requests are split (64k) and chained:

Chained write

Completion

CS CS CS

t

64k

1 server, 256K write3 servers, 256K split/chained write

3 servers, dumb

Write 256Kb req

SSD caching on clients

• 3 parts: filter (RAM), cache, boot cache (1/8, 100MB / 2min)

• Sequential access detection relies on access time

• Write simply invalidates cached blocks

• Avoid interlaced reads: remote block, cached, remote,

cached

• Writes (caching) affect reads performance (user I/O)

21

{fileID, offset}

generation

blockid

accesstime…

8-way hash

and LRU

• SSD bursts performance > 10x times

• Caching on 2nd read access only to avoid excessive caching

• Detect sequential access and avoid caching

SSD journaling at CS

• Bursts write performance, async commit to rotational drives

• Allows to implement data check summing and scrubbing

• Reason: major difference from object storages –

performance and latency are very noticeable and important.

Latency is not hidden by WAN.

22

Summary

Final result

Storage system:

• With file system interface, scalable to Petabytes

• Suitable for running VMs and Containers, Object Storage,

shared hosting and other needs

• SAN like performance over ethernet networks:

• 13K IOPS on 14 machines cluster (4 SATA HDD each)

• 600K IOPS with SSD caching

• 1TB drive recovery takes 10 minutes!

24

Some experience to share

• Asynchronous non-blocking design

• QA: unit tests and stress testing are the must

• QA: how to emulate power off? SIGSTOP+SIGCONT/KILL.

• It hangs connections and avoids RESETs as if host

disappeared

• Drives/RAIDs/SSDs lie about FLUSHes.

• SSD write performance depends on data. Beware compression

inside.

• Checksum everything (4GB/sec) and validate HW memtest86

25


• All queues should be limited and controlled, like in TCP

congestion control is required (both for memory limit and latency

control, i.e. IO queue length)

• One client can fire Nx4K random requests and effectively it’s equivalent to

1MB requests. Congestion should be calculated correctly (taking into

accoung quality of queue).

• In addition to queues limit FDs/connections etc.

• Linux sync() / syncfs() is not usable

• Linux fdatasync() is 3.5-4x faster then fsync(), but should not be

used when file size changed.

• Replication should be done by pairs

26


• Cluster identity: unique ID and name

• Cluster resolving: DNS, manual and zeroconf

• Interesting OSDI’12 Microsoft Research storage: uniform network

topology, 2GB/sec per client, sorting world record 1.4TB in

~60sec.

27

Object storage

28

Image: ext4 filesystem

Object Object Object

Object storage

29


Object Object Object

Object storage

30


Object O O


Object O O


Object O O


Object O O


Object O O


Object O O

Thank You

31

Kirill Korotaev [email protected]

Try Parallels Cloud Server 6.0 beta2 at

http://www.parallels.com/product/pcs

mailto:[email protected]

Присоединяйся к лучшим! [email protected]

• Автоматизация оказания

облачных сервисов

• 10 млн. СМБ в 125 странах

• Виртуализация ПК и Mac

• 3 млн. ПК во всем мире

• 81% рынка в розничных сетях США

Parallels - лидер на международном рынке в своих сегментах

Работа в Parallels - от программирования в ядрах ОС до

создания web интерфейсов и мобильных приложений

• Интересные проекты

• Работа бок о бок с

легендами ИТ-

индустрии

• Процесс разработки

мирового класса

• Карьера, опционы

• 32

mailto:[email protected]

cистема распределенного, масштабируемого и...

Documents