in-memory database 전준민, 정주성, 이한민, 곽하녹 1. table of contents 1. introduction...

56
In-Memory Database 전전전 , 전전전 , 전전전 , 전전전 1

Upload: hollie-reynolds

Post on 12-Jan-2016

226 views

Category:

Documents


7 download

TRANSCRIPT

Page 1: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

1

In-Memory Database전준민 , 정주성 , 이한민 , 곽하녹

Page 2: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

2

Table of Contents

1. Introduction

2. Disk Resident DB vs In-Memory DB

3. Column Store

4. Durability

5. Data Overflow

6. Products of IMDB

7. Optimization Aspects on IMDB

Page 3: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

3

1. IntroductionWhat is In-Memory Database (IMDB) ?

Architecture

Rise of IMDB

Applications

Myths about IMDB

Page 4: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

4

What is In-Memory Database (IMDB)?• An in-memory database system is a database manage-

ment system that stores data entirely in main memory. 

Page 5: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

5

What is In-Memory Database (IMDB)?

Page 6: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

6

Architecture

• Fast data access• Algorithms optimized on

main memory• Efficient memory usage• Durability

Page 7: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

7

Rise of the IMDB

• Multicore Processors• Cheaper and Bigger Memories• Demands on Fast Databases

Page 8: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

8

Rise of the IMDB

Page 9: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

9

Rise of the IMDB

Page 10: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

10

Applications

• Low-latency, high volume systems

Page 11: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

11

Myths about IMDB

• Given the same amount of RAM, disk DBs can perform at the same speed as IMDBs (by using caching technology).

• If a RAM disk is created and a traditional disk DB is de-ployed on it, it delivers the same performance as an in-memory database.

• write on disk• buffer manager• indexes for disk• redundant data

Page 12: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

12

2. Disk Resident DB (DRDB) vs In-Memory DB (IMDB)DRDB vs IMDB : Overview

Indexes

Concurrency Control

Page 13: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

13

DRDB vs IMDB : Overview [1]

DRDB IMDB

File I/O Carries File I/O burden No file I/O burden

Storage UsageAssumes storage is abun-dant

Uses storage more effi-ciently

AlgorithmsAlgorithm optimized for disk

Algorithms optimized for memory

CPU Cycles More CPU cycles Less CPU cycles

Persistence Non-volatile Volatile

Lock Fine Locks Coarse Locks

Page 14: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

14

Indexes: B+-Tree in DRDB [2]

• The redundant data are kept in some index structures, to reduce I/O.

Page 15: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

15

Indexes: T-Tree in IMDB [3]

• The indexes in IMDBs are focused on reduced memory consumption and CPU cycles.

• In the early 90's, Lehman and Carey proposed the T-tree as an index structure for main memory database.

• The T-tree indexes are more efficient than B-trees in that they require less memory space and fewer CPU cy-cles.

Page 16: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

16

Indexes: T-Tree in IMDB

• The T-tree evolved from AVL Trees and B-Trees.

Page 17: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

17

Indexes: Hash indexes in IMDB

• Hash indexes are used for key-value based in-memory databases (cache servers) such as Redis and Mem-cached.

Page 18: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

18

Concurrency Control

• In DRDBs, locking granules are low level.• To reduce contention• To increase parallelism

• In IMDBs, locks are coarse-grained thanks to fast pro-cessing.

• Locking granules like a relation or an entire database• No need to look up hash table• Serial scheduling is enough in most cases

Page 19: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

19

3. Column StoreWhat is Column Store?

Benefits of Column Store

Delta Storage

Page 20: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

20

What is Column Store?

• Column Store• stores data tables as columns

of data rather than as rows of data

Page 21: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

21

Benefits of Column Store [4]

• Column stores are more suitable in IMDB than row stores • Better parallelism• Better compression• Faster data access

• Using parallel processing.• Especially for aggregations.

.

Page 22: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

22

Page 23: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

23

Benefits of Column Store: Parallel-ism [5] • Column storage can easily be separated into equal parts

which leads to effective parallel processing. • Highly parallelized scan operations are available which

are faster than indexed searches.• The row store cannot compete if processing is set-ori-ented and requires column operations, but most appli-cations are based on set-oriented processing and not di-rect tuple access.

Page 24: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

24

Benefits of Column Store: Parallel-ism • Highly parallelized scan operations using column stores

are faster than using just ordinary indexes.

Page 25: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

25

Benefits of Column Store: Compres-sion• Column store allows highly efficient compression be-

cause the columns contain only few distinct values.• Compression

Page 26: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

26

Delta Storage [6]

• Since writing on compressed column stores in real time is inefficient, delta storage techniques are used.

• Delta Storage• optimized for write operations

• Main Storage• compressed column store

Page 27: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

27

Delta Storage

• INSERT • insert a new record in the delta storage. The merge process will

move the record from delta to main.

• DELETE • A DELETE statement will select the record and mark it as invalid by

setting a flag (for main or delta). The merge process will delete the record from memory once there is no open transaction active for it anymore.

• UPDATE• An UPDATE statement will insert a new version of the record. The

merge process will move the latest version from delta to main. Old versions will be deleted  once there is no open transaction active for them anymore.

Page 28: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

28

Delta Storage: Simplified View of In-sert-Only Approach

Page 29: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

29

Delta Storage

• The merge process starts when the delta storage grows big enough.

Page 30: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

30

4. DurabilityLogging and Checkpointing

Command Logging

NVM Logging

Page 31: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

31

Durability• Durability is difficult to support in IMDBs• Many IMDBs have added durability via the following

mechan-isms• Checkpoints• Transaction logging

Page 32: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

32

Checkpointing

• Checkpoints in DRDB• Bring pages on disk up to date• Reduce the work of recovery

• Checkpoints in IMDB• Make a copy of the data on disks (snapshot)• Truncate the logs

Page 33: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

33

Logging and Checkpointing

Transaction

Log Buffer

Mem

ory

Physica

l Disk

Memory Ta-blespace log sync

Checkpoint Image File

REDO Log File

• Problem• Log I/O becomes bot-

tleneck

• How long do we need to keep the log?

• Until the next check-point

Page 34: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

34

• TPCC benchmarking on DRDBs (New Order transaction)

• Logging takes up a non-small portion

• Larger portion for IMDBs

Logging and Checkpointing [7]

Page 35: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

35

Command Logging [8]

• Light-weight, coarse-grained logging technique• Logical logging• Advantages

• Write substantially fewer bytes per transaction than physical logging • Reduce run time overhead

• Disadvantages• Slow recovery

• Failures that require recovery to ensure system availability are much less frequent

• 1.5X higher throughput than main-memory optimized imple-mentation of physical logging

Page 36: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

36

Command Logging

Page 37: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

37

• NVM (Non-Volatile Memory)• low read/write latency like DRAM• persistent write like SSD

NVM Logging [9]

DRAM NAND Flash NVM

Byte-Address-able

Yes No Yes

Capacity 1X 4X 2-4X

Latency 1X 400X 3-5X

Page 38: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

38

• DBMS relies on both DRAM and NVM

NVM+DRAM Architecture

Page 39: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

39

5. Data overflowAnti-caching

Project Siberia

Page 40: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

40

Data overflow

• Datasets may not fit in DRAM• IMDB Solutions

• Anti-caching• Project Siberia

Page 41: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

41

Anti-caching [10]

• Used in H-Store• Cold data is moved to disk in a safe manner• Bloom filter used for tracking data• Manage cold data by maintaining a LRU chain

Page 42: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

42

Anti-caching

Page 43: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

43

Anti-caching

• Fine-grained eviction• eviction is performed at tuple-level, not page-level

• Non-blocking fetches• a transaction that accesses evicted data is simply aborted and

then restarted at a later point

Page 44: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

44

Project Siberia [11]

• Used in Hekaton• Automatically and transparently maintain cold data on

cheaper secondary storage• Allow more data to fit in memory• Log-based management of cold data

Page 45: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

45

6. Products of IMDBH-Store / VoltDB

Hekaton

SAP HANA

In-memory NoSQL Databases

Page 46: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

46

Products of IMDB

Page 47: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

47

H-Store / VoltDB

• Distributed row-based in-memory relational database • Targeted for high-performance OLTP processing• Light-weight logging strategy• Anti-caching

Page 48: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

48

Hekaton

• Memory-optimized OLTP engine• Fully integrated into Microsoft SQL server• Multi-version concurrency control • Project Siberia

Page 49: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

49

SAP HANA

• A distributed in-memory database featured for the inte-gration of OLTP and OLAP

• Provides rich data analytics functionality by offering multiple query language interfaces (e.g., standard SQL, SQLScript, MDX, WIPE, FOX and R)

Page 50: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

50

SAP HANA

• Three-level column-oriented unified table structure

Page 51: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

51

In-memory NoSQL Databases

• RAMCloud• Distributed in-memory key-value store, featured for low la-

tency, high availability and high memory utilization

• Bitsy• Embeddable in-memory graph database that implements the

Blueprints API, with ACID guarantees on transactions based on the optimistic concurrency mode

Page 52: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

52

Comparison of IMDB [12]

Sys-tems Data Model Work-

loads Indexes Fault Toler-ance

Memory Overflow

Relational Databases

H-Store relation(row) OLTPhashing, b+-tree, binary tree

command log-ging, checkpoint, replica

anti-caching

Hekaton relation(row) OLTPlatch-free hashing, Bw-tree

logging, check-point, replica

Project Siberia

SAP HANA

relation, graph, text OLTP, OLAP timeline index

logging, check-point, standby server

table/parti-tion-level swapping

NoSQL Databases

RAM-Cloud key-value object op-

erations hashing logging, replica N/A

Graph Databases Bitsy N/A OLTP

optimistic con-currency con-trol

logging, backup N/A

Page 53: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

53

7. Optimization Aspects on IMDB

Page 54: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

54

Optimization Aspects on IMDB [12]

Aspects Concerns Related Work

Index cache consciousness, time/space efficiency T-Tree, CSS-Trees, CSB+-Trees, BD-Tree

Data Layout cache consciousness, space efficiency

columnar layout, HANA Hybrid Store, log structure

Concurrency Con-trol overhead, correctness virtual snapshot, transaction memory,

MVCC

Query Processing code locality, time efficiency stored procedure, JIT compilation, sort-ing

Fault Tolerance durability, correlated failures, availability

group commit and log coalescing, NVM, command logging, remote logging

Data Overflow locality, paging, hot/cold classification

anti-caching, Hekaton Siberia, data compression, virtual memory manage-ment, pointer swizzling

Page 55: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

55

References[1] Garcia-Molina, Hector, and Kenneth Salem. "Main memory database systems: An overview." Knowledge and Data Engineering, IEEE Transactions on 4.6 (1992): 509-516.

[2] Comer, Douglas. "Ubiquitous B-tree." ACM Computing Surveys (CSUR) 11.2 (1979): 121-137.

[3] Lehman, Tobin J., and Michael J. Carey. "A study of index structures for main memory database management systems." Conference on Very Large Data Bases. Vol. 294. 1986.

[4] Abadi, Daniel J., Samuel R. Madden, and Nabil Hachem. "Column-stores vs. row-stores: how different are they really?." Proceedings of the 2008 ACM SIGMOD international conference on Management of data. ACM, 2008

[5] Plattner, Hasso. "A common database approach for OLTP and OLAP using an in-memory column database." Proceedings of the 2009 ACM SIGMOD International Conference on Management of data. ACM, 2009.

[6] Färber, Franz, et al. "The SAP HANA Database--An Architecture Overview."IEEE Data Eng. Bull. 35.1 (2012): 28-33.

Page 56: In-Memory Database 전준민, 정주성, 이한민, 곽하녹 1. Table of Contents 1. Introduction 2. Disk Resident DB vs In-Memory DB 3. Column Store 4. Durability 5. Data Overflow

56

References[7] Harizopoulos, Stavros, et al. "OLTP through the looking glass, and what we found there." Proceedings of the 2008 ACM SIGMOD international conference on Management of data. ACM, 2008.

[8] Malviya, Nirmesh, et al. "Rethinking main memory oltp recovery." Data Engi-neering (ICDE), 2014 IEEE 30th International Conference on. IEEE, 2014.

[9] DeBrabant, Justin, et al. "A Prolegomenon on OLTP Database Systems for Non-Volatile Memory." Proceedings of the VLDB Endowment 7.14 (2014).

[10] DeBrabant, Justin, et al. "Anti-caching: A new approach to database man-agement system architecture." Proceedings of the VLDB Endowment 6.14 (2013): 1942-1953.

[11] Eldawy, Ahmed, Justin Levandoski, and Paul Larson. "Trekking through siberia: Managing cold data in a memory-optimized database." Proceedings of the VLDB Endowment 7.11 (2014).

[12] Zhang, Hao, et al. "In-memory big data management and processing: A sur-vey." (2015).