A Survey on Efficient Utilization of
Emerging Persistent Memory
Taura Lab.
M1 Makoto Shimazu
2014/11/07
1
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
2
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
3
Development of Computer
“Fast” is the top goal of computation
How to achieve:
CPU: faster clock/more cores
Memory: faster interconnection/more capacity
Storage: broader bandwidth/shorter RW latency
4
CPU Improvements
Many Core Xeon Phi: 60C/240T, x86 Compatible, 1.0GHz, 1TFlops
PEZY-SC: 1024C, 733MHz, 1.5TFlops
TILE-Gx: 72C, 1.2GHz
left) http://www.intel.co.jp/content/www/jp/ja/processors/xeon/xeon-phi-detail.html
center) http://www.tilera.com/products/processors/TILE-Gx_Family
right) http://www.pezy.co.jp/news/PEZY_PR_20140905.pdf
5
Memory Improvements
Bandwidth
HBM (High Bandwidth Memory) up to 256GB/s (DDR3: 12.8GB/s, GDDR5 88GB/s)
Capacity
2.5D/3D Stacking 128GB/card in 2015
fig) http://www.eetimes.com/document.asp?doc_id=1279432
TSV (Through Silicon Via)
6
Storage Improvements
NVM (Non Volatile Memory)
SSD (Flash memory) is one of NVM
OpenNVM: http://opennvm.github.io Flash-aware Linux swap as a transparent extension of DRAM
fig) http://www.hlnand.com/site/ID/applications
Non-Volatile Storage
is still slow! Volatile
Durable
7
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
8
Next Generation NVM
PM (Persistent Memory)
There are many methods: Phase Change Memory
Resistance RAM
Spin Transfer Torque RAM
Memristor
Volatile
Durable!!!
Persistent
Memory
9
Various Types of PM
10
NAND Flash PCM ReRAM STT-RAM
Property Capacitor Phase Resister Magnet
Capacity/Chip ~128Gbit 128Mbit 2Mbit 64Mbit
Latency 10μs 100ns 10ns 7ns
Rewrite cycles 104~5 106 1011~12 1015
Manufacturer Toshiba
SanDisk
Samsung
Micron Panasonic Everspin
Toshiba
Same as DRAM!
Much reliable than Flash!no need to wear leveling
Byte Addressability
Disk: read/write API
PM: load/store instructions
Difference between PM and Disk
11
fig of HDD/SSD) http://storage-system.fujitsu.com/jp/lib-f/tech/beginner/ssd/
load/store to DRAM read/write to
SSD/HDD
load/store to PM
Non-volatile Data
Cache
Per Sector (4k or 512 bytes)
Per Byte (1 byte)
Design Spaces on PM
Durability
V-NV gap is coming up between cache and PM
High Random Access Performance
read/write API are not suitable for PM
Other data structures based on disk are the same
12
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
13
PM-Aware File System
Paper J. Condit, et al., Better I/O Through Byte-Addressable,
Persistent Memory, SOSP’09
Short Summary
Revisit shadow-paging Short-circuit shadow paging instead of WAL1
Introduce two important hardware modifications: Atomic 8-byte writes
Epoch barrier
141) WAL: Write Ahead Logging
PM-Aware File System
Data Structure
15
PM-Aware File System
WAL (Write Ahead Logging)
16
Hello World!
RINKO
NXXXX
hello.txt
1: WRITE “RINKO”
2: WRITE “NOW!!!”
Log
SnapshotLogging is needed
per operation
Unsuitable for
Byte Addressability
CRASH!
Hello World!
RINKO
NOW!!!
PM-Aware File System
Shadow Paging
Safe and consistent method to modify data
Three steps: Copy, Modify, Refer
1: Copy
2: Modify
3: Refer
Recursive Copy!!!
17
PM-Aware File System
Short Circuit Shadow Paging Introduce atomic 8-byte writes
≦ 8 Bytes
File size
Shadow Paging
Atomic
8-byte write
18
PM-Aware File System
Cache EvictionAmbiguous timing of cache eviction causes inconsistency
1: Append
2: Atomic Write
Cache3: Append
Write down to PM19
PM-Aware File System
Epoch Barrier
Problem
Inconsistency caused by timing of cache eviction
Cache Modifications
Introduce Epoch Barrier Software issues the barrier explicitly
Hardware features (for 8 in-flight epochs) 1bit persistent bit+3 bits Epoch Pointer on each cache line
Additional tables to keep the information of epochs
20
Epoch Tables
PM-Aware File System
Cache Eviction w/ EB
1: Append
Cache3: Append
Write down to PM
Epoch 1
Epoch 1
Epoch 2Epoch 3
Epoch 2
Epoch 3
ebarrierebarrier
2: Atomic Write
21
PM-Aware File System
Evaluation
22
Micro benchmarks
BPFS vs. NTFS (RAM/Disk)
Epoch Barrier
SESC simulation
1.5x – 2.9x speed upthan write-through caching
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
23
Heap on PM
Paper J. Coburn, et al., NV-Heaps: Making Persistent Objects Fast
and Safe with Next-Generation, Non-Volatile Memories, ASPLOS’11
Short Summary
Use PM as object-based storage based on the two hardware modifications
Propose three issues coming from PM More significant memory leaks
Restriction of NV-to-V pointer
Transactions on heap area
24
Heap on PM
Strength of Persistent Heap
Motivation
Serialization needs an additional calculation...
However...
Modification of small part of data is heavy Disk must be accessed by each sector (4k or 512 bytes)
Slow seek speed (Disk) / Slow wear leveling speed (SSD)
25
The Area of Byte Addressable
Storage: Persistent Memory!
Heap on PM
Keys of Design
Memory Leaks
Reference count with logging
Pointers
NV-to-V must not be persistent
Transactions
Idea of STM (Software Transactional Memory)
26
Heap on PM
Evaluation
Environments
Linux RAM Diskwith extra delay
Comparison withBerkleyDB andMemcached
Fork from BDB
Results
NV-Heap has the performanceas good as native Memcached
27
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
28
Distributed Logging with PM
Paper T. Wang and R. Johnson, Scalable Logging through Emerging
Non-Volatile Memory, VLDB’14
Short Summary
Distributed logging enhancing PM advantages
Without atomic 8-byte writes and epoch barrier
Two software architecture instead: Global Sequence Number
Passive Group Commit
29
Distributed Logging with PM
Centralized Logging
30Log File
Centralized logging does not suit
massively parallel paradigm
Distributed Logging with PM
Distributed Logging
31
CONTENTION FREE!!!!!
Really?
Distributed Logging with PM
Distributed Logging
32
Remember the importance of
Byte Addressability
How is the order of logs determined?
Distributed Logging with PM
LSN (Log Sequence Number)
33
Hello World!
RINKO
NXXXX
hello.txt
1: WRITE “RINKO”
2: WRITE “NOW!!!”
Log
Hello World!
RINKO
NOW!!!
Snapshot
1: WRITE “RINKO”
2: WRITE “NOW!!!”
Share the counter??
Distributed Logging with PM
Global Sequence Number
GSN must be greater than the GSN of the previous write operation on the same page
34
12345
Tx1 Tx2 Tx3
126
Write to P1
Write to P2
P1 P2
1
2
3
4
1
2
5
6
t = 0
Distributed Logging with PM
Passive Group Commit
Ensure old logs are evicted from cache
Leverage existing hardware support
3 Strategies of Caching
Write-Through
Write-Back
Write-Combining
35
Distributed Logging with PM
Write-Combining
Write back a small block at once
Some adjacent bytes are combined into one
Intel Core series processor has 8 WC buffers/core
36
dgsn: 6dgsn: 9
WC Buffer is not durable
Logging must keep in step on each processor
dgsn: 9dgsn: 10
Distributed Logging with PM
How to Keep Consistency
37
Tx 1 Tx 2 Tx 3
Tx4: dgsn 1
Tx6: dgsn 4
Passive Group Commit Deamon
1: Commit
dgsn: 6 dgsn: 8
Tx2: dgsn 1
Tx1: dgsn 4
Tx1: dgsn 10
2: Commit
dgsn: 12
FULL!
3: Eviction
dgsn: 13
Tx2: dgsn 1
Tx1: dgsn 4
Tx1: dgsn 10
Tx1: dgsn 10
Tx2: dgsn 12
Tx3: dgsn 13
latest: 9 latest: 6 latest: 13latest: 10 latest: 12
Distributed Logging with PM
Time Breakdowns
38
Outline
Background
Trend of Hardware Development
Emerging Persistent Memory
Three usage of Persistent Memory
File System
Heap area
Database
Summary
39
Summary
Three different approaches to PM
File system
Heap area
Database logging
Important features of PM
Volatility of cache
High random access performance
Many design space and revisable viewpoint
40