![Page 1: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/1.jpg)
RECIPE : Converting Concurrent DRAM Indexes to Persistent-Memory Indexes
Se Kwon Lee, Jayashree Mohan, Sanidhya Kashyap, Taesoo Kim, Vijay Chidambaram
1
![Page 2: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/2.jpg)
Persistent Memory (PM)
2
Intel Optane DC Persistent Memory
• New storage class memory technology
• Performance similar to DRAM
• Non-volatile & high-capacity
• Up-to 6TB on a single machine
![Page 3: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/3.jpg)
3
• PM has high capacity and low latency• 6TB on a single machine à 100 billion 64-byte key-value pairs
• Indexing data on PM is crucial for efficient data access
Indexing on PM
![Page 4: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/4.jpg)
PM Indexes need to achieve three goals simultaneously
PM Indexes
4
PM
Crash ConsistencyConcurrencyCache
Efficiency
![Page 5: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/5.jpg)
PM
• Cache Efficiency• Persistent memory is attached to the memory bus• 3x higher latency than DRAM à More cache-sensitive
PM Indexes
5
Crash ConsistencyConcurrencyCache
Efficiency
![Page 6: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/6.jpg)
PM
• Concurrency• High concurrency is necessary for scalability on any modern
multicore platform
PM Indexes
6
Crash ConsistencyConcurrencyCache
Efficiency
![Page 7: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/7.jpg)
PM
• Crash Consistency• CPU cache is still volatile• Arbitrarily-evicted cache lines à Persistence reordering
PM Indexes
7
Crash ConsistencyConcurrencyCache
Efficiency
![Page 8: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/8.jpg)
logcommit
Core
Cache
• Crash Consistency• CPU cache is still volatile• Arbitrarily-evicted cache lines à Persistence reordering
PM Indexes
8
Volatile
Persistent
Program orderwrite (log);write (commit);
① ②
commitlog
![Page 9: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/9.jpg)
commit
Core
Cache
• Crash Consistency• CPU cache is still volatile• Arbitrarily-evicted cache lines à Persistence reordering
PM Indexes
9
Volatile
Persistent
Persistence reorderingwrite (log);write (commit);log Reordered
commit
![Page 10: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/10.jpg)
commit
commit
Core
Cache
• Crash Consistency• CPU cache is still volatile• Arbitrarily-evicted cache lines à Persistence reordering
PM Indexes
10
Volatile
Persistent
Persistence reorderingwrite (log);write (commit);log
commit InconsistencyCrash
![Page 11: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/11.jpg)
Core
Cache
• Crash Consistency• CPU cache is still volatile• Arbitrarily-evicted cache lines à Persistence reordering
• Flush: persist writes to PM• Fence: ensure one write prior another to be persisted first
PM Indexes
11
Volatile
Persistent commitlog
Consistent persistence orderingwrite (log)flush (log)fence ()write (commit)flush (commit)fence ()
![Page 12: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/12.jpg)
Challenge in building PM indexes
12
Correct
Concurrency
CorrectCrash
Consistency
ConsistentData
Conflict Crash
Correctness condition: return previouslyinserted data without data loss or corruption
![Page 13: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/13.jpg)
Bug
Bug
Challenge in building PM indexes
13
Concurrency and crash consistency interact witheach other, a bug in either can lead to data loss
Bug
Concurrency
BugCrash
Consistency
DataLoss
Conflict Crash
![Page 14: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/14.jpg)
• We found bugs in FAST&FAIR [FAST’18] and CCEH [FAST’19]• FAST&FAIR: Concurrent PM-based B+Tree
• One bug in concurrency mechanism• Two bugs in recovery mechanism
• CCEH: Concurrent PM-based dynamic hash table• One bug in concurrency mechanism• One bug in recovery mechanism
Bug in Concurrent PM Index
14
![Page 15: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/15.jpg)
15
How can we reduce the effort involved in building concurrent, crash-consistent PM
indexes?
Answer: We can convert concurrent DRAM indexes to PM indexes with low effort
Insight: Isolation and Crash Consistency are similar
![Page 16: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/16.jpg)
16
How can we reduce the effort involved in building concurrent, crash-consistent PM
indexes?
Approach: Convert concurrent DRAM indexes to PM indexes with low effort
Insight: Isolation and Crash Consistency are similar
![Page 17: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/17.jpg)
• Already designed for cache efficiency and concurrency
DRAM Index
17
Concurrency
CacheEfficiency
T-Tree
CSS-Tree CSB+
Tree
BD-Tree
FAST
Bw-Tree
ART
HOT
Masstree
1986
CLHT
2010 2019
Cac
he E
ffici
ency
Con
curre
ncy
Time
![Page 18: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/18.jpg)
18
Crash
DRAM Index
Volatile
Concurrency
Cache Efficiency
DRAM Index
Concurrency
Cache Efficiency
Crash Vulnerable
DRAM Index
![Page 19: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/19.jpg)
Challenge in Conversion
19
• Require minimal changes to DRAM index• Without modifying the original design principles of DRAM index
DRAM Index
Volatile
Concurrency
Cache Efficiency
PM Index
Concurrency
Cache Efficiency
Crash Consistency
Conversion
![Page 20: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/20.jpg)
201. Steven Pelley et al., Memory Persistency, ISCA’14
• Similar semantics between isolation and consistency1
• Isolation• Return consistent data while multiple active threads are running
• Crash consistency• Return consistent data even after a crash happens at any point
Insight for Conversion
![Page 21: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/21.jpg)
• Similar semantics between isolation and consistency1
• Isolation• Return consistent data while multiple active threads are running
• Crash consistency• Return consistent data even after a crash happens at any point
211. Steven Pelley et al., Memory Persistency, ISCA’14
Insight for Conversion
Approach: reuse mechanisms for isolation in DRAM indexes to obtain crash consistency
![Page 22: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/22.jpg)
RECIPE
• Principled approach to convert DRAM indexes into PM indexes
• Case study of changing five popular DRAM indexes
• Conversion involves different data structures such as Hash Tables, B+ Trees, and Radix Trees
• Conversion required modifying <= 200 LOC
• Up-to 5.2x better performance in multi-threaded evaluation
22
RECIPE
https://github.com/utsaslab/RECIPE
![Page 23: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/23.jpg)
• Overall Intuition• Conversion Conditions• Conversion Example: Masstree• Assumptions & Limitations• Evaluation
Outline
23
![Page 24: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/24.jpg)
• Overall Intuition• Conversion Conditions• Conversion Example: Masstree• Assumptions & Limitations• Evaluation
Outline
24
![Page 25: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/25.jpg)
Overall Intuition for Conversion
25
• Blocking algorithms• Use explicit locks to prevent the conflicts of threads to shared
data• Non-blocking algorithms
• Use well-defined invariants and ordering constraints without locks
• Employed by most high-performance DRAM indexes
![Page 26: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/26.jpg)
Overall Intuition for Conversion
26
• Non-blocking algorithms• Readers Detect and Tolerate inconsistencies
• E.g., Ignore duplicated keys
InconsistencyReader
Tolerate
Detect
![Page 27: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/27.jpg)
Overall Intuition for Conversion
27
• Non-blocking algorithms• Readers Detect and Tolerate inconsistencies
• E.g., Ignore duplicated keys• Writers also Detect, but Fix inconsistencies
• E.g., Eliminate duplicated keys
WriterDetect Fix
ConsistentInconsistency
![Page 28: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/28.jpg)
Overall Intuition for Conversion
28
• Non-blocking algorithms• Readers Detect and Tolerate inconsistencies• Writers also Detect, but Fix inconsistencies• Helping mechanism1 ≈ Crash Recovery2
• Such indexes are *inherently* crash consistent
WriterDetect Fix
ConsistentInconsistency
1. Keren Censor-Hillel et al., Help!, PODC’152. Ryan Berryhill et al., Robust shared objects for non-volatile main memory, OPODIS’15
![Page 29: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/29.jpg)
• Not all DRAM indexes can be converted with low effort
• Exploit inherent crash recovery in the index
• Provide specific conditions that must hold for a DRAM index to be converted
• Provide a matching conversion actions for each condition
29
![Page 30: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/30.jpg)
• Overall Intuition• Conversion Conditions• Conversion Example: Masstree• Assumptions & Limitations• Evaluation
Outline
30
![Page 31: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/31.jpg)
• Condition 1: Updates via Single Atomic Store• Condition 2: Writers fix inconsistencies• Condition 3: Writers don’t fix inconsistencies• Conditions are not exhaustive!
Three Conversion Conditions
31
![Page 32: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/32.jpg)
• Condition 1: Updates via Single Atomic Store• Condition 2: Writers fix inconsistencies• Condition 3: Writers don’t fix inconsistencies
Three Conversion Conditions
32
![Page 33: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/33.jpg)
Condition 1: Updates via Single Atomic Store
33
• Non-blocking readers, (Non-blocking or Blocking) writers• Updates become visible to other threads via single
atomic commit store
Atomic Store
CrashStep 1 Step 2 Step N… Step 1 Step 2 Step N…
Invisible Visible
![Page 34: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/34.jpg)
Condition 1: Updates via Single Atomic Store
34
• Updates become visible to other threads via single atomic commit store
• Conversion: Add flushes after each store and bind final atomic store using fences
Atomic Store
Step 1 Step 2 Step N… Step 1 Step 2 Step N…
Flushes
Flush
FenceFence
![Page 35: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/35.jpg)
• Condition 1: Updates via Single Atomic Store• Condition 2: Writers fix inconsistencies• Condition 3: Writers don’t fix inconsistencies
Three Conversion Conditions
35
![Page 36: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/36.jpg)
Condition 2: Writers fix inconsistencies
36
• Non-blocking readers and writers (don’t hold locks)• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )
Step 1 Step 2 Step 3
Update
Writer 1
A sequence of ordered deterministic steps
CommitStep
![Page 37: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/37.jpg)
Step 3
Condition 2: Writers fix inconsistencies
37
• Non-blocking readers and writers (don’t hold locks)• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )
Step 1 Step 2Writer 1Reader
Detect
CommitStep
Tolerate
![Page 38: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/38.jpg)
Condition 2: Writers fix inconsistencies
38
• Non-blocking readers and writers (don’t hold locks)• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )
Step 1 Step 2Writer 1Writer 2
Step 3
Fix
Detect
CommitStep
![Page 39: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/39.jpg)
PM
Condition 2: Writers fix inconsistencies
39
• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )• Inherently crash recoverable
Step 1 Step 2 Step 3Writer 1 CommitStep
![Page 40: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/40.jpg)
PM
Condition 2: Writers fix inconsistencies
40
• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )• Inherently crash recoverable
Step 1 Step 2 Step 3Writer 1
CrashCommit
Step
![Page 41: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/41.jpg)
PM
Condition 2: Writers fix inconsistencies
41
• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )• Inherently crash recoverable
Step 1 Step 2Thread Step 3
Recover
CommitStep
![Page 42: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/42.jpg)
PM
Condition 2: Writers fix inconsistencies
42
• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )• Inherently crash recoverable• Conversion: Adding flushes and fences after each store and
specific loads
Step 2Thread Step 3
Flushes Flushes Flushes Flushes
Fence Fence Fence
Step 1 CommitStep
![Page 43: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/43.jpg)
• Condition 1: Updates via Single Atomic Store• Condition 2: Writers fix inconsistencies• Condition 3: Writers don’t fix inconsistencies
Three Conversion Conditions
43
![Page 44: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/44.jpg)
• Non-blocking readers, Blocking writers (hold locks)• Readers & Writers à Detect ( ), Tolerate ( ), Fix (X)
Condition 3: Writers don’t fix inconsistencies
44
Step 1 Step 2 Step 3Writer 1
Step 1 Step 2 Step 3Writer 2
Update
A sequence of ordered deterministic steps
UpdateCommit
Step
CommitStep
A sequence of ordered deterministic steps
![Page 45: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/45.jpg)
• Non-blocking readers, Blocking writers (hold locks)• Readers & Writers à Detect ( ), Tolerate ( ), Fix (X)
Condition 3: Writers don’t fix inconsistencies
45
Step 1 Step 2 Step 3Writer 1
Step 1 Step 2 Step 3Writer 2
CommitStep
CommitStep
![Page 46: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/46.jpg)
• Non-blocking readers, Blocking writers (hold locks)• Readers & Writers à Detect ( ), Tolerate ( ), Fix (X)
Condition 3: Writers don’t fix inconsistencies
46
Step 1 Step 2Writer 1
Step 1 Step 2 Step 3Writer 2
Crash
CommitStep
Failure
![Page 47: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/47.jpg)
Writer 2
• Readers & Writers à Detect ( ), Tolerate ( ), Fix ( )• Conversion: Add helping mechanism
• Reuse existing algorithm handling each step
Condition 3: Writers don’t fix inconsistencies
47
Step 1 Step 2 Step 3Thread
Step 1 Step 2
Detect
Fix CommitStep
![Page 48: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/48.jpg)
• Overall Intuition• Conversion Conditions• Conversion Example: Masstree• Assumptions & Limitations• Evaluation
Outline
48
![Page 49: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/49.jpg)
• Example: B-link Tree (Masstree)
Conversion of Masstree
49
10
1 10 15 25
30
30
…
…
High Key
High Key
![Page 50: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/50.jpg)
• Example: B-link Tree (Masstree)
Conversion of Masstree
50
Insert 261510
1 10
30
30
…
15 251. installing new sibling
2. insert middle key to parent
![Page 51: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/51.jpg)
• Example: B-link Tree (Masstree)
Conversion of Masstree
51
Insert 2610
1 10
30
30
…
15 25Intermediate state
![Page 52: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/52.jpg)
• Example: B-link Tree (Masstree)
Conversion of Masstree
52
Insert 2610
1 10
30
30
…
Lookup 25
15 25
![Page 53: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/53.jpg)
• Example: B-link Tree (Masstree)
Conversion of Masstree
53
Insert 2610
1 10
30
30
…
Lookup 25
15 25
Detect(25 > 15)
>>>>Tolerate
![Page 54: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/54.jpg)
• Example: B-link Tree (Masstree)
Conversion of Masstree
54
10
1 10
30
30
…
15 25
Crash Permanent Inconsistency
![Page 55: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/55.jpg)
• Example: B-link Tree (Masstree)• Add helping mechanism to resume split
Conversion of Masstree
55
10
1 10
30
30
…
15 25
Insert 30
Detect
15Resume split (recovery)
![Page 56: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/56.jpg)
Conversion Results of Five DRAM Indexes
56
DRAM Index DS Type
CLHT (Cache-Line Hash Table) [ASPLOS’15] Hash tableHOT (Height Optimized Trie) [SIGMOD’18] TrieBwTree [ICDE’13] B+TreeART (Adaptive Radix Tree) [ICDE’13] Radix TreeMasstree [Eurosys’12] Hybrid (B+Tree & Trie)
![Page 57: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/57.jpg)
Conversion Results of Five DRAM Indexes
57
DRAM Index PM Index ConditionCLHT P-CLHT #1HOT P-HOT #1
BwTree P-BwTree #1, #2ART P-ART #1, #3
Masstree P-Masstree #1, #3
• We produce the P-* family of PM indexes
![Page 58: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/58.jpg)
• Overall Intuition• Conversion Conditions• Conversion Example: Masstree• Assumptions & Limitations• Evaluation
Outline
58
![Page 59: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/59.jpg)
• Assume garbage collection in memory allocator• Assume locks are volatile or re-initialized after a crash• Provide low level of isolation: Read Uncommitted• RECIPE applies only to individual data structures
Assumptions & Limitations
59
![Page 60: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/60.jpg)
• Overall Intuition• Conversion Conditions• Conversion Example: Masstree• Assumptions & Limitations• Evaluation
Outline
60
![Page 61: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/61.jpg)
Evaluation
61
• How much effort is involved in converting indexes?• What is the performance of converted indexes?• Are the converted indexes crash consistent?
![Page 62: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/62.jpg)
Evaluation
62
• How much effort is involved in converting indexes?• What is the performance of converted indexes?• Are the converted indexes crash consistent?
![Page 63: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/63.jpg)
Evaluation
63
• How much effort is involved in converting indexes?• What is the performance of converted indexes?
![Page 64: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/64.jpg)
Modified Lines of Code
64
• Conversion for all indexes à <= 200 LoC changes
RECIPE-converted IndexesLines of Code
Index Core ModifiedP-CLHT 2.8K 30 (1%)P-HOT 2K 38 (2%)
P-BwTree 5.2K 85 (1.6%)P-ART 1.5K 52 (3.4%)
P-Masstree 2.2K 200 (9%)
![Page 65: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/65.jpg)
Modified Lines of Code
65
• Conversion for all indexes à <= 200 LoC changes
RECIPE-converted IndexesLines of Code
Index Core ModifiedP-CLHT 2.8K 30 (1%)P-HOT 2K 38 (2%)
P-BwTree 5.2K 85 (1.6%)P-ART 1.5K 52 (3.4%)
P-Masstree 2.2K 200 (9%)
Conversion for all indexes: <= 200 LoC changes<= 9% from core code base
![Page 66: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/66.jpg)
Evaluation
66
• How much effort is involved in converting indexes?• What is the performance of converted indexes?
![Page 67: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/67.jpg)
• 2-socket 96-core machine with 32MB LLC• 768 GB Intel Optane DC PMM, 378 GB DRAM• YCSB with 16 threads• Ordered/Unordered indexes, Integer/String keys
Performance Evaluation
67
Load Workload A Workload B Workload C Workload E
Insertion 100% Insertion 50%Point Lookup 50%
Insertion 5%Point Lookup 95%
Point Lookup 100%
Insertion 5%Range Scan 95%
![Page 68: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/68.jpg)
Ordered Index
68
• Support both point and range operations• P-HOT
• Persistent Height-Optimized Trie converted by RECIPE• FAST & FAIR [FAST’18]
• Hand-crafted PM-based concurrent B+Tree
![Page 69: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/69.jpg)
Ordered Index
69
• P-HOT produced by RECIPE conversion• P-HOT performs up-to 5.2x better in point operations• Cache-efficient designs of P-HOT à Low cache misses
0123456
Load A B C E
Nor
mal
ized
thro
ughp
ut
FAST&FAIR P-HOT
![Page 70: RECIPE : Converting Concurrent DRAM Indexes to Persistent …nvmw.ucsd.edu/nvmw2020-program/unzip/current2/nvmw2020... · 2020-02-20 · Resume split (recovery) Conversion Results](https://reader033.vdocuments.pub/reader033/viewer/2022060500/5f1a4cd904eeda1d3a410ffc/html5/thumbnails/70.jpg)
RECIPE• Principled approach to convert concurrent DRAM
indexes into PM indexes• Case study of changing five DRAM indexes• Evaluations with YCSB show RECIPE indexes have
better performance than hand-crafted PM indexes• Try our indexes: https://github.com/utsaslab/RECIPE
70
RECIPE