図でわかるhdfs erasure coding
TRANSCRIPT
![Page 1: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/1.jpg)
図でわかるHDFS Erasure Coding
Kai Sasaki Treasure Data Inc.
![Page 2: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/2.jpg)
Who am I
佐々木 海(Kai Sasaki)
Software Engineer at Treasure Data Inc. http://www.treasuredata.com
Hadoop, Spark, DL4J
![Page 3: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/3.jpg)
Agenda
• Erasure Coding
• Under the Namespace
• Writing Side
• Reading Side
![Page 4: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/4.jpg)
Erasure Coding
![Page 5: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/5.jpg)
Replication
Block
HDFS
![Page 6: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/6.jpg)
Replication
Block
Block
Block
Block
HDFS
![Page 7: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/7.jpg)
Replication
Block
Block
Block
Block
HDFS
![Page 8: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/8.jpg)
Replication
Block
Block
Block
Block
HDFS
CapacityOverhead
x3
![Page 9: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/9.jpg)
Replication
Block
Block
Block
Block
HDFS
Redundancy 2
CapacityOverhead
x3
![Page 10: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/10.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block
RS-6-3
![Page 11: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/11.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block6 out of 9
![Page 12: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/12.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block6 out of 9
![Page 13: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/13.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block6 out of 9
![Page 14: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/14.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block6 out of 9
![Page 15: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/15.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
BlockRedundancy 3
![Page 16: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/16.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block
CapacityOverhead
x1.5Redundancy 3
![Page 17: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/17.jpg)
Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block
BlockGroup
![Page 18: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/18.jpg)
Under the Namespace
![Page 19: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/19.jpg)
INode and BlockInfo
BlockInfo
INode
![Page 20: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/20.jpg)
INode and BlockInfo
BlockInfo
INode
BlockInfo BlockInfo…
![Page 21: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/21.jpg)
INode and BlockInfo
BlockInfo
INode
BlockInfo BlockInfo…
…
Block
Block
Block
![Page 22: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/22.jpg)
INode and BlockInfo
BlockInfo
INode
BlockInfo BlockInfo…
…
Block
Block
Block
BlockGroup
![Page 23: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/23.jpg)
BlockInfo
…
Block
Block
Block
BlockGroup
![Page 24: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/24.jpg)
long BlockId0 64
BlockInfo
…
Block
Block
Block
BlockGroup
![Page 25: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/25.jpg)
long BlockId0 64
BlockInfo
…
Block
Block
Block
BlockGroup
index GroupId4bit 60bit
![Page 26: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/26.jpg)
long BlockId0 64
BlockInfo
…
Block
Block
Block
BlockGroup
index GroupId4bit 60bit
index 0
index 2index 1
Saving memory
![Page 27: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/27.jpg)
Writing Side
![Page 28: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/28.jpg)
Data
0
![Page 29: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/29.jpg)
Data64KB
0
![Page 30: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/30.jpg)
Data
BlockGroup
64KB
0
![Page 31: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/31.jpg)
Data
BlockGroup
0
![Page 32: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/32.jpg)
Data
BlockGroup
0
![Page 33: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/33.jpg)
Data
BlockGroup
0
![Page 34: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/34.jpg)
Data
BlockGroup
0
![Page 35: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/35.jpg)
Data
BlockGroup
0
![Page 36: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/36.jpg)
Data
BlockGroup
0
![Page 37: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/37.jpg)
Data
BlockGroup
0
![Page 38: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/38.jpg)
Data
BlockGroup
0
Data Block
![Page 39: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/39.jpg)
Data
BlockGroup
0
Data Block Parity Block
![Page 40: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/40.jpg)
Data
BlockGroup
0
Data Block Parity Block
Stripe
![Page 41: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/41.jpg)
Data
BlockGroup
0
![Page 42: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/42.jpg)
Data
BlockGroup
0
![Page 43: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/43.jpg)
Data
BlockGroup
0
![Page 44: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/44.jpg)
Data
BlockGroup
0
![Page 45: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/45.jpg)
Data
BlockGroup
0
![Page 46: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/46.jpg)
Data
BlockGroup
0
![Page 47: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/47.jpg)
Data
BlockGroup
0
0
![Page 48: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/48.jpg)
Data
BlockGroup
0
0
![Page 49: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/49.jpg)
Data
BlockGroup
0
0
Saving diskspace usage
![Page 50: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/50.jpg)
Reading Side
![Page 51: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/51.jpg)
BlockGroup
0
0
![Page 52: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/52.jpg)
BlockGroup
0
0
200kb 500kb
![Page 53: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/53.jpg)
BlockGroup
0
0
200kb 500kb
![Page 54: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/54.jpg)
BlockGroup
0
0
200kb 500kb
![Page 55: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/55.jpg)
BlockGroup
0
0
200kb 500kb
![Page 56: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/56.jpg)
BlockGroup
0
0
200kb 500kb
![Page 57: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/57.jpg)
BlockGroup
0
0
200kb 500kb
![Page 58: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/58.jpg)
BlockGroup
0
0
200kb 500kb
![Page 59: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/59.jpg)
BlockGroup
0
0
200kb 500kb
Saving reading time
![Page 60: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/60.jpg)
まとめ
• Namespace -> Saving memory BlockInfoStriped, BlockIdManager
• Writing Side -> Saving diskspace usage INodeFile
• Reading Side -> Saving reading time DFSStripedInputStream
![Page 61: 図でわかるHDFS Erasure Coding](https://reader031.vdocuments.pub/reader031/viewer/2022021922/586fde9e1a28ab18428b6c71/html5/thumbnails/61.jpg)
ありがとうございました