Download - OpenStack Swift的性能调优
![Page 1: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/1.jpg)
P e r f o r m a n c e Tu n i n g o f O p e n S t a c k S w i f t
李明宇 奥思数据 创始⼈人& CTO
Email: [email protected]
O p e n S t a c k S w i f t的性能调优
![Page 2: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/2.jpg)
Content
• What is object storage and the object storage APIs? • OpenStack Swift • How to improve the performance of Swift in different
scenarios Basic scenario Large objects Multiple regions Access the data with filesystem APIs
• Will erasure coding degrade or improve the performance of Swift?
![Page 3: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/3.jpg)
Object Storage S3-like Storage
Data are stored in bucketscontainers Flat Namespace
RESTful API
![Page 4: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/4.jpg)
OpenStack Swift• An open source
object storage system.
• One of the first two projects
of OpenStack.
• Very high durability.
• Very high scalability.
• Very high availability.
• Global distributed
(multi-region) cluster.
![Page 5: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/5.jpg)
OpenStack Swift• Users of Swift
![Page 6: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/6.jpg)
The architecture of a Swift cluster
![Page 7: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/7.jpg)
The architecture of a Swift cluster
图引⾃自Intel
![Page 8: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/8.jpg)
OStorage An Object Storage Company
Speed is funSpeed is fun
![Page 9: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/9.jpg)
图引⾃自Intel
How many proxy severs we need to reach the maximum throughput ?
![Page 10: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/10.jpg)
Large objects
• Swift is good at storing small objects like photos and documents.
![Page 11: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/11.jpg)
Large objects
• Swift is good at storing small objects like photos and documents.
• How about large objects?
11
![Page 12: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/12.jpg)
• Large object support APIs
12
Large objects
swift upload test_container -S 1073741824 large_fileswift download test_container large_file# First, upload the segmentscurl -X PUT -H 'X-Auth-Token: <token>' http://<storage_url>/container/myobject/00000001 --data-binary '1'curl -X PUT -H 'X-Auth-Token: <token>' http://<storage_url>/container/myobject/00000002 --data-binary '2'curl -X PUT -H 'X-Auth-Token: <token>' http://<storage_url>/container/myobject/00000003 --data-binary '3'
# Next, create the manifest filecurl -X PUT -H 'X-Auth-Token: <token>' -H 'X-Object-Manifest: container/myobject/' http://<storage_url>/container/myobject --data-binary ''
# And now we can download the segments as a single objectcurl -H 'X-Auth-Token: <token>' http://<storage_url>/container/myobject
![Page 13: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/13.jpg)
Zone 1 Zone 2 ……Zone 3 Zone 1 Zone 2 ……Zone 3
Replication Network
Region 1 Region 2
Multiple Regions
13
[app:proxy-server] write_affinity = r1
[app:proxy-server] …… read_affinity = r1=100
![Page 14: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/14.jpg)
Multiple Regions Experiment
14
![Page 15: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/15.jpg)
15
Multiple Regions Experimental Results
• Writes
![Page 16: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/16.jpg)
16
Multiple Regions Experimental Results
• Reads
![Page 17: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/17.jpg)
17
Zone 1 Zone 2 ……Zone 3 Zone 1 Zone 2 ……Zone 3
Replication Network
Region 1 Region 2
Be careful with the Eventually Consistency!
Only object PUT requests are affected by the write_affinity setting.
Multiple Regions
![Page 18: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/18.jpg)
Access the data with filesystem APIs
18
![Page 19: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/19.jpg)
Proxy FS Proxy
Swift Object Server
File Object
Access the data with filesystem APIs
In Go language
![Page 20: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/20.jpg)
Erasure Coding in Swift
20
![Page 21: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/21.jpg)
21
Erasure Coding in Swift
![Page 22: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/22.jpg)
Erasure codes encode/decode at the proxy servers
EC audit and reconstruct at the object server
22
Erasure Coding in Swift
![Page 23: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/23.jpg)
Erasure codes encode/decode at the proxy servers
EC audit and reconstruct at the object server
23
Erasure Coding in Swift
How many proxy severs we need to reach the maximum throughput
when using EC?
![Page 24: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/24.jpg)
Erasure Coding in Swift
![Page 25: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/25.jpg)
Erasure Coding in Swift
![Page 26: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/26.jpg)
• Compared with replicas … • EC is more suitable for large objects. It
can reach the maximum throughput with less network bandwidth cost.
• But for small objects, the performance will degrade when using EC due to heavier IO cost.
26
Erasure Coding in Swift Conclusion
![Page 27: OpenStack Swift的性能调优](https://reader036.vdocuments.pub/reader036/viewer/2022082207/5871997c1a28ab044e8b5625/html5/thumbnails/27.jpg)
27
Questions ?