netapp ef540 ssd storage test

1 号店

EF540 全闪存阵列性能压力测试

客户:1 号店

版本: 1.1

文档创建日期:2014 年 4 月

1 号店

第 1 页

文档版本历史

版本修改更新日期

1.0 崔彦巍 2014/4/30

1.1 孙龙 2014/5/4

1 号店

第 2 页

目录 1 测试目标 ..................................................................................................................................... 3

2 测试系统介绍 ............................................................................................................................. 3

2.1 测试环境配置 ............................................................................................................. 3

2.2 测试环境拓扑图 ......................................................................................................... 4

3 测试系统配置 ............................................................................................................................. 4

3.1 存储系统参考配置 ..................................................................................................... 4

3.2 RAID Group 布局 ....................................................................................................... 4

3.3 存储控制器管理 IP 设置 ........................................................................................... 5

3.4 Volume/LUN 配置 ...................................................................................................... 6

3.5 Hosts Mapping 定义 ................................................................................................... 6

4 性能、功能测试结果记录 ......................................................................................................... 7

4.1 双控制器下 16 个 LUN 性能测试 ............................................................................. 7

4.1.1 4KB 数据块随机读，随机写，随机读写测试 ................................................ 8

4.1.2 8KB 数据块随机读，随机写，随机读写测试 .............................................. 14

4.1.3 16KB 数据块随机读，随机写，随机读写测试 ............................................ 21

4.2 双控制器下 2 个 LUN 性能测试 ............................................................................. 28

4.3 EF540 全闪存阵列实时采样 ................................................................................... 30

5 性能分析................................................................................................................................... 34

1 号店

第 3 页

1 测试目标

测试全闪存阵列系统在数据库应用场景下推荐存储系统可达到的最大性能。

2 测试系统介绍

2.1 测试环境配置

配置数量备注

服务器 DELL R815 4CPU 12Core(2.5G Hz)/32GB

1 张 2 Port 8Gb FC HBA

CentOS 6.4(64 bit)

DM-Multipath

1

存储 EF540 全闪存阵列

24GB Memory

8*8Gb FC Host Port(实际使用 2 个)

8* 800GB SSD(Sandisk Lighting Enterprise

SSD)

SANTricity Manager 11.10

1

测试工具 FIO

版本：

fio-2.0.10

1 号店

第 4 页

2.2 测试环境拓扑图

3 测试系统配置

3.1 存储系统参考配置

3.2 RAID Group 布局

EF540

RAID Group RAID5 (7D+1P)

Spare 0

1 号店

第 5 页

3.3 存储控制器管理 IP 设置

1 号店

第 6 页

3.4 Volume/LUN 配置

lun 1~16 为 Linux 环境下 FIO 测试使用的 LUN

3.5 Hosts Mapping 定义

CentOS 的主机 mapping 定义

1 号店

第 7 页

4 性能、功能测试结果记录

4.1 双控制器下 16 个 LUN 性能测试

使用一台 DELL 815 服务器通过 FIO 对 EF540 的 16 个 120GB LUN 进行压力测

试, 实际性能压力体现为 EF540 两个存储控制器的性能

测试 LUN 信息

Name: /dev/dm-0 Size: 107g
















1 号店

第 8 页

4.1.1 4KB 数据块随机读，随机写，随机读

写测试

测试命令：

fio -direct=1 -rw=randread -bs=4k -size=50g -numjobs=8 -runtime=100

-group_reporting -ioengine=libaio -iodepth=16 -iodepth_batch=8 -iodepth_low=8

-filename=/dev/dm-0: /dev/dm-1: /dev/dm-2: /dev/dm-3: /dev/dm-10: /dev/dm-11:

/dev/dm-12: /dev/dm-13: /dev/dm-14: /dev/dm-15: /dev/dm-16: /dev/dm-17: /dev/dm-18:

/dev/dm-19: /dev/dm-20: /dev/dm-21

其中 numjobs、iodepth 为变量，范围 4，8，12，16，20，24，28，32

测试结果：

测试 IO IO 大

小

文件

大小

runtim

e 线程数

Iodep

th IOPS

延时

(MS) USR CPU SYS CPU

Iowa

it

randread 4k 50G 100 4 4 44514 0.35 0.60% 3.30% 2.7

randread 4k 50G 100 4 8 87887 0.36 0.60% 4.00% 0.7

randread 4k 50G 100 4 12 91605 0.49 0.50% 3.50% 1.7

randread 4k 50G 100 4 16 107421 0.56 0.40% 4.30% 0.2

randread 4k 50G 100 4 20 104346 0.61 0.40% 4.40% 0.3

randread 4k 50G 100 4 24 111746 0.68 0.40% 4.30% 0.1

randread 4k 50G 100 4 28 110654 0.76 0.40% 4.40% 0.1

randread 4k 50G 100 4 32 118751 0.79 0.40% 4.30% 0.1

randread 4k 50G 100 8 4 88804 0.35 1.00% 6.20% 5.2

randread 4k 50G 100 8 8 152755 0.41 1.10% 8.10% 1.4

randread 4k 50G 100 8 12 140117 0.63 0.80% 7.40% 2.8

randread 4k 50G 100 8 16 181534 0.67 0.60% 8.80% 0.4

randread 4k 50G 100 8 20 178896 0.72 0.70% 8.70% 0.5

randread 4k 50G 100 8 24 185528 0.83 0.60% 8.80% 0.2

randread 4k 50G 100 8 28 188597 0.89 0.60% 8.80% 0.1

randread 4k 50G 100 8 32 187941 1.00 0.60% 8.80% 0.1

randread 4k 50G 100 12 4 127708 0.37 1.30% 8.50% 5.7

randread 4k 50G 100 12 8 204017 0.47 1.40% 11.60% 2.8

randread 4k 50G 100 12 12 185555 0.72 0.90% 9.90% 4.8

randread 4k 50G 100 12 16 192774 0.95 0.70% 8.90% 7.1

1 号店

第 9 页

randread 4k 50G 100 12 20 205552 0.89 0.90% 12.10% 2.7

randread 4k 50G 100 12 24 193162 1.11 0.60% 8.00% 7.9

randread 4k 50G 100 12 28 198372 1.24 0.60% 8.90% 6.5

randread 4k 50G 100 12 32 204008 1.37 0.60% 9.70% 7.3

randread 4k 50G 100 16 4 160077 0.39 1.60% 10.70% 7.2

randread 4k 50G 100 16 8 217644 0.58 1.50% 13.90% 7.3

randread 4k 50G 100 16 12 154769 1.14 0.90% 8.10% 11.2

randread 4k 50G 100 16 16 207130 1.18 0.70% 8.90% 9.2

randread 4k 50G 100 16 20 200772 1.12 0.70% 8.90% 9.2

randread 4k 50G 100 16 24 204499 1.37 0.60% 9.00% 4.9

randread 4k 50G 100 16 28 201872 1.61 0.60% 8.90% 9.1

randread 4k 50G 100 16 32 233941 1.57 0.80% 12.00% 8.8

randread 4k 50G 100 20 4 188200 0.42 1.80% 13.10% 10

randread 4k 50G 100 20 8 174296 0.91 1.10% 8.50% 9.5

randread 4k 50G 100 20 12 156891 1.42 0.90% 8.10% 11

randread 4k 50G 100 20 16 233806 1.30 0.80% 12.60% 11.8

randread 4k 50G 100 20 20 227506 1.23 0.80% 13.40% 10.4

randread 4k 50G 100 20 24 236235 1.47 0.90% 15.30% 9

randread 4k 50G 100 20 28 196363 2.06 0.60% 9.00% 9.1

randread 4k 50G 100 20 32 238106 1.94 0.80% 12.90% 8.7

randread 4k 50G 100 24 4 205661 0.46 1.80% 14.30% 16

randread 4k 50G 100 24 8 195365 0.98 1.40% 11.40% 11.6

randread 4k 50G 100 24 12 157257 1.72 0.90% 7.90% 11.4

randread 4k 50G 100 24 16 242624 1.51 0.90% 14.90% 12.1

randread 4k 50G 100 24 20 210796 1.55 0.70% 10.00% 7.7

randread 4k 50G 100 24 24 230311 1.79 0.80% 12.60% 7.2

randread 4k 50G 100 24 28 249853 1.91 0.90% 14.90% 15.6

randread 4k 50G 100 24 32 237849 2.34 0.80% 12.50% 10.9

randread 4k 50G 100 28 4 187258 0.58 1.70% 12.80% 16

randread 4k 50G 100 28 8 216856 1.10 1.40% 11.80% 11.8

randread 4k 50G 100 28 12 159288 1.99 1.00% 9.00% 12.2

randread 4k 50G 100 28 16 237960 1.81 0.80% 12.60% 11.6

randread 4k 50G 100 28 20 219234 1.72 0.90% 12.80% 9.8

randread 4k 50G 100 28 24 244730 1.94 0.90% 13.80% 13.5

randread 4k 50G 100 28 28 244470 2.29 0.80% 12.70% 12.8

randread 4k 50G 100 28 32 152887 4.20 0.70% 6.10% 17.1

randread 4k 50G 100 32 4 172406 0.74 1.70% 12.40% 15

randread 4k 50G 100 32 8 232623 1.09 1.50% 13.20% 13

randread 4k 50G 100 32 12 170122 2.15 0.90% 7.90% 11.4

randread 4k 50G 100 32 16 242423 2.04 0.90% 13.50% 12.5

randread 4k 50G 100 32 20 238941 1.80 0.90% 13.30% 10.2

randread 4k 50G 100 32 24 225354 2.39 0.80% 11.60% 10.7

randread 4k 50G 100 32 28 145215 4.33 0.70% 7.00% 16.7

1 号店

第 10 页

randread 4k 50G 100 32 32 135341 5.19 0.70% 7.60% 21.1

randwrite 4k 50G 100 4 4 33803 0.46 0.60% 3.10% 3

randwrite 4k 50G 100 4 8 48470 0.65 0.60% 3.50% 2.3

randwrite 4k 50G 100 4 12 47081 0.95 0.50% 3.50% 2.4

randwrite 4k 50G 100 4 16 49728 1.11 0.40% 3.20% 2.5

randwrite 4k 50G 100 4 20 49834 1.20 0.40% 3.30% 2.6

randwrite 4k 50G 100 4 24 50111 1.37 0.40% 3.40% 2.6

randwrite 4k 50G 100 4 28 49963 1.45 0.40% 3.40% 2.6

randwrite 4k 50G 100 4 32 49573 1.61 0.40% 3.40% 2.6

randwrite 4k 50G 100 8 4 49642 0.63 0.90% 4.90% 6.5

randwrite 4k 50G 100 8 8 51074 1.25 0.90% 4.80% 7

randwrite 4k 50G 100 8 12 49394 1.78 0.60% 4.60% 8.1

randwrite 4k 50G 100 8 16 50273 2.09 0.60% 4.50% 7.3

randwrite 4k 50G 100 8 20 50429 2.23 0.60% 4.40% 6.9

randwrite 4k 50G 100 8 24 50791 2.53 0.50% 4.20% 7.8

randwrite 4k 50G 100 8 28 51004 2.59 0.60% 4.50% 7.7

randwrite 4k 50G 100 8 32 50975 2.79 0.50% 4.20% 7.1

randwrite 4k 50G 100 12 4 49820 0.95 1.00% 5.70% 9.3

randwrite 4k 50G 100 12 8 50461 1.89 0.90% 5.30% 10

randwrite 4k 50G 100 12 12 48260 2.70 0.70% 5.20% 10.4

randwrite 4k 50G 100 12 16 50838 3.01 0.60% 5.00% 10.5

randwrite 4k 50G 100 12 20 50722 3.21 0.60% 4.80% 11

randwrite 4k 50G 100 12 24 50782 3.71 0.60% 4.80% 11.5

randwrite 4k 50G 100 12 28 50651 3.73 0.60% 4.70% 11.5

randwrite 4k 50G 100 12 32 50805 3.98 0.60% 4.80% 11.9

randwrite 4k 50G 100 16 4 51150 1.24 1.20% 6.30% 11.9

randwrite 4k 50G 100 16 8 50750 2.51 1.00% 5.80% 13.3

randwrite 4k 50G 100 16 12 49641 3.48 0.70% 5.20% 14.4

randwrite 4k 50G 100 16 16 50973 3.98 0.70% 5.40% 14.3

randwrite 4k 50G 100 16 20 51315 4.19 0.70% 5.30% 14.4

randwrite 4k 50G 100 16 24 51613 4.87 0.60% 5.20% 14.6

randwrite 4k 50G 100 16 28 51316 4.77 0.60% 4.90% 14.6

randwrite 4k 50G 100 16 32 51324 5.01 0.60% 5.10% 14.5

randwrite 4k 50G 100 20 4 51456 1.54 1.10% 6.30% 16.2

randwrite 4k 50G 100 20 8 51859 3.07 1.10% 6.30% 16.7

randwrite 4k 50G 100 20 12 49828 4.32 0.80% 5.80% 17.3

randwrite 4k 50G 100 20 16 51574 4.88 0.80% 5.80% 17.4

randwrite 4k 50G 100 20 20 51803 5.13 0.70% 5.60% 17.5

randwrite 4k 50G 100 20 24 51791 6.01 0.80% 5.40% 17.1

randwrite 4k 50G 100 20 28 51785 5.87 0.70% 6.20% 17.2

randwrite 4k 50G 100 20 32 51844 6.09 0.60% 5.10% 17.9

randwrite 4k 50G 100 24 4 52021 1.83 1.20% 6.80% 20

randwrite 4k 50G 100 24 8 51932 3.68 1.10% 6.70% 20.7

1 号店

第 11 页

randwrite 4k 50G 100 24 12 51283 5.03 0.90% 6.80% 20.9

randwrite 4k 50G 100 24 16 52147 5.76 0.70% 5.80% 20.9

randwrite 4k 50G 100 24 20 52363 6.03 0.80% 5.70% 20.8

randwrite 4k 50G 100 24 24 52227 7.03 0.70% 5.30% 21.2

randwrite 4k 50G 100 24 28 52240 6.85 0.70% 5.50% 20

randwrite 4k 50G 100 24 32 52189 7.16 0.70% 5.40% 20.5

randwrite 4k 50G 100 28 4 51870 2.15 1.20% 7.10% 24.1

randwrite 4k 50G 100 28 8 52182 4.28 1.20% 7.00% 25.2

randwrite 4k 50G 100 28 12 52245 5.80 0.90% 6.50% 24

randwrite 4k 50G 100 28 16 52365 6.12 0.90% 6.50% 24

randwrite 4k 50G 100 28 20 52177 7.08 0.90% 6.70% 23.5

randwrite 4k 50G 100 28 24 52562 8.11 0.80% 5.70% 24.3

randwrite 4k 50G 100 28 28 52695 7.83 0.70% 5.70% 24.2

randwrite 4k 50G 100 28 32 52658 8.15 0.70% 5.50% 23.6

randwrite 4k 50G 100 32 4 52205 2.13 0.80% 5.90% 26.7

randwrite 4k 50G 100 32 8 53077 4.36 0.80% 5.80% 26.8

randwrite 4k 50G 100 32 12 54771 5.30 0.80% 5.80% 27.5

randwrite 4k 50G 100 32 16 54009 6.70 0.80% 5.90% 27.9

randwrite 4k 50G 100 32 20 52049 7.50 0.80% 5.90% 26.6

randwrite 4k 50G 100 32 24 52653 8.79 0.80% 5.80% 28

randwrite 4k 50G 100 32 28 52741 8.89 0.80% 5.80% 29

randwrite 4k 50G 100 32 32 52342 9.04 0.80% 5.80% 28.8

randrw 4k 50G 100 4 4 14849 0.74 0.60% 2.80% 2.8

randrw 4k 50G 100 4 8 21475 1.07 0.60% 3.30% 2.5

randrw 4k 50G 100 4 12 25609 1.21 0.50% 3.20% 2.7

randrw 4k 50G 100 4 16 23821 1.57 0.50% 3.20% 2.7

randrw 4k 50G 100 4 20 24522 1.65 0.50% 3.10% 2.8

randrw 4k 50G 100 4 24 25220 1.78 0.40% 2.90% 2.9

randrw 4k 50G 100 4 28 25568 1.88 0.50% 3.20% 2.7

randrw 4k 50G 100 4 32 25767 2.09 0.40% 3.30% 2.6

randrw 4k 50G 100 8 4 20914 1.10 0.90% 4.40% 5.9

randrw 4k 50G 100 8 8 28281 1.74 0.90% 5.00% 7.7

randrw 4k 50G 100 8 12 32567 2.01 0.70% 4.60% 6.9

randrw 4k 50G 100 8 16 30750 2.51 0.70% 5.00% 7.1

randrw 4k 50G 100 8 20 31308 2.70 0.70% 5.10% 7.3

randrw 4k 50G 100 8 24 31485 2.91 0.70% 4.60% 8

randrw 4k 50G 100 8 28 32136 3.05 0.70% 4.80% 7.9

randrw 4k 50G 100 8 32 32351 3.24 0.70% 5.10% 7.3

randrw 4k 50G 100 12 4 24550 1.46 1.10% 5.40% 8.7

randrw 4k 50G 100 12 8 31355 2.48 1.10% 5.80% 8.6

randrw 4k 50G 100 12 12 35698 2.91 0.90% 5.80% 8.8

randrw 4k 50G 100 12 16 33239 3.59 0.80% 5.20% 8.9

randrw 4k 50G 100 12 20 33687 3.85 0.80% 4.90% 8.7

1 号店

第 12 页

randrw 4k 50G 100 12 24 34222 4.15 0.80% 5.00% 8.5

randrw 4k 50G 100 12 28 34418 4.34 0.80% 5.00% 8.8

randrw 4k 50G 100 12 32 34935 4.58 0.80% 5.00% 8.3

randrw 4k 50G 100 16 4 27640 1.74 1.10% 5.60% 10.2

randrw 4k 50G 100 16 8 33610 3.20 1.30% 6.70% 10.3

randrw 4k 50G 100 16 12 37968 3.70 1.00% 5.80% 10.5

randrw 4k 50G 100 16 16 35546 4.62 0.90% 5.70% 10.3

randrw 4k 50G 100 16 20 36536 4.77 0.90% 5.80% 10.3

randrw 4k 50G 100 16 24 37830 4.96 0.90% 5.80% 10

randrw 4k 50G 100 16 28 39140 5.00 0.90% 5.90% 10.4

randrw 4k 50G 100 16 32 39982 5.25 0.90% 6.00% 10.4

randrw 4k 50G 100 20 4 29820 2.03 1.40% 6.60% 12.3

randrw 4k 50G 100 20 8 34975 3.91 1.50% 7.20% 12.2

randrw 4k 50G 100 20 12 41282 4.14 1.30% 7.50% 11.7

randrw 4k 50G 100 20 16 39801 5.10 1.20% 7.20% 11.2

randrw 4k 50G 100 20 20 41374 5.27 1.20% 6.80% 11.7

randrw 4k 50G 100 20 24 42921 5.43 1.00% 6.50% 11.7

randrw 4k 50G 100 20 28 43579 5.50 1.10% 6.60% 12

randrw 4k 50G 100 20 32 43803 5.65 1.10% 7.10% 12.2

randrw 4k 50G 100 24 4 32059 2.33 1.50% 7.00% 14.2

randrw 4k 50G 100 24 8 37177 4.54 1.50% 7.90% 14.2

randrw 4k 50G 100 24 12 43160 4.42 1.50% 8.00% 14

randrw 4k 50G 100 24 16 43326 5.40 1.30% 7.60% 13.5

randrw 4k 50G 100 24 20 44004 5.46 1.40% 8.10% 13.5

randrw 4k 50G 100 24 24 44325 5.63 1.30% 8.20% 13.6

randrw 4k 50G 100 24 28 44566 5.75 1.10% 6.70% 13.8

randrw 4k 50G 100 24 32 45241 5.98 1.20% 7.70% 14

randrw 4k 50G 100 28 4 33275 2.62 1.50% 7.40% 16.4

randrw 4k 50G 100 28 8 40567 4.87 1.80% 9.00% 16.7

randrw 4k 50G 100 28 12 43519 4.44 1.50% 8.60% 16.3

randrw 4k 50G 100 28 16 44002 5.53 1.30% 7.70% 15.7

randrw 4k 50G 100 28 20 45290 5.72 1.30% 7.80% 15.6

randrw 4k 50G 100 28 24 45742 5.93 1.30% 8.40% 15.8

randrw 4k 50G 100 28 28 45649 6.17 1.40% 8.80% 16.2

randrw 4k 50G 100 28 32 46557 6.49 1.30% 8.60% 16.5

randrw 4k 50G 100 32 4 34612 3.03 1.70% 7.90% 18.6

randrw 4k 50G 100 32 8 42695 5.09 2.00% 9.70% 19.2

randrw 4k 50G 100 32 12 44232 4.59 1.50% 8.40% 18.6

randrw 4k 50G 100 32 16 44679 5.70 1.40% 8.50% 18.2

randrw 4k 50G 100 32 20 45811 5.98 1.40% 8.10% 17.8

randrw 4k 50G 100 32 24 45738 6.34 1.40% 9.30% 18.5

randrw 4k 50G 100 32 28 46132 6.69 1.30% 8.20% 18.9

randrw 4k 50G 100 32 32 45891 7.09 1.20% 7.50% 18.6

1 号店

第 13 页

IOPS 趋势图

图为 4k 随机读的 IOPS，线程和队列深度均为是 4~32，以 4 为单位的等差数列。其中横坐标

为线程，纵坐标为 IOPS，不同颜色的线代表不同队列深度所对应的 IOPS。

图为 4k 随机写的 IOPS。

图为 4k 随机读写的 IOPS。

1 号店

第 14 页

Latency 趋势图

图为 4k 随机读的的延时。

图为 4k 随机写的延时。

图为 4k 随机读写的延时。

4.1.2 8KB 数据块随机读，随机写，随机读

写测试测试命令：


1 号店

第 15 页






测试结果：

测试 IO

IO

大

小

文件

大小

runti

me

线程

数

Iodep

th IOPS

延时


Iowai

t

randread 8k 50G 100 4 4 34215 0.46 0.60% 3.00% 3

randread 8k 50G 100 4 8 72023 0.44 0.60% 3.60% 1.5

randread 8k 50G 100 4 12 69337 0.63 0.30% 2.30% 5.1

randread 8k 50G 100 4 16 94895 0.60 0.40% 4.00% 0.9

randread 8k 50G 100 4 20 87915 0.67 0.50% 3.80% 1.1

randread 8k 50G 100 4 24 95250 0.71 0.40% 4.20% 0.7

randread 8k 50G 100 4 28 103981 0.72 0.40% 4.30% 0.5

randread 8k 50G 100 4 32 103252 0.85 0.40% 4.40% 0.3

randread 8k 50G 100 8 4 69325 0.45 1.00% 6.00% 5.8

randread 8k 50G 100 8 8 125421 0.50 1.00% 7.50% 2.8

randread 8k 50G 100 8 12 122402 0.72 0.90% 7.20% 3.4

randread 8k 50G 100 8 16 112851 0.9 0.90% 7.00% 3.6

randread 8k 50G 100 8 20 146347 0.82 0.70% 7.80% 2.1

randread 8k 50G 100 8 24 154291 0.89 0.70% 8.10% 1.7

randread 8k 50G 100 8 28 160384 0.94 0.70% 8.30% 1.4

randread 8k 50G 100 8 32 162723 1.03 0.80% 8.50% 1.4

randread 8k 50G 100 12 4 100335 0.47 1.30% 7.90% 6.3

randread 8k 50G 100 12 8 158117 0.60 1.40% 9.90% 4

randread 8k 50G 100 12 12 142640 0.91 1.00% 8.70% 6.4

randread 8k 50G 100 12 16 163349 1.03 0.99% 9.90% 5.9

randread 8k 50G 100 12 20 162112 1.06 0.80% 9.20% 6.7

randread 8k 50G 100 12 24 160551 1.23 0.70% 8.20% 7.3

randread 8k 50G 100 12 28 162550 1.38 0.80% 8.10% 10.9

randread 8k 50G 100 12 32 163341 1.57 0.70% 8.50% 6.6

randread 8k 50G 100 16 4 125871 0.53 1.70% 10.40% 8.2

randread 8k 50G 100 16 8 163539 0.78 1.40% 10.90% 8.9

randread 8k 50G 100 16 12 157730 1.1 1.20% 10.50% 8.9

randread 8k 50G 100 16 16 159639 1.46 0.80% 9.20% 10.5

randread 8k 50G 100 16 20 153349 1.39 0.80% 8.00% 8.7

randread 8k 50G 100 16 24 161021 1.63 0.80% 8.30% 8.8

randread 8k 50G 100 16 28 163007 1.87 0.80% 8.10% 10

1 号店

第 16 页

randread 8k 50G 100 16 32 163534 2.14 0.80% 8.70% 9.3

randread 8k 50G 100 20 4 145883 0.54 1.50% 10.20% 8.6

randread 8k 50G 100 20 8 147577 1.08 1.50% 10.70% 9.6

randread 8k 50G 100 20 12 133282 1.65 1.20% 10.40% 8.5

randread 8k 50G 100 20 16 133282 1.9 0.80% 9.20% 10.5

randread 8k 50G 100 20 20 156790 1.68 0.70% 7.70% 6.8

randread 8k 50G 100 20 24 162297 2.01 0.70% 8.20% 8.8

randread 8k 50G 100 20 28 163585 2.34 0.80% 8.00% 10.8

randread 8k 50G 100 20 32 164197 2.69 0.80% 8.70% 9.3

randread 8k 50G 100 24 4 157179 0.6 1.90% 12.80% 16.4

randread 8k 50G 100 24 8 141949 1.3 1.10% 8.00% 11.1

randread 8k 50G 100 24 12 133868 2 0.80% 7.10% 13

randread 8k 50G 100 24 16 160819 2.26 0.80% 8.50% 11.9

randread 8k 50G 100 24 20 147941 2.08 0.70% 7.40% 10.4

randread 8k 50G 100 24 24 162490 2.4 0.70% 8.20% 8.8

randread 8k 50G 100 24 28 163948 2.83 0.70% 8.10% 10.7

randread 8k 50G 100 24 32 162944 3.33 0.70% 9.30% 12.1

randread 8k 50G 100 28 4 144739 0.76 1.90% 12.70% 18.2

randread 8k 50G 100 28 8 150063 1.48 1.20% 9.10% 12.1

randread 8k 50G 100 28 12 135442 2.32 0.90% 7.20% 12.9

randread 8k 50G 100 28 16 162023 2.64 0.80% 8.70% 11.6

randread 8k 50G 100 28 20 162247 2.24 0.70% 8.70% 9.9

randread 8k 50G 100 28 24 163651 2.78 0.90% 10.60% 13.3

randread 8k 50G 100 28 28 164464 3.28 0.70% 9.70% 15

randread 8k 50G 100 28 32 122809 5.2 0.60% 6.90% 19.3

randread 8k 50G 100 32 4 129239 0.98 1.50% 10.10% 15.8

randread 8k 50G 100 32 8 157868 1.61 1.30% 9.10% 11

randread 8k 50G 100 32 12 144834 2.48 1.10% 9.20% 13.9

randread 8k 50G 100 32 16 161323 3 0.90% 10.80% 19.7

randread 8k 50G 100 32 20 159631 2.57 0.70% 8.10% 9.1

randread 8k 50G 100 32 24 164876 4 0.80% 9.80% 17

randread 8k 50G 100 32 28 125854 4.98 0.70% 7.70% 26.1

randread 8k 50G 100 32 32 120136 5.78 0.80% 7.70% 21.6

randwrite 8k 50G 100 4 4 24885 0.63 0.50% 2.60% 3.8

randwrite 8k 50G 100 4 8 25232 1.26 0.40% 2.50% 4.2

randwrite 8k 50G 100 4 12 24087 1.8 0.30% 2.10% 4.7

randwrite 8k 50G 100 4 16 25222 1.61 0.30% 2.10% 4.7

randwrite 8k 50G 100 4 20 25234 1.85 0.30% 2.10% 4.9

randwrite 8k 50G 100 4 24 25191 2.48 0.30% 2.10% 4.6

randwrite 8k 50G 100 4 28 25133 2.6 0.30% 2.00% 5

randwrite 8k 50G 100 4 32 24965 2.81 0.20% 2.00% 5.1

randwrite 8k 50G 100 8 4 25377 1.25 0.60% 3.00% 8.5

randwrite 8k 50G 100 8 8 25459 2.5 0.50% 2.70% 11.7

1 号店

第 17 页

randwrite 8k 50G 100 8 12 24727 3.48 0.40% 2.40% 10.8

randwrite 8k 50G 100 8 16 25357 3.9 0.30% 2.40% 10.9

randwrite 8k 50G 100 8 20 25531 4.12 0.30% 2.20% 11

randwrite 8k 50G 100 8 24 25419 4.74 0.30% 2.40% 11.2

randwrite 8k 50G 100 8 28 25498 4.78 0.30% 2.20% 11.2

randwrite 8k 50G 100 8 32 25551 5.03 0.30% 2.30% 11

randwrite 8k 50G 100 12 4 25449 1.87 0.60% 3.20% 11.9

randwrite 8k 50G 100 12 8 25646 3.73 0.60% 2.90% 13.2

randwrite 8k 50G 100 12 12 25007 5.17 0.40% 2.90% 13.1

randwrite 8k 50G 100 12 16 25674 5.71 0.40% 2.70% 14.1

randwrite 8k 50G 100 12 20 25585 6.1 0.40% 2.50% 12.8

randwrite 8k 50G 100 12 24 25684 6.99 0.40% 2.60% 13.4

randwrite 8k 50G 100 12 28 25683 6.85 0.40% 2.50% 13.7

randwrite 8k 50G 100 12 32 25640 7.13 0.40% 2.50% 12.8

randwrite 8k 50G 100 16 4 25790 2.47 0.60% 3.40% 15.4

randwrite 8k 50G 100 16 8 25800 4.95 0.60% 3.20% 15.8

randwrite 8k 50G 100 16 12 25713 6.60 0.50% 2.90% 15.7

randwrite 8k 50G 100 16 16 25784 7.6 0.40% 2.80% 15.4

randwrite 8k 50G 100 16 20 25703 8.14 0.40% 2.60% 15.9

randwrite 8k 50G 100 16 24 25703 9.59 0.40% 2.80% 16.1

randwrite 8k 50G 100 16 28 25814 9.16 0.40% 2.60% 16.1

randwrite 8k 50G 100 16 32 25779 9.14 0.40% 2.60% 14.7

randwrite 8k 50G 100 20 4 25797 3.09 0.70% 3.40% 18.3

randwrite 8k 50G 100 20 8 25679 6.22 0.70% 3.20% 18.6

randwrite 8k 50G 100 20 12 25570 8.49 0.40% 2.90% 19.5

randwrite 8k 50G 100 20 16 25734 9.45 0.40% 2.50% 18

randwrite 8k 50G 100 20 20 25854 10 0.50% 2.70% 18

randwrite 8k 50G 100 20 24 25913 12.03 0.50% 2.80% 18.8

randwrite 8k 50G 100 20 28 25831 11.27 0.40% 2.60% 18.5

randwrite 8k 50G 100 20 32 25841 11.36 0.40% 2.70% 18

randwrite 8k 50G 100 24 4 25977 3.69 0.70% 3.60% 22

randwrite 8k 50G 100 24 8 25885 7.40 0.70% 3.60% 22.1

randwrite 8k 50G 100 24 12 25828 10.1 0.50% 3.20% 21.3

randwrite 8k 50G 100 24 16 25925 11.42 0.50% 2.80% 21.1

randwrite 8k 50G 100 24 20 25766 12 0.40% 2.70% 20.5

randwrite 8k 50G 100 24 24 25908 13.8 0.50% 2.70% 21.3

randwrite 8k 50G 100 24 28 26017 13.2 0.40% 2.50% 20.8

randwrite 8k 50G 100 24 32 26004 13.4 0.40% 2.60% 19.8

randwrite 8k 50G 100 28 4 25944 4.3 0.70% 3.80% 24.9

randwrite 8k 50G 100 28 8 25945 8.6 0.70% 3.70% 25.7

randwrite 8k 50G 100 28 12 25931 11.5 0.60% 3.10% 24.9

randwrite 8k 50G 100 28 16 25925 12.8 0.50% 3.00% 24.7

randwrite 8k 50G 100 28 20 26082 13.48 0.50% 4.80% 24.8

1 号店

第 18 页

randwrite 8k 50G 100 28 24 26043 15.68 0.50% 2.90% 24.4

randwrite 8k 50G 100 28 28 25980 15.2 0.40% 2.60% 20.8

randwrite 8k 50G 100 28 32 25942 16.27 0.50% 2.40% 22

randwrite 8k 50G 100 32 4 26081 4.89 0.70% 3.80% 27.7

randwrite 8k 50G 100 32 8 26107 9.79 0.70% 4.00% 28.4

randwrite 8k 50G 100 32 12 25906 13.22 0.60% 3.20% 29

randwrite 8k 50G 100 32 16 26072 14.62 0.50% 2.90% 28.7

randwrite 8k 50G 100 32 20 26099 15.74 0.50% 2.70% 26.6

randwrite 8k 50G 100 32 24 26104 16.7 0.50% 2.80% 28.3

randwrite 8k 50G 100 32 28 26191 17.77 0.40% 2.80% 26.2

randwrite 8k 50G 100 32 32 26086 19.3 0.50% 2.60% 25.5

randrw 8k 50G 100 4 4 10057 1.16 0.50% 2.20% 4.1

randrw 8k 50G 100 4 8 14020 1.79 0.50% 2.60% 3.7

randrw 8k 50G 100 4 12 16278 2.08 0.40% 2.40% 4.1

randrw 8k 50G 100 4 16 15158 2.57 0.40% 2.50% 3.8

randrw 8k 50G 100 4 20 15493 2.73 0.40% 2.30% 3.7

randrw 8k 50G 100 4 24 15764 2.97 0.40% 2.30% 3.8

randrw 8k 50G 100 4 28 15807 3.12 0.40% 2.30% 4.1

randrw 8k 50G 100 4 32 16209 3.27 0.40% 2.30% 4

randrw 8k 50G 100 8 4 13684 1.8 0.70% 3.20% 6.1

randrw 8k 50G 100 8 8 16704 3.27 0.90% 4.10% 7.9

randrw 8k 50G 100 8 12 19006 3.81 0.60% 3.40% 10.3

randrw 8k 50G 100 8 16 17486 4.73 0.60% 3.40% 10.2

randrw 8k 50G 100 8 20 17714 4.99 0.60% 3.20% 10.5

randrw 8k 50G 100 8 24 17807 5.24 0.60% 3.20% 8.5

randrw 8k 50G 100 8 28 18283 5.30 0.50% 2.90% 9.1

randrw 8k 50G 100 8 32 18676 5.51 0.50% 3.10% 8.7

randrw 8k 50G 100 12 4 16085 2.35 0.90% 4.20% 9.2

randrw 8k 50G 100 12 8 17736 4.81 1.00% 4.40% 9.1

randrw 8k 50G 100 12 12 22147 4.55 0.80% 4.40% 9.4

randrw 8k 50G 100 12 16 21548 5.58 0.70% 3.80% 9.8

randrw 8k 50G 100 12 20 22363 5.8 0.70% 3.70% 10.5

randrw 8k 50G 100 12 24 22645 5.79 0.70% 3.80% 9

randrw 8k 50G 100 12 28 23081 5.98 0.70% 3.80% 9.3

randrw 8k 50G 100 12 32 23202 6.09 0.70% 3.90% 10.1

randrw 8k 50G 100 16 4 17637 3.03 1.10% 4.80% 11.4

randrw 8k 50G 100 16 8 21274 5.42 1.20% 5.30% 11.2

randrw 8k 50G 100 16 12 22403 4.74 0.90% 4.80% 11.1

randrw 8k 50G 100 16 16 23102 5.78 0.90% 4.70% 11.4

randrw 8k 50G 100 16 20 23607 6.07 0.90% 4.60% 10.8

randrw 8k 50G 100 16 24 23370 6.11 0.90% 4.80% 11.6

randrw 8k 50G 100 16 28 23473 6.21 0.80% 4.90% 10.8

randrw 8k 50G 100 16 32 23678 6.31 0.80% 4.70% 11.8

1 号店

第 19 页

randrw 8k 50G 100 20 4 18207 5.4 1.10% 4.90% 12.5

randrw 8k 50G 100 20 8 22165 5.54 1.30% 5.90% 12.6

randrw 8k 50G 100 20 12 22437 4.88 0.80% 4.40% 13.3

randrw 8k 50G 100 20 16 22986 5.9 0.90% 4.60% 12.9

randrw 8k 50G 100 20 20 23720 6.21 0.90% 4.60% 12.8

randrw 8k 50G 100 20 24 23911 6.49 0.80% 4.80% 12.8

randrw 8k 50G 100 20 28 23822 6.48 0.90% 4.70% 12.6

randrw 8k 50G 100 20 32 23978 6.92 0.90% 4.80% 13

randrw 8k 50G 100 24 4 18778 4.54 1.20% 5.40% 14.3

randrw 8k 50G 100 24 8 22711 5.73 1.40% 6.50% 14.5

randrw 8k 50G 100 24 12 22546 4.9 1.00% 5.20% 14.8

randrw 8k 50G 100 24 16 23664 6.22 0.90% 5.10% 14.2

randrw 8k 50G 100 24 20 23999 6.79 0.90% 4.80% 14.1

randrw 8k 50G 100 24 24 23864 6.83 0.80% 4.90% 14.3

randrw 8k 50G 100 24 28 23759 7 0.90% 4.90% 14.5

randrw 8k 50G 100 24 32 23851 7.41 0.90% 5.10% 14.5

randrw 8k 50G 100 28 4 19583 5.1 1.30% 5.50% 15.6

randrw 8k 50G 100 28 8 22964 5.7 1.30% 6.30% 16.2

randrw 8k 50G 100 28 12 22520 5.14 1.10% 5.90% 15.9

randrw 8k 50G 100 28 16 23261 6.4 1.00% 5.30% 15.8

randrw 8k 50G 100 28 20 23662 6.82 0.90% 4.80% 15.7

randrw 8k 50G 100 28 24 24158 7.81 0.90% 5.10% 15.8

randrw 8k 50G 100 28 28 23849 8.05 0.90% 5.00% 15.4

randrw 8k 50G 100 28 32 24147 8.89 0.90% 4.60% 16

randrw 8k 50G 100 32 4 21787 5.33 1.40% 6.20% 17.4

randrw 8k 50G 100 32 8 23768 6 1.40% 6.60% 18.2

randrw 8k 50G 100 32 12 22622 5.7 0.90% 4.90% 17.4

randrw 8k 50G 100 32 16 23292 6.94 1.00% 5.30% 17.5

randrw 8k 50G 100 32 20 24063 8.14 1.00% 5.10% 17

randrw 8k 50G 100 32 24 23897 8.38 1.00% 5.00% 17.6

randrw 8k 50G 100 32 28 23766 8.96 0.90% 5.00% 17.2

randrw 8k 50G 100 32 32 24069 9.81 0.80% 4.90% 17.8

IOPS 趋势图

1 号店

第 20 页

图为 8k 随机读的 IOPS，线程和队列深度均为是 4~32，以 4 为单位的等差数列。其中横坐标

为线程，纵坐标为 IOPS，不同颜色的线代表不同深度，不同线程所对应的 IOPS。



Latency 趋势图


1 号店

第 21 页



4.1.3 16KB 数据块随机读，随机写，随机

读写测试测试命令：







测试结果：

测试 IO IO 大小文件

大小

runt

ime

线程

数

Iodep

th IOPS

延时


Iowai

t

randread 16k 50G 100 4 4 28006 0.57 0.40% 2.40% 4.1

randread 16k 50G 100 4 8 53454 0.60 0.60% 3.40% 2.2

randread 16k 50G 100 4 12 53142 0.82 0.40% 3.10% 2.6

1 号店

第 22 页

randread 16k 50G 100 4 16 71488 0.77 0.40% 3.40% 2.1

randread 16k 50G 100 4 20 67041 0.84 0.40% 3.20% 2.2

randread 16k 50G 100 4 24 74519 0.88 0.40% 3.40% 1.8

randread 16k 50G 100 4 28 80107 0.90 0.40% 3.60% 1.5

randread 16k 50G 100 4 32 84625 0.96 0.40% 3.90% 1.1

randread 16k 50G 100 8 4 52442 0.60 0.90% 5.20% 7.4

randread 16k 50G 100 8 8 87701 0.73 1.00% 6.50% 4.8

randread 16k 50G 100 8 12 83438 1.04 0.90% 6.50% 4.9

randread 16k 50G 100 8 16 89743 1.21 0.70% 6.10% 5.2

randread 16k 50G 100 8 20 89601 1.26 0.80% 6.20% 4.5

randread 16k 50G 100 8 24 89787 1.45 0.80% 6.60% 4.4

randread 16k 50G 100 8 28 89857 1.62 0.80% 6.40% 4.7

randread 16k 50G 100 8 32 89754 1.84 0.80% 6.50% 4.7

randread 16k 50G 100 12 4 73764 0.65 1.30% 7.30% 8.9

randread 16k 50G 100 12 8 89829 1.06 1.50% 8.20% 7.4

randread 16k 50G 100 12 12 89899 1.45 1.10% 7.50% 7.3

randread 16k 50G 100 12 16 90024 1.76 1.10% 7.20% 7.6

randread 16k 50G 100 12 20 89957 1.94 0.90% 7.00% 8.2

randread 16k 50G 100 12 24 90080 2.31 0.90% 7.10% 7.7

randread 16k 50G 100 12 28 89986 2.34 0.90% 6.50% 8.2

randread 16k 50G 100 12 32 90013 2.55 0.80% 6.60% 8.1

randread 16k 50G 100 16 4 87448 0.73 1.40% 8.00% 11

randread 16k 50G 100 16 8 90052 1.42 1.40% 8.00% 10.6

randread 16k 50G 100 16 12 90013 1.90 1.10% 7.20% 9.5

randread 16k 50G 100 16 16 90149 2.30 1.00% 7.00% 10.5

randread 16k 50G 100 16 20 90067 2.72 1.10% 7.60% 9.4

randread 16k 50G 100 16 24 90185 3.26 1.10% 7.40% 9.8

randread 16k 50G 100 16 28 90207 2.94 1.10% 7.00% 9.4

randread 16k 50G 100 16 32 90262 3.17 1.00% 6.90% 8.9

randread 16k 50G 100 20 4 89735 0.89 1.40% 9.00% 13.6

randread 16k 50G 100 20 8 90298 1.77 1.50% 8.80% 11.8

randread 16k 50G 100 20 12 90210 2.36 1.40% 8.10% 11

randread 16k 50G 100 20 16 90311 3.01 1.30% 7.70% 11.8

randread 16k 50G 100 20 20 90349 3.41 1.10% 7.40% 12

randread 16k 50G 100 20 24 90364 4.07 1.00% 6.60% 11.6

randread 16k 50G 100 20 28 90257 3.55 1.00% 7.00% 11.5

randread 16k 50G 100 20 32 90308 3.79 1.00% 6.80% 10.6

randread 16k 50G 100 24 4 90055 1.06 1.70% 9.30% 15.6

randread 16k 50G 100 24 8 90359 2.12 1.80% 9.70% 13.8

randread 16k 50G 100 24 12 90433 2.83 1.30% 8.00% 12.9

randread 16k 50G 100 24 16 90366 3.69 1.30% 8.20% 13.8

randread 16k 50G 100 24 20 90452 4.01 1.20% 7.50% 13.9

randread 16k 50G 100 24 24 90492 4.58 1.20% 7.30% 13.7

1 号店

第 23 页

randread 16k 50G 100 24 28 90518 4.24 1.00% 6.90% 13.3

randread 16k 50G 100 24 32 90462 4.43 1.00% 6.70% 12.9

randread 16k 50G 100 28 4 90295 1.23 2.00% 10.50% 17

randread 16k 50G 100 28 8 90542 2.47 1.90% 10.30% 16.3

randread 16k 50G 100 28 12 90560 3.29 1.50% 8.90% 15.6

randread 16k 50G 100 28 16 90584 4.18 1.40% 7.90% 16.4

randread 16k 50G 100 28 20 90629 4.49 1.20% 7.70% 15.9

randread 16k 50G 100 28 24 90684 5.03 1.20% 7.70% 16.5

randread 16k 50G 100 28 28 90495 5.02 1.10% 7.10% 15.3

randread 16k 50G 100 28 32 90532 5.23 1.00% 7.00% 15.2

randread 16k 50G 100 32 4 90309 1.41 2.20% 11.20% 19

randread 16k 50G 100 32 8 90509 2.82 2.10% 10.40% 18.5

randread 16k 50G 100 32 12 90742 3.76 1.60% 8.90% 18

randread 16k 50G 100 32 16 90726 4.59 1.40% 8.20% 18.6

randread 16k 50G 100 32 20 90659 4.98 1.30% 7.70% 18.6

randread 16k 50G 100 32 24 90532 5.48 1.10% 6.80% 17.8

randread 16k 50G 100 32 28 90731 5.71 0.90% 6.50% 18.9

randread 16k 50G 100 32 32 90667 5.89 1.10% 7.10% 18.3

randwrite 16k 50G 100 4 4 13697 1.16 0.30% 1.60% 5

randwrite 16k 50G 100 4 8 13698 2.11 0.40% 1.80% 5.8

randwrite 16k 50G 100 4 12 13659 3.13 0.20% 1.20% 6.2

randwrite 16k 50G 100 4 16 13890 3.90 0.30% 1.60% 6.9

randwrite 16k 50G 100 4 20 13793 5.01 0.30% 1.60% 7.2

randwrite 16k 50G 100 4 24 13941 4.31 0.20% 1.00% 6.7

randwrite 16k 50G 100 4 28 13665 4.45 0.20% 1.30% 6.4

randwrite 16k 50G 100 4 32 13726 4.76 0.20% 1.20% 6.3

randwrite 16k 50G 100 8 4 13789 2.31 0.40% 1.70% 9.1

randwrite 16k 50G 100 8 8 13819 4.62 0.30% 1.60% 13

randwrite 16k 50G 100 8 12 13590 6.19 0.30% 1.40% 12.1

randwrite 16k 50G 100 8 16 13812 6.93 0.20% 1.40% 13.7

randwrite 16k 50G 100 8 20 13876 7.49 0.20% 1.40% 13.6

randwrite 16k 50G 100 8 24 13876 8.63 0.20% 1.40% 14.1

randwrite 16k 50G 100 8 28 13880 8.49 0.30% 1.30% 10

randwrite 16k 50G 100 8 32 13844 8.74 0.20% 1.30% 14.2

randwrite 16k 50G 100 12 4 13916 3.44 0.40% 1.90% 13.7

randwrite 16k 50G 100 12 8 13888 6.90 0.40% 1.90% 14.8

randwrite 16k 50G 100 12 12 13880 9.36 0.30% 1.70% 15.1

randwrite 16k 50G 100 12 16 13907 10.43 0.30% 1.50% 13.3

randwrite 16k 50G 100 12 20 13899 11.11 0.30% 1.40% 14.2

randwrite 16k 50G 100 12 24 13850 13.13 0.30% 1.40% 14.7

randwrite 16k 50G 100 12 28 13907 12.41 0.20% 1.40% 15

randwrite 16k 50G 100 12 32 13906 12.54 0.20% 1.40% 13.8

randwrite 16k 50G 100 16 4 13973 4.57 0.40% 2.00% 16

1 号店

第 24 页

randwrite 16k 50G 100 16 8 13974 9.15 0.40% 1.80% 16.2

randwrite 16k 50G 100 16 12 13952 12.43 0.30% 1.80% 16.5

randwrite 16k 50G 100 16 16 13981 14.18 0.30% 1.60% 16.1

randwrite 16k 50G 100 16 20 13913 14.43 0.30% 1.40% 15.5

randwrite 16k 50G 100 16 24 13878 18.66 0.30% 1.40% 16.2

randwrite 16k 50G 100 16 28 13983 16.63 0.30% 1.40% 14.8

randwrite 16k 50G 100 16 32 13991 17.61 0.30% 1.40% 15.9

randwrite 16k 50G 100 20 4 14008 5.70 0.40% 2.10% 17.8

randwrite 16k 50G 100 20 8 13997 11.42 0.50% 2.00% 19

randwrite 16k 50G 100 20 12 13940 15.44 0.40% 1.80% 17.7

randwrite 16k 50G 100 20 16 14051 17.23 0.30% 1.60% 17.4

randwrite 16k 50G 100 20 20 14029 18.84 0.30% 1.40% 16.8

randwrite 16k 50G 100 20 24 13997 22.39 0.30% 1.40% 17.1

randwrite 16k 50G 100 20 28 14007 20.42 0.30% 1.30% 15.9

randwrite 16k 50G 100 20 32 13998 20.59 0.30% 1.30% 17

randwrite 16k 50G 100 24 4 14038 6.83 0.40% 2.20% 19.7

randwrite 16k 50G 100 24 8 14030 13.67 0.40% 1.90% 19.2

randwrite 16k 50G 100 24 12 14027 18.57 0.40% 1.80% 20.4

randwrite 16k 50G 100 24 16 14015 20.39 0.30% 1.60% 19.9

randwrite 16k 50G 100 24 20 14032 22.42 0.30% 1.40% 18.3

randwrite 16k 50G 100 24 24 14030 24.95 0.30% 1.40% 19.9

randwrite 16k 50G 100 24 28 14039 26.41 0.30% 1.30% 18.4

randwrite 16k 50G 100 24 32 14080 22.94 0.30% 1.30% 17.9

randwrite 16k 50G 100 28 4 14021 7.98 0.50% 2.10% 22.4

randwrite 16k 50G 100 28 8 14053 15.93 0.50% 2.20% 22.3

randwrite 16k 50G 100 28 12 13976 21.00 0.40% 1.70% 22.6

randwrite 16k 50G 100 28 16 14045 23.42 0.30% 1.50% 21.6

randwrite 16k 50G 100 28 20 14078 27.31 0.30% 1.40% 20.1

randwrite 16k 50G 100 28 24 14102 29.31 0.30% 1.50% 22.2

randwrite 16k 50G 100 28 28 14063 27.91 0.30% 1.30% 18.9

randwrite 16k 50G 100 28 32 14056 31.24 0.30% 1.40% 20.4

randwrite 16k 50G 100 32 4 14108 9.06 0.50% 2.10% 24.8

randwrite 16k 50G 100 32 8 14108 18.13 0.40% 1.90% 21.8

randwrite 16k 50G 100 32 12 14091 24.02 0.40% 1.60% 24.9

randwrite 16k 50G 100 32 16 14103 26.67 0.40% 1.60% 22

randwrite 16k 50G 100 32 20 14023 28.61 0.30% 1.50% 23.7

randwrite 16k 50G 100 32 24 14090 30.10 0.30% 1.50% 23.3

randwrite 16k 50G 100 32 28 14073 31.71 0.30% 1.40% 22.6

randwrite 16k 50G 100 32 32 14057 35.45 0.30% 1.30% 22.7

randrw 16k 50G 100 4 4 7357 1.72 0.40% 1.90% 4

randrw 16k 50G 100 4 8 9675 2.78 0.50% 2.20% 3.5

randrw 16k 50G 100 4 12 10878 3.32 0.40% 1.90% 5.1

randrw 16k 50G 100 4 16 10310 4.01 0.30% 1.80% 4.4

1 号店

第 25 页

randrw 16k 50G 100 4 20 10422 4.26 0.40% 1.80% 4.4

randrw 16k 50G 100 4 24 10343 4.56 0.30% 1.90% 4.2

randrw 16k 50G 100 4 28 10403 4.76 0.30% 1.60% 5

randrw 16k 50G 100 4 32 10600 4.95 0.30% 1.70% 4.2

randrw 16k 50G 100 8 4 10155 2.63 0.70% 2.90% 7.1

randrw 16k 50G 100 8 8 11027 5.34 0.70% 3.00% 7.9

randrw 16k 50G 100 8 12 13597 5.12 0.50% 2.30% 9.2

randrw 16k 50G 100 8 16 12765 6.09 0.50% 2.90% 9.6

randrw 16k 50G 100 8 20 12833 6.09 0.50% 2.70% 10.6

randrw 16k 50G 100 8 24 12865 6.13 0.40% 2.10% 12.4

randrw 16k 50G 100 8 28 12922 6.20 0.40% 2.10% 8.5

randrw 16k 50G 100 8 32 13004 6.31 0.40% 2.10% 10.4

randrw 16k 50G 100 12 4 10971 3.89 0.80% 3.30% 9.8

randrw 16k 50G 100 12 8 12810 5.62 0.80% 3.60% 10.8

randrw 16k 50G 100 12 12 13713 4.92 0.50% 2.40% 10.9

randrw 16k 50G 100 12 16 13007 6.13 0.50% 2.60% 11.4

randrw 16k 50G 100 12 20 13002 6.36 0.50% 2.40% 10.5

randrw 16k 50G 100 12 24 12962 6.35 0.40% 2.50% 10.7

randrw 16k 50G 100 12 28 13070 6.51 0.40% 2.30% 9.7

randrw 16k 50G 100 12 32 13108 6.79 0.50% 2.40% 11.7

randrw 16k 50G 100 16 4 11251 5.24 0.90% 3.40% 11.7

randrw 16k 50G 100 16 8 12865 6.05 0.80% 3.40% 11.3

randrw 16k 50G 100 16 12 13869 5.16 0.50% 2.70% 12.2

randrw 16k 50G 100 16 16 13118 6.55 0.50% 2.80% 13

randrw 16k 50G 100 16 20 13111 6.87 0.50% 2.50% 12

randrw 16k 50G 100 16 24 13103 7.05 0.50% 2.70% 11.8

randrw 16k 50G 100 16 28 13095 7.23 0.50% 2.50% 11.7

randrw 16k 50G 100 16 32 13141 7.43 0.50% 2.60% 11.7

randrw 16k 50G 100 20 4 12661 5.73 0.90% 4.00% 13.3

randrw 16k 50G 100 20 8 13140 6.28 0.90% 3.80% 13.6

randrw 16k 50G 100 20 12 13892 5.33 0.70% 3.40% 13.8

randrw 16k 50G 100 20 16 13147 7.17 0.60% 2.90% 14

randrw 16k 50G 100 20 20 13157 7.31 0.60% 2.80% 14.1

randrw 16k 50G 100 20 24 13146 7.63 0.50% 2.80% 14.2

randrw 16k 50G 100 20 28 13109 7.76 0.50% 2.70% 14.2

randrw 16k 50G 100 20 32 13194 8.09 0.50% 2.60% 13

randrw 16k 50G 100 24 4 12983 5.91 0.90% 3.70% 14.8

randrw 16k 50G 100 24 8 13124 6.57 0.90% 3.90% 14.9

randrw 16k 50G 100 24 12 13931 5.49 0.70% 3.60% 15

randrw 16k 50G 100 24 16 13193 7.60 0.60% 2.90% 15.1

randrw 16k 50G 100 24 20 13244 8.01 0.60% 2.80% 14.8

randrw 16k 50G 100 24 24 13193 8.67 0.60% 2.90% 14.8

randrw 16k 50G 100 24 28 13164 9.61 0.50% 2.70% 15

1 号店

第 26 页

randrw 16k 50G 100 24 32 13279 10.11 0.50% 2.60% 14.4

randrw 16k 50G 100 28 4 13173 5.97 0.90% 4.20% 15.9

randrw 16k 50G 100 28 8 13210 6.82 0.90% 3.90% 16.3

randrw 16k 50G 100 28 12 14032 5.88 0.80% 3.70% 16.4

randrw 16k 50G 100 28 16 13269 10.01 0.60% 2.90% 17.2

randrw 16k 50G 100 28 20 13231 11.21 0.70% 2.90% 17.3

randrw 16k 50G 100 28 24 13257 12.15 0.60% 2.80% 17.2

randrw 16k 50G 100 28 28 13281 12.92 0.60% 2.70% 17.8

randrw 16k 50G 100 28 32 13235 9.75 0.60% 2.80% 16.7

randrw 16k 50G 100 32 4 13112 6.02 0.90% 4.10% 17.5

randrw 16k 50G 100 32 8 13284 7.32 0.90% 4.20% 17.7

randrw 16k 50G 100 32 12 13966 6.61 0.80% 3.50% 17.2

randrw 16k 50G 100 32 16 13227 9.54 0.60% 2.90% 17.4

randrw 16k 50G 100 32 20 13292 11.01 0.70% 2.80% 17

randrw 16k 50G 100 32 24 13222 12.06 0.60% 2.70% 17.4

randrw 16k 50G 100 32 28 13268 13.25 0.60% 2.70% 17

randrw 16k 50G 100 32 32 13264 13.98 0.50% 2.70% 18.3

IOPS 趋势图

图为 16k 随机读的 IOPS，线程和队列深度均为是 4~32，以 4 为单位的等差数列。其中横坐

标为线程，纵坐标为 IOPS，不同颜色的线代表不同深度，不同线程所对应的 IOPS。


1 号店

第 27 页


Latency 趋势图



1 号店

第 28 页


4.2 双控制器下 2 个 LUN 性能测试

使用一台 DELL 815 服务器通过 FIO 对 EF540 的 2 个 1TB 的 LUN 进行压力测

试, 实际性能压力体现为 EF540 两个存储控制器的性能。

测试命令：



-filename=/dev/dm-0: /dev/dm-1

其中 numjobs 为 8、16、24，iodepth 为 8，16，bs 为 4k，8k。

测试结果：

测试 IO IO大

小

文件

大小

runti

me

线程

数

Iodep

th IOPS 延时(MS) USR CPU SYS CPU

Iowai

t

randread 4k 50G 100 8 8 123002 0.52 0.80% 8.80% 0.1

randread 4k 50G 100 8 16 112112 1.08 0.40% 8.00% 0.8

randread 4k 50G 100 16 8 62631 2.04 0.60% 6.80% 12.9

randread 4k 50G 100 16 16 66048 3.35 0.40% 6.30% 15.8

randread 4k 50G 100 24 8 60668 3.16 0.60% 8.80% 17.1

randread 4k 50G 100 24 16 63491 5.20 0.40% 8.60% 19.1

randwrite 4k 50G 100 8 8 46354 1.37 0.60% 4.20% 8

randwrite 4k 50G 100 8 16 41851 2.56 0.50% 5.30% 4.8

randwrite 4k 50G 100 16 8 41825 3.05 0.50% 7.30% 12.5

randwrite 4k 50G 100 16 16 38003 5.60 0.40% 8.10% 12.3

randwrite 4k 50G 100 24 8 44956 4.26 0.50% 13.30% 17.9

randwrite 4k 50G 100 24 16 41892 8.00 0.40% 9.80% 19.2

randrw 4k 50G 100 8 8 24980 1.84 0.80% 5.10% 6

randrw 4k 50G 100 8 16 28053 2.47 0.50% 3.80% 7.2

1 号店

第 29 页

randrw 4k 50G 100 16 8 24409 3.24 0.70% 10.50% 9.1

randrw 4k 50G 100 16 16 25828 4.63 0.50% 4.20% 11.6

randrw 4k 50G 100 24 8 23410 4.69 0.70% 13.20% 16.4

randrw 4k 50G 100 24 16 25272 7.15 0.50% 4.70% 15.9

randread 8k 50G 100 8 8 96976 0.66 0.80% 7.00% 3.7

randread 8k 50G 100 8 16 103148 1.14 0.90% 8.20% 1.6

randread 8k 50G 100 16 8 56584 2.26 0.70% 7.00% 13.3

randread 8k 50G 100 16 16 62287 3.39 0.40% 5.90% 15.3

randread 8k 50G 100 24 8 57496 3.33 0.60% 9.10% 17.7

randread 8k 50G 100 24 16 60856 5.39 0.50% 9.40% 20

randwrite 8k 50G 100 8 8 24623 2.59 0.50% 3.30% 9.9

randwrite 8k 50G 100 8 16 23568 4.22 0.30% 2.80% 10.6

randwrite 8k 50G 100 16 8 24621 5.20 0.50% 5.30% 14.2

randwrite 8k 50G 100 16 16 22977 8.91 3.00% 3.50% 13.5

randwrite 8k 50G 100 24 8 25036 7.66 0.50% 8.90% 18.7

randwrite 8k 50G 100 24 16 23736 14.00 0.30% 5.60% 22.4

randrw 8k 50G 100 8 8 15582 3.31 0.70% 4.20% 8.1

randrw 8k 50G 100 8 16 17365 4.23 0.50% 2.80% 7.6

randrw 8k 50G 100 16 8 15793 5.55 0.50% 5.30% 14.2

randrw 8k 50G 100 16 16 16731 7.37 0.40% 1.80% 11.9

randrw 8k 50G 100 24 8 15966 7.51 0.80% 6.60% 15.7

randrw 8k 50G 100 24 16 16693 11.20 0.40% 2.00% 14.8

IOPS 趋势图

图为 4k 随机读、随机写、随机读写的 IOPS。

1 号店

第 30 页

图为 8k 随机读、随机写、随机读写的 IOPS。

Latency 趋势图

图为 4k 随机读、随机写、随机读写的延时。

图为 8k 随机读、随机写、随机读写的延时。

4.3 EF540 全闪存阵列实时采样

1 号店

第 31 页

图为 4k 8 线程 8 队列深度随机读的 IOPS 和延时


1 号店

第 32 页


1 号店

第 33 页



1 号店

第 34 页


5 性能分析

通过对各项测试指标的观察我们发现，在 16 个 lun 的压力测试下，4K 的数据块在 4 线

程和 8 线程随机读的时候 iowait 最低，平均只有 1%左右，Iops 达到 18 万左右，syscpu 则在

10%以下。测试表明 iodepth 对 iowait 的影响不大，线程数对 Iowait 有较大影响。

1 号店

第 35 页

如图所示，区间 1 的 Iowait 在 2%以下，区间 2 的 syscpu 均在 15%以下，所以区间 3 即

为理论可用值范围，小块的 4k 随机读 iops 最高可以达到 18 万左右。其他线程的 iops 虽然可

以达到 25 万的高负载，但是平均 iowait 在 10%左右，对生产系统性能会略有影响，所以实

际测试可用值如图区间三所示。8k 的 iops 随机读实际测试可用值在 16 万左右，16k 的 iops

为 8 万左右。

区间 1

区间 2

区间 3

Iodepth

Iodepth

Iodepth

4k 随机读

4k 随机读

4k 随机读

1 号店

第 36 页

在 2 个 1TB 的 lun 压力测试下，4k 的数据块在 8 线程随机读的时候 Iowait 最低，不到

1%，iops为 11万左右，syscpu在 8%左右，比同期的 16个 lun的压力测试 iops低，是因为

多个 LUN 更有利于 NetApp 的并发处理，获得更高性能，或者，NetApp 的设计面向共享的存

储设施，单个 LUN 难以使系统饱和。

如下图所示：

4k 随机读

4k 随机读

4k 随机读

Iodepth

Iodepth

Iodepth

netapp ef540 ssd storage test

Technology