network test at chep
Post on 22-Jan-2018
329 Views
Preview:
TRANSCRIPT
Network Tests at CHEPNetwork Tests at CHEP
K. Kwon, D. Han, K. Cho, J.S. Suh, D. SonCenter for High Energy Physics, KNU, Korea
H. ParkSupercomputing Center, KISTI, Korea
Representing the HEP Working Group for ANF/the HEP Data Grid WG
The 3rd International Workshop on HEP Data Grid August 26, 2004, Daegu, Korea
2
IntroductionIntroduction Network Tests using Iperf
Domestic tests International tests (USA, Europe)
Real File Transfer Tests using bbFTP Domestic tests International tests (Europe)
Summary & Future Works
3
HEP Data GridHEP Data Grid Implementation of the Tier-1 Regional Data Center of LHC
Networking Tier0 (CERN)–Tier1 (CHEP) : ~2.5Gbps via TEIN Tier1(CHEP)–Tier1(US and Japan): ~Gbps via APII Tier1(CHEP), Tier2 or 3(inside Korea): 0.155~1 Gbps via
KOREN/KREONET
Computing(1000 CPU Linux clusters)
Data Storage Capability Storage: 1.1 PB Raid Type Disk (Tier1+Tier2) Tape Drive: ~ 3.2 PB HPSS Servers
4
Network at CHEP & Available Research Network at CHEP & Available Research NetworksNetworks
Servers/PCs……
Servers
JapanAPII/H-G
1 G
2.5G
Cisco7606
1 G
Kreonet
CHEP
Clustered PCs ……
APII/KREONET22 * 622 M
USA
TEIN34 M
KORENCERN/Geneva
…100 M
1 G
HSM
TransPAC 2.5G
KNU CC
1 G
Cisco6509
DataTaG 2.5G
Network Test PC
5
Test Tools Test Tools Iperf
A tool for measuring TCP and bandwidth performance. Data sent by default from the client’s memory to the server memory
bbFTP File transfer software that is optimized for large files.
Supports multi-stream transfer and big windows
TCP Reno: Linux 2.4.26 TCP protocol
6
Factors affecting TCP Factors affecting TCP performanceperformance
Window Size
Number of Streams
MTU – We have not tried yet.
Txqueuelen – No gain in performance
SACK – No gain in performance
7
CHEP CHEP KOREN-NOC Iperf TestKOREN-NOC Iperf TestKNU- > KOREN- NOC (Single Stream)
0100200300400500600700800900
1000
256 KB 512 KB 1 MB 2 MB 3 MB 4 MBTCP Window Size
Ach
ieva
ble
ban
dw
idth
in M
bp
s
KNU->KOREN- NOC (5 Parallel TCP Streams)
0
50
100
150
200
250
A B C D ETCP Stream
Achie
va
ble
Ba
nd
wid
th in M
bp
s
RTT: ~2ms
BDP=0.002*1000Mbps=0.25MB
Max throughput: 920Mbps
Throughput of five streams: 916Mbps
KORENKNU
KOREN-NOC1 G
CHEP1 G
GigabitEthernet3/1 -- KOREN
8
Single Stream Tests Single Stream Tests between CHEP between CHEP Caltech Caltech
Duration: 10min each, 10min interval, over KOREN-TransPAC path (1Gbps), 20MB window, TCP(Linux 2.4.26)
KNU- > Caltech(June 23)
0
50
100
150
200
1 2 3 4 5No. of Trial
Ban
dwid
th(M
bps)
KNU Genkai LA CaltechBusan
1 G 2.5G1 G
KOREN H-G transPAC CalREN2
Tokyo1G 1 G
RTT: ~130ms
BDP: 0.13*1000Mbps=16MB
Max throughput: 146Mbps
9
Multi-Stream Tests between Multi-Stream Tests between CHEP CHEP Caltech Caltech
CHEP- >Caltech (June 23)
0
200
400
600
800
1000
10 20 30 40 50 60 70 80 90 100No. of Streams
Band
wid
th(M
bps)
Duration: 10min each over KOREN-TransPAC path (1Gbps),
(Stream* Window ) <=100MB
APII/Genkai Link (APII-Juniper ge-0/1/0.1)
TransPAC LA link (TPR2 so-2/0/0.0)
Max throughput: 783Mbps
10
Single Stream Tests between Single Stream Tests between CHEP CHEP CERN CERN
KNU- >CERN(Single Stream)
020406080
100120
1 2 3 4 5No. of Trials
Thro
ughp
ut(M
bps)
KNU Genkai LA ChicagoBusan
1 G 2.5G1 G
KOREN H-G transPAC Abilene
Tokyo1G 1 G
CERN2.5 G
DataTag
Duration: 10min each, 10min interval, over KOREN-TransPAC path
40MB window, TCP(Linux 2.4.26)
RTT: ~370ms
BDP: 0.37*1000Mbps=46MB
Max throughput: 99Mbs
11
Multi-Stream Tests between Multi-Stream Tests between CHEP CHEP CERN CERN
CHEP- >CERN(Multistream)
0100200300400500600700800
10 20 30 40 50 60 70 80 90 100No. of Streams
Thro
ughp
ut(M
bps)
APII/Genkai Link (APII-Juniper ge-0/1/0.1)
TransPAC LA link (TPR2 so-2/0/0.0)
Duration: 10min each over KOREN-TransPAC path (1Gbps),
(Stream* Window ) <=100MB
Max throughput:714Mbps
12
Other TCP StacksOther TCP Stacks
CHEP->CERN(Single Stream)
050
100150200250300350
1 2 3 4 5No. of Trials
Throu
ghpu
t(Mbp
s) HS-TCPTCP RenoFAST
Setup for HS-TCP:net.ipv4.tcp_rmem= 4096 87380 67108864net.ipv4.tcp_wmem= 4096 87380 67108864net.ipv4.tcp_mem= 8388608 8388608 67108864txqueuelen =1000
Setup for FAST TCP:net.ipv4.tcp_rmem= 4096 33554422 134217728net.ipv4.tcp_wmem= 4096 33554422 134217728net.ipv4.tcp_mem= 4096 33554422 134217728txqueuelen =1000
CHEP- >Caltech(Single Stream)
050
100150200250300350400450
1 2 3 4 5No. of Trials
Thro
ughp
ut (
Mbp
s)
HS- TCPTCP RenoFAST
13
Real File Transfer between Real File Transfer between KNU & KOREN over Linux TCP KNU & KOREN over Linux TCP [root@cluster90 bbftpc]# ./bbftp -V -e 'setrecvwinsize 1024; setsendwinsize 1024; put ams' -u root 203.255.252.26
Password:
>> USER root PASS
<< bbftpd version 3.0.2 : OK
>> COMMAND : setremotecos 0
<< OK : COS set
>> COMMAND : setrecvwinsize 1024
<< OK
>> COMMAND : setsendwinsize 1024
<< OK
>> COMMAND : put ams ams
<< OK
1024000000 bytes send in 19.8 secs (5.06e+04 Kbytes/sec or 395 Mbits/s)
14
I/O test run rulesI/O test run rules The maximum file size to be greater than the
total physical memory to get accurate results (Iozone file system benchmark)
Perform 40X physical RAM size worth of IO to minimize the percentage of error due to IO being read out of cache(3ware white paper)
15
Real File Transfer (100GB) between Real File Transfer (100GB) between KNU & KORENKNU & KOREN
KOREN(2.5G)
AMD Opteron Dual, Tuned RAID 0, (Read 197MB/s with Iozone, Write 178MB/s)
Xeon 2GHz Dual, ATA DISK Drive at KOREN-NOC, Daejon
1G 1G
Time Taken: 1 hour 20 min 58
Average throughput: 164Mbps
16
Real File Transfer(1TB) between Real File Transfer(1TB) between KNU & KORENKNU & KOREN
KOREN
KNU File Server
(Tuned RAID 0, Read 197MB/s with Iozone, Write 178MB/s)
machines at KOREN-NOC in
Daejon
1G
1G
1G
1G
1G
1G
200GB each
A: 3 hr. 13 min. 20
B: 3 hr. 13 min. 31
C: 3 hr. 14 min. 36
D: 3 hr. 14 min. 46
E: 3 hr. 11 min. 22
E
D
C
A
B
Throughput: 701Mbps
17
File Transfer (100GB)with LustreFile Transfer (100GB)with Lustre
Lustre(Linux + Cluster): Distributed file system for large clusters
OST
OST
OST
OST
OSTClient
MDS
GigE
100GB
1G
KNU File Server
Time Take: 52min 55
Throughput: 251Mbps
18
Real File Transfer between KNU Real File Transfer between KNU & CERN over HS-TCP& CERN over HS-TCP
[kihwan@w01gva bbftpc]$ ./bbftp -V -e 'setrecvwinsize 41024; setsendwinsize 41024; cd /d/Bandwidth/BBftp/bbftp-3.0.2/bbftpd; get ams' -u root cluster90.knu.ac.krPassword: >> USER root PASS<< bbftpd version 3.0.2 : OK>> COMMAND : setremotecos 0<< OK : COS set>> COMMAND : setrecvwinsize 41024<< OK>> COMMAND : setsendwinsize 41024<< OK>> COMMAND : cd /d/Bandwidth/BBftp/bbftp-3.0.2/bbftpd<< OK : Current remote directory is /d/Bandwidth/BBftp/bbftp-3.0.2/bbftpd>> COMMAND : get ams ams<< OK1024000000 bytes got in 47.7 secs (2.1e+04 Kbytes/sec or 164 Mbits/s)
SummarySummary High Bandwidth Network is essential for HEP Data Grid
Domestic Links Only Window size need to be adjusted to fully utilize available
bandwidth Real file transfer shows that speed is limited by physical I/O,
rather than network
International Links Single stream: ~100Mbps Parallel streams needed to achieve significant throughput Other TCP Stacks may help improve performance Further tests and investigations should be done
20
Future WorksFuture Works
Jumbo Frame (9000Byte MTU)
File transfer using RAID disks between KNU and CERN
Tests over Lambda Networks
top related