site report: tokyo tomoaki nakamura icepp, the university of tokyo 2014/12/10tomoaki nakamura1

Site report: Tokyo

Tomoaki Nakamura

ICEPP, The University of Tokyo

2014/12/10 Tomoaki Nakamura 1

Update from the last year


No HW upgrade from the last year for Grid resources- 2560 CPU cores (18.03 HS06/core)- RAM (2GB/core for 1280CPU, 4GB/core for 1280CPU)- No memory upgrade until the end of 2015 (considered at last year)- 2000PB for pledged Disk (2014) and ~600TB for LocalGroupDisk

All service instance have been migrated to EMI3- CREAM, DPM, BDII (site/top), Arugus, gLexec-WN, APEL - WMS, LB, MyProxy: can be decommissioned for ATLAS

The other service instance- perfSONAR (latency 1G, bandwidth 1G, bandwidth 10G)- Squid (condDB x 2 + CVMFS x 2)

Services for ATLAS have been deployed- DPM-WebdDAV: used for Rucio renaming, will be used for central deletion- DPM-XrootD and FAX setup: connected with Asia redirector- Multi core queuex: 512 cores, 20% of resources, 64 static 8-core slots

FAX remote access


4TB / day = ~46 MB / sec

ASAP (all data)


(ATLAS Site Availability Performance)

99.77%

Pledge for the next year and beyond


2013 2014 2015

CPU pledge

16000 [HS06]

20000 [HS06] 24000

CPU deployed

43673.6 [HS06-SL5] (2560core)

46156.8 [HS06-SL6] (2560core)

-

Disk pledge 1600 [TB] 2000 [TB] 2400 [TB]

Disk deployed 2000 [TB] 2000 [TB] -

For FY2015- Increase 400TB to pledge- 528TB (8 servers) will be added to DPM by the end of Mar. 2015- Total DPM capacity: 3168TB (~750TB for LocalGroupDisk)

End of 2015 - End of this system- Procurement work will start from the next spring- If we can get 6TB HDD, total storage capacity can be doubled at 4th system

International network for Tokyo


TOKYO

ASGC

BNL

TRIUMF

NDGF

RALCCIN2P3CERNCANFPIC

SARANIKEF

LA

PacificAtlantic

10Gbps

10Gbps

WIX

New line (10Gbps)since May. 2013

OSAKA

40Gbps

10x3 Gbps

10x3 Gbps

10 Gbps

Amsterdam

Geneva

Dedicated line

Frankfurt

Configuration for the LHCONE evaluation


MLXe32 (10G)

Dell8024 (10G)

Dell 5448 (1G)

Catalyst 6500 (10G)

Catalyst 3750 (10G)

NY

DC

LA

Dell8024 (10G)

UI (Gridftp)

perfSONAR(Latency)

perfSONAR(Bandwidth)

perfSONAR (Latency/Bandwidth)

UI (Gridftp)

ICEPP (production)157.82.112.0/21

UTnet SINET

IPv4/v6

LHCONE BGP peering

ICEPP (LHCONE evaluation)157.82.118.0/24

10Gbps 1Gbps

Stability on packet loss (CC-IN2P3)


Directly affect to transfer rate.

Fraction of packet loss (NY vs. DC)


Comparable each other.

Minimum latency (CC-IN2P3)


Useful to know the typical latency and stability.

Minimum latency (CC-IN2P3)


Originating from other group in Univ. of Tokyo.

Distribution of Minimum latency (CC-IN2P3)


Distribution of Minimum latency (CC-IN2P3)


originating from other group.miss measurement.

Maximum latency (CC-IN2P3)


Useful to find problems.

Maximum latency (CC-IN2P3)


Also have spikes.Additional periodic noise.

Distribution of Maximum latency (CC-IN2P3)


Distribution of Maximum latency (CC-IN2P3)


Discrepancy due to the periodic noise.

Also for the other sites


(US)

(FR)

• One of the perfsonar instance in Tokyo seems to fall into the busy state once in a day.

• It is independent of source sites.

• But, no significant errors in system and service logs.

Maximum latency (masked by time)


Periodic nose can be cleaned up.

Maximum latency by mask (CC-IN2P3)


Still remaining, but comparable.

Bandwidth measurement (CC-IN2P3 and CNAF)


Asymmetric~38 MB/s (incoming)~28 MB/s (outgoing)

Symmetric, but unstable ~34 MB/s (incoming)~35 MB/s (outgoing)

Minimum latency (CC-IN2P3 in 2014)


Minimum latency (CC-IN2P3 in 2014)


Spikes were gone.

Average value is split.

Latency in one day (CC-IN2P3)


Both production line via NY

Incoming

Outgoing

Load balancing somewhere in NY or GEANT?

Maximum latency (CC-IN2P3, 2014)


Some improvement in FR-Geneva?

Bandwidth measurement (latest data)


Still asymmetric~35 MB/s (incoming)~24 MB/s (outgoing)

Symmetric, and very stable ~32 MB/s (incoming)~30 MB/s (outgoing)

Configuration for the LHCONE evaluation


MLXe32 (10G)

Dell8024 (10G)

Dell 5448 (1G)

Catalyst 6500 (10G)

Catalyst 3750 (10G)

NY

DC

LA

Dell8024 (10G)

UI (Gridftp)

perfSONAR(Latency)



UI (Gridftp)


UTnet SINET

IPv4/v6

LHCONE BGP peering


10Gbps 1Gbps

LHCONE (EU sites) for all production servers


MLXe32 (10G)

Dell8024 (10G)

Dell 5448 (1G)

Catalyst 6500 (10G)

Catalyst 3750 (10G)

NY

DC

LA

Dell8024 (10G)

UI (Gridftp)

perfSONAR(Latency)



UI (Gridftp)


UTnet SINET

IPv4/v6

LHCONE BGP peering


10Gbps 1Gbps

Nov. 11, 2014 (latency for CCIN2P3)


Nov. 11, 2014 (latency for CNAF)


Nov. 11 (throughput for CCIN2P3)


Nov. 11 (throughput for CNAF)


Dec. 7, 2014 (incoming B.W. is saturated)


User subscription of AOD via DaTri physics.Egampa, 8TeV all period: ~150TB

Still on going today (continuously several days)

Breakdown from GridFTP log


Part of LHCONE contribution

Mainly FTS3 and direct transfer from multiple sites

10 min. bin

1 min. bin

Near future and Concerns


LHCONE- Next for US and Canada- And then, for Asisa (ASGC, IHEP)

Network Bandwidth- 2015: more 10G from ICEPP to SINET? UTokyo is offering, but depends on

them.- JFY2016: SINET will be upgraded (SINET5)

• 100G for US (LA)• 20G for EU (reverse around)

EMI3- End of full support April 30, 2014- End of standard update October 31, 2014- End of security update April 30, 2015

Batch job systemTroque/Maui, no more support, not effective dynamic multi-core allocationHTCondor, SLURM or the other commercial product (UNIVA GE, LSF)

site report: tokyo tomoaki nakamura icepp, the university of tokyo 2014/12/10tomoaki nakamura1

Documents