Gungwon Kang & Jiwoong Kim (KISTI)

LIGO Data Grid and KISTI

June 27, 2015 at 8th J-K Joint Workshop on KAGRA, Gwangju in Korea

2

OUTLINE

I. KISTI GSDC Overview

II. KISTI LDG (LIGO Data Grid)

III. Conclusion

3

• National research institute for information technology since 1962• About 600 people working for Supercomputing & Networking and

National Information Service (development & analysis)

• Running High-Performance Computing Facility• Total 3,398 Nodes (30,592 CPUs, 360 TFlops at peak), 1,667 TB

storage (introduced from 2008)

KISTI: Korea Institute of Science and Technology Information

Intel Xeon X5570 2.93GHz (Nehalem)

Rpeak: 300TFCPU: 25,408Memory: 76.8TBStorage: 1,061TB

I. KISTI GSDC Overview

4

GSDC: Global Science experiment Data hub Center

• National project to promote data-based science research experiments by pro-viding computing and storage resources: HEP and other fields

• Running Data-Intensive Computing Facility• ~20 Staffs: system administration, experiment support, external-relations, administra-

tion and students• CPU: ~5,900 cores. Storage: ~6.8PB• Budget: ~ 6M$/year

• Supporting experiments: ALICE, CMS, Belle, LIGO, RENO, Genomic Medicine, etc.

GSDC Facility

HP Servers

Hitachi VSP Storage

National Institute of Supercomputing and Networking 2013. 10 . 28.

CPU (cores)

2014.12

Field / Experiment

Particle/Nu-clear Physics

CERN (ALICE) 2,520 ~2,000

KiAF 108 100

CERN (T3) 240 110

KEK 224 110

BNL (STAR) 540 110

RENO 280 252

HCP 346 180

Astrophysics LIGO 420 152

Medical Sci-ence

Genom 144 150

G-brain 208 52

Meteorology PCMI 0 380

Etc. 870 126

Total 5,900 3,722

5

Storage (TB)

Resource allocations

6

Model Physical size Usable size

NetApp FAS2050

(SAN only, RAID6)104TB 50TB

NetApp FAS6080

(SAN & NAS, RAID6)334TB 200TB

Hitachi USP-V

(SAN & NAS, RAID6)960TB 600TB

EMC CX4-960C(SAN, RAID6) 1,920TB 1,250TB

EMC Isilon 108NL 1,620TB 1,400TB

Hitach VSP 1

(SAN, NAS, RAID6)

2013 758TB 500TB

2014 320TB 214TB

Hitach VSP 2

(SAN, NAS, RAID6)857TB 570TB

Total 6,873TB 4,784TB

GSDC CPU/Storage(2014.12.30)

구 분 내역

Model Hitach VSP, HNAS4080(4Node)

Disk Usable 700TB

RAID RAID6(6D+2P)

Cache Mem-

ory512GB

Front-end In-

terface8Gbps FC 32 ports

구 분 세 부 사 항 수 량

Com-

puting

Server

(Pur-

chase

d in

2014)

☐ 모델명 : HP DL360G8(1U)

☐ 사양 :

- E5-2680v2 2.8GHz

* 2P(20core)

- 128GB DDR3 1600Mhz SDRAM

- 600GB 10K SAS * 4EA

- 1GbE 2Port, 10GbE 2Port

- 8G HBA

- Redundant Power Supply

1,100

Core

(10-

Core

x 55

Nodes

)

7

II. LIGO Data Grid (LDG)

+ 8KISTI LDG T3

Result AnalysisDiscussion

User Authentication based on GSI

Storage

155 TBs

Tier1/2/3

LIGO / VIRGO Data

576 cores (780)wn3076~3110.sdfarm.kr (Condor)

Cluster

Login Node Web Server

KGWG F2F Meeting 8

ui04.sdfarm.kr (Condor)

ldas.ligo.kisti.re.kr

lgm.sdfarm.kr (Intel-Compiler-License)

ce04.sdfarm.kr (Condor)

ldr.sdfarm.kr(GridFTP server)

9

ui04.sdfarm.kr{Condor}

wn3076~3110.sdfarm.kr{Condor}

Central storage150TB

ldr.ligo.caltech.edu

….

ldr.sdfarm.kr{GridFTP server}

ldas.ligo.kisti.re.kr{web publication}

Connection/Job Submission

lgm.sdfarm.kr{Intel-Compiler-License}

ui test{Condor}

• System Configuration in more detail:

+KISTI LDG Resources (2015) 10

KGWG F2F Meeting

• Computation Resources(Worker Node)• 48 Node : 780cores (Hyperthread-17node) / 576 cores (Physical)

• Storage Resources(Only Data - /data/ligo/archive)• 155 TB (Expandable to 200 TB)

Size Used Avail Use (%)

/data/ligo/home 786GB 471GB 315GB 60

/data/ligo/lib 100GB 71GB 20GB 71

/data/ligo/scratch 4.73TB 3.96TB 784G 84

/data/ligo/archive

150TB 123TB 28TB 82

Total 155TB 127TB 29TB 8210

Cores RAM

Worker Node 780 / 576 72GB(Hyperthread)48GB(W/O

Hyperthread)

UI,CE,LGM,LDAS,LDR

60(12core per server) 24GB

Total 840 / 636

+Stored Data

- hoft frame: LIGO S5~6. Virgo VSR1~4

- RDS L1 frame: LIGO S6

11

+KISTI LDG Usage: 12

KGWG F2F Meeting12

기간 1 월 2 월 3 월 4 월 5 월 6 월 7 월 8 월 9 월 10 월 11 월 12 월 합계

2011

건수( 컴퓨팅 시간 , 일 )

124(81)

362(309)

80(171)

70(726)

44(500)

30(4)

187(193)

128(159)

1,025(2,143

)

2012

62 55 22 10 19 30 30 1 0 0 ? ? 229

2013

1,287 1,951 927 268 1 9 5,430 1,129 26 ? ? ?11,02

82014

331 918 921 703 367 881 1214 545 120 124 ? ? 6,124

2015

9,335 21,683 4,257 109 1 1435,40

0

• Work Node Usage: CPU 384 cores - 2014.11~2015.02: Max 78.9%, Ave 13% - 2015.01~2015.02: Max 60%, Ave 35% (※ Monitored by KISTI)

* Job 건수 만이 아닌 새로운 모니터링 Measure 필요 ( 예 , 2011 년 6 월 , 8 월 비교 )

13

• KGWG Korean Gravitational Wave Group (2008~): • ~30 people working in 8 universities and 3 government-funded institutes

• LIGO-Virgo and KAGRA

User groups (1)

KGWG

한국중력파연구협력단

소속 이름서울대 이형목 (PI) NIMS 오정근연세대 김정리 오상훈한양대 이현규 손재주

김경민 김환선이철훈 추형석

서강대 조규만 KISTI 강궁원부산대 이창환 장행진

김영민 김지웅김명국 윤희준

인제대 이형원 조희석김정초 KAERI 차용호

고려대 윤태현 GIST 강훈수조동현 경북대 박명구

명지대 김재완 군산대 김상표

+

Parameter estimation:

Chunglee Kim(Yonsei U/ KISTI GSDC), Hyungwon Lee, Chungcho Kim (Inje U)

+ LSC collaborators (Caltech, NU, UWM, Monclair State Univ.)

+ KAGRA collaborators (Osaka University)

User groups (2)14

CBC Signal and Noise Identification:

오정근 , 오상훈 , 손재주 , 김환선 , 추형석 (NIMS), 이창환 , 김영민 ( 부산대 )

※ KISTI 연구원 : 강궁원 , 장행진 , 김지웅 , 윤희준 , 조희석 (KISTI-GSDC)

iDQ pipeline 의 개선 (Deep Learning) 및 인공신경망 모듈 이식 Bank chisq 를 이용한 detection statistic 의 개선연구 중력파 채널과 보조채널간의 Correlation Analysis (CAGMon) HHT 를 이용한 Trigger generation 연구

- Mostly used by domestic researchers!

중력파 분석을 통한 모수추정 (parameter estimation) 보다 천체물리학적으로 “실제와 가깝고” , 계산적으로 효율적인 중력파형 개발

+User groups (3)

이름 소속 e-mail 계정

Kazuhiro

Hayama

Osaka

Universitykazuhiro.hayama@gmail.com …

Tatsusya

Narikawa

Osaka

Universitynarikawat@vega.ess.sci.osaka-u.ac.jp …

Hideyuki

Tagoshi

Osaka

Universitytagoshi@vega.ess.sci.osaka-u.ac.jp …

Koh UenoOsaka

Universityueno@vega.ess.sci.osaka-u.ac.jp …

Hirotaka

Yuzurihara

Osaka

University

yuzurihara@yukimura.hep.osaka-

cu.ac.jp…

+이름 접속기록

Kazuhiro Hayama -

Tatsusya Narikawa

(588 분 )

narikawa pts/7 :pts/8:S.0 Wed Jun 3 16:49 - 16:51 (00:01)

narikawa pts/8 pascal.hep.osaka Wed Jun 3 16:49 - 16:51 (00:01)

narikawa pts/7 :pts/6:S.0 Thu May 21 12:12 - 12:43 (00:30)

narikawa pts/7 :pts/6:S.0 Thu May 21 11:20 - 11:51 (00:30)

narikawa pts/6 pascal.hep.osaka Thu May 21 11:20 - 12:45 (01:24)

narikawa pts/7 :pts/6:S.0 Thu May 21 10:52 - 10:52 (00:00)

Tue May 12 18:07 - 18:11 (00:04)

narikawa pts/18 :pts/14:S.0 Tue May 12 17:48 - 17:57 (00:09)

narikawa pts/14 pascal.hep.osaka Tue May 12 17:48 - 17:57 (00:09)

narikawa pts/19 :pts/16:S.0 Mon Apr 27 17:29 - 18:02 (00:32)

narikawa pts/19 :pts/16:S.0 Mon Apr 27 15:50 - 16:54 (01:04)

narikawa pts/16 pascal.hep.osaka Mon Apr 27 15:50 - 18:02 (02:11)

narikawa pts/19 :pts/16:S.0 Mon Apr 27 14:59 - 14:59 (00:00)

narikawa pts/19 :pts/16:S.0 Mon Apr 27 12:38 - 13:10 (00:31)

narikawa pts/16 pascal.hep.osaka Mon Apr 27 12:38 - 14:59 (02:20)

narikawa pts/5 :pts/1:S.0 Thu Apr 23 17:59 - 18:04 (00:05)

narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:59 - 18:04 (00:05)

narikawa pts/5 :pts/1:S.0 Thu Apr 23 17:56 - 17:56 (00:00)

narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:55 - 17:56 (00:00)

narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:54 - 17:55 (00:00)

+이름 접속기록

Kazuhiro Hayama -

Tatsusya

Narikawa

(588 분 )

narikawa pts/7 :pts/8:S.0 Wed Jun 3 16:49 - 16:51 (00:01)

narikawa pts/8 pascal.hep.osaka Wed Jun 3 16:49 - 16:51 (00:01)

…

narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:55 - 17:56 (00:00)

narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:54 - 17:55 (00:00)

Hideyuki Tagoshi -

Koh Ueno

(250 분 )

ueno pts/19 :pts/18:S.0 Sat May 2 19:37 - 20:21 (00:43)

ueno pts/18 pascal.hep.osaka Sat May 2 19:37 - 20:21 (00:43)

…

ueno pts/8 pascal.hep.osaka Tue Apr 21 21:00 - 21:07 (00:06)

Hirotaka

Yuzurihara

(28 분 )

yuzu pts/19 :pts/3:S.0 Tue Jun 9 06:09 - 06:09 (00:00)

yuzu pts/3 pascal.hep.osaka Tue Jun 9 06:09 - 06:09 (00:00)

…

yuzu pts/24 pascal.hep.osaka Mon Apr 27 17:39 - 17:48 (00:09)

18

• LDG central monitoring system• Ganglia installed on all KISTI

resources• ce04.sdfarm.kr, a condor server,

gathers ganglia information of LDG WNs.• 8649 port is open to the watchtower.

• 129.89.57.50(watchtower.phys.uwm.edu)

LDG WatchTower

19

• OS: Scientific Linux 6.1 ( ?)• Batch system for managing compute jobs: Condor-7.8.7 ( ?)• LIGO Data Grid : 5.2.2 https://www.lsc-group.phys.uwm.edu/daswg/download/repositories.html• More than 200 packages

Soft Wares Deployed

Packages Description

FrameL for data_frame manipulation MetaIO for LIGO_LW files metadata manipulation LAL Suite Related Project: PYLAL

LIGO Algorithm Library [LAL] + LAL based Applications [LALApps]

GLUE Grid LSC User Environment FrameCPP (deprecated) LDAS-TOOLS

C++ interface to access frame structures

GDS LIGO Global Diagnostics System NDS2-Client Related Project: PYNDS

Part of DMT offline that allow the user to down-load LIGO data from the V2 LIGO Network Data Servers.

Matapps-utilitites A collection of MATLAB® based applications for LIGO data analysis

LVAlert LIGO/Virgo Alert Tools GRACEdb Gravitational Wave Candidate Event Database LARS LIGO Archival Service LIGO-common Simple setup of Python ligo namespace

GSTLAL GSTLAL provides a self-contained suite of GStreamer elements (and dependencies) that

expose gravitational-wave data analysis tools from the LAL library for use in GStreamer signal-processing pipelines.

GST-Plugins A collection of scientific visualization plugins for GStreamer using Cairo-powered graphics

Mathematical operations plugins for GStreamer Low-Latency Ligo low-latency data distribution server initialization

VOEvent VOEvent is the standardized language used to report observation and for describing observations of immediate astronomical events

20

III. Conclusion• We have briefly introduced computing resources, en-

vironments and operation status of the KISTI GSDC LDG T3 center.

• We hope to develop a good collaboration in the com-puting and data management of KAGRA in the future.

21

THANK YOU감사 ( 感謝 ) 합니다