Toward Stable Operationof Genkai/Hyeonhae/APII link

Yoshinori Kitatsuji

APAN Tokyo XPKDDI R&D Laboratories, Inc.

Overview• Test before starting service

– Checking Availability

• Staff in Service– Routes– Link Utility– Bandwidth availability

• Troubleshoot– Tools– status to public

• Future arrangement– Resource management

• Scheduling– Project– Maintenance

• Making requirement clear– Protocol, Path, Traffic bandwidth, Schedule

Test before starting service

• Understanding availability is much important– Spec. written in Catalogue doesn't’ match with actual performance

sometime.– Combination of gears usually perform like the greatest common measure.– It’s comfortable for both users and operators to use networks if the

limitation is know well.• It helps scheduling upgrade network and enhancing the performance.

• Performance known at the moment– Meinohama-Busan

• 930Mbit/s (IP packet, 1000 byte length)– Meinohama-Tokyo

• 250Mbit/s (UDP/TCP), because shaper (271Mbit/s) applied• 430Mbit/s (TCP over a loop Tokyo-Tenjin-Tokyo)

– Should be known how much improved• between QGPOP and Seoul• between Seoul and Tokyo

Genkai Network Configuration at L3

SuperSINET

Busan XP

fidc-juniper

hakozaki-juniper

Genkai XP (Meinohama)

Kyushu U

hakozaki-cisco2

APII-Juniper

ATM/OC12

ATM/OC3

GbE

GbE

Cisco 　 GSR tpr2

GbE

203.181.249.168/29

169

170

203.181.249.160/29

161

162

GbE

Tokyo XP (KDDI Otemachi)

203.181.249.104/30

105

106

203.181.249.184/29

203.181.249.176/29

177

178

AS 9270

AS 7660

185

AS 2523GbE

133.69.152.0/26

13

15

6

133.69.164.0/28

203.181.248.192/30

193

194

tpr3

AS 7660

AS 2523

AS 2907

Presented in NOC meeting, APAN Fukuoka Meeting

250Mbit/s250Mbit/s

430Mbit/s430Mbit/s

APANTokyo

XP

JGNATM

network

JGNATM

network

CRL

CRL

PC

PC

APANTokyo

XP

CRL

APII Juniper

Orenge: ATM SwitchBlue: Ethernet SwitchOrenge: ATM SwitchBlue: Ethernet Switch

GigabitEthernet

OC12

OC12

OC12

OC12

OC12

OC12

OC12

Configuration of Loopback test

Staff prepared in service

• Making acceptable usage policy clear– Simple and Opened AUP helps for operators to manage trouble.

• Monitoring in APAN Tokyo XP– Traffic, Number of routes

• MRTG, RRDTool, OCXmon/DAG– Reachability

• Ping and report for major targets every 5minutes– Performance

• SLAC’s performance monitor using iperf

• Opening operational information to public– Assignment information

• IP address, ATM PVC, VLAN ID, Port utility of equips– Monitoring result

MRTG

• Multi Router Traffic Grapher– Most common tool for operation– Plots 2 data source against time– Use SNMP to collect counters of interfaces– 5min average and 2 data sources in a figure

• Advantages– Ability to create scripts to feed data into MRTG– Mature code because of very wide deployment in large networks– Any variable can be graphed– Easy to understand configuration

• Disadvantages– Can’t create custom graph periods– Only keeps 5 minute data for 2.5 days, after witch it is aggregated to longer period.– Too hard to hack to modify higher late than 5 minutes

RRDTool

• Round Robin Database Tool– Successor of MRTG– Ability to customize graphs across user-defined interval– Constant file size independs on the number of feeding time because of binary file format

• Advantages– 25-30 percent faster than MRTG– Ability to have multiple data sources in a single graph.– Ability to generate graph from multiple data files.

• Disadvantages– No collector– Doesn’t create graph and web-pages without additional scripts– Hard to merge multiple files into one.

ATM　Switch

CoralREEF/DAG• TCPDUMP for ATM/POS

– Packet capturing system– POS OC48 is under development

• Unknown how much perform actually– CoralREEF

• Successor of OC3mon developed by MCI & NLANR• run with Fore PCI200 or Point ATM card

– DAG• developed by Wikato University in New Zealand • run with DAG card developed by Wikato University

• Advantages– Header or whole packets are captured– Ability to convert original format to TCPDUMP (pcap

library)

• Disadvantages– Run on only FreeBSD or Linux– Splitter divide the laser and power degrades to half– Need multiple PCs to capture a high speed link.

• May need GPS clock than NTP to get high accuracy– Need interface to generating graphs continuously like

MRTG

Router

Traffic since …Hyeonhea/Genkai Link

TransPAC link to Seattle

Here operation started

Here Genkailink operation started

SLAC performance monitor

• TCP performance monitor─ Generate TCP traffic for 1-2 minutes.─ Measure RTT and bandwidth of

iperf, BBCPMEM, BBFTP─ http://perf3-fe.jp.apan.net/~cottrell/html/slac_wan_bw_tests.html

• Advantage− requires only ssh and iperf− multiple targets are measured round robin every two hours automatically

• Disadvantage─ Generated traffic effects regular traffic.─ Takes some time (2-3minutes) to measure.

APAN Tokyo XP

QGPOPAIST

Stanford Univ

Here operation started

Open Operation

• Opening assignment policy and operation status– Reducing query– Make troubleshoot easier

• Sharing account between neighbor networks– Consider to cause the lack of security,

but fruitful.– Ability to understand neighbor networks deeply– Account holder do troubleshoot by themselves

• Opening status of trouble– Users and neighbor networks feel relieved.– May have help from outside

Troubleshoot

• Tool– ping/traceroute for unicast– mtrace, multicast beacon and debug output for multicast– MRTG or more accurate graph

• Sharing information– Maintenance report

• schedule, status, and after maintenance• Traceable status board

– Make any information opened as much as possible• Neighbor may do troubleshoot

Multicast Beacon

• Every beacon sends and receives periodical packets to some group and all of them make reports the statistics to MB server.– Developed by NLANR

• http://dast.nlanr.net/Projects/Beacon/

– Popular in AccessGrid

• MB server in XP may helps to debug the multicast trouble for especially seeking where problem lie.

Future operation

• Resource Scheduling will become important– Now we can generate 100s Mbit/s speed traffic between Korea and

Europe through Japan and USA– More important to schedule events than sharing resources over events– Multiple Traffic easily break each other.

• Will take more time to support the Events– Demonstration will requires the high performance

• Ex. AIST tried to record high speed TCP transmission test with 2 TransPAC links in parallel in SC2002.

– Need to Collect detail of requirement – Regular traffic increase gradually

• Installing Tools and Devices– XPs are required to equips tools and devices to support all kind of services

run by projects