Experiences with 100Gbps Network Applications

Analyzing Data Movements and Identifying Techniques for

Next-generation High-bandwidth Networks

Mehmet Balman Computational Research Division

Lawrence Berkeley National Laboratory

Outline Climate Data as a typical science scenario

• 100Gbps Climate100 demo MemzNet: Memory-mapped zero-copy Network Channel Advance Network Initiative (ANI) Testbed

• 100Gbps Experiments

Future Research Plans

100Gbps networking has finally arrived! Applications’ Perspective Increasing the bandwidth is not sufficient by itself; we need careful evaluation of high-bandwidth networks from the applications’ perspective.

1Gbps to 10Gbps transition (10 years ago)

Application did not run 10 times faster because there was more bandwidth available

The need for 100Gbps Modern science is Data driven and Collaborative in nature • The largest collaborations are most likely to

depend on distributed architectures. • LHC (distributed architecture) data generation,

distribution, and analysis. • The volume of data produced by of genomic

sequencers is rising exponentially. •  In climate science, researchers must analyze

observational and simulation data located at facilities around the world

ANI 100Gbps Demo

•  100Gbps demo by ESnet and Internet2

•  Application design issues and

host tuning strategies to scale to 100Gbps rates

•  Visualization of remotely located data (Cosmology)

•  Data movement of large datasets with

many files (Climate analysis)

Late 2011 – early 2012

ESG (Earth Systems Grid) • Over 2,700 sites • 25,000 users

•  IPCC Fifth Assessment Report (AR5) 2PB

•  IPCC Forth Assessment Report (AR4) 35TB

Data distribution for climate science How scientific data movement and analysis between geographically disparate supercomputing facilities can benefit from high-bandwidth networks?

•  Local copies •  data files are copied

into temporary storage in HPC centers for post-processing and further climate analysis.

Climate Data Analysis Petabytes of data •  Data is usually in a remote location • How can we access efficiently?

Tuning Network Transfers

Adaptive Tuning

Worked well with large files !

Climate Data over 100Gbps • Data volume in climate applications is increasing

exponentially. • An important challenge in managing ever increasing data

sizes in climate science is the large variance in file sizes. •  Climate simulation data consists of a mix of relatively small and

large files with irregular file size distribution in each dataset.

• Many small files

lots-of-small-files problem!

file-centric tools?

FTP RPC

request a file

request a file

send file

send file

request data

send data

•  Concurrent transfers •  Parallel streams

Moving climate files efficiently

MemzNet: memory-mapped zero-copy network channel

network

Front-­‐end  threads  

(access  to  memory  blocks)

Front-­‐end  threads  (access  to  memory  blocks)

Memory  blocks Memory  

blocks

memory caches are logically mapped between client and server

Advantages •  Decoupling I/O and network operations

•  front-end (I/O processing) •  back-end (networking layer)

•  Not limited by the characteristics of the file sizes

•  On the fly tar approach, bundling and sending many files together

•  Dynamic data channel management Can increase/decrease the parallelism level both in the network communication and I/O read/write operations, without closing and reopening the data channel connection (as is done in regular FTP variants).

Advantages The synchronization of the memory cache is accomplished based on the tag header. •  Application processes interact with the memory

blocks. •  Enables out-of-order and asynchronous send

receive MemzNet is is not file-centric. Bookkeeping information is embedded inside each block.

•  Can increase/decrease the number of parallel streams without closing and reopening the data channel.

100Gbps demo environment

RRT: Seattle – NERSC 16ms NERSC – ANL 50ms NERSC – ORNL 64ms

100Gbps Demo • CMIP3 data (35TB) from the GPFS filesystem at NERSC

• Block size 4MB • Each block’s data section was aligned according to the

system pagesize. •  1GB cache both at the client and the server

• At NERSC, 8 front-end threads on each host for reading

data files in parallel. •  At ANL/ORNL, 4 front-end threads for processing

received data blocks. •  4 parallel TCP streams (four back-end threads) were

used for each host-to-host connection.

83Gbps throughput

Framework for the Memory-mapped Network Channel

memory caches are logically mapped between client and server

MemzNet’s Features Data files are aggregated and divided into simple blocks. Blocks are tagged and streamed over the network. Each data block’s tag includes information about the content inside. Decouples disk and network IO operations; so, read/write threads can work independently. Implements a memory cache managements system that is accessed in blocks. These memory blocks are logically mapped to the memory cache that resides in the remote site.

Performance at SC11 demo

TCP buffer size is set to 50MB

MemzNet GridFTP

Advance Network Initiative

Esnets’s 100Gbps Testbed

Performance results in the 100Gbps ANI testbed

MemzNet’s Performance

TCP buffer size is set to 50MB

MemzNet GridFTP

100Gbps demo

ANI Testbed

Challenge? • High-bandwidth brings new challenges!

•  We need substantial amount of processing power and involvement of multiple cores to fill a 40Gbps or 100Gbps network

•  Fine-tuning, both in network and application layers, to take advantage of the higher network capacity.

•  Incremental improvement in current tools? •  We cannot expect every application to tune and improve every time

we change the link technology or speed.

MemzNet • MemzNet: Memory-mapped Network Channel

•  High-performance data movement

MemzNet is an initial effort to put a new layer between the application and the transport layer.

• Main goal is to define a network channel so applications can directly use it without the burden of managing/tuning the network communication.

Framework for the Memory-mapped Network Channel

memory caches are logically mapped between client and server

Advance Network Initiatieve

Esnets’s 100Gbps Testbed

Initial Tests ²  Many TCP sockets oversubscribe the network

and cause performance degradation.

²  Host system performance could easily be the bottleneck.

•  TCP/UDP buffer tuning, using jumbo frames, and

interrupt coalescing.

• Multi-core systems: IRQ binding is now essential for maximizing host performance.

Many Concurrent Streams

(a) total throughput vs. the number of concurrent memory-to-memory transfers, (b) interface traffic, packages per second (blue) and bytes per second, over a single NIC with different number of concurrent transfers. Three hosts, each with 4 available NICs, and a total of 10 10Gbps NIC pairs were used to saturate the 100Gbps pipe in the ANI Testbed. 10 data movement jobs, each corresponding to a NIC pair, at source and destination started simultaneously. Each peak represents a different test; 1, 2, 4, 8, 16, 32, 64 concurrent streams per job were initiated for 5min intervals (e.g. when concurrency level is 4, there are 40 streams in total).

ANI testbed 100Gbps (10x10NICs, three hosts): Interrupts/CPU vs the number of concurrent transfers [1, 2, 4, 8, 16, 32 64 concurrent jobs - 5min intervals], TCP buffer size is 50M

Effects of many concurrent streams

Parallel Streams - 10Gbps

ANI testbed 10Gbps connection: Interface traffic vs the number of parallel streams [1, 2, 4, 8, 16, 32 64 streams -5min intervals], TCP buffer size is set to 50MB

Parallel Streams – 40Gbps

ANI testbed 40Gbps (4x10NICs, single host): Interface traffic vs the number of parallel streams [1, 2, 4, 8, 16, 32 64 streams - 5min intervals], TCP buffer size is set to 50MB

Parallel streams - 40Gbps

Figure 5: ANI testbed 40Gbps (4x10NICs, single host): Throughput vs the number of parallel streams [1, 2, 4, 8, 16, 32 64 streams - 5min intervals], TCP buffer size is set to 50MB

Host tuning With proper tuning, we achieved 98Gbps using only 3 sending hosts, 3 receiving hosts, 10 10GE NICS, and 10 TCP flows

Image source: “Experiences with 100Gbps network applications” In Proceedings of the fifth international workshop on Data-Intensive Distributed Computing, 2012

NIC/TCP Tuning •  We are using Myricom 10G NIC (100Gbps testbed)

•  Download latest drive/firmware from vendor site •  Version of driver in RHEL/CentOS fairly old

•  Enable MSI-X •  Increase txgueuelen /sbin/ifconfig eth2 txqueuelen 10000!

•  Increase Interrupt coalescence!/usr/sbin/ethtool -C eth2 rx-usecs 100!

•  TCP Tuning: net.core.rmem_max = 67108864!net.core.wmem_max = 67108864!net.core.netdev_max_backlog = 250000

From “Experiences with 100Gbps network applications” In Proceedings of the fifth international workshop on Data-Intensive Distributed Computing, 2012

100Gbps = It’s full of frames ! •  Problem:

•  Interrupts are very expensive •  Even with jumbo frames and driver optimization, there is still

too many interrupts.

•  Solution: •  Turn off Linux irqbalance (chkconfig irqbalance off) •  Use /proc/interrupt to get the list of interrupts •  Dedicate an entire processor core for each 10G interface •  Use /proc/irq/<irq-number>/smp_affinity to bind rx/tx queues to

a specific core.

From “Experiences with 100Gbps network applications” In Proceedings of the fifth international workshop on Data-Intensive Distributed Computing, 2012

Tuning Setting without tuning with tuning % improvementInterrupt coalescing (TCP) 24 36.8 53.3333333

Interrupt coalescing (UDP) 21.1 38.8 83.8862559

IRQ Binding (TCP) 30.6 36.8 20.2614379

IRQ Binding (UDP) 27.9 38.5 37.9928315

0

5

10

15

20

25

30

35

40

45

Interrupt coalescing

(TCP)

Interrupt coalescing

(UDP)

IRQ Binding (TCP)

IRQ Binding (UDP)

Gbp

s

Host Tuning Results

without tuning

with tuning

Image source: “Experiences with 100Gbps network applications” In Proceedings of the fifth international workshop on Data-Intensive Distributed Computing, 2012

Initial Tests ²  Many TCP sockets oversubscribe the network

and cause performance degradation.

²  Host system performance could easily be the bottleneck.

• TCP/UDP buffer tuning, using jumbo frames, and

interrupt coalescing. • Multi-core systems: IRQ binding is now essential for

maximizing host performance.

End-to-end Data Movement • Tuning parameters

• Host performance

• Multiple streams on the host systems

• Multiple NICs and multiple cores

• Effect of the application design

MemzNet’s Architecture for data streaming

MemzNet • MemzNet: Memory-mapped Network Channel

•  High-performance data movement

MemzNet is an initial effort to put a new layer between the application and the transport layer.

• Main goal is to define a network channel so applications can directly use it without the burden of managing/tuning the network communication.

Framework for the Memory-mapped Network Channel

memory caches are logically mapped between client and server

Related Work • Luigi Rizzo ’s netmap

• proposes a new API to send/receive data over the network

• http://info.iet.unipi.it/~luigi/netmap/

• RDMA programming model • MemzNet can be used for RDMA-enabled transfers

• memory-management middleware

• GridFTP popen + reordering ?

Future Work

•  Integrate MemzNet with scientific data formats for remote data analysis

• Provide a user library for filtering and subsetting of data.

Testbed Results http://www.es.net/RandD/100g-testbed/

•  Proposal process •  Testbed Description, etc.

•  http://www.es.net/RandD/100g-testbed/results/

•  RoCE (RDMA over Ethernet) tests •  RDMA implementation (RFPT100) •  100Gbps demo paper •  Efficient Data Transfer Protocols •  MemzNet (Memory-mapped Zero-copy Network Channel)

Acknowledgements Collaborators: Eric Pouyoul, Yushu Yao, E. Wes Bethel Burlen Loring, Prabhat, John Shalf, Alex Sim, Arie Shoshani, and Brian L. Tierney

Special Thanks: Peter Nugent, Zarija Lukic , Patrick Dorn, Evangelos Chaniotakis, John Christman, Chin Guok, Chris Tracy, Lauren Rotman, Jason Lee, Shane Canon, Tina Declerck, Cary Whitney, Ed Holohan, Adam Scovel, Linda Winkler, Jason Hill, Doug Fuller, Susan Hicks, Hank Childs, Mark Howison, Aaron Thomas, John Dugan, Gopal Vaswani

NDM 2013 Workshop (tentative)

The 3rd International Workshop on Network-aware Data Management

to be held in conjunction with the EEE/ACM International

Conference for High Performance Computing, Networking, Storage and

Analysis (SC’13)

http://sdm.lbl.gov/ndm

Editing a book (CRC press): NDM research frontiers (tentative)