lagopus sdn/openflow switch: yet another sdn/of switch agent

35
0 Copyright©2014 NTT corp. All Rights Reserved. High-performance vSwitch of the user, by the user, for the user A bird in cloud Yoshihiro Nakajima, Wataru Ishida, Tomonori Fujita, Takahashi Hirokazu, Tomoya Hibi, Hitoshi Matsutahi, Katsuhiro Shimano NTT Labs

Upload: phamtu

Post on 04-Feb-2017

225 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

0 Copyright©2014 NTT corp. All Rights Reserved.

High-performance vSwitch

of the user, by the user, for the user

A bird in cloud

Yoshihiro Nakajima, Wataru Ishida, Tomonori Fujita, Takahashi Hirokazu, Tomoya Hibi, Hitoshi Matsutahi, Katsuhiro Shimano

NTT Labs

Page 2: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

1 Copyright©2014 NTT corp. All Rights Reserved.

Agenda

vSwitch of the user Background and motivation

vSwitch by the user High-performance software-switch

• Concept

• Design

• Implementation & techniques

• Performance Evaluation

vSwitch for the user PoC of carrier usecase

• Segment Routing or Service chaining (Tentative)

Open source community

Page 3: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

2 Copyright©2014 NTT corp. All Rights Reserved.

Our mission

Open innovation

with

open source

for networking

Page 4: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

3 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch of the user

• Background

• Motivation

Page 5: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

4 Copyright©2014 NTT corp. All Rights Reserved.

Trend shift in networking

Closed (Vender lock-in)

Yearly dev cycle

Waterfall dev

Standardization

Protocol

Special purpose HW / appliance

Distributed cntrl

Custom ASIC / FPGA

Open (lock-in free)

Monthly dev cycle

Agile dev

DE fact standard

API

Commodity HW/ Server

Logically centralized cntrl

Merchant Chip

Page 6: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

5 Copyright©2014 NTT corp. All Rights Reserved.

What is Software-Defined Networking? 5

Innovate services and applications in software development speed.

Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf

Decouple control plane and data plane →Free control plane out of the box (OpenFlow is one of APIs)

Logically centralized view →Hide and abstract complexity of networks, provide entire view of the network

Programmability via abstraction layer →Enables flexible and rapid service/application development

SDN conceptual model

Page 7: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

6 Copyright©2014 NTT corp. All Rights Reserved.

OpenFlow vs conventional NW node

OpenFlow controller

OpenFlow switch

Data-plane

Flow Table

Control plane (Routing /

swithching)

Data-plane (ASIC, FPGA)

Control plane (routing/ switching)

Flow match action

Flow match action counter

OpenFlow Protocol

Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #)

Action (frame processing) (output port, drop, modify/pop/push header)

Flow statistics (# of packet, byte size, flow duration,… )

counter

Conventional NW node OpenFlow

Flow Table #2

Flow Table #3

Page 8: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

7 Copyright©2014 NTT corp. All Rights Reserved.

7

Evaluate the benefits of SDN by implementing

our control plane and switch

Page 9: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

8 Copyright©2014 NTT corp. All Rights Reserved.

Motivation of vSwitch

Agile and flexible networking Full automation in provisioning, operation and

management

Seamless networking for customers

Server virtualization and NFV needs a high-performance software switch Short latency

Wire-rate in case of short packet (64B)

No High-performance OpenFlow 1.3 software switch for wide-area networks 1M flow rules

10Gbps-wire-rate

Multiple table supports

Page 10: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

9 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch requirement from user side

1. Run on the commodity PC server and NIC

2. Provide a gateway function to allow connect different various network domains Support of packet frame type in DC, IP-VPN, MPLS and

access NW

3. Achieve 10Gbps-wire rate with >= 1M flow rules low-latency packet processing flexible flow lookup using multiple-tables High performance flow rule setup/delete

4. Run in userland and decrease tight-dependency to OS kernel easy software upgrade and deployment

5. Support various management and configuration protocols.

Page 11: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

10 Copyright©2014 NTT corp. All Rights Reserved.

Why Intel x86 server for networking?

Strong processing power Many CPU cores

• 8 cores/CPU (2012), 12 cores/CPU (2013),…

Rapid development cycle

Available everywhere Can be purchased in Akihabara!

3 month lead time in case of special purpose HW

Reasonable price and flexible configuration 8 core CPU (ATOM) with 1,000 USD

Can chose configuration according to your budget

Many programmers and engineers Porting to a NPU needs special skill and knowledge

Page 12: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

11 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch by the user

• Concept

• Design

• Implementation & techniques

• Performance Evaluation

Page 13: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

12 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch by the user

Concept

Page 14: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

13 Copyright©2014 NTT corp. All Rights Reserved.

Target of Lagopus vSwitch

High performance soft-based OpenFlow switch 10Gbps wire-rate packet processing / port

1M flow rules

Expands the SDN apps to wide-area networks Not only for data centers

WAN protocols, e.g. MPLS and PBB

Various management /configuration interfaces

Open Innovation Community-based development

Page 15: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

14 Copyright©2014 NTT corp. All Rights Reserved.

Approach for vSwitch development

Simple is better for everything Packet processing

Protocol handling

Straight forward approach No OpenFlow 1.0 support

Full scratch (No use of existing vSwitch code)

User land packet processing as much as

possible, keep kernel code small Kernel module update is hard for operation

Every component & algorithm can be replaced Flow lookup, library, protocol handling

Page 16: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

15 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch design

Simple modular-based design Switch control agent

Data-plane

Switch control agent Unified configuration data

store

Multiple management IF

Data-plane control with HAL

Data-plane High performance I/O

library (Intel DPDK)

Raw socket for test

Page 17: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

16 Copyright©2014 NTT corp. All Rights Reserved.

Implementation strategy for vSwitch

Massive RX interrupts handling for NIC device

=> Polling-based packet receiving

Heavy overhead of task switch

=> Thread assignment (one thread/one physical CPU)

Lower performance of PCI-Express I/O and memory bandwidth compared with CPU

=> Reduction of # of access in I/O and memory

Shared data access is bottleneck between threads

=> Lockless-queue, RCU, batch processing

Page 18: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

17 Copyright©2014 NTT corp. All Rights Reserved.

Processing bypass for speed

NIC

skb_buf

Ethernet Driver API

Socket API

vswitch

packet

buffer

packet buffer

memory

Standard linux application

1. Interrupt & DMA

2. system call (read)

User space

Kernel

space

Driver

4. DMA

3. system call (write)

NIC

Ethernet Driver API

User-mode I/O & HAL

vswitch

packet

buffer

Application with intel DPDK

1. DMA Write 2. DMA READ

DPDK Library

Polling-base

packet handling

Event-base

packet handling

Page 19: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

18 Copyright©2014 NTT corp. All Rights Reserved.

Packet processing using multi core CPUs

Exploit many core CPUs Reduce data copy & move (reference access)

Simple packet classifier for parallel processing in I/O RX

Decouple I/O processing and flow processing Improve D-cache efficiency

Explicit thread assign to CPU core

NIC 1 RX

NIC 2 RX

I/O RX CPU0

I/O RX CPU1

NIC 1 TX

NIC 2 TX

I/O TX CPU6

I/O TX

CPU7

Flow lookup packet processing

CPU2

Flow lookup packet processing

CPU4

Flow lookup packet processing

CPU3

Flow lookup packet processing

CPU5

NIC 3 RX

NIC 4 RX

NIC 3 TX

NIC 4 TX

NIC RX buffer

Ring buffer

Ring buffer NIC TX buffer

Page 20: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

19 Copyright©2014 NTT corp. All Rights Reserved.

Packet batching

Reduce # of lock in flow lookup table Frequent locks are required

• Switch OpenFlow agent and counter retrieve for SNMP

• Packet processing

Input packet

Lookup table

Input packet

Lookup table

Lock Unlock

ロック Unlock

●Naïve implementation

●Packet batching implementation

Lock is required for each packet

Lock can be reduced due to packet batching

Page 21: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

20 Copyright©2014 NTT corp. All Rights Reserved.

Bypass pipeline with flow $

20

●Naïve implementation table1 table2 table3

Input packet

table1 table2 table3

Input packet

Flow cache

1. New flow

2. Success flow

Output packet

Multiple flow $ generation & mngmt Write flow $

●Lagopus implementation

Output packet

Packet

Reduce # of flow lookup in multiple table Exploit flow $ for

Page 22: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

21 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch by the user

Performance Evaluation

Page 23: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

22 Copyright©2014 NTT corp. All Rights Reserved.

Performance Evaluation

Summary Throughput: 10Gbps wire-rate

Flow rules: 1M flow rules

Evaluation models WAN-DC gateway

• MPLS-VLAN mapping

L2 switch

• Mac address switching

Page 24: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

23 Copyright©2014 NTT corp. All Rights Reserved.

Performance Evaluation

Evaluation setup

Server spec. CPU: Dual Intel Xeon E5-2660

• 8 core(16 thread), 20M Cache, 2.2 GHz, 8.00GT/s QPI, Sandy bridge

Memory: DDR3-1600 ECC 64GB

• Quad-channel 8x8GB Chipset: Intel C602 NIC: Intel Ethernet Converged Network Adapter X520-DA2

• Intel 82599ES, PCIe v2.0

Server

Lagopus

Flow

table

tester Flows

Throughput (bps/pps/%)

Flow

rules

Packet size

Flow cache

(on/off)

Page 25: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

24 Copyright©2014 NTT corp. All Rights Reserved.

WAN-DC Gateway

0

1

2

3

4

5

6

7

8

9

10

0 200 400 600 800 1000 1200 1400 1600

Throughput(Gbps)

Packetsize(byte)

10flowrules

100flowrules

1kflowrules

10kflowrules

100kflowrules

1Mflowrules

Throughput vs packet size, 1 flow, flow-cache

0

1

2

3

4

5

6

7

8

9

10

1 10 100 1000 10000 100000 1000000

Throughput(Gbps)

flows

10kflowrules

100kflowrules

1Mflowrules

Throughput vs flows, 1518 bytes packet

Page 26: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

25 Copyright©2014 NTT corp. All Rights Reserved.

L2 switch performance (Mbps) 10GbE x 2 (RFC2889 test)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

LINC OVS

(netdev)

OVS

(kernel)

Lagopus

(software)

Mb

ps

72

128

256

512

1024

1280

1518

Intel xeon E5-2660 (8cores, 16threads),

Intel X520-DA2, DDR3-1600 64GB

Packet size

Page 27: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

26 Copyright©2014 NTT corp. All Rights Reserved.

vSwitch for the user

PoC of carrier usecase

Open source community

Page 28: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

27 Copyright©2014 NTT corp. All Rights Reserved.

SPRING

alternative way of IP forwarding use “labels” to forward packets

much simpler architecture than MPLS network

can utilize traditional MPLS routers

good chemistry with NFV

For what ? policy-based routing

• explicit routing instead of shortest path routing

service-chaining (NFV)

• apply services to specific traffic pattern

Page 29: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

28 Copyright©2014 NTT corp. All Rights Reserved.

Career Network with SPRING

Source

Destination

NFV server

use lagopus as NFV server with career grade

routers

all nodes include NFV server are labeled

Page 30: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

29 Copyright©2014 NTT corp. All Rights Reserved.

Career Network with SPRING

Source

Destination

NFV server

at the ingress router, label stack is pushed

label stack represents the policy

IP

Page 31: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

30 Copyright©2014 NTT corp. All Rights Reserved.

PoC Implementation

Page 32: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

31 Copyright©2014 NTT corp. All Rights Reserved.

Demo

Page 33: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

32 Copyright©2014 NTT corp. All Rights Reserved.

Open Source Activity for the user

Lagopus goes wild since this summer

Your contribution are very welcome!

http://lagopus.github.io/

Twitter: lagopusvswitch

Ryu: component-based SDN framework

OpenFlow 1.3, 1.4, BGP, BMP, etc..

http://osrg.github.io/ryu/

Page 34: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

33 Copyright©2014 NTT corp. All Rights Reserved.

Summary

We contribute vSwitch for user Improve performance and scalability

Enhance the functionality and capability

DPDK is great library for user Easy to develop & implement

Easy to deploy like UNIX apps

Page 35: Lagopus SDN/OpenFlow switch: yet another SDN/OF switch agent

34 Copyright©2014 NTT corp. All Rights Reserved.

Thank you!

This research is a part of the project for “Research

and Development of Network Virtualization Technology” supported by the Ministry of Internal

Affairs and Communications.