howto createopenflow switchusing fpga (at fpgax#6)

How to createOpenFlow Switch

using FPGAPresented at FPGA Extreme Computing #6

2015/02/01

Kentaro Ebisawa |海老澤健太郎 | Twitter: @ebiken

Disclaimer

• The objective of this presentation is to “share information” related to OpenFlow Switch development and increase people interested in OpenFlow and developing Network Equipments.

• Thus, it is not directly related to specification or design of any existing product.

How to create OpenFlow Switch using FPGA | FPGA Extreeme Computing #6 | FPGAX 2014/02/01 | Twitter: @ebiken 2

Self-introductionCurrent

• Dev manager of OpenFlow Switch@ start-up company (Riava Networks, Inc.)

Past• Several Technical role at multiple startup

companies to help booting up team or company, growing presence in Asia market.• Head of support team• Sales Engineer (SE)• Solution Architect designing OSS/BSS.• Product design and research.• Development management.

• Twitter: @ebiken

• Technical Area• ATM / Ethernet / PPPoE / L2TP / RADIUS• Streaming / Web Proxy• Virtualization

• OpenVZ / Parallels Cloud Server• Hosting Service. OSS/BSS.

• Operation / Business Support System• IPv6/v4 Translation (nat64)• OpenFlow


List of Contents

1. What is “OpenFlow Switch” ?(Comparing with Layer 2 / 3 switch)

2. Why use FPGA to build OpenFlow Switch?

3. OpenFlow Switch components

4. OpenFlow Switch design trade-offs



What is “OpenFlow Switch” ?

Comparing with Layer 2 / 3 switch.

How traditional network switch works (Layer 2)


MAC Address Port

00-00-5E-00-53-01 1

00-00-5E-00-53-02 2

00-00-5E-00-53-03 3

00-00-5E-00-53-04 4

Layer 2 forwarding (MAC learning) table

1 2 3 4 5

-01 -02 -03 -04 -05

Table Entry• Self learning (Source MAC)

Action (match)• Forward to Port

Action (no-match)• Flooding

VLAN 1 VLAN 2

How traditional network switch works (Layer 3)


Subnet (LPM) Gateway Port

192.168.10.0 /24 10.0.0.5 5

192.168.20.0 /24 172.16.0.1 1

... ... ...

0.0.0.0/0 172.16.0.1 1

Layer 3 forwarding table

1 2 3 4 5

172.16.0.0/24 10.0.0.0/24

Table Entry• Routing Protocol etc.

Action (match)• Decrement TTL• Modify Destination MAC• Forward to Port

Action (no-match)• Drop• Send “ICMP Dest untreachable”

How OpenFlow switch works


Multiple Flow (Lookup) Tables

1 2 3 4 5

Match Field Instruction / Action

MPLS Label pop MPLS label, meter 10, goto-table 4

VLAN output port 2, ...

IPv4 Address, TCP port set IPv4 Address, Queue 10, goto-table 3

... ...

Table Entry• Inject from Controller





... ...





... ...



VLAN output port 2


... ...

OpenFlow Controller

Which packet? (OpenFlow Match Field : 40 Fields, 1261 bits)


Field Name bits

OXM_OF_IN_PORT 32

OXM_OF_IN_PHY_PORT 32

OXM_OF_METADATA 64

OXM_OF_ETH_DST 48

OXM_OF_ETH_SRC 48

OXM_OF_ETH_TYPE 16

OXM_OF_VLAN_VID 12+1

OXM_OF_VLAN_PCP 3

OXM_OF_IP_DSCP 6

OXM_OF_IP_ECN 2

OXM_OF_IP_PROTO 8

OXM_OF_IPV4_SRC 32

OXM_OF_IPV4_DST 32

OXM_OF_TCP_SRC 16

Field Name bits

OXM_OF_TCP_DST 16

OXM_OF_UDP_SRC 16

OXM_OF_UDP_DST 16

OXM_OF_SCTP_SRC 16

OXM_OF_SCTP_DST 16

OXM_OF_ICMPV4_TYPE 8

OXM_OF_ICMPV4_CODE 8

OXM_OF_ARP_OP 16

OXM_OF_ARP_SPA 32

OXM_OF_ARP_TPA 32

OXM_OF_ARP_SHA 48

OXM_OF_ARP_THA 48

OXM_OF_IPV6_SRC 128

Field Name bits

OXM_OF_IPV6_DST 128

OXM_OF_IPV6_FLABEL 20

OXM_OF_ICMPV6_TYPE 8

OXM_OF_ICMPV6_CODE 8

OXM_OF_IPV6_ND_TARGET 128

OXM_OF_IPV6_ND_SLL 48

OXM_OF_IPV6_ND_TLL 48

OXM_OF_MPLS_LABEL 20

OXM_OF_MPLS_TC 3

OXM_OF_MPLS_BOS 1

OXM_OF_PBB_ISID 24

OXM_OF_TUNNEL_ID 64

OXM_OF_IPV6_EXTHDR 9

How to process it? (OpenFlow Instruction / Action)


Instruction

Meter Policing（Rate-Limit）

Apply-Actions Apply Imidiately

Clear-Actions Initialize Action Set

Write-Actions Add to Action Set

Write-Metadata Stamp metadata

Goto-Table <ID>Go through another table

Action （アクション）

Output <port_no> Forward packet

Group <group_id> Group processing

Set-Queue <queue_id> Assign Queue

Push-Tag <ethertype>MPLS,PBB,VLAN

Pop-Tag <ethertype>

Set-Field<field_type> <value>

Set value to arbitrary fields.

Set TTL <ttl>

MPLS, IPTTL (Time To Live)

Decrement TTL

Copy TTL out/inwards

OpenFlow Pipeline


OpenFlow Pipeline

Flow Table #0

Flow Entry

Flow Entry

Flow Table #n

Flow Entry

Flow Entry

Match

Flow Table #1

Flow Entry

Action Set(associated with each packet)

Flow Entry

Action

Action

Action

PacketIngress

Match

Match

Goto-Table Goto-Table PacketOutput

ExecuteAction

Set(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

Comparing Layer 2/3 Switch and OpenFlow Switch


Which packet? How to process it?

Layer 2 MAC AddressForward to port where

Host with Dest MAC exists

Layer 3 IP AddressSet Dest MAC AddressForward to Next Hop

OpenFlowSet from Controller

(Based on Packet header)

Set from Controller(Set header fields,forward, QoS etc.)

Which packet? How to process it?

Layer 2 MAC AddressForward to port where

Host with Dest MAC exists

Layer 3 IP AddressSet Dest MAC AddressForward to Next Hop

OpenFlowSet from Controller

(Based on Packet header)

Set from Controller(Set header fields,forward, QoS etc.)

Comparing Layer 2/3 Switch and OpenFlow Switch


How to Match/Process is pre-defined.

How to Match/Process is programmable.


Why use FPGA to buildOpenFlow Switch?

3 reasons for using FPGA to build OpenFlow Switch


Avoid restriction of existing Network Chip

Keep up with new version of standard.

Continuous improvements based on new use cases.

Avoid restriction of existing Network Chip

• Difficult to implement Full OpenFlow Pipeline using existing network chip aimed for Layer 2/3 switch.

• TCAM is small• Mostly used for ACL only.

• Re-use of L2/L3 Forwarding Table• Limited match fields available on each table.

• Less Instruction / Action supported.• Some tables cannot use most Instructions / Actions.


Continuous improvements based on new use cases.

• OpenFlow will make network programmable.

• Programmable = expected behavior / feature depends more on Business requirements. (Compared to L2 / L3 switch)

• Sizing balance (where to put more HW resource) depends on business model.• Group, Meter, QoS, Packet-in/out … which one is more important??• Datacenter CLOS Network ?• Carrier / Provider Edge Gateway ?• Security Gateway ?• Wireless ?


Keep up with new version of standard.


2010 2011 2012 2013 2014

(Dec. 31, 2009)1.0.0

(Jun. 7, 2012)1.0.1 Errata

(Nov. 1, 2013)1.0.2 Errata

(Apr. 13, 2012)1.3.0

1.3.1(Sept. 6, 2012)

(Apr. 25, 2013)1.3.2

(Mar. 27, 2014)1.3.4

1.3.3(Dec. 18, 2013)

(Feb. 28, 2011)1.1.0

(Dec. 5, 2011)1.2

(Oct. 15, 2013)1.4.0

0x01

0x02

0x04

0x05

0x03

Wire Protocol #

(Jan. 9, 2015)1.5.0

0x06History of OpenFlow Standard

2015

https://www.opennetworking.org/sdn-resources/technical-library#tech-spec

https://www.opennetworking.org/sdn-resources/technical-library#tech-spec


OpenFlow Switch components

Using Hardware Datapath

OpenFlow Switch components | Software


Hardware Abstraction Layer

Hardware Specific Driver

DatapathPCIe

Open Flow Agent

Protocol Interface

Admin Interface• Switch/Controller IP Address configuration• Showing Status / Statistics• Debug, Log

OF Protocol <-> Datapath data format conversion.Managing Flow Entry Layout in Datapath (Stored in TCAM/RAM）Packet-in/out processing.

PCIe Driver.FPGA packet processing board (Datapath) will be recognized as PCI express device from OS.

Admin Interface(CLI, syslog, SNMP)

CPU Board

FPGA / TCAM / RAM(*)Physical Network Ports

(*) RAM = SRAM/DRAM/BlockRAM etc.

Protocol level interface between controller.Managing Flow Table data coupled with OpenFlow protocol.

OpenFlow Switch ~ Hardware Datapath Implementation Types


Datapath (FPGA etc.)

FlowTable #0

FlowTable #x

FlowTable #N

Software (CPU)

FlowTable #0

FlowTable #x

FlowTable #N

Insert Flow Table

OpenFlow Agent

Packet

Type 1: Flow Installation




Flow CacheTable

Software (CPU)

FlowTable #0

FlowTable #x

FlowTable #N

OpenFlow Agent

Packet

Type 2: Flow Cache (No Cache Entry)

Insert Flow CacheRequest




Flow CacheTable

Software (CPU)

FlowTable #0

FlowTable #x

FlowTable #N

OpenFlow Agent

Packet

Type 2: Flow Cache (With Cache Entry)



Type 1: Flow Installation Type 2: Flow Cache

Packet in same flow has same latency Large delay processing initial packet of

each flow

(Relatively) Easy to expect performance. Bottleneck between Datapath/CPU Significant performance impact when

cache over flows

(Relatively) easy to add / mod Flow Entry

Require Flushing Cache when add / mod Flow Entry (Multi Table)

(Could be avoided with smart algorism but complex)

Consume more HW resource (for non-used Flow Entry)

Consume less HW resource (only active flow)



Type 1: Flow Installation Type 2: Flow Cache

Packet in same flow has same latency Large delay processing initial packet of

each flow

(Relatively) Easy to expect performance. Bottleneck between Datapath/CPU Significant performance impact when

cache over flows

(Relatively) easy to add / mod Flow Entry

Require Flushing Cache when add / mod Flow Entry (Multi Table)

(Could be avoided with smart algorism but complex)

Consume more HW resource (for non-used Flow Entry)

Consume less HW resource (only active flow)

Today we will discuss Type 1 only.

OpenFlow Switch components | Hardware Datapath


Hardware Abstraction Layer

Hardware Specific Driver

DatapathPCIe

Open Flow Agent

Protocol Interface

Admin Interface• Switch/Controller IP Address configuration• Showing Status / Statistics• Debug, Log

OF Protocol <-> Datapath data format conversion.Managing Flow Entry Layout in Datapath (Stored in TCAM/RAM）Packet-in/out processing.

PCIe Driver.FPGA packet processing board (Datapath) will be recognized as PCI express device from OS.

Admin Interface(CLI, syslog, SNMP)

CPU Board

FPGA / TCAM / RAM(*)Physical Network Ports

(*) RAM = SRAM/DRAM/BlockRAM etc.

Protocol level interface between controller.Managing Flow Table data coupled with OpenFlow protocol. Now Moving to

Datapath.


CPU Board(OpenFlow Agent)

Physical Ports (10Gbps / 1Gbps)

PCIe Core

DMA ControlHost Interface

MAC (XAUI / QSGMII)

TCAM(Match Field)

SRAM | DRAM(Instruction, Action)

(Packet Buffer)

Packet FIFO

OpenFlow Pipeline Processor

Flow Table ProviderFPGA Switch / Queue

PCIe

Insert Flow Entry


CPU Board(OpenFlow Agent)

Physical Ports (10Gbps / 1Gbps)

PCIe Core

DMA ControlHost Interface

MAC (XAUI / QSGMII)

TCAM(Match Field)

SRAM | DRAM(Instruction, Action)

(Packet Buffer)

Packet FIFO

OpenFlow Pipeline Processor

Flow Table ProviderFPGA Switch / Queue

PCIe

Packet Input


OpenFlow Switch design trade-offs

a. Long Search / TCAM Entry Key Length

b. Update considering Priority

c. A lot of packet fields to process

Layer 2 Forwarding (MAC learning) Table Search (Using CAM)


MAC Address (48 bit) Port

00-00-5E-00-53-FE 5

00-15-17-ED-9E-5B 3

6C-F0-49-7F-79-AA 1

00-00-5E-00-53-01 1

00-15-17-ED-9E-01 2

… …

00-00-5E-00-53-01

Search Key

Forward to Port #1

CAM (Exact Match)

Layer 2 Forwarding (MAC learning) Table Search (Hash Table)


MAC Address Port MAC Address Port MAC Address Port

0 ⇒ 00-00-5E-00-53-FE 5

1 ⇒ 00-15-17-ED-9E-00 3 00-15-17-ED-9E-2F 2 00-15-17-ED-9E-8B 4

2 ⇒ 6C-F0-49-7F-79-AA 1

3 ⇒ 00-00-5E-00-53-08 3 00-00-5E-00-53-01 1

4 ⇒ 00-15-17-ED-9E-01 2 00-00-5E-00-53-F0 4 00-15-17-ED-9E-32 8

… ⇒ … … … … … …

00-00-5E-00-53-01

Search Key

Forward to Port #1

HASH

Hash Table (on SRAM)

Layer 3 Forwarding Table Search (TCAM - Longest Prefix Match)


TCAM (with Don’t care bit)

IPv4 Address (32 bit) Gateway

192 168 10 * 10.0.0.5

10 128 0 * 172.16.0.1

192 168 20 * 172.16.0.2

192 168 * * 10.0.0.100

... ... ... ... ...

* * * * 172.16.0.1

192.168.20.100

Search Key

• TCAM is required for LPM• Search Key has fixed length• Only 64bit even for IPv6

※ Actually masked per bit

OpenFlow Flow Table (TCAM – Simple design)


IN_PORT METADATA ETH_DST ETH_SRC VLAN_VID MPLS_LBL IP_PROTO IPV4_SRC …

10 * * * 1 * * 10.1.1.* …

* * * * * * 6 10.1.1.1 …

* * * * 10 * * * …

5 * * * 5 * * * …

* 0xABAB0010 * * * * 17 * …

… … … … … … … … …

PORT, METADATA, ETH_DST …

Search Key (1248 bit)

Instruction/ Action

pop VLAN, …

push vlan, goto-table 1

Output 5, …

set-metadata, …

set-field …

…

TCAM (40 Fields, 1261 bits)

• Inefficient use of TCAM• 1261 bit per TCAM Entry => about 40Mbit per 32K Entry• Only a few fields used per Flow Entry

• Entry has Priority => Could require shifting when insert.

How to deal with Long Search Key (example 1)


参考文献：”OF-DPA Abstract Switch Specification ver1.0” by Broadcom Corporation.Simplified “Figure 2: OpenFlow OF-DPA Abstract Switch Pipeline“https://github.com/Broadcom-Switch/of-dpa/blob/master/doc/OFDPA_OASS-ETP101-R.PDF

IngressPort

Table 1MAC

Table 3

UnicastRouting

Table 4a

MulticastRouting

Table 4b

Bridge

Table 4c

ACL Policy

Table 5

VLAN

Table 2

Match limited field per Table Some table does not require TCAM Cannot match entry using AND operation

(ex：MAC_DST==XXX && VLAN_ID==YY)

SearchProcess

CAM orSRAM

TCAM

https://github.com/Broadcom-Switch/of-dpa/blob/master/doc/OFDPA_OASS-ETP101-R.PDF

Update considering Priority (using TCAM)


Strict Priority for all entries Loose Priority (*1)

Resource efficiency (BAD GOOD) | Management Logic (Simple Complex)

Priority PORT VLAN IP Addr TCP Port

255 3 * * 80

255 1 20 10.0.0.* *

50 * * * 80

10 1 * * 443

0 * * * *


255 3 * * 80

255 1 20 10.0.0.* *

50 * * * 80

10 1 * * 443

0 * * * *


10 * 10 10.0.0.* 22

(*1) Sort in priority order for each set of rules which can match same packet.

Trade-off depending on how to process packet fields.


Layer 3 Switch• Modify MAC DA

• DA: Destination Address• Decrement TTL• Layer 2 Processing

MAC DA MAC SA VLAN EtherType MPLS IPv4 / IPv6 / ARP TCP / UDP / ICMP Payload

Layer 2 Switch• VLAN Tag Push/Pop

Layer 2 / 3 Switch modifies only a few packet fields.




OpenFlow Switch• VLAN Tag Push/Pop• MPLS Push / Pop• Modification of almost any header fields





parallel processing Delay: small (1 ~ a few cycles) Logic size: Huge

Sequentially processing each field Delay: big (10s ~ 100+ cycles) Logic size: Small


For people who wants to try it…

Packet processing using FPGA is FUN!!

• Can buy at Digilent Inc.• Part# 410-301P-KIT

• Academic：$1,695• List price: $24,500

How to try?• NetFPGA SUME

• NIC type FPGA board for research and experimentation.

• FPGA: Virtex-7 690T• Network:

• 4 x 10Gbps (SFP+)

• Memory:• 2 x 4GB DDR3 SoDIMM• 3 x 72Mbits QDR II SRAM

• PCIe x8 Gen. 3 (8Gbps x8)• 2 x SATA-III


Photo from Digilent Inc.http://digilentinc.com/Products/Detail.cfm?NavPath=2,1301,1311&Prod=NETFPGA-10G-SUME

http://digilentinc.com/Products/Detail.cfm?NavPath=2,1301,1311&Prod=NETFPGA-10G-SUME

How to try?• OpenFlow implementation using

FPGA available on GitHub• A bit old but good start.

• Design Document• Block Diagram• State Machine

• Source Code• GitHub

• OpenFlow 1.0 only• Single Table (No multi-table)


https://github.com/NetFPGA/NetFPGA-public/wiki/NetFPGA-10G-OpenFlow-Switch

https://github.com/NetFPGA/NetFPGA-public/wiki/NetFPGA-10G-OpenFlow-Switch


!! Let’s have fun together !!

FPGA + OpenFlow

海老澤健太郎｜ Kentaro Ebisawa

Twitter: @ebiken

howto createopenflow switchusing fpga (at fpgax#6)

Technology