open stackdaykorea2016 wedge

OpenNSL으로 브로드콤 기반 네트워크 스위치 제어하기

OpenStack Day Korea 2016

NIC 기술원

SDI Tech. Lab

Manager 서준호2016. 2. 18

1

Contents

본 발표는 지난 한달 간 Facebook이 제공한 Wedge를 가지고 이리 저리

가지고 논 경험을 담고 있음

1. OCP 소개

2. Facebook Wedge H/W 소개

3. Networking Operating System – Open Network Linux (ONL)

4. Broadcom Trident2 & OpenNSL 소개

5. OpenNSL 튜토리얼 (L2 스위칭, L3 라우팅)

2

OCP 소개

OCP (Open Compute Project)

§ Facebook 주도의 데이터센터 인프라 설계 기술 공유를 위한 오픈

하드웨어 프로젝트

§Facebook 은 Oregon 주 Prineville 데이터센터 인프라 스펙을 공개

§IT 기술을 접목시켜 경제적이고 효율적 관리를 통해 $20억 절감

§ 프로젝트

§Server/Storage/Networking/Open Rack/Data Center/Management

§ SKT 는 올해 2월 가입

§OCP Summit 2016 (3월) 에서 발표 예정

3

OCP Networking 소개

Project Chairs: Omar Baldonado | Carlos Cardenas

네트워크 하드웨어 + 네트워킹 소프트웨어

§ Disaggregation + Open

§ 현재는 ToR 스위치와 Spine 스위치 스펙 정의에 집중하고 있음

§ 또한 각 칩 벤더간의 서로 다른 Hardware Abstract Layer (HAL)

영역을 하나의 공통된 인터페이스 Service Abstract Interface (SAI)

로 스펙화 진행 중

4

Facebook이 개발한 1RU(Rack Unit) 크기의 ToR Switch

Facebook Wedge 소개 (1/3)

Modular Design

§ Micro-server module

§Intel Avoton C2550 (Atom)

§ Main switch module

§16 QSFP

§4x Fan module

§ BCM56850 (Trident2)

§ Open BMC module

Spine Switch

5


High Level Block Diagram

기존서버 아키텍처와다를게없음!

6

High Speed Interface Connectivity§ 16x QSFP (each x4 SerDes 10GbE)§ 16x (not used), but for 6-Pack


Main Switching ASIC - BCM56850 (Trident2)

2U Yosemite

Wedge

16 x Yosemite

7

Power on & Boot up Wedge

ONIE: Boot Loader contributed by CumulusNetworks§ GRUB2 (x86) or U-Boot (powerpc or other

architectures)§ Reinstall/Uninstall/Rescue/Boot NOSes on

OCP bare metal switches over IPv6 and HTTP

8

Installing Networking Operating System

Switch Light

Current ONIE compatibleNOS List

We choose Open Network Linux (ONL)

§ OCP 커뮤니티에서 베어 메탈 스위치를 위한 레퍼런스 네트워크 운영체제§ Big Switch가 OCP 커뮤니티 네트워킹 그룹에 오픈소스 기여§ 데비안 리눅스 기반 + 네트워크 ASIC 칩 드라이버 + ɑ (vendor specific)§ 현재 Big Switch Networks, 페이스북, NTT 사용 중§ 현재 베어 메탈 스위치 제조 벤더들이 지원 à 앞으로도 계속 ONL이

지원되는 OCP 표준 스위치 증가할 예정 (OCP summit `15)

9

Installing Networking Operating System

10

네트워크 스위치가 되기 위해서는…

지금까지 한 일

네트워크 스위치가 되기위해서 할 일

Wedge

11

Facebook - the next-generation Facebook data center network

Wedge (ToR 스위치) 6-pack (Spine 스위치)

Clos networks

FBOSS

Facebook ONL Use Case

12

Use Cases: NTT

NTT’s EVPN

NTT ONL Use Case

13

OpenNSL 소개

Broadcom Switching ASIC (i.e., Trident+, Trident2) Chip 제어를 위한오픈소스 기반 Hardware Abstraction Layer (HAL)

Licence Policy§ OEM/ODM Development Package§ Community Development PackageFeatures§ L2/L3/VLAN management§ Port and switch management§ Link monitoring§ Packet transmit and receive§ Trunking§ …Supported Platforms§ Acction Wedge, AS5712§ Broadcom reference platform (SVK)

14

OpenNSL 소개

Broadcom Switching ASIC (i.e., Trident+, Trident2) Chip 제어를 위한오픈소스 기반 Hardware Abstraction Layer (HAL)

OpenNSL Use Cases§ FBOSS (Facebook 주도)§ Open Network Linux (Big Switch 주도)§ BroadView§ Open L3 Routing (NTT 주도)§ OpenSwitch (HP주도)

15

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

Input Arbiter(Packet Header Parser)

Packet Processing Pipeline

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

High Level Block Diagram of Broadcom ASIC

OpenNSL 소개

PCI-

E

Kernel SpaceUser Space

OpenNSL

Broadcom SDKKernel BDE

Kernel KNET

DM

A

eth0

eth1

eth2 SRAM

TCAM

DRAM

16

Example – L2 Switching

Basic Three Operations of L2 Switch§ Address learning

§Src MAC matching§Automatically learning

§ L2 Forwarding§ Address ageing

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


Address Resolution Logic(ARL)

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

ARL Table

Memory

MAC | VLAN | PORT | TIME

opennsl_l2_addr_add(mac, vlan, port, flags);

flags: OPENNSL_L2_DISCARD_SRCOPENNSL_L2_DISCARD_DSTOPENNSL_L2_L3LOOKUPOPENNSL_L2_MOVE_PORTOPENNSL_L2_COPY_TO_CPUCALLBACK…

opennsl_l2_addr_add()opennsl_l2_addr_delete()opennsl_l2_addr_get()opennsl_l2_addr_register()opennsl_l2_addr_t_init()

17

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


ARL

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

Basic Three Operations of L2 Switch§ Address learning§ L2 Forwarding

§Dst MAC matching§Default - Flooding

§ Address ageing


Forwarding Logic

Forwarding Table

Memory

MAC | MAC_MASK

opennsl_l2_station_add(mac, mask, flags);

flags: OPENNSL_L2_STATION_ARP_RARPOPENNSL_L2_STATION_IPV4

opennsl_l2_station_add()opennsl_l2_station_delete()opennsl_l2_station_get()opennsl_l2_station_t_init()

18

Example – L3 Routing

L3 Routing§ Routing between different sub-networks§ Longest Prefix Matching (LPM)

192.168.1.0/24

192.168.2.0/24

192.168.3.0/24

Subnet 1 Subnet 2 Subnet 3

192.168.1.1

192.168.2.1

192.168.3.1

19


MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


L2 Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

L3 Logic

L3 Routing1. Set L3 egress mode2. Create IP interfaces3. Make the address learn on a VLAN and port4. Create egress object5. Add host point6. Add default route

opennsl_switch_control_set(opennslSwitchL3EgressMode);

20


MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


L2 Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

L3 Logic

L3 Routing1. Set L3 egress mode2. Create Ingress/Egress IP interfaces3. Make the address learn on a VLAN and port4. Create egress object5. Add host point6. Add default route

opennsl_l3_intf_create(vlan, mac, in_port);opennsl_l3_intf_create(vlan, mac, out_port);

21


MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


L2 Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

L3 Logic


opennsl_l2_addr_addr(vlan, in_port, mac);opennsl_l2_addr_addr(vlan, out_port, mac);flags |= OPENNSL_L2_L3LOOKUP

ARL Table

Memory

MAC | VLAN | PORT | TIME

22


MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


L2 Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

L3 Logic


opennsl_l3_egress_create(vlan, out_port, out_intf, nexthop_mac);

L3 Forwarding Table

out_inf | nexthop_mac

23


MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


L2 Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

L3 Logic


opennsl_l3_host_add(host_ip)

L3 Forwarding Table

host_ip | out_inf | nexthop_mac

24


MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ

MACRxQ

CPURxQ


L2 Logic

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

MACTxQ

CPUTxQ

Output Queues

L3 Logic


opennsl_l3_route_add(default_ip, out_inf)

L3 Forwarding Table

host_ip | out_inf | nexthop_mac

default_ip | out_inf | nexthop_mac

25

Thanks

감사합니다.

26


Basic Three Operations of L2 Switch§ Address learning – Src MAC matching§ L2 Forwarding§ Address ageing

Host A Host B

00:00:0A 00:00:0B

1

MAC ADD VLAN PORT

00:00:0A 1 1

00:00:0B 1 1

00:00:0C 1 2

00:00:0D 1 2

Address Learning Table

Host C

00:00:0C 00:00:0D

Host D

L2 Switch

opennsl_l2_addr_t addr;opennsl_l2_addr_t_init(&addr, mac, vlan);addr.flags |= SOME_ACTION;opennsl_l2_addr_add(&addr);

12

VLAN

OPENNSL_L2_DISCARD_SRCOPENNSL_L2_DISCARD_DSTOPENNSL_L2_COPY_TO_CPU…

27


Basic Three Operations of L2 Switch§ Address learning§ L2 Forwarding (Bridging) – Dst MAC matching§ Address ageing

Host A Host B

00:00:0A 00:00:0B

1

MAC ADD PORT VLAN

00:00:0A 1 1

00:00:0B 1 1

00:00:0C 2 2

00:00:0D 2 2

Address Learning Table

Host C

00:00:0C 00:00:0D

Host D

L2 Switch12

VLAN1VLAN2 opennsl_l2_station_t sta;

opennsl_l2_station_t_init(&sta);sta.dst_mac = 00:00:0Dsta.dst_mac_mask = FF:FF:FFopennsl_l2_station_add(&sta);

MAC ADD

00:00:00D

Station Table

28


Routing between different networks

Host A Host B

00:00:0A 00:00:0B

1

Host C

00:00:0C 00:00:0D

Host D

L3 Router12

Net1Net2

IP ADDR Intf

00:00:0A 1

00:00:0B 1

00:00:0C 2

00:00:0D 2

29


Routing between different networks1. Set L3 egress mode2. Create IP interfaces3. Make the address learn on a VLAN and port4. Create egress object5. Add host point6. Add default route

opennsl_switch_control_set(opennslSwitchL3EgressMode);

L3 Routing Logic

30


Routing between different networks1. Set L3 egress mode2. Create ingress & egress IP interfaces3. Make the address learn on a VLAN and port4. Create egress object5. Add host point6. Add default route

opennsl_l3_intf_create(vlan, mac, in_port);opennsl_l3_intf_create(vlan, mac, out_port);

L3 Routing LogicIn_intf out_intfin_port out_port

31



opennsl_l2_addr_addr(vlan, in_port, mac);opennsl_l2_addr_addr(vlan, out_port, mac);flags |= OPENNSL_L2_L3LOOKUP

L3 Routing LogicIn_intf out_intfin_port out_port

32



opennsl_l3_egress_create(vlan, out_port, out_intf, nexthop_mac);

L3 Routing LogicIn_intf out_intfin_port out_portEgress Object

33



opennsl_l3_host_add()


IP ADDR Intf

HOST_A out_intf

Routing Table

34



opennsl_l3_route_add()


IP ADDR Intf

HOST_A out_intf

DEFAULT out_intf

Routing Table

open stackdaykorea2016 wedge

Engineering