integrate kubernetes into cord(central office re-architected as a datacenter)

KUBERNETES INTEGRATEWITH CORDHUNG-WEI CHIU

LINKER NETWORKS.INC

WHO AM I

• Hung-Wei Chiu (邱宏瑋)

• hwchiu@linkernetworks.com

• hwchiu.com

• Experience

• Software Engineer at Linker Netowrks

• Software Engineer at Synology (2014~2017)

• Co-Found of SDNDS-TW

• Open Source experience

• SDN related projects (mininet, ONOS, Floodlight, awesome-sdn)

OUTLINE

• What is CORD

• Challenge of kubernetes with CORD.

• What have we done now

• Next steps

What is CORD ?

CENTRAL OFFICE RE-ARCHITECTED AS A DATACENTER

5

SDN + NFV + Cloud

Open Source Software

Commodity Hardware

(Servers, White-Box Switches, I/O Blades)

Large number of

COs

Evolved over 40-50 years

300+ Types of

equipmentHuge source

of CAPEX/OPEX

DATA CENTER LEAF-SPINE FABRIC

6

Reliability 😃 Flexibility 😃 Bandwidth 😃

Spine

Leaf

Me

troR

ou

ter

White Box White Box

White Box

White Box

White Box White Box White Box White Box

White Box White Box White Box

White Box

White Box

White Box

Open SourceSDN-basedBare-metal

White Box

White Box

R,E,M-Access

7

ONOS Controller Cluster

vRouterControl

XOS (Orchestrator)

vSG

VNF

VNF VNFVNFvSG VNF

VNF

VNFVNF

vSG VNF

VNF VNF

VNF

OVS OVS OVS OVS OVS

Residential Mobile Enterprise

Underlay Control

Underlay

OverlayControl

Overlay

vOLTControl

Final CORD Architecture

Me

troR

ou

ter

White Box White Box

White Box

White Box

White Box White Box White Box White Box

White Box White Box White Box

White Box

White Box

White Box

Open SourceSDN-basedBare-metal

White Box

White Box

R,E,M-Access

8

ONOS Controller Cluster

vRouterControl

XOS (Orchestrator)

vSG

VNF

VNF VNFVNFvSG VNF

VNF

VNFVNF

vSG VNF

VNF VNF

VNF

OVS OVS OVS OVS OVS

Residential Mobile Enterprise

Underlay Control

Underlay

OverlayControl

Overlay

vOLTControl

Final CORD Architecture

Me

troR

ou

ter

White Box White Box

White Box

White Box

White Box White Box White Box White Box

White Box White Box White Box

White Box

White Box

White Box

Open SourceSDN-basedBare-metal

White Box

White Box

R,E,M-Access

9

ONOS Controller Cluster

vRouterControl

XOS (Orchestrator)

vSG

VNF

VNF VNFVNFvSG VNF

VNF

VNFVNF

vSG VNF

VNF VNF

VNF

OVS OVS OVS OVS OVS

Residential Mobile Enterprise

Underlay Control

Underlay

OverlayControl

Overlay

vOLTControl

Final CORD Architecture

SUMMARY

• VM-based NFV (Network Function Virtualization)

• Use ONOS (SDN controller) + Open VSwitch to control packets.

• Use XOS (Service Orchestration) to control all services (VNF)

• ONOS/XOS need to communicate with OpenStack component.

Why kubernetes CORD ?

WHO?

• Ciena

• Hate OpenStack

• Intel China

• Some Telecom service providers

• China unicom

CHANGE TO KUBERNETES

• VM-based NFV (Network Function Virtualization)

• vSG, vPGW, vSGW, etc

• Who owns the NFV?

• Vendors

• We can’t force them to convert all NFVs to container.

• It’s impossible to have a kubernetes solution for CORD now.

CHANGE TO KUBERNETES

• How about hybrid architecture?

• Support OpenStack + kubernetes

CHANGE TO KUBERNETES

• Use ONOS (SDN controller) + Open vSwitch to control packets.

• There’re so many CNI for kubernetes now.

• None of them are Open vSwitch based solution.

• Linen-CNI is Open vSwitch + Linux Bridge solution

• Same subnet traffics between POD are controlled by Bridge

NODE

POD POD

OVS

br0

ens0p3

CHENLAGE TO KUBERNETES

• Use ONOS (SDN controller) + Open vSwitch to control packets.

• Create our own CNI to support pure Open vSwitch version.

• Still some problem need to conquer future. NODE

POD POD

OVS

ens0p3

？

CHANGE TO KUBERNETES

NODE

POD POD

Bridge

ens0p3

many iptable rules.

many iptable rules.

CHANGE TO KUBERNETES

• Use XOS (Service Orchestration) to control all services (VNF)

• Kubernetes can handle most thing.

• XOS should communicate with Kubernetes via its API service.

CHANGE TO KUBERNETES

• ONOS/XOS need to communicate with OpenStack component.

• ONOS need to know the IP information of each host (VMs) from neutron component.

• In kubernetes, we need to provide the IP information of each POD.

• Since we implement our CNI, we can send the IP information after CNI assigns IP to POD.

• Send the information via Restful API/gRPC

PROBLEMS WE MET

• Deploy ONOS controller as container

• Multiple network interface for POD

• Centralized IP management

DEPLOY PROBLEM

• We need to deploy ONOS as container

• The chicken-and-egg conundrum!

• Hard to solve, we need to work-around now.

• We decide to move the ONOS out of POD.

• For each node, it should have multiple network interface including data network and control

network.

• Out of band.

DEPLOY PROBLEM

NODE

POD POD

OVS

ens0p3

ens0p4

NODE

POD POD

OVS

ens0p3

ens0p4

NODE

POD POD

OVS

ens0p3

ens0p4

Data network

Control network

DEPLOY PROBLEM

NODE

POD POD

OVS

ens0p3

ens0p4

NODE

POD POD

OVS

ens0p3

ens0p4

NODE

POD POD

OVS

ens0p3

ens0p4

Data network

Control network

MULTIPLE NETWORK INTERFACE

• For some NFV,(vSG) they need to multiple interface in its POD.

• For example.

NODE

POD POD

OVS

ens0p3

MULTIPLE NETWORK INTERFACE

• We found a open source project (multus-CNI)

• Provides the multi interface support in a pod

• We don’t figure out how to use it first.

• We thought it’s a global setting.

• We try to implement by ourself.

MULTIPLE NETWORK INTERFACE

• Multiple network interface means call CNI multiple times.

• For CNI, we need to know the namespace location for each POD.

• We want to provide a interface to dynamically call CNI for any existing POD.

• Input

• Pod Name, Network configuration (CNI name, configuration)

• Output

• Success (add another interface for existing POD)

• Error

MULTIPLE NETWORK INTERFACE

• For each CNI, it should know the namespace location of each POD.

• We also need to know the POD name

• In the CNI plugin, we can get above information via

• Args.Args (many information, separate by semi-colon)

• Args.Netns

• We store those information in ETCD.

MULTIPLE NETWORK INTERFACE

• After I have finished almost tasks.

• One intel guy tell us multus-cni support pod configuration.

• …..…..

• Ok, we use multus-cni.

CENTRALIZED IP MANAGEMENT

• Our CNI plugin use IPAM to handle the IP management.

• Official IPAM support two types.

• Host-local

• DHCP

IPAM DHCP

• Requirement

• You should run a IPAM DHCP daemon on each node.

• You should setup a DHCP server on your network.

• How it works.

• Start a DHCP client when a POD is be created.

• Forward the DHCP packet to DHCP server(it depends on your CNI forward L2 broadcast)

• Official recommend to use MacVLan as CNI.

• Limitation

• All node in same subnet.

• Simple configuration. You only specify “type=dhcp” in CNI configuration.

IPAM HOST-LOCAL

• Requirement

• None

• How it works

• Use a local file to record how many IP address has been used.

• Lookup the file and choose an available IP address for CNI.

• Limitation.

• You should prepare a configuration for each Node with different setting.

• Complex configuration. You need to specify which subnet the Node will use.

• Make sure no duplicate for each Node.

IPAM

• What we want ?

• Simple configuration

• Support multiple subnet

• We decided to create new IPAM

DHCP

• Refer to trellis (CORD network infrastructure)• Requirement

• DHCP server

• How it works• We set the Gateway Address for each dhcp request to support multiple subnet via L3 unicast.• It looks like

DHCP

NODE 1

POD POD

OVS

ens0p3

POD POD

OVS

ens0p3NODE 2

network

Master DHCPServer

DHCP relay DHCP relay192.168.1.1 192.168.2.1

Subnet 192.168.1.0/24 {

}Subnet 192.168.2.0/24 {

}

DHCP – PROBLEM

• The Problem is..• How to decide the IP address of each Open vSwitch?

• Gateway address of each subnet.

• Use ETCD• Maybe we can use ETCD to replace the DHCP

ETCD

• Implement a new IPAM which use etcd to record the subnet of each node.• Simple configuration

• Network: 10.12.0.0/16• Subnet length: 24• etcd address.

• The subnet will from 10.12.1.0/24 to 10.12.255.0/24• Simple and easy to implement.

NEXT STEP

• Integrate the ONOS (SDN controller) with our CNI.

• Make sure the ONOS can control the network.

• For the kubernetes internal communication, the ONOS should implement all of them by

Openflow, not iptables.NODE

POD POD

OVS

ens0p3

Use openflow rules

many iptable rules.

NEXT STEP

Fix Bugs

Q&A