open network linux€“ second stage more feature-full boot loader ... debian wheezy • need to...

Open Network Linux

A Path to an Open Source Network OS Rob Sherwood Big Switch Networks CTO

Outline: Open Network Linux (ONL)

•  Context: What is ONL? •  Network OS Architecture View •  What does ONL provide?

–  ONIE Compatible NOS Installer –  Supported Hardware –  Basic L3 Routing with Open Route Cache –  Platform Abstraction Layer – ONLP –  “Server-like” Management –  Second stage more feature-full boot loader

•  Plans for the Future •  How to Get Involved

©2014 B IG SWITCH NETWORKS , INC . WWW.B IGSWITCH .COM 3

ISN’T NETWORKING ALREADY OPEN? Faster Throughput = More commercial value = More Proprietary

Aggregate Throughput à

ß Open

10 Gb/s 100 Gb/s 1 Tb/s 10 Tb/s

All Hardware Specs Public X86 Servers

Closed OS, Open APIs for Policy, Stats

MulT-‐chassis Routers

Closed OS, Open Control

Plane APIs

DC – 1U ToRs, Spines

Binary SDK for Packet

Forwarding NPU, FPGA



ISN’T NETWORKING ALREADY OPEN? Faster Throughput = More commercial value = More Proprietary

Aggregate Throughput à

ß Open

10 Gb/s 100 Gb/s 1 Tb/s 10 Tb/s

All Hardware Specs Public X86 Servers

Closed OS, Open APIs for Policy, Stats

MulT-‐chassis Routers

Closed OS, Open Control

Plane APIs


Binary SDK for Packet

Forwarding NPU, FPGA DC – 1U

ToRs, Spines This talk


LOTS OF OPEN SOURCE CONTROLLERS But what are they controlling?

Open Source SDN

Controllers

SDN OSS Software Switches SDN OSS Hardware Switches

??? This talk

ONL is a Linux Distribution for Bare Metal Switches

•  A collection of software packages, utilities, drivers, and abstractions to run on OCP, bare metal, “brite box” hardware

•  i.e., a “NOS” that ONIE would install •  Why not use an existing Linux distribution?

•  Does build on existing distribution – Debian Wheezy •  Need to create ONIE installers for many platforms •  Need to manage switch-specific hardware (e.g.,SFPs) •  Switches are very similar to servers, but not quite

ONL Example Use-Cases

1.  DIY packet forwarding platform, e.g., for students •  Platform driver code too complicated for most hobbyists

2.  Reference hardware testing platform, e.g., OCP Certification •  Common core in open source for verification, reference

3.  Building Block for Commercial Software •  Pull ONL in to a larger, commercially supported solution

•  Not a Big Switch talk, but we use ONL in our products

Why Use ONL?

•  Help ecosystem focus on innovation •  Many annoying software details to run an OCP switch •  Building platform drivers not high value asset; should be common

•  Enables a reference NOS implementation •  Hardware without software is not useful •  Package up details and best practices into one place

•  Help bootstrap the Open ecosystem and OCP adoption •  Allows commercial companies and DYI-folks to build OCP-based

products faster

CPU Misc Hardware (fans, LEDs, SFP, sensors) ASIC

Platform Abstraction Layer ASIC SDK

Forwarding Agent

Environment Agent(s)

Per Switch Stack

OS Distribution (Linux kernel, etc.)

Background: Generic NOS Architecture

Hardware

Platform

Applications

CPU (x86,PPC)

Misc Hardware (fans, LEDs, SFP, sensors) BRCM

ONL Platform APIs BRCM SDK

SensorD

Per Switch Stack

ONL Distribution (Linux + stuff)

Shipping now: ONL + Open Route Cache

Hardware

Platform

Applications Open Route Cache

Orc-brcm Driver

Quagga/Xorp/Etc.

ONLP Dump

S N M P

ORC Driver API

This API looks a lot like SAI

ONLP APIs

Closed Source

ONL Contributors/Supporters (So Far)

OTHERS Provides Platform Drivers

Adds Forwarding Agents

Open Network Linux

Additional Drivers

Linux 3.9.6

Linux 3.8.13

Open Route Cache

FaultD

Architecture Infrastructure Module (AIM)

Plaaorm-‐* (installers, etc.)

ONLP (Plaaorm API Library)

ONLP-‐$vendor

ONL-‐Specific .debs

$ONL/debian/repo

Generic .debs From

Emdebian.org and Debian.org

Via apt-‐cacher-‐ng

Switch Image (.swi)

ONL Loader

Using MulTstrap and

`onl-‐mkws`

ONIE Installer SHAR

wrapper

ONL Loader

ONL ONIE

Installer

All components: `make –C $ONL/builds/components`

SWI: `make –C $ONL/builds/swi/$arch/all` Installer:

`make –C $ONL/builds/

installer/$arch/all`

Anatomy of `make onl-‐$arch`

Specific component:

`make –C $ONL/components/ $arch/$component/

deb`

Busybox Buildroot

Etc.

New from ONL

3rd Party OSS

Linux 3.9.6 Kernel Patches

Mixed

ONL Relative to ONIE

• “An Installer Needs a NOS”

ONIE One Time Installer

~3MB

Normal Full-‐featured Boot Loader (w/BusyBox)

Main Network OS Image (.swi) (w/real binaries)

~16MB

~160 MB

Open Network Linux Github.com/ onie/onie

Common kernel and DTS files??

Open Network Linux Community

•  Project has been public for ~1 year •  Code – http://github.com/opennetworklinux/ONL •  Website – http://opennetlinux.org •  ~50K LOC (without Linux patches); 56K Makefile lines

•  Community Traction •  Six platforms added by community; more coming •  Three companies have commercial projects w/ONL •  ~14 code contributors from four companies

ONL Provides an ONIE-Compatible Installer

•  ONIE is great – but needs an installer

•  Installer packages entire NOS into a single image •  Installer is the glue between Platform and NOS •  Sets the $nos_bootcmd magic

•  e.g., nos_bootcmd= diskboot 0x10000000 0:1 ; bootm 0x10000000 •  Formats local storage, e.g., flash or local disk •  Installs ONL loader onto local storage (see next)

Install Using ONIE then Boot ONL

Boot Flash

~64MB

uBoot ENVs ONIE

Free Space

Boot Logic: 1. uBoot POSTs 2. $nos_bootcmd is read from ENVs 3.  run $nos_bootcmd

•  If $nos_bootcmd returns, run ONIE •  On install, ONIE sets $nos_bootcmd

to load ONL loader 4.  Loader downloads specified SWI URL

if not cached 5.  Loader mounts rootfs as ramdisk with

overlayfs 6.  ONL loader kexec’s SWI kernel

Mass Storage

~2GB

ONL Loader

SWI’s: ONL

SWI #1 (cached)

ONL config


ONL – SUPPORTED HARDWARE X86 and PPC Support

BRCM

BRCM SDK

Basic L3 Routing With Open Route Cache (ORC)

Open Route Cache

Orc-brcm Driver

Quagga/Xorp/Etc. 1.  ORC creates a orcXX interface for each port

2.  Quagga (or Xorp, etc.) installs routes into Linux kernel

3.  Linux kernel sends RT Netlink Updates to ORC

4.  ORC translates to SDK calls •  Hardware accelerated routing! •  Only unicast IPv4 now •  No IPv6, Multicast, ECMP support

5.  Linux kernel is slow path to ASIC’s fast path

ONL has a Platform Abstraction Layer

•  BIOS does not provide inventory/drivers for all devices –  SFP has a life cycle, like USB; F2B vs. B2F power supplies –  Hot Pluggable devices: PSU, Fan trays, AC/DC power

•  Not all platforms have BIOSs –  Need for per-platform memory maps of LEDs, Fans, Temp

•  ONL Platform provides an Platform Abstraction Layer –  Not tied to Linux subsystem (but could be integrated) –  Goal is for trivial driver impls; leverage existing diag code –  OID based; single threaded, multi-application –  https://github.com/opennetworklinux/ONLP/tree/master/modules/onlp/module/inc/onlp

ONL Provides “Server-like” Management

•  Switches have flash, not hard drives ¡  Problem 1: Maximum flash cycle time limit disk writes ¡  Problem 2: Flash and ram more limited than typical servers ¡  Fix: Use overlayfs to overlay copy-on-write ram disk over flash

•  ONL uses full-featured binaries ¡  For size, most switch OS’s use stripped binaries, e.g., busybox ¡  Bigger binaries uses additional space, but ok with overlayfs ¡  Install/use proper Debian binaries using apt-get ¡  Useful for development or operations, e.g., gcc or Chef/Puppet

ONL Provides A Second-Stage Loader

•  uBoot (PPC) or Grub (x86) is the first stage loader •  ONIE is a 2nd-stage loader; only runs at install-time

•  ONL 2nd Stage Loader •  Runs every time before NOS boots •  Built on Linux – enables a full shell for rescue mode •  Load NOS via ssh/scp/http/tftp/nfs/ftp/local flash

•  Net boot allows centralized NOS image management •  NFS book is very useful

•  (future) Initialize ASIC/front panel ports – inline boot

CPU (x86,PPC)

Misc Hardware (fans, LEDs, SFP, sensors) BRCM

ONL Platform APIs

BRCM SDK

SensorD

Per Switch Stack

ONL Distribution (Linux + stuff)

Next Target: ONL + OpenFlow Agent

Hardware

Platform

Applications Indigo OpenFlow Agent

Indigo OF-DPA Driver

OpenFlow Controller

ONLP Dump

S N M P

Indigo API

Closed Source

OF-DPA

OpenFlow Protocol: v1.0-v1.3


OpenFlow Datapath AbtracTon: OF-‐DPA

Chip SDK

OF-‐DPA

Indigo driver

closed

OF-‐DPA API open

How to Get Involved •  Join the mailing list and provide feedback

–  Compile the code –  Try a new platform –  “Feedback is fine, but patches are preferred” J

•  ONL is a Linux Distribution for bare metal switches •  Growing support for OCP and non-OCP switches •  Now supports ORC forwarding agent •  Indigo-based OpenFlow agent in progress •  Find out more at http://opennetlinux.org

•  Documentation •  Videos •  Pre-compiled binaries

open network linux€“ second stage more feature-full boot loader ... debian wheezy • need to...

Documents