osdc 2014: christopher kunz - software defined networking in an open-source compute cloud
DESCRIPTION
Big networking vendors have discovered network virtualization for themselves. However, not only hardware appliances, but also open-source solutions have various means of virtualising networks.Hosting an IaaS cloud, you are faced with the challenge of isolating VMs, implementing private internal networks, billing and accounting, firewalls and shaping. And all these challenges should not affect the rest of your (non-virtualized) network. Using OpenVSwitch, you can tackle many of these tasks. In this session, we show you the caveats, but also the exciting possibilities of open-source network virtualization in practical examples.TRANSCRIPT
Heartbleed
...and why yours should, too
You are in the right session
_ This is an emergency service announcement _ Due to events that transpired on Tuesday _ I thought it‘d be good to have some info
10.04.14 OSDC 2014 2
About me
_ Dr. Christopher Kunz _ Studied CompSci in Hannover, PhD in 2012 _ Works as a hoster for 15 years
_ Some admin experience
_ Used to do a lot of PHP _ Author, „PHP-‐Sicherheit“, ed. 1-‐3
_ And don‘t get me started about swords!
10.04.14 OSDC 2014 3
About filoo
_ hQps://www.filoo.de _ Quickly-‐growing hosVng company _ Data center in Frankfurt, Germany _ Developed own IaaS middleware
_ QEMU/KVM, OVS, Ceph
_ Offer hosVng, co-‐locaVon, cloud services _ 100% subsidiary of Thomas-‐Krenn.AG _ Visit their booth!
10.04.14 OSDC 2014 4
Heartbleed in a nutshell
_ A bug with a cute name _ ...and not so cute effects _ Pre-‐auth, pre-‐logging universal TLS/SSL bug _ Introduced in OpenSSL 1.0.1a (2012) _ Allows to make 64kb memory dumps of the server‘s memory
10.04.14 OSDC 2014 5
Wait. What?
_ Yes, remote memory dumps _ Due to an unchecked buffer length, a TLS enabled server may dump memory contents to the client _ Limit of 64k per reply _ MulVple replies possible _ Memdump may contain...
_ URLs and GET / POST variables _ Random excerpts from whatever _ Source code of scripts/whatever else _ SSL cerVficate private keys
10.04.14 OSDC 2014 6
About DTLS heartbeats
_ RFC 6520, Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension _ Provides a heartbeat for TLS (TCP) and DTLS (mostly UDP) sessions _ Intended to add stability to unstable connecVons and prevent renegoVaVons _ Implemented in OpenSSL as part of a PhD thesis _ Patch commiQed Dec 15, 2011
10.04.14 OSDC 2014 7
What this bug is not
_ This is not a crypto bug _ At least not in its primary funcVon
_ This is not a fully arbitrary mem disclosure _ Only memory belonging to aQacked daemon can be dumped
_ This is not a remote root hole _ Hence the relaVvely low CVE score of 5.0
10.04.14 OSDC 2014 8
Anatomy of the bug 1
struct { HeartbeatMessageType type; uint16 payload_length; opaque payload[HeartbeatMessage.payload_length]; opaque padding[padding_length];
} HeartbeatMessage;
_ From RFC6520: _ payload_length: The length of the payload. _ payload: The payload consists of arbitrary content.
10.04.14 OSDC 2014 9
Anatomy of the bug 2
_ ssl/d1_both.c, line 1474+: buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;
[..]
memcpy(bp, pl, payload);
_ From: https://github.com/openssl/openssl/commit/4817504d069b4c5082161b02a22116ad75f822b1
10.04.14 OSDC 2014 10
Anatomy of the bug
_ The heartbeat extension allocates payload+19 bytes of memory _ Copies pl bytes of arbitrary user-‐supplied data payload via memcpy() to construct response _ Client sets pl to 65535 _ Client sends only 1 byte of data in payload
_ Response contains 1 byte of client-‐supplied payload _ ...and 64K of RAM from the memcpy() call _ Analysis in: hQp://blog.existenValize.com/diagnosis-‐of-‐the-‐openssl-‐heartbleed-‐bug.html
10.04.14 OSDC 2014 11
Test vulnerability
_ Python script at: hQps://gist.github.com/takeshixx/10107280 _ Can test any SSL/TLS enabled TCP service
_ Has support for StartTLS (-‐s opVon) _ Conveniently dumps 64kb of memory for you
10.04.14 OSDC 2014 12
00d0: 10 00 11 00 23 00 00 00 0F 00 01 01 33 41 31 25 ....#.......3A1% 00e0: 32 43 25 32 32 5F 6D 6F 64 65 25 32 32 25 33 41 2C%22_mode%22%3A 00f0: 25 32 32 6A 73 6F 6E 25 32 32 25 32 43 25 32 32 %22json%22%2C%22 0100: 5F 69 64 25 32 32 25 33 41 25 32 32 70 5F 33 30 _id%22%3A%22p_30 0110: 33 34 35 38 31 38 25 32 32 25 32 43 25 32 32 5F 345818%22%2C%22_ 0120: 63 6F 6E 74 61 69 6E 65 72 25 32 32 25 33 41 30 container%22%3A0 0130: 25 32 43 25 32 32 5F 61 63 74 69 6F 6E 25 32 32 %2C%22_action%22 0140: 25 33 41 25 32 32 76 69 65 77 25 32 32 25 32 43 %3A%22view%22%2C
Memdump
_ From: hQps://twiQer.com/markloman/status/453502888447586304
10.04.14 OSDC 2014 13
Memdump
_ Memory contents is non-‐determinisVc _ SomeVmes exciVng, mostly boring
_ while true do python hb-test.py yahoo.com | grep -C 2 login >> /tmp/out; sleep 1; done"
_ Profit!
10.04.14 OSDC 2014 14
Detect exploitation
_ No logging on the machine _ All exploitaVon is pre-‐logging, pre-‐applicaVon _ IDS vendors are pushing out signatures already
10.04.14 OSDC 2014 15
Affected services
_ Above all, SSL-‐enabled web servers _ Any that uses OpenSSL, anyway
_ Mail servers _ IMAP over SSL, POP over SSL, SMTP over SSL, StartTLS
_ VPN tunnels _ OpenVPN when using cert auth (maybe?) _ PotenVally others
_ IRC servers, XMPP, FTP over TLS _ Android 4.1.1 is vulnerable _ OpenSSH is not vulnerable
10.04.14 OSDC 2014 16
Linux versions affected
_ OpenSSL 1.0.1 a thru f _ Debian Wheezy, Jessie, Sid
_ Fixed for Wheezy & Sid
_ Ubuntu 10.04, 12.04, 12.10, 13.10, 14.04 _ Fixed packages exist
_ RHEL 6 _ Patch exists
_ And all others that ship OpenSSL _ Clients are also vulnerable!
10.04.14 OSDC 2014 17
Other affected stuff
_ Cisco devices: „We use Cisco SSL which is not OpenSSL.“; SSL VPN products potenVally affected _ Juniper has released fixes for their SSL VPN, none for J-‐Web etc. yet _ Big IP? Kemp? Fritz.Box? Your home NAS? _ More info (hopefully) here: hQp://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4
10.04.14 OSDC 2014 18
Mitigation & cleanup
_ First, upgrade to fixed openssl _ apt-‐get install openssl libssl-‐1.0.0
_ Next, restart all services that load old lib _ Use checkrestart or lsof –n | grep DEL | grep ssl
_ If you use staVc binaries, recompile everything _ If you use Google‘s mod_spdy on Apache2.2, don‘t
_ It has its own staVcally linked mod_ssl which is shamefully out of date
10.04.14 OSDC 2014 19
What about certs?
_ It is possible that privkeys have leaked _ If so, you need to revoke&reissue certs _ Some CAs offer free reissue _ If you don‘t have PFS, you have a problem _ AQackers who sniffed your traffic might be able to decode it
10.04.14 OSDC 2014 20
Thank you
_ .Do not despair, there is hope!
_ ...and now, back to our regular scheduled programme!
10.04.14 OSDC 2014 21
hQp://xkcd.com/1353/
Software-defined Networking
In an open-source cloud
Agenda
_ High-‐Level overview: What is this about? _ The use case – virtualized networks for IaaS _ Intro to OpenVSwitch _ How-‐to: Deploy OpenVSwitch _ Frontnet, Backnet, public net _ Firewalling _ Tying it all together
10.04.14 OSDC 2014 23
So what‘s the hype?
_ Sovware-‐Defined Networking is the hype _ I‘m not good with hype
_ Networking is decoupled from bare metal _ EssenVally you virtualize parts of your network _ Control and data plane are decoupled
_ Many vendors jumped on the train _ HP, Cisco, VMWare, you name it
10.04.14 OSDC 2014 24
OpenFlow
_ ImperaVve control _ Switches are dumb – they only forward according to rules _ OpenFlow controllers make the rules _ First packet of each type is sent thru OpenFlow controller _ Subsequent ones go directly through switch
10.04.14 OSDC 2014 25
OpFlex
_ Cisco‘s answer to OpenFlow _ Other vendors on board: Citrix, MSFT, RHAT, Canonical _ Not on board: J, HP, Huawei, vmWare
_ Balance intelligence between switch and controller _ „DeclaraVve control“; just declare how you want it and the switch interprets that rule
_ IETF proposed standard _ Drav-‐smith-‐opflex _ Open APIs
_ AltruisVc goal: Eliminate SPOF (the controller) _ EgoisVc goal: Sell smarter (=$++) switches
10.04.14 OSDC 2014 26
The OSS Contender
_ OpenVSwitch _ Openvswitch.org
_ Open Source _ Apache 2.0 license, non-‐viral _ GPLv2
_ MulVlayer (2,3) virtual switch
_ Supports lots of interesVng features _ VLANs, Ne{low, sFlow, LACP, filtering, ...
10.04.14 OSDC 2014 27
OVS Overview
_ Shamelessly lived from [1]
10.04.14 OSDC 2014 28
ovs-vswitchd
OVS Kernel Module
Control Cluster
ovsdb-server
Off-box
User Kernel
Management Protocol (6632/TCP)OpenFlow (6633/TCP)Netlink
OSVDB
_ Database holds configuraVon items _ DefiniVons for bridges, tunnels, interfaces _ Controller addresses
_ ConfiguraVon is reboot-‐safe _ Custom database system, not MySQLiteMongoDB _ Speaks custom protocol (OSVDB) _ Log based
_ osvdb-‐tool show-‐log shows all changes _ Nivy for debug / change management!
10.04.14 OSDC 2014 29
How ovs works
_ ImperaVve control _ All intelligence is in the controller _ Data path only carries out instrucVons
_ Data Path _ Kernel module _ Licensed under GPLv2
_ Controller _ Lives in userland _ Licensed under Apache 2.0
10.04.14 OSDC 2014 30
Flow flow
_ Everything is a flow _ CombinaVon of input port, VLAN, MAC, IP, TCP/UDP port
10.04.14 OSDC 2014 31
OVS management
_ Command-‐line tools _ Ovs-‐vsctl for switch management _ Ovs-‐ofctl for flow management _ Ovsdb-‐tool for database management
10.04.14 OSDC 2014 32
What‘s our angle here?
_ filoo is a hoster.
_ We host VMs.
_ VMs need networking.
_ See where this goes?
10.04.14 OSDC 2014 33
What we wanted
_ Internet-‐facing front-‐net interface _ Private LAN for VMs _ VM isolaVon _ Firewalling _ Traffic shaping _ Fine-‐grained accounVng _ Live migraVon
10.04.14 OSDC 2014 34
Overview - physical
10.04.14 OSDC 2014 35
Back-‐end switch
Front-‐end switch
Overview - virtual
10.04.14 OSDC 2014 36
Firewall Firewall Firewall
Overview – OVS stack
10.04.14 OSDC 2014 37
OVS OVS OVS
Let‘s get started
_ We usually compile ovs ourselves _ There are also packages in apt _ Those might work or not
_ Download & compile OVS _ Latest stable: 2.1.0, latest LTS: 1.9.3 _ ./boot.sh && ./configure && make && make install
_ Kernel module from 3.3+ _ Enable in Kernel Networking -‐> OpVons -‐> Open Vswitch _ modprobe openvswitch
10.04.14 OSDC 2014 38
Let‘s get started 2
_ Set up ovs db _ Ovsdb-‐tool create conf.db vswitch.ovsschema _ Conf.db is in /usr/localetc/openvswitch _ /usr/src/openvswitch-‐1.9.3/vswitchd/vswitch.ovsschema
_ Make sure ovs-‐vswitchd and ovsdb-‐server start before networking _ Add startup entries to rc.local _ Remove networking from rc.d _ start networking in rc.local
10.04.14 OSDC 2014 39
Initial bridges
_ Front-‐net vlan: 199 _ Same procedure for back-‐net VLAN _ Add bridges
_ ovs-‐vsctl add-‐br vmbr1 _ ovs-‐vsctl add-‐port vmbr1 vlan199 tag=199 _ ovs-‐vsctl set interface vlan199 type=internal
_ Log in via IPMI _ ovs-‐vsctl add-‐port vmbr1 eth1 _ Machine is offline now
_ Modify physical switching
10.04.14 OSDC 2014 40
VM networking
_ We use KVM/QEMU _ Add the TAP interface
_ /sbin/ip tuntap add dev tap1i0d0 mode tap user fcms _ qemu-‐system-‐x86_64 ... -‐device rtl8139,mac=00:F1:70:00:00:10,netdev=vlan0d0 -‐netdev type=tap,id=vlan0d0,ifname=tap1i0d0
_ Bring up the port _ /usr/local/bin/ovs-‐vsctl add-‐port vmbr0 tap1i0d0 199 other_config:stp-‐enable=false
10.04.14 OSDC 2014 41
From TAP to port to flow
_ We have a tap interface tap1i0d0
_ Find the corresponding bridge port: _ ovs-‐ofctl show vmbr0 | grep tap1i0d0 _ 1820(tap1i0d0): addr:fa:7a:67:e3:5d:�
_ Now we have a port number: 1820
_ We use this port for flow management
10.04.14 OSDC 2014 42
Multiple interfaces
_ Add more TAP interfaces _ Assign one VLAN per customer _ Internal network across VMs on same node
_ Make VLAN known on inter-‐node switches _ Via whatever switch automaVon you have
_ Cross-‐node internal networking _ VLAN limits apply – hard cut at ~4090 _ Overlay networks to the rescue
10.04.14 OSDC 2014 43
Prevent MAC spoofing
_ PORT=1820 _ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" arp idle_Vmeout=0 priority=39500 acVon=resubmit("${PORT}",2)“ _ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=2 arp priority=200 idle_Vmeout=0 arp_sha=00:F1:70:00:00:10 nw_src=192.168.1.1 acVon=normal" _ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=2 priority=100 idle_Vmeout=0 acVon=drop"
10.04.14 OSDC 2014 44
We know this MAC because we control the hypervisor!
We know this address too!
Caveats for MAC/ARP
_ SomeVmes you want customers to spoof _ HA soluVons that switch „cluster IP addresses“ _ You can cater for this in case you know the corresponding MACs _ Assign sequenVal MACs and wildcard _ Or set specific rules _ OpVonal „HA feature“ for VMs _ Never allow customers to wildcard here!
10.04.14 OSDC 2014 45
Firewalling with flows
_ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=1 tcp idle_Vmeout=0 nw_dst=192.168.12.13/32 nw_src=192.168.1.123/32 tp_dst="80" priority=38000 acVon=drop“ _ From 192.168.1.123 _ To 192.168.12.13 _ Port 80 _ Drop
10.04.14 OSDC 2014 46
Port ranges
_ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=1 tcp idle_Vmeout=0 nw_src=192.168.1.123/32 nw_dst=192.168.12.13/24 tp_src="0x05E8/0xFFFC" priority=37960 acVon=drop“ _ Source 192.168.1.123 _ DesVnaVon 192.168.12.0/24 _ Source port = 0x05E8/0xFFFC _ 0x05E8/0xFFFC = 1512/65532 _ Port 1512 – 1516
_ OVS 1.11 supports „Megaflows“, i.e universal wildcarding
10.04.14 OSDC 2014 47
Default accept
_ ovs-‐ofctl add-‐flow vmbr0 "in_port="${PORT}" table=1 priority=100 acVon=normal“ _ Fallthru rule _ Match everything else
10.04.14 OSDC 2014 48
Accounting
_ We grab interface counters from the tap interfaces _ You can also use Ne{low/sFlow or ipfix _ We didn‘t go there yet, experiences welcome
10.04.14 OSDC 2014 49
Shaping
_ Simple shaping: _ ovs-‐vsctl set Interface tap0 ingress_policing_rate=100000 _ ovs-‐vsctl set Interface tap0 ingress_policing_burst=1000
_ QoS policies: _ ovs-‐vsctl set port eth1 qos=@newqos \ id=@newqos create qos type=linux-‐htb \ other-‐config:max-‐rate=200000000 queues=0=@q0,1=@q1 \
_ We don‘t do QoS policies, shaping works mostly as intended
10.04.14 OSDC 2014 50
Live migration
_ We don‘t actually do OVS‘s own live migraVon _ Start VM on target host in suspend-‐to-‐RAM mode _ Stop VM on losing host; down interface _ Resume VM on target host
_ There are live migraVon mechanisms in OVS _ L2 based _ Inter-‐OVS GRE tunnel _ Honestly, I have no clue.
10.04.14 OSDC 2014 51
Thank you
_ I hope you learned something _ If not, I hope you had a laugh at my expense _ If neither, I‘m really sorry. Beer?
_ QuesVons?
10.04.14 OSDC 2014 52
Literature
_ [1] hQp://openvswitch.org/slides/OpenStack-‐131107.pdf – OVS Deep Dive _ OVS IntroducVon: hQp://horms.net/projects/openvswitch/2010-‐10/openvswitch.en.pdf
10.04.14 OSDC 2014 53