vmug - nsx architettura e design
TRANSCRIPT
VMware NSX
Architettura e Design
Luca Morelli Senior Systems Engineer
Qualche Info sullo Speaker…
© 2016 VMware Inc. All rights reserved. 2
• Nato a Catanzaro, la città delle 3 V, circa 38 anni fà
• Ingegnere Informatico – Università di Rende
• Nell’IT da circa 15 anni – Esperienze in Spagna, Francia, Olanda e altri paesi
• Iniziato con lo sviluppo software quindi prevendita da quasi 9 anni
• Quasi 7 anni con un vendor di rete “fisica”
• “Virtualizzato” dal Gennaio 2015
• Appassionato di subacquea, apnea, running (sto preparando la mia prima maratona), arrampicata e della mia splendida compagna
https://it.linkedin.com/in/lumorelli
1 NSX Refresher and Use Cases
2 NSX Components
3 NSX Design
4 Summary
Agenda
3
“Non” Agenda
4
• Marketing (Well, I will do my best but sometimes I cannot control it J)
• A full NSX overview
• Security design and operations recommendations
• Multi-sites scenarios (e.g. Disaster recovery)
• Deep technical aspects
• Specific jargon and acronyms (see Marketing comments)
5
La virtualizzazione del Networkè divenuta una tendenza.
• Modello operativo collaudato
• Facile dispiegamento
• Strumenti per la gestione operativa
Evoluzione del Software-Defined Networking
2008 2015
Ricerca• OpenFlow• Ambito Universitario• Sperimentale
Prodotti e Architetture• Overlay networks • Control planes centralizzati• Service providers & enterprise
2010 2012 2014
Battaglie Architetturali
2016
NSX - Distributed Services in the Hypervisor
Applications
Virtual Machines
Virtual Networks
Virtual Storage
Data Center Virtualization
Location Independence
Software
Hardware
L2 Switching
L3 Routing
Firewalling/ACLs
Load Balancing
Automated operational model of the SDDC
Network & Security Services Now in the Hypervisor
Pooled compute, network and storage capacity; Vendor independent, best price/perf; Simplified config and mgt.
ComputeCapacity
NetworkCapacity
Storage Capacity
© 2016 VMware Inc. All rights reserved.
Virtual Network Services to the Virtual Switch
© 2016 VMware Inc. All rights reserved.
NSX vSwitch
With NSXBefore NSX
Default Gateway
UCS Fabric A UCS Fabric B
UCS Blade 1
vswitch
6 wire hops 6 wire hops
UCS Fabric A UCS Fabric B
UCS Blade 1 UCS Blade 2
vswitch vswitch
UCS Fabric A UCS Fabric B
0 wire hops
UCS Fabric A UCS Fabric B
UCS Blade 1 UCS Blade 2
With NSXBefore NSX
East-West Routing / Same host East-West Routing / Host to host
2 wire hops
NSX vSwitch
UCS Blade 1
The Advantage of Distributing ServicesDistributed Logical Routing - More efficient networking, fewer hops
Default Gateway Default Gateway Default Gateway
© 2016 VMware Inc. All rights reserved.
NSX Edge Services Gateway: Integrated Network Services
….
FirewallLoad Balancer
VPN
Routing/NATDHCP/DNS relayDDI
VM VM VM VM VM
• Integrated L3 – L7 services• Virtual appliance model to provide
rapid deployment and scale-out
Overview
• Real time service instantiation
• Support for dynamic service differentiation per tenant/application
• Uses x86 compute capacity
Benefits
VLAN 20Edge Uplink
External Network
Physical Router
Web1 App1 DB1 Webn Appn DBn
NSX Edge
VXLAN 5020Transit Link
Distributed Routing
Routing Peering
10
How it looks like a Basic NSX Topology
…
High Scale Multi Tenant Topology
External Network
Tenant 1Web Logical Switch App Logical Switch DB Logical Switch
…
Web Logical Switch App Logical Switch DB Logical Switch
Tenant NSX Edge Services Gateway
NSX Edge X-Large (Route Aggregation Layer)
Tenant NSX Edge Services Gateway
VXLAN Uplinks (or VXLAN Trunk)
VXLAN Uplinks (or VXLAN Trunk)
VXLAN 5100 Transit
11
1 NSX Refresher and Use Cases
2 NSX Components
3 NSX Design
4 Summary
Agenda
12
NSX Architecture and ComponentsCloud Consumption • Self Service Portal
• vCloud Automation Center, OpenStack, Custom
Data Plane
ESXi Hypervisor Kernel Modules
Distributed Services
• High – Performance Data Plane• Scale-out Distributed Forwarding Model
Management PlaneNSX Manager
• Single configuration portal• REST API entry-point
Control Plane• Manages Logical networks• Control-Plane Protocol• Separation of Control and Data Plane
Logi
cal N
etw
ork
Phys
ical
Net
wor
k
vCenter Server
13
NSX Logical Router Control VM
NSX Controller
NSX Edge
FirewallDistributed
Logical RouterLogicalSwitch
VPN
…
…
NSX is AGNOSTIC to Underlay Network Topology
§ NSX Manager deployed as a virtual appliance• Minimum 16 GB and 4 vCPU• Larger scale use 24 GB and 8 vCPU
§ 1:1 relation with vCenter§ Availability Considerations – same as any appliance model
• Resiliency of NSX Manager provided by vSphere HA • Catastrophic failure of NSX Manager is rare, however periodic
backup is recommended to restore to the last known configurations• During the failure all existing data plane connectivity continue to
work since data and management plane are separated
§ NSX Manager also holds DFW rules logic, flow monitoring and local logging data
NSX ManagerDeployment Considerations
14
Management Cluster
WANInternet
L3
L2
Host 1
Host 3
Host 2
Host 4
§ Provide control plane to distribute network information to ESXi hosts
§ NSX Controllers are clustered for scale out and high availability
§ Network information is distributed across nodes in a Controller Cluster (slicing)
§ Remove the VXLAN dependency on multicast routing/PIM in the physical network
§ Provide suppression of ARP broadcast traffic in VXLAN networks
VXLAN 5000
Logical Router 1VXLAN 5001
Logical Router 2
VXLAN - 5002
Logical Router 3
Controller VXLAN Directory Service
MAC table
ARP table
VTEP table
NSX Controllers Functions
§ Controller nodes are deployed as virtual appliances• 4 vCPU, 4GB of RAM per node• CPU Reservation of 2048 MHz• No memory reservation required
§ 3 Controller nodes is the only supported configuration§ Storage resiliency to ensure not all nodes fails due to storage component failure
§ Controller majority is required for having a functional controller cluster• Data plane activity maintained even under complete controller cluster failure
§ By default the DRS and anti-affinity rules are not enforced for controller VMs• The recommendation is to manually enable DRS and anti-affinity rules• Minimum 3 host is required to enforce the anti-affinity rule
§ Storage resiliency is required in terms of access, paths, disks and LUNs• Making sure not all three controller does not end in the same storage or partition such
a failure of storage component should not constitute a failure of all three control VM• Oversubscription of IO does not result in contention among three controller VMs
NSX Controllers
16
Management Cluster
WANInternet
L3
L2
Host 1
Host 3
Host 2
Host 4
Distributed Logical Routing Components
§ The Distributed Logical Router Control Plane is provided by a per instance DLR Control VM and the NSX Controller
§ Dynamic Routing Protocols supported with DLR –OSPF and BGP
• Control VM forms the adjacencies with Edge node
§ Communicates with NSX Manager and Controller Cluster
• NSX Manager sends LIF information to the Control VM and Controller Cluster
• Control VM sends Routing updates to the Controller Cluster
§ DLR Control VM and NSX Controller are not in the data path
§ High availability supported through Active-Standby configuration
§ Can exist in edge cluster or in compute cluster
Logical Router Control VM
§ Logical Interfaces (LIFs) on a Distributed Logical Router Instance• There are internal LIFs and uplink LIFs• VM Default Gateway traffic is handled by LIFs on the
appropriate network• LIFs are distributed across every hypervisor prepared for
NSX• An ARP table is maintained per LIF
§ vMAC is the MAC address of an internal LIF• vMAC is same across all hypervisors and it is never seen
by the physical network (only by VMs)
• Routing table on each ESXi hosts is programed via controller
DLR Kernel Module
vSphere Host
LIF1 LIF2
Transit VXLAN
Uplink
1 NSX Refresher and Use Cases
2 NSX Components
3 NSX Design
4 Summary
Agenda
18
NSX sizing is based on a modular footprint• NSX footprint is sized based on customer requirements
• Once these requirements are defined, then map NSX components to infrastructure resources
• Similarly separate VCs for Management and Compute is not an NSX requirement
§ Network Virtualization with NSX enables greater flexibility regardless of physical network design
§ NSX capabilities are independent of network topology
19
• Flexibility with NSX components– Controller are in management cluster with single VC– Edge Resources are flexible in terms of vCPU and memory– NSX stack is flexible – DFW only vs full stack– Three tire licensing allow flexibility that maps to cost and growth
VMware NSX for vSphere Network Virtualization Design Guide ver 3.0https://communities.vmware.com/docs/DOC-27683
VMkernel Networking – L2 OR L3 Topology Agnostic
20
vSphere Host (ESXi)
Layer 2 or Layer 3 Uplinksbased on topology
VLAN Trunk (802.1Q)
VLAN 100
Mgmt10.100.1.25/26DGW: 10.66.1.1
VLAN 101
vMotion10.101.1.25/26DGW: 10.77.1.1
VLAN 102
Storage10.102.1.25/26DGW: 10.88.1.1
VLAN 103
VXLAN10.103.1.25/26DGW: 10.99.1.1
SVI 100: 10.100.1.1/26SVI 101: 10.101.1.1/26SVI 102: 10.102.1.1/26SVI 103: 10.103.9.1.1/26
Span
of V
LAN
s
Span
of V
LAN
s
VMkernel Interface
Transport Zone, VTEP, Logical Networks and VDS § Transport Zone: collection of VXLAN prepared ESXi
clusters
§ Normally a TZ defines the span of Logical Switches (Layer 2 communication domains)
§ VTEP (VXLAN Tunnel EndPoint) is a logical interface (VMkernel) connects to TZ for encap/decap VXLAN traffic
§ VTEP VMkernel interface belongs to a specific VLAN backed port-group dynamically created during the cluster VXLAN preparation
§ One or more VDS can be part of the same TZ
§ A given Logical Switch can span multiple VDS
§ Multiple VDS allows flexibility, connectivity and operational control
§ Need to consider L2 vs L3 VTEP Design
§ IP addressing
§ Bandwidth and availably considerations VTEP Design21
vSphere Host
VXLAN Transport Network
10.20.10.10
Host 1
10.20.10.11VTEP1 VTEP2
VM
VXLAN 5002 MAC2
vSphere Host10.20.10.12
Host 2
10.20.10.13
VM
MAC4
VM
MAC1
VM
MAC3
VTEP3 VTEP4
vSphere Distributed Switch vSphere Distributed Switch
vSphere Host
VXLAN Transport Network
10.20.10.10
Host 1
10.20.10.11VTEP1 VTEP2
VM
VXLAN 5002 MAC2
vSphere Host10.20.10.12
Host 2
10.20.10.13
VM
MAC4
VM
MAC1
VM
MAC3
VTEP3 VTEP4
vSphere Distributed Switch vSphere Distributed Switch
VTEP Design § Number of VTEPs deployed depends on teaming mode
• Single VTEP for LACP or Explicit Failover• Multiple VTEPs (based on number of host uplinks) for Src-ID teaming
option
§ Single VTEP is sufficient for • Workloads do not drive more than 10G of throughput• Simple operational model • Deterministic traffic mapping to uplink is desired (Explicit Failover only)
§ Multiple VTEPs or LACP is required for § Workloads require > 10G of throughput• Allows flexibility of choosing teaming mode for other traffic types (non-
LACP)
• IP addressing for VTEP • Common VTEP subnet for a L2 fabric• Multiple VTEP subnets (one per rack) for L3 fabrics
§ IP Pools or DHCP can be use for IP address assignment
22
Design Considerations – VDS and Transport ZoneManagement Cluster
Edge Cluster
WebVM
WebVM
VM
VM
WebVM
WebVM
VM
VM
Compute A Compute N
NSX ManagerController Cluster NSX Edges
VXLAN Transport Zone Spanning Three Clusters
Compute VDS Edge VDS
VTEPvSphere Host
vSphere Host
192.168.230.100
192.168.230.101
Compute Cluster 1vSphere Host
192.168.240.100
Compute Cluster N
vSphere Host
192.168.240.101
vSphere Host
vSphere Host
192.168.220.100
192.168.220.101
VTEP VTEP
Active
VPN
VDS Uplink Design• NSX create dvUplink port-groups for VXLAN enabled hosts. This
uplink connectivity carrying VXLAN traffic
• Must be consistent for all hosts belonging to the VDS
• For the VXLAN traffic, the choice in teaming mode depends on• Simplicity• Bandwidth requirement
• Recommended teaming mode is “route Based on Originating Port”.
• LACP teaming mode is discouraged:• LACP teaming mode only allows single VTEP• LACP teaming forces all the other traffic types to use the
same teaming mode• LACP requires additional configuration and operational
consistency • Having multiple VDS for compute and Edge allow flexibility
of teaming more for uplink configuration
24
Teaming and Failover Mode
NSX Support
Multi-VTEPSupport
Uplink Behavior2 x 10G
Route based on Originating Port
✓ ✓ Both Active
Route based on Source MAC hash
✓ ✓ Both Active
LACP ✓ × Flow based –both active
Route based on IP Hash (Static
EtherChannnel)
✓ × Flow based –both active
Explicit Failover Order ✓ × Only one link is active
Route based on Physical NIC Load
(LBT)
× × ×
DC Topologies – NSX is Agnostic
25
VLANs & IP Subnet Defined at each ToR SVI Interface VLAN ID IP Subnet
Management 100 10.100.R_ID.x/24
vMotion 101 10.101.R_ID.x/24
Storage 102 10.102.R_ID.x/24
VXLAN 103 10.103.R_ID.x/24
VLANs & IP Subnet Defined for POD A
SVI Interface VLAN ID IP Subnet
Management 100 10.100.A.x/24
vMotion 101 10.101.A.x/24
Storage 102 10.102.A.x/24
VXLAN 103 10.103.A.x/24
VLANs & IP Subnet Defined for POD B
SVI Interface VLAN ID IP Subnet
Management 200 10.200.B.x/24
vMotion 201 10.201.B.x/24
Storage 202 10.202.B.x/24
VXLAN 103 10.103.B.x/24
VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)
VLAN ID 100, 101 & 102 Scope VLAN ID 200, 201 and 203 Scope
Compute Cluster A32 Hosts
Compute Cluster B32 Hosts
POD A POD BL3
L2
VLAN ID 100, 101 & 102 Scope VLAN ID 200, 201 and 203 Scope
VXLAN VLAN ID 103 - Transport Zone Scope (extends across ALL PODs/clusters)
Compute Cluster A32 Hosts
Compute Cluster B32 Hosts
L3
L2
Cluster Design with NSX
26
• Flexible DC Design– Dedicated VC for management cluster– Dedicated VC that reside in management cluster but
manages NSX domain (Edge and Compute)• Avoid circular dependencies – manager should be always
be outside of the infrastructure it manages• Mobility of mgmt. cluster for remote operations• Integration with existing VC • Ability to have more than one NSX-domains with a
dedicated VC for mgmt • Upgrade of VC not affecting – non-NSX domains. • SRM and other explicit state management is possible
– Separate Rack and ToR for• Management Cluster • Edge Cluster – Covered under the Edge design
• Simple DC Design§ Single VC for managing all components§ Separate cluster for compute § Separate Edge & MGMT Cluster§ Two VDS that spans VXLAN – compute
and Edge§ The Edge and MGMT can share the
same rack/ToR
§ Resources and components that may be hostsed Edge clusters
• Edge VMs for N-S Traffic
• Control-VM for DLR routing
• Services VM - Load Balancer and VPN
• Optional - Controllers, Monitoring, Log Insight VMs
§ Type of workload & oversubscriptions desired
• N-S vs. east-west
• Host uplinks, NIC & CPU
§ Size of DC – Small, Medium & Large
§ Single vs. multi-VC and DR with SRM
§ Availability and Scaling
• Limiting VLANs sprawl for peering with ToR
• Rack Availability and BW considerations
• Oversubscription options
27
DC Design ConsiderationEdge Cluster Design
Management&
Edge Clusters
Separate ComputeCommon Edge and Management Cluster
with NSX
WANInternet
L3
L2
ComputeCluster
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
§ Edge Cluster design and capacity planning depends on many factors
§ Components in Edge clusters• Edge VMs for N-S Traffic
• Control-VM for DLR routing
• Services VM - Load Balancer and VPN
• Optional - Controllers, Monitoring, Log Insight VMs
§ Type of workload & oversubscriptions desired• N-S vs. east-west
• Host uplinks, NIC & CPU
§ Single vs. multi-VC and DR with SRM
§ Availability and Scaling• Limiting VLANs sprawl for peering with ToR
• HA, DRS, BW and Rack Availability
§ Size of DC – Small, Medium & Large
28
Edge Cluster Design
Should I mix clusters?
Cluster Design Choices and Components Sizing
WANInternet
L3
L2
Host 1
Host 3
Host 2
Host z
Host y
Host x
What type of Edge mode is right?
Where would I put control VM?
How do I scale?
Growth & advance functions?
What is the minimum configuration?
Edge Cluster Design
29
• Minimum four hosts Cluster• Two host to hold two ECMP Edge VMs• Other two for DLR Control-VM• Not to mix ECMP and DLR-Control VM
• Avoiding race condition due to dual failures of components during host failure
• Anti-affinity is automatically enabled for DLR Control-VM• Need anti-affinity and DRS protection group for ECMP VMs
• Host Uplink & VDS • Use “SRC_ID with Failover” teaming for VXLAN traffic • Route peering maps to unique link
• Performance & Sizing• Intel, Broadcom or Emulex supporting VXLAN offload
including RSS and TSO offload• Higher CPU Clock is better• Up to 4 ECMP VMs per hosts with 4x10 GB NIC
• Oversubscription Dependent on• Upstream connectivity from the ToR• Application requirements• Density of Edge VM per hosts
L3
L2
VLAN 10 VLAN 20
Host 1 Host 2
ESG-01 ESG-02 ESG-03 ESG-04
Host 3 Host 4
VM DRS Group 1
VM DRS Group 3
VM DRS Group 2
Bridge Instance
L3
L2
Host 1
VLAN 10 VLAN 20
L3
L2
VLAN 10 VLAN 20
No over subscription 1:2 over subscription
Host 2
Host 3 Host 4Host 1 Host 2
30
Active-Standby Edge Design
L3 - ToR
Routing Adjacency
vSphere Host vSphere Host
VXLAN 5020Transit Link
Active-StandbyStateful
FW/NAT/LB
• Active-Standby Edge Services Gateway enables stateful services • Perimeter FW, NAT, LB, SSL-VPN, North-South routing• Deployed in pair with heartbeat and synchronization of
services state• Heartbeat and sync both use the same internal vNIC • L2 connectivity required between active and standby• Form factor – X-Large to Compact (one license)
• Multi-interface routing Support – OSPF & BGP• Must tune protocol timers to 40/120(hello/hold timer)
• Anti-affinity is automatically created• Active and Standby Edges are placed on different hosts• Minimum three hosts are recommended• Automatic resource reservation – from 6.2.4 onward
• Multiple instance to Edge can be deployed• LB Edge can be deployed near application tire• Multiple tenants can have separate Edge services
31
ECMP Based Edge Design -
ECMP EdgesNon-Stateful
VXLAN
VLAN
Transit VXLAN
E1 E2 E7 E8
R1 R2
External Network
VLAN 10
VLAN 20
ECMP Active NSX Edges
Customer Routers
…
• ECMP Edge enables scalable north-south traffic forwarding services • 8 instances of Edge • Stateful services are not supported due to
asymmetric traffic behavior• No heartbeat and sync between Edge nodes • L2 connectivity required for peering to
physical• Form factor – X-Large to Compact (one
license)• Multi-interface routing Support – OSPF &
BGP• Aggressing timers tuning supported 1/3
(hello/hold timer)• Anti-affinity configuration is required
• Minimum three hosts are recommended• Multiple tenants can have separate Edge
services
High Availability with Edge ClusterEdge Active-Standby Guideline• Minimum three hosts• Anti-affinity automatic enforced for Edge
Active-Standby mode • Heart beat timers can be reduced from 15
seconds default to 9 seconds– Internal link for heartbeat for failure detection
• vSphere HA and standby takeover with HB failure routing timer 40/120 Sec.
• Control VM active-standby anti-affinity is automatic– HB timers for control VM typically do not
requires modification
32
ECMP HA Guideline• Minimum 3 hosts, more if higher BW (VM)
is required• DRS and HA policy governed by how
many VMs per host (dependency on BW and uplinks per hosts – 2x10 or 4x10)
• Active Control-VM and Edge VM can not be in same host to avoid dual failure
• Aggressive timers support up to 1/3 sec• Control VM active-standby anti-affinity is
automatic– HB timers for control VM typically do not
requires modification
• In either Edge mode, the vMotion is supported and does not have impact on control plan
ECMP with DLR and Edge§ ECMP support on the DLR and on the NSX Edge
• Both have the capability of installing in their forwarding tables up to 8 equal cost routes toward a given destination
• 8 NSX Edges can be simultaneously deployed for a given DLR+Edge combination - per tenant
§ Increase the available bandwidth for North-South communication • Reduces the traffic outage in an ESG failure scenario
(only 1/N of the flows are affected)
§ The routing protocol timer can be reduced up to 1 second for hello and 3 second for hold for both protocol OSPF and BGP
§ Recommended best practices is not to have ECMP Edge VM and DLR control VM on the same host to avoid the dual failure and triggering much longer outage due to aggressive timers • Anti-affinity is automatic between active-standby
control VM and enable DRS protection between ECMP edge VM and DLR control-VM
• Disabled GR on ECMP Edge to improve convergence
Web DB
DLR
E1
Physical Router
App
DC Fabric
Active ECMPE8
VLAN 20
Transit VXLAN
Active Standby
RoutingAdjacency…
VLAN 10
Edge HA Models Comparison – BW, Services & ConvergenceActive/Standby HA Model
Bandwidth Single Path(~10 Gbps/Tenant)
Stateful Services Supported - NAT, SLB, FW
Availability Low convergence with stateful services enabled
ECMP Yes with Two Uplinks on Edge HA
ECMP Model
Bandwidth Up to 8 Paths(~80 Gbps/Tenant)
Stateful Services Not Supported
AvailabilityHigh
~ 3-4 sec with (1,3 sec) timers tuning
E1
Physical Router
Active StandbyE2
Routing Adjacency
Web DB
DLR
AppActive Standby
DLRControl VM
…E8E3E1
Physical Router
E2
Routing Adjacencies
Web DB
DLR
App
Active Standby
DLRControl VM
Routing Protocol & Topology• Enterprise vs. DC Routing
– Core & Aggregation Routing – Handled by backbone team– DC Spine-Leaf handled by DC Team– Protocol choices and its connectivity differs in DC
• 10G Links, Convergence is TCP centric – not sub-seconds
• DC Routing Protocol Choices– Mostly OSPF, BGP is prevalent in large scale design and
growing acceptance in enterprise DC fabric
• DC Routing Design and NSX Scope– NSX domain is connected as Edge routing
• Edge routing features is sufficient for most needs• Does not act as transport network
– Simplified protocol features and connectivity for NSX• Use consistent one protocol end-to-end between ESG and
physical and DLR to ESG• Adopting the established routing best practices
– Summarization– reducing the routing churn as well as protecting black hole of traffic– Treating NSX as another autonomous system or stub topology
35
OSPF/BGP
L3
L2
WAN/Internet
AS 65501
AS 65002
NSX Routing Domain
DLR
Compute
NSX Component Mapping
36
Separation of compute, management and Edge function with following design advantage
• Management Cluster• Can co-exist with Edge Cluster in same UCS Chassis• Minimum three hosts – more if needed• LACP can be used on rack-mount
• Edge Cluster• Should be independent UCS C series• Edge VM for North-south traffic • Active-standby Control-VM• Can hold NSX Controller is optimization of resources
is desired
• Compute Cluster
• VXLAN is enabled per cluster• Can co-exist with physical bare-metal compute
Function NSX ComponentsRecommended
Cluster Designation
ManagementPlane NSX Manager & VC Management
Cluster
Control Plane
NSX Controller Cluster
Management or Edge Cluster
Logical Router Control VM Edge Cluster
Data Plane East-West
VXLAN forwarding -compute and edge
VDS kernelcomponents &
DLR(Distributed Logical Routers)
Compute and Edge Cluster
Data PlaneNorth-South
ECMP Edge or Edge Services Edge Cluster
Bridging Traffic DLR Control VM Edge Cluster
§ Mgmt Cluster is typically provisioned with limited severs:• Follows minimum CPU & Memory for management components
• VC, Mgmt, vRA, vROPs etc.
• Static allocation, idle capacity can be used for other resources
§ The single rack design still requires redundant uplinks from host to ToR carrying VLANs for management• LACP teaming mode can be helpful
§ Larger design tends to have dedicated management cluster • Exclude management cluster from preparing VXLAN
§ In a smaller design, management and Edge cluster can be combined
• NSX manager and Controllers automatically excluded from DFW functions.
• Put vCenter server in DFW exclusion list
• Use resource reservation to protect the Edge VMs
LeafL2
L3
VMkernel VLANs
VLANs for Management VMs
L2
VMkernel VLANs
DC Fabric
802.1Q Trunk
Management Cluster Connectivity
L2
37
Management Cluster Deployment Considerations
• Rack based vs. multi-rack (horizontal) stripping• Availability vs. localized domain – CPU & mobility constraint &
simplification of connectivity (IP, VTEP, Automation)
• Lifecycle of the workload drives the consideration for• Growth, availability and changes in the application flows• Multi-rack, zoning ( type of customer, tenancy etc.)
• Typically rack connectivity is streamlined and repeated • Same four VLANs typically streamlines the configuration of ToR• Connectivity to the fabric and requirement for additional capacity
remains the same since its abstracted from infrastructure
• Workloads type, compliance and SLA can be met via• Cluster separation• Separate VXLAN network• Per tenant separation routing domains• DRS
Compute Cluster Connectivity
38
DC Design ConsiderationCompute Cluster
Management
WANInternet
L3
L2
ComputeCluster
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Compute Clusters
L3
L2
DC Fabric
Edge Clusters
Organizing Compute, Management & Edge Separation of compute, management and Edge function with following design advantage
• Managing life-cycle of resources for compute and Edge functions
• Ability to isolate and develop span of control• Capacity planning – CPU, Memory & NIC• Upgrades & migration flexibility
• High availability based on functional need• Workload specific SLA (DRS & FT)• Network centric connectivity – P/V, ECMP• vMotion boundary
• Automation control over area or function that requires frequent changes
• app-tier, micro-segmentation & load-balancer
Three areas of technology require considerations• Interaction with physical network• Overlay (VXLAN) impact• Integration with vSphere clustering
Management
WANInternet
L3
L2
Compute
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Compute
L3
L2
DC Fabric
Edge
NSXEDGE
NSXEDGE
NSXEDGE
NSXEDGE
Large DC Cluster Design
40
• Workload characteristics– Variable– On-demand– Compliance requirements
• For cross-VC and SRM Deployment– Separation of management cluster is inevitable
• Large scale Edge Cluster Design– Dedicated minimum four hosts– Minimum four ECMP Edge (Quad Large) 40 GB total BW
– Separate host with DRS protection between ECMP Edge VM and Active Control-VM
– Capacity for services VMs
• Edge VM CPU Guideline– Ideally >= 2.6 GHz with 10 core to hold min two ECMP VMs
for 20 GB (2x10 NIC) bandwidth– Higher cores can be used to consolidate VMs but may need
4x10 GB network ports– Keep the CPU/Socket consistent for Edge cluster to have
flexibility
Edge Cluster
Separate Management Compute& Edge and Cluster with NSX
Management
WANInternet
L3
L2
Compute
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Host 1
Host 3
Host 2
Host 6
Host 5
Host 4
Compute
L3
L2
DC Fabric
Cluster Design with Medium DC
41
• Mixing compute and edge workload requires– Balanced Compute workload can be mixed with Edge VM
resources– However the growth of compute can put additional burden on
managing resource reservation to protect the Edge VM CPU
• Collapsing edge OR compute with management components (VC and NSX manager) – Requires management component to be dependent on VXLAN since
VXLAN enablement is per cluster bases– Expansion or decoupling of management required for growth
• moving management cluster to remote location• Having multiple VCs to manage separation
• Mixing Edge and Management is a better strategy– Consistent static requirements of the resources – mgmt. is relatively
time idle resources compared to compute workload
• For growth consider separation of edge and mgmt. cluster
Management&
Edge Clusters
Separate ComputeCommon Edge and Management Cluster
with NSX
WANInternet
L3
L2
ComputeCluster
Host 1
Host 3
Host 2
Host 32
Host y
Host x
Cluster Design with Medium DC - Continue
42
• Small to medium cluster can utilize the edge service gateway features where – N-S BW is not more then 10 G– Desire to reduce external FW usage with Edge FW
functionality– Using built in Load Balancer– Use VPN or SSL functionality
• Edge Services Sizing– Start with Large (2 vCPU) if the line rate BW is not
required– Can be upgraded to Quad-Large (4 vCPU) for growth
in BW
• Consider LB in single arm mode to be near the application segments
• Decouple the need to Edge service mode choice if only LB service is required
Management&
Edge Clusters
Separate ComputeCommon Edge and Management Cluster
with NSX
WANInternet
L3
L2
ComputeCluster
Host 1
Host 3
Host 2
Host 32
Host y
Host x
Small Design Considerations• Understanding the workload that impact NSX
component selection – Edge workload is CPU centric with consistent memory –
except in L7 load-balancer Edge services– Edge resources requires external connectivity to VLANs
thus restricting it location to avoid VLAN sprawl
• Single cluster for small design, expand to medium with separation of compute cluster– Single cluster can start with DFW only design
• NSX Manger is the only component required• VDS license comes with NSX
– Progress to full stack for other services such as FW, LB, VPN and VxLAN• ESG in active-standby – Large form factor• Quad-large if needed for firewall throughput• Static routing for simplicity and reduced need of
deploying control-VM
43
Single Cluster with NSX
WANInternet
L3
L2
Host 1
Host 3
Host 2
Host 32
Host y
Host x
Small, Medium and Large Virtualized DC – NSX Scales Consistently
44
Sizing VC Workload Edge Type N-S BW GB Cluster Choice Requirement ResourceReservation
Small 1 ConsistentLarge - 2 vCPU
< 20 G Collapsed Harder to
separate Mgmt. later
Need for Edge VMsESG or ECMP
Med 1Consistence
Some on-demand
Large to Quad 2 or 4 vCPU
< 40G Mgmt./EdgeGrowth not likely, No other smaller
DC
Need for Edge VMs
ESG or ECMP
Med (with multiple DC or compute
growth)
>1 On-demandWith DR
Quad – 4 vCPU
<= 40G
Separate Mgmt., Edge and ComputeCluster
Growth or other DC integration
must
If needed formix useECMP for N-S
ESG for local LB
Large >1
Variable, on-demand, DR,Inter-site and
dev-ops
Quad – 4 vCPU> 40G and multi-tenant
Separate Mgmt., Edge and ComputeCluster
Scale & Availability NA
Multi-tier for services
1 NSX Refresher and Use Cases
2 NSX Components
3 NSX Design
4 Summary
Agenda
45
SecurityInherently secure infrastructure
Automation IT at the speed of business
Application continuityData center anywhere
NSX customer use cases
Micro-segmentation
DMZ anywhere
Secure end user
IT automating IT
Multi-tenant infrastructure
Developer cloud
Disaster recovery
Cross cloud
Multi data center pooling
47
NSX Reference Designs
NSX Platform
Hardening
NSX Getting StartedGuides
SDDCValidatedSolutions
NSX PartnerWhite
papers
Reference Designs & Technical Papers on VMware Communities: https://communities.vmware.com/docs
Reference Designs and Technical Papers on the NSX Portal: http://www.vmware.com/products/nsx/resources.html
NSX andFabric
Vendors
VMware NSX Collateral Landscape
Resources
Read vmware.com
Design, Admin, and User Guides
Ops Guides (Monitoring,
Troubleshooting)
White PapersTrylabs.hol.vmware.com
Hands-on Labs
Takevmware.com/education
Courses – ICM, Design and Deploy, Troubleshooting and OperationsCertifications – VCA, VCP, VCIX, VCDX
Operations Transformation Services (OTS)
Usevmware.com/consulting
Thank you.