the limits of switch bandwidth scott kipp february 28, 2011
TRANSCRIPT
THE LIMITS OF SWITCH BANDWIDTHScott Kipp
February 28, 2011
2
Increasing Bandwidth
• Switch designs are limited by:• ASICS that are getting faster with process improvements• 45 nm processes can support 25Gbps lanes
• Backplanes and connectors are getting faster and support 25 Gbps• FR4 will probably need to be replaced with Nelco13
• Front panel is set by 19” rack width and limits the port count• The height is optimized on 1.75” (1RU) increments and is a
differentiator
• Power of blade – optics usually only dissipate 50W of power max
• Higher speed designs will need new optical module and electrical interface standards
• 25-28 Gbps electrical signals is needed to increase the bandwidth of the switch
3
10GbE ASIC and Module Design
QSFP – 4 lanes of 10GbE
1st Generation
2nd Generation 3rd Generation
Electrical / Optical Interface
XSBI / 300 pin MSA
XAUI / X2, XENPAK, XPAK
XFI, SFI /XFP, SFP+, QSFP+
Electrical Lanes and Speeds
16 lanes at 644 MHz
4 lanes at 3.125Gbps
1 lane at 10.3125 Gbps
10GE Ports/ASIC 3 12 48 - 128
Throughput/ASIC 30 Gbps 120 Gbps 480 to 1280 Gbps
300 Pin MSA
X2
SFP+
XFPXENPAK XPAK
4
100GbE ASIC and Module Design
1st Generation 2nd Generation 3rd Generation
Electrical / Optical Interface
CAUI - CFP, CPPI – CXP, CXP2?
CAUI-4 - CFP2, CPPI-4 – QSFP, CFP4
?
Electrical Interface
10 lanes at 10 Gbps
4 lanes at 25Gbps
?
100GE Ports/ASIC 4 with CFP32 with CXP
12 ?
Throughput/ASIC 480 to 1280 Gbps
1200 to 2560 Gbps
?
CFP QSFP+CFP2
CXP
CFP4
Under DevelopmentBy CFP MSA
CXP2?
5
Switch Design• Switches are typically designed for a 19” rack
and this allows about 16-17” of port space for ports• Highest densities so far are 64 mini SFP+ (mSFP)
ports at 10GE or 48 SFP+ and 4 QSFP ports• Blades inserted horizontally or vertically
1U Switch
Modular Switch with 64 mSFP+ ports
Modular Switch with 48 SFP+ ports
6
Switch Designs with 10Gbps interfaces
48 SFP+ Design - 480 Gbps of throughput, 48 Watts* of powerLimited by size of SFP+ with 15mm pitch
4 CFP Design - 400 Gbps of throughput, 60 Watts** of powerLimited by ASICs now, but soon limited by size of CFP Port with 84mm pitch
*Based on 1 Watt of power/SFP per SFF-8431 *** Based on 2 Watts of power/QSFP+**Based on 15 Watts/CFP for 10km links ****Based on 2.5 Watts of power / CXP
CFP CFP CFP CFP
40 QSFP+ Design - 1600 Gbps of throughput, 80 Watts*** of powerLimited by ASICs now, size of QSFP+ with 21mm pitch
32 CXP Design - 3200 Gbps of throughput, 80 Watts**** of powerLimited by ASICs, CXP has 27mm pitch
7
Summary of 10 Gbps Switch Designs
• Support of single-mode applications requires larger modules because of higher power dissipation
• Multimode modules support less power and are denser Throughput Max Power
/ Module10km Support
SFP+ 48*10Gbps = 480 Gbps
1.5 W Yes
CFP for 100GbE 4*100Gbps = 400 Gbps
20 W for 10km 32W for 40km
Yes
QSFP+ for 40GbE
40*40Gbps = 1.6Tbps
2 W Not without exotic cooling
CXP for 100GbE 32*100Gbps = 3.2 Tbps
2 W Not without exotic cooling
8
Switch Designs with 25-28 Gbps interfaces48 SFP+ Design - 1.3 Tbps of throughput at 32GFC, 48 Watts* of powerLimited by size of SFP+ with 15mm pitch
8 CFP2 Design - 800 Gbps of throughput, 96 Watts** of powerLimited by ASICs now, but soon limited by size of CFP2 Port with 42mm pitch
*Based on 1 Watt of power/SFP per SFF-8431 **Based on 12 Watts/CFP2 for 10km links
CFP2 CFP2 CFP2 CFP2
40 QSFP+ Design - 4 Tbps of throughput, 80 Watts*** of powerLimited by ASICs and power, QSFP+ pitch is 21mm
16 CFP4 Design - 1.6 Tbps of throughput, 96 Watts**** of powerLimited by ASICs, CXP pitch is 27mm
CFP2 CFP2 CFP2 CFP2
***Based on 2 Watt of power/QSFP****Based on 6 Watts/CFP2 for 10km links
9
Summary of 25Gbps Switch Designs
• Support of single-mode applications requires larger modules because of higher power dissipation
• Multimode modules support less power and are denser Throughput Max Power
/ Module10km Support
SFP+ for 32GFC 48*28Gbps = 1.3Tbps
1.5 W Yes, but challenging at 1.5W
CFP2 for 100GbE
8*100Gbps = 800 Gbps
6 W Yes
QSFP+ for 100GbE
40*100Gbps = 4Tbps
2 W Not without exotic cooling
CFP4 for 100GbE
16*100Gbps = 1.6 Tbps
3 W Probably 2014
Tbps Bandwidths require faster speeds
• 10Gbps lanes will limit the throughput of the switch/blade in the next few years• Difficult to expand beyond 128 ports / ASIC
• Module size limits bandwidth and causes long trace lengths
• In the next few years, 25-28Gbps signaling needed to support Tbps switch/blade bandwidths• 4X25Gbps easier to support than 10X10Gbps in near
future
• Cost of 25-28G signaling compared to 10Gbps will be an interesting battle in the next few years
• Terabit Ethernet will require major innovation• 100x10G or 40x25G not practical for pluggable modules
• 10X100G not practical from signaling perspective
11
Thank You