green data centers - jlworkshop.com center-ken.pdfgreen data centers ken hall, rcdd nts september...

Green Data CentersKen Hall, RCDD NTSSeptember 2008

August, 2008page 2 /

Green Initiatives: Materials

� Lead-Free, RoHS product set� Reusable products

� Modular Copper and Optical Fiber Preterminated Communications cabling


What is a �Green� Data Center?


Next Generation Ethernet Networks

Power Consumption

Cooling Requirements

Transceiver Size

~2W ~8-15W

Fiber Copper

Data Center Area $ $$$$


Green Initiatives: Power

� High-density Cabling � Enables 7-27% power reduction on per port basis� Smaller cable than six 4-pair cables

� Better airflow = better cooling = less power

� Shielded Cat6A Cabling� Up to 40% smaller cable than Cat 6a U/UTP� Potential power savings on network electronics

� XG Optical Fiber Cabling� Optical ports draw less power than copper ports


Data Center Standards

�TIA 942 Telecommunications Infrastructure Standard, April 2005

� Substantial, available, North America

�EN 50173-5 Information technology � Data Centers

� Smaller scope, Europe

�BICSI 942, �complementary� to TIA 942

� ISO/IEC Draft 24764 Information technology

� Global Generic Cabling for Data Centers


Hierarchical Structure of Data Centers


Different Tiers of Data Centers (TIA942)


Annual Downtime

24 min

1 hour

36 min

22 hours

1 day

4 hours

48 min

Downtime/

Month

Downtime/

Year

2 min8 min

1 hour

50 min

2 hours

24 min

0,4 hours1,6 hours22,0 hours28,8 hours

Tier 4Tier 3Tier 2Tier 1

Not much time for Re-cabling!


Tiering For Cabling Systems (TIA942)

Security/Availability

Structured Cabling Trends for SANs and Data Centers


Data Center Dynamics Survey


Power�.

� Demand for electricity is projected to grow 19% in next ten years� Generation capacity is projected to grow only 6% (Source: NA Electric Reliability Council)

� Virtualization & multi-core processors are driving dramatic increases in cabinet density� 2006: US servers consumed $14 B in power & cooling� 2009 Projection: $50 B (Source: IDC)

� 2011 Projection: $100 B � In 2008, half of the world�s data centers will not be able to utilize the

latest high-density equipment (Source: Gartner)

� Their power infrastructure simply won�t support it� 1W at the server takes ~ 2 watts at the meter

� Network power - Cooling of network equipment - utilization


Availability, Space and Density

45%

52%

83%

0% 20% 40% 60% 80% 100%

Space

Availability

Power consumption

A survey of Data Centers owners and the biggest

concerns

Solution: flexible cabling solution, easy to migrate, just Place and Plug, high density


Passive Cabling Solutions for Data Centers

� Data Center cabling requirements are different from the ones in Office Networks

� Dynamic vs. Static

� Easy deployment (weekend work/testing)

� High performance

� Structured scalability

� Fast & easy MAC�s

� Functional Security (less human errors)

� High density

� Effective cable management

� Availability (less planning, less downtime)

� Cooling support


Passive Solutions for Data Centers

� Market situation� The market changed in 3 years to pre-terminated solutions� Copper

� High density copper 1 Gb/s systems� Integrated cable management

� Fiber� Multimode OM3 systems� MPO systems

Data Center Applications & TechnologiesToday & Tomorrow


What applications are running in Data Centers ?� 10/100Base-T

� GigaBit Ethernet (1000Base-T, 1000Base-SX/LX)

� 10GigaBit Ethernet (10GBase-T, 10GBase-SR)

� Fibre Channel

� InfiniBand

� Coming: 40 & 100 GigaBit Ethernet


0

5

10

15

20

25

3020

00

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

100M 1G 10G 40G 100G

mill

ions

ser

ver u

nits

x86 Servers by Ethernet Connection Speed (100 Mbps through 100 Gbps)

Source: Intel & Broadcom (April 2007)

10 year transition for 1G Ethernet 6 years for 10GBE 5 years for 40GBE


GBE/10GBE

� GigaBit Ethernet

� OM1/OM2/OM3 for fiber (1000Base-SX/LX)� Cat.5e for copper (1000Base-T)

� 10GigaBit Ethernet

� OM3 recommended for fiber (10GBase-SR)� Cat.6A/Cat.7/Cat.7A for copper (10GBase-T)


� Summary of power measurements

0

5

10

15

Link speed (Mb/sec)

Pow

er u

se (W

)

10 100 1000 10000

10G Base-T is a power concern

g00.xls

10G Base-T Application Cabling Power

20082008


10G Copper Cabling

� 10G Copper Expectations� 36% expect to install 10G copper in the next 1-2 years� 61% expect to install 10G copper in the next 3 years

� Out of those expectations, 75% of those plan to choose shielded copper over unshielded

0%

5%

10%

15%

20%

25%

Cat 6 F/UTP Cat 6A U/UTP Cat 6A F/UTP Cat 7 S/FTP Cat 7A S/FTP UndecidedSource: BSRIA: U.S. Data Center Structured Cabling Study, January 2008



� Reason� Signal to noise (S/N) ratio of UTP cabling systems� DSP�s are required to improve the signal quality

� Shielded systems have a much better EMC performance� Better S/N ratio� Less calculation for DSP�s� Less power consumption

� Conclusion� Shielded cabling enables reduction of network power consumption


� 10GB-T PHY optimized for 100m UTP Cabling� 15-20W (in 90nm) (1000B-T: 500mW)

� Problem for High Density Line Cards, PCI Cards, MBs, Modules

� 2.5usec latency (1000B-T: 250ns)

� Too high for some High Performance Applications, FC, IB

� Power Hungry AFE

� 50% of Power is in AFE

� Not likely to improve much over time� Large percentage of the power is in the AFE

� Power reductions in smaller process geometries are questionable



� Power Dissipation� 3W Typical, 4W Worst Case

� Latency� Less than 500ns� Goal of 250ns

� Low Cost� Significant SNR Margin

(1000B-T: 10dB)� Demonstrable Design of 1M

Gates (2x 1000B-T)

� Short Reach� 30m on STP/FTP Cable (field

configurable)� 2 Connector, 3 Links Segments� Goal of 45m

� Auto Negotiation Compatible� RJ-45 Connector� Backwards Compatible with 100B-

T / 1000B-T over UTP

10G Base-T Application Shielded Cabling Power


Characteristics of Data Center Cabling

� 90% of Links are in Server Rooms Less Than 1,524 sq. meters(100% <= 30m)

� 5% of Links are in S. Rooms Between 1,524 � 6,096 sq. meters(80% <= 30m)

� 5% of Links are in Server Rooms Greater Than 6,096 sq. meters (55% <= 30m)

� => 97% Coverage with 30m

� => 99% Coverage with 45m


Fibre Channel


InfiniBand Bandwidth

2004 2005 2006 2007

Spee

d

40G

20G

10G8G4G2G

Infiniband10G EthernetFibre Channel1G Ethernet

120G QDR

DDR

SDR

Parallel lanes means parallel optics


InfiniBand: High Speed Cabling Types

QSFP


40/100 GBE Multimode Fiber (MMF) Solutions

!850nm 12x10G CWDM devices may support 100-300m

!Utilization of MPO connectors is an important factor for

economic feasibility of 100G MMF

!Pre-terminated OM3 ribbon fiber will be cost-effective

!90% data centre links are MMF today and within 150m!High Performance Computing requires links to 100m!Majority of corporate building backbones under 300m!IBM survey found 70 - 80% of above links under 100m!Corning survey of data centre links found 96% <200m


40/100 Gb/s

� Official IEEE project � IEEE802.3ba� Current situation indicates an MPO solution� Reason: Parallel transmission needed

12-fiber MPO 72-fiber MPO

Examples of Tyco Electronics Parallel Optic connectors


ISO/IEC 24764 data center standard

ZDMD LDP EQENI

ISO/IEC 24764 model for Data CentersENI

Network access cabling

subsystem

Main distributioncabling

subsystem

Zone distributioncabling subsystem

� Standard MPO systems allows mostly 2 cassettes for 10 Gb/s� Next Generation allows 6 cassettes

� Complete cabling from MD to EQ� Reduce number of actives� Allows more flexibility in DC design

QSFP-style Active Cable Assemblies for InfiniBand Applications


Infiniband

� The situation:� SAN�s a essential part of Data

centers� More and more data will be

stored� Infiniband and Fiber channel

most common applications� Mostly Copper for Infiniband and

fiber for Fiber channel


Why Active Optical Cable Assemblies?

!Up to 200mtr.!

!Bend radius 25mm!

!20 GB Bi- directional (Up to 40Gb)

!Multi Mode fiber optic cable

Copper compared to Fiber

!Limited lengths, max 20mtr.

!Large Bend radius

!28-26 AWG Copper cable

!Max. 10 GB Bi- directional

Longer lengthsare

not possible due to

Noise and Crosstalk

!28-26 AWG Copper cable


Product Overview

� Products� DDR4 - 4 Channels of 5 Gb/s

� 20 Gb/s Up to 200 Meter Reach

� QDR4 - 4 Channels of 10 Gb/s� 40 Gb/s up to 120 Meter Reach

� Next steps to develop� Product DDR12 -12 Ch 5 Gb/s (60 Gb/s) Up to 120 Meter Reach� Product QDR12 � 12 Ch 10 Gb/s (120 Gb/s) Up to 120 Meter Reach

Data Centers Solutions

How to combine DC cabling infrastructure requirements with today�s and tomorrow�s

applications ?


Recall: Passive Cabling Requirements for Data Centers


� High performance



� Functional Security (less human errors)

� High density


� Availability (less planning, less downtime)

� Cooling support

MRJ21 MPO 1�


Why High Density Technology?

� Eco-Friendly� Assists in efficient power use at the switch� Assists in efficient power use in HVAC� Less material� Reusable

� Reduces need for expensive abandoned cable removal

� Modular� Speeds design & layout� Speeds installation / availability� Speeds training� Speeds MACs� Speeds recovery


High Density Pre-terminated Solutions

� Pre-terminated MRJ21 Copper System

� High Density with 6 GBE Channels per

cable

� ≤ 48 ports in 1 rack unit

� Pre-terminated MPO Fiber System

� High Density with 12 or 72 fibers per

cable

MRJ21 cable is 30% smaller, 20% lighter than 6 4-pair cables


Current GbE application benefits

� Density, modularity & performance on equipment and in infrastructure

� MRJ21� provides Hi-D Green GbE � Connector: 1/3 -1/4 RJ45 cassette

� Cable: 30% smaller, 20% lighter

� 1 cable vs. wrapped 6-cable bundle

� Eco-Friendly

� All components permanent link tested,

documented and serialized

� 7-27% less power than RJ45 @ 2x ports

� 10G Shielded Version in process

1.14�x.74�6 GbE ports

MRJ21�MPO

Euro


High-density Pre-terminated Copper

� The MRJ21� 24-pair Connector� 1�x 5/8� Footprint� Being adopted directly on electronics

12~399 in2RJ-45

8~266 in2RJ-45 (High Density)

6~200 in2MRJ21

Rack UnitsArea288 4-pair Ports

1 MRJ21 6 RJ45s


MRJ21 Energy Efficiency

4.88 Watts234 Watts481 GbE48 port MRJ21

5.25 Watts126 Watts241 GbE24 port RJ 45

3.56 Watts320 Watts901 GbE90 port MRJ21

4.75 Watts190 Watts401 GbE40 port RJ 45

Total power

per port

Max power

per card

# of

ports

Appl.Card

configuration

Combines high density and ease of maintenance with energy savings


Network Vendors with MRJ21 Equipment

� Alcatel:� OmniSwitch� Alcatel TiMetra� Lucent / Riverstone

� Barco Xenia� EIT� EntriSphere � Foundry Networks� Force10 Networks � Fujitsu � IBM � Motorola � NMS Communications � Proworks

� Shanghai Electronics� Stratex Networks� Tellabs / Vivace� UT Starcom

Force10 E120090 GbE ports/card

Foundry BigIron RX-4,8, 16, & 3248 GbE ports/card

Note: The vendors listed and shown have publicly launched MRJ21 equipment into the marketplace. Those not listed should be contacted directly for program status updates.

IBM eServerBladeCenter Copper Pass-

thru module 15 GbE ports

Alcatel OmniSwitchNI48 card


Content of One 7-foot Rack (40 RUs)

~$0.70

~$1.56

~$1.40

~$3.12

~$2.80

~$6.25

Op Cost per

Port (year)

1920960480Cu Ports per

Rack

0 RUs20 RUs20 RUsCable Mgt.

per Rack

484824# Copper

Ports per RU

AngledHigh-densityTraditional

low HIGH

9 sq. ft. Traditional Room � Avg $1,350/yr ops. cost20 sq. ft. Hot/Cold Aisle � Avg $3,000/yr ops. cost



~$0.70

~$1.56

~$1.40

~$3.12

~$2.80

~$6.25

Op Cost per

Port (year)


Rack


per Rack

484824# Copper

Ports per RU


low HIGH



Installation Time of One 7-foot Rack (40 RUs)

256 Hours128 Hours64 hoursInstall Time per

Rack � RJ45

2 minsAvg. Install

Time � MRJ21

10 hours,

40 mins

5 hours,

20 mins

2 hours,

40 mins

Install Time per

Rack � MRJ21

8 minsAvg. Install

Time - RJ45


Cable prep, termination, dress, insert & label


Installation Time of One 7-foot Rack (40 RUs)

480 ports 960 ports 1,920 ports

256 Hours128 Hours64 hoursInstall Time per

Rack � RJ45

2 minsAvg. Install

Time � MRJ21

10 hours,

40 mins

5 hours,

20 mins

2 hours,

40 mins

Install Time per

Rack � MRJ21

8 minsAvg. Install Time

- RJ45


Cable prep, termination, dress, insert & label


Large Job (960 Ports) Cost Model Summary

~10 days less (E/P)~2 days less (NE/NP)

88% lessTotal

Time

~10% less (E/P)+15% more (NE/NP)

27% lessTotal

Cost

Fiber

12F MPO to Small Form

Factor

Copper

MRJ21 vs. RJ45

Product, placing cables, cable preparation, termination

The material referenced on this slide is for comparison purposes only and not for budgetary purposes. Actual costs may differ and may depend on project size, location or purchasing practice as well as other variables.


High-density Pre-terminated Fiber

� The MPO� FOCIS 5, published in 1997

� 12 fibers in one connector� Being adopted directly on electronics

2~66 in212-fiber MPO Fiber

24~798 in2Duplex Fiber

12~399 in2Duplex Fiber (SFF)

<0.25~2 in272-fiber Para-Optix� MPO

Rack UnitsArea288 Duplex Ports

1 MPO

6 SC Duplex

1 Para-Optix

36 LC Duplex


What is the value of time?

0 2 4 6 8 10 12 14 16

On-Site Person Hours

3) Para-Optix 36 DuplexPorts 1U

2) MPO Cassettes 1u (6x12Cable)

1) Field-terminated NE/NPDuplex LC

Installation Time 144 Fibers (72 OM3 fibers / end)

Time (Hrs)

� Installation Time� Uptime� Recovery Time

The Combination of a High Density/High Performance Cabling System Platform with Cable Management


Size Matters�

� How does cable size really affect your pathway costs?� Conduit fill � based on 40% fill ratio� 40% more cables in a conduit

� Patch cord comparison � F/UTP compared to U/UTP

Conduit F/UTP 6A UTPSize 7.6mm (.300") 9.0 mm (0.354")

2 18 133 40 294 71 51


10 Gigabit per Second � Space (Air Handling � Cooling � Power)

ShieldSpaceMitigation

TechniquesANEXT

Reduction

57

.25 inch

48

Cat 6 U/UTP

42

(15 < Cat 6)

28

(29 < Cat 6)3� Conduit Cable Fill

0.290 inch

(Up to 16% > Cat 6)

≤ 0.354 inch

(Up to 41% > Cat 6)Cable OD

4816 � 24Port Density

(1 RU)

Cat 6aF/UTP

Cat 6aU/UTP


Cable Management

� Does this look familiar?

� Unmanaged and unstructured connections

� Poor documentation

� 30-40% abandoned patch cords after 1 year of operation

� Dead ports in switches and cable runs 30-40% unused capacity

� MAC time > 1h per port

� Adds up to 150% of operational cost

� Adds up to 200% of floorspace needed

� Residual cable mass blocks airflow in rack and under floor


Recall: Passive Cabling Requirements




� Functional Security

(less human errors)

� High density


� Availability

(less planning, less downtime)

� Cooling support


AMP Hi-D technology

� Patented Corner Brackets combined with Angled Patch Panels� Eliminates the need for horizontal cable managers� Uses unused valuable cabinet space - increase of cabinet density� Toolless cable management maintenance � increase of operational

speed


End-to-End Data Center Solutions

green data centers - jlworkshop.com center-ken.pdfgreen data centers ken hall, rcdd nts september...

Documents