IP-Based Storage IP-Based Storage NetworkingNetworking

陳文賢陳文賢中興大學資訊科學研究所中興大學資訊科學研究所

OutlineOutline IntroductionIntroduction iSCSI: Architecture and Standardization iSCSI: Architecture and Standardization Issues and Solutions of  IP StorageIssues and Solutions of  IP Storage

PerformancePerformance SecuritySecurity CostCost InteroperabilityInteroperability

Storage over WANStorage over WAN Conclusion Conclusion

IntroductionIntroduction

SAN: Storage Area NetworkSAN: Storage Area Network NAS: Network Attached StorageNAS: Network Attached Storage

PCI Bus speedPCI Bus speed Gigabit and 10Gigabit EthernetGigabit and 10Gigabit Ethernet

Storage is no fun until networking Storage is no fun until networking comes in.comes in.

IntroductionIntroduction

Standardization: The Internet Engineering Task Force (IETF) has approved the iSCSI standard since Feb. 2003.

A mapping of the SCSI remote procedure invocation model on top of the TCP protocol.

A new “SCSI transport” as defined by the SCSI SAM-2 document. Equivalent protocols include “SPI-2”,

“FCP-2”, … To take compelling advantages from the

IP/Ethernet infrastructure.

SCSI-3 Architecture RoadmapSCSI-3 Architecture Roadmap

Common Access Method (CAM)

BlockCommands

(SBC, SBC-2)

ReducedBlock

Commands(RBC)

StreamCommands

(SSC, SSC-2)

MediumChanger

Commands(SMC, SMC-2)

Multi-MediaCommands

(MMC, MMC-2,MMC-3, RMC)

ControllerCommands

(SCC, SCC-2)

EnclosureServices

(SES)

Object-BasedStorageDevice(OSD)

Primary Commands (SPC, SPC-2, SPC-3)

Arc

hite

ctu

re M

od

el (S

AM

, SA

M-2

)

SCSIParallel

Interface(SPI-2, SPI-3,

SPI-4)

(aka, Ultra2,Ultra160,Ultra320)

Serial BusProtocol – 2

(SBP-2)

FibreChannelProtocol

(FCP, FCP-2)

SSASCSI-3

Protocol(SSA-S3P)

SCSIover ST(SST)

SCSIVI Protocol

(SVP)

IEEE 1394Fibre

Channel(FC)

SSA-TL2

SSA-PH1 orSSA-PH2

iSCSI/TCP/IP

ScheduledTransfer

(ST)

VirtualInterface

(VI)

802.2/802.3……

Layers and SessionsLayers and Sessions

SCSI layer (SAM-2)

TCP/IP TCP/IP TCP/IP

SCSI CDB

iSCSI PDUs

TCP packets

iSCSI layer

iSCSI layer

iSCSI layer

iSCSI layer

SCSI layer (SAM-2)

TCP/IP TCP/IP TCP/IP

iSCSI layer

iSCSI layer

iSCSI layer

iSCSI layer

Initiator Target

Session Session

TCP Connection

TCP Connection

TCP Connection

Layer and Session (Cont.)Layer and Session (Cont.)

Conceptual Layering ModelConceptual Layering Model SCSI layer – builds/receives SCSI SCSI layer – builds/receives SCSI

Command Data Blocks (cf. SCSI Command Data Blocks (cf. SCSI Architecture Model - 2)Architecture Model - 2)

iSCSI layer – builds/receives iSCSI PUDsiSCSI layer – builds/receives iSCSI PUDs TCP Connections – form an initiator-target TCP Connections – form an initiator-target

“session”“session”

SessionSession A group of TCP connections linking an A group of TCP connections linking an

initiator with a target.initiator with a target. Defined by a session IDDefined by a session ID

What customer problems does iSCSI What customer problems does iSCSI solve?solve?

iSCSI provides a cost-effective transport for Storage iSCSI provides a cost-effective transport for Storage Area Network (SAN) when compared with Fibre Area Network (SAN) when compared with Fibre Channel.Channel.

iSCSI enables affordable storage consolidation iSCSI enables affordable storage consolidation solutions—particularly in environments populated solutions—particularly in environments populated with mid range servers.with mid range servers.

Together with Storage Management Solution, iSCSI Together with Storage Management Solution, iSCSI also provides affordable disaster recovery, backup, also provides affordable disaster recovery, backup, and secondary storage solutions. and secondary storage solutions.

Performance and Cost: Hardware versus Performance and Cost: Hardware versus software-based solutionssoftware-based solutions

SoftwareSoftware iSCSI initiators provide the lowest cost iSCSI initiators provide the lowest cost iSCSI solution. A software-only iSCSI initiator uses iSCSI solution. A software-only iSCSI initiator uses a standard Ethernet NIC or a NIC with TCP offload a standard Ethernet NIC or a NIC with TCP offload Engine (TOE) to process the iSCSI commands and Engine (TOE) to process the iSCSI commands and the TCP/IP protocol. For workstations/servers with 2 the TCP/IP protocol. For workstations/servers with 2 GHz CPUs, iSCSI protocol processing does not GHz CPUs, iSCSI protocol processing does not impose a significant overhead for most customer impose a significant overhead for most customer workloads. Example: Microsoft iSCSI initiator driver.workloads. Example: Microsoft iSCSI initiator driver.

Hardware:Hardware: With older CPUs and heavily loaded With older CPUs and heavily loaded servers, a hardware-assisted iSCSI initiator is servers, a hardware-assisted iSCSI initiator is appropriate as the iSCSI initiator HBA can offload appropriate as the iSCSI initiator HBA can offload the CPU. Example: Intel iSCSI HBA (Intel Pro 1000T)the CPU. Example: Intel iSCSI HBA (Intel Pro 1000T)

Performance and CPU overhead (For Performance and CPU overhead (For single Gbps connection)single Gbps connection)

iSCSI iSCSI HBAHBA

iSCSI iSCSI driverdriver

SAN/IPSAN/IP

Sequential ReadSequential Read 9494MB/sMB/s 5959MB/sMB/s 105105MB/sMB/s

CPU CPU Utilization(ClientUtilization(Client

))

17%17% 23%23% 35%35%

Note: With multiple connections, the performance can Note: With multiple connections, the performance can be enhanced even more.be enhanced even more.

IPStor ServerWin 2000 Server

Intel IOMeter running on Win2k Server

Testing ConfigurationTesting Configuration

FalconStor’s IPStor Server (iSCSI target)

GBE Switch

Scenarios

1. Intel iSCSI HBA (Pro 1000 T)

2. Microsoft iSCSI initiator (software)

3. SAN/IP Client

iSCSI SecurityiSCSI Security

Fibre Channel is perceived to be more Fibre Channel is perceived to be more secure as it is a private network. However, it secure as it is a private network. However, it is a Layer 2 protocol with no security is a Layer 2 protocol with no security mechanism built in essentially.mechanism built in essentially.

The iSCSI spec, on the other hand, coversThe iSCSI spec, on the other hand, covers initiator and target authentication (using initiator and target authentication (using CHAP, SRP, Kerberos, and SPKM) to prevent CHAP, SRP, Kerberos, and SPKM) to prevent unauthorized access and permit only unauthorized access and permit only trustworthy nodes. In addition, IPsec can trustworthy nodes. In addition, IPsec can be used to provide privacy and prevents be used to provide privacy and prevents eavesdropping.eavesdropping.

The solutions are readily available today.The solutions are readily available today.

Security ConfigurationSecurity Configuration

IPsecIPsec Peers must authenticate each other before data Peers must authenticate each other before data

transfertransfer Data is encrypted on the wireData is encrypted on the wire Operates at IP layerOperates at IP layer

CHAPCHAP One way authentication mechanism, but may be One way authentication mechanism, but may be

done by both Initiator and Targetdone by both Initiator and Target Operates at iSCSI protocol layerOperates at iSCSI protocol layer

iSCSI CHAP and IPsec rely upon the peer iSCSI CHAP and IPsec rely upon the peer knowing knowing a “secret” for authenticationa “secret” for authentication Pre-shared or private keyPre-shared or private key

iSCSI interoperabilityiSCSI interoperability Operating system and application vendors often have a Operating system and application vendors often have a

catalog of qualified hardware solutions. The Microsoft catalog of qualified hardware solutions. The Microsoft Windows Catalog lists iSCSI hardware devices that have been Windows Catalog lists iSCSI hardware devices that have been qualified. In late 2003, more than 14 leading storage vendors qualified. In late 2003, more than 14 leading storage vendors had qualified their iSCSI hardware products under Microsoft had qualified their iSCSI hardware products under Microsoft iSCSI Designed for Windows Logo Program. iSCSI Designed for Windows Logo Program.

Fibre Channel interoperability problems were primarily due to Fibre Channel interoperability problems were primarily due to two issues. First, the vendors implemented the SCSI3 two issues. First, the vendors implemented the SCSI3 command set differently. Secondly, Fibre Channel lacks built-command set differently. Secondly, Fibre Channel lacks built-in networking capabilities.in networking capabilities.

In iSCSI’s case, the interoperability issues are greatly reduced. In iSCSI’s case, the interoperability issues are greatly reduced. In addition, SNIA, SNW, and other labs are continuously In addition, SNIA, SNW, and other labs are continuously working on the interoperability issues.working on the interoperability issues.

IDC’s predictionIDC’s prediction

IDC expects that iSCSI adoption will IDC expects that iSCSI adoption will commence in most countries in the Asia commence in most countries in the Asia Pacific region during 2003 with progressive Pacific region during 2003 with progressive deployment expected in 2004. In many cases, deployment expected in 2004. In many cases, an iSCSI implementation will be an iSCSI implementation will be complementary to existing fibre channel SANs.complementary to existing fibre channel SANs.

Overall, IDC believes the two most likely places Overall, IDC believes the two most likely places where iSCSI will be adopted are:where iSCSI will be adopted are: In smaller organizations that haven't networked In smaller organizations that haven't networked

their storage, yet are familiar with TCP/IP. their storage, yet are familiar with TCP/IP. Large organizations that will use iSCSI to link FC Large organizations that will use iSCSI to link FC

SANs. SANs. Graham Penn, Director, Asia Pacific Storage, IDCGraham Penn, Director, Asia Pacific Storage, IDC

Microsoft’s iSCSI initiatorMicrosoft’s iSCSI initiator

The Microsoft iSCSI Software Initiator The Microsoft iSCSI Software Initiator version 1.0 package was released to the Web version 1.0 package was released to the Web June 25, 2003. The Microsoft iSCSI software June 25, 2003. The Microsoft iSCSI software initiator allows a Windows-based computer initiator allows a Windows-based computer to serve as an iSCSI initiator to connect to to serve as an iSCSI initiator to connect to iSCSI targets on an Internet Protocol Storage iSCSI targets on an Internet Protocol Storage Area Network (IP SAN). Area Network (IP SAN).

All iSCSI devices appear in Windows as a All iSCSI devices appear in Windows as a local disk and can be managed in Disk local disk and can be managed in Disk Administrator as any other local disk. Administrator as any other local disk.

Download:Download:http://www.microsoftcom/downloads/details.asphttp://www.microsoftcom/downloads/details.aspx?FamilyID=12cb3c1a-15d6-4585-b385-befd1319x?FamilyID=12cb3c1a-15d6-4585-b385-befd1319f825&DisplayLang=enf825&DisplayLang=en

Benefits of using SANsBenefits of using SANs Enhance applications performance by Enhance applications performance by

freeing up enterprise networkfreeing up enterprise network Permits more desktop use of RAID Permits more desktop use of RAID

technology technology Consolidated backups and archivesConsolidated backups and archives Disk mirroring, backups to disaster Disk mirroring, backups to disaster

recovery sitesrecovery sites High availability mission critical High availability mission critical

databasesdatabases Distributed (logical) server clusteringDistributed (logical) server clustering Disk virtualisationDisk virtualisation

Consolidated backups and archives Disk mirroring, backups to disaster

recovery sites High availability mission critical databases Distributed (logical) server clustering Disk virtualisation

SAN in the WANSAN in the WAN

Enhance applications performance by Enhance applications performance by freeing up enterprise networkfreeing up enterprise network

Permits more desktop use of RAID Permits more desktop use of RAID technology technology

WAN

Is SAN in the WAN possible ? Is SAN in the WAN possible ?

Yes, and SAN traffic loads are Yes, and SAN traffic loads are typically less than many people typically less than many people think : think : Very few disks or RAID systems can Very few disks or RAID systems can

stream at > 10 Mbytes/sec, although stream at > 10 Mbytes/sec, although peaks of 30 Mbyte/sec are commonpeaks of 30 Mbyte/sec are common

Even high performance UNIX servers Even high performance UNIX servers can rarely exceed 20 Mbyte/sec can rarely exceed 20 Mbyte/sec

NT servers are much worse, typically NT servers are much worse, typically < 10 Mbytes /sec< 10 Mbytes /sec

Is SAN in the WAN possible ?Is SAN in the WAN possible ?

Tape Subsystems are quite slow:Tape Subsystems are quite slow: 4 to 10 Mbyte/sec streaming is normal. 4 to 10 Mbyte/sec streaming is normal.

Peak data of 20 Mbyte/sec maximum per Peak data of 20 Mbyte/sec maximum per interface and driveinterface and drive

Disk mirroring depends upon Disk mirroring depends upon applicationapplication Transaction or database system often Transaction or database system often

below 1 Mbyte/secbelow 1 Mbyte/sec Backups may be faster, but are limited Backups may be faster, but are limited

by system (controller / drive) by system (controller / drive) performanceperformance

Is SAN in the WAN possible ?Is SAN in the WAN possible ?

Given that high bandwidth network Given that high bandwidth network links are increasingly affordable: links are increasingly affordable: T3 (45 Mbit/sec) is capable of around 5 T3 (45 Mbit/sec) is capable of around 5

MBytes /sec : easily enough to run a MBytes /sec : easily enough to run a remote DLT drive or to handle disk remote DLT drive or to handle disk mirroringmirroring

OC3 (155 Mbit/sec) is capable of about 17 OC3 (155 Mbit/sec) is capable of about 17 Mbytes/SecMbytes/Sec

OC-12 (622 Mbit/sec) is capable of about OC-12 (622 Mbit/sec) is capable of about 65 Mbytes/sec65 Mbytes/sec

Is SAN in the WAN possible?Is SAN in the WAN possible?

Yes, but only with pipelined data transfers WAN data takes about 5Secs to travel

1Km, or 5mSec for 1000 Km. For a single disk reading or writing 64KByte blocks at 10 Mbytes/sec, over 1000 Km distance, non pipelined operation will reduce the performance to about 40% of the transfer speed. If 8 such blocks are pipelined, performance will be 84% of transfer speed.

Is SAN in the WAN possible?Is SAN in the WAN possible?

6.4 mSec 5 mSec

5 mSec DiskAck

Total time taken to transmit data block andreturn ack is 6.4 + 5 + 5 mSec = 16.4 mSecLost transmission time due to ack = 10 mSecLost efficiency due to ack = 10/16.4 = 60%

64 Kbyte block

Is SAN in the WAN possible ?Is SAN in the WAN possible ?

Yes, but only with low latency WANsYes, but only with low latency WANs Data must not be held in queues within Data must not be held in queues within

the WANthe WAN

0.000

0.200

0.400

0.600

0.800

1.000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

mS

ec D

elay

OC-3 (Frame)T3 (Frame)

WAN Delays Vs. EfficiencyWAN Delays Vs. Efficiency

0

20

40

60

80

100

1 5 10 15 20

One way delay mSec

Eff

icie

ncy

64 Kbyte Block

128 Kbyte Block256 Kbyte Block

512 Kbyte Block

1MByte Block

Efficiency at 10 Mbytes/sec for different block sizes

Very high speed Restricted

transmission distance Unreliable protocols

not designed for communications use

Primitive windowing

Machine room technologyMachine room technology

Today's Storage Area Networks belong in the Machine Room

SAN

Storage network speedsStorage network speeds

SCSI Type Speed

SCSI-1 5 Mbytes/secSCSI-2 (Narrow) 10 Mbytes/sec

SCSI-2 (Wide) 20 Mbytes/secUltra SCSI (Narrow)Ultra SCSI (Wide)Ultra 2 SCSI 80 Mbytes/sec

40 Mbytes/sec20 Mbytes/sec

Clock Mbytes/sec

5 MHz10 MHz

10 MHz20 MHz20 MHz40 MHz

Ultra 3 SCSI 40 MHz* 160 Mbytes/sec

* Ultra320: 320Mbytes/sec

Storage network speedsStorage network speeds

Fibre ChannelFibre Channel Up to 100 Mbytes/secUp to 100 Mbytes/sec Runs at 1 Gbaud using 8B/10B Runs at 1 Gbaud using 8B/10B

encoding, taken directly from FDDI encoding, taken directly from FDDI standardstandard

Frame based technology based on Frame based technology based on FDDI. Uses FDDI checksumsFDDI. Uses FDDI checksums

FC-AL shared between < 126 devicesFC-AL shared between < 126 devices 2 Gbaud and 4 Gbaud Fibrechannel 2 Gbaud and 4 Gbaud Fibrechannel

comingcoming

Transmission distancesTransmission distances

SCSISCSI Low voltage differential = 25 metresLow voltage differential = 25 metres Single ended = 3 metresSingle ended = 3 metres

FibreChannel (100 Mbytes/sec, 1.06 FibreChannel (100 Mbytes/sec, 1.06 Gbaud)Gbaud) Singlemode, 1300 nM < 10 KmSinglemode, 1300 nM < 10 Km Multimode, 850 nM < 300 metresMultimode, 850 nM < 300 metres

FibreChannel (25 Mbytes/sec, 266 FibreChannel (25 Mbytes/sec, 266 Mbaud)Mbaud) Multimode 850 nM < 2 KmMultimode 850 nM < 2 Km

Storage protocolsStorage protocols

SCSI SCSI defines a simple bus based transmission defines a simple bus based transmission

scheme with limited reliability featuresscheme with limited reliability features

Fibre Channel Fibre Channel is conceived as a high speed carrier is conceived as a high speed carrier

mechanism capable of transporting any bit mechanism capable of transporting any bit stream reliably, but is really a local protocol:stream reliably, but is really a local protocol: Sequence retry is very inefficient (subsequent Sequence retry is very inefficient (subsequent

sequences are repeated)sequences are repeated) Networking layers are missing: FibreChannel is really Networking layers are missing: FibreChannel is really

a layer 2 technologya layer 2 technology

WindowingWindowing Performance at a distance requires Performance at a distance requires

efficient windowingefficient windowing SCSI (and SCSI over FibreChannel) does SCSI (and SCSI over FibreChannel) does

not allow thisnot allow this SCSI has no inherent windowing: commands SCSI has no inherent windowing: commands

are acknowledged individually by the target are acknowledged individually by the target Command tag queuing is a solution, but is Command tag queuing is a solution, but is

not supported by many devices, and is a not supported by many devices, and is a higher level solution to a lower level problemhigher level solution to a lower level problem

FibreChannel, as a transparent transport FibreChannel, as a transparent transport mechanism, does NOT address this problemmechanism, does NOT address this problem

SAN in the WANSAN in the WAN

To build Storage Networks that operate To build Storage Networks that operate over WANs we need:over WANs we need: Realistic data speedsRealistic data speeds Adaptation of SCSI or FCP (SCSI over Adaptation of SCSI or FCP (SCSI over

FibreChannel) to a networking protocolFibreChannel) to a networking protocol Reliable stream transport Reliable stream transport

Disk Profiles operate with FibreChannel Class Disk Profiles operate with FibreChannel Class 3 service, an unacknowledged datagram 3 service, an unacknowledged datagram serviceservice

The only form of ACK is a sequence abortThe only form of ACK is a sequence abort

ApplicationsApplications

Storage Consolidation through IPStorage Consolidation through IP SAN features such as storage virtualization, SAN features such as storage virtualization,

Capacity-on-Demand mirroring, Capacity-on-Demand mirroring, TimeMark/TimeView (Disk Journaling), TimeMark/TimeView (Disk Journaling), Replication, Backup and Recovery, Storage Replication, Backup and Recovery, Storage Vaulting, etc, can be carried out in a cost Vaulting, etc, can be carried out in a cost effective manner.effective manner.

Diskless BladesDiskless Blades Storage Infrastructure for On-Demand/Utility Storage Infrastructure for On-Demand/Utility

ComputingComputing

ConclusionConclusion Simply put, iSCSI provides network storage

connectivity at Ethernet prices: iSCSI brings along a iSCSI brings along a simple and cost-effective solution to storage simple and cost-effective solution to storage networkingnetworking

Easy implementation for diskless servers, Easy implementation for diskless servers, workstations, blades, and utility/on-demand workstations, blades, and utility/on-demand computing.computing.

Create opportunities to the traditional networking Create opportunities to the traditional networking and storage companies alike, as the networking and storage companies alike, as the networking infrastructure can be leveraged.infrastructure can be leveraged.

iSCSI and IP Storage have arrived and will change iSCSI and IP Storage have arrived and will change the perception of computing forever!the perception of computing forever!

Storage Management is the key to success!Storage Management is the key to success!