computer networks & digital lab project

Computer Networks & Digital Lab project.

In cooperation with Mellanox In cooperation with Mellanox Technologies Ltd.Technologies Ltd.

Guided by:Guided by: Crupnicoff Diego & Gurewitz Omer. Crupnicoff Diego & Gurewitz Omer.

Students:Students: Cohen Erez , Gindi Nimrod & Krig Amit. Cohen Erez , Gindi Nimrod & Krig Amit.

Thanks!!!!! תודות

ברצוננו להודות לכל האנשים שבלעדיהם לא היה פרויקט זה יוצא לפועל : ובעזרת הדרכתם וסיועם הגענו עד כאן

על התמיכה והלבביות – חן )מעבדה לרשתות מחשבים(-יורם אור. לאורך כל הדרך

על ההכוונה להתמקד בכיוונים – אלי שושן ) מעבדה ספרתית ( . ובנושאים המעניינים באמת

על העזרה הרבה בתיאום – יורם יחיה )מעבדה לרשתות מחשבים(. כל שלבי הפרוייקט

– על ההדרכה, ההנחיה והעזרה עומר גורביץ ודייגו קופרניקוף. הרבה בכל שלבי הפרוייקט

שי כהן ,סמנכ"ל תפעול בחברת ברצוננו להודות באופו מיוחד ל( על הזמן והמשאבים הרבים )והיקריםמלאנוקס טכנולוגיות,

שאפשר לנו על מנת שנוכל להשלים את הפרוייקט

• Projects Motivation .• Projects objective .• InfiniBand short preview (10 min).• Projects System overview :

• Hardware Project description (10 min).• Software Project description (10 min).

•Projects Demonstration Tools and process explanation (10 min)• Agilent IB tracer usage demonstration .•Specific InfiniBand Terms to be used .• Specific patterns to be sent explanation . • Mellanox InfiniBand Development tool to be used .

AGENDA

• Projects Demonstration (20 min)• Installation Process of the System in ‘Virgin’ system . • Sending Various InfiniBand Packets from the Ibgenerator to Agilent’s IB tracer. • Demonstration of full InfiniBand system flow ( Sending InfiniBand Packets from the Ibgenerator through Agilent’s IB tracer to Mellanox Infinibridge)

AGENDA – Cont’

The InfiniBand world is still in his early days – and as such it requires all sorts of supporting equipment .

As we will show you, analyzing equipment already existed when we started thinking on this project , what we could not locate is dedicated device which purpose is transmitting InfiniBand packets ( and we still can not find such device in the market !!!! – although several companies declared almost a year ago that they are working on this issue ).

That , and our desire to learn more about the InfiniBand standard , brought us to the idea of this project .

Projects Motivation

Learning and understanding deeply, the new InfiniBand™ Architecture.

Implementation of packet generator / transmitter focusing on InfiniBand™ protocol (release 1.0a).

Projects objective

InfiniBand short preview(Refreshing our memory)

In order to refresh our memory without saying what was already presented in former presentations of these projects we will review here the InfiniBand architecture by going over it in ‘title’ level.

The InfiniBand Standard

InfiniBand Overview

Switch fabricSwitch fabricConcurrent data transferConcurrent data transferNo mechanical constrainsNo mechanical constrains

PerformancePerformanceHigh bandwidth (2.5 to 30Gbit/sec)High bandwidth (2.5 to 30Gbit/sec)Fast I/O access by applicationFast I/O access by application

QOSQOSPacket-granularity b/w allocationPacket-granularity b/w allocationPacket-granularity latency decisionPacket-granularity latency decision

Switch

End Node

Switch Switch

Switch

End Node

End Node

End Node

End Node

End Node

End Node

End Node

End Node

End Node

The InfiniBand Architecture Model

RouterRouterNetwork or IB

IB L

ink

IB L

ink

SysSysMemMem

CPUCPU

CPUCPU

MemMemCntlrCntlr HCAHCA

IB LinkIB LinkSwitchSwitch

IB LinkIB Link

IB L

ink

IB L

ink

TCATCA Target

TCATCA

TargetTargetH

ost

In

terc

on

nec

tH

ost

In

terc

on

nec

t

RouterRouterIB Link

Host Channel Adaptersfor computing platforms

Target Channel Adaptersfor Specialized Subsystems

Subnets consist ofLinks & Switches

Routers enableinter-subnet communications

while providing subnet isolation

InfiniBand System Architecture

Decouples CPU and I/ODecouples CPU and I/O

OS-independent.OS-independent.

Scope – from “PCI” to WANScope – from “PCI” to WAN

Same ‘look and feel’ for localSame ‘look and feel’ for local or remote nodes. or remote nodes.

Wide cost/performance range Wide cost/performance range for implementations. for implementations.

CPU

TCA

TCA

IOCntrlr

IOCntrlr

TCAIO

Cntrlr

System Cntrlr

IB Switch

SystemMemory

Host ChannelAdapter

CPU

TCA

TCA

IOCntrlr

IOCntrlr

TCAIO

Cntrlr

System Cntrlr

IB Switch

SystemMemory

Host ChannelAdapter

InfiniBand Overview cont’

ReliabilityReliability

Reliable transport service in HWReliable transport service in HW

Automatic path migration in HW (fault tolerance)Automatic path migration in HW (fault tolerance)

Scalability/flexibilityScalability/flexibility

Up to 64K nodes in subnet, up to 2Up to 64K nodes in subnet, up to 2128128 nodes in network nodes in network

Multiple link width/trace (Cu, Fiber)Multiple link width/trace (Cu, Fiber)

Auto-negotiation of link width and transfer rateAuto-negotiation of link width and transfer rate

InfiniBand architecture.

InfiniBand is a Layered architecture.InfiniBand is a Layered architecture.

Similar to an IP network, each layer supply services to the Similar to an IP network, each layer supply services to the higher layer.higher layer.

An IB packet is build from headers added by each layer.An IB packet is build from headers added by each layer.

The layers responsibilities are:The layers responsibilities are:

Insure correct routing of a packet.Insure correct routing of a packet.

Insure correct data. Insure correct data.

Insure QOS.Insure QOS.

And more ….And more ….

InfiniBand architecture cont.’

IB End node

Application

Upper Layerprotocols

TransportLayer

NetworkLayer

LinkLayer

PhysicalLayer

Application


TransportLayer

NetworkLayer

LinkLayer

PhysicalLayer

IB Switch

Packet relay

PHY

PHY

IB Router

Packet relayPH

YL

ink

PHY

Link

Packet relay

PHY

Lin

k

PHY

Link

Legacy Router

InfiniBand architecture features.

Multiple transport servicesMultiple transport services Reliable and unreliableReliable and unreliable Connected and datagramConnected and datagram

Enables memory exposure to remote nodeEnables memory exposure to remote node RDMA-read and RDMA-writeRDMA-read and RDMA-write

Enables network partitioningEnables network partitioning Partition key and routing programmingPartition key and routing programming

Enables user-level access to I/OEnables user-level access to I/O Adapter validates access rightsAdapter validates access rights Adapter translates memory addressAdapter translates memory address

InfiniBand architecture features cont.’

Enables dynamic load balancingEnables dynamic load balancing

Within end-node or in the fabricWithin end-node or in the fabricMultiple levels of QOS decisionsMultiple levels of QOS decisions

Hardware description

Hardware system description

I2C

connector

Xilinx xcv400e FPGA

(125 MHz DDR)

IB port 2.5 GB/sec

Agilent’s

SerDes

10 1

I2C interface

transmitter

JTAG

JTAG connector

Oscillator

Power unit

PCI connector for power and reset

Mictor

System interface

The board is a standard PCI form factorThe board is a standard PCI form factor.. The board contain the following interfaces:The board contain the following interfaces:

I2C – software interface: This interface is used to load I2C – software interface: This interface is used to load data (10 bit for each byte, by InfiniBand spec), data (10 bit for each byte, by InfiniBand spec), commands and to control the Ibgenerator. We have commands and to control the Ibgenerator. We have used this interface because it’s a very simple and cheap used this interface because it’s a very simple and cheap solution. solution.

InfiniBand connector – This is the interface to the InfiniBand connector – This is the interface to the InfiniBand fabric. We use 1x connector according to InfiniBand fabric. We use 1x connector according to the InfiniBand spec.the InfiniBand spec.

System interface

JTAG – This is a common interface to connect to JTAG – This is a common interface to connect to FPGA’s. In our board we used the JTAG interface to FPGA’s. In our board we used the JTAG interface to program the Xilinx FPGA and to debug the Verilog program the Xilinx FPGA and to debug the Verilog code.code.

PCI interface – We didn’t use a “real” PCI interface. PCI interface – We didn’t use a “real” PCI interface. We have used this interface only to get power from a We have used this interface only to get power from a PC and to get reset signal.PC and to get reset signal.

System flow

Data is received from the I2C interface to the FPGA unit.Data is received from the I2C interface to the FPGA unit. Data can be written to 3 different location in the FPGA:Data can be written to 3 different location in the FPGA:

Data array – This is a 256X32 bit array, it’s holding the Data array – This is a 256X32 bit array, it’s holding the data to be transmitted to the IB fabric.data to be transmitted to the IB fabric.

Command array – This is a 32X32 bit array, it’s Command array – This is a 32X32 bit array, it’s holding in each row (each 32 bit) a command to holding in each row (each 32 bit) a command to execute. Each command hold the following execute. Each command hold the following information:information:

Pointer to the start address in the data array.Pointer to the start address in the data array. Pointer to the end address in the data array.Pointer to the end address in the data array.

System flow con’t

Times to transmit this data.Times to transmit this data. Pointer to the next command to execute.Pointer to the next command to execute.

Status register – This is the “go” command of the Status register – This is the “go” command of the system. To start a transmission we should write the system. To start a transmission we should write the address of the command to be execute and the FPGA address of the command to be execute and the FPGA will start sending this data.will start sending this data.

System flow con’t – Send TS1

CommandArray

DataArray

Status

TS1 Data

To SerDes

System flow con’t

Once the FPGA starts to work on a command, it will send Once the FPGA starts to work on a command, it will send 10 bits of data in a rate of 125 MHz DDR to the SerDes. It 10 bits of data in a rate of 125 MHz DDR to the SerDes. It will also send TBC (Transmit Byte Clock) signal.will also send TBC (Transmit Byte Clock) signal.

The SerDes will send this data in a rate of 2.5 Gbit/sec in a The SerDes will send this data in a rate of 2.5 Gbit/sec in a differential lines.differential lines.

Signal integrity

Software description

SW description

The software project contains 3 major librariesThe software project contains 3 major libraries

MPGA libraryMPGA library

I2C libraryI2C library

IB generator libraryIB generator library

MPGA library

TheThe MPGA MPGA (Management Packet Generator Analyzer) (Management Packet Generator Analyzer) library provides software for generation and analysis of library provides software for generation and analysis of InfiniBand packets.InfiniBand packets.

This library is especially important when using the IB This library is especially important when using the IB generator for sending and receiving all kinds of InfiniBand generator for sending and receiving all kinds of InfiniBand packets. packets.

Refer to mpga.h and the related files packet_append.h, Refer to mpga.h and the related files packet_append.h, packet_utilities and ib_opcodes.h for further details.packet_utilities and ib_opcodes.h for further details.

MPGA structure

This library has 3 hierarchical levels:This library has 3 hierarchical levels:

The first level is the upper level containing the user interface The first level is the upper level containing the user interface functions.functions.

The second level of the library is in charge of the InfiniBand The second level of the library is in charge of the InfiniBand packet generation building blocks.packet generation building blocks.

The last level in MPGA is a hidden part of the library The last level in MPGA is a hidden part of the library containing all of the internal functions used only by this containing all of the internal functions used only by this library.library.

MPGA structure

Hק'Headers in Level 1

Headers in level 2

Headers in level 3

User level

Building blocks

Internal functions

MPGA cont’ - Generating packet flow

Raw data to send Raw data to send

Building a transport packet Building a transport packet

PAYLOAD

BTH PAYLOAD

IB End node

Application


TransportLayer

NetworkLayer

LinkLayer

PhysicalLayer

MPGA cont’ - Generating packet flow

Building a Link layer packet .Building a Link layer packet .

IB End node

Application


TransportLayer

NetworkLayer

LinkLayer

PhysicalLayer

LRH BTH PAYLOAD VCRCICRC

MPGA cont’ - Analyzing packet flow

???? Link layer of the incoming packet.

LRH ???? ICRC VCRC

Analyzing the transport layer of the incoming packet.

LRH BTH DETHH Payload ICRC VCRC

Payload pointer

Packet size

Mpga cont’ – special features

Endian proofEndian proof

ICRC/VCRC calculation (ICRC/VCRC calculation (Cyclic Redundancy CodeCyclic Redundancy Code ))

Error generationError generation

IB generator library

This library is the major driver for the Ibgenerator. This library is the major driver for the Ibgenerator.

The library uses the I2C library to communicate with the The library uses the I2C library to communicate with the Ibgenerator through the I2C master card (CALIBRE).Ibgenerator through the I2C master card (CALIBRE).

All of the basic structures of the Ibgenerator Lib is based All of the basic structures of the Ibgenerator Lib is based on the FPGA I2C interface defined in InfiniBand project on the FPGA I2C interface defined in InfiniBand project HW section.HW section.

IB generator library The main features are :The main features are :

8 to 108 to 10

Link phy sectionLink phy sectionSending TS1/TS2 (Tranning sequenc one/two).Sending TS1/TS2 (Tranning sequenc one/two).

Logical linkLogical link Sending Flow controls init and normal state.Sending Flow controls init and normal state.

PacketsPacketsSending a regular IB packet.Sending a regular IB packet.sending big buffers 4K MTU. sending big buffers 4K MTU.

I2C library

The I2C library is based on the API ofThe I2C library is based on the API of CALIBRE CompanyCALIBRE Company

The library provides I2C interface to theThe library provides I2C interface to the IB generator board.IB generator board.

Demonstration preview


IB tracerIB tracer Init the system.Init the system.

Ibgenerator device installation.Ibgenerator device installation. Caliber card installation.Caliber card installation. Installation of Ibgenerator software package.Installation of Ibgenerator software package.

Connect to the Agilent IB tracer. Connect to the Agilent IB tracer. Start the Ibgenerator GUI:Start the Ibgenerator GUI:

Send TS1.Send TS1. Send TS2.Send TS2. Send idle data.Send idle data. Send credit packet (init and normal state).Send credit packet (init and normal state).


Send the IB golden packet.Send the IB golden packet. Send RDMA write.Send RDMA write. Send Ack.Send Ack. Send errors:Send errors:

ICRC errorICRC error VCRC errorVCRC error Packet length error.Packet length error.

Send big packet (4k MTU).Send big packet (4k MTU).


Connect to Mellanox Infinibridge device through Agilent Connect to Mellanox Infinibridge device through Agilent IB tracer.IB tracer.

Send the following packets:Send the following packets: TS1TS1 TS2.TS2. Idle data.Idle data. Credit packet (init and normal state).Credit packet (init and normal state). Show physical and logical link state.Show physical and logical link state.

Send data packets to Mellanox device.Send data packets to Mellanox device.

Demonstration

computer networks & digital lab project

Documents