cms daq design
DESCRIPTION
Memory pipeline used in the frontend chips to store the incoming data, until a first level trigger decision is made. The trigger pulse marks the memory cell to be read out. CMS DAQ Design. Basic principle: CMS DAQ is a number of functionally identical, parallel, small DAQ systems - PowerPoint PPT PresentationTRANSCRIPT
US CMS DOE/NSF Review: May 19-21, 2003 1
Gig
aB
it
CP
U
Sp
ill
bu
ffe
r
Sp
ill
bu
ffe
r
Detectorfrontends
GEM & siliconAPV 25
Other detectors/frontends
Readoutmodules(~ 100)
S-Links100 - 160 MB/s
40 MB/s
GeSiCA TCS CATCH TCS
Readoutbuffers(16 PCs)
GigabitEthernet
Event builderand filter
Central datarecording
DAQcomputers(8 PCs)
Switch
Schematic view of the COMPASS data acquisition system
Memory pipeline used in the frontend chips to store the incoming data, until a first level trigger decision is made. The trigger pulse marks the memory cell to be read out.
US CMS DOE/NSF Review: May 19-21, 2003 2
ROB
TCSreceiver
Laser
TTClasercrate
Modulator
Encoder
1:32 passive optical coupler
Optical distribution network
Calibrationand monito-ring trigger
Hodoscopes, scintillationcounters, ECAL, HCAL
- Start/stop run - Configuration of controller and receivers- Request for calibration and monitoring triggers- Test and control
Online machineTCS server
38.88 MHzhigh precision
clock
Ch A (FLT) Ch B (data)
TriggerResetClock
BUSY
Frontends
FLT
VME P2 connector
6U VME module
TCScontroller
Spill buffer
OR
DAQRun control
S-Link
CATCH / GeSiCA38.88 MHz TCS clockFirst Level TriggerSpill numberEvent numberTrigger typeResetBegin of the spillEnd of the spillPretrigger
1:32 Coupler
Trigger logicVETO
The architecture of the COMPASS Trigger ControlSystem
US CMS DOE/NSF Review: May 19-21, 2003 3
CMS DAQ DesignCMS DAQ DesignCMS DAQ DesignCMS DAQ Design
Basic principle: • CMS DAQ is a number
of functionally identical, parallel, small DAQ systems
• A 64x64 system is feasible today
US CMS DOE/NSF Review: May 19-21, 2003 4
CMS DAQ DesignCMS DAQ DesignCMS DAQ DesignCMS DAQ Design
Basic principle: • CMS DAQ is a number
of functionally identical, parallel, small DAQ systems
• A 64x64 system is feasible today
US CMS DOE/NSF Review: May 19-21, 2003 5
Detector readout to surfaceDetector readout to surfaceDetector readout to surfaceDetector readout to surface
US CMS DOE/NSF Review: May 19-21, 2003 6
Data to Surface: SLink64/FRLData to Surface: SLink64/FRLData to Surface: SLink64/FRLData to Surface: SLink64/FRL
SLink64: From FED to FRL•Test pattern generator in sender card•Test pattern verification in FRL
FRL (Frontend Readout Link) •Multiple firmware versions•Spy functionality•Event generator allows testing DAQ system without operational FEDs
US CMS DOE/NSF Review: May 19-21, 2003 7
Commodity Computers + commidity networking (Myrinet/GigabitEthernet)
• Myricom’s LANaiX: PCI-X, multiple links, • LANai1XP: since 1 April 2003/LANai2XP: available Oct 2003
Event Builder:Event Builder:All Commodity BackboneAll Commodity Backbone
Event Builder:Event Builder:All Commodity BackboneAll Commodity Backbone
Myricom Switches: currently based on Xbar16Xbar32 under development First products expected 2Q 2004
Measured bandwidth with two LANai1XP NICs serving as PCI-X DMA engines:
“Grand Champion”: 710 MB/s per bus Intel “Pluma”: 426 MB/s per bus
i.e. 1.4 GB/s aggregate dual bus
“Grand Champion”chipset by Serverworks
US CMS DOE/NSF Review: May 19-21, 2003 8
0
20
40
60
80
100
120
140
100 1000 10000 100000
Fragment Size (Byte)
Th
rou
gh
pu
t p
er N
od
e (M
B/s
)
link BW (1Gbps)
8x8 EVB [P4 e1000 Powerconnect 5224]
32x32 EVB [P3 AceNIC FastIron8000]
GB Ethernet TCP EVB 8x8 GB Ethernet TCP EVB 8x8 GB Ethernet TCP EVB 8x8 GB Ethernet TCP EVB 8x8
XDAQ 8x8 EVB using gigabit ethernet
Each BU builds 128 evts Fixed fragment sizes
For fragment size > 4 kB:
Thru /node ~100 MB/s
• ie 80% utilization
(Specification is 200 MB/s)
Working point
US CMS DOE/NSF Review: May 19-21, 2003 9
0
50
100
150
200
250
0 10000 20000 30000 40000 50000 60000
Average Fragment Size (Byte)
Th
rou
gh
pu
t p
er N
od
e (M
B/s
)
link (2 Gbps)
Fixed Size
Variable size
Myricom EVB 8x8Myricom EVB 8x8Myricom EVB 8x8Myricom EVB 8x8
Full XDAQ 8x8 EVB using Myricom (LanAI9)
• software overhead (e.g. EVM included)
Each builder unit builds 128
evts Log-normal fragment
size distribution
At working point(16 kB):
Thu/node = 210 MB/s
(Specification is 200 MB/s)
Working point
US CMS DOE/NSF Review: May 19-21, 2003 10
Filter Farm Test StandFilter Farm Test StandFilter Farm Test StandFilter Farm Test Stand
SM
FDN/FCN
FU1/BU0
FU2-8
Dual 2.4 GHz Xeon with 2 GB memory, 40 GB disks and dual on-board GE.
Ran empty FU application: load BU output to a max at ~100 MB/s (1 FU/node), ~95 MB/s (2 FU/node)
HLT application (small event sample)CPU usage tests ongoing
TIER 0CSN
BU
MSS
FU FU FU FU SM
Filter Data Network
BUBU
FU FU FU FUFilterControlNetwork
US CMS DOE/NSF Review: May 19-21, 2003 11
Building names…Building names…Building names…Building names…Surface DAQ building
Underground Area