kazuro furukawa < kazuro . furukawa @ kek . jp > for linac and kekb control groups oct.2009

Kazuro Furukawa, KEK, Oct.2009.

Event System and Embedded IOCs

1EPICS Meeting, NSRRC, Taiwan

Event System and Embedded IOCs at KEK

Recent Activities at KEKMRF Event System for 50Hz Beam Switching

F3RP61, PLC-Embedded IOCs, etcATCA/TCA for LLRF

Kazuro Furukawa < kazuro . Furukawa @ kek . jp >

For Linac and KEKB Control Groups

Oct.2009.

< kazuro.furukawa @ kek.jp >




Simultaneous Injection Requirements

Linac clientsKEKB

8-GeV e– 1nC x23.5-GeV e+ 1nC x2

(with 10nC primary e–) PF 2.5-GeV e– 0.1nC(PF-AR 3-GeV e– 0.2nC)

At first simultaneous top-up injections to three rings at KEKB and PF Switching beams at 50Hz For stable operation and higher quality exp. results

Event System




KEKB Operation Improvement

May.2000

Apr.2003Dual Bunch

e+

Feb.2005ContinuousInjections

Event System




Linac & PF & KEKBSimultaneous Continuous Injection to PF, KEKB-HER and KEKB-LER

50Hz Beam Pulses are Shared between 3 RingsWith very different Beam Properties, in Energy, Charge, etc.

50Hz Beam Instrumentation (Beam Position Monitor)Only Passive Components other than Oscilloscope (Tek-DPO7104)

Windows-embedded (3GHz Intel), EPICS-3.14.9, VC++One Oscilloscope reads 2-5 BPMs, 24 Oscilloscopes Installed

Synchronized 100-BPM Read-out

Introduction of Event System, EVG230-EVR230RF from MRF10 EVR’s Installed, 1/3 of Old Timing Stations Replaced

VxWorks-5.5.1, EPICS-3.14.9, (Gave-up with RTEMS)Event drives Low-level RF in VME, BPM Oscilloscopes over NetworkGun Parameters, Pulsed Magnets, Kickers, etc are Controlled 50HzBeam Pattern Rules on Client Script, can be Downloaded every second

More Development NeededFlavoured Beam Feedback SystemsEvent System Integrity Monitor

EVG & Timing

EVR & LLRF

Event System




Event SystemMany accelerator system require timing signals and accompanying

information (event) Several facilities combined and used at KEKB and Linac

Fast Timing signals are provided with delay module TD4/TD4V Need timing trigger and rf clock

(Slow) Events are provided in another facility Combining Hardware and Software

Event/Timing Systems which distribute the both timing and event are developed at Argonne/SLS/Diamond, and are employed at many institutes (Event Generator/Receiver)

Fast Timing, rf clock, Hardware event, Software Interrupt, can be handled in one combined system with a single fiber cable

Especially in EPICS, event can be connected EPICS Event directly, so record/database programming is possible

Event generator

Event code

Coax Cable or Optical fiber link

Event receiverFanout Event receiver

Event receiver

Rf

AC or trigger

Output/Input

Event System




Event System Distribution mechanism of timing with data/information Developed based on experiences at several accelerator institutes

APS at Argonne (ANL/APS) Swiss Light Source (PSI/SLS)

New Event System (EVG/EVR-200/230) Employment at many accelerator institutes DIAMOND, SLS, BEPCII, LCLS, Shanghai, KEK-Linac, Australia, … (SNS), (LANL), (BNL), …

Many functionalities Bit rate up to 2.5Gbps, Event rate 50-125MHz, ~10ps precision, 8bit signal, 2kbyte data buffer, EPICS support

The stimulus to send an event can be: -pulse on a hardware input -software event (write to a register) -an entry in an event playback RAM

When an event code is received the receiver can: -output a pulse, of specified delay and width -trigger a software action (process an EPICS record) Each event receiver can be programmed to respond in a different way to the same event code.

Event generator

Event code

Optical fiber links

Event receiverFanout Event receiver

Event receiver

DIAMOND(TRISTAN, KEKB, Linac)

Rf

AC or trigger

Output/Input

Event System




Timing System

Old Sub-Timing StationNew Event Receiver Station (Timing) 16 outputs




Event SystemQuasi-simultaneous Injection

to KEKB-HER, KEKB-LER, and PF2.5GeV to 8GeV, 0.1nC to 10nC

Stable stored beam current at three ringsShould improve collision tuning with Crab cavitiesShould improve the quality of experimental data at PF

Fast switching of many device parametersIn 20ms / 50HzShould be reliable because beam power is much different

MRF Series 230 Event Generator / ReceiverVxWorks 5.5.1, MVME5500 (Originally with RTEMS but…)

Timing precision less than 10ps is sufficient (TD4 provides 3ps)

Multi-mode fiber, and single-mode fiber for longer distance

Event System




Event System Configuration

114.24MHz event rate, 50Hz fiducialsMore than hundred 50Hz-Analog/Timing PVsMulti/single-mode fiberTiming precision is < 10ps.

< 1ps with external module.

MRF’s series-230 Event Generator / Receivers. VME64x and VxWorks v5.5.1. EPICS R3.14.9 with DevSup v2.4.1. 17 event receivers for now.

Event System




Synchronization SchemeSynchronization Req.

KEKB : < 30psPF : < 300~700ps

Linac rf is Synchronized to KEKB rfEvent Clock is 114.24MHzWe have to manage

Circumference compensationBucket selectionInjection phase controlsRubidium

SG

Rubidium SG 10 MHz10 MHz

HER/LER Injection

Phase Control

HER/LER Injection

Phase ControlSGSG

114.24 MHz114.24 MHz

571.2 MHz571.2 MHz

2856 MHz2856 MHz

508.89 MHz508.89 MHz

SHB1 Phase Control

SHB1 Phase Control

SHB2 Phase Control

SHB2 Phase Control

KEKB HER/LER Circumference

Correction

KEKB HER/LER Circumference

Correction

Acc. Phase, Timing Control

Acc. Phase, Timing Control

99.39 kHz99.39 kHzHER/LER

BucketSelection

HER/LERBucket

Selection

EventSystem

EventSystem

508.89 MHz508.89 MHz

Flip-flopFlip-flop

Flip-flopFlip-flopPF Revolution

PF Circumference

Correction

PF Circumference

Correction

Bucket Selection

Bucket Selection

Flip-flop

Flip-flop 50Hz50Hz

AC Line Sync.1.6 MHz1.6 MHz

EventSystem

EventSystem

114.24MHz114.24MHz

Linac SHB

KEKB Revolution

x 5

x 5

X 49 ÷ 275

÷ 5120

for KEKB

Clock

Fiducial

for PF

Event System




Beam Mode Pattern Generation

Every pulse (every 20ms) corresponds to a beam mode. 10 different beam modes are defined (for KEKB e+, etc). One beam mode may contain many event codes.

At least one main code and a preparation code for the next pulse. About 50 event codes are defined.

Some events correspond to many functions, and others to specific devices. Beam pattern buffer length (n) can be 2 to 500 (20ms x 500 = 10 seconds). A new pattern can be loaded at the end of the previous pattern.

Otherwise, the pattern repeats forever. Pattern generator software arbitrates requests from downstream rings.

There are many pattern rules due to pulse device features and limitations. Pattern generator software is written in scripting languages to meet daily changes during the commissioning stage.

Beam Mode 1Beam Mode 1 Beam Mode 2Beam Mode 2 Beam Mode 3Beam Mode 3 ……………… Beam Mode nBeam Mode n

Pulse 1 Pulse 2 Pulse 3 Pulse n

Main eventcodes for ‘n’

Preparation eventcodes for ‘n+1’

Event System




Beam Mode Pattern GeneratorsThere are several versions

Because we were commissioning new pulsed hardware equipment, the beam optics schemes, event system itself, etc, since autumn 2008One of them is mostly used, remote or human controllable, automatic- prioritized arbitrated, etc

Remote controlled automatic pattern arbitrator Manual pattern generator

Typical operation in 2009.~25Hz for KEKB LER~12.5Hz for KEKB HER~0.5Hz for PF

Event System




LLRFLLRF Timing/analog signals are essential for absolute energy,

energy spread, and dual-bunch energy equalization.Signals are switched pulse-by-pulse.Value changes are triggered by a preparation event.Driver klystrons (SB), energy tuner klystron (KL), and

sub-harmonic bunchers (SH) are managed by the event system.

Event System




BPMTektronix DPO7104 can acquire data at >50Hz.

With embedded EPICSBeam modes are recognized by events through CA network.Clients can monitor data of an interested beam mode.26 oscilloscopes are installed.100 BPMs are synchronized. (100 BPMs at BT as well soon)

Event System




Measurement and Data AcquisitionOriginally much efforts to develop detectors, shaping amplifiers

No budget for all BPMsSwitched to direct waveform acquisition

Minimized active components, then minimized calibration tasks, maintenanceEqual-length cablesOne oscilloscope covers about 5 BPMs, or combined 20 (or 40) waveforms5 - 10Gs/s (with additional interpolation)Possible to measure dual bunchesSolved many issues at once!Extract each signal, apply calibration factors, send to upper layer at 50Hz




Database and Calibration FactorsPulse timing value for each electrode, each monitor, each of four beam modesDynamic range (voltage) for each beam modeMapping information up to 3rd order polynomial Cable loss for each electrode, combiner loss, charge conversions for single/multi-bunch beamsAbout 40 coefficients for each BPMProcessed on one of 24 DPO7104s in the framework of EPICS software then served directly to clients at 50Hz

Old system served at 1Hz

100 BPMs 19 x TDS680B 19 VMEs ~5 Unix Clients

100 BPMs 24 x DPO7104 Clients




Embedded IOC on OscilloscopeDPO7104, 10Gs/s, 4ch, 8bitWindows-XPCygwin software development environmentMicrosoft Visual C++ 2008

http://www-linac.kek.jp/cont/epics/win32/EPICS 3.14.8.2Fast data-acquisition at ~150Hz was tricky, but was possibleEvent triggers the data acquisitionBeam positions and charges are calculated based on ~30 coefficients, and tagged with beam modes50Hz processing is stable at LinacVery efficient for us




ParametersParameters switching via Event system

LLRF :14x4HP RF Timing :~60Gun voltages, fast delays, :4Pulsed magnets :14Injection system :4BPM over channel access x~100

Basically sufficient for fast beam mode switchingMore parameters commingIntegrity monitorsImproved slow beam feedback, fast feedback, etc.

Event System




Linac Event SystemSatisfies the requirements

Event rate : 114.24MHzFiducial rate : 50HzTiming jitter (Short term) : ~8psNo. of defined events : ~50No. of receiver stations (now) : 17No. of Fast parameters (now) : ~130

Beam currents are kept within

KEK 2mA (improving)PF 0.1mA (in 450mA)

CPU

EVG

EVR

Opt. Fan-out

Event System




KEKB Operation Improvement

May.2000

Apr.2003Dual Bunch

e+

Feb.2005ContinuousInjections

Dec.2008Crab Cavities and

Quasi-simultaneous Injection

Belle/KEK

Event System




(Initial) PLC usage at KEKAt e–/e+ Linac

We enforced that all the new controllers should be connected over IP/Ethernet since 1993 (instead of other field networks)PLC was much cost-effective compared with VME

if the speed requirement allowsProducts from OMRON, Mitsubishi, Yokogawa, etc. were installed

Only Yokogawa (FAM3) remained and others were removed, because maintenance capability over network was better

Ladder software downloadable over IP/Ethernet, etc.(Recently Mitsubishi also added that feature)

170 PLCs (with Ethernet) used for RF, Magnets, Vacuum, (Safety), etcAt J-PARC

Many installations with the same reasons as e-LinacAt KEKB

Used indirectly at many devices, over serial or GPIB linksEven custom hardware modules can be designed (I/O Open)

PLC embedded IOC




Vacuum Controller Internal Magnet Controller Internal RF Controller Internal

Safety Controller Touch Panel Display for RF




Software management with PLCsIdeal at the beginning with ladder softwareSeparate software developments at control group, at equipment group, or at industrial companyLater, integration test with IP/Ethernet

Logic management, howeverSame logics could be placed at ladder software, and in EPICS database/squencer (or in high-level applications)

Speed requirementClosed loop over Ethernet was slow, sometimes un-reliableInterrupts were possible, but slow and complicated

Thus, hoped to run EPICS on PLC

PLC embedded IOC




EPICS on PLCVxWorks CPU was available on PLC (Yokogawa, Mitsubishi)

Besides normal sequence / ladder CPUHowever, license management of vxWorks …

Yokogawa starts to provide Linux (2.6) on PLC CPU (F3RP61)Brave enough to choose open source environment

We negotiate with Yokogawa to remove any license issues Odagiri/KEK, Uchiyama/SHI-RIKEN, Yamada/KEK made much effort to realize the implementation, (no need for asynchronous records)Takuya-Nakamura/MSC-KEK tailored the environment for KEKB

Procserv, pcmon, NFS, …Six new PLC IOCs are used in KEKB operation

Since September 2008 and later, six in totalMany will be installed in 2010-2011 for vacuum, rf, …

Beam mask controllers and Pulsed-quad controllersNo trouble at all, they run more than 1 year

~20 new IOCs are also used in J-PARC operation now

PLC embedded IOC




F3RP61 (e-RT3 2.0)

Linux 2.6.24PPC 533MHz128Mbyte RAM100BaseTx x 2USBIEEE1394SerialPCII/O Bus for FAM3 Module Interface

can access to mature FAM3 I/O ModulesCan be combined with conventional ladder CPUSoftware development environment (ELDK)

KEKB Beam mask controller

PLC embedded IOC




Simple Usage under EPICSConventional PLC usage with asynchronous access

FAM3 PLCI/O ModulesFAM3 PLC

I/O ModulesF3RP61

IOCF3RP61

IOC


I/O Modules

LadderCPU

(Logics)

LadderCPU

(Logics)

OPIClientsOPI

ClientsIOC

(Logics)IOC

(Logics)

PLC usage with F3RP61 with only synchronous access and maybe with sequencer

OPIClientsOPI

Clients


I/O ModulesF3RP61

IOCF3RP61

IOC

If necessary, we can combine

OPIClientsOPI

ClientsLadderCPU

LadderCPU

PLC embedded IOC



EPICS Meeting, NSRRC, Taiwan 27

Device Support No need for asynchronous accessDirect access to all I/O modules

Can access to registers on ladder CPUIf necessary

Interrupts also possibleLogics can be database links or sequencersDid extend the number of EPICS developers

Source code and documentshttp://www-linac.kek.jp/cont/epics/f3rp61/

PREEMPT_RT realtime developement (Yamada, Yokogawa Co., et al)

PLC embedded IOC




Other Developments at KEKEmbedded IOC on FPGA controller

By A. Akiyama, et alEmbedded IOC on oscilloscopes

By M. Satoh, et alRedundant IOC (RIOC with OSI supports)Redundant GatewayATCA IOC with HPI/SAF support for RIOC

ATCA for STF/ILC-LLRF and TCA for cERL-LLRFAutomatic test system environment

By A. Kazakov, et alWireshark protocol analyzer for CA

By Klemen Zagar, et al




Embedded EPICS with FPGASuzaku/atmark-technoFPGA Vertex-4PPC Linux-2.6EPICS 3.14

J-PARC MPSKEKB MagnetLinac RF




J-PARC MR MPS Operational

Akiyama, Nakagawa, et al.Several Different Interfaces




Thank you




Old FAM3 failure considerations 10Base network interface 15 years ago was weak against broadcast storm

If we make a Ethernet loop, all the FAM3 in the segment had died.Our rf modulators are very noisy because of its grounding scheme and the voltage of 50kV, and 15-year-old PLC sometimes fails/stops and needs reboot

About five PLC failures per 60 rf PLCs per yearAbout one PLC failure per 50 other PLCs per year

Recent version has a redundant memory system in CPU module and relatively strong against noisesChemical capacitors have a lifetime of 8 years

We used more than 10 years, and found one failure for 100 PLCsMechanical relays (if it is used) have lifetime

Electronical lifetime 100k timesMechanical lifetime 10M times

kazuro furukawa < kazuro . furukawa @ kek . jp > for linac and kekb control groups oct.2009

Documents