DAQ and Trigger Systems for the CLAS12 detector

Version 1.1

DAQ and Trigger Systems for the CLAS12 detector

Abstract

The CLAS12 detector Data Aquisition and Trigger System are discussed.

Contents

1 Overview and Requirements 4 1.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Current DAQ System Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.1 Front-End Electronics: ADCs . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.2 Front-End Electronics: TDCs and Discriminators . . . . . . . . . . . . 5 1.2.3 Front-End Electronics: TDCs and ADBs in Drift Chambers . . . . . . 5 1.2.4 Trigger System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.5 Online Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.6 Online Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3 ‘Free-Running’ vs. Traditional DAQ Systems . . . . . . . . . . . . . . . . . . . 6 1.4 SVT Readout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Hardware Design 6 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Front-End for Fast Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Front-End for the Drift Chambers . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 CLAS12 Trigger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.4.1 Trigger Efficiency Study . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4.2 Level1 Trigger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.3 Level 2 Trigger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.4 Level 1/Level 2 Timing Diagram . . . . . . . . . . . . . . . . . . . . . 14

2.5 Readout Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6 Computing and Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Software 17 3.1 CODA Data Acquisition System . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 CLAS12-Specific Components Running as CODA Extensions . . . . . . . . . . 19 3.3 Level 3 Trigger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 Online Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.5 Online Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Conclusion 21

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

1 Overview and Requirements

1.1 Requirements

The following parameters are required for the CLAS12 Data Acquisition (DAQ) System: at least 10 kHz event rate, at least 100 MB/s data rate at the Event Recorder level, and not more than a 15% dead time under the above listed conditions.

1.2 Current DAQ System Analysis

Currently the CLAS DAQ system runs at an 8-10 kHz event rate and a 25-35 MB/s data rate at a dead time of not more than 15%. Since it will be partially re-used in CLAS12, we will discuss its main features and limitations, along with improvements necessary to achieve the CLAS12 requirements.

1.2.1 Front-End Electronics: ADCs

The currently used FASTBUS 1881M ADCs set a limit for the entire DAQ system: 12 μs conversion time projected into 12% dead time at 10 kHz event rate. It can be improved by switching to the low-resolution mode of these units, which sets a 9 μs conversion time. In this case the event rate limit can be pushed up to about 15 kHz. This means that from a performance point of view, the 1881M ADCs can be re-used, however, the number of FAST-BUS ADCs must be doubled to equip all new detectors, and delay cables and pretrigger discriminators must be purchased as well.

We have decided to replace all existing ADCs and equip all new detectors with Flash ADC boards. That decision provides higher performance for traditional DAQ system and free-running DAQ compartibility for future upgrades. Flash ADCs boards with on-board processing units will be used as first stage of Level1 trigger, so no pretrigger discriminators are required. Delay cables will be eliminated as well.

It should be mentioned that removing the 1881Ms will increase the Level 1 trigger decision time from hundreds of nanoseconds to at least 7 μs, making it much more powerful. In addition, Flash ADCs can deliver more detailed information to the trigger logic than the pretrigger discriminators.

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

1.2.2 Front-End Electronics: TDCs and Discriminators

All fast detectors in CLAS are equipped already with new pipeline TDC boards, both low (85 ps) and high (35 ps) resolution, and they will be re-used in CLAS12. All new detectors will be equipped with the same type of boards. We are planning to purchase new VME-based JLABmade discriminators to equip all channels, eliminating the existing CAMAC-based LeCroy discriminators, however, the LeCroy boards can be re-used if necessary.

1.2.3 Front-End Electronics: TDCs and ADBs in Drift Chambers

The drift chamber readout is equipped with FASTBUS 1877 TDCs, and according to our plan, they will be re-used in CLAS12. The conversion time and readout speed of the 1877s will not limit the DAQ readout rate from the 1877s until about 15 kHz, however, we may see aging effects, so a replacement solution for these boards will be provided.

The ADB electronics will be partially re-used but new ADB crate backplanes, MUX, and trigger boards (segment finders) will be designed. All FASTBUS and ADB crates will be equipped with new remotely controlled power supplies.

1.2.4 Trigger System

The current CLAS Level 1/Level 2 trigger system will be completely replaced with a new Level 1/Level 2 system capable of a few microseconds decision time. The new trigger system will include segment and track finding capabilities in the drift chambers, cluster finding in the calorimeters, and matching capabilities between different detectors. In addition, a Level 3 trigger will be running between the Event Builder and the Event Recorder for additional event rate reduction and/or data filtering. Level 3 will perform full event reconstruction, including hit-based tracking and time-based tracking if required. The trigger system in general will be flexible to accommodate different running conditions, with the main goal to search for events with electrons.

1.2.5 Online Monitoring

Four practically independent monitoring systems are currently used in CLAS: Nagios-based computer and network monitoring, SmartSocket-based DAQ and Online monitoring, EPICSbased slow controls monitoring, and data monitoring. In addition, a few smaller hardwarespecific systems are used. The data monitoring system includes visualization only with almost no alarm capabilities.

5  DAQ and Trigger

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

CLAS12 will have one integrated monitoring system, with a standard low-level interface compatible with the CLAS12 equipment. It will also be compatible with the accelerator control and monitoring system on some level. It will have not only monitoring, but also control capabilities. The implementation of such a system is under discussion and concept development will take at least one more year.

1.2.6 Online Calibration

The online calibration system must provide prompt information for the trigger system and online monitoring, and will be used for preliminary offline data processing. Such a system does not exist in CLAS and will be implemented in CLAS12.

1.3 ‘Free-Running’ vs. Traditional DAQ Systems

Since we will employ some old FASTBUS boards in the readout, the new DAQ system cannot be run in ‘free-running’ mode, however all new components will be compatible with the ‘freerunning’ concept. This means that the CLAS12 DAQ system will become ‘free-running’ after all FASTBUS boards are replaced in the future.

1.4 SVT Readout

The Silicon Vertex Tracker is not a part of the CLAS12 trigger system. The SVT readout is not included in the CLAS12 DAQ section, and will be described in the SVT section of this document.

2 Hardware Design

2.1 Overview

The generic scheme for the CLAS12 DAQ system hardware is shown in Fig. 1. This system will be described in detail in the following sections.

2.2 Front-End for Fast Detectors

CLAS12 will have two calorimeters, three time-of-flight systems, and two Čerenkov counters with a more than 4100 total PMT channels. Every PMT will be connected to a Flash ADC

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

Figure 1: Generic scheme for the CLAS12 DAQ system hardware.

and to a TDC through a discriminator (see Figs. 2 and 3). Splitter panels will be used to deliver signals to both ADC and TDC subystems. Four VME64X and nineteen VXS crates will be used to accommodate 56 TDC and 259 ADC boards. Sixteen VME crates will contain 262 discriminator boards.

The 16-channel JLab-made discriminators have a controllable threshold and output signal width. They have build-in scalers, and every VME crate with discriminators will have a readout controller (ROC) so that scalers can be read out. The discriminators were successfully used in the CLAS tagger system. A new revision of the discriminators for CLAS12 will have a test input for calibration purposes and a channel mask.

In CLAS12, 128-channel 85 ps resolution and 32-channel 35 ps resolution pipeline TDCs will be used. The first channel of every board will be connected to the reference signal through a discriminator to provide a timing reference. The CLAS system has been using these TDC boards for several years. A new version of the TDCs for CLAS12 will have improved readout

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

speed due to advanced 2eVME/2eSST data transfer protocols.

Figure 2: Planned scheme for the TDC readout in CLAS12.

The 16-channel JLab-made Flash ADCs are still under development and were not available for testing in CLAS at the time of writing of this document. However, based on our tests with commercially available Flash ADC boards (Struck SIS3320, see Refs. [3, 4]), CLAS12 can be equipped with 12-bit 200 MHz boards. We are expecting that the JLab FADCs will satisfy those requirements.

The FADCs are a significant part of the trigger system and will send prompt information to the Cluster Finder board installed in the same crate. The cluster information will include energy for every channel above threshold. A threshold will be applied to the energy sum of several consecutive hits. Information will be sent to the Cluster Finder over the back-plane serial bus every 16 ns. The Cluster Finder board is described in the Section 2.4.2, along with the energy summing algorithm.

Figure 3: Planned scheme for the ADC readout in CLAS12.

2.3 Front-End for the Drift Chambers

The existing CLAS preamplifier hybrid boards will be re-used for sence wire signal amplification. The existing MUX boards will be redesigned and the ADB crates will be equipped with new backplanes. The ADB crates will be re-used with new power supplies installed. Fig. 4 shows the new ADB electronics layout that is planned. One ADB crate will have 14 96-input preamplifier boards serving a complete drift chamber region, with 7 boards per superlayer (112×6 wires = 96×7 channels). The new MUX boards will send data to the new Segment Finder using a 96-wire bus, and ECL outputs to the TDCs after a 2-to-1 multiplexing. The ADB crate backplane has two 96-line buses, one per superlayer, ending in the middle of crate, where the two-unit-wide Segment Finder is installed. Communication between the various components is described in Section 2.4.2.

18 ADB crates will be connected to 6 FASTBUS crates containing 21 LeCroy 1877 TDC boards each. The FASTBUS crates will be upgraded with new Wiener power supplies including

Figure 4: Planned layout of an ADB crate for CLAS12.

remote control. In the future, the FASTBUS electronics can be replaced by 6 VME64X crates containing 16 v1190 CAEN TDC boards each (see Fig. 5).

In addition to aging problems, the FASTBUS TDCs may set another limitation for the CLAS12 DAQ system. Currently they are programmed to accept up to 4 hits during a 2 μs interval, which corresponds to the maximum drift time in the Region 3 of the CLAS drift chamber. The CLAS12 drift chambers will have similar drift times so TDC window will be set to the similar value of CLAS. Unfortunately, with the FASTBUS 1877 TDCs, we can only program the window size against a Common STOP signal. This signal is derived from the Level 1 trigger signal, and can be delayed few microseconds to the TDC inputs. In that situation, we have to open the TDC window by a few more microseconds, which may lead to more background hits. In that case we may need to allowed 8 or 16 hits per channel to make sure we are not losing useful hits. That may increase the TDC conversion time (which is hit-dependent) and data rate from the FASTBUS crates, and decrease the overall DAQ

Figure 5: Schematic layout of the planned readout configuration for the CLAS12 drift chambers.

performance. Our backup VME-based solution will be activated if the FASTBUS electronics will not be usable for any reason.

2.4 CLAS12 Trigger

CLAS12 will have a three-level trigger system. The Level 3 trigger is a software-based component and will be discussed in Section 3. The following sections contain a detailed description of the hardware-based Level 1 and Level 2 trigger components.

2.4.1 Trigger Efficiency Study

As a first step in the CLAS12 trigger system design, we studied the efficiency of the current CLAS trigger system. In particular, we studied its efficiency for electrons, which is important for most of the physics program. Detailed results can be found in Ref. [5]. We found that by increasing the Čerenkov counter pretrigger discriminator threshold up to 2 photoelectrons,

11  DAQ and Trigger

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

we can make the electron trigger much more selective. However, CLAS has never run with that threshold because of the possibility of losing electrons with low energy deposition in the Čerenkov counter. CLAS12 will have two Čerenkov counters, so even with a lower threshold, it is expected that electron selection will be improved. We also found that many non-electron events can be rejected by doing pattern recognition in the drift chambers and matching tracks with the fast detectors. The new CLAS12 trigger system was designed to be flexible enough to exercise different approaches, depending on the particular run requirements.

2.4.2 Level1 Trigger

The Level 1 trigger system is based on fast detectors equipped with Flash ADCs. The first stage components of the trigger hardware are incorporated into the Data Acquisition elements (VXS crates), while the final decision is made in a specialized trigger crate containing Level 1 Routers. The same crate also contains Level 2 Routers and Level 1/Level 2 matching logic (one per sector), as well as Event Processor and Trigger Supervisor boards as shown in Fig. 6. The system is driven by a 16 ns clock.

The VXS crates shown in Fig. 3 contain two trigger system components: processing units on the FADC boards (Fig. 7) and Cluster Finder (CF) boards collecting data from the Flash ADCs over fast serial lines. The CF boards search for clusters in the electromagnetic calorimeters and other detectors, and send the results to the following stage of the trigger system (see Fig. 8). A signal processing algorithm running in the FADC board processing unit is shown in Fig. 9.

2.4.3 Level 2 Trigger

The ADB crates shown in Fig. 4 contain the Segment Finder (SF) boards that collect data from the set of drift chamber MUX boards corresponding to whole region. The SF boards search for track segments and send them on to the following stage. The MUXs send data with a 15 ns strobe, so in a 105 ns interval, all data from both superlayers is injected into a programmable delay (see Fig. 10). The Level 1-driven switch and latch units accumulates the drift chamber hits during a 2 μs interval and passes the result to the Segment Finder board for processing.

The Segment Finder searches for segments in both superlayers simultaneously, then performs a region-based segment search, and finally builds a list of region-based segments. The system works as a pipeline and sends sets of segments to the Level 2 router. Every segment

Figure 6: Trigger crate logic diagram for CLAS12.

occupies 24 bits: 7 bits for wire numbers in U and 5 bits for the U-V difference for the first and last wires in the region-based segment.

Six Level 2 router boards are installed in the trigger crate along with six Level 1 router boards, an Event Processor (EP) board, and a Trigger Supervisor (TS) board. Each Level 2 router collects all information from the fast detectors and drift chambers installed in a particular sector, along with the information from the central detectors. It performs drift chamberbased track finding and matches the tracks with clusters from the fast detectors. The Event Processor collects the data from the Level 2 routers and produces the Level 2 trigger signals (see Fig. 6).

The Trigger Supervisor generates the Level 1/Level 2 pass/fail and other signals, and controls the entire DAQ system readout through the Trigger Interface units. The Trigger Interface (TI) units installed in every crate participating in the readout process.

Figure 7: Processing unit on the FADC board.

2.4.4 Level 1/Level 2 Timing Diagram

The generic scheme for the Level 1/Level 2 trigger timing is shown in Fig. 10. The entire trigger system can be considered as a three-stage process.

The first stage is the Level 1 sector-based trigger, which includes the FADC on-board processing (T1), data transfer over the VXS backplane serial bus (T2), cluster finding (T3), data transfer to the trigger crate (T4), sector-based Level 1 decision (T5), and data transfer to the following trigger stages. During that period of time (T1+T2+T3+T4+T5+T6), the drift chamber signals are delayed by a programmable delay on the MUX boards in the ADB crates. The T3 and T5 time intervals can be different for different run periods, so the programmable delay must be reprogrammed accordingly.

The second stage starts when the Level 1 signal arrives to the ADB crate initiating the drift chamber hit latching process (T7). The typical T7 timing is about 2 μs and is defined by the maximum drift time. Three latch units will be used for every channel, so up to three

Figure 8: Schematic layout of the Cluster Finder board.

Level 1 triggers can be processed in parallel. If a fourth Level 1 signal arrives while the first one is still latching, the whole trigger system will be stopped, however such events will be extremely rare based on the expected trigger rate. After the latching process is finished, the latched hits will be transferred to the Segment Finder (T8), processed there (T9), and the list of segments will be transferred to the Level 2 routers (T10) installed in the trigger crate. During that period of time (T7+T8+T9+T10), the Level 1 information (clusters from the Level 1 routers) will be delayed by another programmable delay.

Stage three is the final Level 2 trigger processing stage in the sector-based Level 2 routers (track finding and Level 1/Level 2 match), and in the Event Processor (the entire CLAS12 trigger) (T11). The total trigger decision time must be less than 7 μs to satisfy the Flash ADC range.

Figure 9: Signal processing algorithm on the FADC board.

2.5 Readout Controllers

Readout Controllers (ROCs) are installed in every FASTBUS, VME, VME64X, and VXS crate. The ROCs are collecting data from the front-end boards, processing it and sending it to the Event Builder over the network. Currently we employ mvme6100 controllers with a prpmc880 or pmc280 co-processor modules. That configuration is fast enough to meet the CLAS12 requirements and more co-processor modules can be installed if necessary. By the time of CLAS12 startup, a new generation of ROCs (including multi-processor and multi-core units) will be available.

2.6 Computing and Network

A Foundry 1500 switch is currently used as the backbone of the DAQ system, and a similar switch will be used for CLAS12. The Event Builder, Event Recorder, and other critical DAQ

Figure 10: Level 1 and Level 2 trigger timing diagram.

components for CLAS are running on 4-CPU Opteron-based servers, and that configuration is sufficient for CLAS12 as well. ROCs are linked to the 1500 Foundry switch through smaller switches with up to four GBit uplinks (see Fig. 11). The CLAS data storage system (RAID5) is sufficient for up to a 150 MB/sec data rate.

All available equipment is good enough to satisfy the CLAS12 requirements, but most likely we will upgrade most of the components because of aging and maintenance considerations. The performance of the link to the JLab Computer Center tape storage must be increased up to at least 200 MB/s.

3 Software

3.1 CODA Data Acquisition System

CLAS12 will use the CODA data acquisition software [1], similar to what we are using in

Figure 11: Planned scheme for CLAS12 networking.

CLAS. The CLAS version of CODA was modified in recent years to improve performance and reliability, and those changes will be incorporated into the future version of CODA, along with appropriate redesign and cleanup. The most important changes were ‘parallel’ front-end readout, multiprocessor ROC support, and a multi-threaded Event Builder. As a result, CLAS is running at an 8-10 kHz event rate, which is close to the CLAS12 requirements. We are expecting to get new software from the CODA group by the time of CLAS12 startup. Until then we will use the current CLAS version of CODA for detector commissioning and test runs.

Recent upgrades have eliminated most of bottlenecks from the DAQ software, but there are still places where improvements can be realized. One of them is multi-event readout and corresponding data coping inside the ROC memory. That modification will improve DAQ performance significantly for the free-running DAQ mode, but it will not have a big impact while CLAS12 is using FASTBUS 1877 TDCs. Those TDCs have 6 event buffers and need to be readout on an event-by-event basis to avoid back-logs. When the FASTBUS electronics

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

are replaced, we plan on utilizing multi-event readout.

3.2 CLAS12-Specific Components Running as CODA Extensions

Several CODA extensions were developed for the CLAS detector, and some of them will be re-used in CLAS12. The CODA fragment format was expanded by adding an additional data bank header; this way several data banks with different formats could be created and processed by the same ROC. This feature may be implemented into the new CODA version, or we can continue to use it as it is implemented now.

The currently used BOS data format will be obsolete, and the CODA data format will be used instead for all output data files, but data will be organized based on the detectors, not on the readout hardware layout. Again, it can be done by expanding the CODA format or by CLAS12-specific extensions.

The data translation procedure running in the ROCs will be re-used, with possible data format modifications. Two advantages of this method should be mentioned: the data volume reduction (especially in the drift chambers) and the ability to access data using the ‘offline’ format on the very first stage of the data acquisition process. Also the ROC-level histogram package played a critical role in the DAQ performance analysis, it will be re-used after adjustment to the new data format. The DAQ performance profiler will be re-used, although it maybe part of the new CODA version as well.

The Event Transfer (ET) system in CLAS was extended by adding new a package to handle multi-threaded attachments to a single ET station, with special care taken for preserving the event order. This method was successfully used by the Event Builder, Level 3, and the fast online reconstruction software running on an 8-CPU Sparc Server, and will be re-used in some form in CLAS12.

3.3 Level 3 Trigger

A multi-threaded Level 3 (software) trigger was implemented in CLAS but was used in tagging mode only. We will redesign that component for CLAS12 implementing full reconstruction up to at least the hit-based tracking level.

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

3.4 Online Monitoring

Online Monitoring will be addressed in CLAS12 as a part of the Experiment Control System (ECS) concept. This approach was used in previous years by other collaborations. We will use their experience to design the CLAS12 ECS.

ECS will collect all information coming from various CLAS12 components into one processing center, which will perform analysis and report problems with maximum possible precision. To make the whole system work successfully, all information from the low-level components will be reported in standard form, EPICS-channel access format is currently being considered. Special care will be taken to avoid false alarms.

The generic scheme for the CLAS12 Experiment Control System is shown in Fig. 12. Lowlevel hardware and data quality control components will communicate with top-level state machines using EPICS channel access and/or similar protocol. The data acquisition components will have a DAQ-specific communication system. Most state machines will be detector-specific with one central component supervising the entire CLAS12 setup. A set of logic diagrams will be developed to handle different situations during CLAS12 experiments, such as production data taking, calibration runs, test runs, troubleshooting, recovery, etc. These diagrams will be processed by the state machines.

Figure 12: Generic scheme of the CLAS12 Experiment Control System.

CLAS12 Technical Design Report  Version 1.1 June 18, 2007

The Experiment Control System for CLAS12 is currently under development. We are looking into several frameworks that can be used to build the CLAS12 ECS, such as CERN PVSS (used by all major CERN experiments), AFECS (under development at JLab), and others.

It may take a while to complete the design because of the dependency on the accelerator’s control system, which we have to be compatible with on some level. Several approaches are being considering by other JLab group, and we are planning to participate in joint efforts to obtain the system mostly suitable for CLAS12.

3.5 Online Calibration

The main goal of the online calibration system is to deliver the calibration parameters for the trigger components. Normally special runs will be taken and analyzed online before production data taking starts. Those runs will be processed and analyzed by software included into the Level 3 trigger component running in tagging mode.

4 Conclusion

Since the current CLAS DAQ system already has a performance close to that needed for CLAS12, we are confident that the CLAS12 goals will be achieved with available solutions. The main effort in the upcoming years will be spent to build the sophisticated trigger system that is required.

References

[1] CODA DAQ System.

[2] CLAS DAQ System.

[3] S.Pozdnyakov, CLAS-Note 2006-023.

[4] C.Smith, CLAS12-Note 2007-xxx.

[5] K.Mikhailov, CLAS-Note 2007-009.