| CLAS DAQ upgrade |
| Part2: Readout Controllers | |
| S.Boyarinov | |
| (CLAS online) | |
| D. Abbott (CODA) | |
| Project motivation |
| CLAS DAQ event rate was limited by 4.2kHz; typically 2.5-3.5kHz at deadtime below 20% | |
| single-ROC test shown that one of the limitations is in Readout Controller | |
| Goals |
| Make it possible to increase event rate for entire CLAS up to 10kHz at dead time below 15% | |
| New features |
| New VME controllers | |
| ÔParallelÕ first readout list | |
| Secondary CPU: hardware and BSP | |
| Dual-CPU CODA support | |
| New Ôcoda_rocÕ design | |
| Time profiling system | |
| New VME controllers |
| New Motorola VME controllers mv5500 and mv6100 were purchased, tested and installed | |
| Hardware problem was found in mv5500 power circuit, it was fixed by Motorola | |
| MotorolaÕs BSP for mv5500 was almost ready, few changes were applied | |
| MotorolaÕs BSP for mv6100 required some work, in particular new Tempe VME interface; some problems were fixed but one issue still remains (SFI-mv6100 incompatibility), it is under investigation | |
| Both mv5500 and mv6100 flavors are used in VME and FASTBUS crates | |
| ÔParallelÕ first readout list |
| Main idea: start FASTBUS/VME DMA in N-event interrupt routine and check DMA completion in (N+1)-event interrupt routine, so in most cases data is ready and we do not wait in interrupt | |
| Algorithm description: two new libraries were developed: directories ÔsfiDmaLibÕ and ÔvmeIntLibÕ; polling ROL1 and user interrupt library; new event queue; trigger routine structure (interrupt acknowledgement logic); ÔEndÕ transition logic; etc ÉÉÉÉ | |
| Figure: time profiling histograms for regular and parallel ROL1 | |
| Secondary CPU - hardware and BSP |
| prpmc800: first experience using carrier system, BSP adjustment, first working prototype with mv5500; switch to faster models | |
| prpmc880: BSP adjustment, working configurations on mv5100, mv5500 and mv6100; product dropped after 10 boards were delivered but replacement was promised | |
| pmc280: BSP adjustment, single- and dual-CPU versions, working configurations on mv2306, mv2432, mv5100, mv5500, mv6100; 14 boards delivered | |
| Dual-CPU CODA support: version 1 |
| Main idea: interrupt routine on CPU1, coda_roc on CPU2 | |
| Algorithm description: ÉÉÉÉ | |
| Figure: general scheme | |
| Dual-CPU CODA support: version 2 |
| Main idea: redesigned coda_roc on CPU1, second readout list and networking on CPU2 | |
| Algorithm description: ÉÉÉ. | |
| Figure: general scheme | |
| Slide 10 |
| New coda_roc design |
| Support running second readout list as thread (proc_thread) | |
| Support proc_thread and net_thread execution on CPU2 | |
| New coda_pmc main loop on CPU2 | |
| Support transition passing between CPU1 and CPU2 | |
| Replace message queue-based interprocess communication with simpler version to support communication over PCI bus | |
| New Ôbig buffersÕ library including DMA-based data transfer over PCI bus in ÔparallelÕ mode | |
| ÔSide effectsÕ: remnants of ÔtokenÕ were removed, TCL elimination except constructor/destructor just to support TCL prompt, C-based mSQL interface everywhere instead of TCL-based, new functions to replace TCL calls (string parsing etc), execute TCP and UDP services as separate threads, better parameter definitions (event size is defined in one place etc), CRL elimination in readout lists, general CODA source tree cleanup, etc | |
| Time profiling system |
| Time profiling was implemented for all three critical processes: interrupt routine, second readout list and networking | |
| System is based on CPU high-frequency clock and normalized to microseconds | |
| Measurement results are inserted into data stream as histograms to be viewed on ROOT-based presenter running downstream | |
| Simple histogram package was developed | |
| System allows real time ROC performance monitoring and can be used to make quick DAQ upgrades (install faster controller into busiest ROC, split ROC in two parts etc) | |
| Figures: ROL2 execution time (several peaks) |
| Project in numbers |
| 7 mv5500s and 11 mv6100s purchased, all of them in use | |
| 10 PrPMC880s and 14 PMC280s purchased, 19 of them currently installed and in use | |
| Software | |
| Overall cost over $100k |
| Project timeline |
| jun 2002: 2 prpmc800 boards ordered | |
| jun 2003: prpmc800 incompatible with mv5100, but will work with new mv5500 | |
| aug 2003: 7 mv5500s ordered | |
| oct 2003: hardware problem discovered with newly arrived mv5500s, and software problem related to PPC command swap discovered | |
| nov 2003: Motorola opened case for 7 mv5500s | |
| jan 2004: Motorola fixed mv5500 problem | |
| jan 2004: download prpmc800 BSP, ask for mv5500 or carrier board necessity | |
| jan 2004: ÔparallelÕ ROL1 and possibly dualCPUs projects deadline set to EG3 run | |
| feb 2004: still talking to Motorola how to use prpmc800, ask for prpmc880 availability; mv5500s installed in CLAS | |
| feb 2004: carrier board arrived, vxworks installed on prpmc800; waiting for documentation to set shared memory mv5500+prpmc800; two prpmc880s ordered | |
| mar 2004: 2 prpmc800s works on carrier after BSP fixes, will try it on mv5500 | |
| apr 2004: mv5500+prpmc800 with shared memory works, CODA implementation is considered | |
| Project timeline (cont.) |
| may 2004: prpmc880 arrived with BSPs, studies started; first version of 'parallel' ROL1 is ready, tests started; new mv6100 VME controlles are ordered | |
| jun 2004: prpmc880 under vxworks ported on mv5500 and mv5100; network design for PMCs started | |
| jun 2004: 'parallel' ROL1 successfully tested | |
| jul 2004: DMA works for prpmc880s, 22 more boards ordered | |
| PMC network installation started | |
| aug 2004: first 4 mv6100 boards arrived, BSP work started including prpmc880 porting | |
| sep 2004: 2 versions of CODA dual-cpu support are under development | |
| oct 2004: Motorola solved mv6100+prpmc880 problem; prpmc880 dropped, we will have 10 boards only; replacement promised | |
| nov 2004: CODA support for mv5500+prpmc800 released | |
| nov 2004: dual-cpu CODA tests started in tests setup | |
| nov-dec 2004: 'parallel' ROL1 and dual-cpu readout (version 2, network only on CPU2) were included in eg3 DAQ | |
| jan 2005: 'parallel' readout for VME; nothing more for eg3 | |
| Project timeline (cont.) |
| feb 2005: dual-cpu version 2 adopted for future use | |
| may 2005: new Motorola PMC280 sample board arrived, BSP work started | |
| july 2005: PMC280 ported on all mvme's flavors, 14 boards ordered | |
| sep 2005: serial/ethernet adapter under development in Fast Electronics Group | |
| dec 2005: adapters are ready, PMC280s tests resumed | |
| mar 2006: 11 pmc280s installed in CLAS, total number of PMCs is 19; project completed | |
| Conclusion |
| ÔParallelÕ first readout list and Dual-CPU solutions along with redesigned coda_roc increased CLAS DAQ system performance up to 7kHz at deadtime below 12% (final tests to be performed) | |
| Figure: event rate vs beam current for ÔparallelÕ ROL1 only, for dual-CPU only, for both |
| References |
| CLAS DAQ paper | |
| CODA paper | |
| Acknowledgments |
| CODA group | |
| Fast Electronics Group (PMC280 serial/ethernet adapters) | |
| Hall B Instrumentation Group (PrPMC880 cable adapters) | |