ATCA

ATCA electronics for HPS run designed in SLAC

• packages needed:

yum install cppzmq-devel

• SLAC confluence instructions for EEL Setup

• serial connection to atca1 crate: connect atca's serial port to computer USB using appropriate adapter, and use minicom:

minicom -D /dev/ttyUSB0 (set 115200 8N1)
root/HpsRoot
# reboot <- will recycle atca crate power

#ifconfig eth0 down
#ifconfig eth0 up 129.57.86.147

• atca crate is connected to the power strip, following command will recycle power:

roc_reboot atca1

• ethernet connection to atca1 crate:

ssh -o RSAMinSize=1024 -o HostKeyAlgorithms=+ssh-rsa root@atca1

• serial connection to COB blade (port labeled 'console', need usb-micro cable; NOTE: some cables we tried were not good, not sure why

minicom -D /dev/ttyUSB0 (set 115200 8N1)

• old (nfs-based) bootloader:

U-Boot 2013.04-00036-gbc6012b (Nov 03 2016 - 08:23:40)

Memory: ECC disabled
DRAM:  512 MiB
WARNING: Caches not enabled
MMC:   zynq_sdhci: 0
reading uboot.env
In:    serial
Out:   serial
Err:   serial
Net:   eFUSE mac 08:00:56:00:46:0d
Net:   cm_net_init completed in 142 ms
Net:   Gem.e000b000
Hit any key to stop autoboot:  0 
zynq-uboot> printenv
autoload=no
baudrate=115200
bootcmd=run $modeboot
bootdefs=console=ttyPS0,115200 rw rootwait earlyprintk
bootdelay=3
devicetree_image=devicetree.dtb
ethact=Gem.e000b000
ethaddr=08:00:56:00:46:0d
kernel_image=uImage
loadbit=0
modeboot=nfsboot
nfsargs=setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=dhcp
nfsboot=dhcp && run nfsload && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000
nfsfpga=nfs 0x1000000 ${rootpath}/boot/fpga.bit && fpga loadb 0 0x1000000 ${filesize}
nfsload=nfs 0x3000000 ${rootpath}/boot/${kernel_image} && nfs 0x2A00000 ${rootpath}/boot/${devicetree_image}
phycfg=1
ramdisk_devicetree=devicetree_ramdisk.dtb
ramdisk_image=uramdisk.image.gz
ramdisk_kernel=uImage
ramdisk_rootfs=/dev/ram
rtems_image=urtems.elf
sd_rootfs=/dev/mmcblk0p3
sdboot_linux=setenv modeboot sdboot_linux && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${kernel_image} && fatload mmc 0:7 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000
sdboot_rdisk=setenv modeboot sdboot_ramdisk && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${ramdisk_kernel} && fatload mmc 0:7 0x2A00000 ${ramdisk_devicetree} && fatload mmc 0:7 0x2000000 ${ramdisk_image} && bootm 0x3000000 0x2000000 0x2A00000
sdboot_rtems=echo Copying RTEMS from SD to RAM... && mmcinfo && fatload mmc 0:5 0x3000000 ${rtems_image} && bootm 0x3000000
stderr=serial
stdin=serial
stdout=serial

Environment size: 1527/131068 bytes
zynq-uboot> 

• new (tftp-based) bootloader:

U-Boot 2013.04-00036-gbc6012b (Nov 03 2016 - 08:23:40)

Memory: ECC disabled
DRAM:  512 MiB
WARNING: Caches not enabled
MMC:   zynq_sdhci: 0
reading uboot.env
In:    serial
Out:   serial
Err:   serial
Net:   eFUSE mac 08:00:56:00:46:0d
Net:   cm_net_init completed in 141 ms
Net:   Gem.e000b000
Hit any key to stop autoboot:  0 
zynq-uboot> printenv
autoload=no
baudrate=115200
bootcmd=run $modeboot
bootdefs=console=ttyPS0,115200 rw rootwait earlyprintk
bootdelay=3
bootfile=linux-slac/boot/uImage
devicetree_image=devicetree.dtb
dnsip=129.57.90.255
enable-tftp=enp16s0f0
ethact=Gem.e000b000
ethaddr=08:00:56:00:46:0d
gatewayip=129.57.86.1
hostname=dtm1
ipaddr=129.57.86.180
kernel_image=uImage
loadbit=0
modeboot=nfsboot
netmask=255.255.255.0
nfsargs=setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${nfsip}:${nfspath},vers=3,proto=tcp ip=dhcp
nfsboot=dhcp && run nfsload && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000
nfsfpga=tftpboot 0x1000000 ${rootpath}/boot/fpga.bit && fpga loadb 0 0x1000000 ${filesize}
nfsip=129.57.167.152
nfsload=tftpboot 0x3000000 ${rootpath}/boot/${kernel_image} && tftpboot 0x2A00000 ${rootpath}/boot/${devicetree_image}
nfspath=/vol/diskless/ArchLinux/DTM
phycfg=1
ramdisk_devicetree=devicetree_ramdisk.dtb
ramdisk_image=uramdisk.image.gz
ramdisk_kernel=uImage
ramdisk_rootfs=/dev/ram
rootpath=/tftpboot/linux-slac
rtems_image=urtems.elf
sd_rootfs=/dev/mmcblk0p3
sdboot_linux=setenv modeboot sdboot_linux && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${kernel_image} && fatload mmc 0:7 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000
sdboot_rdisk=setenv modeboot sdboot_ramdisk && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${ramdisk_kernel} && fatload mmc 0:7 0x2A00000 ${ramdisk_devicetree} && fatload mmc 0:7 0x2000000 ${ramdisk_image} && bootm 0x3000000 0x2000000 0x2A00000
sdboot_rtems=echo Copying RTEMS from SD to RAM... && mmcinfo && fatload mmc 0:5 0x3000000 ${rtems_image} && bootm 0x3000000
serverip=129.57.167.4
stderr=serial
stdin=serial
stdout=serial

Environment size: 1839/131068 bytes
zynq-uboot> 

• ??? to support newer NFS versions, following has to be in boot settings (in CC ?):

enable-tftp=<ethernet port>
tftp-root=<location of diskless mount>

• DTMs/DPMs nfs mount

Clonfarm1 machine is used as file server for DTMs/DPMs, because they are not supporting newer nfs versions required by modern file servers like clonfs1. Make sure nfs server is set and running on clonfarm1. File /etc/exports suppose to contain following lines:

/data/hps/slac_svt/daq 129.57.0.0/16(rw,sync,no_subtree_check,no_root_squash)
#/data/hps/slac_svt/cache 129.57.0.0/16(rw,sync,no_subtree_check,no_root_squash)
/data/hps/slac_svt/server 129.57.0.0/16(rw,sync,no_subtree_check,no_root_squash)
/data/hps/slac_svt/diskless 129.57.0.0/16(rw,insecure,async,no_subtree_check,no_root_squash)

Run following to make sure, restart if needed:

systemctl status rpcbind
systemctl status nfs-server

• DTMs/DPMs boot

They are dhcp-boot devices, all settings are done by jlab computer center (Carl Bolitho). After boot, check it by logging in, for example ssh root@dtm1 with pass root (be patient, may take some seconds to login).

• some useful scripts in $CLAS/slac_svt/svtdaq/daqV2/rceScripts/:

reboot_cobs.sh - crate power recycle
shutdown_cobs.sh - turn power off (can be turned back on by reboot_cobs.sh)
rem_udp_server.sh - ... (have to run it)
rem_control_server.sh - start GUI server
start_gui.sh - start gui

EEL building test setup

machine used: clondaq10 with TIPCIe card to connect atca blades, and svt3 as master if clondaq10 is slave
run as 'hpsrun', pass the same as clasrun
EXPID=hpsrun, SESSION=clastest10
coda config: clondaq10_er, svt3_clondaq10_er (rol1=hpssvt1_master.so or hpssvt1_slave.so, rol2=rol2.so)
trigger files: HPS/EEL2021/hps_v1_noThr_TOPRCE_TIMaster.trg, HPS/EEL2021/hps_v1_noThr_TOPRCE_VMEMaster.trg
data files monitoring: $CODA/src/bosio/main/evio_hpssvthist.c, kumac hps.kumac

Cable TIPCIe-ATCA blade pinout ('new' TIPCIe):

signal        tipcie(40-pin)       atca(26-pin)
-------------------------------------------------
syncreset      1/2           --->      3/4
trigger        3/4           --->      5/6
busy          17/18          <---     11/12
clock         39/40          --->      7/8

Login and use of the test setup at SLAC

Tunneling to and starting VNC server at SLAC:

ssh -L 5905:localhost:1234 jlabl1
ssh -L 1234:localhost:5905 rdsrv117.slac.stanford.edu
vncserver :5 -geometry 1800x1000 -localhost -nolisten tcp
top

vncserver -kill :5

To have write access on develpc:

ssh clasrun@develpc
sudo tcsh

Running DAQ:

clasrun account can get root access with:
sudo bash

ssh clasrun@slac1
killall runcontrol rcServer rocs (cleanup just in case)
runcontrol -rocs -log
click on "rocs" tab on the right-top of the screen to observe built-in xterms
click "Configure", pick configuration "slac", processes have to be started in xterms
click "Download" and pick trigger file "slac1.trg"
click "Prestart"
click "Go"

• trigger file "slac1.trg" is located in default directory

$CLON_PARMS/trigger/. It contains electronics settings overloading defaults from $CLON_PARMS/ti, $CLON_PARMS/fadc250/ etc. In particular

  TI_RANDOM_TRIGGER 1 3

enables TI internal random pulser with prescale 2**3, with prescale 3 event rate will be about 50kHz.

Rogue GUI:

ssh clasrun@develpc
bash
cd /u1/hps/server/heavy-photon-daq/software
conda activate rogue-hps-dev-rth
python scripts/SvtCodaRun.py

Reboot RCE:

source /u1/hps/setup_env.sh
cob_rce_reset dev-crate1-sm/2

RCE boot status:

cob_dump -rce dev-crate1-sm

Log files if SLAC guys running daq:

/home/clasrun/Xterm.log.slac1.2019.06.07.14.44.29.pE87M0