ATCA

ATCA electronics for HPS run designed in SLAC

• packages needed:

yum install cppzmq-devel

• SLAC confluence instructions for EEL Setup

Booting info: tftpboot from clon01, directory /tftpboot/slac-install/boot, contains kernel and 3 subdirectories containing files fpga.bit and devicetree.dtb. Subdirectory names DTM, ctrlDPM and dataDPM are set in computer center dhcp config file and available in bootloader as rootpath. Root tree is located on clonfs1 file server in directory /vol/diskless/ArchLinux/zynq_slac for nfs-mount.

• serial connection to atca1 crate: connect atca's serial port to computer USB using appropriate adapter, and use minicom from root@clondaq9:

minicom -D /dev/ttyUSB0 (set 115200 8N1)
root/HpsRoot
# reboot <- will recycle atca crate power

#ifconfig eth0 down
#ifconfig eth0 up 129.57.86.147

• atca crate is connected to the power strip, following command will recycle power:

roc_reboot atca1

• ethernet connection to atca1 crate:

ssh -o RSAMinSize=1024 -o HostKeyAlgorithms=+ssh-rsa root@atca1

DTMs

• connection to COB blade from root@clondaq10 (port labeled 'console', need usb-micro cable; NOTE: some cables we tried were not good, not sure why)

minicom -D /dev/ttyUSB0 (set 115200 8N1)

• DTM old (nfs-based) bootloader:

zynq-uboot> printenv
autoload=no
baudrate=115200
bootcmd=run $modeboot
bootdefs=console=ttyPS0,115200 rw rootwait earlyprintk
bootdelay=3
devicetree_image=devicetree.dtb
ethact=Gem.e000b000
ethaddr=08:00:56:00:46:0d
kernel_image=uImage
loadbit=0
modeboot=nfsboot
nfsargs=setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=dhcp
nfsboot=dhcp && run nfsload && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000
nfsfpga=nfs 0x1000000 ${rootpath}/boot/fpga.bit && fpga loadb 0 0x1000000 ${filesize}
nfsload=nfs 0x3000000 ${rootpath}/boot/${kernel_image} && nfs 0x2A00000 ${rootpath}/boot/${devicetree_image}
phycfg=1
ramdisk_devicetree=devicetree_ramdisk.dtb
ramdisk_image=uramdisk.image.gz
ramdisk_kernel=uImage
ramdisk_rootfs=/dev/ram
rtems_image=urtems.elf
sd_rootfs=/dev/mmcblk0p3
sdboot_linux=setenv modeboot sdboot_linux && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${kernel_image} && fatload mmc 0:7 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000
sdboot_rdisk=setenv modeboot sdboot_ramdisk && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${ramdisk_kernel} && fatload mmc 0:7 0x2A00000 ${ramdisk_devicetree} && fatload mmc 0:7 0x2000000 ${ramdisk_image} && bootm 0x3000000 0x2000000 0x2A00000
sdboot_rtems=echo Copying RTEMS from SD to RAM... && mmcinfo && fatload mmc 0:5 0x3000000 ${rtems_image} && bootm 0x3000000
stderr=serial
stdin=serial
stdout=serial

Environment size: 1527/131068 bytes
zynq-uboot> 

• DTM new (tftp-based) bootloader:

zynq-uboot> printenv
autoload=no
baudrate=115200
bootcmd=run $modeboot
bootdefs=console=ttyPS0,115200 rw rootwait earlyprintk
bootdelay=3
devicetree_image=devicetree.dtb
dnsip=129.57.90.255
ethact=Gem.e000b000
ethaddr=08:00:56:00:46:0d
fileaddr=2a00000
filesize=2a1d
gatewayip=129.57.86.1
hostname=dtm1
ipaddr=129.57.86.180
kernel_image=uImage
kernel_path=slac-install
loadbit=0
modeboot=tftp_boot
netmask=255.255.255.0
nfsargs=setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${nfsip}:${nfspath}/${rootpath},vers=3,proto=tcp ip=dhcp
nfsfpga=tftpboot 0x1000000 ${kernel_path}/boot/${rootpath}/fpga.bit && fpga loadb 0 0x1000000 ${filesize}
nfsip=129.57.167.152
nfspath=vol/diskless/ArchLinux/zynq_slac
phycfg=1
ramdisk_devicetree=devicetree_ramdisk.dtb
ramdisk_image=uramdisk.image.gz
ramdisk_kernel=uImage
ramdisk_rootfs=/dev/ram
rootpath=slac-install
rtems_image=urtems.elf
sd_rootfs=/dev/mmcblk0p3
sdboot_linux=setenv modeboot sdboot_linux && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${kernel_image} && fatload mmc 0:7 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000
sdboot_rdisk=setenv modeboot sdboot_ramdisk && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${ramdisk_kernel} && fatload mmc 0:7 0x2A00000 ${ramdisk_devicetree} && fatload mmc 0:7 0x2000000 ${ramdisk_image} && bootm 0x3000000 0x2000000 0x2A00000
sdboot_rtems=echo Copying RTEMS from SD to RAM... && mmcinfo && fatload mmc 0:5 0x3000000 ${rtems_image} && bootm 0x3000000
serverip=129.57.167.4
stderr=serial
stdin=serial
stdout=serial
tftp_boot=dhcp && run tftp_load && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000
tftp_load=tftpboot 0x3000000 ${kernel_path}/boot/${kernel_image} && tftpboot 0x2A00000 ${kernel_path}/boot/${rootpath}/${devicetree_image}

Environment size: 1917/131068 bytes
zynq-uboot>

DPMs

• usb connection to DPMs: ssh dtm1 as root/root, and from there, run

minicom bayX.Y             # bay <N>.<RCE>, see below

(names bayX.Y are defined in /etc/minirc.bayX.Y).

To reboot any DPM, run following on regular server:

cd /usr/clas12/release/1.4.0/slac_svt_new/V3.4.0/i86-linux-64/bin/
./cob_rce_reset atca1/2/3/0

DPM's list:

dpm8:
  minicom bay0.0
    ./cob_rce_reset atca1/2/0/0
dpm9:
  minicom bay0.2
    ./cob_rce_reset atca1/2/0/2
dpm10:
  minicom bay1.0
    ./cob_rce_reset atca1/2/1/0
dpm11:
  minicom bay1.2
    ./cob_rce_reset atca1/2/1/2
dpm12:
  minicom bay2.0
    ./cob_rce_reset atca1/2/2/0
dpm13:
  minicom bay2.2
    ./cob_rce_reset atca1/2/2/2
dpm14:
  minicom bay3.0
    ./cob_rce_reset atca1/2/3/0
dpm15:
  minicom bay3.2
    ./cob_rce_reset atca1/2/3/2

• to convert old bootloader to the new one do this:

setenv kernel_path 'slac-install'
setenv modeboot 'tftp_boot'
setenv nfsargs 'setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${nfsip}:${nfspath}/${rootpath},vers=3,proto=tcp ip=dhcp'
setenv nfsload
setenv tftp_load 'tftpboot 0x3000000 ${kernel_path}/boot/${kernel_image} && tftpboot 0x2A00000 ${kernel_path}/boot/${rootpath}/${devicetree_image}'
setenv nfsboot
setenv tftp_boot 'dhcp && run tftp_load && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000'
setenv nfsip '129.57.167.152'
setenv nfspath 'vol/diskless/ArchLinux/zynq_slac'
setenv nfsfpga 'tftpboot 0x1000000 ${kernel_path}/boot/${rootpath}/fpga.bit && fpga loadb 0 0x1000000 ${filesize}'
saveenv

• DPM old (nfs-based) bootloader:

zynq-uboot> 
autoload=no
bootcmd=run $modeboot
bootdefs=console=ttyPS0,115200 rw rootwait earlyprintk
bootdelay=3
devicetree_image=devicetree.dtb
ethact=Gem.e000b000
ethaddr=08:00:56:00:46:09
kernel_image=uImage
loadbit=0
modeboot=nfsboot
nfsargs=setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=dhcp
nfsboot=dhcp && run nfsload && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000
nfsfpga=nfs 0x1000000 ${rootpath}/boot/fpga.bit && fpga loadb 0 0x1000000 ${filesize}
nfsload=nfs 0x3000000 ${rootpath}/boot/${kernel_image} && nfs 0x2A00000 ${rootpath}/boot/${devicetree_image}
phycfg=0x1
ramdisk_devicetree=devicetree_ramdisk.dtb
ramdisk_image=uramdisk.image.gz
ramdisk_kernel=uImage
ramdisk_rootfs=/dev/ram
rtems_image=urtems.elf
sd_rootfs=/dev/mmcblk0p3
sdboot_linux=setenv modeboot sdboot_linux && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${kernel_image} && fatload mmc 0:7 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000
sdboot_rdisk=setenv modeboot sdboot_ramdisk && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${ramdisk_kernel} && fatload mmc 0:7 0x2A00000 ${ramdisk_devicetree} && fatload mmc 0:7 0x2000000 ${ramdisk_image} && boot0
sdboot_rtems=echo Copying RTEMS from SD to RAM... && mmcinfo && fatload mmc 0:5 0x3000000 ${rtems_image} && bootm 0x3000000
stderr=serial
stdin=serial
stdout=serial

Environment size: 1511/131068 bytes
zynq-uboot> 

• DPM new (tftp-based) bootloader:

zynq-uboot> printenv
autoload=no
bootcmd=run $modeboot
bootdefs=console=ttyPS0,115200 rw rootwait earlyprintk
bootdelay=3
devicetree_image=devicetree.dtb
ethact=Gem.e000b000
ethaddr=08:00:56:00:46:09
kernel_image=uImage
kernel_path=slac-install
loadbit=0
modeboot=tftp_boot
nfsargs=setenv bootargs ${bootdefs} root=/dev/nfs nfsroot=${nfsip}:${nfspath}/${rootpath},vers=3,proto=tcp ip=dhcp
nfsfpga=tftpboot 0x1000000 ${kernel_path}/boot/${rootpath}/fpga.bit && fpga loadb 0 0x1000000 ${filesize}
nfsip=129.57.167.152
nfspath=vol/diskless/ArchLinux/zynq_slac
phycfg=0x1
ramdisk_devicetree=devicetree_ramdisk.dtb
ramdisk_image=uramdisk.image.gz
ramdisk_kernel=uImage
ramdisk_rootfs=/dev/ram
rtems_image=urtems.elf
sd_rootfs=/dev/mmcblk0p3
sdboot_linux=setenv modeboot sdboot_linux && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${kernel_image} && fatload mmc 0:7 0x2A00000 ${devicetree_image} && bootm 0x3000000 - 0x2A00000
sdboot_rdisk=setenv modeboot sdboot_ramdisk && echo Copying Linux from SD to RAM... && mmcinfo && fatload mmc 0:7 0x3000000 ${ramdisk_kernel} && fatload mmc 0:7 0x2A00000 ${ramdisk_devicetree} && fatload mmc 0:7 0x2000000 ${ramdisk_image} && bootm 0x3000000 0x2000000 0x2A00000
sdboot_rtems=echo Copying RTEMS from SD to RAM... && mmcinfo && fatload mmc 0:5 0x3000000 ${rtems_image} && bootm 0x3000000
stderr=serial
stdin=serial
stdout=serial
tftp_boot=dhcp && run tftp_load && run nfsargs && bsir && run nfsfpga && bsic && bootm 0x3000000 - 0x2A00000
tftp_load=tftpboot 0x3000000 ${kernel_path}/boot/${kernel_image} && tftpboot 0x2A00000 ${kernel_path}/boot/${rootpath}/${devicetree_image}

Environment size: 1685/131068 bytes
zynq-uboot>

some old stuff

• DTMs/DPMs nfs mount

Clonfarm1 machine is used as file server for DTMs/DPMs, because they are not supporting newer nfs versions required by modern file servers like clonfs1. Make sure nfs server is set and running on clonfarm1. File /etc/exports suppose to contain following lines:

/data/hps/slac_svt/daq 129.57.0.0/16(rw,sync,no_subtree_check,no_root_squash)
#/data/hps/slac_svt/cache 129.57.0.0/16(rw,sync,no_subtree_check,no_root_squash)
/data/hps/slac_svt/server 129.57.0.0/16(rw,sync,no_subtree_check,no_root_squash)
/data/hps/slac_svt/diskless 129.57.0.0/16(rw,insecure,async,no_subtree_check,no_root_squash)

Run following to make sure, restart if needed:

systemctl status rpcbind
systemctl status nfs-server

• DTMs/DPMs boot

They are dhcp-boot devices, all settings are done by jlab computer center (Carl Bolitho). After boot, check it by logging in, for example ssh root@dtm1 with pass root (be patient, may take some seconds to login).

• some useful scripts in $CLAS/slac_svt/svtdaq/daqV2/rceScripts/:

reboot_cobs.sh - crate power recycle
shutdown_cobs.sh - turn power off (can be turned back on by reboot_cobs.sh)
rem_udp_server.sh - ... (have to run it)
rem_control_server.sh - start GUI server
start_gui.sh - start gui

EEL building test setup

machine used: clondaq10 with TIPCIe card to connect atca blades, and svt3 as master if clondaq10 is slave
run as 'hpsrun', pass the same as clasrun
EXPID=hpsrun, SESSION=clastest10
coda config: clondaq10_er, svt3_clondaq10_er (rol1=hpssvt1_master.so or hpssvt1_slave.so, rol2=rol2.so)
trigger files: HPS/EEL2021/hps_v1_noThr_TOPRCE_TIMaster.trg, HPS/EEL2021/hps_v1_noThr_TOPRCE_VMEMaster.trg
data files monitoring: $CODA/src/bosio/main/evio_hpssvthist.c, kumac hps.kumac

Cable TIPCIe-ATCA blade pinout ('new' TIPCIe):

signal        tipcie(40-pin)       atca(26-pin)
-------------------------------------------------
syncreset      1/2           --->      3/4
trigger        3/4           --->      5/6
busy          17/18          <---     11/12
clock         39/40          --->      7/8

Login and use of the test setup at SLAC

Tunneling to and starting VNC server at SLAC:

ssh -L 5905:localhost:1234 jlabl1
ssh -L 1234:localhost:5905 rdsrv117.slac.stanford.edu
vncserver :5 -geometry 1800x1000 -localhost -nolisten tcp
top

vncserver -kill :5

To have write access on develpc:

ssh clasrun@develpc
sudo tcsh

Running DAQ:

clasrun account can get root access with:
sudo bash

ssh clasrun@slac1
killall runcontrol rcServer rocs (cleanup just in case)
runcontrol -rocs -log
click on "rocs" tab on the right-top of the screen to observe built-in xterms
click "Configure", pick configuration "slac", processes have to be started in xterms
click "Download" and pick trigger file "slac1.trg"
click "Prestart"
click "Go"

• trigger file "slac1.trg" is located in default directory

$CLON_PARMS/trigger/. It contains electronics settings overloading defaults from $CLON_PARMS/ti, $CLON_PARMS/fadc250/ etc. In particular

  TI_RANDOM_TRIGGER 1 3

enables TI internal random pulser with prescale 2**3, with prescale 3 event rate will be about 50kHz.

Rogue GUI:

ssh clasrun@develpc
bash
cd /u1/hps/server/heavy-photon-daq/software
conda activate rogue-hps-dev-rth
python scripts/SvtCodaRun.py

Reboot RCE:

source /u1/hps/setup_env.sh
cob_rce_reset dev-crate1-sm/2

RCE boot status:

cob_dump -rce dev-crate1-sm

Log files if SLAC guys running daq:

/home/clasrun/Xterm.log.slac1.2019.06.07.14.44.29.pE87M0