https://clonwiki0.jlab.org/wiki/api.php?action=feedcontributions&feedformat=atom&user=129.57.167.4
CLONWiki - User contributions [en]
2026-05-07T11:08:32Z
User contributions
MediaWiki 1.42.1
https://clonwiki0.jlab.org/wiki/index.php?title=GE_XVB601&diff=5178
GE XVB601
2011-03-28T19:21:31Z
<p>129.57.167.4: /* Customization for the CLON cluster */</p>
<hr />
<div>Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]<br />
<br />
Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]<br />
<br />
== Customization for the CLON cluster ==<br />
<br />
'''List of boards'''<br />
<br />
* [01] 00:20:38:04:23:5E / 129.57.68.89 / croctest2<br />
<br />
'''Customizing kernel installed on the FLASH'''<br />
<br />
* ''/usr/bin/system-config-network''<br />
<br />
* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''<br />
<br />
* fix ''/etc/nsswitch.conf''<br />
<br />
* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')<br />
<br />
* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')<br />
<br />
* ''/etc/init.d/ypbind start''<br />
<br />
* ''chkconfig ypbind on''<br />
<br />
<br />
* set RHEL-style [[automounts]]<br />
<br />
* ''/etc/init.d/autofs restart''<br />
<br />
* fix symbolic links to /apps, /home, /usr/local etc<br />
<br />
<br />
<br />
'''Trying netboot following Bryan's instructions, using clonpc3'''<br />
<br />
* ''yum install system-config-netboot''<br />
<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* ''service xinetd restart''<br />
<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.167.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.167.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.167.0/255.255.255.0</code> only allows the 167 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
''service nfs reload''<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
* Enable remote syslog (optional)<br />
<br />
Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
Restart syslogd<br />
service syslog restart<br />
<br />
'''DHCP Server Setup'''<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
'''Configure Client OS'''<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
'''Setup of Diskless Kernel'''<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
'''Setup of chroot environment'''<br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
'''Setup of diskless system configurations'''<br />
<br />
''' Which NFS directories to mount '''<br />
<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
''' Configure a NEW diskless client '''<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
''' Configuration on the tftp/NFS server '''<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
''' Configuration on the Diskless Client '''<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
''' Other useful packages to install '''<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
== Network-based booting ==<br />
<br />
The page is geared towards the setup of Diskless nodes for Intel-based VME Controllers, but may be useful for similar other applications. Much of what is here was derived from the [http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/ch-diskless.html Redhat Enterprise Linux 3 Documentation].<br />
<br />
This examples assumes:<br />
* The DHCP server (phecda: 129.57.29.104) has been installed and works. Here we're using the dhcp 3.0.1 package distributed by RHEL4.<br />
* The TFTP and NFS server (mizar: 129.57.29.100) is running RHEL5.5 x86_64.<br />
* The Diskless Client (dafarm44: 129.57.29.54) will run Centos 5.5 x86_64.<br />
<br />
There's nothing stopping you from having the DHCP server on the same machine as that servering TFTP and NFS. You can obviously have more than one diskless client.<br />
<br />
Once the first four steps have been completed, new clients can be configured individually with step 5.<br />
<br />
= Client Setup of OS in VirtualBox =<br />
Here, we've downloaded the latest and greatest Centos 5.5 x86_64 ISO and have set up a VirtualBox Client to use it to boot. We will not attempt to describe the VirtualBox installation or Setup of the ISO to boot as the VirtualBox Guest.<br />
<br />
The Centos 5.5 installation proceeds as normal installations go, but we opt to do a minimal installation (excluding KDE and/or GNOME components) since we're running on a VME controller. Other packages will be installed after first boot.<br />
<br />
=== First Boot ===<br />
At first boot (after installation), a UI will appear and provide you with some options.:<br />
* Disable firewall and SELinux. SELinux setup can be double checked in the file:<br />
/etc/selinux/config<br />
* Disable sendmail, and other services not required for a VME controller<br />
<br />
=== Update packages and transfer the OS to Server ===<br />
Login as root<br />
* Update all installed packages:<br />
yum update<br />
* Install the following packages, with:<br />
yum install <i>package</i><br />
:: busybox-anaconda<br />
:: gcc<br />
:: kernel-devel<br />
:: gdb<br />
:: valgrind<br />
:: cvs<br />
:: subversion<br />
:: compat-libstdc++-33<br />
:: redhat-rpm-config<br />
<br />
* Copy the Client OS to the tftp/NFS Server.<br />
** On The Server:<br />
mkdir -p /diskless/x86_64/Centos5.5/root<br />
: In The VirtualBox Guest:<br />
rsync -Pav -e ssh --exclude='/proc/*' --exclude='/sys/*' / mizar:/diskless/x86_64/Centos5.5/root/<br />
<br />
You are now done with the Centos5.5 Guest installed in the VirtualBox. It can be closed down now.<br />
<br />
= tftp and NFS Server Setup =<br />
Follow these steps on the machine serving out tftp and NFS:<br />
<br />
===Install <code>system-config-netboot</code> ===<br />
(This should automatically install the tftp server)<br />
yum install system-config-netboot<br />
<br />
=== Enable the tftp server to start at boot ===<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* Restart <code>xinetd</code><br />
service xinetd restart<br />
<br />
=== Configure NFS ===<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.29.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.29.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.29.0/255.255.255.0</code> only allows the 29 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
service nfs reload<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
=== Enable remote syslog (optional) ===<br />
* Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
* Restart syslogd<br />
service syslog restart<br />
<br />
= DHCP Server Setup =<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
= Configure Client OS =<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
=== Setup of Diskless Kernel ===<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
=== Setup of chroot environment === <br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
=== Setup of diskless system configurations ===<br />
''' Which NFS directories to mount '''<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
= Configure a NEW diskless client =<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
=== Configuration on the tftp/NFS server ===<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
=== Configuration on the Diskless Client ===<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
= Other useful packages to install =<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=GE_XVB601&diff=5177
GE XVB601
2011-03-28T19:21:12Z
<p>129.57.167.4: /* Customization for the CLON cluster */</p>
<hr />
<div>Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]<br />
<br />
Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]<br />
<br />
== Customization for the CLON cluster ==<br />
<br />
'''List of boards'''<br />
<br />
* [01] 00:20:38:04:23:5E - 129.57.68.89 - croctest2<br />
<br />
'''Customizing kernel installed on the FLASH'''<br />
<br />
* ''/usr/bin/system-config-network''<br />
<br />
* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''<br />
<br />
* fix ''/etc/nsswitch.conf''<br />
<br />
* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')<br />
<br />
* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')<br />
<br />
* ''/etc/init.d/ypbind start''<br />
<br />
* ''chkconfig ypbind on''<br />
<br />
<br />
* set RHEL-style [[automounts]]<br />
<br />
* ''/etc/init.d/autofs restart''<br />
<br />
* fix symbolic links to /apps, /home, /usr/local etc<br />
<br />
<br />
<br />
'''Trying netboot following Bryan's instructions, using clonpc3'''<br />
<br />
* ''yum install system-config-netboot''<br />
<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* ''service xinetd restart''<br />
<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.167.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.167.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.167.0/255.255.255.0</code> only allows the 167 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
''service nfs reload''<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
* Enable remote syslog (optional)<br />
<br />
Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
Restart syslogd<br />
service syslog restart<br />
<br />
'''DHCP Server Setup'''<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
'''Configure Client OS'''<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
'''Setup of Diskless Kernel'''<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
'''Setup of chroot environment'''<br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
'''Setup of diskless system configurations'''<br />
<br />
''' Which NFS directories to mount '''<br />
<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
''' Configure a NEW diskless client '''<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
''' Configuration on the tftp/NFS server '''<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
''' Configuration on the Diskless Client '''<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
''' Other useful packages to install '''<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
== Network-based booting ==<br />
<br />
The page is geared towards the setup of Diskless nodes for Intel-based VME Controllers, but may be useful for similar other applications. Much of what is here was derived from the [http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/ch-diskless.html Redhat Enterprise Linux 3 Documentation].<br />
<br />
This examples assumes:<br />
* The DHCP server (phecda: 129.57.29.104) has been installed and works. Here we're using the dhcp 3.0.1 package distributed by RHEL4.<br />
* The TFTP and NFS server (mizar: 129.57.29.100) is running RHEL5.5 x86_64.<br />
* The Diskless Client (dafarm44: 129.57.29.54) will run Centos 5.5 x86_64.<br />
<br />
There's nothing stopping you from having the DHCP server on the same machine as that servering TFTP and NFS. You can obviously have more than one diskless client.<br />
<br />
Once the first four steps have been completed, new clients can be configured individually with step 5.<br />
<br />
= Client Setup of OS in VirtualBox =<br />
Here, we've downloaded the latest and greatest Centos 5.5 x86_64 ISO and have set up a VirtualBox Client to use it to boot. We will not attempt to describe the VirtualBox installation or Setup of the ISO to boot as the VirtualBox Guest.<br />
<br />
The Centos 5.5 installation proceeds as normal installations go, but we opt to do a minimal installation (excluding KDE and/or GNOME components) since we're running on a VME controller. Other packages will be installed after first boot.<br />
<br />
=== First Boot ===<br />
At first boot (after installation), a UI will appear and provide you with some options.:<br />
* Disable firewall and SELinux. SELinux setup can be double checked in the file:<br />
/etc/selinux/config<br />
* Disable sendmail, and other services not required for a VME controller<br />
<br />
=== Update packages and transfer the OS to Server ===<br />
Login as root<br />
* Update all installed packages:<br />
yum update<br />
* Install the following packages, with:<br />
yum install <i>package</i><br />
:: busybox-anaconda<br />
:: gcc<br />
:: kernel-devel<br />
:: gdb<br />
:: valgrind<br />
:: cvs<br />
:: subversion<br />
:: compat-libstdc++-33<br />
:: redhat-rpm-config<br />
<br />
* Copy the Client OS to the tftp/NFS Server.<br />
** On The Server:<br />
mkdir -p /diskless/x86_64/Centos5.5/root<br />
: In The VirtualBox Guest:<br />
rsync -Pav -e ssh --exclude='/proc/*' --exclude='/sys/*' / mizar:/diskless/x86_64/Centos5.5/root/<br />
<br />
You are now done with the Centos5.5 Guest installed in the VirtualBox. It can be closed down now.<br />
<br />
= tftp and NFS Server Setup =<br />
Follow these steps on the machine serving out tftp and NFS:<br />
<br />
===Install <code>system-config-netboot</code> ===<br />
(This should automatically install the tftp server)<br />
yum install system-config-netboot<br />
<br />
=== Enable the tftp server to start at boot ===<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* Restart <code>xinetd</code><br />
service xinetd restart<br />
<br />
=== Configure NFS ===<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.29.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.29.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.29.0/255.255.255.0</code> only allows the 29 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
service nfs reload<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
=== Enable remote syslog (optional) ===<br />
* Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
* Restart syslogd<br />
service syslog restart<br />
<br />
= DHCP Server Setup =<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
= Configure Client OS =<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
=== Setup of Diskless Kernel ===<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
=== Setup of chroot environment === <br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
=== Setup of diskless system configurations ===<br />
''' Which NFS directories to mount '''<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
= Configure a NEW diskless client =<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
=== Configuration on the tftp/NFS server ===<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
=== Configuration on the Diskless Client ===<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
= Other useful packages to install =<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=GE_XVB601&diff=5176
GE XVB601
2011-03-28T19:20:55Z
<p>129.57.167.4: /* Customization for the CLON cluster */</p>
<hr />
<div>Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]<br />
<br />
Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]<br />
<br />
== Customization for the CLON cluster ==<br />
<br />
'''List of boards'''<br />
<br />
* 1 - 00:20:38:04:23:5E - 129.57.68.89 - croctest2<br />
<br />
'''Customizing kernel installed on the FLASH'''<br />
<br />
* ''/usr/bin/system-config-network''<br />
<br />
* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''<br />
<br />
* fix ''/etc/nsswitch.conf''<br />
<br />
* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')<br />
<br />
* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')<br />
<br />
* ''/etc/init.d/ypbind start''<br />
<br />
* ''chkconfig ypbind on''<br />
<br />
<br />
* set RHEL-style [[automounts]]<br />
<br />
* ''/etc/init.d/autofs restart''<br />
<br />
* fix symbolic links to /apps, /home, /usr/local etc<br />
<br />
<br />
<br />
'''Trying netboot following Bryan's instructions, using clonpc3'''<br />
<br />
* ''yum install system-config-netboot''<br />
<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* ''service xinetd restart''<br />
<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.167.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.167.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.167.0/255.255.255.0</code> only allows the 167 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
''service nfs reload''<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
* Enable remote syslog (optional)<br />
<br />
Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
Restart syslogd<br />
service syslog restart<br />
<br />
'''DHCP Server Setup'''<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
'''Configure Client OS'''<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
'''Setup of Diskless Kernel'''<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
'''Setup of chroot environment'''<br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
'''Setup of diskless system configurations'''<br />
<br />
''' Which NFS directories to mount '''<br />
<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
''' Configure a NEW diskless client '''<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
''' Configuration on the tftp/NFS server '''<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
''' Configuration on the Diskless Client '''<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
''' Other useful packages to install '''<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
== Network-based booting ==<br />
<br />
The page is geared towards the setup of Diskless nodes for Intel-based VME Controllers, but may be useful for similar other applications. Much of what is here was derived from the [http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/ch-diskless.html Redhat Enterprise Linux 3 Documentation].<br />
<br />
This examples assumes:<br />
* The DHCP server (phecda: 129.57.29.104) has been installed and works. Here we're using the dhcp 3.0.1 package distributed by RHEL4.<br />
* The TFTP and NFS server (mizar: 129.57.29.100) is running RHEL5.5 x86_64.<br />
* The Diskless Client (dafarm44: 129.57.29.54) will run Centos 5.5 x86_64.<br />
<br />
There's nothing stopping you from having the DHCP server on the same machine as that servering TFTP and NFS. You can obviously have more than one diskless client.<br />
<br />
Once the first four steps have been completed, new clients can be configured individually with step 5.<br />
<br />
= Client Setup of OS in VirtualBox =<br />
Here, we've downloaded the latest and greatest Centos 5.5 x86_64 ISO and have set up a VirtualBox Client to use it to boot. We will not attempt to describe the VirtualBox installation or Setup of the ISO to boot as the VirtualBox Guest.<br />
<br />
The Centos 5.5 installation proceeds as normal installations go, but we opt to do a minimal installation (excluding KDE and/or GNOME components) since we're running on a VME controller. Other packages will be installed after first boot.<br />
<br />
=== First Boot ===<br />
At first boot (after installation), a UI will appear and provide you with some options.:<br />
* Disable firewall and SELinux. SELinux setup can be double checked in the file:<br />
/etc/selinux/config<br />
* Disable sendmail, and other services not required for a VME controller<br />
<br />
=== Update packages and transfer the OS to Server ===<br />
Login as root<br />
* Update all installed packages:<br />
yum update<br />
* Install the following packages, with:<br />
yum install <i>package</i><br />
:: busybox-anaconda<br />
:: gcc<br />
:: kernel-devel<br />
:: gdb<br />
:: valgrind<br />
:: cvs<br />
:: subversion<br />
:: compat-libstdc++-33<br />
:: redhat-rpm-config<br />
<br />
* Copy the Client OS to the tftp/NFS Server.<br />
** On The Server:<br />
mkdir -p /diskless/x86_64/Centos5.5/root<br />
: In The VirtualBox Guest:<br />
rsync -Pav -e ssh --exclude='/proc/*' --exclude='/sys/*' / mizar:/diskless/x86_64/Centos5.5/root/<br />
<br />
You are now done with the Centos5.5 Guest installed in the VirtualBox. It can be closed down now.<br />
<br />
= tftp and NFS Server Setup =<br />
Follow these steps on the machine serving out tftp and NFS:<br />
<br />
===Install <code>system-config-netboot</code> ===<br />
(This should automatically install the tftp server)<br />
yum install system-config-netboot<br />
<br />
=== Enable the tftp server to start at boot ===<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* Restart <code>xinetd</code><br />
service xinetd restart<br />
<br />
=== Configure NFS ===<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.29.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.29.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.29.0/255.255.255.0</code> only allows the 29 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
service nfs reload<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
=== Enable remote syslog (optional) ===<br />
* Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
* Restart syslogd<br />
service syslog restart<br />
<br />
= DHCP Server Setup =<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
= Configure Client OS =<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
=== Setup of Diskless Kernel ===<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
=== Setup of chroot environment === <br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
=== Setup of diskless system configurations ===<br />
''' Which NFS directories to mount '''<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
= Configure a NEW diskless client =<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
=== Configuration on the tftp/NFS server ===<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
=== Configuration on the Diskless Client ===<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
= Other useful packages to install =<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=GE_XVB601&diff=5175
GE XVB601
2011-03-28T19:15:31Z
<p>129.57.167.4: </p>
<hr />
<div>Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]<br />
<br />
Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]<br />
<br />
== Customization for the CLON cluster ==<br />
<br />
'''List of boards'''<br />
<br />
<br />
<br />
'''Customizing kernel installed on the FLASH'''<br />
<br />
* ''/usr/bin/system-config-network''<br />
<br />
* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''<br />
<br />
* fix ''/etc/nsswitch.conf''<br />
<br />
* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')<br />
<br />
* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')<br />
<br />
* ''/etc/init.d/ypbind start''<br />
<br />
* ''chkconfig ypbind on''<br />
<br />
<br />
* set RHEL-style [[automounts]]<br />
<br />
* ''/etc/init.d/autofs restart''<br />
<br />
* fix symbolic links to /apps, /home, /usr/local etc<br />
<br />
<br />
<br />
'''Trying netboot following Bryan's instructions, using clonpc3'''<br />
<br />
* ''yum install system-config-netboot''<br />
<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* ''service xinetd restart''<br />
<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.167.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.167.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.167.0/255.255.255.0</code> only allows the 167 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
''service nfs reload''<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
* Enable remote syslog (optional)<br />
<br />
Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
Restart syslogd<br />
service syslog restart<br />
<br />
'''DHCP Server Setup'''<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
'''Configure Client OS'''<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
'''Setup of Diskless Kernel'''<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
'''Setup of chroot environment'''<br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
'''Setup of diskless system configurations'''<br />
<br />
''' Which NFS directories to mount '''<br />
<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
''' Configure a NEW diskless client '''<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
''' Configuration on the tftp/NFS server '''<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
''' Configuration on the Diskless Client '''<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
''' Other useful packages to install '''<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
== Network-based booting ==<br />
<br />
The page is geared towards the setup of Diskless nodes for Intel-based VME Controllers, but may be useful for similar other applications. Much of what is here was derived from the [http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/ch-diskless.html Redhat Enterprise Linux 3 Documentation].<br />
<br />
This examples assumes:<br />
* The DHCP server (phecda: 129.57.29.104) has been installed and works. Here we're using the dhcp 3.0.1 package distributed by RHEL4.<br />
* The TFTP and NFS server (mizar: 129.57.29.100) is running RHEL5.5 x86_64.<br />
* The Diskless Client (dafarm44: 129.57.29.54) will run Centos 5.5 x86_64.<br />
<br />
There's nothing stopping you from having the DHCP server on the same machine as that servering TFTP and NFS. You can obviously have more than one diskless client.<br />
<br />
Once the first four steps have been completed, new clients can be configured individually with step 5.<br />
<br />
= Client Setup of OS in VirtualBox =<br />
Here, we've downloaded the latest and greatest Centos 5.5 x86_64 ISO and have set up a VirtualBox Client to use it to boot. We will not attempt to describe the VirtualBox installation or Setup of the ISO to boot as the VirtualBox Guest.<br />
<br />
The Centos 5.5 installation proceeds as normal installations go, but we opt to do a minimal installation (excluding KDE and/or GNOME components) since we're running on a VME controller. Other packages will be installed after first boot.<br />
<br />
=== First Boot ===<br />
At first boot (after installation), a UI will appear and provide you with some options.:<br />
* Disable firewall and SELinux. SELinux setup can be double checked in the file:<br />
/etc/selinux/config<br />
* Disable sendmail, and other services not required for a VME controller<br />
<br />
=== Update packages and transfer the OS to Server ===<br />
Login as root<br />
* Update all installed packages:<br />
yum update<br />
* Install the following packages, with:<br />
yum install <i>package</i><br />
:: busybox-anaconda<br />
:: gcc<br />
:: kernel-devel<br />
:: gdb<br />
:: valgrind<br />
:: cvs<br />
:: subversion<br />
:: compat-libstdc++-33<br />
:: redhat-rpm-config<br />
<br />
* Copy the Client OS to the tftp/NFS Server.<br />
** On The Server:<br />
mkdir -p /diskless/x86_64/Centos5.5/root<br />
: In The VirtualBox Guest:<br />
rsync -Pav -e ssh --exclude='/proc/*' --exclude='/sys/*' / mizar:/diskless/x86_64/Centos5.5/root/<br />
<br />
You are now done with the Centos5.5 Guest installed in the VirtualBox. It can be closed down now.<br />
<br />
= tftp and NFS Server Setup =<br />
Follow these steps on the machine serving out tftp and NFS:<br />
<br />
===Install <code>system-config-netboot</code> ===<br />
(This should automatically install the tftp server)<br />
yum install system-config-netboot<br />
<br />
=== Enable the tftp server to start at boot ===<br />
* Change line (<code>/etc/xinetd.d/tftp</code>):<br />
disable = yes<br />
: To<br />
disable = no<br />
<br />
* Restart <code>xinetd</code><br />
service xinetd restart<br />
<br />
=== Configure NFS ===<br />
* Add these entries to <code>/etc/exports</code><br />
/diskless/x86_64/Centos5.5/root/ 129.57.29.0/255.255.255.0(ro,sync,no_root_squash)<br />
/diskless/x86_64/Centos5.5/snapshot/ 129.57.29.0/255.255.255.0(rw,sync,no_root_squash)<br />
: Where the <code>129.57.29.0/255.255.255.0</code> only allows the 29 subnet to access these mounts.<br />
<br />
* Reload the NFS configuration or start NFS, if it's not started<br />
service nfs reload<br />
<br />
* Make sure NFS is started at boot<br />
/sbin/chkconfig --level 345 nfs on<br />
<br />
=== Enable remote syslog (optional) ===<br />
* Change syslog argument in <code>/etc/sysconfig/syslog</code> from:<br />
SYSLOGD_OPTIONS="-m 0"<br />
:: To<br />
SYSLOGD_OPTIONS="-m 0 -r"<br />
<br />
* Restart syslogd<br />
service syslog restart<br />
<br />
= DHCP Server Setup =<br />
On the machine hosting the DHCP server.<br />
<br />
* Add an entry in <code>/etc/dhcpd.conf</code>:<br />
<pre><br />
host dafarm44<br />
{<br />
filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/<br />
next-server 129.57.29.38; # TFTP server<br />
hardware ethernet 00:20:38:03:D8:52; # Client MAC<br />
fixed-address 129.57.29.54; # Client IP<br />
}<br />
</pre><br />
<br />
* Restart dhcpd<br />
service dhcpd restart<br />
<br />
= Configure Client OS =<br />
<br />
Here, we configure the OS that each and every Diskless Client will enjoy.<br />
<br />
=== Setup of Diskless Kernel ===<br />
Run system-config-netboot (on TFTP server):<br />
system-config-netboot<br />
<br />
* In "First Time Druid" window. Click on "Diskless"<br />
# Click "Forward" on first page<br />
# Enter Name and Description.<br />
# Enter IP address of TFTP server and the "root" directory exported via NFS<br />
# Select a kernel<br />
# Click on Apply.<br />
: This will take a minute or two to compile a boot kernel for the diskless client.<br />
<br />
=== Setup of chroot environment === <br />
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).<br />
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.<br />
chroot /diskless/x86_64/Centos5.5/root<br />
mount /proc /proc -t proc<br />
mkdir /u <br />
mkdir /group<br />
mkdir /daqfs<br />
mv /home /home.old<br />
ln -s /u/apps /apps<br />
ln -s /u/site /site<br />
ln -s /daqfs/home /home<br />
<br />
=== Setup of diskless system configurations ===<br />
''' Which NFS directories to mount '''<br />
* Add entry to <code>/etc/fstab</code><br />
phecda:/raid /daqfs nfs rw,soft,intr 0 0<br />
<br />
* Mount daqfs<br />
mount /daqfs<br />
<br />
''' Which users/groups to allow access '''<br />
* Add entries to /etc/passwd<br />
+@da::::::<br />
+@ccc::::::<br />
+::::::/bin/false<br />
<br />
''' A proxy for yum ''' (may be optional)<br />
* Add entry to /etc/yum.conf<br />
proxy=http://jprox:8080<br />
<br />
''' domain servers for ypbind '''<br />
* Add entries to /etc/yp.conf<br />
domain CCCHP server nis2.jlab.org<br />
domain CCCHP server nis3.jlab.org<br />
<br />
* Make sure ypbind started at boot<br />
/sbin/chkconfig --level 345 nfs on <br />
<br />
''' NTP setup ''' (optional... ntp must be installed)<br />
* Add entries to <code>/etc/ntp.conf</code><br />
server 127.0.0.1 # local clock<br />
fudge 127.0.0.1 stratum 10<br />
server ntp1.jlab.org<br />
server ntp2.jlab.org<br />
driftfile /etc/ntp/drift<br />
broadcastdelay 0.008<br />
authenticate no<br />
<br />
''' Some additional JLab CUE directories to mount '''<br />
* Add entries to <code>/etc/auto.master</code><br />
/group /etc/auto.group --timeout=0<br />
/u auto.u.bb --DOSNAME=Linux-RHEL5<br />
<br />
''' Additional files, not included in the standard diskless filesystem '''<br />
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>. Entry:<br />
/tmp/<br />
/etc/minirc.dfl<br />
<br />
= Configure a NEW diskless client =<br />
This operation must be peformed to add a new Diskless Client. <br />
<br />
=== Configuration on the tftp/NFS server ===<br />
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:<br />
# Click on "New"<br />
# Enter hostname<br />
# Select OS<br />
# Enter "CCCHP" in Enable NISDOMAIN box.<br />
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code><br />
# Enter hostname as snapshot name. Make sure "generate" is check-marked<br />
# Click "OK"<br />
<br />
=== Configuration on the Diskless Client ===<br />
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.<br />
<br />
= Other useful packages to install =<br />
Later on... I installed some other useful packages using the chroot environment. Here they are:<br />
<br />
* <code>xorg-x11-xauth</code> - Required to run X-applications remotely<br />
* <code>ntp</code> - Synchronize system clock with a central server<br />
: To Execute <code>ntpdate</code> on server startup:<br />
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in<br />
/etc/sysconfig/ntpd<br />
:* Make sure it starts on boot<br />
/sbin/chkconfig --level 345 nfs on</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=CLON_Standard_Procedures&diff=4946
CLON Standard Procedures
2010-10-01T21:55:25Z
<p>129.57.167.4: </p>
<hr />
<div>'''Contacts:'''<br />
* Computer Center 24/7 phone: 876-1794<br />
* Sergey Boyarinov: USA cell: 757-869-2188 <br />
* Nerses: approximately up to 15:00 eastern time: home number is +374 10 425 049, cell phone +374 91 206 217 (Armenia), USA cell: 757-358-3042</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=PERL&diff=4913
PERL
2010-09-08T14:31:07Z
<p>129.57.167.4: </p>
<hr />
<div>We are installing PERL modules directly into the PERL system location. Normally PERL modiles will be obtained from [http://www.cpan.org/] in a form of tar files and placed in /usr/local/downloads. For every platform they must be copied to /usr/local/src, untared and installed. Following modules must be installed:<br />
Pezca<br />
Tk<br />
DBI<br />
DBD::mysql<br />
Time<br />
URI<br />
HTML::Tagset<br />
HTML::Parser<br />
IO-Compress-Base<br />
Compress-Raw-Zlib (see note below)<br />
Compress-Raw-Bzip2<br />
IO-Compress-Bzip2<br />
IO-Compress-Zlib<br />
Compress-Zlib<br />
LWP (libwww-perl) (use default options when asked)<br />
XML-Parser<br />
<br />
For installation instructions see README inside those packages. Usually you have to type:<br />
perl Makefile.PL<br />
gmake<br />
gmake test (optionally)<br />
gmake install<br />
<br />
Following was noticed:<br />
<br />
* Pezca_0.3: After running 'perl Makefile.PL' and before 'gmake', modify 'Makefile': after '-Iinclude', add '-Iinclude/os/solaris -Iinclude/O.solaris' (adjust for OS; do not know why it did not happened during 'perl Makefile.PL' ...). Also, module 'osdTime.cc' must be compiled by 'CC', not by 'cc' as it is happening (just go to 'libCom/os/generic', type 'gmake', copy last failed command, replace 'cc' by 'CC' and execute).<br />
<br />
* Pezca_0.4: After running 'perl Makefile.PL.ontop' and before 'gmake', modify 'Makefile': after -I.../include/os add '/solaris' with corresponding flavor. Also, version 0.4 can be (and must be) compiled using our EPICS, not the one which comes with Pezca. To do so EPICS_BASE and EPICS_EXTENSIONS environment variables must be defined, and configuring must be done by command ''perl Makefile.PL.ontop''. On solaris-x86, modify ''platform_ontop.pm'' as following:<br />
# platform dependant sub-directory for include files (on sparc-solaris probably the same using ''solaris-sparc''):<br />
my %os_inc_subdir= ('linux-x86' => 'Linux',<br />
'win32-x86' => 'WIN32',<br />
'hpux-11' => 'hp700',<br />
'solaris' => 'solaris', <----- added<br />
);<br />
# platform dependant sub-directory for libraries in the base:<br />
my %os_b_lib_subdir= ('linux-x86' => 'linux-x86',<br />
'win32-x86' => 'win32-x86',<br />
'hpux-11' => 'hp700',<br />
'solaris' => 'solaris-x86', <----- added<br />
);<br />
# platform dependant sub-directory for libraries in the extensions directory:<br />
my %os_e_lib_subdir= ('linux-x86' => 'linux-x86',<br />
'win32-x86' => 'win32-x86',<br />
'hpux-11' => 'hp700',<br />
'solaris' => 'solaris-x86', <----- added<br />
);<br />
<br />
* DBI:<br />
<br />
clon01:DBI-1.52> perl Makefile.PL<br />
*** Note:<br />
The optional PlRPC-modules (RPC::PlServer etc) are not installed.<br />
If you want to use the DBD::Proxy driver and DBI::ProxyServer<br />
modules, then you'll need to install the RPC::PlServer, RPC::PlClient,<br />
Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you.<br />
You can install them any time after installing the DBI.<br />
You do *not* need these modules for typical DBI usage.<br />
Optional modules are available from any CPAN mirror, in particular<br />
http://search.cpan.org/<br />
http://www.perl.com/CPAN/modules/by-module<br />
http://www.perl.org/CPAN/modules/by-module<br />
ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module<br />
<br />
* Compress-Raw-Zlib:<br />
<br />
sed -e "s|BUILD_ZLIB\s*= True|BUILD_ZLIB = False|" -e "s|INCLUDE\s*= ./zlib-src|INCLUDE = /usr/include|" -e "s|LIB\s*= ./zlib-src|LIB = /usr/lib|" config.in<br />
<br />
Install all modules on one machine of each flavor and execute following for the rest:<br />
<br />
cd /usr/local/src/Pezca-0.4<br />
gmake install<br />
cd /usr/local/src/Tk-804.027/<br />
gmake install<br />
cd /usr/local/src/DBI-1.52/<br />
gmake install<br />
cd /usr/local/src/DBD-mysql-3.0008/<br />
gmake install<br />
cd /usr/local/src/Time-modules-2006.0814/<br />
gmake install<br />
cd /usr/local/src/URI-1.35/<br />
gmake install<br />
cd /usr/local/src/HTML-Tagset-3.10/<br />
gmake install<br />
cd /usr/local/src/HTML-Parser-3.56/<br />
gmake install<br />
cd /usr/local/src/IO-Compress-Base-2.005/<br />
gmake install<br />
cd /usr/local/src/Compress-Raw-Zlib-2.005/<br />
gmake install<br />
cd /usr/local/src/Compress-Raw-Bzip2-2.005/<br />
gmake install<br />
cd /usr/local/src/IO-Compress-Bzip2-2.005/<br />
gmake install<br />
cd /usr/local/src/IO-Compress-Zlib-2.005/<br />
gmake install<br />
cd /usr/local/src/Compress-Zlib-2.005/<br />
gmake install<br />
cd /usr/local/src/libwww-perl-5.808/<br />
gmake install<br />
cd /usr/local/src/XML-Parser-2.36/<br />
gmake install</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=PERL&diff=4912
PERL
2010-09-08T14:29:35Z
<p>129.57.167.4: </p>
<hr />
<div>We are installing PERL modules directly into the PERL system location. Normally PERL modiles will be obtained from [http://www.cpan.org/] in a form of tar files and placed in /usr/local/downloads. For every platform they must be copied to /usr/local/src, untared and installed. Following modules must be installed:<br />
Pezca<br />
Tk<br />
DBI<br />
DBD::mysql<br />
Time<br />
URI<br />
HTML::Tagset<br />
HTML::Parser<br />
IO-Compress-Base<br />
Compress-Raw-Zlib (see note below)<br />
Compress-Raw-Bzip2<br />
IO-Compress-Bzip2<br />
IO-Compress-Zlib<br />
Compress-Zlib<br />
LWP (libwww-perl) (use default options when asked)<br />
XML-Parser<br />
<br />
For installation instructions see README inside those packages. Usually you have to type:<br />
perl Makefile.PL<br />
gmake<br />
gmake test (optionally)<br />
gmake install<br />
<br />
Following was noticed:<br />
<br />
* Pezca_0.3: After running 'perl Makefile.PL' and before 'gmake', modify 'Makefile': after '-Iinclude', add '-Iinclude/os/solaris -Iinclude/O.solaris' (adjust for OS; do not know why it did not happened during 'perl Makefile.PL' ...). Also, module 'osdTime.cc' must be compiled by 'CC', not by 'cc' as it is happening (just go to 'libCom/os/generic', type 'gmake', copy last failed command, replace 'cc' by 'CC' and execute).<br />
<br />
* Pezca_0.4: After running 'perl Makefile.PL.ontop' and before 'gmake', modify 'Makefile': after -I.../include/os add '/solaris' with corresponding flavor. Also, version 0.4 can be (and must be) compiled using our EPICS, not the one which comes with Pezca. To do so EPICS_BASE and EPICS_EXTENSIONS environment variables must be defined, and configuring must be done by command ''perl Makefile.PL.ontop''. On solaris-x86, modify ''platform_ontop.pm'' as following:<br />
# platform dependant sub-directory for include files (on sparc-solaris probably the same using ''solaris-sparc''):<br />
my %os_inc_subdir= ('linux-x86' => 'Linux',<br />
'win32-x86' => 'WIN32',<br />
'hpux-11' => 'hp700',<br />
'solaris' => 'solaris', <----- added<br />
);<br />
# platform dependant sub-directory for libraries in the base:<br />
my %os_b_lib_subdir= ('linux-x86' => 'linux-x86',<br />
'win32-x86' => 'win32-x86',<br />
'hpux-11' => 'hp700',<br />
'solaris' => 'solaris-x86', <----- added<br />
);<br />
# platform dependant sub-directory for libraries in the extensions directory:<br />
my %os_e_lib_subdir= ('linux-x86' => 'linux-x86',<br />
'win32-x86' => 'win32-x86',<br />
'hpux-11' => 'hp700',<br />
'solaris' => 'solaris-x86', <----- added<br />
);<br />
<br />
* DBI:<br />
<br />
clon01:DBI-1.52> perl Makefile.PL<br />
*** Note:<br />
The optional PlRPC-modules (RPC::PlServer etc) are not installed.<br />
If you want to use the DBD::Proxy driver and DBI::ProxyServer<br />
modules, then you'll need to install the RPC::PlServer, RPC::PlClient,<br />
Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you.<br />
You can install them any time after installing the DBI.<br />
You do *not* need these modules for typical DBI usage.<br />
Optional modules are available from any CPAN mirror, in particular<br />
http://search.cpan.org/<br />
http://www.perl.com/CPAN/modules/by-module<br />
http://www.perl.org/CPAN/modules/by-module<br />
ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module<br />
<br />
* Compress-Raw-Zlib:<br />
<br />
sed -e "s|BUILD_ZLIB\s*= True|BUILD_ZLIB = False|" -e "s|INCLUDE\s*= ./zlib-src|INCLUDE = /usr/include|" -e "s|LIB\s*= ./zlib-src|LIB = /usr/lib|" config.in<br />
<br />
Install all modules on one machine of each flavor and execute following for the rest:<br />
<br />
cd /usr/local/src/Pezca-0.4<br />
gmake install<br />
cd /usr/local/src/Tk-804.027/<br />
gmake install<br />
cd /usr/local/src/DBI-1.52/<br />
gmake install<br />
cd /usr/local/src/DBD-mysql-3.0008/<br />
gmake install<br />
cd /usr/local/src/Time-modules-2006.0814/<br />
gmake install<br />
cd /usr/local/src/URI-1.35/<br />
gmake install<br />
cd /usr/local/src/HTML-Tagset-3.10/<br />
gmake install<br />
cd /usr/local/src/HTML-Parser-3.56/<br />
gmake install<br />
cd /usr/local/src/IO-Compress-Base-2.005/<br />
gmake install<br />
cd /usr/local/src/Compress-Raw-Zlib-2.005/<br />
gmake install<br />
cd /usr/local/src/Compress-Raw-Bzip2-2.005/<br />
gmake install<br />
cd /usr/local/src/IO-Compress-Bzip2-2.005/<br />
gmake install<br />
cd /usr/local/src/IO-Compress-Zlib-2.005/<br />
gmake install<br />
cd /usr/local/src/Compress-Zlib-2.005/<br />
gmake install<br />
cd /usr/local/src/libwww-perl-5.808/<br />
gmake install</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=CLAS_Online_Software&diff=4889
CLAS Online Software
2010-07-19T19:32:43Z
<p>129.57.167.4: </p>
<hr />
<div>'''If you want information about DAQ/Online Systems startup and recover procedures, use following link:'''<br />
<br />
[[DAQ/Online Quick Reference]]<br />
<br />
[[DAQ/Online Software Stricture]]<br />
<br />
<br />
== PRIMEX setup ==<br />
<br />
ssh to clon10 as primrun, pass 'primextocome'<br />
primex_start<br />
primex_exit<br />
primex_xterms_this<br />
primex_xterms_kill<br />
<br />
== Wolfram test setup ==<br />
<br />
* NOTE: EXPID must be clasdev, SESSION must be claswolf<br />
<br />
* serial connection to the VME crate: on wolfram: ''cu -l /dev/cua/b''<br />
<br />
* to start DAQ:<br />
ssh vpk@clon00<br />
wolfram_start<br />
<br />
* to start ROOT:<br />
ssh vpk@clon00<br />
cmroot claswolf hist_claswolf<br />
<br />
* to kill DAQ:<br />
wolfram_exit<br />
<br />
<br />
== Expert Information ==<br />
<br />
[[ADC Pedestals Measurement Procedure]]<br />
<br />
[[Alarm Browser]]<br />
<br />
[[Alarm Server]]<br />
<br />
[[Data Monitor]]<br />
<br />
[[dbrouter]]<br />
<br />
[[Epics Server]]<br />
<br />
[[ET System]]<br />
<br />
[[Event Monitor]]<br />
<br />
[[info_server]]<br />
<br />
[[Logbook]]<br />
<br />
[[msql database]]<br />
<br />
[[Process Management]]<br />
<br />
[[Recsis]]<br />
<br />
[[sgutil]]<br />
<br />
[[Stadis]]<br />
<br />
[[CMON]] CLAS Online Data Monitoring<br />
<br />
[[Runlog]] programs</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Info_server&diff=4888
Info server
2010-07-19T19:21:57Z
<p>129.57.167.4: </p>
<hr />
<div>''info_server'' is actually smart sockets router, it collects messages from various sources, form new messages and sends them in according to the configuration file ''$CLON_PARMS/info_server/info_server.cfg''.<br />
<br />
To monitor its activity ''ipc_monitor'' command can be used, for example following command<br />
<br />
java clonjava/ipc_monitor -a clasprod -t info_server -d /info_server/out/epics_server -dump<br />
<br />
will dump all messages sent from ''info_server'' to the ''epics_server''. Use<br />
<br />
java clonjava/ipc_monitor -h<br />
<br />
to get more detailes.<br />
<br />
Another example: command<br />
<br />
java clonjava/ipc_monitor -a clasprod -d /dbrouter -dump<br />
<br />
will dump all messages sent to dbrouter.<br />
<br />
Dump all messages sent by ''run_log_begin'':<br />
<br />
java clonjava/ipc_monitor -a clasprod -s /run_log_begin -dump<br />
<br />
'''NOTE''': if someting is wrong in ''info_server.cfg'' file, for example message description does not match with actual message sent, ''info_server'' will print error message similar to<br />
ProcessInputMsg - expecting an integer but didnt get it (run_log_begin::helclockd)<br />
InfoServer: Message run_log_begin did not match definition.<br />
and message will be NOT accepted. Check to those messages in the log file !!! Only following message shoul be there which is Ok:<br />
ProcessInputMsg - expecting an integer but didnt get it (never_comes::never)<br />
InfoServer: Message did not match definition.<br />
<br />
To see all processes: ''ipc_info -a clasprod''.</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4887
DAQ/Online Quick Reference
2010-07-19T19:18:00Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel'<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar<br />
<br />
* click 'Reset'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)<br />
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4886
DAQ/Online Quick Reference
2010-07-19T19:17:45Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel'<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar;<br />
<br />
* click 'Reset'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)<br />
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4885
DAQ/Online Quick Reference
2010-07-19T19:16:46Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel'<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar; if rebooting any ROC, reboot clastrig2 as well<br />
<br />
* click 'Reset'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)<br />
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4884
DAQ/Online Quick Reference
2010-07-19T19:15:59Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel'<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar<br />
<br />
* click 'Reset'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)<br />
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4883
DAQ/Online Quick Reference
2010-07-19T19:15:08Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel'<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* click 'Reset'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)<br />
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4882
DAQ/Online Quick Reference
2010-07-19T19:04:55Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)<br />
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4881
DAQ/Online Quick Reference
2010-07-19T18:54:39Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
'''Useful commands'''<br />
<br />
r_ts_prescale - reads prescales from TS2<br />
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4880
DAQ/Online Quick Reference
2010-07-19T18:48:00Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type following command:<br />
<br />
daq_start this<br />
<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
<br />
daq_exit<br />
<br />
To start/stop serial connections to ROCs and PMCs, type following on clon00:<br />
<br />
roc_xterm_this<br />
pmc_xterms_this<br />
<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4879
DAQ/Online Quick Reference
2010-07-19T18:45:29Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
'''Logbook and/or run database does not work'''<br />
<br />
Contact Marty Wise.<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4878
DAQ/Online Quick Reference
2010-07-19T18:44:05Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmartSockets<br />
Mysql<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4877
DAQ/Online Quick Reference
2010-07-19T18:43:36Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''Process management'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Services on clondb1'''<br />
<br />
Clondb1 is running several important servers which suppose to be started automatically on reboot:<br />
SmatrSockets<br />
Mysql<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4876
DAQ/Online Quick Reference
2010-07-19T18:40:17Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4875
DAQ/Online Quick Reference
2010-07-19T18:36:56Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4874
DAQ/Online Quick Reference
2010-07-19T18:35:17Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569, 9-011-7-499-189-0189<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4873
DAQ/Online Quick Reference
2010-07-19T18:34:10Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
Stadis to be started on clon05:0.1.<br />
<br />
DCHV to be started on clon06:0.1.<br />
<br />
Diman to be started on clon03:0.0.<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569, 9-011-7-499-189-0189<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4872
DAQ/Online Quick Reference
2010-07-19T18:31:49Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clonmon2 - online recsis, ced)<br />
<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clonmon2)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569, 9-011-7-499-189-0189<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4871
DAQ/Online Quick Reference
2010-07-19T18:28:30Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon0_start (starts everything on clonmon0 - fast online reconstruction;<br />
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod'<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clon04 - recsis, ced)<br />
they will start everything on clon00, clonmon1 and clon04 respectively.<br />
<br />
To kill online monitoring processes use following commands:<br />
mon0_exit (kills everything on clonmon0)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clon04)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569, 9-011-7-499-189-0189<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=DAQ/Online_Quick_Reference&diff=4870
DAQ/Online Quick Reference
2010-07-19T18:24:56Z
<p>129.57.167.4: </p>
<hr />
<div>'''DAQ Recovery Procedure'''<br />
<br />
In most cases DAQ will crash because of ROC problem. Do following:<br />
<br />
* click 'End Run' or 'Abort', whatever button is active<br />
<br />
* click 'Cancel' and then 'Reset'<br />
<br />
* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')<br />
<br />
* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'<br />
<br />
* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'<br />
<br />
* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage<br />
<br />
* start new run from 'Download'<br />
<br />
If it does not help, page DAQ pager.<br />
<br />
<br />
'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''<br />
<br />
'''DAQ procedures'''<br />
<br />
First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.<br />
<br />
To start all DAQ processes goto clon10 as ''clasrun'' and type 3 following commands on 3 different workspaces:<br />
daq_start this<br />
roc_xterm_this<br />
pmc_xterms_this<br />
to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.<br />
<br />
To kill all DAQ processes goto clon10 as ''clasrun'' and type<br />
daq_exit<br />
roc_xterms_kill<br />
pmc_xterms_kill<br />
<br />
<br />
'''Online procedures'''<br />
<br />
To start online monitoring processes use following commands:<br />
mon_start (starts everything on clon00 - root)<br />
mon1_start (starts everything on clonmon1 - event_monitor)<br />
mon2_start (starts everything on clon04 - recsis, ced)<br />
they will start everything on clon00, clonmon1 and clon04 respectively.<br />
<br />
To kill online monitoring processes use following commands:<br />
mon_exit (kills everything on clon00)<br />
mon1_exit (kills everything on clonmon1)<br />
mon2_exit (kills everything on clon04)<br />
<br />
Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type<br />
monb<br />
<br />
<br />
'''Background information'''<br />
<br />
* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:<br />
<br />
login as 'clascron' to the clon10<br />
goto /home/clascron/backup<br />
type 'crontab -l' to see what is running<br />
type 'crontab -r' to erase all existing cron jobs<br />
type 'crontab clon10'<br />
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively<br />
<br />
'''<br />
<br />
Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !<br />
<br />
* Process management: SmartSockets-based process management procedures must be started as ''clasrun'' using following command:<br />
control_process_manager start clasprod clasprod<br />
control_procmgr_manager start clasprod clasprod<br />
control_ipc_process_manager start clasprod clasprod<br />
and killed by following:<br />
control_process_manager stop clasprod clasprod<br />
control_procmgr_manager stop clasprod clasprod<br />
control_ipc_process_manager stop clasprod clasprod<br />
<br />
<br />
'''Sergey Boyarinov''' (from JLAB local phone):<br />
Moscow cell phone: 9-011-7-916-729-8203<br />
Moscow home phones: 9-011-7-495-392-3569, 9-011-7-499-189-0189<br />
SKYPE: boiarino</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=CLAS_Online_Software&diff=4869
CLAS Online Software
2010-07-19T18:14:49Z
<p>129.57.167.4: </p>
<hr />
<div>'''If you want information about DAQ/Online Systems startup and recover procedures, use following link:'''<br />
<br />
[[DAQ/Online Quick Reference]]<br />
<br />
[[DAQ/Online Software Stricture]]<br />
<br />
<br />
== Wolfram test setup ==<br />
<br />
* NOTE: EXPID must be clasdev, SESSION must be claswolf<br />
<br />
* serial connection to the VME crate: on wolfram: ''cu -l /dev/cua/b''<br />
<br />
* to start DAQ:<br />
ssh vpk@clon00<br />
wolfram_start<br />
<br />
* to start ROOT:<br />
ssh vpk@clon00<br />
cmroot claswolf hist_claswolf<br />
<br />
* to kill DAQ:<br />
wolfram_exit<br />
<br />
<br />
== Expert Information ==<br />
<br />
[[ADC Pedestals Measurement Procedure]]<br />
<br />
[[Alarm Browser]]<br />
<br />
[[Alarm Server]]<br />
<br />
[[Data Monitor]]<br />
<br />
[[dbrouter]]<br />
<br />
[[Epics Server]]<br />
<br />
[[ET System]]<br />
<br />
[[Event Monitor]]<br />
<br />
[[info_server]]<br />
<br />
[[Logbook]]<br />
<br />
[[msql database]]<br />
<br />
[[Process Management]]<br />
<br />
[[Recsis]]<br />
<br />
[[sgutil]]<br />
<br />
[[Stadis]]<br />
<br />
[[CMON]] CLAS Online Data Monitoring<br />
<br />
[[Runlog]] programs</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Linux_Installation_Procedure&diff=4860
Linux Installation Procedure
2010-07-09T19:22:25Z
<p>129.57.167.4: /* RHEL5 */</p>
<hr />
<div>== RHEL5 ==<br />
<br />
Installed using CD provided by Computer Center. Requires MAC address to be registered with Computer Center (email to helpdesk@jlab.org) because it is network-based installation.<br />
<br />
Dialog:<br />
<br />
* boot: <enter><br />
<br />
* ...[D]esktop: Server (it is ignored, will do Desktop anyway ...)<br />
<br />
* root password: enter root password<br />
<br />
* system user: <enter><br />
<br />
* manually partition disk: 100MB /boot, 2048 etc MB swap, the rest /.<br />
<br />
'''CFENGINE''': there are procesess restoring some system config files (in particular ''/etc/ssh/sshd_config'') in according to the Computer Center - maintained templete. To shut them down comment out ''cfengine''-related stuff in root cronjobs and do ''/etc/init.d/cfengine stop''. Ask CC (Sherman White, Paul Letta) to fix templete.<br />
<br />
Run ''/usr/bin/system-config-authentication'' and set NIS servers list as ''clon00,clon10,nis1''.<br />
<br />
Add following in the end of ''/etc/passwd'':<br />
clasrun:x:2508:9998:Online DAQ:/home/clasrun:/bin/tcsh<br />
clascron:x:6005:146:CLAS Cron Jobs:/home/clascron:/bin/tcsh<br />
nagios:x:6246:9997:Nagios:/home/nagios:/bin/tcsh<br />
+@clon_cluster::::::<br />
+@ccc::::::<br />
<br />
Add following in the end of ''/etc/group'':<br />
perl::9999:freyberg<br />
onliners::9998:heyes,hujl,rwm,timmer,wolin,taylorw,cuevas,abbottd,abulafia,anciant,anghi,aswin,auger,avakian,banta,barrow,battagli,bjlee,bogdan,brooksw,bruhwel,boiarino,buckle,burchesk,burin,bzh,cetina,chen,cole,cords,corvi,costy,davidl,costy,davidl,dcarter,dcurry,dennisl,deppman,dhlee,doughty,drago,dytman,dugger,dugger,elton,fedro,ficenec,fklein,france,freyberg,gholden,gilfoyle,golovach,gurjyan,haddock,heddle,jenkins,jhardie,jmorris,kellyk,klusman,kmkim,kossov,lcsmith,longhi,manak,marchand,marki,mastersj,mjohnson,mosbackr,mueller,muguira,murphyl,necaise,odonnell,opper,parkkj,pasyuk,phkim,piot,riccardi,richardt,ripani,rossi,saw,sergpozd,serov,sholt,staylor,taiuti,tang,tedeschi,vineyard,vlassov,vvsap,weygand,wooyoung,burchesk,clasrun,giovanet,clasboot,clascoda,clasroot,clasmail,clascron,trigger,epics,southern,carstens,ccutter,danagu,heyes,huertas,jacobsg,kliv,lachniet,mestayer,primex,nerses<br />
clon::383:clascoda,clascron,clasmail<br />
nagios::9997:nagios<br />
<br />
Create ''/etc/shadow'' files with entries for userd clasrun, epics and nagios (copy from another machine).<br />
<br />
Install updates: normally all updates should be installed by now, it can be done manually by commands<br />
yum list updates<br />
yum update<br />
<br />
Configure [[SSH]].<br />
<br />
Install missing packages:<br />
yum install openmotif-devel<br />
yum install xinetd<br />
<br />
Proceed to the [[Linux Customization on CLON Cluster]].<br />
<br />
'''PROBLEMS''':<br />
<br />
1. /usr/lib/libg2c.so link is missing<br />
<br />
FIXED ??? 2. /usr/lib/libXm.so link is missing<br />
<br />
3. Created /etc/shadow file for clasrun, epics and nagios<br />
<br />
== RHEL4 ==<br />
<br />
Get RHEL4 CDs from Paul Letta.<br />
<br />
Connect terminal, keyboard and mouse.<br />
<br />
Boot from CD1. Follow instructions. When asked, choose following options:<br />
<br />
* Choose 'Manually partition with Disk Druid', set /boot to 100MB, then / to 40GB, then swap to 8192MB or at least double memory size. The rest of disk can be set as /space.<br />
<br />
* Network: choose manually, type clonpc2.jlab.org; click 'EDIT', deactivate 'Configure using DHCP',<br />
type IP address and netmask (255.255.255.0), click Ok.<br />
Type Gateway (for example 129.57.167.99) and 3 DNS servers (129.57.167.5,129.57.167.14,129.57.32.100).<br />
<br />
* Choose 'No firewall'<br />
<br />
* Disable 'SELinux'<br />
<br />
* Choose 'Customize software packages to be installed', click 'Everything'.<br />
<br />
You will be asked to install CD2, CD3 and CD4, then CD1 again.<br />
Then you will be asked to click 'Reboot'. First remove CD, then click 'Reboot'.<br />
You will be asked few more questions. When asked to activate service, choose<br />
'Tell me why I need to register ..' and click 'Next'. Choose 'I can not complete registration ..'<br />
and click 'Next'. You will see 'System user' screen, click 'Use network login..'.<br />
Choose 'Enable NIS support' and click 'Configure NIS'. Type NIS Domain: 'CCCHP' and NIS Server: clon00.jlab.org.<br />
Finish installation process.<br />
<br />
* fix files ''/etc/passwd'', ''/etc/shadow'' and ''/etc/group'' adding clasrun etc (see examples on another machines).<br />
<br />
'''NOTE''': if machine has multiple network ports, Linux may not assign them properly to specified IP names and addresses. If problem occures, open network GUI and delete all existing records. After that, click ''New'' and pick hadrware port you want to configure from the list of available ports (tricky part is to know which RJ-45 connector corresponds to which port ..). Configure and activate it, make sure it works. Repeat for other ports. Results will be stored in directory ''/etc/sysconfig/network-scripts'', file names will be ''ifcfg-eth0'', ''ifcfg-eth1'' etc. If modifying those files manually, restart network by ''/etc/init.d/network restart''. Finally, set routing, for example for ''clonmon'' command ''route -n'' shows following:<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
129.57.160.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1<br />
129.57.68.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3<br />
129.57.69.0 0.0.0.0 255.255.255.0 U 0 0 0 dev5937<br />
129.57.64.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0<br />
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth3<br />
0.0.0.0 129.57.68.100 0.0.0.0 UG 0 0 0 eth3<br />
Run following commands:<br />
/sbin/route add -net 129.57.160.0 netmask 255.255.255.0 gw 129.57.160.5 dev eth1<br />
/sbin/route add -net 129.57.68.0 netmask 255.255.255.0 gw 129.57.68.26 dev eth3<br />
/sbin/route add -net 129.57.69.0 netmask 255.255.255.0 gw 129.57.69.206 dev dev5937<br />
/sbin/route add -net 129.57.64.0 netmask 255.255.255.0 gw 129.57.64.57 dev eth0<br />
Now ''route -n'' shows following:<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
129.57.160.0 129.57.160.5 255.255.255.0 UG 0 0 0 eth1<br />
129.57.160.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1<br />
129.57.68.0 129.57.68.26 255.255.255.0 UG 0 0 0 eth3<br />
129.57.68.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3<br />
129.57.69.0 129.57.69.206 255.255.255.0 UG 0 0 0 dev5937<br />
129.57.69.0 0.0.0.0 255.255.255.0 U 0 0 0 dev5937<br />
129.57.64.0 129.57.64.57 255.255.255.0 UG 0 0 0 eth0<br />
129.57.64.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0<br />
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth3<br />
0.0.0.0 129.57.68.100 0.0.0.0 UG 0 0 0 eth3<br />
In that example automount will do configured using ''clonmon-daq1'' interface, because default ''clonmon'' interface is on external subnet. To make that routing table permanent (retore it after reboot) I tried to add four files with following contents to the ''/etc/sysconfig/network-scripts'' directory:<br />
route-eth1:<br />
GATEWAY0=129.57.160.5<br />
NETMASK0=255.255.255.0<br />
ADDRESS0=129.57.160.0<br />
route-eth3:<br />
GATEWAY0=129.57.68.26<br />
NETMASK0=255.255.255.0<br />
ADDRESS0=129.57.68.0<br />
route-dev5937:<br />
GATEWAY0=129.57.69.206<br />
NETMASK0=255.255.255.0<br />
ADDRESS0=129.57.69.0<br />
route-eth0:<br />
GATEWAY0=129.57.64.57<br />
NETMASK0=255.255.255.0<br />
ADDRESS0=129.57.64.0<br />
It did not worked. Then I added four ''/sbin/route add ..'' lines to ''/etc/rc.local'' script, and it worked.<br />
<br />
From now on you can use remote 'ssh' connection.<br />
<br />
* To configure your system to use the patch server, issue the following commands as root:<br />
# mount jlabsite:/site /mnt<br />
# /mnt/CC/linux/satellite-connect --patchall<br />
The script will prompt you for an activation key, but if you don't know what this is, just hit ENTER and it will use a sensible default. Depending on how many patches need to be applied to your system to bring it up to the current level, this script may take a very long time to run. When it finishes, you should reboot your machine to make the new patches take effect.<br />
<br />
To install updates later run ''up2date -uf'' as ''root''.<br />
<br />
Configure [[SSH]].<br />
<br />
* Proceed with [[Linux Customization on CLON Cluster]] if applicable.<br />
<br />
To boot Linux in single user mode (no graphics), add ''single'' to the end of boot line during boot process.<br />
<br />
<br />
<br />
== '''background info''' ==<br />
<br />
'''Add startup script'''<br />
<br />
Create script in ''/etc/init.d/'' directory (use existing scripts as example). Must have a line like<br />
# chkconfig: - 91 09<br />
which specifies start and stop order. Run commands<br />
chkconfig --add <scriptname><br />
chkconfig --level 3 <scriptname> on<br />
for every level you want that script to be executed (usualy 3 and/or 4 and/or 5). Check if it os done using command<br />
chkconfig --list | grep <scriptname><br />
<br />
<br />
'''xxx'''<br />
<br />
You can choose to download the RHEL operating system directly from the Red Hat Web site and not receive a media kit from HP containing the software. This scenario occurs when the media option AJR is deselected from the order. (The media kit option is always selected by default. )<br />
<br />
It is necessary to register at the Red Hat Web site prior to download. Registration requires the activation code that each customer receives from HP when ordering either RHEL.<br />
<br />
You can download RHEL for each available platform as four CD ISO images or one DVD ISO image.<br />
<br />
Use the following procedure to download RHEL and create the installation media:<br />
<br />
1.<br />
<br />
Register and log in at the following Red Hat Web site:<br />
<br />
www.redhat.com/rhn<br />
2.<br />
<br />
Click on channels and then select Red Hat Enterprise Linux ES (v.4 for 64–bit Intel Itanium).<br />
3.<br />
<br />
Select Downloads and then download the ISO images of install, compatibility and source disks<br />
<br />
Files with ia64 in the name are for the Itanium platform.<br />
4.<br />
<br />
Create the CDs or a DVD from the ISO images.<br />
<br />
<br />
IMPORTANT: Create the CDs or DVD using the contents of the ISO images. Do not create the CDs or DVD by burning the .iso files themselves to CD or DVD. For example, if you are using K3b to burn a CD, click Tools>CD>Burn CD Image, select the .iso image, then click Burn.<br />
<br />
'''Bootup scripts''' for example for level 5 execution (multiuser with X):<br />
<br />
cp <script-file> /etc/init.d/<br />
chmod +x /etc/init.d/<script-file><br />
ln -s /etc/init.d/<scriptfile> /etc/rc.d/rc5.d/S50<scriptfile><br />
ln -s /etc/init.d/<scriptfile> /etc/rc.d/rc5.d/K50<scriptfile><br />
<br />
<br />
'''Changing hostname/IP address'''<br />
<br />
Changing hostname on RHEL<br />
1. Change the HOSTNAME line in /etc/sysconfig/network<br />
2. Change the hostname (FQDN and alias) in /etc/hosts<br />
3. Run '/bin/hostname new_hostname' for the hostname change to take effect immediately.<br />
4. Run '/sbin/service syslog restart' for syslog to log using the new hostname.<br />
A reboot is not required to change the system hostname.<br />
<br />
IP address: You have to change the IP address entry in<br />
/etc/sysconfig/network-scripts/ifcfg-eth0.</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Clontrig0&diff=4859
Clontrig0
2010-07-09T18:56:10Z
<p>129.57.167.4: </p>
<hr />
<div>'''Clontrig0'''<br />
<br />
IP: 129.57.167.7<br />
<br />
MAC: 84:2B:2B:01:6A:93<br />
<br />
Taq: F425336</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=CLON_Index&diff=4858
CLON Index
2010-07-09T18:54:54Z
<p>129.57.167.4: </p>
<hr />
<div>[[Apache]]<br />
<br />
[[automounts]]<br />
<br />
[[Bertan MWPC]] High Voltage Power Supplies<br />
<br />
[[Bootp]]<br />
<br />
[[Cacti]]<br />
<br />
[[CAEN A464]] caenet board (installed in sy527 mainframes)<br />
<br />
[[CAEN SY1527 Mainframe]]<br />
<br />
[[CAEN SY527 Mainframe]] and modules<br />
<br />
[[CAEN V1190]] TDC board (128 ch 100 ps)<br />
<br />
[[CAEN V1290]] TDC board (32 ch 25 ps)<br />
<br />
[[CAEN V1495]] General Purpose VME Board<br />
<br />
[[CAEN V288]]<br />
<br />
[[CAEN V560]] scaler board<br />
<br />
[[CAEN V775]] TDC Board<br />
<br />
[[CAEN V895]] 16 ch. Leading Edge Discriminator with individual thresholds<br />
<br />
[[Capfast]]<br />
<br />
[[Cisco Switches]]<br />
<br />
[[Clasweb]]<br />
<br />
[[Clasxt]]<br />
<br />
[[Clon00]]<br />
<br />
[[Clon01]]<br />
<br />
[[Clon02]]<br />
<br />
[[Clon03]]<br />
<br />
[[Clon04]]<br />
<br />
[[Clon05]]<br />
<br />
[[Clon06]]<br />
<br />
[[Clon10]]<br />
<br />
[[Clon20]]<br />
<br />
[[Clondaq1]]<br />
<br />
[[Clondaq2]]<br />
<br />
[[Clondb1]]<br />
<br />
[[Clondb2]]<br />
<br />
[[Clonfs]] Main RAID System<br />
<br />
[[Clonfs1]]<br />
<br />
[[Clonfs2]]<br />
<br />
[[Clonmon]]<br />
<br />
[[Clonmon0]]<br />
<br />
[[Clonmon1]]<br />
<br />
[[Clonmon2]]<br />
<br />
[[Clonpc1]]<br />
<br />
[[Clonpc2]]<br />
<br />
[[Clonpc3]]<br />
<br />
[[Clonpc5]]<br />
<br />
[[Clonpc6]]<br />
<br />
[[Clonpc8]]<br />
<br />
[[Clontrig0]]<br />
<br />
[[Clonweb]]<br />
<br />
[[Clonwiki]]<br />
<br />
[[clonhp]]<br />
<br />
[[clonhp2]]<br />
<br />
[[Coda]]<br />
<br />
[[Compcontrol CC121]]<br />
<br />
[[CycladesTS1000]]<br />
<br />
[[DATEL DVME-628]]<br />
<br />
[[DNS server]]<br />
<br />
[[Electronics Documentation]]<br />
<br />
[[EtherLite32]]<br />
<br />
[[EPICS Base Installation]]<br />
<br />
[[EPICS: Generation of IOC with different options and customization for HallB]]<br />
<br />
[[EPICS: Software IOC]]<br />
<br />
[[EPICS: vxWorks IOC]]<br />
<br />
[[EPICS: Software IOC for CAEN HV]]<br />
<br />
[[EPICS: generation of DBs and GUIs for HV]]<br />
<br />
[[EPICS: CVS usage examples]]<br />
<br />
[[EPICS: Timeline Histories]]<br />
<br />
[[EPICS: Gamma Profiler]]<br />
<br />
[[Event Monitor]]<br />
<br />
[[Fiber Channel Switches]]<br />
<br />
[[FieldPoint]]<br />
<br />
[[Flash ADC Studies]]<br />
<br />
[[Foundry BigIron 15000 Switch]]<br />
<br />
[[Foundry Switches]]<br />
<br />
[[Helicity Signals from MCC]]<br />
<br />
[[HYTEC VSD2992]]<br />
<br />
[[JAVA installation]]<br />
<br />
[[JLAB Discriminators]]<br />
<br />
[[JLAB FLEXIO]]<br />
<br />
[[JLAB LF]] VME Latch FIFO Board<br />
<br />
[[JLAB LD]] VME Latch Driver Board<br />
<br />
[[JLAB Level2 Majority]] Logic Unit<br />
<br />
[[JLAB MLU]] VME Memory Lookup Unit (MLU)<br />
<br />
[[JLAB OR]] VME Programmable “OR” Module<br />
<br />
[[JLAB TS2]] VXI D-size Trigger Supervisor Module<br />
<br />
[[JLAB TI]] CODA Trigger Interface Cards<br />
<br />
[[JLAB PCI TI]] PCI Trigger Interface Card<br />
<br />
[[JLAB VME TI]] VME Trigger Interface Module<br />
<br />
[[Joerger VSC16]]<br />
<br />
[[KineticSystems 2917]]<br />
<br />
[[Labeling Machines]]<br />
<br />
[[Linux Installation Procedure]]<br />
<br />
[[Linux Customization on CLON Cluster]]<br />
<br />
[[Lnf cluster]]<br />
<br />
[[Mac OS X Installation Procedure]]<br />
<br />
[[Motorola MVME5500]]<br />
<br />
[[MRTG]]<br />
<br />
[[mSQL]]<br />
<br />
[[MySQL Installation]]<br />
<br />
[[MySQL Usage]]<br />
<br />
[[Nagios]]<br />
<br />
[[Nagios Plugins]]<br />
<br />
[[Net-SNMP]]<br />
<br />
[[NFS]]<br />
<br />
[[NIS server]]<br />
<br />
[[Nrpe]]<br />
<br />
[[Ntp]]<br />
<br />
[[OMS VME44]] by Oregon Micro Systems<br />
<br />
[[OMS VS4]] by Oregon Micro Systems<br />
<br />
[[OpenSSL]]<br />
<br />
[[PERL]]<br />
<br />
[[Procmail]]<br />
<br />
[[PrPMC]]<br />
<br />
[[raidold]]<br />
<br />
[[Routing]]<br />
<br />
[[Rrdtool]]<br />
<br />
[[SBS VIPC616]]<br />
<br />
[[Serial Connections from Computer Ports]]<br />
<br />
[[Sfs61]]<br />
<br />
[[Slow Controls Time History]]<br />
<br />
[[SmartSockets]]<br />
<br />
[[SNMP]]<br />
<br />
[[Software Documentation]]<br />
<br />
[[Solaris Installation Procedure]]<br />
<br />
[[Solaris Customization on CLON Cluster]]<br />
<br />
[[Solaris Patches]]<br />
<br />
[[SSH]]<br />
<br />
[[Struck SIS3801]]<br />
<br />
[[Sudo]]<br />
<br />
[[SUN Microsystems]]<br />
<br />
[[SYSTRAN VMESC5]]<br />
<br />
[[Tcl/Tk]]<br />
<br />
[[Tftp]]<br />
<br />
[[Troubleshooting printing]]<br />
<br />
[[UNIX Automount on CLON Cluster]]<br />
<br />
[[UPS (Uninterruptible Power Supplies)]]<br />
<br />
[[UPS CHB1]]<br />
<br />
[[UPS CHB2]]<br />
<br />
[[UPS Troubleshooting]]<br />
<br />
[[VI editor]]<br />
<br />
[[VMIC VMIVME-3122]]<br />
<br />
[[VMIC VMIVME-2120]]<br />
<br />
[[VMIC VMIVME-1129]]<br />
<br />
[[VMIC VMIVME-2232]]<br />
<br />
[[VMIC VMIVME-6015]]<br />
<br />
[[VMIC VMIVME-6016]]<br />
<br />
[[VNC]]<br />
<br />
[[WIENER]]<br />
<br />
[[Wiki]]<br />
<br />
[[Wiring]]<br />
<br />
[[X server]]<br />
<br />
[[Xycom DIO XVME-240]]<br />
<br />
[[Xycom XVME-564]]<br />
<br />
[[Xycom XVME-560]]<br />
<br />
[[ZFS]]</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4743
ERROR Book
2010-04-16T18:40:53Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!???:<br />
clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod<br />
-m eb -v<br />
et_nodename >eb<<br />
et_start: asking for 200000 byte events.<br />
et_start: asking for 4000 events.<br />
removing file >/tmp/et_sys_clasprod<<br />
file >/tmp/et_sys_clasprod< removed<br />
et_start: starting ET system /tmp/et_sys_clasprod<br />
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/<br />
libet_user.so< will be used<br />
et INFO: et_add_stations, nstations = 2, stats_max = 10<br />
et INFO: et_conductor 1, found station cec1d440<br />
et INFO: et_station_create_at, created station LEVEL3<br />
et INFO: et_add_stations, nstations = 3, stats_max = 10<br />
et INFO: et_conductor 2, found station cec1d860<br />
et INFO: et_station_create_at, created station ET2ER<br />
unique name >et_start_eb_clasprod<<br />
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R<br />
Tserver<br />
11:45:20: TAL-SS-00089-I Using local protocol<br />
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver<br />
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve<br />
r<br />
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1><br />
RTserver<br />
11:45:20: TAL-SS-00089-I Using local protocol<br />
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1><br />
RTserver<br />
11:45:20: TAL-SS-00089-I Using tcp protocol<br />
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established<br />
.<br />
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_cl<br />
asprod><br />
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab<br />
.org><br />
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all><br />
<br />
ipc_init called at Fri Apr 16 11:45:20 2010<br />
<br />
et_start: pthread_create(0x0000000e,...) done<br />
et ERROR: et_netserver: ET server being probed by non-ET client or rea<br />
d failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or rea<br />
d failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,22078)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments<br />
<br />
<br />
<br />
<br />
clondaq2:clasrun> cd et2_start<br />
clondaq2:et2_start> <br />
clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod<br />
-m er -v<br />
et_nodename >er<<br />
et_start: asking for 200000 byte events.<br />
et_start: asking for 4000 events.<br />
removing file >/tmp/et_sys_clasprod<<br />
file >/tmp/et_sys_clasprod< removed<br />
et_start: starting ET system /tmp/et_sys_clasprod<br />
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/<br />
libet_user.so< will be used<br />
et INFO: et_add_stations, nstations = 2, stats_max = 10<br />
et INFO: et_conductor 1, found station cec1d440<br />
et INFO: et_station_create_at, created station TAPE<br />
et INFO: et_add_stations, nstations = 3, stats_max = 10<br />
et INFO: et_conductor 2, found station cec1d860<br />
et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur<br />
rent/coda/SunOS_i86pc/lib/libet_user.so<br />
et INFO: et_conductor 2, loaded function et2et_function<br />
et INFO: et_station_create_at, created station ET2ET10<br />
unique name >et_start_er_clasprod<<br />
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R<br />
Tserver<br />
11:45:20: TAL-SS-00089-I Using local protocol<br />
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver<br />
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve<br />
r<br />
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1><br />
RTserver<br />
11:45:20: TAL-SS-00089-I Using local protocol<br />
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1><br />
RTserver<br />
11:45:20: TAL-SS-00089-I Using tcp protocol<br />
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established<br />
.<br />
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl<br />
asprod><br />
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab<br />
.org><br />
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all><br />
<br />
ipc_init called at Fri Apr 16 11:45:20 2010<br />
<br />
et_start: pthread_create(0x0000000e,...) done<br />
et INFO: et_netserver: magic numbers do match, accept ET client<br />
et INFO: et_station_attach, done<br />
et INFO: et_netserver: magic numbers do match, accept ET client<br />
et INFO: et_station_attach, done<br />
et INFO: et_netserver: magic numbers do match, accept ET client<br />
et INFO: et_station_attach, done<br />
et INFO: et_restore_events, no events found belonging to attachment 4 <br />
et INFO: et_station_detach, done<br />
et WARN: et_command_loop: remote client connection broken<br />
et ERROR: et_netserver: ET server being probed by non-ET client or rea<br />
d failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or rea<br />
d failure<br />
et INFO: et_sys_heartmonitor, kill bad process (1,8898)<br />
et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1<br />
et INFO: et_flush_events, read 4000 events<br />
et INFO: et_flush_events, wrote 4000 events<br />
et INFO: et_station_detach, make station "TAPE" (cec1d440) idle<br />
et INFO: et_restore_events, no events found belonging to attachment 3 <br />
et INFO: et_station_detach, done<br />
et INFO: et_sys_heartmonitor, cleanup process 1<br />
et INFO: set_fix_nprocs, change # of ET processes from 1 to 1<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments<br />
et INFO: set_fix_natts, station ET2ET10 has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 3 -> 3<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
<br />
<br />
<br />
<br />
clondaq1:dac> <br />
clondaq1:dac> <br />
clondaq1:dac> <br />
clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod<br />
LOCALITY = 1<br />
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)<br />
(tcp port 0) (udp port 11111) (multicast port 11111)<br />
(pid 22276) (lang C) (local) (period = 5 sec)<br />
<br />
STATIC INFO - maximum of:<br />
events(4000), event size(200000), temps(4000)<br />
stations(10), attaches(20), procs(10), concurrency added(16)<br />
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,<br />
<br />
DYNAMIC INFO - currently there are:<br />
processes(4), attachments(4), temps(0)<br />
stations(3), hearbeat(1)<br />
<br />
STATIONS:<br />
"GRAND_CENTRAL" (id = 0)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)<br />
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(2), ids(2,3,)<br />
input list: cnt = 4000, events in = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"LEVEL3" (id = 1)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)<br />
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(1), ids(1,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"ET2ER" (id = 2)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)<br />
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(1), ids(0,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
LOCAL USERS:<br />
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)<br />
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)<br />
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)<br />
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)<br />
<br />
ATTACHMENTS:<br />
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)<br />
proc(0), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)<br />
proc(1), blocked(YES)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)<br />
proc(2), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)<br />
proc(3), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
<br />
EVENTS OWNED BY:<br />
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),<br />
<br />
EVENT RATE of GC = 0 events/sec<br />
<br />
CREATING STATIONS: <br />
IDLE STATIONS: <br />
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER, <br />
LOCKED MUTEXES: <br />
<br />
*****************************************<br />
<br />
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)<br />
(tcp port 0) (udp port 11111) (multicast port 11111)<br />
(pid 22276) (lang C) (local) (period = 5 sec)<br />
<br />
STATIC INFO - maximum of:<br />
events(4000), event size(200000), temps(4000)<br />
stations(10), attaches(20), procs(10), concurrency added(16)<br />
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,<br />
<br />
DYNAMIC INFO - currently there are:<br />
processes(4), attachments(4), temps(0)<br />
stations(3), hearbeat(1)<br />
<br />
STATIONS:<br />
"GRAND_CENTRAL" (id = 0)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)<br />
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(2), ids(2,3,)<br />
input list: cnt = 4000, events in = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"LEVEL3" (id = 1)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)<br />
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(1), ids(1,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"ET2ER" (id = 2)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)<br />
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(1), ids(0,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
LOCAL USERS:<br />
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)<br />
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)<br />
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)<br />
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)<br />
<br />
ATTACHMENTS:<br />
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)<br />
proc(0), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)<br />
proc(1), blocked(YES)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)<br />
proc(2), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)<br />
proc(3), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
<br />
EVENTS OWNED BY:<br />
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),<br />
<br />
EVENT RATE of GC = 0 events/sec<br />
<br />
CREATING STATIONS: <br />
IDLE STATIONS: <br />
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER, <br />
LOCKED MUTEXES: <br />
<br />
*****************************************<br />
<br />
ET SYSTEM is DEAD!<br />
*****************************************<br />
<br />
^C<br />
clondaq1:dac> <br />
<br />
<br />
<br />
<br />
<br />
<br />
clondaq2:stage_in> <br />
clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod<br />
LOCALITY = 1<br />
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)<br />
(tcp port 0) (udp port 11111) (multicast port 11111)<br />
(pid 16127) (lang C) (local) (period = 5 sec)<br />
<br />
STATIC INFO - maximum of:<br />
events(4000), event size(200000), temps(4000)<br />
stations(10), attaches(20), procs(10), concurrency added(16)<br />
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,<br />
<br />
DYNAMIC INFO - currently there are:<br />
processes(3), attachments(3), temps(0)<br />
stations(3), hearbeat(1)<br />
<br />
STATIONS:<br />
"GRAND_CENTRAL" (id = 0)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)<br />
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(0), ids()<br />
input list: cnt = 4000, events in = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"TAPE" (id = 1)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)<br />
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(2), ids(1,2,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"ET2ET10" (id = 2)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)<br />
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)<br />
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class = <br />
dynamic info<br />
attachments: total#(1), ids(0,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
LOCAL USERS:<br />
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)<br />
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)<br />
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)<br />
<br />
ATTACHMENTS:<br />
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)<br />
proc(0), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)<br />
proc(1), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)<br />
proc(2), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
<br />
EVENTS OWNED BY:<br />
system (4000), att0 (0), att1 (0), att2 (0),<br />
<br />
EVENT RATE of GC = 0 events/sec<br />
<br />
CREATING STATIONS: <br />
IDLE STATIONS: <br />
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10, <br />
LOCKED MUTEXES: <br />
<br />
*****************************************<br />
<br />
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)<br />
(tcp port 0) (udp port 11111) (multicast port 11111)<br />
(pid 16127) (lang C) (local) (period = 5 sec)<br />
<br />
STATIC INFO - maximum of:<br />
events(4000), event size(200000), temps(4000)<br />
stations(10), attaches(20), procs(10), concurrency added(16)<br />
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,<br />
<br />
DYNAMIC INFO - currently there are:<br />
processes(3), attachments(3), temps(0)<br />
stations(3), hearbeat(1)<br />
<br />
STATIONS:<br />
"GRAND_CENTRAL" (id = 0)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)<br />
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(0), ids()<br />
input list: cnt = 4000, events in = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"TAPE" (id = 1)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)<br />
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)<br />
dynamic info<br />
attachments: total#(2), ids(1,2,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
"ET2ET10" (id = 2)<br />
static info<br />
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)<br />
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)<br />
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class = <br />
dynamic info<br />
attachments: total#(1), ids(0,)<br />
input list: cnt = 0, events in = 0, events try = 0<br />
output list: cnt = 0, events out = 0<br />
<br />
LOCAL USERS:<br />
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)<br />
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)<br />
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)<br />
<br />
ATTACHMENTS:<br />
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)<br />
proc(0), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)<br />
proc(1), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)<br />
proc(2), blocked(NO)<br />
events: make(0), get(0), put(0), dump(0)<br />
<br />
EVENTS OWNED BY:<br />
system (4000), att0 (0), att1 (0), att2 (0),<br />
<br />
EVENT RATE of GC = 0 events/sec<br />
<br />
CREATING STATIONS: <br />
IDLE STATIONS: <br />
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10, <br />
LOCKED MUTEXES: <br />
<br />
*****************************************<br />
<br />
ET SYSTEM is DEAD!<br />
*****************************************<br />
<br />
^C<br />
clondaq2:stage_in> <br />
<br />
<br />
* Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem:<br />
<br />
wait: coda request >end< in progress<br />
0x916e80 (twork0028): TS GO bit cleared<br />
0x916e80 (twork0028): TS csr: 0xfc070000<br />
0x916e80 (twork0028): TS csr2: 0xffff1c1f<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0<br />
0x916e80 (twork0028): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-apr-2010: message from runcontrol during cancel-reset:<br />
Get all components failed: Lost connection to MySQL server during query<br />
<br />
* Sergey B. 6-apr-2010: EB waiting for following ROCs: <br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
Files produced:<br />
<br />
clon10:ccscans> ll | grep Apr<br />
drwxrwxrwx 2 clasrun onliners 4096 Apr 2 16:23 ./<br />
drwxr-xr-x 163 clasrun 100 73728 Apr 6 2010 ../<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_cc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:09 good_clastrig2*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest10*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest2*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:21 good_croctest3*<br />
-rwxrwxrwx 1 baturin clas 34 Apr 2 13:26 good_ctoftest1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc10*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc11*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc5*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc6*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc7*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc8*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc9*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_polar*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_sc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_sc2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_scaler1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage3* <br />
clon10:ccscans> more good_dc7<br />
accepted link from >129.57.68.80<<br />
clon10:ccscans> more good_dc8<br />
accepted link from >129.57.68.74<<br />
clon10:ccscans><br />
<br />
On 'Reset' EB crashed, outputs:<br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.167.14< port=34468<br />
wait: coda request >prestart< in progress<br />
codaExecute reached, message >exit<, len=4<br />
codaExecute: 'exit' transition<br />
debShutdownBuild 1<br />
cb_delete roc=0<br />
cb_delete roc=1<br />
cb_delete roc=2<br />
cb_delete roc=3<br />
cb_delete roc=4<br />
cb_delete roc=5<br />
cb_delete roc=6<br />
cb_delete roc=7<br />
cb_delete roc=8<br />
cb_delete roc=9<br />
cb_delete roc=10<br />
cb_delete roc=11<br />
cb_delete roc=12<br />
cb_delete roc=13<br />
cb_delete roc=14<br />
cb_delete roc=15<br />
cb_delete roc=16<br />
cb_delete roc=17<br />
cb_delete roc=18<br />
cb_delete roc=19<br />
cb_delete roc=20<br />
cb_delete roc=21<br />
cb_delete roc=22<br />
cb_delete roc=23<br />
cb_delete roc=24<br />
cb_delete roc=25<br />
cb_delete roc=26<br />
cb_delete roc=27<br />
cb_delete roc=28<br />
cb_delete roc=29<br />
cb_delete roc=30<br />
cb_delete roc=31<br />
debShutdownBuild 2<br />
Stop waiting for the rocs because of force_end condition<br />
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0<br />
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!<br />
[0] et_event_new() 1<br />
[0] EVENT_ENCODE_SPEC 2 ..<br />
BOS_encode_spec reached<br />
BOS_encode_spec done: len=116<br />
[0] .. done<br />
[0] handle_build_cleanup<br />
[0] flush ET system<br />
[0] detached from ET<br />
[0] remove mutex locks and shutdown fifos<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[00] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[00] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[00] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[01] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[01] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[01] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[02] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[02] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[02] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[03] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[03] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[03] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[04] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[04] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[04] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[05] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[05] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[05] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[06] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[06] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[06] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[07] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[07] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[07] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[08] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[08] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[08] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[09] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[09] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[09] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[10] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[10] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[10] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 2<br />
[0] count for roc[11] = 2<br />
CCCCCCCCCC: 2<br />
CCCCCCCCCC: 0<br />
[0] count for roc[11] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 2<br />
[0] count for roc[12] = 2<br />
CCCCCCCCCC: 2<br />
CCCCCCCCCC: 0<br />
[0] count for roc[12] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 2<br />
[0] count for roc[13] = 2<br />
CCCCCCCCCC: 2<br />
CCCCCCCCCC: 0<br />
[0] count for roc[13] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[14] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[15] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[16] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[17] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[18] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[19] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[20] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[21] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[22] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[23] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[24] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[25] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[26] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[27] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[28] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[29] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[30] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[31] = 0<br />
CCCCCCCCCC: 0<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] build thread exiting: 1 1<br />
[0] The implementation has detected that the value specified by threa<br />
d does not refer to a joinable thread.<br />
wait: coda request >exit< in progress<br />
polling_routine ================!!!!!!!!!!!!!!!!!!=====<br />
cancel building threads<br />
=c=============================================<br />
=c=============================================<br />
=c=============================================<br />
=c====<br />
>>>>> close link from >dc1< link=0x081c2af0<br />
debCloseLink: theLink=0x081c2af0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc2< link=0x081c4a90<br />
debCloseLink: theLink=0x081c4a90 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc3< link=0x081c4b28<br />
debCloseLink: theLink=0x081c4b28 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc4< link=0x081c14f8<br />
debCloseLink: theLink=0x081c14f8 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc5< link=0x081c1590<br />
debCloseLink: theLink=0x081c1590 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc6< link=0x081c1628<br />
debCloseLink: theLink=0x081c1628 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc8< link=0x081c16c0<br />
debCloseLink: theLink=0x081c16c0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >cc1< link=0x081c1a08<br />
debCloseLink: theLink=0x081c1a08 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >sc1< link=0x081c1aa0<br />
debCloseLink: theLink=0x081c1aa0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >ec1< link=0x081c1b38<br />
debCloseLink: theLink=0x081c1b38 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >ec2< link=0x081c1bd0<br />
debCloseLink: theLink=0x081c1bd0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >scaler2< link=0x681d95c8<br />
debCloseLink: theLink=0x681d95c8 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >scaler3< link=0x681d9660<br />
debCloseLink: theLink=0x681d9660 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >scaler4< link=0x681d96f8<br />
debCloseLink: theLink=0x681d96f8 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >ec4< link=0x00000001<br />
debCloseLink: theLink=0x00000001 -> closing<br />
Segmentation fault<br />
clondaq1:coda_eb> <br />
<br />
DC7:<br />
<br />
codaExecute done<br />
wait: coda request >download prod< in progress<br />
codaExecute reached, message >prestart<, len=8<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ? <br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410567520<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc7<<br />
LINK_constructor_C: set name to >dc7->EB1< <br />
3121: dbsock=410567520<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda req<<<<br />
uest >fields [pr2estart] >>>< in prodowngress<br />
<<<<br />
fields [3] >>>37683<<<<br />
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683<br />
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x18772230 (twork0028INFO: Entering Us): er <br />
Prestart 2<br />
INIT_NAME: rolp->daproc = <br />
trans_rol2:2<br />
adr(nddl)=1db60750 NWBOS=16384<br />
uthbook1: WARN id=1047 already exist - will be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x18772230INFO: User Prest (art 2 executed<br />
twork0028): CTRIGRSS: set handler and done, <br />
code=1<br />
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c<br />
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4130) returns 490106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a580) returns 490106296<br />
0x18772230 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x18772230 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1<br />
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc<br />
0x1882d864 0x00000001 0x1882b0c8<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = 00000019<br />
spds_mask1: 0xfdfbf0<br />
0x18772230_tdc1877 ===> (4twork0028wait: co)<br />
: Use 32-bit Ram List<br />
da request >firsprestartt< in progre csr0=ss<br />
100 (1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0=100 (900) csr18=13<br />
bignetptr=0x641d366fe0 csr1= offset=0x00bigp<br />
rocptr=0x000001d366db80 offset=0x6400000000<br />
00001b<br />
bigproc at 0x<br />
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =<br />
==>1d366dc818 -> 0x1d366dc8<br />
proc_thread reacheUDP_cancel: cancel >inf:d<br />
c7 sys 0, mask 1d<br />
<<br />
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached<br />
<<br />
wait: cproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 1875 1881(2) 1877(2) 1877S(2)<br />
[ 0] 0 0 0 0bignetptr=0x <br />
1d366fe00 offset=0x0000000 0<br />
0 0 0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda reques0t > prestart<br />
0< in progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 0 0 0 1 0 0 <br />
0<br />
[20] bignet at 0x 1d366fe00, bignet.gbig<br />
Buffer at 0x 1d366ff00 -> 0x1d366ff0<br />
0 0 1 0 0 0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC07<<br />
0x18772230 (twork0028111): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <<br />
<<<<<<<<<<<<<<<<<br />
s: >UPDATE process SET state='paused' WHERE <br />
name='dc7'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request .. <br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_cancel: cancel >sta:dc7 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
-> <br />
-> <br />
<br />
DC8:<br />
<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410505056<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc8<<br />
LINK_constructor_C: set name to >dc8->EB1<<br />
3121: dbsock=410505056<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda re<<<<br />
quest >fields [p2restart] >>>< in prupogress<br />
<<<<br />
fields [3] >>>37772<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>up< port=>37772< -> 37772<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37772<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc8->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x187ef280INFO: Entering U (ser Prestart 2<br />
twork0028<br />
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da<br />
proc = 1d2b60750<br />
NWBOS=16384<br />
uthbook1: WARN id=1048 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x187ef280 (twork0028INFO: User Pr): estart <br />
2 executedCTRIGRSS: set handler and done, co<br />
de=<br />
1<br />
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac<br />
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4110) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a560) returns 49<br />
0106296<br />
0x187ef280 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x187ef280 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x187ef280 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c<br />
0x1882d1e4 0x00000001 0x1882aa48<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = wait: coda req00<br />
0000uest >19presta<br />
rtspds_mask1: 0x< ifdfbf0n progress <br />
0x187ef280_tdc1877 === (>twork0028): Use 32-<br />
bit Ram List<br />
4 first csr0=100 (1100) csr18=1364 csr1=6400<br />
001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f<br />
e0100 offset=0x0000000 (0<br />
900) csr18=1364bigprocptr=0x csr1=1d366db864<br />
00001b offset=0x<br />
00000000<br />
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.<br />
gbigBuffer at 0x181d3UDP_cancel: cancel >inf<br />
:dc8 sys 0, mask 166dc8<<br />
wait: coda rUDP_cancel: cancel >inf:dc8 sys <br />
0, mask 1 -> 0x<<br />
1d366dc8eq<br />
firstproc_thread reached<br />
uest >proc_thread reached<br />
csr0=proc_thread reached<br />
prproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
100estart (< in prog1100ress<br />
) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 bignetptr=0x1875 181d366fe081<br />
(2) 1877( offset=0x2) 1877S(00000002)<br />
0<br />
[ 0] 0 0 0 0 0 0 0 <br />
0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda request0 > prestart<br />
0< i n progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 bignet at 0x 1d366fe00, bigne<br />
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366<br />
ff0<br />
0 1 0 0 0<br />
[20] 0 0 0 0 1 0 0 <br />
0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC08<<br />
0x187ef280 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state=<br />
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<<br />
<<<<<<<<<<<<<br />
ERE name='dc8'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_cancel: cancel >sta:dc8 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
.....<br />
<br />
<br />
* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:<br />
........<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
3123: dbsock=59553504<br />
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<<br />
====1==== !!! object->runType=47<br />
token_interval=64<br />
object->name >polar<<br />
big0.proc_on_pmc=0, big0.net_on_pmc=0<br />
big1.proc_on_pmc=0, big1.net_on_pmc=0<br />
name selected<br />
nrow=1<br />
download: name >polar<<br />
code selected<br />
nrow=1<br />
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <<br />
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<<br />
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
set this_roc_id = 25, rocId() can be called from now on<br />
ObjInitName >_moller__init< <br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
INFO: >_moller__init()< routine found<br />
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded<br />
allocate 200 buffers with length 65536 bytes<br />
partCreate1() reached<br />
partCreate1(): pPart=0x008e7570<br />
listInit: cleanup 24 bytes starting from address 0x008e75a0<br />
partCreate1() done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0<br />
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<<br />
0x81c7e0 (twork0004): Init - Done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
Exception at interrupt level:<br />
Exception next instruction address: 0xfff901f0<br />
Fixed Point Register: 0x00000012<br />
Condition Register: 0x40482044<br />
Regs at 0x64e498<br />
Press any key to stop auto-boot...<br />
2<br />
[VxWorks Boot]: <br />
<br />
It suppose to be as following:<br />
.....<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0<br />
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
rol1: downloading DDL table ...<br />
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<<br />
0x91ae30 (twork0001): Init - Done<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1<br />
wait: coda request >download prod< in progress<br />
adr1(nddl_)=0x037b69c8<br />
N name (nname) fmt (nfmt) ncol<br />
[ 1] >PTRN< (4) >B32< (3) 1<br />
[ 2] >PSYN< (4) >B32< (3) 1<br />
[ 3] >RC00< (4) >B32< (3) 1<br />
[ 4] >RC01< (4) >B32< (3) 1<br />
.....<br />
<br />
<br />
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4737
ERROR Book
2010-04-07T00:46:09Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: EB waiting for following ROCs: <br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
Files produced:<br />
<br />
clon10:ccscans> ll | grep Apr<br />
drwxrwxrwx 2 clasrun onliners 4096 Apr 2 16:23 ./<br />
drwxr-xr-x 163 clasrun 100 73728 Apr 6 2010 ../<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_cc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:09 good_clastrig2*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest10*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest2*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:21 good_croctest3*<br />
-rwxrwxrwx 1 baturin clas 34 Apr 2 13:26 good_ctoftest1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc10*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc11*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc5*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc6*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc7*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc8*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc9*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_polar*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_sc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_sc2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_scaler1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage3* <br />
clon10:ccscans> more good_dc7<br />
accepted link from >129.57.68.80<<br />
clon10:ccscans> more good_dc8<br />
accepted link from >129.57.68.74<<br />
clon10:ccscans><br />
<br />
On 'Reset' EB crashed, outputs:<br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.167.14< port=34468<br />
wait: coda request >prestart< in progress<br />
codaExecute reached, message >exit<, len=4<br />
codaExecute: 'exit' transition<br />
debShutdownBuild 1<br />
cb_delete roc=0<br />
cb_delete roc=1<br />
cb_delete roc=2<br />
cb_delete roc=3<br />
cb_delete roc=4<br />
cb_delete roc=5<br />
cb_delete roc=6<br />
cb_delete roc=7<br />
cb_delete roc=8<br />
cb_delete roc=9<br />
cb_delete roc=10<br />
cb_delete roc=11<br />
cb_delete roc=12<br />
cb_delete roc=13<br />
cb_delete roc=14<br />
cb_delete roc=15<br />
cb_delete roc=16<br />
cb_delete roc=17<br />
cb_delete roc=18<br />
cb_delete roc=19<br />
cb_delete roc=20<br />
cb_delete roc=21<br />
cb_delete roc=22<br />
cb_delete roc=23<br />
cb_delete roc=24<br />
cb_delete roc=25<br />
cb_delete roc=26<br />
cb_delete roc=27<br />
cb_delete roc=28<br />
cb_delete roc=29<br />
cb_delete roc=30<br />
cb_delete roc=31<br />
debShutdownBuild 2<br />
Stop waiting for the rocs because of force_end condition<br />
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0<br />
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!<br />
[0] et_event_new() 1<br />
[0] EVENT_ENCODE_SPEC 2 ..<br />
BOS_encode_spec reached<br />
BOS_encode_spec done: len=116<br />
[0] .. done<br />
[0] handle_build_cleanup<br />
[0] flush ET system<br />
[0] detached from ET<br />
[0] remove mutex locks and shutdown fifos<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[00] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[00] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[00] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[01] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[01] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[01] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[02] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[02] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[02] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[03] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[03] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[03] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[04] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[04] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[04] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[05] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[05] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[05] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[06] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[06] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[06] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[07] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[07] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[07] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[08] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[08] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[08] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[09] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[09] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[09] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 3<br />
[0] count for roc[10] = 3<br />
CCCCCCCCCC: 3<br />
CCCCCCCCCC: 1<br />
[0] count for roc[10] = 1<br />
CCCCCCCCCC: 1<br />
CCCCCCCCCC: 0<br />
[0] count for roc[10] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 2<br />
[0] count for roc[11] = 2<br />
CCCCCCCCCC: 2<br />
CCCCCCCCCC: 0<br />
[0] count for roc[11] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 2<br />
[0] count for roc[12] = 2<br />
CCCCCCCCCC: 2<br />
CCCCCCCCCC: 0<br />
[0] count for roc[12] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 2<br />
[0] count for roc[13] = 2<br />
CCCCCCCCCC: 2<br />
CCCCCCCCCC: 0<br />
[0] count for roc[13] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[14] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[15] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[16] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[17] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[18] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[19] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[20] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[21] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[22] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[23] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[24] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[25] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[26] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[27] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[28] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[29] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[30] = 0<br />
CCCCCCCCCC: 0<br />
[0] flush input streams<br />
CCCCCCCCCC: 0<br />
[0] count for roc[31] = 0<br />
CCCCCCCCCC: 0<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] ============= Build threads cleaned<br />
[0] build thread exiting: 1 1<br />
[0] The implementation has detected that the value specified by threa<br />
d does not refer to a joinable thread.<br />
wait: coda request >exit< in progress<br />
polling_routine ================!!!!!!!!!!!!!!!!!!=====<br />
cancel building threads<br />
=c=============================================<br />
=c=============================================<br />
=c=============================================<br />
=c====<br />
>>>>> close link from >dc1< link=0x081c2af0<br />
debCloseLink: theLink=0x081c2af0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc2< link=0x081c4a90<br />
debCloseLink: theLink=0x081c4a90 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc3< link=0x081c4b28<br />
debCloseLink: theLink=0x081c4b28 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc4< link=0x081c14f8<br />
debCloseLink: theLink=0x081c14f8 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc5< link=0x081c1590<br />
debCloseLink: theLink=0x081c1590 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc6< link=0x081c1628<br />
debCloseLink: theLink=0x081c1628 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >dc8< link=0x081c16c0<br />
debCloseLink: theLink=0x081c16c0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >cc1< link=0x081c1a08<br />
debCloseLink: theLink=0x081c1a08 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >sc1< link=0x081c1aa0<br />
debCloseLink: theLink=0x081c1aa0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >ec1< link=0x081c1b38<br />
debCloseLink: theLink=0x081c1b38 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >ec2< link=0x081c1bd0<br />
debCloseLink: theLink=0x081c1bd0 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >scaler2< link=0x681d95c8<br />
debCloseLink: theLink=0x681d95c8 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >scaler3< link=0x681d9660<br />
debCloseLink: theLink=0x681d9660 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >scaler4< link=0x681d96f8<br />
debCloseLink: theLink=0x681d96f8 -> closing<br />
debCloseLink: link is down<br />
debCloseLink: free memory<br />
debCloseLink: done.<br />
=c====<br />
>>>>> close link from >ec4< link=0x00000001<br />
debCloseLink: theLink=0x00000001 -> closing<br />
Segmentation fault<br />
clondaq1:coda_eb> <br />
<br />
DC7:<br />
<br />
codaExecute done<br />
wait: coda request >download prod< in progre<br />
ss<br />
codaExecute reached, message >prestart<, len<br />
=8<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410567520<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc7<<br />
LINK_constructor_C: set name to >dc7->EB1<<br />
3121: dbsock=410567520<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda req<<<<br />
uest >fields [pr2estart] >>>< in prodowngres<br />
s<br />
<<<<br />
fields [3] >>>37683<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>down< port=>37683< -> 37683<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37683<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc7->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x18772230 (twork0028INFO: Entering Us): er <br />
Prestart 2<br />
INIT_NAME: rolp->daproc = <br />
trans_rol2:2<br />
adr(nddl)=1db60750 NWBOS=16384<br />
uthbook1: WARN id=1047 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x18772230INFO: User Prest (art 2 executed<br />
twork0028): CTRIGRSS: set handler and done, <br />
code=1<br />
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c<br />
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4130) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a580) returns 49<br />
0106296<br />
0x18772230 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x18772230 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x18772230 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc<br />
0x1882d864 0x00000001 0x1882b0c8<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = 00000019<br />
spds_mask1: 0xfdfbf0<br />
0x18772230_tdc1877 ===> (4twork0028wait: co)<br />
: Use 32-bit Ram List<br />
da request >firsprestartt< in progre csr0=ss<br />
100 (1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0=100 (900) csr18=13<br />
bignetptr=0x641d366fe0 csr1= offset=0x00bigp<br />
rocptr=0x000001d366db80 offset=0x6400000000<br />
00001b<br />
bigproc at 0x<br />
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =<br />
==>1d366dc818 -> 0x1d366dc8<br />
proc_thread reacheUDP_cancel: cancel >inf:d<br />
c7 sys 0, mask 1d<br />
<<br />
firstUDP_cancel: cancel >proc_threainf:dc7 s<br />
ys 0, mask 1d reached<br />
<<br />
wait: cproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
csr0=oda request >1prestart00< in progr (es<br />
s<br />
1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 1875 1881(2) 1877(2) 1877S(<br />
2)<br />
[ 0] 0 0 0 0bignetptr=0x <br />
1d366fe00 offset=0x0000000 0<br />
0 0 0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda reques0t > prestart<br />
0< in progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 0 0 0 1 0 0 <br />
0<br />
[20] bignet at 0x 1d366fe00, bignet.gbig<br />
Buffer at 0x 1d366ff00 -> 0x1d366ff0<br />
0 0 1 0 0 0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC07<<br />
0x18772230 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 7 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <<br />
<<<<<<<<<<<<<<<<<br />
s: >UPDATE process SET state='paused' WHERE <br />
name='dc7'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_cancel: cancel >sta:dc7 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
<br />
-> <br />
-> <br />
<br />
DC8:<br />
<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410505056<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc8<<br />
LINK_constructor_C: set name to >dc8->EB1<<br />
3121: dbsock=410505056<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda re<<<<br />
quest >fields [p2restart] >>>< in prupogress<br />
<<<<br />
fields [3] >>>37772<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>up< port=>37772< -> 37772<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37772<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc8->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x187ef280INFO: Entering U (ser Prestart 2<br />
twork0028<br />
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da<br />
proc = 1d2b60750<br />
NWBOS=16384<br />
uthbook1: WARN id=1048 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x187ef280 (twork0028INFO: User Pr): estart <br />
2 executedCTRIGRSS: set handler and done, co<br />
de=<br />
1<br />
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac<br />
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4110) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a560) returns 49<br />
0106296<br />
0x187ef280 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x187ef280 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x187ef280 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c<br />
0x1882d1e4 0x00000001 0x1882aa48<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = wait: coda req00<br />
0000uest >19presta<br />
rtspds_mask1: 0x< ifdfbf0n progress<br />
<br />
0x187ef280_tdc1877 === (>twork0028): Use 32-<br />
bit Ram List<br />
4 first csr0=100 (1100) csr18=1364 csr1=6400<br />
001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f<br />
e0100 offset=0x0000000 (0<br />
900) csr18=1364bigprocptr=0x csr1=1d366db864<br />
00001b offset=0x<br />
00000000<br />
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.<br />
gbigBuffer at 0x181d3UDP_cancel: cancel >inf<br />
:dc8 sys 0, mask 166dc8<<br />
wait: coda rUDP_cancel: cancel >inf:dc8 sys <br />
0, mask 1 -> 0x<<br />
1d366dc8eq<br />
firstproc_thread reached<br />
uest >proc_thread reached<br />
csr0=proc_thread reached<br />
prproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
100estart (< in prog1100ress<br />
) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 bignetptr=0x1875 181d366fe081<br />
(2) 1877( offset=0x2) 1877S(00000002)<br />
0<br />
[ 0] 0 0 0 0 0 0 0 <br />
0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda request0 > prestart<br />
0< i n progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 bignet at 0x 1d366fe00, bigne<br />
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366<br />
ff0<br />
0 1 0 0 0<br />
[20] 0 0 0 0 1 0 0 <br />
0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC08<<br />
0x187ef280 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state=<br />
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<<br />
<<<<<<<<<<<<<br />
ERE name='dc8'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_cancel: cancel >sta:dc8 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
<br />
<br />
<br />
<br />
* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:<br />
........<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
3123: dbsock=59553504<br />
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<<br />
====1==== !!! object->runType=47<br />
token_interval=64<br />
object->name >polar<<br />
big0.proc_on_pmc=0, big0.net_on_pmc=0<br />
big1.proc_on_pmc=0, big1.net_on_pmc=0<br />
name selected<br />
nrow=1<br />
download: name >polar<<br />
code selected<br />
nrow=1<br />
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <<br />
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<<br />
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
set this_roc_id = 25, rocId() can be called from now on<br />
ObjInitName >_moller__init< <br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
INFO: >_moller__init()< routine found<br />
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded<br />
allocate 200 buffers with length 65536 bytes<br />
partCreate1() reached<br />
partCreate1(): pPart=0x008e7570<br />
listInit: cleanup 24 bytes starting from address 0x008e75a0<br />
partCreate1() done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0<br />
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<<br />
0x81c7e0 (twork0004): Init - Done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
Exception at interrupt level:<br />
Exception next instruction address: 0xfff901f0<br />
Fixed Point Register: 0x00000012<br />
Condition Register: 0x40482044<br />
Regs at 0x64e498<br />
Press any key to stop auto-boot...<br />
2<br />
[VxWorks Boot]: <br />
<br />
It suppose to be as following:<br />
.....<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0<br />
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
rol1: downloading DDL table ...<br />
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<<br />
0x91ae30 (twork0001): Init - Done<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1<br />
wait: coda request >download prod< in progress<br />
adr1(nddl_)=0x037b69c8<br />
N name (nname) fmt (nfmt) ncol<br />
[ 1] >PTRN< (4) >B32< (3) 1<br />
[ 2] >PSYN< (4) >B32< (3) 1<br />
[ 3] >RC00< (4) >B32< (3) 1<br />
[ 4] >RC01< (4) >B32< (3) 1<br />
.....<br />
<br />
<br />
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4736
ERROR Book
2010-04-07T00:36:16Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: EB waiting for following ROCs: <br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
Files produced:<br />
<br />
clon10:ccscans> ll | grep Apr<br />
drwxrwxrwx 2 clasrun onliners 4096 Apr 2 16:23 ./<br />
drwxr-xr-x 163 clasrun 100 73728 Apr 6 2010 ../<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_cc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:09 good_clastrig2*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest10*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest2*<br />
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:21 good_croctest3*<br />
-rwxrwxrwx 1 baturin clas 34 Apr 2 13:26 good_ctoftest1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc10*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc11*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc5*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc6*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc7*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc8*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc9*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_polar*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_sc1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_sc2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_scaler1*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler3*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler4*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage2*<br />
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage3* <br />
clon10:ccscans> more good_dc7<br />
accepted link from >129.57.68.80<<br />
clon10:ccscans> more good_dc8<br />
accepted link from >129.57.68.74<<br />
clon10:ccscans><br />
<br />
<br />
DC7:<br />
<br />
codaExecute done<br />
wait: coda request >download prod< in progre<br />
ss<br />
codaExecute reached, message >prestart<, len<br />
=8<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410567520<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc7<<br />
LINK_constructor_C: set name to >dc7->EB1<<br />
3121: dbsock=410567520<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda req<<<<br />
uest >fields [pr2estart] >>>< in prodowngres<br />
s<br />
<<<<br />
fields [3] >>>37683<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>down< port=>37683< -> 37683<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37683<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc7->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x18772230 (twork0028INFO: Entering Us): er <br />
Prestart 2<br />
INIT_NAME: rolp->daproc = <br />
trans_rol2:2<br />
adr(nddl)=1db60750 NWBOS=16384<br />
uthbook1: WARN id=1047 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x18772230INFO: User Prest (art 2 executed<br />
twork0028): CTRIGRSS: set handler and done, <br />
code=1<br />
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c<br />
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4130) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a580) returns 49<br />
0106296<br />
0x18772230 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x18772230 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x18772230 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc<br />
0x1882d864 0x00000001 0x1882b0c8<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = 00000019<br />
spds_mask1: 0xfdfbf0<br />
0x18772230_tdc1877 ===> (4twork0028wait: co)<br />
: Use 32-bit Ram List<br />
da request >firsprestartt< in progre csr0=ss<br />
100 (1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0=100 (900) csr18=13<br />
bignetptr=0x641d366fe0 csr1= offset=0x00bigp<br />
rocptr=0x000001d366db80 offset=0x6400000000<br />
00001b<br />
bigproc at 0x<br />
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =<br />
==>1d366dc818 -> 0x1d366dc8<br />
proc_thread reacheUDP_cancel: cancel >inf:d<br />
c7 sys 0, mask 1d<br />
<<br />
firstUDP_cancel: cancel >proc_threainf:dc7 s<br />
ys 0, mask 1d reached<br />
<<br />
wait: cproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
csr0=oda request >1prestart00< in progr (es<br />
s<br />
1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 1875 1881(2) 1877(2) 1877S(<br />
2)<br />
[ 0] 0 0 0 0bignetptr=0x <br />
1d366fe00 offset=0x0000000 0<br />
0 0 0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda reques0t > prestart<br />
0< in progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 0 0 0 1 0 0 <br />
0<br />
[20] bignet at 0x 1d366fe00, bignet.gbig<br />
Buffer at 0x 1d366ff00 -> 0x1d366ff0<br />
0 0 1 0 0 0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC07<<br />
0x18772230 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 7 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <<br />
<<<<<<<<<<<<<<<<<br />
s: >UPDATE process SET state='paused' WHERE <br />
name='dc7'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_cancel: cancel >sta:dc7 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
<br />
-> <br />
-> <br />
<br />
DC8:<br />
<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410505056<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc8<<br />
LINK_constructor_C: set name to >dc8->EB1<<br />
3121: dbsock=410505056<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda re<<<<br />
quest >fields [p2restart] >>>< in prupogress<br />
<<<<br />
fields [3] >>>37772<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>up< port=>37772< -> 37772<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37772<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc8->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x187ef280INFO: Entering U (ser Prestart 2<br />
twork0028<br />
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da<br />
proc = 1d2b60750<br />
NWBOS=16384<br />
uthbook1: WARN id=1048 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x187ef280 (twork0028INFO: User Pr): estart <br />
2 executedCTRIGRSS: set handler and done, co<br />
de=<br />
1<br />
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac<br />
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4110) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a560) returns 49<br />
0106296<br />
0x187ef280 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x187ef280 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x187ef280 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c<br />
0x1882d1e4 0x00000001 0x1882aa48<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = wait: coda req00<br />
0000uest >19presta<br />
rtspds_mask1: 0x< ifdfbf0n progress<br />
<br />
0x187ef280_tdc1877 === (>twork0028): Use 32-<br />
bit Ram List<br />
4 first csr0=100 (1100) csr18=1364 csr1=6400<br />
001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f<br />
e0100 offset=0x0000000 (0<br />
900) csr18=1364bigprocptr=0x csr1=1d366db864<br />
00001b offset=0x<br />
00000000<br />
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.<br />
gbigBuffer at 0x181d3UDP_cancel: cancel >inf<br />
:dc8 sys 0, mask 166dc8<<br />
wait: coda rUDP_cancel: cancel >inf:dc8 sys <br />
0, mask 1 -> 0x<<br />
1d366dc8eq<br />
firstproc_thread reached<br />
uest >proc_thread reached<br />
csr0=proc_thread reached<br />
prproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
100estart (< in prog1100ress<br />
) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 bignetptr=0x1875 181d366fe081<br />
(2) 1877( offset=0x2) 1877S(00000002)<br />
0<br />
[ 0] 0 0 0 0 0 0 0 <br />
0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda request0 > prestart<br />
0< i n progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 bignet at 0x 1d366fe00, bigne<br />
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366<br />
ff0<br />
0 1 0 0 0<br />
[20] 0 0 0 0 1 0 0 <br />
0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC08<<br />
0x187ef280 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state=<br />
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<<br />
<<<<<<<<<<<<<br />
ERE name='dc8'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_cancel: cancel >sta:dc8 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
<br />
<br />
<br />
<br />
* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:<br />
........<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
3123: dbsock=59553504<br />
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<<br />
====1==== !!! object->runType=47<br />
token_interval=64<br />
object->name >polar<<br />
big0.proc_on_pmc=0, big0.net_on_pmc=0<br />
big1.proc_on_pmc=0, big1.net_on_pmc=0<br />
name selected<br />
nrow=1<br />
download: name >polar<<br />
code selected<br />
nrow=1<br />
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <<br />
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<<br />
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
set this_roc_id = 25, rocId() can be called from now on<br />
ObjInitName >_moller__init< <br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
INFO: >_moller__init()< routine found<br />
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded<br />
allocate 200 buffers with length 65536 bytes<br />
partCreate1() reached<br />
partCreate1(): pPart=0x008e7570<br />
listInit: cleanup 24 bytes starting from address 0x008e75a0<br />
partCreate1() done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0<br />
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<<br />
0x81c7e0 (twork0004): Init - Done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
Exception at interrupt level:<br />
Exception next instruction address: 0xfff901f0<br />
Fixed Point Register: 0x00000012<br />
Condition Register: 0x40482044<br />
Regs at 0x64e498<br />
Press any key to stop auto-boot...<br />
2<br />
[VxWorks Boot]: <br />
<br />
It suppose to be as following:<br />
.....<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0<br />
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
rol1: downloading DDL table ...<br />
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<<br />
0x91ae30 (twork0001): Init - Done<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1<br />
wait: coda request >download prod< in progress<br />
adr1(nddl_)=0x037b69c8<br />
N name (nname) fmt (nfmt) ncol<br />
[ 1] >PTRN< (4) >B32< (3) 1<br />
[ 2] >PSYN< (4) >B32< (3) 1<br />
[ 3] >RC00< (4) >B32< (3) 1<br />
[ 4] >RC01< (4) >B32< (3) 1<br />
.....<br />
<br />
<br />
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4735
ERROR Book
2010-04-07T00:32:05Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: EB waiting for following ROCs: <br />
<br />
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 21 22 25 26 28 29 30<br />
<br />
DC7:<br />
<br />
codaExecute done<br />
wait: coda request >download prod< in progre<br />
ss<br />
codaExecute reached, message >prestart<, len<br />
=8<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410567520<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc7'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc7<<br />
LINK_constructor_C: set name to >dc7->EB1<<br />
3121: dbsock=410567520<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda req<<<<br />
uest >fields [pr2estart] >>>< in prodowngres<br />
s<br />
<<<<br />
fields [3] >>>37683<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>down< port=>37683< -> 37683<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37683<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc7->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x18772230 (twork0028INFO: Entering Us): er <br />
Prestart 2<br />
INIT_NAME: rolp->daproc = <br />
trans_rol2:2<br />
adr(nddl)=1db60750 NWBOS=16384<br />
uthbook1: WARN id=1047 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x18772230INFO: User Prest (art 2 executed<br />
twork0028): CTRIGRSS: set handler and done, <br />
code=1<br />
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c<br />
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4130) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a580) returns 49<br />
0106296<br />
0x18772230 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x18772230 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x18772230 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc<br />
0x1882d864 0x00000001 0x1882b0c8<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = 00000019<br />
spds_mask1: 0xfdfbf0<br />
0x18772230_tdc1877 ===> (4twork0028wait: co)<br />
: Use 32-bit Ram List<br />
da request >firsprestartt< in progre csr0=ss<br />
100 (1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0=100 (900) csr18=13<br />
bignetptr=0x641d366fe0 csr1= offset=0x00bigp<br />
rocptr=0x000001d366db80 offset=0x6400000000<br />
00001b<br />
bigproc at 0x<br />
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =<br />
==>1d366dc818 -> 0x1d366dc8<br />
proc_thread reacheUDP_cancel: cancel >inf:d<br />
c7 sys 0, mask 1d<br />
<<br />
firstUDP_cancel: cancel >proc_threainf:dc7 s<br />
ys 0, mask 1d reached<br />
<<br />
wait: cproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
csr0=oda request >1prestart00< in progr (es<br />
s<br />
1100) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 1875 1881(2) 1877(2) 1877S(<br />
2)<br />
[ 0] 0 0 0 0bignetptr=0x <br />
1d366fe00 offset=0x0000000 0<br />
0 0 0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda reques0t > prestart<br />
0< in progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 0 0 0 1 0 0 <br />
0<br />
[20] bignet at 0x 1d366fe00, bignet.gbig<br />
Buffer at 0x 1d366ff00 -> 0x1d366ff0<br />
0 0 1 0 0 0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC07<<br />
0x18772230 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 7 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <<br />
<<<<<<<<<<<<<<<<<br />
s: >UPDATE process SET state='paused' WHERE <br />
name='dc7'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_standard_request >sta:dc7 paused<<br />
UDP_cancel: cancel >sta:dc7 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
<br />
-> <br />
-> <br />
<br />
DC8:<br />
<br />
codaExecute: 'prestart' transition<br />
roc_prestart reached<br />
partReInitAll() reached ?<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
rocCleanup reached<br />
partReInitAll() reached ?<br />
wait: coda request >prestart< in progress<br />
partReInitAll() reached !<br />
partReInitAll() done<br />
315: dbsock=410505056<br />
>>> prestarting, run 62496, type 47<br />
rocMask=0x76fefffe<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod<br />
WHERE name='dc8'<<br />
++++> output to network<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1< <br />
3<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<<br />
rocCloselink reached<br />
rocOpenLinks reached<br />
LINK_constructor_C: set host to >dc8<<br />
LINK_constructor_C: set name to >dc8->EB1<<br />
3121: dbsock=410505056<br />
name selected<br />
nrow=1<br />
fields [0] >>>TCP<<<<br />
fields [1] >>>clondaq1-daq1wait: coda re<<<<br />
quest >fields [p2restart] >>>< in prupogress<br />
<<<<br />
fields [3] >>>37772<<<<br />
parsing results: type=>TCP< host=>clondaq1-d<br />
aq1< state=>up< port=>37772< -> 37772<br />
LINK_establish: socket # 17<br />
LINK_establish: socket buffer size is 48000(<br />
0x0000bb80) bytes<br />
LINK_establish: keepAlive is 8<br />
LINK_establish: socket 17 is ready: host 129<br />
.57.68.22 port 37772<br />
DB command >UPDATE links SET state='up' WHER<br />
E name='dc8->EB1'<<br />
UPDATE process success<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
Spawn proc/net threads<br />
0x187ef280INFO: Entering U (ser Prestart 2<br />
twork0028<br />
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da<br />
proc = 1d2b60750<br />
NWBOS=16384<br />
uthbook1: WARN id=1048 already exist - will <br />
be replaced<br />
uthbook1: free histogram buffer ... done.<br />
RAW=0 PROFILE=1 NOTRANS=0<br />
MAX_EVENT_LENGTH = 65536 NWBOS = 16384<br />
0x187ef280 (twork0028INFO: User Pr): estart <br />
2 executedCTRIGRSS: set handler and done, co<br />
de=<br />
1<br />
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac<br />
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8<br />
wait: coda request >prestart< in progress<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
taskSpawn("coda_net",0x187d4110) returns 490<br />
106848<br />
wait: coda request >prestart< in progress<br />
setHeartError: 0 >sys 0, mask 1<<br />
WARN: HeartBeat[0]: heartbeat=19424(19424) h<br />
eartmask=1<br />
bignetptr=0x1d366fe0 offset=0x00000000<br />
bigprocptr=0x1d366db8 offset=0x00000000<br />
taskSpawn("coda_proc",0x1874a560) returns 49<br />
0106296<br />
0x187ef280 (twork0028): INIT_NAME: rolp->dap<br />
roc = 2<br />
0x187ef280 (twork0028): INFO: Entering User <br />
Prestart<br />
disconnecting vector 236 <br />
0x187ef280 (twork0028): CTRIGRSA: set handle<br />
r and done, code=1<br />
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c<br />
0x1882d1e4 0x00000001 0x1882aa48<br />
Sequencer Status Register = FFFFA000<br />
Last Sequencer KeyAddress = FFFF0003<br />
Fastbus Status Register1 = FFFFF000<br />
Fastbus Status Register2 = FFFF0000<br />
Fastbus Last Primary Addr = wait: coda req00<br />
0000uest >19presta<br />
rtspds_mask1: 0x< ifdfbf0n progress<br />
<br />
0x187ef280_tdc1877 === (>twork0028): Use 32-<br />
bit Ram List<br />
4 first csr0=100 (1100) csr18=1364 csr1=6400<br />
001b<br />
_tdc1877 ===>5 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>6 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>7 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>8 middle csr0=100 (1900) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>9 last csr0=100 (900) csr18=136<br />
4 csr1=6400001b<br />
_tdc1877 ===>11 first csr0=100 (1100) csr18=<br />
1364 csr1=6400001b<br />
_tdc1877 ===>12 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>13 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>14 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>15 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f<br />
e0100 offset=0x0000000 (0<br />
900) csr18=1364bigprocptr=0x csr1=1d366db864<br />
00001b offset=0x<br />
00000000<br />
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.<br />
gbigBuffer at 0x181d3UDP_cancel: cancel >inf<br />
:dc8 sys 0, mask 166dc8<<br />
wait: coda rUDP_cancel: cancel >inf:dc8 sys <br />
0, mask 1 -> 0x<<br />
1d366dc8eq<br />
firstproc_thread reached<br />
uest >proc_thread reached<br />
csr0=proc_thread reached<br />
prproc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
proc_thread reached<br />
100estart (< in prog1100ress<br />
) csr18=1364 csr1=6400001b<br />
_tdc1877 ===>19 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>20 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>21 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>22 middle csr0=100 (1900) csr18<br />
=1364 csr1=6400001b<br />
_tdc1877 ===>23 last csr0=100 (900) csr18=13<br />
64 csr1=6400001b<br />
DEBUG in SFIRamLoad => 0 1 0 0 0<br />
slot# 1872 bignetptr=0x1875 181d366fe081<br />
(2) 1877( offset=0x2) 1877S(00000002)<br />
0<br />
[ 0] 0 0 0 0 0 0 0 <br />
0<br />
[ 1] 0 0 0 0 0 0 0 <br />
0<br />
[ 2] 0 0 0 0 0 0 0 <br />
0<br />
[ 3] 0 0 0 0 0 0 0 <br />
0<br />
[ 4] 0 0 0 0 1 0 0 <br />
0<br />
[ 5] 0 0 0 0 1 0 0 <br />
0<br />
[ 6] 0 0 0 0 1 0 0 <br />
0<br />
[ 7] 0 0 0 0 1 0 0 <br />
0<br />
[ 8] 0 0 0 0 1 0 0 <br />
0<br />
[ 9] 0 0 0 0 1 2 0 <br />
0<br />
[10] wait: coda request0 > prestart<br />
0< i n progress<br />
0 0 0 0 0 0<br />
[11] 0 0 0 0 1 0 0 <br />
0<br />
[12] 0 0 0 0 1 0 0 <br />
0<br />
[13] 0 0 0 0 1 0 0 <br />
0<br />
[14] 0 0 0 0 1 0 0 <br />
0<br />
[15] 0 0 0 0 1 0 0 <br />
0<br />
[16] 0 0 0 0 1 2 0 <br />
0<br />
[17] 0 0 0 0 0 0 0 <br />
0<br />
[18] 0 0 0 0 1 0 0 <br />
0<br />
[19] 0 bignet at 0x 1d366fe00, bigne<br />
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366<br />
ff0<br />
0 1 0 0 0<br />
[20] 0 0 0 0 1 0 0 <br />
0<br />
[21] 0 0 0 0 1 0 0 <br />
0<br />
[22] 0 0 0 0 1 0 0 <br />
0<br />
[23] 0 0 0 0 1 2 0 <br />
0<br />
[24] 0 0 0 0 0 0 0 <br />
0<br />
[25] 0 0 0 0 0 0 0 <br />
0<br />
DEBUG 0 1 0 0 <br />
INFO: datascan = 0x00000000<br />
INFO: datascan&spds_mask1 = 0x00000000<br />
rcname >RC08<<br />
0x187ef280 (twork0028111<br />
): INFO: User Prestart Executed<br />
333: rocp->primwait: coda request >efd=prest<br />
art17< in p<br />
rogress<br />
informEB reached<br />
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011<br />
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000<br />
0000<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state=<br />
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<<br />
<<<<<<<<<<<<<br />
ERE name='dc8'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_standard_request >sta:dc8 paused<<br />
UDP_cancel: cancel >sta:dc8 downloaded<<br />
codaUpdateStatus: updating request done<br />
prestarted<br />
POLLS: 0 0<br />
codaExecute done<br />
<br />
<br />
<br />
<br />
* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:<br />
........<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
3123: dbsock=59553504<br />
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<<br />
====1==== !!! object->runType=47<br />
token_interval=64<br />
object->name >polar<<br />
big0.proc_on_pmc=0, big0.net_on_pmc=0<br />
big1.proc_on_pmc=0, big1.net_on_pmc=0<br />
name selected<br />
nrow=1<br />
download: name >polar<<br />
code selected<br />
nrow=1<br />
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <<br />
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<<br />
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
set this_roc_id = 25, rocId() can be called from now on<br />
ObjInitName >_moller__init< <br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
INFO: >_moller__init()< routine found<br />
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded<br />
allocate 200 buffers with length 65536 bytes<br />
partCreate1() reached<br />
partCreate1(): pPart=0x008e7570<br />
listInit: cleanup 24 bytes starting from address 0x008e75a0<br />
partCreate1() done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0<br />
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<<br />
0x81c7e0 (twork0004): Init - Done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
Exception at interrupt level:<br />
Exception next instruction address: 0xfff901f0<br />
Fixed Point Register: 0x00000012<br />
Condition Register: 0x40482044<br />
Regs at 0x64e498<br />
Press any key to stop auto-boot...<br />
2<br />
[VxWorks Boot]: <br />
<br />
It suppose to be as following:<br />
.....<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0<br />
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
rol1: downloading DDL table ...<br />
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<<br />
0x91ae30 (twork0001): Init - Done<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1<br />
wait: coda request >download prod< in progress<br />
adr1(nddl_)=0x037b69c8<br />
N name (nname) fmt (nfmt) ncol<br />
[ 1] >PTRN< (4) >B32< (3) 1<br />
[ 2] >PSYN< (4) >B32< (3) 1<br />
[ 3] >RC00< (4) >B32< (3) 1<br />
[ 4] >RC01< (4) >B32< (3) 1<br />
.....<br />
<br />
<br />
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4734
ERROR Book
2010-04-07T00:16:12Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:<br />
........<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
3123: dbsock=59553504<br />
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<<br />
====1==== !!! object->runType=47<br />
token_interval=64<br />
object->name >polar<<br />
big0.proc_on_pmc=0, big0.net_on_pmc=0<br />
big1.proc_on_pmc=0, big1.net_on_pmc=0<br />
name selected<br />
nrow=1<br />
download: name >polar<<br />
code selected<br />
nrow=1<br />
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <<br />
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<<br />
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
set this_roc_id = 25, rocId() can be called from now on<br />
ObjInitName >_moller__init< <br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
INFO: >_moller__init()< routine found<br />
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded<br />
allocate 200 buffers with length 65536 bytes<br />
partCreate1() reached<br />
partCreate1(): pPart=0x008e7570<br />
listInit: cleanup 24 bytes starting from address 0x008e75a0<br />
partCreate1() done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0<br />
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<<br />
0x81c7e0 (twork0004): Init - Done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
Exception at interrupt level:<br />
Exception next instruction address: 0xfff901f0<br />
Fixed Point Register: 0x00000012<br />
Condition Register: 0x40482044<br />
Regs at 0x64e498<br />
Press any key to stop auto-boot...<br />
2<br />
[VxWorks Boot]: <br />
<br />
It suppose to be as following:<br />
.....<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0<br />
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
rol1: downloading DDL table ...<br />
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<<br />
0x91ae30 (twork0001): Init - Done<br />
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1<br />
wait: coda request >download prod< in progress<br />
adr1(nddl_)=0x037b69c8<br />
N name (nname) fmt (nfmt) ncol<br />
[ 1] >PTRN< (4) >B32< (3) 1<br />
[ 2] >PSYN< (4) >B32< (3) 1<br />
[ 3] >RC00< (4) >B32< (3) 1<br />
[ 4] >RC01< (4) >B32< (3) 1<br />
.....<br />
<br />
<br />
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4733
ERROR Book
2010-04-07T00:11:00Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:<br />
........<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
3123: dbsock=59553504<br />
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<<br />
====1==== !!! object->runType=47<br />
token_interval=64<br />
object->name >polar<<br />
big0.proc_on_pmc=0, big0.net_on_pmc=0<br />
big1.proc_on_pmc=0, big1.net_on_pmc=0<br />
name selected<br />
nrow=1<br />
download: name >polar<<br />
code selected<br />
nrow=1<br />
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <<br />
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<<br />
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<<br />
set this_roc_id = 25, rocId() can be called from now on<br />
ObjInitName >_moller__init< <br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
INFO: >_moller__init()< routine found<br />
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded<br />
allocate 200 buffers with length 65536 bytes<br />
partCreate1() reached<br />
partCreate1(): pPart=0x008e7570<br />
listInit: cleanup 24 bytes starting from address 0x008e75a0<br />
partCreate1() done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0<br />
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1<br />
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200<br />
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<<br />
0x81c7e0 (twork0004): Init - Done<br />
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...<br />
wait: coda request >download prod< in progress<br />
wait: coda request >download prod< in progress<br />
Exception at interrupt level:<br />
Exception next instruction address: 0xfff901f0<br />
Fixed Point Register: 0x00000012<br />
Condition Register: 0x40482044<br />
Regs at 0x64e498<br />
Press any key to stop auto-boot...<br />
2<br />
[VxWorks Boot]: <br />
<br />
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4732
ERROR Book
2010-04-07T00:06:54Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4731
ERROR Book
2010-04-07T00:06:27Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:<br />
-> ts2status<br />
CSR 1 (0x0):<br />
Go : 1<br />
Pause on Next scheduled Sync : 0<br />
Sync and Pause : 0<br />
Initiate Sync Event : 0<br />
Initiate Program 1 Event : 0<br />
Initiate Program 2 Event : 1<br />
Enable Level 1 (drives outputs) : 1<br />
Override Inhibit : 0<br />
Test Mode : 0<br />
Reserved : 0<br />
Reset : 0<br />
Initialize : 0<br />
<br />
Sync Event occurred : 0<br />
Program 1 Event occurred : 1<br />
Program 2 Event occurred : 0<br />
Late Fail occurred : 0<br />
Inhibit occurred : 1<br />
Write FIFO error occurred : 0<br />
Read FIFO error occurreds : 1<br />
CSR 2 (0x4):<br />
Enable Scheduled Sync : 1<br />
Use Clear Permit Timer : 1<br />
Use Front Busy Timer : 1<br />
Use Clear Hold Timer : 1<br />
Use External Front Busy : 1<br />
Lock ROC Branch 1 : 0<br />
Lock ROC Branch 2 : 0<br />
Lock ROC Branch 3 : 0<br />
Lock ROC Branch 4 : 0<br />
Lock ROC Branch 5 : 0<br />
Enable Program 1 front panel input : 1<br />
Enable Interrupt : 1<br />
Enable local ROC (branch 5) : 1<br />
Trigger Control Register (0x8): 0x00001fff<br />
ROC Enable Register (0xc) val=0xf07fff7f:<br />
Branch 1: 0x7f bits: 01111111<br />
Branch 2: 0xff bits: 11111111<br />
Branch 3: 0x7f bits: 01111111<br />
Branch 4: 0xf0 bits: 11110000<br />
Synchronization Interval Register (0x10): 1000<br />
Trigger Word Count Register (0x14): 0<br />
Trigger Data Register (0x18): 256<br />
Local ROC (Branch 5) Data Register (0x1c): 60<br />
Synchronization Flag : 0<br />
Late Fail Flag : 0<br />
ROC code : 15<br />
Input Trigger Prescale Registers:<br />
Input 1 Prescale Factor : 0<br />
Input 2 Prescale Factor : 0<br />
Input 3 Prescale Factor : 0<br />
Input 4 Prescale Factor : 0<br />
Input 5 Prescale Factor : 0<br />
Input 6 Prescale Factor : 0<br />
Input 7 Prescale Factor : 0<br />
Input 8 Prescale Factor : 0<br />
<br />
Clear Permit Timer Register = 0<br />
Level2 Accept Timer Register = 83<br />
Level3 Accept Timer Register = 0<br />
Front Busy Timer Register = 325<br />
Clear Hold Timer Register = 50<br />
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040<br />
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0<br />
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086<br />
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0<br />
ROC Acknowledge Status Register (0x60): val=0x0<br />
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)<br />
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)<br />
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)<br />
State Register (0x6C):<br />
Level 1 Accept : 0<br />
Start Level 2 Trigger : 0<br />
Level 2 Pass Latched : 0<br />
Level 2 Fail Latched : 0<br />
Level 2 Accept : 0<br />
Start Level 3 Trigger : 0<br />
Level 3 Pass Latched : 0<br />
Level 3 Fail Latched : 0<br />
Level 3 Accept : 0<br />
Clear : 0<br />
Front End Busy (external) : 0<br />
External Inhibit : 0<br />
Latched Trigger : 0<br />
TS Busy : 1<br />
TS Active : 1<br />
TS Ready : 0<br />
Main Sequencer Active : 0<br />
Synchronization Sequencer Active : 0<br />
Program 1 Event Sequencer Active : 1<br />
Program 2 Event Sequencer Active : 0<br />
Event Count (0xc8): 135<br />
Live 1 Count (0xcc): 54866<br />
Live 2 Count (0xd0): 9290375<br />
value = 1 = 0x1<br />
-> exit<br />
<br />
clasprod::clastrig2> clastrig2 statistics<br />
200 0 0 0<br />
<br />
<br />
* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:<br />
<br />
interrupt: Program Events 1-2: 247 0<br />
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)<br />
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)<br />
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)<br />
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)<br />
wait: coda request >go< in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
roc_end reached ??<br />
roc_end reached !!<br />
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3<br />
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <<br />
interrupt: inputs 1-6: 0 0 0 0 0 0<br />
interrupt: inputs 7-12: 0 0 339 0 0 0<br />
interrupt: Program Events 1-2: 172 0<br />
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)<br />
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)<br />
wait: coda request >end< in progress<br />
0x18c6f5c0 (twork0003): TS GO bit cleared<br />
0x18c6f5c0 (twork0003): TS csr: 0xfc000000<br />
interrupt: TRIG ERROR: no pool buffer available<br />
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0<br />
0x18c6f5c0 (twork0003): INFO: User End Executed<br />
codaUpdateStatus: dbConnecting ..<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_standard_request >sta:clastrig2 ending<<br />
UDP_cancel: cancel >sta:clastrig2 active<<br />
codaUpdateStatus: updating request done<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
NOW !!!!!!!!!!!!!!!!!!!<br />
roc_end done<br />
codaExecute done<br />
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
coda_roc: process rol1 buffer by rol2 completed<br />
coda_roc: break loop<br />
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: in DA_ENDING last event=12313 nevents=12314<br />
coda_roc: rocp->state == ENDING and input list is empty<br />
<br />
<br />
<br />
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):<br />
.....<br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING'<br />
coda_roc: processing case 'DA_ENDING': last <br />
event=2405 nevents=2405<br />
coda_roc: processing case 'DA_ENDING'<br />
.....<br />
<br />
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:<br />
<br />
file[ 0]->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: splitted files handling ..<br />
bosopen.c: >/raid/stage_in/clas_062414.A00<<br />
bosopen.c: len1=30<br />
bosopen.c: len2=30<br />
bosopen.c: len3=14<br />
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14<br />
bosopen.c: checking how much disk space left in directory /raid/stage_in<br />
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available <br />
1100988411 blocks (1 block = 512 bytes)<br />
bosopen.c: so we have 550493696 kbytes available space<br />
bosopen.c: we can continue to write file.<br />
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-<br />
bosopen.c: system call: >checkdisk 4094<<br />
[1] 9253<br />
bosopen.c: system call completed<br />
codaUpdateStatus: dbConnecting ..<br />
ER: Not attached to ET system<br />
er_write_thread loop ended (0 145452664)<br />
codaUpdateStatus: dbConnect done<br />
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<<br />
codaUpdateStatus: dbDisconnecting ..<br />
codaUpdateStatus: dbDisconnect done<br />
codaUpdateStatus: updating request ..<br />
UDP_standard_request >sta:ER3 paused<<br />
UDP_standard_request >sta:ER3 paused<<br />
<br />
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.<br />
<br />
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Slow_Controls_Time_History&diff=4716
Slow Controls Time History
2010-03-19T22:07:23Z
<p>129.57.167.4: </p>
<hr />
<div>Produces Web-based timeline plots, can be viewed on http://clasweb.jlab.org/cgi-bin/epics_history/epics.pl.<br />
<br />
Resides in CVSROOT=/home/epics/cvs, normally checked out to /home/epics/history and running on clon01 as user ''epics''.<br />
<br />
Main script is:<br />
<br />
get_latest_cont.pl<br />
<br />
It reads EPICS channels and updates ''epics'' database on ''clondb1''. <br />
<br />
Cron job is checking if main script is running (if not it will try to restart it):<br />
<br />
0,15,30,45 * * * * (/bin/tcsh -c /home/epics/history/www/check_get.pl >> /dev/null)<br />
<br />
If started by cron job, log file ''/home/epics/history/log/get_latest_cont.log'' will be produced<br />
<font color=red><br />
(CURRENTLY GOES TO /dev/null !!!)<br />
</font><br />
<br />
Database ''epics'' on ''clondb1'' was creates as follwoing:<br />
<br />
mysql> create database epics;<br />
mysql> grant ALL on epics.* to marki@clon01.jlab.org identified by '********';<br />
mysql> grant ALL on epics2.* to marki@clon01.jlab.org identified by '********';<br />
mysql> grant SELECT,INSERT,UPDATE on epics.* to epics@clon01.jlab.org;<br />
mysql> grant SELECT,INSERT,UPDATE on epics2.* to epics@clon01.jlab.org;<br />
mysql> grant SELECT on epics.* to clasweb@claspc28.jlab.org;<br />
mysql> grant SELECT on epics2.* to clasweb@claspc28.jlab.org;<br />
<br />
Web software runs on clasweb machine. It can be found in /u/group/clas/www/clasweb/htbin/epics_history' directory, in particular file 'db_parms.pm' contains the name of database machine (in our case clondb1) and other related attributes.<br />
<br />
<br />
IMPORTANT: DUMPING FROM clasweb-bck:<br />
<br />
cd /home<br />
mysqldump -uroot -pe1tocome epics | gzip > /home/epics.sql.gz<br />
mysqldump: Can't get CREATE TABLE for table `hallb_sf_xy560_0_12_1min` (Can't open file: 'hallb_sf_xy560_0_12_1min.MYI'. (errno: 145))<br />
one table:<br />
mysqldump -uroot -pe1tocome epics hallb_sf_xy560_0_12_1hr > 1hr.sql<br />
(works)<br />
mysqldump -uroot -pe1tocome epics hallb_sf_xy560_0_12_1min > 1min.sql<br />
mysqldump: Can't get CREATE TABLE for table `hallb_sf_xy560_0_12_1min` (Can't open file: 'hallb_sf_xy560_0_12_1min.MYI'. (errno: 145))<br />
<br />
<br />
Following command produce error message but runs (took about 33 minutes):<br />
<br />
mysqldump -f -l --opt -uroot -pe1tocome epics | gzip > /home/epics.sql.gz<br />
mysqldump: Got error: 1016: Can't open file: 'hallb_sf_xy560_0_12_1min.MYI'. (errno: 145) when using LOCK TABLES<br />
mysqldump: Can't get CREATE TABLE for table `hallb_sf_xy560_0_12_1min` (Can't open file: 'hallb_sf_xy560_0_12_1min.MYI'. (errno: 145))<br />
mysqldump: Can't get CREATE TABLE for table `hallc_MeV_1min` (Can't open file: 'hallc_MeV_1min.MYI'. (errno: 145))<br />
mysqldump: Can't get CREATE TABLE for table `ioctorus1colonloadPercent_1min` (Can't open file: 'ioctorus1colonloadPercent_1min.MYI'. (errno: 145))<br />
<br />
Import into ''clondb1'': type following on ''clondb1'':<br />
<br />
cd /database<br />
scp clasweb-bck:/home/epics.sql.gz .<br />
gunzip epics.sql.gz<br />
mysql -u root -p epics < epics.sql<br />
<br />
(last command took about 19 minutes).<br />
<br />
<font color=red><br />
<br />
TODO:<br />
<br />
understand errors reported by 'mysqldump'; fix database if necessary<br />
<br />
when script 'get_latest_cont.pl' was started it produced the number of errors into log file<br />
'/home/epics/history/log/get_latest_cont.log1'; currently script is writing to /dev/null<br />
so log file can be used to see what was wrong<br />
<br />
</font><br />
<br />
'''NOTE''': password to slow control timeline to add or delete: ''arne''.</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Clasxt&diff=4715
Clasxt
2010-03-19T18:29:47Z
<p>129.57.167.4: </p>
<hr />
<div>'''Clasxt''' machines are diskless 'thin clients' from neoware, running newlinux. We have 5 those machines still in use:<br />
<br />
* clasxt23<br />
<br />
* clasxt33<br />
<br />
* clasxt35<br />
<br />
* clasxt37<br />
<br />
* clasxt39<br />
<br />
* clasxt40<br />
<br />
To reboot in single user mode do following:<br />
<br />
* Power on the EON and hold the left shift key down until you are shown a boot prompt<br />
* The boot prompt should display 'Software'<br />
* Enter ''Software -s ramdisk=12000'' and press enter<br />
* As the OS boots, press alt-F2 (possibly more than once) until you are greeted with a bash prompt<br />
<br />
<br />
<br />
'''IMPORTANT:'''<br />
<br />
On clon machines where clasxt's suppose to login, XDM server must be running. On Solaris 10 XDM server is running as part of ''dtlogin''. By default it has following options:<br />
<br />
clon00:/etc> ps -ef | grep dtlogin<br />
root 7799 1 0 17:25:38 ? 0:00 /usr/dt/bin/dtlogin -daemon -udpPort 0<br />
<br />
which means upd connection is not allowed. Looking into options will see following:<br />
<br />
clon00:/etc> svcprop svc:/application/graphical-login/cde-login | grep udpPort<br />
dtlogin/args astring \ -udpPort\ 0<br />
<br />
To change that use following command:<br />
<br />
clon00:/etc> svccfg -s svc:/application/graphical-login/cde-login setprop dtlogin/args=\"\"<br />
<br />
Now that option is gone:<br />
<br />
clon00:/etc> svcprop svc:/application/graphical-login/cde-login | grep udpPort<br />
clon00:/etc><br />
<br />
To place that option back type following:<br />
<br />
clon00:/etc> svccfg -s svc:/application/graphical-login/cde-login setprop dtlogin/args=\" -udpPort 0\"<br />
<br />
Restart dtlogin server:<br />
<br />
/etc/init.d/dtlogin stop<br />
/etc/init.d/dtlogin start<br />
<br />
Now it is running without upd restriction:<br />
<br />
clon00:/etc> ps -ef | grep dtlogin<br />
root 11503 1 0 17:30:46 ? 0:00 /usr/dt/bin/dtlogin -daemon<br />
<br />
<br />
In addition to that, to make fonts loading process work enable xfs service. Normally it will be enabled on the machines with monitors, but on servers like clon00 and clon10 it will be disabled by default:<br />
<br />
clon00:/etc> svcs -a | grep xfs<br />
disabled Jan_04 svc:/application/x11/xfs:default<br />
clon00:/etc><br />
<br />
so enable it:<br />
<br />
clon00:/etc> svcadm enable svc:/application/x11/xfs<br />
clon00:/etc> svcs | grep xfs<br />
online 12:55:53 svc:/application/x11/xfs:default<br />
clon00:/etc> <br />
<br />
<br />
<br />
<br />
<br />
Some useful information can be found in following paper:<br />
<br />
Some reports indicate that 3.x builds do not work. The error that is reported is<br />
FATAL ERROR<br />
The device that contains the configuration file (config.xml) could not be found. m0n0wall cannot continue booting.<br />
I do not yet know of a solution. This tutorial should be construed to apply only to the 2.x builds in the meantime. Please email me if you get 3.x working on an EON.<br />
<br />
Here's how to install the latest build of the excellent m0n0wall firewall on a Neoware EON 4000 thin client. These machines make excellent firewalls for home users and small businesses because they are cheap, compact, sturdy, and powerful enough to run a several megabit connection. The hardware is available in a variety of configurations, with CPUs roughly equivalent to a 150-200Mhz Pentium. Before embarking on this little journey, you should be aware of a few caveats:<br />
<br />
* Some Neoware boxes come with a 32-pin Disk-on-Chip module in a socket on the motherboard. m0n0wall does not ship with the required driver to run this module. Therefore, on these systems you will need a hard drive, compact flash to IDE adapter, or similar device to boot from. You may also be able to netboot your firewall (this is supported on the built-in network device), though I do not cover that in this tutorial.<br />
* The motherboard accepts both PC133 DIMMs or SO-DIMMs. It will use both at the same time. However, when using a DIMM, be sure it is less than 1.1" (28mm) tall, or you will not be able to install a second network card.<br />
* There are many different revisions of the NeoLinux operating system and not all include the necessary software to complete this tutorial.<br />
* You may need to purchase several EON 4000s to get suitable software and hardware.<br />
* These instructions were based on NeoLinux 2.4-100802, with kernel 2.2.20. Your particular configuration may vary slightly.<br />
<br />
With those potential issues in mind, follow along!<br />
<br />
There are three main steps to be completed to get m0n0wall running on your EON. First, you will need to get single-user access to the NeoLinux installation. Next, you need to enable networking and fetch and install m0n0wall. Third, you need to configure it to fit your requirements. I will address the first two of these, as adequate resources exist to help with the latter.<br />
<br />
To begin, you will need:<br />
<br />
* One or more Neoware EONs with disk-on-chip modules that plug into the IDE header<br />
* Access to the internet from your EON (or to a web server on a LAN)<br />
* A supported PCI network card. Common Intel or 3Com cards work fine.<br />
* At least 64MB of suitable memory. Some EONs ship with SO-DIMMs, others with DIMMs. These should be short enough to accomodate a PCI network card.<br />
* See the m0n0wall supported hardware page for more information.<br />
<br />
Step ONE: Get single-user access to the NeoLinux distribution.<br />
<br />
* Power on the EON and hold the left shift key down until you are shown a boot prompt<br />
* The boot prompt may vary, but pressing tab should display either 'Software' or 'Eon_Software'.<br />
* Enter, as appropriate, either<br />
Software -s ramdisk=12000<br />
or<br />
Eon_Software -s ramdisk=12000<br />
and press enter.<br />
* As the OS boots, press alt-F2 (possibly more than once) until you are greeted with a bash prompt.<br />
<br />
Step TWO: Fetch and install m0n0wall.<br />
<br />
* Check that your installation has the necessary software:<br />
bash# ls /usr/bin/wget<br />
. If NeoLinux reports that it is not found, you will have to try a different EON. Keep this hardware however: a single working EON can program m0n0wall onto several disk-on-chip modules, even those without the proper software. More on this later.<br />
* Plug the EON into a network which provides DHCP information and enable the ethernet connection:<br />
bash# cp /etc/sysconfig/network-scripts/ifcfg-eth0.default /etc/sysconfig/network-scripts/ifcfg-eth0<br />
bash# ifup eth0<br />
* Create a ramdisk on which to place temporary files during the installation:<br />
bash# mkfs.ext2 /dev/ram0<br />
bash# mount /dev/ram0 /mnt/usbcdrom<br />
bash# cd /mnt/usbcdrom<br />
bash# /usr/bin/wget http://hostname/path/to/m0n0wall<br />
bash# cp /bin/gunzip .<br />
bash# cp /bin/dd .<br />
* Now you're ready. You have created a ramdisk with the two utilities you will need to rewrite IDE disk-on-chip modules with m0n0wall. At this point, if you have modules containing copies of NeoLinux without wget, you should try using them first. That way if something goes wrong you can repeat these steps later. With the machine running, remove the IDE DOC that you booted from and replace it with the one you want to reprogram.<br />
* Then,<br />
gunzip -c m0n0wall-file-name.img | dd of=/dev/hda bs=16k<br />
Ignore the warning about trailing garbage.<br />
<br />
That's it! At this point you can remove the DOC and replace it with another one and repeat the reprogramming step, as many times as needed.<br />
<br />
When you next boot the machine, it should load m0n0wall exactly as it would on any other x86-compatible system. Jump right in with the m0n0wall quick-start guide for PC platforms. Chapter 3: Initial Configuration is probably where you will want to start. Good luck!<br />
<br />
Version 0.3 | Content date: 05 July 2008 | Page last generated: 2009-01-18 13:11 CST</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4714
ERROR Book
2010-03-19T00:37:47Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:<br />
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:<br />
7 9 10 11 17 20 22 25 26 28 29<br />
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???<br />
<br />
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=JLAB_Level2_Majority&diff=4712
JLAB Level2 Majority
2010-03-17T20:13:55Z
<p>129.57.167.4: </p>
<hr />
<div>Manual in [https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_Level2MajorityLogic_Manual.pdf (pdf)]</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=CLON_Index&diff=4711
CLON Index
2010-03-17T20:09:56Z
<p>129.57.167.4: </p>
<hr />
<div>[[Apache]]<br />
<br />
[[automounts]]<br />
<br />
[[Bertan MWPC]] High Voltage Power Supplies<br />
<br />
[[Bootp]]<br />
<br />
[[Cacti]]<br />
<br />
[[CAEN A464]] caenet board (installed in sy527 mainframes)<br />
<br />
[[CAEN SY1527 Mainframe]]<br />
<br />
[[CAEN SY527 Mainframe]] and modules<br />
<br />
[[CAEN V1190]] TDC board (128 ch 100 ps)<br />
<br />
[[CAEN V1290]] TDC board (32 ch 25 ps)<br />
<br />
[[CAEN V1495]] General Purpose VME Board<br />
<br />
[[CAEN V288]]<br />
<br />
[[CAEN V775]] TDC Board<br />
<br />
[[CAEN V895]] 16 ch. Leading Edge Discriminator with individual thresholds<br />
<br />
[[Capfast]]<br />
<br />
[[Cisco Switches]]<br />
<br />
[[Clasweb]]<br />
<br />
[[Clasxt]]<br />
<br />
[[Clon00]]<br />
<br />
[[Clon01]]<br />
<br />
[[Clon02]]<br />
<br />
[[Clon03]]<br />
<br />
[[Clon04]]<br />
<br />
[[Clon05]]<br />
<br />
[[Clon06]]<br />
<br />
[[Clon10]]<br />
<br />
[[Clon20]]<br />
<br />
[[Clondaq1]]<br />
<br />
[[Clondaq2]]<br />
<br />
[[Clondb1]]<br />
<br />
[[Clondb2]]<br />
<br />
[[Clonfs]] Main RAID System<br />
<br />
[[Clonfs1]]<br />
<br />
[[Clonfs2]]<br />
<br />
[[Clonmon]]<br />
<br />
[[Clonmon0]]<br />
<br />
[[Clonmon1]]<br />
<br />
[[Clonmon2]]<br />
<br />
[[Clonpc1]]<br />
<br />
[[Clonpc2]]<br />
<br />
[[Clonpc3]]<br />
<br />
[[Clonpc5]]<br />
<br />
[[Clonpc6]]<br />
<br />
[[Clonpc8]]<br />
<br />
[[Clonweb]]<br />
<br />
[[Clonwiki]]<br />
<br />
[[clonhp]]<br />
<br />
[[clonhp2]]<br />
<br />
[[Coda]]<br />
<br />
[[Compcontrol CC121]]<br />
<br />
[[CycladesTS1000]]<br />
<br />
[[DATEL DVME-628]]<br />
<br />
[[DNS server]]<br />
<br />
[[Electronics Documentation]]<br />
<br />
[[EtherLite32]]<br />
<br />
[[EPICS Base Installation]]<br />
<br />
[[EPICS: Generation of IOC with different options and customization for HallB]]<br />
<br />
[[EPICS: Software IOC]]<br />
<br />
[[EPICS: vxWorks IOC]]<br />
<br />
[[EPICS: Software IOC for CAEN HV]]<br />
<br />
[[EPICS: generation of DBs and GUIs for HV]]<br />
<br />
[[EPICS: CVS usage examples]]<br />
<br />
[[EPICS: Timeline Histories]]<br />
<br />
[[EPICS: Gamma Profiler]]<br />
<br />
[[Event Monitor]]<br />
<br />
[[Fiber Channel Switches]]<br />
<br />
[[FieldPoint]]<br />
<br />
[[Flash ADC Studies]]<br />
<br />
[[Foundry BigIron 15000 Switch]]<br />
<br />
[[Foundry Switches]]<br />
<br />
[[Helicity Signals from MCC]]<br />
<br />
[[HYTEC VSD2992]]<br />
<br />
[[JAVA installation]]<br />
<br />
[[JLAB Discriminators]]<br />
<br />
[[JLAB FLEXIO]]<br />
<br />
[[JLAB LF]] VME Latch FIFO Board<br />
<br />
[[JLAB LD]] VME Latch Driver Board<br />
<br />
[[JLAB Level2 Majority]] Logic Unit<br />
<br />
[[JLAB MLU]] VME Memory Lookup Unit (MLU)<br />
<br />
[[JLAB OR]] VME Programmable “OR” Module<br />
<br />
[[JLAB TS2]] VXI D-size Trigger Supervisor Module<br />
<br />
[[JLAB TI]] CODA Trigger Interface Cards<br />
<br />
[[JLAB PCI TI]] PCI Trigger Interface Card<br />
<br />
[[JLAB VME TI]] VME Trigger Interface Module<br />
<br />
[[Joerger VSC16]]<br />
<br />
[[KineticSystems 2917]]<br />
<br />
[[Labeling Machines]]<br />
<br />
[[Linux Installation Procedure]]<br />
<br />
[[Linux Customization on CLON Cluster]]<br />
<br />
[[Lnf cluster]]<br />
<br />
[[Mac OS X Installation Procedure]]<br />
<br />
[[Motorola MVME5500]]<br />
<br />
[[MRTG]]<br />
<br />
[[mSQL]]<br />
<br />
[[MySQL Installation]]<br />
<br />
[[MySQL Usage]]<br />
<br />
[[Nagios]]<br />
<br />
[[Nagios Plugins]]<br />
<br />
[[Net-SNMP]]<br />
<br />
[[NFS]]<br />
<br />
[[NIS server]]<br />
<br />
[[Nrpe]]<br />
<br />
[[Ntp]]<br />
<br />
[[OMS VME44]] by Oregon Micro Systems<br />
<br />
[[OMS VS4]] by Oregon Micro Systems<br />
<br />
[[OpenSSL]]<br />
<br />
[[PERL]]<br />
<br />
[[Procmail]]<br />
<br />
[[PrPMC]]<br />
<br />
[[raidold]]<br />
<br />
[[Routing]]<br />
<br />
[[Rrdtool]]<br />
<br />
[[SBS VIPC616]]<br />
<br />
[[Serial Connections from Computer Ports]]<br />
<br />
[[Sfs61]]<br />
<br />
[[Slow Controls Time History]]<br />
<br />
[[SmartSockets]]<br />
<br />
[[SNMP]]<br />
<br />
[[Software Documentation]]<br />
<br />
[[Solaris Installation Procedure]]<br />
<br />
[[Solaris Customization on CLON Cluster]]<br />
<br />
[[Solaris Patches]]<br />
<br />
[[SSH]]<br />
<br />
[[Struck SIS3801]]<br />
<br />
[[Sudo]]<br />
<br />
[[SUN Microsystems]]<br />
<br />
[[SYSTRAN VMESC5]]<br />
<br />
[[Tcl/Tk]]<br />
<br />
[[Tftp]]<br />
<br />
[[Troubleshooting printing]]<br />
<br />
[[UNIX Automount on CLON Cluster]]<br />
<br />
[[UPS (Uninterruptible Power Supplies)]]<br />
<br />
[[UPS CHB1]]<br />
<br />
[[UPS CHB2]]<br />
<br />
[[UPS Troubleshooting]]<br />
<br />
[[VI editor]]<br />
<br />
[[VMIC VMIVME-3122]]<br />
<br />
[[VMIC VMIVME-2120]]<br />
<br />
[[VMIC VMIVME-1129]]<br />
<br />
[[VMIC VMIVME-2232]]<br />
<br />
[[VMIC VMIVME-6015]]<br />
<br />
[[VMIC VMIVME-6016]]<br />
<br />
[[VNC]]<br />
<br />
[[WIENER]]<br />
<br />
[[Wiki]]<br />
<br />
[[Wiring]]<br />
<br />
[[X server]]<br />
<br />
[[Xycom DIO XVME-240]]<br />
<br />
[[Xycom XVME-564]]<br />
<br />
[[Xycom XVME-560]]<br />
<br />
[[ZFS]]</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4710
ERROR Book
2010-03-17T19:31:28Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up. <br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Stadis&diff=4709
Stadis
2010-03-16T23:27:37Z
<p>129.57.167.4: </p>
<hr />
<div>'''Stadis''' takes livetime data from scaler_server-produced file ''$CLON_PARMS/scalers/archive/scalers_clasprod_0<runnum>.txt''.</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=Stadis&diff=4708
Stadis
2010-03-16T23:25:20Z
<p>129.57.167.4: </p>
<hr />
<div>'''Stadis''' takes livetime data from scaler_server-produced file ''$CLON_PARMS/scalers/archive/scstatmon_clasprod_0<runnum>.txt''.</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=CLAS_Online_Software&diff=4707
CLAS Online Software
2010-03-16T23:22:28Z
<p>129.57.167.4: </p>
<hr />
<div>'''If you want information about DAQ/Online Systems startup and recover procedures, use following link:'''<br />
<br />
[[DAQ/Online Quick Reference]]<br />
<br />
[[DAQ/Online Software Stricture]]<br />
<br />
'''Expert Information:'''<br />
<br />
[[ADC Pedestals Measurement Procedure]]<br />
<br />
[[Alarm Browser]]<br />
<br />
[[Alarm Server]]<br />
<br />
[[dbrouter]]<br />
<br />
[[Epics Server]]<br />
<br />
[[ET System]]<br />
<br />
[[Event Monitor]]<br />
<br />
[[info_server]]<br />
<br />
[[Logbook]]<br />
<br />
[[msql database]]<br />
<br />
[[Process Management]]<br />
<br />
[[Recsis]]<br />
<br />
[[Stadis]]<br />
<br />
[[CMON]] CLAS Online Data Monitoring<br />
<br />
[[Runlog]] programs</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4706
ERROR Book
2010-03-15T02:22:58Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash<br />
<br />
* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4
https://clonwiki0.jlab.org/wiki/index.php?title=ERROR_Book&diff=4705
ERROR Book
2010-03-14T17:04:33Z
<p>129.57.167.4: </p>
<hr />
<div>* Sergey B. 14-mar-2010: during reboot dc9 gives following:<br />
.................<br />
ppc/bootscripts/boot_dc9<br />
taskSpawn("TCPSERVER") returns 475458320<br />
bind on port 5001<br />
myname >dc9<<br />
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
333333333333333<br />
program<br />
Exception current instruction address: 0x1c5<br />
e8734<br />
Machine Status Register: 0x00081000<br />
Condition Register: 0x40000085<br />
Task: 0x1c5e8e50 "ROC"<br />
interrupt: Unconnected main interrupt 0<br />
logTask: 3222 log messages lost.<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
interrupt: Unconnected main interrupt 1<br />
interrupt: Unconnected main interrupt 0<br />
..............<br />
<br />
probably mysql connection problem<br />
<br />
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing <br />
wait: coda request >exit< in progress<br />
indefinitely, must check<br />
<br />
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:<br />
<br />
codaEnd 10<br />
codaEnd 11<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<br />
codaExecute done<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59319<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
codaExecute: ERROR: len=0 - do nothing<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=59342<br />
wait: coda request >< in progress<br />
Executing >TP/1.0<br />
<br />
Probably last string contains something, will remove printing<br />
<br />
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:<br />
.....<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
case 0: no swap<br />
.. done.<br />
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) <br />
sent -1 (type 18)<br />
[0] ERROR: Discard data until next control event<br />
clondaq1:coda_eb><br />
<br />
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,<br />
reboot all rocs again - not it works.<br />
<br />
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...<br />
<br />
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:<br />
<br />
.....<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_standard_request >sta:EB1 booted<<br />
UDP_cancel: cancel >sta:EB1 booted<<br />
codaUpdateStatus: updating request done<br />
CODA_Init 14<br />
2<br />
clasprod::EB1> bind on port 5001<br />
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.28< port=37258<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
wait: coda request >< in progress<br />
Executing >< (len=0)<br />
Segmentation fault (core dumped)<br />
clondaq1:coda_eb> <br />
<br />
In the same time ''et_start'' complains (we have now ET system protected against CC scans):<br />
<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
et ERROR: et_netserver: ET server being probed by non-ET client or read failure<br />
<br />
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already<br />
<br />
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:<br />
<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e404190 "coda_pmc"<br />
<br />
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.<br />
<br />
* Sergey B. 1-sep-2009: during croctest1 reboot:<br />
<br />
INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)<br />
dbConnect ERROR: mysql == NULL<br />
<br />
second reboot went fine<br />
<br />
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?<br />
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.<br />
<br />
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:<br />
<br />
..........................<br />
taskDelay(sysClkRateGet()*5)<br />
Args = -session clasprod -objects sc2 ROC -i<br />
CODA_Init reached<br />
CODA_Init 1<br />
CODA_Init 2<br />
sc2<br />
CODA_Init 11<br />
CODA_Init: objectTy >(null)<<br />
CODA_Init 12<br />
CODA_Init: use 'SESSION' as >clasprod<<br />
Tcl_AppInit CALLS !!!!!<br />
11-11<br />
11-12<br />
11-13<br />
11111111111111111<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
value = 0 = 0x0<br />
taskSpawn "TCP_SERVER",250,0,100000,tcpServer<br />
value = 231091088 = 0xdc62b90<br />
proconhost<br />
value = 0 = 0x0<br />
......................<br />
<br />
Second reboot shows the same:<br />
<br />
...................<br />
22222222222222222<br />
0xdd0e880 (tNetTask): arptnew failed on 8139a743<br />
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24<br />
list >clon10 clasrun 2508 9998<<br />
machine name = clon10<br />
user ID = 2508<br />
group ID = 9998<br />
.................<br />
<br />
<br />
* Sergey B. around 6-June-2009: ec2pmc1 error message:<br />
<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: about to print: 255115 20<br />
write_thread: wait= 199 send= 8 microsec per event (nev=12755)<br />
CALLING customized :<br />
machine check <br />
SRAM ERROR :<br />
SRAM Error Address : 0x00000000.f2000504<br />
SRAM Error Data Low : 0x006805e0<br />
SRAM Error Data High : 0x00420000<br />
SRAM Error Parity : 0x00000004<br />
SRAM Error Cause : 0x00000001<br />
PCI_0 DEVICE ERROR :<br />
PCI_0 Error: Status=0x00000100<br />
Master abort<br />
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0<br />
Error Address High: 0x00000000<br />
Error Address Low : 0xf0d10128<br />
CALLING generic :<br />
machine check<br />
Exception next instruction address: 0x0001f058<br />
Machine Status Register: 0x0012b030<br />
Condition Register: 0x40222042<br />
Task: 0x1e3b0020 "coda_net"<br />
<br />
reboot did not helped, have to turn pmc off and run on host only<br />
<br />
<br />
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:<br />
<br />
0x1e42a360 (coda_pmc): bb_new: 'big' buffer <br />
created (addr=0x1eba3590, 16 bufs, 3145728 size)<br />
coda_pmc: big buffer1 allocated<br />
myname >ec1pmc1<<br />
Done executing startup script $CODA/VXWORKS_<br />
ppc/bootscripts/boot_pmc1<br />
-> INFO(mysql_real_connect9): errno=0<br />
INFO(mysql_real_connect9): OK<br />
program<br />
Exception current instruction address: 0x00000000<br />
Machine Status Register: 0x0008b030<br />
Condition Register: 0x40000085<br />
Task: 0x1e42a360 "coda_pmc"<br />
-> tt<br />
14cad8 vxTaskEntry +68 : coda_pmc ()<br />
1e2275c0 coda_pmc +144: mysql_query ()<br />
1e258520 mysql_query +54 : mysql_real_query ()<br />
1e249850 mysql_real_query+110: mysql_send_query ()<br />
1e249718 mysql_send_query+1d0: 0 ()<br />
value = 0 = 0x0<br />
-> <br />
<br />
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.<br />
<br />
<br />
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):<br />
<br />
write_thread: about to print: 394603 20<br />
write_thread: about to print: 394603 20<br />
write_thread: wait= 176 send= 6 microsec per event (nev=19730)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)<br />
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)<br />
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)<br />
bosMgetid: ERROR: lenn=0<br />
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
try again ..<br />
bosMgetid: ERROR: lenn=0<br />
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00<br />
bosMgetid: lenn=4, len=4<br />
bosMgetid: ERROR: name >RC-2< does not descr<br />
ibed, update clonbanks.ddl file !!!<br />
bosMgetid: lenn=4, len=4, nddl1=116<br />
bosMgetid: name=>RC-286331154<<br />
no way !!<br />
bosMlink: ERROR: bosMgetid returns -99<br />
<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)<br />
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)<br />
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)<br />
setHeartError: 0 >sys 0, mask 14<<br />
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<<br />
<br />
<br />
<br />
<br />
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:<br />
<br />
clon00:clasrun> ipc_info -a clasprod<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver<br />
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using local protocol<br />
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver<br />
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver<br />
08:08:31: TAL-SS-00089-I Using tcp protocol<br />
^C<br />
<br />
and it waits here. After restarting server on clondb1 by commands<br />
<br />
/etc/init.d/smartsockets stop<br />
/etc/init.d/smartsockets start<br />
<br />
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:<br />
<br />
top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64<br />
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie<br />
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si<br />
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers<br />
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached<br />
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br />
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x <br />
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init <br />
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0 <br />
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0 <br />
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1 <br />
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1 <br />
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2<br />
<br />
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.<br />
<br />
<br />
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.<br />
<br />
<br />
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':<br />
rcConfigure::loadRcDbaseCbk: Loading database failed !!!<br />
On second 'Configure' click it worked.<br />
<br />
* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!<br />
<br />
* Sergey B. 7-may-2008: EB1 crashed, last messages:<br />
<br />
EB: WARNING - resyncronization in crate controller number 5 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
EB: WARNING - resyncronization in crate controller number 6 ... fixed<br />
.<br />
ERROR: lfmt=0 bankid=0<br />
<br />
Found that tage2 and tage3 must be rebooted<br />
<br />
<br />
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that<br />
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything<br />
works fine<br />
<br />
<br />
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU<br />
<br />
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:<br />
<br />
#configuration for gamin<br />
# Can be used to override the default behaviour.<br />
# notify filepath(s) : indicate to use kernel notification<br />
# poll filepath(s) : indicate to use polling instead<br />
# fsset fsname method poll_limit : indicate what method of notification for the filesystem<br />
# kernel - use the kernel for notification<br />
# poll - use polling for notification<br />
# none - don't use any notification<br />
#<br />
# the poll_limit is the number of seconds<br />
# that must pass before a resource is polled again.<br />
# It is optional, and if it is not present the previous<br />
# value will be used or the default.<br />
fsset nfs poll 10<br />
# use polling on nfs mounts and poll once every 10 seconds<br />
# This will limit polling to every 10 seconds and seams to prevent it from running away<br />
<br />
Created file, will see if it helped ..<br />
<br />
<br />
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)<br />
<br />
.................................................<br />
write_thread: wait= 135 send= 28 microsec per event (nev=12351)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)<br />
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: about to print: 252148 20<br />
write_thread: wait= 135 send= 31 microsec per event (nev=12607)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)<br />
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)<br />
ERROR: bufout overflow - skip the rest ...<br />
bufout=485322144 hit=485387636 endofbufout=485387680<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: about to print: 255564 20<br />
write_thread: wait= 128 send= 42 microsec per event (nev=12778)<br />
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
.................................................. <br />
<br />
<br />
At the same time at sc2:<br />
<br />
......................................<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)<br />
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 220 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 692 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 544 log messages lost.<br />
..................................<br />
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums<br />
logTask: 562 log messages lost.<br />
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear<br />
logTask: 707 log messages lost.<br />
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear<br />
logTask: 517 log messages lost.<br />
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear<br />
logTask: 620 log messages lost.<br />
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear<br />
logTask: 559 log messages lost.<br />
interrupt: SYNC: scan_flag=0x00390000<br />
logTask: 512 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 671 log messages lost.<br />
......................................<br />
logTask: 563 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
logTask: 385 log messages lost.<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
wait: coda request in progress<br />
codaExecute reached, message >end<, len=3<br />
codaExecute: 'end' transition<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.......................................<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums<br />
interrupt: TRIGGER ERROR: no pool buffer available<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
.....................................<br />
<br />
Note: sc2 never warn about wrong slot number before that moment.<br />
<br />
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time<br />
<br />
<br />
* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:<br />
<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59063<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60187<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60295<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60418<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=60823<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=32961<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33122<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33795<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34529<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35811<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36134<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41569<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53913<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=57701<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33357<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=36136<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39509<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=39900<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45174<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=45909<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=34521<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=49526<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=53555<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55694<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=56833<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=58251<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=59023<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=33687<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=35731<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=51581<br />
wait: coda request in progress<br />
wait: coda request in progress<br />
Error(old rc): nRead=18, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=41371<br />
wait: coda request in progress<br />
Error(old rc): nRead=6, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=43198<br />
wait: coda request in progress<br />
Error(old rc): nRead=10, must be 1032<br />
CODAtcpServer: start work thread<br />
befor: socket=5 address>129.57.71.38< port=55260<br />
wait: coda request in progress<br />
Error(old rc): nRead=0, must be 1032<br />
clasprod::EB1> ^C<br />
<br />
Similar messages were coming from ER.<br />
<br />
ET reported following:<br />
<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure <br />
<br />
and another ET:<br />
.......<br />
et_start: pthread_create(0x0000000d,...) done<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
TCP server got a connection so spawn thread<br />
et ERROR: et_client_thread: read failure<br />
et INFO: et_sys_heartmonitor, kill bad process (2,3063)<br />
et INFO: et_sys_heartmonitor, cleanup process 2<br />
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2<br />
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments<br />
et INFO: set_fix_natts, station LEVEL3 has 1 attachments<br />
et INFO: set_fix_natts, station ET2ER has 1 attachments<br />
et INFO: set_fix_natts, # total attachments 2 -> 2<br />
et INFO: set_fix_natts, proc 0 has 1 attachments<br />
et INFO: set_fix_natts, proc 1 has 1 attachments <br />
<br />
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':<br />
<br />
Query test_ts2 table failed: Error reading table'test_ts2' definition<br />
ec3 ec3<br />
<br />
If one runcontrol is restarted, then it works, but another one complains !!!<br />
<br />
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.<br />
<br />
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters<br />
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window<br />
was not big enough; set to 100 and 900 respectively, will ask Jie<br />
<br />
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:<br />
(dbx) where<br />
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 <br />
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d <br />
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a <br />
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 <br />
[5] 0xfea60020(), at 0xfea60020 <br />
(dbx)<br />
<br />
* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far<br />
<br />
* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!<br />
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!<br />
end run failed. Reboot sc2.<br />
During end transition ec2 froze with message:<br />
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)<br />
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)<br />
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181<br />
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000<br />
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11<br />
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)<br />
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled<br />
Reboot ec2. Started new run 55463, everything looks normal.<br />
<br />
* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
Unknown error errno=65<br />
data taking continues, but runcontrol printed message:<br />
WARN : ec2 has not reported status for 1516 seconds<br />
ERROR : ec2 is in state disconnected should be active<br />
ec2pmc1 looked fine; end run failed, need to reboot ec2</div>
129.57.167.4