CLONWiki - User contributions [en]

MSQL

2008-06-13T18:11:08Z

129.57.121.200:

mSQL database is used by CODA to store configurations.
There is a plan to move to mySQL. To dump mSQL following
command was used (in release 2.1 only !!!):

msqldump clasrun > clasrun.sql
msqldump clasdev > clasdev.sql
mysql -h clondb1 -u clasrun daq_clasrun < clasrun.sql
mysql -h clondb1 -u clasrun daq_clasdev < clasdev.sql

NOTE: databases 'daq_clasrun' and 'daq_clasdev' must be created and user 'clasrun' must have an appropriate privileges to access those databases (normally 'ALL_PRIVILEGES'). It can be useful to re-create entire database before filling it up, for example:
mysql -h clondb1 -u clasrun
mysql> drop database daq_clasrun;
mysql> create database daq_clasrun;
mysql> exit

Observed problem: 'BONUS' and 'bonus' recognized by mySQL as identical key;
To fix it 'msqldump' was modified to use VARCHAR() BINARY instead of CHAR()
for primary keys. Some other changes were applied as well (see code).

MSQL

2008-06-11T16:27:46Z

129.57.121.200:

MSQL

2008-06-11T16:19:51Z

129.57.121.200:

mSQL database is used by CODA to store configurations.
There is a plan to move to mySQL. To dump mSQL following
command was used (in release 2.1 only !!!):

msqldump clasrun > clasrun.sql
mysql -h clondb1 -u clasrun daq_clasrun < clasrun.sql

NOTE: database 'daq_clasrun' must be created and have appropriate privileges.

Observed problem: 'BONUS' and 'bonus' recognized by mySQL as identical key;
To fix it 'msqldump' was modified to use VARCHAR() BINARY instead of CHAR()
for primary keys. Some other changes were applied as well (see code).

MSQL

2008-06-11T16:14:17Z

129.57.121.200:

mSQL database is used by CODA to store configurations.
There is a plan to move to mySQL. To dump mSQL following
command was used (in release 2.1 only !!!):

msqldump clasrun > msqldump.sql
mysql -h clondb1 -u clasrun daq_clasrun < msqldump.sql

Observed problem: 'BONUS' and 'bonus' recognized by mySQL as identical key;
To fix it 'msqldump' was modified to use VARCHAR() BINARY instead of CHAR()
for primary keys. Some other changes were applied as well (see code).

Electronics Documentation

2008-05-07T16:31:35Z

129.57.121.200:

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_NIM_LVL1FWDCarriage.pdf ADC Gate Electronics (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_FIFO_board.pdf JLAB VME FIFO Board (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_Latch_driver.pdf JLAB VME Latch Driver Board (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_MLU_board.pdf JLAB VME MLU Board (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_OR_board.pdf JLAB VME OR Board (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_ts2_board.doc JLAB VME TS2 Board (doc)]

Electronics Documentation

2008-05-07T16:31:14Z

129.57.121.200:

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_NIM_LVL1FWDCarriage.pdf ADC Gate Electronics (pdf)]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_FIFO_board.pdf JLAB VME FIFO Board (pdf)]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_Latch_driver.pdf JLAB VME Latch Driver Board (pdf)]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_MLU_board.pdf JLAB VME MLU Board (pdf)]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_OR_board.pdf JLAB VME OR Board (pdf)]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_VME_ts2_board.doc JLAB VME TS2 Board (doc)]

Electronics Documentation

2008-05-07T16:22:47Z

129.57.121.200:

[https://clonwiki.jlab.org/wiki/clondocs/Docs/JLAB_NIM_LVL1FWDCarriage.pdf ADC Gate Electronics]

Electronics Documentation

2008-05-07T16:20:34Z

129.57.121.200:

[https://clonwiki.jlab.org/]

CLON Index

2008-05-07T16:19:58Z

129.57.121.200:

[[Apache]]

[[Bootp]]

[[CAEN SY527 Mainframe]]

[[Capfast]]

[[Cisco Switches]]

[[Clasweb]]

[[Clon00]]

[[Clon01]]

[[Clon02]]

[[Clon03]]

[[Clon04]]

[[Clon05]]

[[Clon06]]

[[Clon10]]

[[Clondaq1]]

[[Clondaq2]]

[[Clondb1]]

[[Clonfs]] Main RAID System

[[Clonfs2]]

[[Clonmon]]

[[Clonmon1]]

[[Clonpc2]]

[[Clonweb]]

[[Clonwiki]]

[[clonhp]]

[[clonhp2]]

[[CycladesTS1000]]

[[DNS server]]

[[Electronics Documentation]]

[[EtherLite32]]

[[EPICS Base Installation]]

[[EPICS: Software IOC]]

[[EPICS: vxWorks IOC]]

[[EPICS: Software IOC for CAEN HV]]

[[EPICS: generation of DBs and GUIs for HV]]

[[EPICS: CVS usage examples]]

[[EPICS: Gamma Profiler]]

[[Fiber Channel Switches]]

[[Foundry BigIron 15000 Switch]]

[[Foundry Switches]]

[[JAVA installation]]

[[JLAB Discriminators]]

[[Labeling Machines]]

[[Linux Installation Procedure]]

[[Linux Customization on CLON Cluster]]

[[Lnf cluster]]

[[Mac OS X Installation Procedure]]

[[MRTG]]

[[mSQL]]

[[MySQL Installation]]

[[MySQL Usage]]

[[Nagios]]

[[Nagios Plugins]]

[[NFS]]

[[NIS server]]

[[Nrpe]]

[[Ntp]]

[[OpenSSL]]

[[PERL]]

[[Procmail]]

[[Routing]]

[[Rrdtool]]

[[Serial Connections from Computer Ports]]

[[Slow Controls Time History]]

[[SmartSockets]]

[[SNMP]]

[[Solaris Installation Procedure]]

[[Solaris Customization on CLON Cluster]]

[[Sudo]]

[[SUN Microsystems]]

[[Tcl/Tk]]

[[Tftp]]

[[Troubleshooting printing]]

[[UNIX Automount on CLON Cluster]]

[[UPS (Uninterruptible Power Supplies)]]

[[UPS CHB1]]

[[UPS CHB2]]

[[UPS Troubleshooting]]

[[VNC]]

[[WIENER]]

[[Wiki]]

[[Wiring]]

[[X server]]

[[ZFS]]

ERROR Book

2008-05-07T15:03:52Z

129.57.121.200:

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2008-05-07T14:58:25Z

129.57.121.200:

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2008-05-07T14:53:46Z

129.57.121.200:

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2008-04-30T21:30:47Z

129.57.121.200:

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

New CLAS Level1 Trigger System

2008-04-23T16:27:52Z

129.57.121.200:

During G12 run preparations on spring 2008 we decided to upgrade photon part of CLAS Level1 trigger system using CAEN v1495 logic unit. Eventually those efforts resulted in full Level1 system replacement: 6 level1 router boards and event processor board were replaced with one single v1495 unit equipped with 2 extra daughter modules. Trigger was commissioned during first phase of G12 run, last significant change was applied April 20, 2008. Most of the job was completed by Ben Raydo from Fast Electronics group. Valery Kubarovsky and Pawel Nadel-Turonski made generic trigger design and conducted trigger testing. Sergey Boyarinov, Ivan Bedlinskiy, Nerses Gevorgyan and Hovanes Egiyan integrated new hardware and software components into CLAS online and slow control systems. Technical assistance was provided by Fast Electronic Group and CODA Group personal. Current document contains basic information about the system.

* [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/V1495largekit.jpg boards pictures], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/layout.jpg general layout]

* v1495-based logic: see GUI snapshots below; total propagation time is about 75ns, similar to old Level1 system

* Windows-based control GUI (see [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_gui_1.jpg snapshot1], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_gui_2.jpg snapshot2] and [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_gui_3.jpg snapshot3]) were developed by Ben Raydo; it displays basic trigger logic and allows user to control and monitor various parts of the system using controlled logic elements, delays and lookup tables; built-in scalers are providing useful information about system functionality; system includes an multi-channel logic analyzer which was extremely useful to set appropriate delays (see [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_1.jpg snapshot1], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_2.jpg snapshot2], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_3.jpg snapshot3] and [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_4.jpg snapshot4])

* system incorporation into CLAS Online and Slow control: 9 OR modules were installed in sc2 crate, controlling script was provided to set appropriate masks into those units; v1495 was installed into clastrig2 trigger supervisor crate, and tcp server was developed and started under vxWorks to provide communication with Windows GUI and any other control and monitoring software running on UNIX machines; vxWorks timer-driven process is reading and reseting v1495 scalers every second placing data into semaphore-protected memory, so EPICS-related functions can extract necessary information from there; to provide EPICS support EPICS IOC was lunched along with CODA in clastrig2 crate, both systems are running together without problems (clastrig2 'i' command [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/clastrig2_i.txt snapshot]); EPICS GUIs were developed (see [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/ts_bits.png snapshot1] and [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/ts_st_view.png snapshot2]) to display trigger system scalers; Windows-based GUI was installed on clonwin2 desktop which was accessed using remote desktop software from clon04 Linux machine

April 23, 2008 online meeting minutes

2008-04-23T16:26:30Z

129.57.121.200:

present: Sergey Boyarinov, Sergey Pozdnyakov, David Abbott

1. G12 run finally have new v1495-based level1 trigger commissioned, it is running at 8kHz with 12% deadtime and 35MB/s data rate. Level2 is used as well. DAQ looks stable. EPICS histograms for new trigger system were reportedly implemented by Nerses, we'll activate them shortly.

New Scintillator Hodoscope ADCs, TDCs and discriminators were connected and programmed, still need to finisg scalers wiring. Data need to be analysed to make sure everything connected properly. Discriminators control is in 'dimanv', Sergey P. will send to Stepan short instruction howto use it.

Inventory database was presented on last Monday meeting. More improvements needed for front end, in particular the date of last change must be implemented, and search for the range of those dates will be useful. It is probably a good idea to make separate maintenance history table instead of ascii records.

2. We discussed our experience with [[New CLAS Level1 Trigger System]] implementation. David will create generic library for v1495 which can be used by other Halls personal. We need to create an documentation and provide basic training (if Ben have a time !) so people can use those boards for at least simple applications. Appropriate machine and lincensed software from Altera maybe necessary as well, although free versions (including simulation tools) can be used on Windows machines.

April 23, 2008 online meeting minutes

2008-04-23T16:21:40Z

129.57.121.200:

April 23, 2008 online meeting minutes

2008-04-23T16:20:51Z

129.57.121.200:

April 23, 2008 online meeting minutes

2008-04-23T16:18:26Z

129.57.121.200:

present: Sergey Boyarinov, Sergey Pozdnyakov, David Abbott

1. G12 run finally have new v1495-based level1 trigger commissioned, it is running at 8kHz with 12% deadtime and 35MB/s data rate. Level2 is used as well. DAQ looks stable. EPICS histograms for new trigger system were reportedly implemented by Nerses, we'll activate them shortly.

New Scintillator Hodoscope ADCs, TDCs and discriminators were connected and programmed, still need to finisg scalers wiring. Data need to be analysed to make sure everything connected properly. Discriminators control is in 'dimanv', Sergey P. will send to Stepan short instruction howto use it.

Inventory database was presented on last Monday meeting. More improvements needed for front end, in particular the date of last change must be implemented, and search for the range of those dates will be useful. It is probably a good idea to make separate maintenance history table instead of ascii records.

2. We discussed an experience with new v1495-based trigger system development and implementation, it can be found [[here]].

April 23, 2008 online meeting minutes

2008-04-23T16:13:57Z

129.57.121.200:

April 23, 2008 online meeting minutes

2008-04-23T16:12:39Z

129.57.121.200:

New CLAS Level1 Trigger System

2008-04-23T16:06:11Z

129.57.121.200:

During G12 run preparations on spring 2008 we decided to upgrade photon part of CLAS Level1 trigger system using CAEN v1495 logic unit. Eventually those efforts resulted in full Level1 system replacement: 6 level1 router boards and event processor board were replaced with one single v1495 unit equipped with 2 extra daughter modules. Trigger was commissioned during first phase of G12 run, last significant change was applied April 20, 2008. Most of the job was completed by Ben Raydo from Fast Electronics group. Valery Kubarovsky and Pawel Nadel-Turonski made generic trigger design and conducted trigger testing. Sergey Boyarinov, Ivan Bedlinskiy, Nerses Gevorgyan and Hovanes Egiyan integrated new hardware and software components into CLAS online and slow control systems. Technical assistance was provided by Fast Electronic Group and CODA Group personal. Current document contains basic information about the system.

* [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/V1495largekit.jpg boards pictures], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/layout.jpg general layout]

* v1495-based logic: see GUI snapshots below

* Windows-based control GUI (see [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_gui_1.jpg snapshot1], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_gui_2.jpg snapshot2] and [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_gui_3.jpg snapshot3]) were developed by Ben Raydo; it displays basic trigger logic and allows user to control and monitor various parts of the system using controlled logic elements, delays and lookup tables; built-in scalers are providing useful information about system functionality; system includes an multi-channel logic analyzer which was extremely useful to set appropriate delays (see [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_1.jpg snapshot1], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_2.jpg snapshot2], [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_3.jpg snapshot3] and [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/trig_display_4.jpg snapshot4])

* system incorporation into CLAS Online and Slow control: 9 OR modules were installed in sc2 crate, controlling script was provided to set appropriate masks into those units; v1495 was installed into clastrig2 trigger supervisor crate, and tcp server was developed and started under vxWorks to provide communication with Windows GUI and any other control and monitoring software running on UNIX machines; vxWorks timer-driven process is reading and reseting v1495 scalers every second placing data into semaphore-protected memory, so EPICS-related functions can extract necessary information from there; to provide EPICS support EPICS IOC was lunched along with CODA in clastrig2 crate, both systems are running together without problems (clastrig2 'i' command [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/clastrig2_i.txt snapshot]); EPICS GUIs were developed (see [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/ts_bits.png snapshot1] and [https://clonwiki.jlab.org/wiki/clondocs/V1495CLASTrigger/ts_st_view.png snapshot2]) to display trigger system scalers; Windows-based GUI was installed on clonwin2 desktop which was accessed using remote desktop software from clon04 Linux machine

April 16, 2008 online meeting minutes

2008-04-17T18:38:17Z

129.57.121.200:

present: Sergey Boyarinov, Chris Cuevas, Ben Raydo, Sergey Pozdnyakov

CLON meeting April 16, 2008

2008-04-17T18:37:31Z

129.57.121.200:

Clas On-line Meeting, Wednesday, April 16 at 10am in Room A110
------------------------------------------------------------------

1. CLAS12 electronics

CLON Group Meetings

2008-04-17T18:36:12Z

129.57.121.200:

[[CLON meeting December 13, 2006]] >>>>> [[December 13, 2006 online meeting minutes]]

==== 2007

[[CLON meeting September 5, 2007]] >>>>> [[September 5, 2007 online meeting minutes]]

[[CLON meeting September 12, 2007]] >>>>> [[September 12, 2007 online meeting minutes]]

[[CLON meeting September 19, 2007]] >>>>> [[September 19, 2007 online meeting minutes]]

[[CLON meeting September 26, 2007]] >>>>> [[September 26, 2007 online meeting minutes]]

[[CLON meeting October 3, 2007]] >>>>> [[October 3, 2007 online meeting minutes]]

[[CLON meeting October 10, 2007]] >>>>> [[October 10, 2007 online meeting minutes]]

[[CLON meeting October 17, 2007]] >>>>> [[October 17, 2007 online meeting minutes]]

[[CLON meeting October 24, 2007]] >>>>> [[October 24, 2007 online meeting minutes]]

[[CLON meeting October 31, 2007]] >>>>> [[October 31, 2007 online meeting minutes]]

[[CLON meeting November 7, 2007]] >>>>> [[November 7, 2007 online meeting minutes]]

[[CLON meeting November 14, 2007]] >>>>> [[November 14, 2007 online meeting minutes]]

[[CLON meeting November 28, 2007]] >>>>> [[November 28, 2007 online meeting minutes]]

[[CLON meeting December 12, 2007]] >>>>> [[December 12, 2007 online meeting minutes]]

==== 2008

[[CLON meeting January 16, 2008]] >>>>> [[January 16, 2008 online meeting minutes]]

[[CLON meeting January 23, 2008]] >>>>> [[January 23, 2008 online meeting minutes]]

[[CLON meeting January 30, 2008]] >>>>> [[January 30, 2008 online meeting minutes]]

[[CLON meeting February 13, 2008]] >>>>> [[February 13, 2008 online meeting minutes]]

[[CLON meeting March 5, 2008]] >>>>> [[March 5, 2008 online meeting minutes]]

[[CLON meeting March 12, 2008]] >>>>> [[March 12, 2008 online meeting minutes]]

[[CLON meeting March 26, 2008]] >>>>> [[March 26, 2008 online meeting minutes]]

[[CLON meeting April 16, 2008]] >>>>> [[April 16, 2008 online meeting minutes]]

MSQL

2008-03-11T16:38:42Z

129.57.121.200:

mSQL database is used by CODA to store configurations.
There is a plan to move to mySQL. To dump mSQL following
command was used:

msqldump clasrun > msqldump.sql
mysql -h clondb1 -u clasrun daq_clasrun < msqldump.sql

Observed problem: 'BONUS' and 'bonus' recognized by mySQL as identical key;
To fix it 'msqldump' was modified to use VARCHAR() BINARY instead of CHAR()
for primary keys. Some other changes were applied as well (see code).

MSQL

2008-03-11T16:08:49Z

129.57.121.200:

mSQL database is used by CODA to store configurations.
There is a plan to move to mySQL. To dimp mSQL following
command can be used:

msqldump -c clasrun > msqldump.sql
mysql -h clondb1 -u clasrun daq_clasrun < msqldump.sql

Observed problem: 'BONUS' and 'bonus' recognized by mySQL as identical key;
probably 'msqldump' must be modified to use VARCHAR() BINARY instead of CHAR().
Also, index is not defined ???

CLON Index

2008-03-11T16:05:10Z

129.57.121.200:

[[Apache]]

[[Bootp]]

[[CAEN SY527 Mainframe]]

[[Cisco Switches]]

[[Clasweb]]

[[Clon00]]

[[Clon01]]

[[Clon02]]

[[Clon03]]

[[Clon04]]

[[Clon05]]

[[Clon06]]

[[Clon10]]

[[Clondaq1]]

[[Clondaq2]]

[[Clondb1]]

[[Clonfs]] Main RAID System

[[Clonfs2]]

[[Clonmon]]

[[Clonmon1]]

[[Clonpc2]]

[[Clonweb]]

[[Clonwiki]]

[[clonhp]]

[[clonhp2]]

[[CycladesTS1000]]

[[DNS server]]

[[EtherLite32]]

[[EPICS Base Installation]]

[[EPICS: Software IOC]]

[[EPICS: vxWorks IOC]]

[[EPICS: Software IOC for CAEN HV]]

[[EPICS: generation of DBs and GUIs for HV]]

[[EPICS: CVS usage examples]]

[[EPICS: Gamma Profiler]]

[[Fiber Channel Switches]]

[[Foundry BigIron 15000 Switch]]

[[Foundry Switches]]

[[JAVA installation]]

[[JLAB Discriminators]]

[[Labeling Machines]]

[[Linux Installation Procedure]]

[[Linux Customization on CLON Cluster]]

[[Lnf cluster]]

[[Mac OS X Installation Procedure]]

[[MRTG]]

[[mSQL]]

[[MySQL Installation]]

[[MySQL Usage]]

[[Nagios]]

[[Nagios Plugins]]

[[NFS]]

[[NIS server]]

[[Nrpe]]

[[Ntp]]

[[OpenSSL]]

[[PERL]]

[[Procmail]]

[[Routing]]

[[Rrdtool]]

[[Serial Connections from Computer Ports]]

[[Slow Controls Time History]]

[[SmartSockets]]

[[SNMP]]

[[Solaris Installation Procedure]]

[[Solaris Customization on CLON Cluster]]

[[Sudo]]

[[SUN Microsystems]]

[[Tcl/Tk]]

[[Tftp]]

[[Troubleshooting printing]]

[[UNIX Automount on CLON Cluster]]

[[UPS (Uninterruptible Power Supplies)]]

[[UPS CHB1]]

[[UPS CHB2]]

[[UPS Troubleshooting]]

[[VNC]]

[[WIENER]]

[[Wiki]]

[[Wiring]]

[[X server]]

[[ZFS]]