CLONWiki - User contributions [en]

DAQ/Online Quick Reference

2012-02-17T22:29:14Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey '''office 5795 cell 757-869-2188 (if no responce, home 757-988-0680)'''

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Call DAQ pager..

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
office 5795
US cell 757-869-2188
(if no response, home 757-988-0680)
Moscow cell phone: 9-011-7-903-190-2196 (January 11, 2012)
Moscow cell phone: 9-011-7-916-729-8203 (*** old one ***)
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2012-01-11T15:08:34Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey '''office 5795 cell 757-869-2188 (if no responce, home 757-988-0680)'''

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
office 5795
US cell 757-869-2188
(if no response, home 757-988-0680)
Moscow cell phone: 9-011-7-903-190-2196 (January 11, 2012)
Moscow cell phone: 9-011-7-916-729-8203 (*** old one ***)
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

ZFS

2012-01-06T16:06:32Z

129.57.167.14:

'''RAID partitions on clondaq2'''

Four zfs partitions made on clonfs raid system connected to clondaq2: raid0, raid1, raid2 and raid3.

NOTE: at some point (after machine was renamed from clonxt3 and several reboots) raid3 dissapeared. When I tried to created it said it exist, but mount did not worked. Maybe it happened when I was playing with the same partition from clondaq1. Anyway, I created raid3 again, will watch it ...

To create ''raid0-raid3'' on clondaq2:

zpool create -f raid0 c0t600C0FF00000000009839205A879C400d0s2
zpool create -f raid1 c0t600C0FF0000000000983926ED5760700d0s2

zpool create -f raid2 c0t600C0FF000000000098392374595CB00d0s2
zpool create -f raid3 c0t600C0FF00000000009839252F7ADDC00d0s2

#zpool create -f raid2 c0t600C0FF00000000009839260D9481700d0s2
#zpool create -f raid3 c0t600C0FF0000000000983927F9F306900d0s2

#zpool create -f raid2 c0t600C0FF00000000009839223AFDC0A00d0s2
#zpool create -f raid3 c0t600C0FF0000000000983920939EB6300d0s2

zfs set mountpoint=/mnt/raid0 raid0
zfs set mountpoint=/mnt/raid1 raid1
zfs set mountpoint=/mnt/raid2 raid2
zfs set mountpoint=/mnt/raid3 raid3

To destroy for example ''raid3'' on clondaq2:

zfs umount /mnt/raid3
zpool destroy raid3

To display the list of all zfs partitions:

zfs list

NOTE: After reboot zfs partitions maybe (will be ?) unmounted, for example on clon10 'zfs list' returns following:

NAME USED AVAIL REFER MOUNTPOINT
data 134G 214G 134G /data
data1 131K 182G 49K /data1

but ''/data'' and ''/data1'' do not exists or empty. To mount those two partitions, do following as 'root':

zfs mount data
zfs mount data1

'''RAIDOLD on clon10'''

mkdir /raidold
zpool create -f raidold \
c1t32d0 c1t33d0 c1t34d0 c1t35d0 c1t36d0 c1t37d0 c1t38d0 c1t39d0 c1t40d0 c1t41d0 c1t42d0 \
c1t48d0 c1t49d0 c1t50d0 c1t51d0 c1t52d0 c1t53d0 c1t54d0 c1t55d0 c1t56d0 c1t57d0 c1t58d0 \
c1t64d0 c1t65d0 c1t66d0 c1t67d0 c1t68d0 c1t69d0 c1t70d0 c1t71d0 c1t72d0 c1t73d0 c1t74d0 \
c1t80d0 c1t81d0 c1t82d0 c1t83d0 c1t84d0 c1t85d0 c1t86d0 c1t87d0 c1t88d0 c1t89d0 c1t90d0
zfs set mountpoint=/raidold raidold

'''extra disks on clons'''

clon01:
zpool create -f space c1t1d0
zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 17.9G 49.0G 94K /rpool
rpool/ROOT 8.94G 49.0G 18K legacy
rpool/ROOT/s10s_u6wos_07b 8.94G 49.0G 6.42G /
rpool/ROOT/s10s_u6wos_07b/var 2.53G 49.0G 2.53G /var
rpool/dump 1.00G 49.0G 1.00G -
rpool/export 38K 49.0G 20K /export
rpool/export/home 18K 49.0G 18K /export/home
rpool/swap 8G 57.0G 24K -
space 89.5K 66.9G 1K /space

To change swap size for example, do following:

zfs set volsize=4G rpool/swap

To add new disk to the existing zfs pool, do following:

iostat -E (shows all disks available)
zpool add rpool ...

'''Hardware Maintenance'''

To check disks status, run:
zpool status

To offline a failing drive, run:
zpool offline pool-name disk-name
(A -t flag allows the disk to come back online after a reboot.)

Unconfigure disk. For example if disk name is 'c0t1d0', run:
cfgadm -c unconfigure c0::dsk/c0t1d0

On sfs61: Open a cover and look for the disk with blue led on. Replace it with new disk. '''Do not keep cover opened more then 90 sec !'''.

Once the drive has been physically replaced, run the replace command against the device:
zpool replace pool-name device-name
After an offlined drive has been replaced, it can be brought back online:
zpool online pool-name disk-name

It is good idea to run:
zpool scrub pool-name

Firmware upgrades may cause the disk device ID to change. ZFS should be able to update the device ID automatically, assuming that the disk was not physically moved during the update. If necessary, the pool can be exported and re-imported to update the device IDs.

'''from the web'''

Useful link: http://www.princeton.edu/~unix/Solaris/troubleshoot/zfs.html

Ok, ZFS is now in the tree, what's now? Below you'll find some
instructions how to quickly make it up and running.

First of all you need some disks. Let's assume you have three spare SCSI
disks: da0, da1, da2.

Add a line to your /etc/rc.conf to start ZFS automatically on boot:

# echo 'zfs_enable="YES"' >> /etc/rc.conf

Load ZFS kernel module, for the first time by hand:

# kldload zfs.ko

Now, setup one pool using RAIDZ:

# zpool create tank raidz da0 da1 da2

It should automatically mount /tank/ for you.

Ok, now put /usr/ on ZFS and propose some file systems layout. I know
you probably have some files already, so we will work on /tank/usr
directory and once we ready, we will just change the mountpoint to /usr.

# zfs create tank/usr

Create ports/ file system and enable gzip compression on it, because
most likely we will have only text files there. On the other hand, we
don't want to compress ports/distfiles/, because we keep compressed
stuff already in-there:

# zfs create tank/usr/ports
# zfs set compression=gzip tank/usr/ports
# zfs create tank/usr/ports/distfiles
# zfs set compression=off tank/usr/ports/distfiles

(You do see how your life is changing, don't you?:))

Let's create home file system, my own home/pjd/ file system. I know we
use RAIDZ, but I want to have directory where I put extremly important
stuff, you I'll define that each block has to be stored in tree copies:

# zfs create tank/usr/home
# zfs create tank/usr/home/pjd
# zfs create tank/usr/home/pjd/important
# zfs set copies=3 tank/usr/home/pjd/important

I'd like to have directory with music, etc. that I NFS share. I don't
really care about this stuff and my computer is not very fast, so I'll
just turn off checksumming (this is only for example purposes! please,
benchmark before doing it, because it's most likely not worth it!):

# zfs create tank/music
# zfs set checksum=off tank/music
# zfs set sharenfs=on tank/music

Oh, I almost forget. Who cares about access time updates?

# zfs set atime=off tank

Yes, we set it only on tank and it will be automatically inherited by
others.

Will be also good to be informed if everything is fine with our pool:

# echo 'daily_status_zfs_enable="YES"' >> /etc/periodic.conf

For some reason you still need UFS file system, for example you use ACLs
or extended attributes which are not yet supported by our ZFS. If so,
why not just use ZFS to provide storage? This way we gain cheap UFS
snapshots, UFS clones, etc. by simply using ZVOLs.

# zfs create -V 10g tank/ufs
# newfs /dev/zvol/tank/ufs
# mount /dev/zvol/tank/ufs /ufs

# zfs snapshot tank/ufs at 20070406
# mount -r /dev/zvol/tank/ufs at 20070406 /ufs20070406

# zfs clone tank/ufs at 20070406 tank/ufsok
# fsck_ffs -p /dev/zvol/tank/ufsok
# mount /dev/zvol/tank/ufsok /ufsok

Want to encrypt your swap and still use ZFS? Nothing more trivial:

# zfs create -V 4g tank/swap
# geli onetime -s 4096 /dev/zvol/tank/swap
# swapon /dev/zvol/tank/swap.eli

Trying to do something risky with your home? Snapshot it first!

# zfs snapshot tank/home/pjd at justincase

Turns out it was more stupid than risky? Rollback your snapshot!

# zfs rollback tank/home/pjd at justincase
# zfs destroy tank/home/pjd at justincase

Ok, everything works, we may set tank/usr as our real /usr:

# zfs set mountpoint=/usr tank/usr

Don't forget to read zfs(8) and zpool(8) manual pages and SUN's ZFS
administration guide:

http://www.opensolaris.org/os/community/zfs/docs/zfsadmin.pdf

--

======================================== from the web =============================================

ZFS and Containers: An Example
This section is a step-by-step guide that shows how to perform certain ZFS file system tasks inside of Solaris Containers; for example, taking snapshots and managing data compression. It does this by going through the following steps:

* Creating a zpool
* Creating a Zone
* Allocating a ZFS File System to a Zone
* Creating New File Systems
* Applying Quota to the File Systems
* Changing the Mountpoint of a File System
* Setting the Compression Property
* Taking a Snapshot

Each of these steps is described in detail below.

Creating a zpool
ZFS uses device names or partition names when dealing with pools and devices. For a device, this will be something like c1t0d0 (for a SCSI device) or c1d1 (for an IDE device). For a partition, it will be something like c1t0d0s0 (for a SCSI device) or c1d1s0 (for an IDE device). This example creates a pool that is mirrored using two disks.

1. To create a zpool in the global zone, use the zpool create command. Typically, you use two devices to provide redundancy.

Global# zpool create mypool mirror c2t5d0 c2t6d0

Note that the zpool create command may fail if the devices are in use or contain some types of existing data (e.g. UFS file system). If they are in use, you will need to unmount them or otherwise stop using them. If they contain existing data you can use the -f(force) flag to override the safety check, but be sure that you are not destroying any data you want to retain.
2. Examine the pool properties using the zpool list command.

Global# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool

199G

164K

199G

0%

online

--
This shows you that there is one zpool, named mypool,with a capacity of 199GBytes.

Creating a Zone
To show ZFS working in an environment that is isolated from other applications on the system, you need to create a zone. To create a zone:

1. Create a directory where the zone file system will reside, using the mkdir command. Be sure to chose a location where the filesystem has at least 80MBytes of available space.

Global# mkdir /zones

Note that in this example, for the sake of brevity, the root file system of the zone is a UFS file system.

2. Configure the zone (myzone), using the zonecfg command, and specify the location of the zone's files (/zones/myzone). Use the following series of commands.

Global# zonecfg -z myzone
myzone: No such zone configured
Use 'create' to begin configuring a new zone
zonecfg:myzone< create
zonecfg:myzone< set zonepath=/zones/myzone
zonecfg:myzone< verify
zonecfg:myzone< commit
zonecfg:myzone< exit

Again, for the purpose of streamlining, this example uses a very minimal zone. For more details on creating zones see the Solaris Containers How To Do Stuff guide at: sun.com/software/solaris/howtoguides/containersLowRes.jsp

3. Install the zone by using the zoneadm.

Global# zoneadm -z myzone install
Preparing to install zone >myzone<
[Output from zoneadm, this may take a few mins]

4. Boot the zone to complete the installation, using the zoneadm command.

Global# zoneadm -z myzone boot

5. Use the zlogin command to connect to the zone console.

Global# zlogin -C myzone
[Connected to zone 'myzone' console]
[Initial zone boot output, service descriptions are loaded etc.]
It may take a short while for the first boot to configure everything, load all the service descriptors, and so on. You will need to answer the system configuration details. Some suggestions are:

Terminal=(12)X Terminal Emulator (xterms)
Not Networked
No Kerberos
Name service = None
Time Zone = your-time-zone
root passwd = (Your choice—remember it though!)

The zone will reboot after you have provided the configuration information.

6. Before you can proceed to the next stage, the configured zone needs to be shutdown (configuration changes are only applied when the zone boots).

Global# zlogin myzone init 5

Allocating a ZFS File System to a Zone
Now that you have a zpool (mypool) and a zone (myzone) you are ready to allocate a ZFS file system to the zone.

1. To create a ZFS file system, use the zfs create command.

Global# zfs create mypool/myzonefs

2. To apply a quota to the file system, use the zfs set quota command.

Global# zfs set quota=5G mypool/myzonefs

The file system and all of its child file systems can be no larger than the designated quota. Note that both these steps must be performed in the global zone. Also notice that creating the file system in ZFS is much simpler than with a traditional file system/volume manager combination.

To illustrate the isolation/security properties of containers with ZFS this example now creates a ZFS file system that will remain outside the container. There is no need to apply a quota to this outside file system.

3. To create this other file system, again use the zfs create command.

Global# zfs create mypool/myfs

4. To show the properties of all the pool and the file systems, use the zfs list command.

Global# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

396G

197G

99.5K

/mypool
mypool/myfs

98.5K

197GK

98.5K

/mypool/myfs
mypool/myzonefs

98.5K

5G

98.5K

/mypool/myzonefs

To make the file system (myzonefs) available in the zone (myzone), the zone configuration needs to be updated.

5. To update the zone configuration, use the zonecfg command.

Global# zonecfg -z myzone
zonecfg:myzone> add dataset
zonecfg:myzone:dataset> set name=mypool/myzonefs
zonecfg:myzone:dataset> end
zonecfg:myzone> commit
zonecfg:myzone> exit

The mypool/myzonefs file system is now added to the zone configuration. Note that these steps must be performed with the zone shut down, otherwise the zone configuration changes would not be visible until the next reboot. To check that the zone is shut down try logging into it using zlogin myzone. If the zone is shut down the login will fail; if the zone is running you will see a login prompt—login as root and shut the zone down with init 5. These steps are performed in the global zone.

6. Now boot the zone.

Global# zoneadm -z myzone boot

7. Log in to the zone. (Leave a few seconds for the zone to boot.)

Global# zlogin -C myzone
[Connected to zone 'myzone' pts/3]
[Usual Solaris login sequence]

8. List the ZFS file systems in the zone.

NAME USED AVAIL REFER MOUNTPOINT
mypool

0M

200B

--

/mypool
mypool/myzonefs

8K

5G

8K

/mypool/myzonefs

Note the 5GByte maximum available from the external quota and that the other file systems in the pool (mypool/myfs) are not visible. This demonstrates the isolation property that Containers provide.

Creating New File Systems
Administering ZFS file systems from the non-global zone is done just like it is in the global zone, although you are limited to operating within the file system that is allocated to the zone (mypool/myzonefs). New ZFS file systems are always created as a child of this file system because this is the only ZFS file system the non-global zone can see. The administrator in the non-global zone can create these file systems. There is no need to involve administrator of the global zone, though the administer could do so if it were necessary.

1. To create a newfile system, use the zfs create command.

MyZone# zfs create mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

594M

197G

99K

/mypool
mypool/myzonefs

197K

5.00G

98.5K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

5.00G

98.5K

/mypool/myzonefs/tim

The non-global zone administrator can create as many child file systems as s/he wants and each child file system can have its own file systems, and in that way form a hierarchy.

As a demonstration that the non-global zone administrator is limited to the assigned file systems, this example demonstrates trying to break security by creating a file system outside the container's "space".

2. Try to create another file system outside of mypool/myzonefs, using the zfs create command.

MyZone# zfs create mypool/myzonefs1
cannot create 'mypool/myzonefs1': permission denied

As you can see, ZFS and zones security denies permission for the non-global zone to access resources it has not been allocated and the operation fails.

Applying Quota to the File Systems
Typically, to prevent the user consuming all of the space, a non-global zone administrator will want to apply a quota to the new file system. Of course, the child's quota can't be more than 5GByte as that's the quota specified by the global zone administrator to all of the file systems below mypool/myzonefs.

1. To set a quota on our new file system, use the zfs set quota command.

MyZone# zfs set quota=1G mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

508M

197G

99K

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

1024M

98.5K

/mypool/myzonefs/tim

The administrator of the non-global zone has set the quota of the child file system to be 1G. They have full authority to do this because they are operating on their delegated resources and do not need to involve the global zone administrator.

The ZFS property inheritance mechanism applies across zone boundaries, so the non-global zone administrator can specify his/her own property values should s/he wish to do so. As with normal ZFS property inheritance, these override inherited values.

Changing the Mountpoint of a File System
Now that the file system is set up and has the correct quota assigned to it, it is ready for use. However, the place where the file system appears (the mountpoint) is partially dictated by what the global zone administrator initially chose as the pool name (in this example, mypool/myzonefs). But typically, a non-global zone administrator would want to change it.

1. To change the mountpoint, use the zfs set mountpoint command.

MyZone# zfs set mountpoint=/export/home/tim mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

508M

197G

99K

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

1024M

98.5K

/export/home/tim

Note that the mountpoint can be changed for any file system independently.

Setting the Compression Property
The next example demonstrates the compression property. If compression is enabled, ZFS will transparently compress all of the data before it is written to disk.

The benefits of compression are both saved disk space and possible write speed improvements.

1. To see what the current compression setting is, use the zfs get command.

MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim

NAME PROPERTY VALUE SOURCE
mypool

compression

off

default
mypool/myzonefs

compression

off

default
mypool/myzonefs/tim

compression

off

default

Beaware that the compression property on the pool is inherited by the file system and child file system. So if the non- global zone administrator sets the compression property for the delegated file system, it will set it for everything below, as well.

2. To set the compression for the file system, use the zfs set command.

MyZone# zfs set compression=on mypool/myzonefs

3. Examine the compression property again in the non-global zone.

MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim

NAME PROPERTY VALUE SOURCE
mypool

compression

off

default
mypool/myzonefs

compression

on

local
mypool/myzonefs/tim

compression

off

Inherited from mypool/myzonefs

Note the compression property has been inherited by mypool/myzonefs/tim as with normal ZFS administration.

Taking a Snapshot
One of the major advantages of ZFS is the ability to create an instant snapshot of any file system. By delegating a file system to a non-global zone this feature becomes available as an option for the non-global zone administrator.

1. To take a snapshot named "1st" of the file system, use the zfs snapshot command.

MyZone# zfs snapshot mypool/myzonefs@1st
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

512K

99K

default

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs@1st

OK

--

99K

--
mypool/myzonefs/tim

98.5K

1024M

98.5K

/export/home/tim

As with ZFS file systems in the global zone, this snapshot is now accessible from the root of the file system in .zfs/snapshot/1st.

Back To Top

Summary
Once a zone has been created and a ZFS file system has been allocated to it, the administrator for that (non-global) zone can create file systems, take snapshots, create clones, and perform all the other functions of an administrator—within that zone. Yet the global zone, and any other zones, are fully isolated from whatever happens in that zone.

The integration of Solaris Containers and Solaris ZFS is just another way that the Solaris 10 OS is providing cost benefits to customers by allowing them to safely consolidate applications and more easily manage the data those applications use.

ZFS

2012-01-06T15:54:12Z

129.57.167.14:

'''RAID partitions on clondaq2'''

Four zfs partitions made on clonfs raid system connected to clondaq2: raid0, raid1, raid2 and raid3.

NOTE: at some point (after machine was renamed from clonxt3 and several reboots) raid3 dissapeared. When I tried to created it said it exist, but mount did not worked. Maybe it happened when I was playing with the same partition from clondaq1. Anyway, I created raid3 again, will watch it ...

To create ''raid0-raid3'' on clondaq2:

zpool create -f raid0 c0t600C0FF00000000009839205A879C400d0s2
c0t600C0FF00000000009839205A879C400d0s2

zpool create -f raid1 c0t600C0FF0000000000983926ED5760700d0s2
c0t600C0FF0000000000983926ED5760700d0s2

zpool create -f raid2 c0t600C0FF00000000009839260D9481700d0s2
c0t600C0FF00000000009839260D9481700d0s2

zpool create -f raid3 c0t600C0FF0000000000983927F9F306900d0s2
c0t600C0FF0000000000983927F9F306900d0s2

#zpool create -f raid2 c0t600C0FF00000000009839223AFDC0A00d0s2
c0t600C0FF00000000009839223AFDC0A00d0s2

#zpool create -f raid3 c0t600C0FF0000000000983920939EB6300d0s2
c0t600C0FF0000000000983920939EB6300d0s2

zfs set mountpoint=/mnt/raid0 raid0
zfs set mountpoint=/mnt/raid1 raid1
zfs set mountpoint=/mnt/raid2 raid2
zfs set mountpoint=/mnt/raid3 raid3

To destroy for example ''raid3'' on clondaq2:

zfs umount /mnt/raid3
zpool destroy raid3

To display the list of all zfs partitions:

zfs list

NOTE: After reboot zfs partitions maybe (will be ?) unmounted, for example on clon10 'zfs list' returns following:

NAME USED AVAIL REFER MOUNTPOINT
data 134G 214G 134G /data
data1 131K 182G 49K /data1

but ''/data'' and ''/data1'' do not exists or empty. To mount those two partitions, do following as 'root':

zfs mount data
zfs mount data1

'''RAIDOLD on clon10'''

mkdir /raidold
zpool create -f raidold \
c1t32d0 c1t33d0 c1t34d0 c1t35d0 c1t36d0 c1t37d0 c1t38d0 c1t39d0 c1t40d0 c1t41d0 c1t42d0 \
c1t48d0 c1t49d0 c1t50d0 c1t51d0 c1t52d0 c1t53d0 c1t54d0 c1t55d0 c1t56d0 c1t57d0 c1t58d0 \
c1t64d0 c1t65d0 c1t66d0 c1t67d0 c1t68d0 c1t69d0 c1t70d0 c1t71d0 c1t72d0 c1t73d0 c1t74d0 \
c1t80d0 c1t81d0 c1t82d0 c1t83d0 c1t84d0 c1t85d0 c1t86d0 c1t87d0 c1t88d0 c1t89d0 c1t90d0
zfs set mountpoint=/raidold raidold

'''extra disks on clons'''

clon01:
zpool create -f space c1t1d0
zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 17.9G 49.0G 94K /rpool
rpool/ROOT 8.94G 49.0G 18K legacy
rpool/ROOT/s10s_u6wos_07b 8.94G 49.0G 6.42G /
rpool/ROOT/s10s_u6wos_07b/var 2.53G 49.0G 2.53G /var
rpool/dump 1.00G 49.0G 1.00G -
rpool/export 38K 49.0G 20K /export
rpool/export/home 18K 49.0G 18K /export/home
rpool/swap 8G 57.0G 24K -
space 89.5K 66.9G 1K /space

To change swap size for example, do following:

zfs set volsize=4G rpool/swap

To add new disk to the existing zfs pool, do following:

iostat -E (shows all disks available)
zpool add rpool ...

'''Hardware Maintenance'''

To check disks status, run:
zpool status

To offline a failing drive, run:
zpool offline pool-name disk-name
(A -t flag allows the disk to come back online after a reboot.)

Unconfigure disk. For example if disk name is 'c0t1d0', run:
cfgadm -c unconfigure c0::dsk/c0t1d0

On sfs61: Open a cover and look for the disk with blue led on. Replace it with new disk. '''Do not keep cover opened more then 90 sec !'''.

Once the drive has been physically replaced, run the replace command against the device:
zpool replace pool-name device-name
After an offlined drive has been replaced, it can be brought back online:
zpool online pool-name disk-name

It is good idea to run:
zpool scrub pool-name

Firmware upgrades may cause the disk device ID to change. ZFS should be able to update the device ID automatically, assuming that the disk was not physically moved during the update. If necessary, the pool can be exported and re-imported to update the device IDs.

'''from the web'''

Useful link: http://www.princeton.edu/~unix/Solaris/troubleshoot/zfs.html

Ok, ZFS is now in the tree, what's now? Below you'll find some
instructions how to quickly make it up and running.

First of all you need some disks. Let's assume you have three spare SCSI
disks: da0, da1, da2.

Add a line to your /etc/rc.conf to start ZFS automatically on boot:

# echo 'zfs_enable="YES"' >> /etc/rc.conf

Load ZFS kernel module, for the first time by hand:

# kldload zfs.ko

Now, setup one pool using RAIDZ:

# zpool create tank raidz da0 da1 da2

It should automatically mount /tank/ for you.

Ok, now put /usr/ on ZFS and propose some file systems layout. I know
you probably have some files already, so we will work on /tank/usr
directory and once we ready, we will just change the mountpoint to /usr.

# zfs create tank/usr

Create ports/ file system and enable gzip compression on it, because
most likely we will have only text files there. On the other hand, we
don't want to compress ports/distfiles/, because we keep compressed
stuff already in-there:

# zfs create tank/usr/ports
# zfs set compression=gzip tank/usr/ports
# zfs create tank/usr/ports/distfiles
# zfs set compression=off tank/usr/ports/distfiles

(You do see how your life is changing, don't you?:))

Let's create home file system, my own home/pjd/ file system. I know we
use RAIDZ, but I want to have directory where I put extremly important
stuff, you I'll define that each block has to be stored in tree copies:

# zfs create tank/usr/home
# zfs create tank/usr/home/pjd
# zfs create tank/usr/home/pjd/important
# zfs set copies=3 tank/usr/home/pjd/important

I'd like to have directory with music, etc. that I NFS share. I don't
really care about this stuff and my computer is not very fast, so I'll
just turn off checksumming (this is only for example purposes! please,
benchmark before doing it, because it's most likely not worth it!):

# zfs create tank/music
# zfs set checksum=off tank/music
# zfs set sharenfs=on tank/music

Oh, I almost forget. Who cares about access time updates?

# zfs set atime=off tank

Yes, we set it only on tank and it will be automatically inherited by
others.

Will be also good to be informed if everything is fine with our pool:

# echo 'daily_status_zfs_enable="YES"' >> /etc/periodic.conf

For some reason you still need UFS file system, for example you use ACLs
or extended attributes which are not yet supported by our ZFS. If so,
why not just use ZFS to provide storage? This way we gain cheap UFS
snapshots, UFS clones, etc. by simply using ZVOLs.

# zfs create -V 10g tank/ufs
# newfs /dev/zvol/tank/ufs
# mount /dev/zvol/tank/ufs /ufs

# zfs snapshot tank/ufs at 20070406
# mount -r /dev/zvol/tank/ufs at 20070406 /ufs20070406

# zfs clone tank/ufs at 20070406 tank/ufsok
# fsck_ffs -p /dev/zvol/tank/ufsok
# mount /dev/zvol/tank/ufsok /ufsok

Want to encrypt your swap and still use ZFS? Nothing more trivial:

# zfs create -V 4g tank/swap
# geli onetime -s 4096 /dev/zvol/tank/swap
# swapon /dev/zvol/tank/swap.eli

Trying to do something risky with your home? Snapshot it first!

# zfs snapshot tank/home/pjd at justincase

Turns out it was more stupid than risky? Rollback your snapshot!

# zfs rollback tank/home/pjd at justincase
# zfs destroy tank/home/pjd at justincase

Ok, everything works, we may set tank/usr as our real /usr:

# zfs set mountpoint=/usr tank/usr

Don't forget to read zfs(8) and zpool(8) manual pages and SUN's ZFS
administration guide:

http://www.opensolaris.org/os/community/zfs/docs/zfsadmin.pdf

--

======================================== from the web =============================================

ZFS and Containers: An Example
This section is a step-by-step guide that shows how to perform certain ZFS file system tasks inside of Solaris Containers; for example, taking snapshots and managing data compression. It does this by going through the following steps:

* Creating a zpool
* Creating a Zone
* Allocating a ZFS File System to a Zone
* Creating New File Systems
* Applying Quota to the File Systems
* Changing the Mountpoint of a File System
* Setting the Compression Property
* Taking a Snapshot

Each of these steps is described in detail below.

Creating a zpool
ZFS uses device names or partition names when dealing with pools and devices. For a device, this will be something like c1t0d0 (for a SCSI device) or c1d1 (for an IDE device). For a partition, it will be something like c1t0d0s0 (for a SCSI device) or c1d1s0 (for an IDE device). This example creates a pool that is mirrored using two disks.

1. To create a zpool in the global zone, use the zpool create command. Typically, you use two devices to provide redundancy.

Global# zpool create mypool mirror c2t5d0 c2t6d0

Note that the zpool create command may fail if the devices are in use or contain some types of existing data (e.g. UFS file system). If they are in use, you will need to unmount them or otherwise stop using them. If they contain existing data you can use the -f(force) flag to override the safety check, but be sure that you are not destroying any data you want to retain.
2. Examine the pool properties using the zpool list command.

Global# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool

199G

164K

199G

0%

online

--
This shows you that there is one zpool, named mypool,with a capacity of 199GBytes.

Creating a Zone
To show ZFS working in an environment that is isolated from other applications on the system, you need to create a zone. To create a zone:

1. Create a directory where the zone file system will reside, using the mkdir command. Be sure to chose a location where the filesystem has at least 80MBytes of available space.

Global# mkdir /zones

Note that in this example, for the sake of brevity, the root file system of the zone is a UFS file system.

2. Configure the zone (myzone), using the zonecfg command, and specify the location of the zone's files (/zones/myzone). Use the following series of commands.

Global# zonecfg -z myzone
myzone: No such zone configured
Use 'create' to begin configuring a new zone
zonecfg:myzone< create
zonecfg:myzone< set zonepath=/zones/myzone
zonecfg:myzone< verify
zonecfg:myzone< commit
zonecfg:myzone< exit

Again, for the purpose of streamlining, this example uses a very minimal zone. For more details on creating zones see the Solaris Containers How To Do Stuff guide at: sun.com/software/solaris/howtoguides/containersLowRes.jsp

3. Install the zone by using the zoneadm.

Global# zoneadm -z myzone install
Preparing to install zone >myzone<
[Output from zoneadm, this may take a few mins]

4. Boot the zone to complete the installation, using the zoneadm command.

Global# zoneadm -z myzone boot

5. Use the zlogin command to connect to the zone console.

Global# zlogin -C myzone
[Connected to zone 'myzone' console]
[Initial zone boot output, service descriptions are loaded etc.]
It may take a short while for the first boot to configure everything, load all the service descriptors, and so on. You will need to answer the system configuration details. Some suggestions are:

Terminal=(12)X Terminal Emulator (xterms)
Not Networked
No Kerberos
Name service = None
Time Zone = your-time-zone
root passwd = (Your choice—remember it though!)

The zone will reboot after you have provided the configuration information.

6. Before you can proceed to the next stage, the configured zone needs to be shutdown (configuration changes are only applied when the zone boots).

Global# zlogin myzone init 5

Allocating a ZFS File System to a Zone
Now that you have a zpool (mypool) and a zone (myzone) you are ready to allocate a ZFS file system to the zone.

1. To create a ZFS file system, use the zfs create command.

Global# zfs create mypool/myzonefs

2. To apply a quota to the file system, use the zfs set quota command.

Global# zfs set quota=5G mypool/myzonefs

The file system and all of its child file systems can be no larger than the designated quota. Note that both these steps must be performed in the global zone. Also notice that creating the file system in ZFS is much simpler than with a traditional file system/volume manager combination.

To illustrate the isolation/security properties of containers with ZFS this example now creates a ZFS file system that will remain outside the container. There is no need to apply a quota to this outside file system.

3. To create this other file system, again use the zfs create command.

Global# zfs create mypool/myfs

4. To show the properties of all the pool and the file systems, use the zfs list command.

Global# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

396G

197G

99.5K

/mypool
mypool/myfs

98.5K

197GK

98.5K

/mypool/myfs
mypool/myzonefs

98.5K

5G

98.5K

/mypool/myzonefs

To make the file system (myzonefs) available in the zone (myzone), the zone configuration needs to be updated.

5. To update the zone configuration, use the zonecfg command.

Global# zonecfg -z myzone
zonecfg:myzone> add dataset
zonecfg:myzone:dataset> set name=mypool/myzonefs
zonecfg:myzone:dataset> end
zonecfg:myzone> commit
zonecfg:myzone> exit

The mypool/myzonefs file system is now added to the zone configuration. Note that these steps must be performed with the zone shut down, otherwise the zone configuration changes would not be visible until the next reboot. To check that the zone is shut down try logging into it using zlogin myzone. If the zone is shut down the login will fail; if the zone is running you will see a login prompt—login as root and shut the zone down with init 5. These steps are performed in the global zone.

6. Now boot the zone.

Global# zoneadm -z myzone boot

7. Log in to the zone. (Leave a few seconds for the zone to boot.)

Global# zlogin -C myzone
[Connected to zone 'myzone' pts/3]
[Usual Solaris login sequence]

8. List the ZFS file systems in the zone.

NAME USED AVAIL REFER MOUNTPOINT
mypool

0M

200B

--

/mypool
mypool/myzonefs

8K

5G

8K

/mypool/myzonefs

Note the 5GByte maximum available from the external quota and that the other file systems in the pool (mypool/myfs) are not visible. This demonstrates the isolation property that Containers provide.

Creating New File Systems
Administering ZFS file systems from the non-global zone is done just like it is in the global zone, although you are limited to operating within the file system that is allocated to the zone (mypool/myzonefs). New ZFS file systems are always created as a child of this file system because this is the only ZFS file system the non-global zone can see. The administrator in the non-global zone can create these file systems. There is no need to involve administrator of the global zone, though the administer could do so if it were necessary.

1. To create a newfile system, use the zfs create command.

MyZone# zfs create mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

594M

197G

99K

/mypool
mypool/myzonefs

197K

5.00G

98.5K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

5.00G

98.5K

/mypool/myzonefs/tim

The non-global zone administrator can create as many child file systems as s/he wants and each child file system can have its own file systems, and in that way form a hierarchy.

As a demonstration that the non-global zone administrator is limited to the assigned file systems, this example demonstrates trying to break security by creating a file system outside the container's "space".

2. Try to create another file system outside of mypool/myzonefs, using the zfs create command.

MyZone# zfs create mypool/myzonefs1
cannot create 'mypool/myzonefs1': permission denied

As you can see, ZFS and zones security denies permission for the non-global zone to access resources it has not been allocated and the operation fails.

Applying Quota to the File Systems
Typically, to prevent the user consuming all of the space, a non-global zone administrator will want to apply a quota to the new file system. Of course, the child's quota can't be more than 5GByte as that's the quota specified by the global zone administrator to all of the file systems below mypool/myzonefs.

1. To set a quota on our new file system, use the zfs set quota command.

MyZone# zfs set quota=1G mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

508M

197G

99K

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

1024M

98.5K

/mypool/myzonefs/tim

The administrator of the non-global zone has set the quota of the child file system to be 1G. They have full authority to do this because they are operating on their delegated resources and do not need to involve the global zone administrator.

The ZFS property inheritance mechanism applies across zone boundaries, so the non-global zone administrator can specify his/her own property values should s/he wish to do so. As with normal ZFS property inheritance, these override inherited values.

Changing the Mountpoint of a File System
Now that the file system is set up and has the correct quota assigned to it, it is ready for use. However, the place where the file system appears (the mountpoint) is partially dictated by what the global zone administrator initially chose as the pool name (in this example, mypool/myzonefs). But typically, a non-global zone administrator would want to change it.

1. To change the mountpoint, use the zfs set mountpoint command.

MyZone# zfs set mountpoint=/export/home/tim mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

508M

197G

99K

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

1024M

98.5K

/export/home/tim

Note that the mountpoint can be changed for any file system independently.

Setting the Compression Property
The next example demonstrates the compression property. If compression is enabled, ZFS will transparently compress all of the data before it is written to disk.

The benefits of compression are both saved disk space and possible write speed improvements.

1. To see what the current compression setting is, use the zfs get command.

MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim

NAME PROPERTY VALUE SOURCE
mypool

compression

off

default
mypool/myzonefs

compression

off

default
mypool/myzonefs/tim

compression

off

default

Beaware that the compression property on the pool is inherited by the file system and child file system. So if the non- global zone administrator sets the compression property for the delegated file system, it will set it for everything below, as well.

2. To set the compression for the file system, use the zfs set command.

MyZone# zfs set compression=on mypool/myzonefs

3. Examine the compression property again in the non-global zone.

MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim

NAME PROPERTY VALUE SOURCE
mypool

compression

off

default
mypool/myzonefs

compression

on

local
mypool/myzonefs/tim

compression

off

Inherited from mypool/myzonefs

Note the compression property has been inherited by mypool/myzonefs/tim as with normal ZFS administration.

Taking a Snapshot
One of the major advantages of ZFS is the ability to create an instant snapshot of any file system. By delegating a file system to a non-global zone this feature becomes available as an option for the non-global zone administrator.

1. To take a snapshot named "1st" of the file system, use the zfs snapshot command.

MyZone# zfs snapshot mypool/myzonefs@1st
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

512K

99K

default

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs@1st

OK

--

99K

--
mypool/myzonefs/tim

98.5K

1024M

98.5K

/export/home/tim

As with ZFS file systems in the global zone, this snapshot is now accessible from the root of the file system in .zfs/snapshot/1st.

Back To Top

Summary
Once a zone has been created and a ZFS file system has been allocated to it, the administrator for that (non-global) zone can create file systems, take snapshots, create clones, and perform all the other functions of an administrator—within that zone. Yet the global zone, and any other zones, are fully isolated from whatever happens in that zone.

The integration of Solaris Containers and Solaris ZFS is just another way that the Solaris 10 OS is providing cost benefits to customers by allowing them to safely consolidate applications and more easily manage the data those applications use.

ERROR Book

2011-12-11T04:21:20Z

129.57.167.14:

* Sergey B. 10-dec-2011: nothing bad happened, security scan got through, need to block it better:

.....
FFFFFFFFFFFFFFFFFF
WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=29 address>129.57.71.28< port=59401
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
accept: Too many open files
.....

* Sergey B. 23-nov-2011: EB1 hung, probably because if security scan:

.....
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
tcl7.4(init.tcl): invalid command name "GET"
clasprod::EB1> WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=6 address>129.57.71.28< port=47514
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
<HUNG>

* Sergey B. 7-nov-2011: dc12, error message from from rol.h (10kHz pulser):

codaUpdateStatus: updating request done
active, events so far 0
wait: coda request >go< in progress
0x3e55cd0 (twork0047): INIT_NAME: INFO: rolp->daproc = 5
0x3e55cd0 (twork0047): INFO: User Go ...
codaExecute done
SEND_BUFFER_ROC 4 (0x017f4050 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x017f4050 11)
SSS2 4 (0x017f4050 11)
SEND_BUFFER_ROC 4 (0x016bd8e0 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x016bd8e0 11)
SSS2 4 (0x016bd8e0 11)
SEND_BUFFER_ROC 4 (0x01587170 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01587170 11)
SSS2 4 (0x01587170 11)
SEND_BUFFER_ROC 4 (0x01450a00 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01450a00 11)
SSS2 4 (0x01450a00 11)
SEND_BUFFER_ROC 4 (0x0131a290 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x0131a290 11)
SSS2 4 (0x0131a290 11)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
interrupt: INFO from bb_write: NO BUFS1
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=21263(21263) heartmask=4
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
...........

Click end - ended fine (probably network was stuck ?? or proc ???).

* Sergey B. 8-jun-2011: test setup croctest2: error on croctest2 after 99M events, DMAing from single v1720 (NOTE: first test of ROL1 writing directly to the big buffer):
proc_thread: waiting= 846 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 192 processing= 460 microsec per event (nev=189)
rols_thread: waiting= 194 processing= 457 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 846 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 185 processing= 469 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 200 processing= 449 microsec per event (nev=188)
proc_thread: waiting= 847 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
net_thread: waiting= 720 sending= 0 microsec per event (nev=2218)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 847 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 205 processing= 450 microsec per event (nev=188)
machine check
Exception next instruction address: 0x0946e298
Machine Status Register: 0x0010b030
Condition Register: 0x80000088
Task: 0x946c670 "ROLS_LOOP"
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=54077(54077) heartmask=4
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
-> tt
1668c8 vxTaskEntry +68 : rols_loop ()
d2e5290 rols_loop +b4 : output_proc_network ()
d2e5574 output_proc_network+298: vmenrol1__init ()
946ccb8 vmenrol1__init +19c: 946cafc ()
946cb0c end +91cb68c: 946ca1c ()
946cabc end +91cb63c: 946c890 ()
946c920 end +91cb4a0: 946e298 ()
value = 0 = 0x0
->

* Sergey B. 10-feb-2011: test setup croctest2: coda_net on pmc dies:
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6559)
net_thread: waiting= 131 sending= 2 microsec per event (nev=12708)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6474)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting= 129 processing= 3 microsec per event (nev=6452)
net_thread: waiting= 133 sending= 2 microsec per event (nev=12598)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6553)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=16)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=12)
net_thread: waiting= 131 sending= 2 microsec per event (nev=12714)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6489)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=8)
proc_thread: waiting= 128 processing= 3 microsec per event (nev=6478)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
net_thread: waiting= 134 sending= 2 microsec per event (nev=12377)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=0)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6554)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=3)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6470)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting= 128 processing= 3 microsec per event (nev=6516)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
ERROR2: LINK_sized_write() returns errno=32
Broken pipe
ERROR: net_thread failed (in LINK_sized_write).
->
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 PEND 13f934 1ed0f190 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_proc proc_thread 1aacca00 115 DELAY 144bb4 1aacc900 0 10
coda_pmc coda_pmc 1e3f9720 200 DELAY 144bb4 1e3f9480 0 5
value = 0 = 0x0
-> printErrno 32
S_errno_EPIPE
value = 14 = 0xe
-> tt
Sorry, traces of my own stack begin at tt ().
value = -1 = 0xffffffff
-> memShow
invalid block at 0x1ec60000 deleted
value = -1 = 0xffffffff
-> memShow
status bytes blocks avg block max block
------ ---------- --------- ---------- ----------
current
free 421348656 56 7524083 411333840
alloc 79807344 1986 40184 -
cumulative
alloc 87490320 7392 11835 -
value = 0 = 0x0
-> usrNetStackSysPoolStatus
value = 0 = 0x0
-> usrNetStackSysPoolStatus("CODAtcpServer",0)
value = 0 = 0x0
-> usrNetStackDataPoolStatus("CODAtcpServer",0)
value = 454823200 = 0x1b1c0d20
-> usrNetStackDataPoolStatus("bla",0)
value = 454823168 = 0x1b1c0d00
-> usrNetStackDataPoolStatus("bla",0)
value = 454823040 = 0x1b1c0c80
-> memShow
status bytes blocks avg block max block
------ ---------- --------- ---------- ----------
current
free 421348528 55 7660882 411333840
alloc 79807472 1990 40104 -
cumulative
alloc 87496528 7401 11822 -
value = 0 = 0x0
->

EB looks fine, message was:
-- Got all fragments of go
BOS_encode_spec reached
BOS_encode_spec done: len=116
FFFFFFFFFFFFFFFFFF
PUT: wait while icb=8 == cbp->read=8 -------
clastest::EB5>

Tried to reset-download-prestart and got error on pmc at Download:
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 PEND 13f934 1ed0f190 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_pmc coda_pmc 1e3f9720 200 DELAY 144bb4 1e3f9480 30065 1
value = 0 = 0x0
-> Download command received
-> bignet at 0x0d3a6e68
<- bignet at 0xcd3a6e68
bignet at 0xcd3a6e68, bignet.gbigBuffer at 0xcd3a6e78 -> 0xcd3a6e78
proc_on_pmc=1, net_on_pmc=1
rocID=0
ObjInitName >_rol2_tt_testsetup__init<
data access
Exception current instruction address: 0x000d2090
Machine Status Register: 0x00003030
Data Access Register: 0x3d40001c
Condition Register: 0x40002083
Data storage interrupt Register: 0x40000000
Task: 0x1ed0f280 "tNetTask"
-> tt
14cad8 vxTaskEntry +68 : netTask ()
c3e88 netTask +70 : 209c0 ()
20a14 mgiEndLoad +cc8: 20a88 ()
20bd8 mgiEndLoad +e8c: muxReceive ()
bc638 muxReceive +13c: b002c ()
b02c4 random +358: do_protocol_with_type ()
9c920 do_protocol_with_type+2c : ipintr (1ec60e04)
9fb28 ipintr +c8c: dbbb4 ()
dbbb4 tcp_input +2968: tcp_output ()
dca24 tcp_output +5c8: mHdrClGet ()
aa940 mHdrClGet +8c : d1eb4 ()
value = 0 = 0x0
->
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 SUSPEND d2090 1ed0efd0 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_pmc coda_pmc 1e3f9720 200 PEND 1415a0 1e3f8ab0 3d0004 0
value = 0 = 0x0
->

* Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!???:
clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod -m eb -v
et_nodename >eb<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station LEVEL3
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_station_create_at, created station ET2ER
unique name >et_start_eb_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_clasprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
ipc_init called at Fri Apr 16 11:45:20 2010
et_start: pthread_create(0x0000000e,...) done
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et INFO: et_sys_heartmonitor, kill bad process (2,22078)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

clondaq2:clasrun> cd et2_start
clondaq2:et2_start>
clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod
-m er -v
et_nodename >er<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station TAPE
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur
rent/coda/SunOS_i86pc/lib/libet_user.so
et INFO: et_conductor 2, loaded function et2et_function
et INFO: et_station_create_at, created station ET2ET10
unique name >et_start_er_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R
Tserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve
r
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established
.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl
asprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab
.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>

ipc_init called at Fri Apr 16 11:45:20 2010

et_start: pthread_create(0x0000000e,...) done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_restore_events, no events found belonging to attachment 4
et INFO: et_station_detach, done
et WARN: et_command_loop: remote client connection broken
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et INFO: et_sys_heartmonitor, kill bad process (1,8898)
et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1
et INFO: et_flush_events, read 4000 events
et INFO: et_flush_events, wrote 4000 events
et INFO: et_station_detach, make station "TAPE" (cec1d440) idle
et INFO: et_restore_events, no events found belonging to attachment 3
et INFO: et_station_detach, done
et INFO: et_sys_heartmonitor, cleanup process 1
et INFO: set_fix_nprocs, change # of ET processes from 1 to 1
et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments
et INFO: set_fix_natts, station ET2ET10 has 1 attachments
et INFO: set_fix_natts, # total attachments 3 -> 3
et INFO: set_fix_natts, proc 0 has 1 attachments

clondaq1:dac>
clondaq1:dac>
clondaq1:dac>
clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 22276) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,

DYNAMIC INFO - currently there are:
processes(4), attachments(4), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(2,3,)
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"LEVEL3" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(1,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ER" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)

ATTACHMENTS:
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
proc(1), blocked(YES)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
proc(3), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER,
LOCKED MUTEXES:

*****************************************

ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 22276) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,

DYNAMIC INFO - currently there are:
processes(4), attachments(4), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(2,3,)
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"LEVEL3" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(1,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ER" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)

ATTACHMENTS:
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
proc(1), blocked(YES)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
proc(3), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER,
LOCKED MUTEXES:

*****************************************

ET SYSTEM is DEAD!
*****************************************

^C
clondaq1:dac>

clondaq2:stage_in>
clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 16127) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,

DYNAMIC INFO - currently there are:
processes(3), attachments(3), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(0), ids()
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"TAPE" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(1,2,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ET10" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class =
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)

ATTACHMENTS:
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
proc(1), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10,
LOCKED MUTEXES:

*****************************************

ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 16127) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,

DYNAMIC INFO - currently there are:
processes(3), attachments(3), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(0), ids()
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"TAPE" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(1,2,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ET10" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class =
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)

ATTACHMENTS:
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
proc(1), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10,
LOCKED MUTEXES:

*****************************************

ET SYSTEM is DEAD!
*****************************************

^C
clondaq2:stage_in>

* Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem:

wait: coda request >end< in progress
0x916e80 (twork0028): TS GO bit cleared
0x916e80 (twork0028): TS csr: 0xfc070000
0x916e80 (twork0028): TS csr2: 0xffff1c1f
interrupt: TRIG ERROR: no pool buffer available
0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0
0x916e80 (twork0028): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done

* Sergey B. 14-apr-2010: message from runcontrol during cancel-reset:
Get all components failed: Lost connection to MySQL server during query

* Sergey B. 6-apr-2010: EB waiting for following ROCs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

Files produced:

clon10:ccscans> ll | grep Apr
drwxrwxrwx 2 clasrun onliners 4096 Apr 2 16:23 ./
drwxr-xr-x 163 clasrun 100 73728 Apr 6 2010 ../
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_cc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:09 good_clastrig2*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest10*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest2*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:21 good_croctest3*
-rwxrwxrwx 1 baturin clas 34 Apr 2 13:26 good_ctoftest1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc10*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc11*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc5*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc6*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc7*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc8*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc9*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_polar*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_sc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_sc2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_scaler1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage3*
clon10:ccscans> more good_dc7
accepted link from >129.57.68.80<
clon10:ccscans> more good_dc8
accepted link from >129.57.68.74<
clon10:ccscans>

On 'Reset' EB crashed, outputs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

CODAtcpServer: start work thread
befor: socket=5 address>129.57.167.14< port=34468
wait: coda request >prestart< in progress
codaExecute reached, message >exit<, len=4
codaExecute: 'exit' transition
debShutdownBuild 1
cb_delete roc=0
cb_delete roc=1
cb_delete roc=2
cb_delete roc=3
cb_delete roc=4
cb_delete roc=5
cb_delete roc=6
cb_delete roc=7
cb_delete roc=8
cb_delete roc=9
cb_delete roc=10
cb_delete roc=11
cb_delete roc=12
cb_delete roc=13
cb_delete roc=14
cb_delete roc=15
cb_delete roc=16
cb_delete roc=17
cb_delete roc=18
cb_delete roc=19
cb_delete roc=20
cb_delete roc=21
cb_delete roc=22
cb_delete roc=23
cb_delete roc=24
cb_delete roc=25
cb_delete roc=26
cb_delete roc=27
cb_delete roc=28
cb_delete roc=29
cb_delete roc=30
cb_delete roc=31
debShutdownBuild 2
Stop waiting for the rocs because of force_end condition
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!
[0] et_event_new() 1
[0] EVENT_ENCODE_SPEC 2 ..
BOS_encode_spec reached
BOS_encode_spec done: len=116
[0] .. done
[0] handle_build_cleanup
[0] flush ET system
[0] detached from ET
[0] remove mutex locks and shutdown fifos
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[00] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[00] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[00] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[01] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[01] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[01] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[02] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[02] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[02] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[03] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[03] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[03] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[04] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[04] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[04] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[05] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[05] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[05] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[06] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[06] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[06] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[07] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[07] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[07] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[08] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[08] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[08] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[09] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[09] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[09] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[10] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[10] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[10] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[11] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[11] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[12] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[12] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[13] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[13] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[14] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[15] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[16] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[17] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[18] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[19] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[20] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[21] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[22] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[23] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[24] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[25] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[26] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[27] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[28] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[29] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[30] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[31] = 0
CCCCCCCCCC: 0
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] build thread exiting: 1 1
[0] The implementation has detected that the value specified by threa
d does not refer to a joinable thread.
wait: coda request >exit< in progress
polling_routine ================!!!!!!!!!!!!!!!!!!=====
cancel building threads
=c=============================================
=c=============================================
=c=============================================
=c====
>>>>> close link from >dc1< link=0x081c2af0
debCloseLink: theLink=0x081c2af0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc2< link=0x081c4a90
debCloseLink: theLink=0x081c4a90 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc3< link=0x081c4b28
debCloseLink: theLink=0x081c4b28 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc4< link=0x081c14f8
debCloseLink: theLink=0x081c14f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc5< link=0x081c1590
debCloseLink: theLink=0x081c1590 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc6< link=0x081c1628
debCloseLink: theLink=0x081c1628 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc8< link=0x081c16c0
debCloseLink: theLink=0x081c16c0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >cc1< link=0x081c1a08
debCloseLink: theLink=0x081c1a08 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >sc1< link=0x081c1aa0
debCloseLink: theLink=0x081c1aa0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec1< link=0x081c1b38
debCloseLink: theLink=0x081c1b38 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec2< link=0x081c1bd0
debCloseLink: theLink=0x081c1bd0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler2< link=0x681d95c8
debCloseLink: theLink=0x681d95c8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler3< link=0x681d9660
debCloseLink: theLink=0x681d9660 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler4< link=0x681d96f8
debCloseLink: theLink=0x681d96f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec4< link=0x00000001
debCloseLink: theLink=0x00000001 -> closing
Segmentation fault
clondaq1:coda_eb>

DC7:

codaExecute done
wait: coda request >download prod< in progress
codaExecute reached, message >prestart<, len=8
codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410567520
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc7<
LINK_constructor_C: set name to >dc7->EB1<
3121: dbsock=410567520
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda req<<<
uest >fields [pr2estart] >>>< in prodowngress
<<<
fields [3] >>>37683<<<
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x18772230 (twork0028INFO: Entering Us): er
Prestart 2
INIT_NAME: rolp->daproc =
trans_rol2:2
adr(nddl)=1db60750 NWBOS=16384
uthbook1: WARN id=1047 already exist - will be replaced
uthbook1: free histogram buffer ... done.
RAW=0 PROFILE=1 NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x18772230INFO: User Prest (art 2 executed
twork0028): CTRIGRSS: set handler and done,
code=1
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4130) returns 490106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a580) returns 490106296
0x18772230 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x18772230 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc
0x1882d864 0x00000001 0x1882b0c8
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1 = FFFFF000
Fastbus Status Register2 = FFFF0000
Fastbus Last Primary Addr = 00000019
spds_mask1: 0xfdfbf0
0x18772230_tdc1877 ===> (4twork0028wait: co)
: Use 32-bit Ram List
da request >firsprestartt< in progre csr0=ss
100 (1100) csr18=1364 csr1=6400001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0=100 (900) csr18=13
bignetptr=0x641d366fe0 csr1= offset=0x00bigp
rocptr=0x000001d366db80 offset=0x6400000000
00001b
bigproc at 0x
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =
==>1d366dc818 -> 0x1d366dc8
proc_thread reacheUDP_cancel: cancel >inf:d
c7 sys 0, mask 1d
<
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached
<
wait: cproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot# 1872 1875 1881(2) 1877(2) 1877S(2)
[ 0] 0 0 0 0bignetptr=0x
1d366fe00 offset=0x0000000 0
0 0 0
[ 1] 0 0 0 0 0 0 0
0
[ 2] 0 0 0 0 0 0 0
0
[ 3] 0 0 0 0 0 0 0
0
[ 4] 0 0 0 0 1 0 0
0
[ 5] 0 0 0 0 1 0 0
0
[ 6] 0 0 0 0 1 0 0
0
[ 7] 0 0 0 0 1 0 0
0
[ 8] 0 0 0 0 1 0 0
0
[ 9] 0 0 0 0 1 2 0
0
[10] wait: coda reques0t > prestart
0< in progress
0 0 0 0 0 0
[11] 0 0 0 0 1 0 0
0
[12] 0 0 0 0 1 0 0
0
[13] 0 0 0 0 1 0 0
0
[14] 0 0 0 0 1 0 0
0
[15] 0 0 0 0 1 0 0
0
[16] 0 0 0 0 1 2 0
0
[17] 0 0 0 0 0 0 0
0
[18] 0 0 0 0 1 0 0
0
[19] 0 0 0 0 1 0 0
0
[20] bignet at 0x 1d366fe00, bignet.gbig
Buffer at 0x 1d366ff00 -> 0x1d366ff0
0 0 1 0 0 0
[21] 0 0 0 0 1 0 0
0
[22] 0 0 0 0 1 0 0
0
[23] 0 0 0 0 1 2 0
0
[24] 0 0 0 0 0 0 0
0
[25] 0 0 0 0 0 0 0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC07<
0x18772230 (twork0028111): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <
<<<<<<<<<<<<<<<<
s: >UPDATE process SET state='paused' WHERE
name='dc7'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_cancel: cancel >sta:dc7 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
->
->

DC8:

codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410505056
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc8<
LINK_constructor_C: set name to >dc8->EB1<
3121: dbsock=410505056
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda re<<<
quest >fields [p2restart] >>>< in prupogress
<<<
fields [3] >>>37772<<<
parsing results: type=>TCP< host=>clondaq1-d
aq1< state=>up< port=>37772< -> 37772
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(
0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129
.57.68.22 port 37772
DB command >UPDATE links SET state='up' WHER
E name='dc8->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x187ef280INFO: Entering U (ser Prestart 2
twork0028
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da
proc = 1d2b60750
NWBOS=16384
uthbook1: WARN id=1048 already exist - will
be replaced
uthbook1: free histogram buffer ... done.
RAW=0 PROFILE=1 NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x187ef280 (twork0028INFO: User Pr): estart
2 executedCTRIGRSS: set handler and done, co
de=
1
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4110) returns 490
106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a560) returns 49
0106296
0x187ef280 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x187ef280 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x187ef280 (twork0028): CTRIGRSA: set handle
r and done, code=1
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c
0x1882d1e4 0x00000001 0x1882aa48
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1 = FFFFF000
Fastbus Status Register2 = FFFF0000
Fastbus Last Primary Addr = wait: coda req00
0000uest >19presta
rtspds_mask1: 0x< ifdfbf0n progress
0x187ef280_tdc1877 === (>twork0028): Use 32-
bit Ram List
4 first csr0=100 (1100) csr18=1364 csr1=6400
001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=136
4 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f
e0100 offset=0x0000000 (0
900) csr18=1364bigprocptr=0x csr1=1d366db864
00001b offset=0x
00000000
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.
gbigBuffer at 0x181d3UDP_cancel: cancel >inf
:dc8 sys 0, mask 166dc8<
wait: coda rUDP_cancel: cancel >inf:dc8 sys
0, mask 1 -> 0x<
1d366dc8eq
firstproc_thread reached
uest >proc_thread reached
csr0=proc_thread reached
prproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
100estart (< in prog1100ress
) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot# 1872 bignetptr=0x1875 181d366fe081
(2) 1877( offset=0x2) 1877S(00000002)
0
[ 0] 0 0 0 0 0 0 0
0
[ 1] 0 0 0 0 0 0 0
0
[ 2] 0 0 0 0 0 0 0
0
[ 3] 0 0 0 0 0 0 0
0
[ 4] 0 0 0 0 1 0 0
0
[ 5] 0 0 0 0 1 0 0
0
[ 6] 0 0 0 0 1 0 0
0
[ 7] 0 0 0 0 1 0 0
0
[ 8] 0 0 0 0 1 0 0
0
[ 9] 0 0 0 0 1 2 0
0
[10] wait: coda request0 > prestart
0< i n progress
0 0 0 0 0 0
[11] 0 0 0 0 1 0 0
0
[12] 0 0 0 0 1 0 0
0
[13] 0 0 0 0 1 0 0
0
[14] 0 0 0 0 1 0 0
0
[15] 0 0 0 0 1 0 0
0
[16] 0 0 0 0 1 2 0
0
[17] 0 0 0 0 0 0 0
0
[18] 0 0 0 0 1 0 0
0
[19] 0 bignet at 0x 1d366fe00, bigne
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366
ff0
0 1 0 0 0
[20] 0 0 0 0 1 0 0
0
[21] 0 0 0 0 1 0 0
0
[22] 0 0 0 0 1 0 0
0
[23] 0 0 0 0 1 2 0
0
[24] 0 0 0 0 0 0 0
0
[25] 0 0 0 0 0 0 0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC08<
0x187ef280 (twork0028111
): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000
0000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state=
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<
<<<<<<<<<<<<
ERE name='dc8'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_cancel: cancel >sta:dc8 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
.....

* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:
........
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
3123: dbsock=59553504
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<
====1==== !!! object->runType=47
token_interval=64
object->name >polar<
big0.proc_on_pmc=0, big0.net_on_pmc=0
big1.proc_on_pmc=0, big1.net_on_pmc=0
name selected
nrow=1
download: name >polar<
code selected
nrow=1
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
set this_roc_id = 25, rocId() can be called from now on
ObjInitName >_moller__init<
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
INFO: >_moller__init()< routine found
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded
allocate 200 buffers with length 65536 bytes
partCreate1() reached
partCreate1(): pPart=0x008e7570
listInit: cleanup 24 bytes starting from address 0x008e75a0
partCreate1() done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<
0x81c7e0 (twork0004): Init - Done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
Exception at interrupt level:
Exception next instruction address: 0xfff901f0
Fixed Point Register: 0x00000012
Condition Register: 0x40482044
Regs at 0x64e498
Press any key to stop auto-boot...
2
[VxWorks Boot]:

It suppose to be as following:
.....
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
rol1: downloading DDL table ...
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<
0x91ae30 (twork0001): Init - Done
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1
wait: coda request >download prod< in progress
adr1(nddl_)=0x037b69c8
N name (nname) fmt (nfmt) ncol
[ 1] >PTRN< (4) >B32< (3) 1
[ 2] >PSYN< (4) >B32< (3) 1
[ 3] >RC00< (4) >B32< (3) 1
[ 4] >RC01< (4) >B32< (3) 1
.....

* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:
-> ts2status
CSR 1 (0x0):
Go : 1
Pause on Next scheduled Sync : 0
Sync and Pause : 0
Initiate Sync Event : 0
Initiate Program 1 Event : 0
Initiate Program 2 Event : 1
Enable Level 1 (drives outputs) : 1
Override Inhibit : 0
Test Mode : 0
Reserved : 0
Reset : 0
Initialize : 0
Sync Event occurred : 0
Program 1 Event occurred : 1
Program 2 Event occurred : 0
Late Fail occurred : 0
Inhibit occurred : 1
Write FIFO error occurred : 0
Read FIFO error occurreds : 1
CSR 2 (0x4):
Enable Scheduled Sync : 1
Use Clear Permit Timer : 1
Use Front Busy Timer : 1
Use Clear Hold Timer : 1
Use External Front Busy : 1
Lock ROC Branch 1 : 0
Lock ROC Branch 2 : 0
Lock ROC Branch 3 : 0
Lock ROC Branch 4 : 0
Lock ROC Branch 5 : 0
Enable Program 1 front panel input : 1
Enable Interrupt : 1
Enable local ROC (branch 5) : 1
Trigger Control Register (0x8): 0x00001fff
ROC Enable Register (0xc) val=0xf07fff7f:
Branch 1: 0x7f bits: 01111111
Branch 2: 0xff bits: 11111111
Branch 3: 0x7f bits: 01111111
Branch 4: 0xf0 bits: 11110000
Synchronization Interval Register (0x10): 1000
Trigger Word Count Register (0x14): 0
Trigger Data Register (0x18): 256
Local ROC (Branch 5) Data Register (0x1c): 60
Synchronization Flag : 0
Late Fail Flag : 0
ROC code : 15
Input Trigger Prescale Registers:
Input 1 Prescale Factor : 0
Input 2 Prescale Factor : 0
Input 3 Prescale Factor : 0
Input 4 Prescale Factor : 0
Input 5 Prescale Factor : 0
Input 6 Prescale Factor : 0
Input 7 Prescale Factor : 0
Input 8 Prescale Factor : 0
Clear Permit Timer Register = 0
Level2 Accept Timer Register = 83
Level3 Accept Timer Register = 0
Front Busy Timer Register = 325
Clear Hold Timer Register = 50
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0
ROC Acknowledge Status Register (0x60): val=0x0
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)
State Register (0x6C):
Level 1 Accept : 0
Start Level 2 Trigger : 0
Level 2 Pass Latched : 0
Level 2 Fail Latched : 0
Level 2 Accept : 0
Start Level 3 Trigger : 0
Level 3 Pass Latched : 0
Level 3 Fail Latched : 0
Level 3 Accept : 0
Clear : 0
Front End Busy (external) : 0
External Inhibit : 0
Latched Trigger : 0
TS Busy : 1
TS Active : 1
TS Ready : 0
Main Sequencer Active : 0
Synchronization Sequencer Active : 0
Program 1 Event Sequencer Active : 1
Program 2 Event Sequencer Active : 0
Event Count (0xc8): 135
Live 1 Count (0xcc): 54866
Live 2 Count (0xd0): 9290375
value = 1 = 0x1
-> exit

clasprod::clastrig2> clastrig2 statistics
200 0 0 0

* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:

interrupt: Program Events 1-2: 247 0
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)
wait: coda request >go< in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
roc_end reached ??
roc_end reached !!
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <
interrupt: inputs 1-6: 0 0 0 0 0 0
interrupt: inputs 7-12: 0 0 339 0 0 0
interrupt: Program Events 1-2: 172 0
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)
wait: coda request >end< in progress
0x18c6f5c0 (twork0003): TS GO bit cleared
0x18c6f5c0 (twork0003): TS csr: 0xfc000000
interrupt: TRIG ERROR: no pool buffer available
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0
0x18c6f5c0 (twork0003): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_cancel: cancel >sta:clastrig2 active<
codaUpdateStatus: updating request done
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
roc_end done
codaExecute done
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty

* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):
.....
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
.....

* Sergey B. 2-apr-2010: event recorder gives error during Prestart:

file[ 0]->/raid/stage_in/clas_062414.A00<-
bosopen.c: splitted files handling ..
bosopen.c: >/raid/stage_in/clas_062414.A00<
bosopen.c: len1=30
bosopen.c: len2=30
bosopen.c: len3=14
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14
bosopen.c: checking how much disk space left in directory /raid/stage_in
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available
1100988411 blocks (1 block = 512 bytes)
bosopen.c: so we have 550493696 kbytes available space
bosopen.c: we can continue to write file.
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-
bosopen.c: system call: >checkdisk 4094<
[1] 9253
bosopen.c: system call completed
codaUpdateStatus: dbConnecting ..
ER: Not attached to ET system
er_write_thread loop ended (0 145452664)
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:ER3 paused<
UDP_standard_request >sta:ER3 paused<

Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.

* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 22 25 26 28 29
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???

* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash

* Sergey B. 14-mar-2010: during reboot dc9 gives following:
.................
ppc/bootscripts/boot_dc9
taskSpawn("TCPSERVER") returns 475458320
bind on port 5001
myname >dc9<
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded
INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
333333333333333
program
Exception current instruction address: 0x1c5
e8734
Machine Status Register: 0x00081000
Condition Register: 0x40000085
Task: 0x1c5e8e50 "ROC"
interrupt: Unconnected main interrupt 0
logTask: 3222 log messages lost.
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
..............

probably mysql connection problem

'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up.

* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing
wait: coda request >exit< in progress
indefinitely, must check

* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:

codaEnd 10
codaEnd 11
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
codaExecute done
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59319
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
codaExecute: ERROR: len=0 - do nothing
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59342
wait: coda request >< in progress
Executing >TP/1.0

Probably last string contains something, will remove printing

* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:
.....
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
.. done.
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30)
sent -1 (type 18)
[0] ERROR: Discard data until next control event
clondaq1:coda_eb>

After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,
reboot all rocs again - not it works.

In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...

* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:

.....
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_cancel: cancel >sta:EB1 booted<
codaUpdateStatus: updating request done
CODA_Init 14
2
clasprod::EB1> bind on port 5001
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=37258
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
Segmentation fault (core dumped)
clondaq1:coda_eb>

In the same time ''et_start'' complains (we have now ET system protected against CC scans):

et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure

'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already

* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:

Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e404190 "coda_pmc"

It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.

* Sergey B. 1-sep-2009: during croctest1 reboot:

INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL

second reboot went fine

* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.

* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:

..........................
taskDelay(sysClkRateGet()*5)
Args = -session clasprod -objects sc2 ROC -i
CODA_Init reached
CODA_Init 1
CODA_Init 2
sc2
CODA_Init 11
CODA_Init: objectTy >(null)<
CODA_Init 12
CODA_Init: use 'SESSION' as >clasprod<
Tcl_AppInit CALLS !!!!!
11-11
11-12
11-13
11111111111111111
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
value = 0 = 0x0
taskSpawn "TCP_SERVER",250,0,100000,tcpServer
value = 231091088 = 0xdc62b90
proconhost
value = 0 = 0x0
......................

Second reboot shows the same:

...................
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
list >clon10 clasrun 2508 9998<
machine name = clon10
user ID = 2508
group ID = 9998
.................

* Sergey B. around 6-June-2009: ec2pmc1 error message:

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait= 199 send= 8 microsec per event (nev=12755)
CALLING customized :
machine check
SRAM ERROR :
SRAM Error Address : 0x00000000.f2000504
SRAM Error Data Low : 0x006805e0
SRAM Error Data High : 0x00420000
SRAM Error Parity : 0x00000004
SRAM Error Cause : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2011-12-11T04:20:52Z

129.57.167.14:

ERROR Book

2011-12-11T04:20:13Z

129.57.167.14:

* Sergey B. 10-dec-2011: nothing bad happened, security scan got through, need to block it better:

.....
FFFFFFFFFFFFFFFFFF
WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=29 address>129.57.71.28< port=59401
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
accept: Too many open files

.....

* Sergey B. 23-nov-2011: EB1 hung, probably because if security scan:

.....
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
tcl7.4(init.tcl): invalid command name "GET"
clasprod::EB1> WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=6 address>129.57.71.28< port=47514
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
<HUNG>

* Sergey B. 7-nov-2011: dc12, error message from from rol.h (10kHz pulser):

codaUpdateStatus: updating request done
active, events so far 0
wait: coda request >go< in progress
0x3e55cd0 (twork0047): INIT_NAME: INFO: rolp->daproc = 5
0x3e55cd0 (twork0047): INFO: User Go ...
codaExecute done
SEND_BUFFER_ROC 4 (0x017f4050 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x017f4050 11)
SSS2 4 (0x017f4050 11)
SEND_BUFFER_ROC 4 (0x016bd8e0 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x016bd8e0 11)
SSS2 4 (0x016bd8e0 11)
SEND_BUFFER_ROC 4 (0x01587170 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01587170 11)
SSS2 4 (0x01587170 11)
SEND_BUFFER_ROC 4 (0x01450a00 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01450a00 11)
SSS2 4 (0x01450a00 11)
SEND_BUFFER_ROC 4 (0x0131a290 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x0131a290 11)
SSS2 4 (0x0131a290 11)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
interrupt: INFO from bb_write: NO BUFS1
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=21263(21263) heartmask=4
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
...........

Click end - ended fine (probably network was stuck ?? or proc ???).

* Sergey B. 8-jun-2011: test setup croctest2: error on croctest2 after 99M events, DMAing from single v1720 (NOTE: first test of ROL1 writing directly to the big buffer):
proc_thread: waiting= 846 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 192 processing= 460 microsec per event (nev=189)
rols_thread: waiting= 194 processing= 457 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 846 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 185 processing= 469 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 200 processing= 449 microsec per event (nev=188)
proc_thread: waiting= 847 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
net_thread: waiting= 720 sending= 0 microsec per event (nev=2218)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 847 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 205 processing= 450 microsec per event (nev=188)
machine check
Exception next instruction address: 0x0946e298
Machine Status Register: 0x0010b030
Condition Register: 0x80000088
Task: 0x946c670 "ROLS_LOOP"
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=54077(54077) heartmask=4
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
-> tt
1668c8 vxTaskEntry +68 : rols_loop ()
d2e5290 rols_loop +b4 : output_proc_network ()
d2e5574 output_proc_network+298: vmenrol1__init ()
946ccb8 vmenrol1__init +19c: 946cafc ()
946cb0c end +91cb68c: 946ca1c ()
946cabc end +91cb63c: 946c890 ()
946c920 end +91cb4a0: 946e298 ()
value = 0 = 0x0
->

* Sergey B. 10-feb-2011: test setup croctest2: coda_net on pmc dies:
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6559)
net_thread: waiting= 131 sending= 2 microsec per event (nev=12708)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6474)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting= 129 processing= 3 microsec per event (nev=6452)
net_thread: waiting= 133 sending= 2 microsec per event (nev=12598)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6553)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=16)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=12)
net_thread: waiting= 131 sending= 2 microsec per event (nev=12714)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6489)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=8)
proc_thread: waiting= 128 processing= 3 microsec per event (nev=6478)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
net_thread: waiting= 134 sending= 2 microsec per event (nev=12377)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=0)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6554)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=3)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6470)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting= 128 processing= 3 microsec per event (nev=6516)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
ERROR2: LINK_sized_write() returns errno=32
Broken pipe
ERROR: net_thread failed (in LINK_sized_write).
->
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 PEND 13f934 1ed0f190 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_proc proc_thread 1aacca00 115 DELAY 144bb4 1aacc900 0 10
coda_pmc coda_pmc 1e3f9720 200 DELAY 144bb4 1e3f9480 0 5
value = 0 = 0x0
-> printErrno 32
S_errno_EPIPE
value = 14 = 0xe
-> tt
Sorry, traces of my own stack begin at tt ().
value = -1 = 0xffffffff
-> memShow
invalid block at 0x1ec60000 deleted
value = -1 = 0xffffffff
-> memShow
status bytes blocks avg block max block
------ ---------- --------- ---------- ----------
current
free 421348656 56 7524083 411333840
alloc 79807344 1986 40184 -
cumulative
alloc 87490320 7392 11835 -
value = 0 = 0x0
-> usrNetStackSysPoolStatus
value = 0 = 0x0
-> usrNetStackSysPoolStatus("CODAtcpServer",0)
value = 0 = 0x0
-> usrNetStackDataPoolStatus("CODAtcpServer",0)
value = 454823200 = 0x1b1c0d20
-> usrNetStackDataPoolStatus("bla",0)
value = 454823168 = 0x1b1c0d00
-> usrNetStackDataPoolStatus("bla",0)
value = 454823040 = 0x1b1c0c80
-> memShow
status bytes blocks avg block max block
------ ---------- --------- ---------- ----------
current
free 421348528 55 7660882 411333840
alloc 79807472 1990 40104 -
cumulative
alloc 87496528 7401 11822 -
value = 0 = 0x0
->

EB looks fine, message was:
-- Got all fragments of go
BOS_encode_spec reached
BOS_encode_spec done: len=116
FFFFFFFFFFFFFFFFFF
PUT: wait while icb=8 == cbp->read=8 -------
clastest::EB5>

Tried to reset-download-prestart and got error on pmc at Download:
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 PEND 13f934 1ed0f190 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_pmc coda_pmc 1e3f9720 200 DELAY 144bb4 1e3f9480 30065 1
value = 0 = 0x0
-> Download command received
-> bignet at 0x0d3a6e68
<- bignet at 0xcd3a6e68
bignet at 0xcd3a6e68, bignet.gbigBuffer at 0xcd3a6e78 -> 0xcd3a6e78
proc_on_pmc=1, net_on_pmc=1
rocID=0
ObjInitName >_rol2_tt_testsetup__init<
data access
Exception current instruction address: 0x000d2090
Machine Status Register: 0x00003030
Data Access Register: 0x3d40001c
Condition Register: 0x40002083
Data storage interrupt Register: 0x40000000
Task: 0x1ed0f280 "tNetTask"
-> tt
14cad8 vxTaskEntry +68 : netTask ()
c3e88 netTask +70 : 209c0 ()
20a14 mgiEndLoad +cc8: 20a88 ()
20bd8 mgiEndLoad +e8c: muxReceive ()
bc638 muxReceive +13c: b002c ()
b02c4 random +358: do_protocol_with_type ()
9c920 do_protocol_with_type+2c : ipintr (1ec60e04)
9fb28 ipintr +c8c: dbbb4 ()
dbbb4 tcp_input +2968: tcp_output ()
dca24 tcp_output +5c8: mHdrClGet ()
aa940 mHdrClGet +8c : d1eb4 ()
value = 0 = 0x0
->
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 SUSPEND d2090 1ed0efd0 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_pmc coda_pmc 1e3f9720 200 PEND 1415a0 1e3f8ab0 3d0004 0
value = 0 = 0x0
->

* Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!???:
clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod -m eb -v
et_nodename >eb<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station LEVEL3
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_station_create_at, created station ET2ER
unique name >et_start_eb_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_clasprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
ipc_init called at Fri Apr 16 11:45:20 2010
et_start: pthread_create(0x0000000e,...) done
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et INFO: et_sys_heartmonitor, kill bad process (2,22078)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

clondaq2:clasrun> cd et2_start
clondaq2:et2_start>
clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod
-m er -v
et_nodename >er<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station TAPE
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur
rent/coda/SunOS_i86pc/lib/libet_user.so
et INFO: et_conductor 2, loaded function et2et_function
et INFO: et_station_create_at, created station ET2ET10
unique name >et_start_er_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R
Tserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve
r
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established
.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl
asprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab
.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>

ipc_init called at Fri Apr 16 11:45:20 2010

et_start: pthread_create(0x0000000e,...) done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_restore_events, no events found belonging to attachment 4
et INFO: et_station_detach, done
et WARN: et_command_loop: remote client connection broken
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et INFO: et_sys_heartmonitor, kill bad process (1,8898)
et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1
et INFO: et_flush_events, read 4000 events
et INFO: et_flush_events, wrote 4000 events
et INFO: et_station_detach, make station "TAPE" (cec1d440) idle
et INFO: et_restore_events, no events found belonging to attachment 3
et INFO: et_station_detach, done
et INFO: et_sys_heartmonitor, cleanup process 1
et INFO: set_fix_nprocs, change # of ET processes from 1 to 1
et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments
et INFO: set_fix_natts, station ET2ET10 has 1 attachments
et INFO: set_fix_natts, # total attachments 3 -> 3
et INFO: set_fix_natts, proc 0 has 1 attachments

clondaq1:dac>
clondaq1:dac>
clondaq1:dac>
clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 22276) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,

DYNAMIC INFO - currently there are:
processes(4), attachments(4), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(2,3,)
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"LEVEL3" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(1,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ER" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)

ATTACHMENTS:
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
proc(1), blocked(YES)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
proc(3), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER,
LOCKED MUTEXES:

*****************************************

ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 22276) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,

DYNAMIC INFO - currently there are:
processes(4), attachments(4), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(2,3,)
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"LEVEL3" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(1,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ER" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)

ATTACHMENTS:
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
proc(1), blocked(YES)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
proc(3), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER,
LOCKED MUTEXES:

*****************************************

ET SYSTEM is DEAD!
*****************************************

^C
clondaq1:dac>

clondaq2:stage_in>
clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 16127) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,

DYNAMIC INFO - currently there are:
processes(3), attachments(3), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(0), ids()
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"TAPE" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(1,2,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ET10" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class =
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)

ATTACHMENTS:
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
proc(1), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10,
LOCKED MUTEXES:

*****************************************

ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 16127) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,

DYNAMIC INFO - currently there are:
processes(3), attachments(3), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(0), ids()
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"TAPE" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(1,2,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ET10" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class =
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)

ATTACHMENTS:
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
proc(1), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10,
LOCKED MUTEXES:

*****************************************

ET SYSTEM is DEAD!
*****************************************

^C
clondaq2:stage_in>

* Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem:

wait: coda request >end< in progress
0x916e80 (twork0028): TS GO bit cleared
0x916e80 (twork0028): TS csr: 0xfc070000
0x916e80 (twork0028): TS csr2: 0xffff1c1f
interrupt: TRIG ERROR: no pool buffer available
0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0
0x916e80 (twork0028): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done

* Sergey B. 14-apr-2010: message from runcontrol during cancel-reset:
Get all components failed: Lost connection to MySQL server during query

* Sergey B. 6-apr-2010: EB waiting for following ROCs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

Files produced:

clon10:ccscans> ll | grep Apr
drwxrwxrwx 2 clasrun onliners 4096 Apr 2 16:23 ./
drwxr-xr-x 163 clasrun 100 73728 Apr 6 2010 ../
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_cc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:09 good_clastrig2*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest10*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest2*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:21 good_croctest3*
-rwxrwxrwx 1 baturin clas 34 Apr 2 13:26 good_ctoftest1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc10*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc11*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc5*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc6*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc7*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc8*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc9*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_polar*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_sc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_sc2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_scaler1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage3*
clon10:ccscans> more good_dc7
accepted link from >129.57.68.80<
clon10:ccscans> more good_dc8
accepted link from >129.57.68.74<
clon10:ccscans>

On 'Reset' EB crashed, outputs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

CODAtcpServer: start work thread
befor: socket=5 address>129.57.167.14< port=34468
wait: coda request >prestart< in progress
codaExecute reached, message >exit<, len=4
codaExecute: 'exit' transition
debShutdownBuild 1
cb_delete roc=0
cb_delete roc=1
cb_delete roc=2
cb_delete roc=3
cb_delete roc=4
cb_delete roc=5
cb_delete roc=6
cb_delete roc=7
cb_delete roc=8
cb_delete roc=9
cb_delete roc=10
cb_delete roc=11
cb_delete roc=12
cb_delete roc=13
cb_delete roc=14
cb_delete roc=15
cb_delete roc=16
cb_delete roc=17
cb_delete roc=18
cb_delete roc=19
cb_delete roc=20
cb_delete roc=21
cb_delete roc=22
cb_delete roc=23
cb_delete roc=24
cb_delete roc=25
cb_delete roc=26
cb_delete roc=27
cb_delete roc=28
cb_delete roc=29
cb_delete roc=30
cb_delete roc=31
debShutdownBuild 2
Stop waiting for the rocs because of force_end condition
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!
[0] et_event_new() 1
[0] EVENT_ENCODE_SPEC 2 ..
BOS_encode_spec reached
BOS_encode_spec done: len=116
[0] .. done
[0] handle_build_cleanup
[0] flush ET system
[0] detached from ET
[0] remove mutex locks and shutdown fifos
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[00] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[00] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[00] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[01] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[01] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[01] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[02] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[02] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[02] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[03] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[03] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[03] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[04] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[04] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[04] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[05] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[05] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[05] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[06] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[06] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[06] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[07] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[07] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[07] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[08] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[08] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[08] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[09] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[09] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[09] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[10] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[10] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[10] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[11] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[11] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[12] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[12] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[13] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[13] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[14] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[15] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[16] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[17] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[18] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[19] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[20] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[21] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[22] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[23] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[24] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[25] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[26] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[27] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[28] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[29] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[30] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[31] = 0
CCCCCCCCCC: 0
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] build thread exiting: 1 1
[0] The implementation has detected that the value specified by threa
d does not refer to a joinable thread.
wait: coda request >exit< in progress
polling_routine ================!!!!!!!!!!!!!!!!!!=====
cancel building threads
=c=============================================
=c=============================================
=c=============================================
=c====
>>>>> close link from >dc1< link=0x081c2af0
debCloseLink: theLink=0x081c2af0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc2< link=0x081c4a90
debCloseLink: theLink=0x081c4a90 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc3< link=0x081c4b28
debCloseLink: theLink=0x081c4b28 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc4< link=0x081c14f8
debCloseLink: theLink=0x081c14f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc5< link=0x081c1590
debCloseLink: theLink=0x081c1590 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc6< link=0x081c1628
debCloseLink: theLink=0x081c1628 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc8< link=0x081c16c0
debCloseLink: theLink=0x081c16c0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >cc1< link=0x081c1a08
debCloseLink: theLink=0x081c1a08 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >sc1< link=0x081c1aa0
debCloseLink: theLink=0x081c1aa0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec1< link=0x081c1b38
debCloseLink: theLink=0x081c1b38 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec2< link=0x081c1bd0
debCloseLink: theLink=0x081c1bd0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler2< link=0x681d95c8
debCloseLink: theLink=0x681d95c8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler3< link=0x681d9660
debCloseLink: theLink=0x681d9660 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler4< link=0x681d96f8
debCloseLink: theLink=0x681d96f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec4< link=0x00000001
debCloseLink: theLink=0x00000001 -> closing
Segmentation fault
clondaq1:coda_eb>

DC7:

codaExecute done
wait: coda request >download prod< in progress
codaExecute reached, message >prestart<, len=8
codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410567520
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc7<
LINK_constructor_C: set name to >dc7->EB1<
3121: dbsock=410567520
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda req<<<
uest >fields [pr2estart] >>>< in prodowngress
<<<
fields [3] >>>37683<<<
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x18772230 (twork0028INFO: Entering Us): er
Prestart 2
INIT_NAME: rolp->daproc =
trans_rol2:2
adr(nddl)=1db60750 NWBOS=16384
uthbook1: WARN id=1047 already exist - will be replaced
uthbook1: free histogram buffer ... done.
RAW=0 PROFILE=1 NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x18772230INFO: User Prest (art 2 executed
twork0028): CTRIGRSS: set handler and done,
code=1
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4130) returns 490106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a580) returns 490106296
0x18772230 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x18772230 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc
0x1882d864 0x00000001 0x1882b0c8
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1 = FFFFF000
Fastbus Status Register2 = FFFF0000
Fastbus Last Primary Addr = 00000019
spds_mask1: 0xfdfbf0
0x18772230_tdc1877 ===> (4twork0028wait: co)
: Use 32-bit Ram List
da request >firsprestartt< in progre csr0=ss
100 (1100) csr18=1364 csr1=6400001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0=100 (900) csr18=13
bignetptr=0x641d366fe0 csr1= offset=0x00bigp
rocptr=0x000001d366db80 offset=0x6400000000
00001b
bigproc at 0x
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =
==>1d366dc818 -> 0x1d366dc8
proc_thread reacheUDP_cancel: cancel >inf:d
c7 sys 0, mask 1d
<
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached
<
wait: cproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot# 1872 1875 1881(2) 1877(2) 1877S(2)
[ 0] 0 0 0 0bignetptr=0x
1d366fe00 offset=0x0000000 0
0 0 0
[ 1] 0 0 0 0 0 0 0
0
[ 2] 0 0 0 0 0 0 0
0
[ 3] 0 0 0 0 0 0 0
0
[ 4] 0 0 0 0 1 0 0
0
[ 5] 0 0 0 0 1 0 0
0
[ 6] 0 0 0 0 1 0 0
0
[ 7] 0 0 0 0 1 0 0
0
[ 8] 0 0 0 0 1 0 0
0
[ 9] 0 0 0 0 1 2 0
0
[10] wait: coda reques0t > prestart
0< in progress
0 0 0 0 0 0
[11] 0 0 0 0 1 0 0
0
[12] 0 0 0 0 1 0 0
0
[13] 0 0 0 0 1 0 0
0
[14] 0 0 0 0 1 0 0
0
[15] 0 0 0 0 1 0 0
0
[16] 0 0 0 0 1 2 0
0
[17] 0 0 0 0 0 0 0
0
[18] 0 0 0 0 1 0 0
0
[19] 0 0 0 0 1 0 0
0
[20] bignet at 0x 1d366fe00, bignet.gbig
Buffer at 0x 1d366ff00 -> 0x1d366ff0
0 0 1 0 0 0
[21] 0 0 0 0 1 0 0
0
[22] 0 0 0 0 1 0 0
0
[23] 0 0 0 0 1 2 0
0
[24] 0 0 0 0 0 0 0
0
[25] 0 0 0 0 0 0 0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC07<
0x18772230 (twork0028111): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <
<<<<<<<<<<<<<<<<
s: >UPDATE process SET state='paused' WHERE
name='dc7'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_cancel: cancel >sta:dc7 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
->
->

DC8:

codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410505056
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc8<
LINK_constructor_C: set name to >dc8->EB1<
3121: dbsock=410505056
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda re<<<
quest >fields [p2restart] >>>< in prupogress
<<<
fields [3] >>>37772<<<
parsing results: type=>TCP< host=>clondaq1-d
aq1< state=>up< port=>37772< -> 37772
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(
0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129
.57.68.22 port 37772
DB command >UPDATE links SET state='up' WHER
E name='dc8->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x187ef280INFO: Entering U (ser Prestart 2
twork0028
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da
proc = 1d2b60750
NWBOS=16384
uthbook1: WARN id=1048 already exist - will
be replaced
uthbook1: free histogram buffer ... done.
RAW=0 PROFILE=1 NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x187ef280 (twork0028INFO: User Pr): estart
2 executedCTRIGRSS: set handler and done, co
de=
1
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4110) returns 490
106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a560) returns 49
0106296
0x187ef280 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x187ef280 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x187ef280 (twork0028): CTRIGRSA: set handle
r and done, code=1
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c
0x1882d1e4 0x00000001 0x1882aa48
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1 = FFFFF000
Fastbus Status Register2 = FFFF0000
Fastbus Last Primary Addr = wait: coda req00
0000uest >19presta
rtspds_mask1: 0x< ifdfbf0n progress
0x187ef280_tdc1877 === (>twork0028): Use 32-
bit Ram List
4 first csr0=100 (1100) csr18=1364 csr1=6400
001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=136
4 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f
e0100 offset=0x0000000 (0
900) csr18=1364bigprocptr=0x csr1=1d366db864
00001b offset=0x
00000000
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.
gbigBuffer at 0x181d3UDP_cancel: cancel >inf
:dc8 sys 0, mask 166dc8<
wait: coda rUDP_cancel: cancel >inf:dc8 sys
0, mask 1 -> 0x<
1d366dc8eq
firstproc_thread reached
uest >proc_thread reached
csr0=proc_thread reached
prproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
100estart (< in prog1100ress
) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot# 1872 bignetptr=0x1875 181d366fe081
(2) 1877( offset=0x2) 1877S(00000002)
0
[ 0] 0 0 0 0 0 0 0
0
[ 1] 0 0 0 0 0 0 0
0
[ 2] 0 0 0 0 0 0 0
0
[ 3] 0 0 0 0 0 0 0
0
[ 4] 0 0 0 0 1 0 0
0
[ 5] 0 0 0 0 1 0 0
0
[ 6] 0 0 0 0 1 0 0
0
[ 7] 0 0 0 0 1 0 0
0
[ 8] 0 0 0 0 1 0 0
0
[ 9] 0 0 0 0 1 2 0
0
[10] wait: coda request0 > prestart
0< i n progress
0 0 0 0 0 0
[11] 0 0 0 0 1 0 0
0
[12] 0 0 0 0 1 0 0
0
[13] 0 0 0 0 1 0 0
0
[14] 0 0 0 0 1 0 0
0
[15] 0 0 0 0 1 0 0
0
[16] 0 0 0 0 1 2 0
0
[17] 0 0 0 0 0 0 0
0
[18] 0 0 0 0 1 0 0
0
[19] 0 bignet at 0x 1d366fe00, bigne
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366
ff0
0 1 0 0 0
[20] 0 0 0 0 1 0 0
0
[21] 0 0 0 0 1 0 0
0
[22] 0 0 0 0 1 0 0
0
[23] 0 0 0 0 1 2 0
0
[24] 0 0 0 0 0 0 0
0
[25] 0 0 0 0 0 0 0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC08<
0x187ef280 (twork0028111
): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000
0000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state=
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<
<<<<<<<<<<<<
ERE name='dc8'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_cancel: cancel >sta:dc8 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
.....

* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:
........
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
3123: dbsock=59553504
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<
====1==== !!! object->runType=47
token_interval=64
object->name >polar<
big0.proc_on_pmc=0, big0.net_on_pmc=0
big1.proc_on_pmc=0, big1.net_on_pmc=0
name selected
nrow=1
download: name >polar<
code selected
nrow=1
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
set this_roc_id = 25, rocId() can be called from now on
ObjInitName >_moller__init<
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
INFO: >_moller__init()< routine found
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded
allocate 200 buffers with length 65536 bytes
partCreate1() reached
partCreate1(): pPart=0x008e7570
listInit: cleanup 24 bytes starting from address 0x008e75a0
partCreate1() done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<
0x81c7e0 (twork0004): Init - Done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
Exception at interrupt level:
Exception next instruction address: 0xfff901f0
Fixed Point Register: 0x00000012
Condition Register: 0x40482044
Regs at 0x64e498
Press any key to stop auto-boot...
2
[VxWorks Boot]:

It suppose to be as following:
.....
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
rol1: downloading DDL table ...
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<
0x91ae30 (twork0001): Init - Done
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1
wait: coda request >download prod< in progress
adr1(nddl_)=0x037b69c8
N name (nname) fmt (nfmt) ncol
[ 1] >PTRN< (4) >B32< (3) 1
[ 2] >PSYN< (4) >B32< (3) 1
[ 3] >RC00< (4) >B32< (3) 1
[ 4] >RC01< (4) >B32< (3) 1
.....

* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:
-> ts2status
CSR 1 (0x0):
Go : 1
Pause on Next scheduled Sync : 0
Sync and Pause : 0
Initiate Sync Event : 0
Initiate Program 1 Event : 0
Initiate Program 2 Event : 1
Enable Level 1 (drives outputs) : 1
Override Inhibit : 0
Test Mode : 0
Reserved : 0
Reset : 0
Initialize : 0
Sync Event occurred : 0
Program 1 Event occurred : 1
Program 2 Event occurred : 0
Late Fail occurred : 0
Inhibit occurred : 1
Write FIFO error occurred : 0
Read FIFO error occurreds : 1
CSR 2 (0x4):
Enable Scheduled Sync : 1
Use Clear Permit Timer : 1
Use Front Busy Timer : 1
Use Clear Hold Timer : 1
Use External Front Busy : 1
Lock ROC Branch 1 : 0
Lock ROC Branch 2 : 0
Lock ROC Branch 3 : 0
Lock ROC Branch 4 : 0
Lock ROC Branch 5 : 0
Enable Program 1 front panel input : 1
Enable Interrupt : 1
Enable local ROC (branch 5) : 1
Trigger Control Register (0x8): 0x00001fff
ROC Enable Register (0xc) val=0xf07fff7f:
Branch 1: 0x7f bits: 01111111
Branch 2: 0xff bits: 11111111
Branch 3: 0x7f bits: 01111111
Branch 4: 0xf0 bits: 11110000
Synchronization Interval Register (0x10): 1000
Trigger Word Count Register (0x14): 0
Trigger Data Register (0x18): 256
Local ROC (Branch 5) Data Register (0x1c): 60
Synchronization Flag : 0
Late Fail Flag : 0
ROC code : 15
Input Trigger Prescale Registers:
Input 1 Prescale Factor : 0
Input 2 Prescale Factor : 0
Input 3 Prescale Factor : 0
Input 4 Prescale Factor : 0
Input 5 Prescale Factor : 0
Input 6 Prescale Factor : 0
Input 7 Prescale Factor : 0
Input 8 Prescale Factor : 0
Clear Permit Timer Register = 0
Level2 Accept Timer Register = 83
Level3 Accept Timer Register = 0
Front Busy Timer Register = 325
Clear Hold Timer Register = 50
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0
ROC Acknowledge Status Register (0x60): val=0x0
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)
State Register (0x6C):
Level 1 Accept : 0
Start Level 2 Trigger : 0
Level 2 Pass Latched : 0
Level 2 Fail Latched : 0
Level 2 Accept : 0
Start Level 3 Trigger : 0
Level 3 Pass Latched : 0
Level 3 Fail Latched : 0
Level 3 Accept : 0
Clear : 0
Front End Busy (external) : 0
External Inhibit : 0
Latched Trigger : 0
TS Busy : 1
TS Active : 1
TS Ready : 0
Main Sequencer Active : 0
Synchronization Sequencer Active : 0
Program 1 Event Sequencer Active : 1
Program 2 Event Sequencer Active : 0
Event Count (0xc8): 135
Live 1 Count (0xcc): 54866
Live 2 Count (0xd0): 9290375
value = 1 = 0x1
-> exit

clasprod::clastrig2> clastrig2 statistics
200 0 0 0

* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:

interrupt: Program Events 1-2: 247 0
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)
wait: coda request >go< in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
roc_end reached ??
roc_end reached !!
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <
interrupt: inputs 1-6: 0 0 0 0 0 0
interrupt: inputs 7-12: 0 0 339 0 0 0
interrupt: Program Events 1-2: 172 0
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)
wait: coda request >end< in progress
0x18c6f5c0 (twork0003): TS GO bit cleared
0x18c6f5c0 (twork0003): TS csr: 0xfc000000
interrupt: TRIG ERROR: no pool buffer available
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0
0x18c6f5c0 (twork0003): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_cancel: cancel >sta:clastrig2 active<
codaUpdateStatus: updating request done
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
roc_end done
codaExecute done
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty

* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):
.....
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
.....

* Sergey B. 2-apr-2010: event recorder gives error during Prestart:

file[ 0]->/raid/stage_in/clas_062414.A00<-
bosopen.c: splitted files handling ..
bosopen.c: >/raid/stage_in/clas_062414.A00<
bosopen.c: len1=30
bosopen.c: len2=30
bosopen.c: len3=14
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14
bosopen.c: checking how much disk space left in directory /raid/stage_in
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available
1100988411 blocks (1 block = 512 bytes)
bosopen.c: so we have 550493696 kbytes available space
bosopen.c: we can continue to write file.
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-
bosopen.c: system call: >checkdisk 4094<
[1] 9253
bosopen.c: system call completed
codaUpdateStatus: dbConnecting ..
ER: Not attached to ET system
er_write_thread loop ended (0 145452664)
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:ER3 paused<
UDP_standard_request >sta:ER3 paused<

Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.

* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 22 25 26 28 29
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???

* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash

* Sergey B. 14-mar-2010: during reboot dc9 gives following:
.................
ppc/bootscripts/boot_dc9
taskSpawn("TCPSERVER") returns 475458320
bind on port 5001
myname >dc9<
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded
INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
333333333333333
program
Exception current instruction address: 0x1c5
e8734
Machine Status Register: 0x00081000
Condition Register: 0x40000085
Task: 0x1c5e8e50 "ROC"
interrupt: Unconnected main interrupt 0
logTask: 3222 log messages lost.
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
..............

probably mysql connection problem

'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up.

* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing
wait: coda request >exit< in progress
indefinitely, must check

* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:

codaEnd 10
codaEnd 11
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
codaExecute done
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59319
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
codaExecute: ERROR: len=0 - do nothing
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59342
wait: coda request >< in progress
Executing >TP/1.0

Probably last string contains something, will remove printing

* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:
.....
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
.. done.
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30)
sent -1 (type 18)
[0] ERROR: Discard data until next control event
clondaq1:coda_eb>

After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,
reboot all rocs again - not it works.

In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...

* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:

.....
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_cancel: cancel >sta:EB1 booted<
codaUpdateStatus: updating request done
CODA_Init 14
2
clasprod::EB1> bind on port 5001
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=37258
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
Segmentation fault (core dumped)
clondaq1:coda_eb>

In the same time ''et_start'' complains (we have now ET system protected against CC scans):

et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure

'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already

* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:

Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e404190 "coda_pmc"

It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.

* Sergey B. 1-sep-2009: during croctest1 reboot:

INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL

second reboot went fine

* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.

* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:

..........................
taskDelay(sysClkRateGet()*5)
Args = -session clasprod -objects sc2 ROC -i
CODA_Init reached
CODA_Init 1
CODA_Init 2
sc2
CODA_Init 11
CODA_Init: objectTy >(null)<
CODA_Init 12
CODA_Init: use 'SESSION' as >clasprod<
Tcl_AppInit CALLS !!!!!
11-11
11-12
11-13
11111111111111111
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
value = 0 = 0x0
taskSpawn "TCP_SERVER",250,0,100000,tcpServer
value = 231091088 = 0xdc62b90
proconhost
value = 0 = 0x0
......................

Second reboot shows the same:

...................
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
list >clon10 clasrun 2508 9998<
machine name = clon10
user ID = 2508
group ID = 9998
.................

* Sergey B. around 6-June-2009: ec2pmc1 error message:

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait= 199 send= 8 microsec per event (nev=12755)
CALLING customized :
machine check
SRAM ERROR :
SRAM Error Address : 0x00000000.f2000504
SRAM Error Data Low : 0x006805e0
SRAM Error Data High : 0x00420000
SRAM Error Parity : 0x00000004
SRAM Error Cause : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

DAQ/Online Quick Reference

2011-12-06T14:41:44Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey '''office 5795 cell 757-869-2188 (if no responce, home 757-988-0680)'''

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
office 5795
US cell 757-869-2188
(if no response, home 757-988-0680)
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2011-12-06T14:38:33Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey '''cell 757-869-2188 (if no responce, home 757-988-0680)'''

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2011-12-06T14:32:09Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey cell 757-869-2188 (if no responce, home 757-988-0680)

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2011-12-06T14:31:19Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey cell 757-869-2188.

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2011-12-04T23:41:41Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -o "tage ROC" -i
TAGE2: coda_roc -s clasprod -o "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey cell 757-232-6221 (day time) or home 757-988-0680 (night time).

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

ERROR Book

2011-11-23T19:51:19Z

129.57.167.14:

* Sergey B. 23-nov-2011: EB1 hung, probably because if security scan:

.....
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
tcl7.4(init.tcl): invalid command name "GET"
clasprod::EB1> WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=6 address>129.57.71.28< port=47514
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
<HUNG>

* Sergey B. 7-nov-2011: dc12, error message from from rol.h (10kHz pulser):

codaUpdateStatus: updating request done
active, events so far 0
wait: coda request >go< in progress
0x3e55cd0 (twork0047): INIT_NAME: INFO: rolp->daproc = 5
0x3e55cd0 (twork0047): INFO: User Go ...
codaExecute done
SEND_BUFFER_ROC 4 (0x017f4050 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x017f4050 11)
SSS2 4 (0x017f4050 11)
SEND_BUFFER_ROC 4 (0x016bd8e0 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x016bd8e0 11)
SSS2 4 (0x016bd8e0 11)
SEND_BUFFER_ROC 4 (0x01587170 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01587170 11)
SSS2 4 (0x01587170 11)
SEND_BUFFER_ROC 4 (0x01450a00 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01450a00 11)
SSS2 4 (0x01450a00 11)
SEND_BUFFER_ROC 4 (0x0131a290 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x0131a290 11)
SSS2 4 (0x0131a290 11)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
interrupt: INFO from bb_write: NO BUFS1
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=21263(21263) heartmask=4
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
...........

Click end - ended fine (probably network was stuck ?? or proc ???).

* Sergey B. 8-jun-2011: test setup croctest2: error on croctest2 after 99M events, DMAing from single v1720 (NOTE: first test of ROL1 writing directly to the big buffer):
proc_thread: waiting= 846 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 192 processing= 460 microsec per event (nev=189)
rols_thread: waiting= 194 processing= 457 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 846 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 185 processing= 469 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 200 processing= 449 microsec per event (nev=188)
proc_thread: waiting= 847 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
net_thread: waiting= 720 sending= 0 microsec per event (nev=2218)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 451 microsec per event (nev=188)
proc_thread: waiting= 635 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
rols_thread: waiting= 199 processing= 450 microsec per event (nev=188)
proc_thread: waiting= 847 processing= 3 microsec per event (nev=188)
rols_thread: waiting= 205 processing= 450 microsec per event (nev=188)
machine check
Exception next instruction address: 0x0946e298
Machine Status Register: 0x0010b030
Condition Register: 0x80000088
Task: 0x946c670 "ROLS_LOOP"
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=54077(54077) heartmask=4
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
-> tt
1668c8 vxTaskEntry +68 : rols_loop ()
d2e5290 rols_loop +b4 : output_proc_network ()
d2e5574 output_proc_network+298: vmenrol1__init ()
946ccb8 vmenrol1__init +19c: 946cafc ()
946cb0c end +91cb68c: 946ca1c ()
946cabc end +91cb63c: 946c890 ()
946c920 end +91cb4a0: 946e298 ()
value = 0 = 0x0
->

* Sergey B. 10-feb-2011: test setup croctest2: coda_net on pmc dies:
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6559)
net_thread: waiting= 131 sending= 2 microsec per event (nev=12708)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6474)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting= 129 processing= 3 microsec per event (nev=6452)
net_thread: waiting= 133 sending= 2 microsec per event (nev=12598)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6553)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=16)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=12)
net_thread: waiting= 131 sending= 2 microsec per event (nev=12714)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6489)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=8)
proc_thread: waiting= 128 processing= 3 microsec per event (nev=6478)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
net_thread: waiting= 134 sending= 2 microsec per event (nev=12377)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=0)
proc_thread: waiting= 127 processing= 3 microsec per event (nev=6554)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=3)
proc_thread: waiting= 139 processing= 3 microsec per event (nev=6470)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting= 128 processing= 3 microsec per event (nev=6516)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
ERROR2: LINK_sized_write() returns errno=32
Broken pipe
ERROR: net_thread failed (in LINK_sized_write).
->
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 PEND 13f934 1ed0f190 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_proc proc_thread 1aacca00 115 DELAY 144bb4 1aacc900 0 10
coda_pmc coda_pmc 1e3f9720 200 DELAY 144bb4 1e3f9480 0 5
value = 0 = 0x0
-> printErrno 32
S_errno_EPIPE
value = 14 = 0xe
-> tt
Sorry, traces of my own stack begin at tt ().
value = -1 = 0xffffffff
-> memShow
invalid block at 0x1ec60000 deleted
value = -1 = 0xffffffff
-> memShow
status bytes blocks avg block max block
------ ---------- --------- ---------- ----------
current
free 421348656 56 7524083 411333840
alloc 79807344 1986 40184 -
cumulative
alloc 87490320 7392 11835 -
value = 0 = 0x0
-> usrNetStackSysPoolStatus
value = 0 = 0x0
-> usrNetStackSysPoolStatus("CODAtcpServer",0)
value = 0 = 0x0
-> usrNetStackDataPoolStatus("CODAtcpServer",0)
value = 454823200 = 0x1b1c0d20
-> usrNetStackDataPoolStatus("bla",0)
value = 454823168 = 0x1b1c0d00
-> usrNetStackDataPoolStatus("bla",0)
value = 454823040 = 0x1b1c0c80
-> memShow
status bytes blocks avg block max block
------ ---------- --------- ---------- ----------
current
free 421348528 55 7660882 411333840
alloc 79807472 1990 40104 -
cumulative
alloc 87496528 7401 11822 -
value = 0 = 0x0
->

EB looks fine, message was:
-- Got all fragments of go
BOS_encode_spec reached
BOS_encode_spec done: len=116
FFFFFFFFFFFFFFFFFF
PUT: wait while icb=8 == cbp->read=8 -------
clastest::EB5>

Tried to reset-download-prestart and got error on pmc at Download:
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 PEND 13f934 1ed0f190 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_pmc coda_pmc 1e3f9720 200 DELAY 144bb4 1e3f9480 30065 1
value = 0 = 0x0
-> Download command received
-> bignet at 0x0d3a6e68
<- bignet at 0xcd3a6e68
bignet at 0xcd3a6e68, bignet.gbigBuffer at 0xcd3a6e78 -> 0xcd3a6e78
proc_on_pmc=1, net_on_pmc=1
rocID=0
ObjInitName >_rol2_tt_testsetup__init<
data access
Exception current instruction address: 0x000d2090
Machine Status Register: 0x00003030
Data Access Register: 0x3d40001c
Condition Register: 0x40002083
Data storage interrupt Register: 0x40000000
Task: 0x1ed0f280 "tNetTask"
-> tt
14cad8 vxTaskEntry +68 : netTask ()
c3e88 netTask +70 : 209c0 ()
20a14 mgiEndLoad +cc8: 20a88 ()
20bd8 mgiEndLoad +e8c: muxReceive ()
bc638 muxReceive +13c: b002c ()
b02c4 random +358: do_protocol_with_type ()
9c920 do_protocol_with_type+2c : ipintr (1ec60e04)
9fb28 ipintr +c8c: dbbb4 ()
dbbb4 tcp_input +2968: tcp_output ()
dca24 tcp_output +5c8: mHdrClGet ()
aa940 mHdrClGet +8c : d1eb4 ()
value = 0 = 0x0
->
-> i
NAME ENTRY TID PRI STATUS PC SP ERRNO DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask excTask 1effe6c0 0 PEND 14e508 1effe5a0 3d0001 0
tLogTask logTask 1effbb20 0 PEND 14e508 1effba10 0 0
tShell shell 1eb8cb70 1 READY 145290 1eb8c750 d 0
tRlogind rlogind 1ebb3850 2 PEND 13f934 1ebb33f0 0 0
tWdbTask wdbTask 1ebaa530 3 PEND 13f934 1ebaa400 0 0
tNetTask netTask 1ed0f280 50 SUSPEND d2090 1ed0efd0 0 0
tPortmapd portmapd 1ebb1690 54 PEND 13f934 1ebb1430 3d0002 0
tFtpdTask b6064 1ebaebc0 55 PEND 13f934 1ebaea10 0 0
coda_pmc coda_pmc 1e3f9720 200 PEND 1415a0 1e3f8ab0 3d0004 0
value = 0 = 0x0
->

* Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!???:
clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod -m eb -v
et_nodename >eb<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station LEVEL3
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_station_create_at, created station ET2ER
unique name >et_start_eb_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_clasprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
ipc_init called at Fri Apr 16 11:45:20 2010
et_start: pthread_create(0x0000000e,...) done
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et INFO: et_sys_heartmonitor, kill bad process (2,22078)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

clondaq2:clasrun> cd et2_start
clondaq2:et2_start>
clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod
-m er -v
et_nodename >er<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station TAPE
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur
rent/coda/SunOS_i86pc/lib/libet_user.so
et INFO: et_conductor 2, loaded function et2et_function
et INFO: et_station_create_at, created station ET2ET10
unique name >et_start_er_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R
Tserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve
r
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established
.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl
asprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab
.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>

ipc_init called at Fri Apr 16 11:45:20 2010

et_start: pthread_create(0x0000000e,...) done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_restore_events, no events found belonging to attachment 4
et INFO: et_station_detach, done
et WARN: et_command_loop: remote client connection broken
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et INFO: et_sys_heartmonitor, kill bad process (1,8898)
et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1
et INFO: et_flush_events, read 4000 events
et INFO: et_flush_events, wrote 4000 events
et INFO: et_station_detach, make station "TAPE" (cec1d440) idle
et INFO: et_restore_events, no events found belonging to attachment 3
et INFO: et_station_detach, done
et INFO: et_sys_heartmonitor, cleanup process 1
et INFO: set_fix_nprocs, change # of ET processes from 1 to 1
et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments
et INFO: set_fix_natts, station ET2ET10 has 1 attachments
et INFO: set_fix_natts, # total attachments 3 -> 3
et INFO: set_fix_natts, proc 0 has 1 attachments

clondaq1:dac>
clondaq1:dac>
clondaq1:dac>
clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 22276) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,

DYNAMIC INFO - currently there are:
processes(4), attachments(4), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(2,3,)
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"LEVEL3" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(1,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ER" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)

ATTACHMENTS:
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
proc(1), blocked(YES)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
proc(3), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER,
LOCKED MUTEXES:

*****************************************

ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 22276) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,

DYNAMIC INFO - currently there are:
processes(4), attachments(4), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(2,3,)
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"LEVEL3" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(1,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ER" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)

ATTACHMENTS:
att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
proc(1), blocked(YES)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
proc(3), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0), att3 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, LEVEL3, ET2ER,
LOCKED MUTEXES:

*****************************************

ET SYSTEM is DEAD!
*****************************************

^C
clondaq1:dac>

clondaq2:stage_in>
clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 16127) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,

DYNAMIC INFO - currently there are:
processes(3), attachments(3), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(0), ids()
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"TAPE" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(1,2,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ET10" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class =
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)

ATTACHMENTS:
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
proc(1), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10,
LOCKED MUTEXES:

*****************************************

ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
(tcp port 0) (udp port 11111) (multicast port 11111)
(pid 16127) (lang C) (local) (period = 5 sec)

STATIC INFO - maximum of:
events(4000), event size(200000), temps(4000)
stations(10), attaches(20), procs(10), concurrency added(16)
network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,

DYNAMIC INFO - currently there are:
processes(3), attachments(3), temps(0)
stations(3), hearbeat(1)

STATIONS:
"GRAND_CENTRAL" (id = 0)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(0), ids()
input list: cnt = 4000, events in = 0
output list: cnt = 0, events out = 0

"TAPE" (id = 1)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
dynamic info
attachments: total#(2), ids(1,2,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

"ET2ET10" (id = 2)
static info
status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so, function = et2et_function, class =
dynamic info
attachments: total#(1), ids(0,)
input list: cnt = 0, events in = 0, events try = 0
output list: cnt = 0, events out = 0

LOCAL USERS:
process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)

ATTACHMENTS:
att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
proc(0), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
proc(1), blocked(NO)
events: make(0), get(0), put(0), dump(0)
att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
proc(2), blocked(NO)
events: make(0), get(0), put(0), dump(0)

EVENTS OWNED BY:
system (4000), att0 (0), att1 (0), att2 (0),

EVENT RATE of GC = 0 events/sec

CREATING STATIONS:
IDLE STATIONS:
STATION CHAIN: GRAND_CENTRAL, TAPE, ET2ET10,
LOCKED MUTEXES:

*****************************************

ET SYSTEM is DEAD!
*****************************************

^C
clondaq2:stage_in>

* Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem:

wait: coda request >end< in progress
0x916e80 (twork0028): TS GO bit cleared
0x916e80 (twork0028): TS csr: 0xfc070000
0x916e80 (twork0028): TS csr2: 0xffff1c1f
interrupt: TRIG ERROR: no pool buffer available
0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0
0x916e80 (twork0028): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done

* Sergey B. 14-apr-2010: message from runcontrol during cancel-reset:
Get all components failed: Lost connection to MySQL server during query

* Sergey B. 6-apr-2010: EB waiting for following ROCs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

Files produced:

clon10:ccscans> ll | grep Apr
drwxrwxrwx 2 clasrun onliners 4096 Apr 2 16:23 ./
drwxr-xr-x 163 clasrun 100 73728 Apr 6 2010 ../
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_cc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:09 good_clastrig2*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest10*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:25 good_croctest2*
-rwxrwxrwx 1 boiarino clas 34 Apr 2 16:21 good_croctest3*
-rwxrwxrwx 1 baturin clas 34 Apr 2 13:26 good_ctoftest1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc10*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc11*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc5*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc6*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc7*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_dc8*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_dc9*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_ec2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_ec4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_polar*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_sc1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_sc2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_scaler1*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler3*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:26 good_scaler4*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage2*
-rwxrwxrwx 1 clasrun onliners 34 Apr 6 20:08 good_tage3*
clon10:ccscans> more good_dc7
accepted link from >129.57.68.80<
clon10:ccscans> more good_dc8
accepted link from >129.57.68.74<
clon10:ccscans>

On 'Reset' EB crashed, outputs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 21 22 25 26 28 29 30

CODAtcpServer: start work thread
befor: socket=5 address>129.57.167.14< port=34468
wait: coda request >prestart< in progress
codaExecute reached, message >exit<, len=4
codaExecute: 'exit' transition
debShutdownBuild 1
cb_delete roc=0
cb_delete roc=1
cb_delete roc=2
cb_delete roc=3
cb_delete roc=4
cb_delete roc=5
cb_delete roc=6
cb_delete roc=7
cb_delete roc=8
cb_delete roc=9
cb_delete roc=10
cb_delete roc=11
cb_delete roc=12
cb_delete roc=13
cb_delete roc=14
cb_delete roc=15
cb_delete roc=16
cb_delete roc=17
cb_delete roc=18
cb_delete roc=19
cb_delete roc=20
cb_delete roc=21
cb_delete roc=22
cb_delete roc=23
cb_delete roc=24
cb_delete roc=25
cb_delete roc=26
cb_delete roc=27
cb_delete roc=28
cb_delete roc=29
cb_delete roc=30
cb_delete roc=31
debShutdownBuild 2
Stop waiting for the rocs because of force_end condition
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!
[0] et_event_new() 1
[0] EVENT_ENCODE_SPEC 2 ..
BOS_encode_spec reached
BOS_encode_spec done: len=116
[0] .. done
[0] handle_build_cleanup
[0] flush ET system
[0] detached from ET
[0] remove mutex locks and shutdown fifos
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[00] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[00] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[00] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[01] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[01] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[01] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[02] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[02] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[02] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[03] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[03] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[03] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[04] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[04] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[04] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[05] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[05] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[05] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[06] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[06] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[06] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[07] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[07] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[07] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[08] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[08] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[08] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[09] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[09] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[09] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[10] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[10] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[10] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[11] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[11] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[12] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[12] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[13] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[13] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[14] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[15] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[16] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[17] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[18] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[19] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[20] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[21] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[22] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[23] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[24] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[25] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[26] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[27] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[28] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[29] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[30] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[31] = 0
CCCCCCCCCC: 0
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] build thread exiting: 1 1
[0] The implementation has detected that the value specified by threa
d does not refer to a joinable thread.
wait: coda request >exit< in progress
polling_routine ================!!!!!!!!!!!!!!!!!!=====
cancel building threads
=c=============================================
=c=============================================
=c=============================================
=c====
>>>>> close link from >dc1< link=0x081c2af0
debCloseLink: theLink=0x081c2af0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc2< link=0x081c4a90
debCloseLink: theLink=0x081c4a90 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc3< link=0x081c4b28
debCloseLink: theLink=0x081c4b28 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc4< link=0x081c14f8
debCloseLink: theLink=0x081c14f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc5< link=0x081c1590
debCloseLink: theLink=0x081c1590 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc6< link=0x081c1628
debCloseLink: theLink=0x081c1628 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc8< link=0x081c16c0
debCloseLink: theLink=0x081c16c0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >cc1< link=0x081c1a08
debCloseLink: theLink=0x081c1a08 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >sc1< link=0x081c1aa0
debCloseLink: theLink=0x081c1aa0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec1< link=0x081c1b38
debCloseLink: theLink=0x081c1b38 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec2< link=0x081c1bd0
debCloseLink: theLink=0x081c1bd0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler2< link=0x681d95c8
debCloseLink: theLink=0x681d95c8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler3< link=0x681d9660
debCloseLink: theLink=0x681d9660 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler4< link=0x681d96f8
debCloseLink: theLink=0x681d96f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec4< link=0x00000001
debCloseLink: theLink=0x00000001 -> closing
Segmentation fault
clondaq1:coda_eb>

DC7:

codaExecute done
wait: coda request >download prod< in progress
codaExecute reached, message >prestart<, len=8
codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410567520
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc7<
LINK_constructor_C: set name to >dc7->EB1<
3121: dbsock=410567520
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda req<<<
uest >fields [pr2estart] >>>< in prodowngress
<<<
fields [3] >>>37683<<<
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x18772230 (twork0028INFO: Entering Us): er
Prestart 2
INIT_NAME: rolp->daproc =
trans_rol2:2
adr(nddl)=1db60750 NWBOS=16384
uthbook1: WARN id=1047 already exist - will be replaced
uthbook1: free histogram buffer ... done.
RAW=0 PROFILE=1 NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x18772230INFO: User Prest (art 2 executed
twork0028): CTRIGRSS: set handler and done,
code=1
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4130) returns 490106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a580) returns 490106296
0x18772230 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x18772230 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc
0x1882d864 0x00000001 0x1882b0c8
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1 = FFFFF000
Fastbus Status Register2 = FFFF0000
Fastbus Last Primary Addr = 00000019
spds_mask1: 0xfdfbf0
0x18772230_tdc1877 ===> (4twork0028wait: co)
: Use 32-bit Ram List
da request >firsprestartt< in progre csr0=ss
100 (1100) csr18=1364 csr1=6400001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0=100 (900) csr18=13
bignetptr=0x641d366fe0 csr1= offset=0x00bigp
rocptr=0x000001d366db80 offset=0x6400000000
00001b
bigproc at 0x
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =
==>1d366dc818 -> 0x1d366dc8
proc_thread reacheUDP_cancel: cancel >inf:d
c7 sys 0, mask 1d
<
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached
<
wait: cproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot# 1872 1875 1881(2) 1877(2) 1877S(2)
[ 0] 0 0 0 0bignetptr=0x
1d366fe00 offset=0x0000000 0
0 0 0
[ 1] 0 0 0 0 0 0 0
0
[ 2] 0 0 0 0 0 0 0
0
[ 3] 0 0 0 0 0 0 0
0
[ 4] 0 0 0 0 1 0 0
0
[ 5] 0 0 0 0 1 0 0
0
[ 6] 0 0 0 0 1 0 0
0
[ 7] 0 0 0 0 1 0 0
0
[ 8] 0 0 0 0 1 0 0
0
[ 9] 0 0 0 0 1 2 0
0
[10] wait: coda reques0t > prestart
0< in progress
0 0 0 0 0 0
[11] 0 0 0 0 1 0 0
0
[12] 0 0 0 0 1 0 0
0
[13] 0 0 0 0 1 0 0
0
[14] 0 0 0 0 1 0 0
0
[15] 0 0 0 0 1 0 0
0
[16] 0 0 0 0 1 2 0
0
[17] 0 0 0 0 0 0 0
0
[18] 0 0 0 0 1 0 0
0
[19] 0 0 0 0 1 0 0
0
[20] bignet at 0x 1d366fe00, bignet.gbig
Buffer at 0x 1d366ff00 -> 0x1d366ff0
0 0 1 0 0 0
[21] 0 0 0 0 1 0 0
0
[22] 0 0 0 0 1 0 0
0
[23] 0 0 0 0 1 2 0
0
[24] 0 0 0 0 0 0 0
0
[25] 0 0 0 0 0 0 0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC07<
0x18772230 (twork0028111): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <
<<<<<<<<<<<<<<<<
s: >UPDATE process SET state='paused' WHERE
name='dc7'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_cancel: cancel >sta:dc7 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
->
->

DC8:

codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410505056
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
WHERE name='dc8'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc8<
LINK_constructor_C: set name to >dc8->EB1<
3121: dbsock=410505056
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda re<<<
quest >fields [p2restart] >>>< in prupogress
<<<
fields [3] >>>37772<<<
parsing results: type=>TCP< host=>clondaq1-d
aq1< state=>up< port=>37772< -> 37772
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(
0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129
.57.68.22 port 37772
DB command >UPDATE links SET state='up' WHER
E name='dc8->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x187ef280INFO: Entering U (ser Prestart 2
twork0028
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da
proc = 1d2b60750
NWBOS=16384
uthbook1: WARN id=1048 already exist - will
be replaced
uthbook1: free histogram buffer ... done.
RAW=0 PROFILE=1 NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x187ef280 (twork0028INFO: User Pr): estart
2 executedCTRIGRSS: set handler and done, co
de=
1
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac
0x187704d0 0x1876fdf0 0x00000001 0x1876fde8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4110) returns 490
106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a560) returns 49
0106296
0x187ef280 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x187ef280 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x187ef280 (twork0028): CTRIGRSA: set handle
r and done, code=1
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c
0x1882d1e4 0x00000001 0x1882aa48
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1 = FFFFF000
Fastbus Status Register2 = FFFF0000
Fastbus Last Primary Addr = wait: coda req00
0000uest >19presta
rtspds_mask1: 0x< ifdfbf0n progress
0x187ef280_tdc1877 === (>twork0028): Use 32-
bit Ram List
4 first csr0=100 (1100) csr18=1364 csr1=6400
001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=136
4 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f
e0100 offset=0x0000000 (0
900) csr18=1364bigprocptr=0x csr1=1d366db864
00001b offset=0x
00000000
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.
gbigBuffer at 0x181d3UDP_cancel: cancel >inf
:dc8 sys 0, mask 166dc8<
wait: coda rUDP_cancel: cancel >inf:dc8 sys
0, mask 1 -> 0x<
1d366dc8eq
firstproc_thread reached
uest >proc_thread reached
csr0=proc_thread reached
prproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
100estart (< in prog1100ress
) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot# 1872 bignetptr=0x1875 181d366fe081
(2) 1877( offset=0x2) 1877S(00000002)
0
[ 0] 0 0 0 0 0 0 0
0
[ 1] 0 0 0 0 0 0 0
0
[ 2] 0 0 0 0 0 0 0
0
[ 3] 0 0 0 0 0 0 0
0
[ 4] 0 0 0 0 1 0 0
0
[ 5] 0 0 0 0 1 0 0
0
[ 6] 0 0 0 0 1 0 0
0
[ 7] 0 0 0 0 1 0 0
0
[ 8] 0 0 0 0 1 0 0
0
[ 9] 0 0 0 0 1 2 0
0
[10] wait: coda request0 > prestart
0< i n progress
0 0 0 0 0 0
[11] 0 0 0 0 1 0 0
0
[12] 0 0 0 0 1 0 0
0
[13] 0 0 0 0 1 0 0
0
[14] 0 0 0 0 1 0 0
0
[15] 0 0 0 0 1 0 0
0
[16] 0 0 0 0 1 2 0
0
[17] 0 0 0 0 0 0 0
0
[18] 0 0 0 0 1 0 0
0
[19] 0 bignet at 0x 1d366fe00, bigne
t.gbigBuffer at 0x 1d366ff00 -> 0x1d366
ff0
0 1 0 0 0
[20] 0 0 0 0 1 0 0
0
[21] 0 0 0 0 1 0 0
0
[22] 0 0 0 0 1 0 0
0
[23] 0 0 0 0 1 2 0
0
[24] 0 0 0 0 0 0 0
0
[25] 0 0 0 0 0 0 0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC08<
0x187ef280 (twork0028111
): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000
0000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state=
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<
<<<<<<<<<<<<
ERE name='dc8'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_cancel: cancel >sta:dc8 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
.....

* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:
........
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
3123: dbsock=59553504
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<
====1==== !!! object->runType=47
token_interval=64
object->name >polar<
big0.proc_on_pmc=0, big0.net_on_pmc=0
big1.proc_on_pmc=0, big1.net_on_pmc=0
name selected
nrow=1
download: name >polar<
code selected
nrow=1
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
set this_roc_id = 25, rocId() can be called from now on
ObjInitName >_moller__init<
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
INFO: >_moller__init()< routine found
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded
allocate 200 buffers with length 65536 bytes
partCreate1() reached
partCreate1(): pPart=0x008e7570
listInit: cleanup 24 bytes starting from address 0x008e75a0
partCreate1() done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<
0x81c7e0 (twork0004): Init - Done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
Exception at interrupt level:
Exception next instruction address: 0xfff901f0
Fixed Point Register: 0x00000012
Condition Register: 0x40482044
Regs at 0x64e498
Press any key to stop auto-boot...
2
[VxWorks Boot]:

It suppose to be as following:
.....
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
rol1: downloading DDL table ...
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<
0x91ae30 (twork0001): Init - Done
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1
wait: coda request >download prod< in progress
adr1(nddl_)=0x037b69c8
N name (nname) fmt (nfmt) ncol
[ 1] >PTRN< (4) >B32< (3) 1
[ 2] >PSYN< (4) >B32< (3) 1
[ 3] >RC00< (4) >B32< (3) 1
[ 4] >RC01< (4) >B32< (3) 1
.....

* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:
-> ts2status
CSR 1 (0x0):
Go : 1
Pause on Next scheduled Sync : 0
Sync and Pause : 0
Initiate Sync Event : 0
Initiate Program 1 Event : 0
Initiate Program 2 Event : 1
Enable Level 1 (drives outputs) : 1
Override Inhibit : 0
Test Mode : 0
Reserved : 0
Reset : 0
Initialize : 0
Sync Event occurred : 0
Program 1 Event occurred : 1
Program 2 Event occurred : 0
Late Fail occurred : 0
Inhibit occurred : 1
Write FIFO error occurred : 0
Read FIFO error occurreds : 1
CSR 2 (0x4):
Enable Scheduled Sync : 1
Use Clear Permit Timer : 1
Use Front Busy Timer : 1
Use Clear Hold Timer : 1
Use External Front Busy : 1
Lock ROC Branch 1 : 0
Lock ROC Branch 2 : 0
Lock ROC Branch 3 : 0
Lock ROC Branch 4 : 0
Lock ROC Branch 5 : 0
Enable Program 1 front panel input : 1
Enable Interrupt : 1
Enable local ROC (branch 5) : 1
Trigger Control Register (0x8): 0x00001fff
ROC Enable Register (0xc) val=0xf07fff7f:
Branch 1: 0x7f bits: 01111111
Branch 2: 0xff bits: 11111111
Branch 3: 0x7f bits: 01111111
Branch 4: 0xf0 bits: 11110000
Synchronization Interval Register (0x10): 1000
Trigger Word Count Register (0x14): 0
Trigger Data Register (0x18): 256
Local ROC (Branch 5) Data Register (0x1c): 60
Synchronization Flag : 0
Late Fail Flag : 0
ROC code : 15
Input Trigger Prescale Registers:
Input 1 Prescale Factor : 0
Input 2 Prescale Factor : 0
Input 3 Prescale Factor : 0
Input 4 Prescale Factor : 0
Input 5 Prescale Factor : 0
Input 6 Prescale Factor : 0
Input 7 Prescale Factor : 0
Input 8 Prescale Factor : 0
Clear Permit Timer Register = 0
Level2 Accept Timer Register = 83
Level3 Accept Timer Register = 0
Front Busy Timer Register = 325
Clear Hold Timer Register = 50
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0
ROC Acknowledge Status Register (0x60): val=0x0
Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)
Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)
Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)
Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)
State Register (0x6C):
Level 1 Accept : 0
Start Level 2 Trigger : 0
Level 2 Pass Latched : 0
Level 2 Fail Latched : 0
Level 2 Accept : 0
Start Level 3 Trigger : 0
Level 3 Pass Latched : 0
Level 3 Fail Latched : 0
Level 3 Accept : 0
Clear : 0
Front End Busy (external) : 0
External Inhibit : 0
Latched Trigger : 0
TS Busy : 1
TS Active : 1
TS Ready : 0
Main Sequencer Active : 0
Synchronization Sequencer Active : 0
Program 1 Event Sequencer Active : 1
Program 2 Event Sequencer Active : 0
Event Count (0xc8): 135
Live 1 Count (0xcc): 54866
Live 2 Count (0xd0): 9290375
value = 1 = 0x1
-> exit

clasprod::clastrig2> clastrig2 statistics
200 0 0 0

* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:

interrupt: Program Events 1-2: 247 0
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)
proc_thread: waiting= 3743 processing= 30 microsec per event (nev=404)
rols_thread: waiting= 754 processing= 21 microsec per event (nev=454)
wait: coda request >go< in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
roc_end reached ??
roc_end reached !!
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <
interrupt: inputs 1-6: 0 0 0 0 0 0
interrupt: inputs 7-12: 0 0 339 0 0 0
interrupt: Program Events 1-2: 172 0
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)
wait: coda request >end< in progress
0x18c6f5c0 (twork0003): TS GO bit cleared
0x18c6f5c0 (twork0003): TS csr: 0xfc000000
interrupt: TRIG ERROR: no pool buffer available
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0
0x18c6f5c0 (twork0003): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_cancel: cancel >sta:clastrig2 active<
codaUpdateStatus: updating request done
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
roc_end done
codaExecute done
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty

* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):
.....
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
.....

* Sergey B. 2-apr-2010: event recorder gives error during Prestart:

file[ 0]->/raid/stage_in/clas_062414.A00<-
bosopen.c: splitted files handling ..
bosopen.c: >/raid/stage_in/clas_062414.A00<
bosopen.c: len1=30
bosopen.c: len2=30
bosopen.c: len3=14
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14
bosopen.c: checking how much disk space left in directory /raid/stage_in
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available
1100988411 blocks (1 block = 512 bytes)
bosopen.c: so we have 550493696 kbytes available space
bosopen.c: we can continue to write file.
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-
bosopen.c: system call: >checkdisk 4094<
[1] 9253
bosopen.c: system call completed
codaUpdateStatus: dbConnecting ..
ER: Not attached to ET system
er_write_thread loop ended (0 145452664)
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:ER3 paused<
UDP_standard_request >sta:ER3 paused<

Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.

* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:
7 9 10 11 17 20 22 25 26 28 29
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???

* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash

* Sergey B. 14-mar-2010: during reboot dc9 gives following:
.................
ppc/bootscripts/boot_dc9
taskSpawn("TCPSERVER") returns 475458320
bind on port 5001
myname >dc9<
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded
INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
333333333333333
program
Exception current instruction address: 0x1c5
e8734
Machine Status Register: 0x00081000
Condition Register: 0x40000085
Task: 0x1c5e8e50 "ROC"
interrupt: Unconnected main interrupt 0
logTask: 3222 log messages lost.
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
..............

probably mysql connection problem

'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up.

* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing
wait: coda request >exit< in progress
indefinitely, must check

* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:

codaEnd 10
codaEnd 11
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
codaExecute done
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59319
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
codaExecute: ERROR: len=0 - do nothing
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59342
wait: coda request >< in progress
Executing >TP/1.0

Probably last string contains something, will remove printing

* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:
.....
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
.. done.
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30)
sent -1 (type 18)
[0] ERROR: Discard data until next control event
clondaq1:coda_eb>

After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,
reboot all rocs again - not it works.

In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...

* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:

.....
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_cancel: cancel >sta:EB1 booted<
codaUpdateStatus: updating request done
CODA_Init 14
2
clasprod::EB1> bind on port 5001
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=37258
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
Segmentation fault (core dumped)
clondaq1:coda_eb>

In the same time ''et_start'' complains (we have now ET system protected against CC scans):

et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure

'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already

* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:

Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e404190 "coda_pmc"

It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.

* Sergey B. 1-sep-2009: during croctest1 reboot:

INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL

second reboot went fine

* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.

* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:

..........................
taskDelay(sysClkRateGet()*5)
Args = -session clasprod -objects sc2 ROC -i
CODA_Init reached
CODA_Init 1
CODA_Init 2
sc2
CODA_Init 11
CODA_Init: objectTy >(null)<
CODA_Init 12
CODA_Init: use 'SESSION' as >clasprod<
Tcl_AppInit CALLS !!!!!
11-11
11-12
11-13
11111111111111111
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
value = 0 = 0x0
taskSpawn "TCP_SERVER",250,0,100000,tcpServer
value = 231091088 = 0xdc62b90
proconhost
value = 0 = 0x0
......................

Second reboot shows the same:

...................
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
list >clon10 clasrun 2508 9998<
machine name = clon10
user ID = 2508
group ID = 9998
.................

* Sergey B. around 6-June-2009: ec2pmc1 error message:

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait= 199 send= 8 microsec per event (nev=12755)
CALLING customized :
machine check
SRAM ERROR :
SRAM Error Address : 0x00000000.f2000504
SRAM Error Data Low : 0x006805e0
SRAM Error Data High : 0x00420000
SRAM Error Parity : 0x00000004
SRAM Error Cause : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

DAQ/Online Quick Reference

2011-11-23T14:58:19Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -s "tage ROC" -i
TAGE2: coda_roc -s clasprod -s "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey cell 757-232-6221 (day time) or home 757-988-0680 (night time).

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2011-11-23T14:57:28Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
TAGE: coda_roc -s clasprod -s "tage ROC" -i
TAGE2: coda_roc -s clasprod -s "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey home 757-988-0680.

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

DAQ/Online Quick Reference

2011-11-23T14:57:05Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and type command:
for tage: coda_roc -s clasprod -s "tage ROC" -i
for tage2: coda_roc -s clasprod -s "tage2 ROC" -i

If it does not help, page DAQ pager. If no reponce during 10 minutes, call Sergey home 757-988-0680.

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

GE XVB601

2011-11-22T17:38:09Z

129.57.167.14: /* List of boards */

Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]

== Module manufacturer information ==

Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]

Documentation:
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_xvb601_ds_gfa1299.pdf Datasheet]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_xvb601_hr_500-930007815-000_E.pdf Hardware Reference]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_522-9300800627-000_A.book.pdf Transition Module Installation Guide]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_Configuration_Guide_XVB601.pdf Product Configurations]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_Ruggedization_Levels_GFA-926a.pdf Ruggedization]

== List of boards ==

* [01] 00:20:38:04:23:5E / 129.57.68.88 / croctest1

* [02] 00:20:38:04:23:60 / 129.57.68.90 / croctest3 (curently: 129.57.68.79 / dvcs2)

* [03] 00:20:38:04:29:2E / 129.57.68.68 / tage

* [04] 00:20:38:04:23:64 / 129.57.68.86 / tage2

== Initial settings ==

Install new board into VME crate. Connect ethernet to the upper LAN port, rs232 9600N1 serial line (optional), VGA monitor and USB keyboard. Turn on power, setup page should appear (for future reboots hold '''DEL''' key to enter setup). Goto '''Advanced->Onboard Device''', set memory size to '''512MB''' and enable '''Onboard Gigabit LAN 1'''. Goto '''Advanced->Serial Port Console Redirection''', enable '''COM1 Console Redirection''' and in '''Console Redirection Settings''' set '''Bits per second=9600'''. Do '''Save & Exit'''.

Sergey Boyarinov only: open web browser from '''clonweb''' and goto '''jnet.jlab.org'''. Login using CUE login and password. Click on '''Machine Registration''' button. Click '''Add Device'''. Fill in a form, for example:
Connection Type: wired
MAC address: 00:20:38:04:23:64
VLAN: VLAN 568 : Hall B Data Acquisition <choose from the list>
Hostname: tage2.jlab.org
IP address: 129.57.68.86 <unselect 'Automatically Assign Statis IP>
JLAB Username: boiarino
Has PII: no
Machine Type: other
Property Tag: F
Click '''Submit'''. Following message must shows up:
The device (00:20:38:04:23:64) has been successfully added to the registration table.
-----------------
The device (00:20:38:04:23:64) has been successfully added to the assignment table.
-----------------
(DHCP) This entry will be added to DHCP the next time the server restarts (about 10 seconds).
-----------------
hallb-rtr.jlab.org ethernet8/9 is not an access port, blip failed...

Add IP name to the [[clonfs1]] export list, if not done yet.

Recycle power. Messages on monitor must show successful DHCP connection. Go to the following section to setup TFTP etc.

== Software structure description ==

* following is installed in ''/tftpboot'' area on clon10 server:

linux-install/Centos5PAE_new/initrd.img
/initrd.img.backup
/vmlinuz
/msgs/boot.msg
/expert.msg
/general.msg
/param.msg
/rescue.msg
/snake.msg
/pxelinux.0
/pxelinux.cfg/81394458 -> croctest1
/81394459 -> croctest2
/croctest1
/croctest2
/default
/pxeos.xml

Every client must has a file in ''/pxelinux.cfg/'' area with the name containing IP address in HEX format, for example croctest1's IP is 129.57.68.88, so file name must be 81394458. It can be a symbolic link. The contents of the file for croctest1:

default Centos5PAE_new
label Centos5PAE_new
kernel Centos5PAE_new/vmlinuz
append initrd=Centos5PAE_new/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=129.57.167.16:/vol/diskless/Centos5 ramdisk_size=31809 ETHERNET=eth0 SNAPSHOT=croctest1.jlab.org NISDOMAIN=CCCHP acpi=force vga=0x305 vmalloc=256MB console=ttyS0,9600

All settings in that file are more or less generic. To customize it for another client only ''SNAPSHOT=croctest1.jlab.org'' must be changed. Statement ''NFSROOT=129.57.167.16:/vol/diskless/Centos5'' defines the fileserver IP and directory where diskless Linux is located, in our case it is ''clonfs1''. Statement ''console=ttyS0,9600'' force Centos5 to send boot messages to the com port so they can be monitored using ''tsconnect''.

* diskless Linux is installed on ''clonfs1'' fileserver in ''/vol/diskless/Centos5'' volume. That volume is mounted on the clonpc3 RHEL machine as well, and all Centos5 installation was performed on clonpc3, as it described in EXPERT PART section below. Directory ''/diskless/Centos5/root'' will be mounted as ''/'' level on all diskless clients, so that part is generic for all clients. Custom areas in ''/diskless/Centos5/snapshot'' will be client-specific, for every new client new directory with appropriate name must be creates there using templete tar ball, for example:
tar xvf templete.jlab.org.tar
mv templete.jlab.org croctest1.jlab.org
* File ''files'' contains the list of all files/directories to be used from snapshot area rather then from root area, that file was created during Centos5 installation. File ''files.custom'' can be used to specify additional files/directories, not in use right now.

== Customizing kernel installed in ''/diskless/Centos5/root'' ==

After expert installation is done, following changes must be applied. They are similar to the RHEL installation on clon cluster PCs and servers:

* ''/usr/bin/system-config-network''

* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''

'''NOTE''': to add new used you may use command ''useradd'', for example
useradd -u 6246 -g nagios -d /home/nagios -c "Nagios" -s /bin/tcsh nagios
(add flag '-m' if want to force home directory creation).

* fix ''/etc/nsswitch.conf''

* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')

* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')

* ''/etc/init.d/ypbind start''

* ''chkconfig ypbind on''

* set RHEL-style [[automounts]]

* ''/etc/init.d/autofs restart''

* fix symbolic links to /apps, /home, /usr/local etc

To install updates, ssh to clonpc3 as ''root'' and start chroot environment: ''chroot /misc/diskless/Centos5/root''. After that you can use ''yum'', for example:
yum install gcc-c++
yum install mysql-devel
yum install compat-gcc-34-g77
yum install libXpm-devel
yum install libXtst
yum install java

In case of yum, rpm etc problems usually you have to rebuild dictionary, google the error message.

== EXPERT PART: setting everything from scratch using clonpc3 ==

GE XVB601

2011-11-22T17:37:54Z

129.57.167.14: /* List of boards */

Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]

== Module manufacturer information ==

Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]

Documentation:
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_xvb601_ds_gfa1299.pdf Datasheet]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_xvb601_hr_500-930007815-000_E.pdf Hardware Reference]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_522-9300800627-000_A.book.pdf Transition Module Installation Guide]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_Configuration_Guide_XVB601.pdf Product Configurations]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_Ruggedization_Levels_GFA-926a.pdf Ruggedization]

== List of boards ==

* [01] 00:20:38:04:23:5E / 129.57.68.88 / croctest1

* [02] 00:20:38:04:23:60 / 129.57.68.90 / croctest3 (currently: 129.57.68.79 / dvcs2)

* [03] 00:20:38:04:29:2E / 129.57.68.68 / tage

* [04] 00:20:38:04:23:64 / 129.57.68.86 / tage2

== Initial settings ==

Install new board into VME crate. Connect ethernet to the upper LAN port, rs232 9600N1 serial line (optional), VGA monitor and USB keyboard. Turn on power, setup page should appear (for future reboots hold '''DEL''' key to enter setup). Goto '''Advanced->Onboard Device''', set memory size to '''512MB''' and enable '''Onboard Gigabit LAN 1'''. Goto '''Advanced->Serial Port Console Redirection''', enable '''COM1 Console Redirection''' and in '''Console Redirection Settings''' set '''Bits per second=9600'''. Do '''Save & Exit'''.

Sergey Boyarinov only: open web browser from '''clonweb''' and goto '''jnet.jlab.org'''. Login using CUE login and password. Click on '''Machine Registration''' button. Click '''Add Device'''. Fill in a form, for example:
Connection Type: wired
MAC address: 00:20:38:04:23:64
VLAN: VLAN 568 : Hall B Data Acquisition <choose from the list>
Hostname: tage2.jlab.org
IP address: 129.57.68.86 <unselect 'Automatically Assign Statis IP>
JLAB Username: boiarino
Has PII: no
Machine Type: other
Property Tag: F
Click '''Submit'''. Following message must shows up:
The device (00:20:38:04:23:64) has been successfully added to the registration table.
-----------------
The device (00:20:38:04:23:64) has been successfully added to the assignment table.
-----------------
(DHCP) This entry will be added to DHCP the next time the server restarts (about 10 seconds).
-----------------
hallb-rtr.jlab.org ethernet8/9 is not an access port, blip failed...

Add IP name to the [[clonfs1]] export list, if not done yet.

Recycle power. Messages on monitor must show successful DHCP connection. Go to the following section to setup TFTP etc.

== Software structure description ==

* following is installed in ''/tftpboot'' area on clon10 server:

linux-install/Centos5PAE_new/initrd.img
/initrd.img.backup
/vmlinuz
/msgs/boot.msg
/expert.msg
/general.msg
/param.msg
/rescue.msg
/snake.msg
/pxelinux.0
/pxelinux.cfg/81394458 -> croctest1
/81394459 -> croctest2
/croctest1
/croctest2
/default
/pxeos.xml

Every client must has a file in ''/pxelinux.cfg/'' area with the name containing IP address in HEX format, for example croctest1's IP is 129.57.68.88, so file name must be 81394458. It can be a symbolic link. The contents of the file for croctest1:

default Centos5PAE_new
label Centos5PAE_new
kernel Centos5PAE_new/vmlinuz
append initrd=Centos5PAE_new/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=129.57.167.16:/vol/diskless/Centos5 ramdisk_size=31809 ETHERNET=eth0 SNAPSHOT=croctest1.jlab.org NISDOMAIN=CCCHP acpi=force vga=0x305 vmalloc=256MB console=ttyS0,9600

All settings in that file are more or less generic. To customize it for another client only ''SNAPSHOT=croctest1.jlab.org'' must be changed. Statement ''NFSROOT=129.57.167.16:/vol/diskless/Centos5'' defines the fileserver IP and directory where diskless Linux is located, in our case it is ''clonfs1''. Statement ''console=ttyS0,9600'' force Centos5 to send boot messages to the com port so they can be monitored using ''tsconnect''.

* diskless Linux is installed on ''clonfs1'' fileserver in ''/vol/diskless/Centos5'' volume. That volume is mounted on the clonpc3 RHEL machine as well, and all Centos5 installation was performed on clonpc3, as it described in EXPERT PART section below. Directory ''/diskless/Centos5/root'' will be mounted as ''/'' level on all diskless clients, so that part is generic for all clients. Custom areas in ''/diskless/Centos5/snapshot'' will be client-specific, for every new client new directory with appropriate name must be creates there using templete tar ball, for example:
tar xvf templete.jlab.org.tar
mv templete.jlab.org croctest1.jlab.org
* File ''files'' contains the list of all files/directories to be used from snapshot area rather then from root area, that file was created during Centos5 installation. File ''files.custom'' can be used to specify additional files/directories, not in use right now.

== Customizing kernel installed in ''/diskless/Centos5/root'' ==

After expert installation is done, following changes must be applied. They are similar to the RHEL installation on clon cluster PCs and servers:

* ''/usr/bin/system-config-network''

* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''

'''NOTE''': to add new used you may use command ''useradd'', for example
useradd -u 6246 -g nagios -d /home/nagios -c "Nagios" -s /bin/tcsh nagios
(add flag '-m' if want to force home directory creation).

* fix ''/etc/nsswitch.conf''

* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')

* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')

* ''/etc/init.d/ypbind start''

* ''chkconfig ypbind on''

* set RHEL-style [[automounts]]

* ''/etc/init.d/autofs restart''

* fix symbolic links to /apps, /home, /usr/local etc

To install updates, ssh to clonpc3 as ''root'' and start chroot environment: ''chroot /misc/diskless/Centos5/root''. After that you can use ''yum'', for example:
yum install gcc-c++
yum install mysql-devel
yum install compat-gcc-34-g77
yum install libXpm-devel
yum install libXtst
yum install java

In case of yum, rpm etc problems usually you have to rebuild dictionary, google the error message.

== EXPERT PART: setting everything from scratch using clonpc3 ==

GE XVB601

2011-11-22T17:37:19Z

129.57.167.14: /* List of boards */

Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]

== Module manufacturer information ==

Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]

Documentation:
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_xvb601_ds_gfa1299.pdf Datasheet]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_xvb601_hr_500-930007815-000_E.pdf Hardware Reference]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_522-9300800627-000_A.book.pdf Transition Module Installation Guide]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_Configuration_Guide_XVB601.pdf Product Configurations]
[https://clonwiki.jlab.org/wiki/clondocs/Docs/ge_Ruggedization_Levels_GFA-926a.pdf Ruggedization]

== List of boards ==

* [01] 00:20:38:04:23:5E / 129.57.68.88 / croctest1

* [02] 00:20:38:04:23:60 / 129.57.68.90 / croctest3
* [02] 00:20:38:04:23:60 / 129.57.68.79 / dvcs2

* [03] 00:20:38:04:29:2E / 129.57.68.68 / tage

* [04] 00:20:38:04:23:64 / 129.57.68.86 / tage2

== Initial settings ==

Install new board into VME crate. Connect ethernet to the upper LAN port, rs232 9600N1 serial line (optional), VGA monitor and USB keyboard. Turn on power, setup page should appear (for future reboots hold '''DEL''' key to enter setup). Goto '''Advanced->Onboard Device''', set memory size to '''512MB''' and enable '''Onboard Gigabit LAN 1'''. Goto '''Advanced->Serial Port Console Redirection''', enable '''COM1 Console Redirection''' and in '''Console Redirection Settings''' set '''Bits per second=9600'''. Do '''Save & Exit'''.

Sergey Boyarinov only: open web browser from '''clonweb''' and goto '''jnet.jlab.org'''. Login using CUE login and password. Click on '''Machine Registration''' button. Click '''Add Device'''. Fill in a form, for example:
Connection Type: wired
MAC address: 00:20:38:04:23:64
VLAN: VLAN 568 : Hall B Data Acquisition <choose from the list>
Hostname: tage2.jlab.org
IP address: 129.57.68.86 <unselect 'Automatically Assign Statis IP>
JLAB Username: boiarino
Has PII: no
Machine Type: other
Property Tag: F
Click '''Submit'''. Following message must shows up:
The device (00:20:38:04:23:64) has been successfully added to the registration table.
-----------------
The device (00:20:38:04:23:64) has been successfully added to the assignment table.
-----------------
(DHCP) This entry will be added to DHCP the next time the server restarts (about 10 seconds).
-----------------
hallb-rtr.jlab.org ethernet8/9 is not an access port, blip failed...

Add IP name to the [[clonfs1]] export list, if not done yet.

Recycle power. Messages on monitor must show successful DHCP connection. Go to the following section to setup TFTP etc.

== Software structure description ==

* following is installed in ''/tftpboot'' area on clon10 server:

linux-install/Centos5PAE_new/initrd.img
/initrd.img.backup
/vmlinuz
/msgs/boot.msg
/expert.msg
/general.msg
/param.msg
/rescue.msg
/snake.msg
/pxelinux.0
/pxelinux.cfg/81394458 -> croctest1
/81394459 -> croctest2
/croctest1
/croctest2
/default
/pxeos.xml

Every client must has a file in ''/pxelinux.cfg/'' area with the name containing IP address in HEX format, for example croctest1's IP is 129.57.68.88, so file name must be 81394458. It can be a symbolic link. The contents of the file for croctest1:

default Centos5PAE_new
label Centos5PAE_new
kernel Centos5PAE_new/vmlinuz
append initrd=Centos5PAE_new/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=129.57.167.16:/vol/diskless/Centos5 ramdisk_size=31809 ETHERNET=eth0 SNAPSHOT=croctest1.jlab.org NISDOMAIN=CCCHP acpi=force vga=0x305 vmalloc=256MB console=ttyS0,9600

All settings in that file are more or less generic. To customize it for another client only ''SNAPSHOT=croctest1.jlab.org'' must be changed. Statement ''NFSROOT=129.57.167.16:/vol/diskless/Centos5'' defines the fileserver IP and directory where diskless Linux is located, in our case it is ''clonfs1''. Statement ''console=ttyS0,9600'' force Centos5 to send boot messages to the com port so they can be monitored using ''tsconnect''.

* diskless Linux is installed on ''clonfs1'' fileserver in ''/vol/diskless/Centos5'' volume. That volume is mounted on the clonpc3 RHEL machine as well, and all Centos5 installation was performed on clonpc3, as it described in EXPERT PART section below. Directory ''/diskless/Centos5/root'' will be mounted as ''/'' level on all diskless clients, so that part is generic for all clients. Custom areas in ''/diskless/Centos5/snapshot'' will be client-specific, for every new client new directory with appropriate name must be creates there using templete tar ball, for example:
tar xvf templete.jlab.org.tar
mv templete.jlab.org croctest1.jlab.org
* File ''files'' contains the list of all files/directories to be used from snapshot area rather then from root area, that file was created during Centos5 installation. File ''files.custom'' can be used to specify additional files/directories, not in use right now.

== Customizing kernel installed in ''/diskless/Centos5/root'' ==

After expert installation is done, following changes must be applied. They are similar to the RHEL installation on clon cluster PCs and servers:

* ''/usr/bin/system-config-network''

* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''

'''NOTE''': to add new used you may use command ''useradd'', for example
useradd -u 6246 -g nagios -d /home/nagios -c "Nagios" -s /bin/tcsh nagios
(add flag '-m' if want to force home directory creation).

* fix ''/etc/nsswitch.conf''

* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver clon10.jlab.org'')

* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')

* ''/etc/init.d/ypbind start''

* ''chkconfig ypbind on''

* set RHEL-style [[automounts]]

* ''/etc/init.d/autofs restart''

* fix symbolic links to /apps, /home, /usr/local etc

To install updates, ssh to clonpc3 as ''root'' and start chroot environment: ''chroot /misc/diskless/Centos5/root''. After that you can use ''yum'', for example:
yum install gcc-c++
yum install mysql-devel
yum install compat-gcc-34-g77
yum install libXpm-devel
yum install libXtst
yum install java

In case of yum, rpm etc problems usually you have to rebuild dictionary, google the error message.

== EXPERT PART: setting everything from scratch using clonpc3 ==

VNC

2011-11-22T14:29:06Z

129.57.167.14:

'''VNC''' stands for ''Virtual Network Computing package'' and is included into both Solaris and Linux installations.

To start ''vncserver'' do following:

'''LINUX RHEL5''': as 'root' do following:

1. Edit ''/etc/sysconfig/vncservers'', two lines must be uncommented (will be running as ''boiarino''):

VNCSERVERS="2:boiarino"
VNCSERVERARGS[2]="-geometry 1600x1024 -nolisten tcp -nohttpd -localhost"

2. Edit ''boiarino/.vnc/xstartup'': change window manager from ''twm'' to ''startkde'' (for KDE) or to ''exec gnome-session'' (for GNOME)

3. As root run ''/etc/init.d/vncserver start''

4. As root run ''vncpasswd'' and set password

5. Make it start automatically at boot time:
/sbin/chkconfig --level 3 vncserver on
/sbin/chkconfig --level 4 vncserver on
/sbin/chkconfig --level 5 vncserver on
/sbin/chkconfig --list | grep vncserver
vncserver 0:off 1:off 2:off 3:on 4:on 5:on 6:off

6. Since 590X ports blocked by computer center, use tunneling; for example to connect to vncserver running on clontrig0, run following on your desktop:
ssh -L 5902:localhost:5902 clontrig0
and keep it running; type:
vncviewer localhost:5902
enter password and VNC window will shows up.

'''LINUX RHEL4''': as 'root' do following:

/etc/init.d/vncserver start
Starting VNC server: [ OK ]
vncpasswd
Password:
Verify:
vncserver
New 'clonweb:1 (boiarino)' desktop is clonweb:1
Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/clonweb:1.log

So what did we do there? First, we started the vncserver service. It may or may not have already been running on your system. Next we set a password to access the VNC desktop. When you set the password, you will not see any characters on the screen, and you must enter the password twice. You will only need to do this the very first time you run vncserver. The password will be saved in the Linux filesystem, and you can change it at any time by invoking the vncpasswd command again. Last, to activate the VNC desktop, we simply invoked the vncserver command. Notice the output; the desktop is named "clonweb:1" which can also be replaced via the machines IP address.

To kill server type:
vncserver -kill :1

NOTE: do we need to do following ??? :
/sbin/chkconfig --level 3 vncserver on
/sbin/chkconfig --level 4 vncserver on
/sbin/chkconfig --level 5 vncserver on
/sbin/chkconfig --list | grep vncserver
vncserver 0:off 1:off 2:off 3:on 4:on 5:on 6:off

'''SOLARIS 10''':

Login as ''clasrun'' and the do ''su''.

Download, install and use ''x11vnc'' as following (as 'root'):

cd /usr/local/src/x11vnc-0.9.13
./configure <<<--without-ssl>>>
gmake
gmake install

Now create password (we assume that x1vnc server will be running as 'clasrun'; it is important that all clon workstations usually logged in as clasrun):

x11vnc -storepasswd

It will create password which will be used by vncviewer. Now start server(s) (one per screen), for example if two monitors connected to the workstation run two servers:

x11vnc -rfbauth /home/clasrun/.vnc/passwd -display :0.0 -forever -shared
x11vnc -rfbauth /home/clasrun/.vnc/passwd -display :0.1 -forever -shared

Those commands are collected in script ''$CODA/common/scripts/vnc_start'', run it as ''root'' to start all necessary vnc servers.

Option '-shared' allows more then one client to connect, and '-forever' will prevent server from dying when last client disconnected. Now you can connect from another machine, for example to connect to two 'clon06' screens type follwoing on your machine:

vncviewer clon06:0
vncviewer clon06:1

If running 'vncviewer' from Solaris but from non-solaris terminal it maybe useful to specify flag '-bgr233' to get better colors.

TRIED, DID NOT WORK:

Tell dtlogin to run Xvnc rather than the normal X server:

# mkdir /etc/dt/config
# cat >/etc/dt/config/Xservers <<EOF
:0 Local local_uid@none root /opt/csw/bin/Xvnc :0 -geometry 1024x768 -depth 24
EOF
#

Restart dtlogin:

# /etc/init.d/dtlogin stop
# /etc/init.d/dtlogin start
#

After a couple of seconds, it’s worth checking that Xvnc is actually running:

# ps -ef | grep Xvnc
root 102525 102484 0 15:48:15 ? 0:00 /opt/csw/bin/Xvnc :0 -auth /var/dt/A:0-7Zavqg
#

======================

[Apr 2, 2007] Solaris 10, gcc and VNC server by jalzabrewer

May 28, 2005 (forum.java.sun.com) VNC is on the SUN Supplimental SFW disk for Soalris 10 system SFWvnc vnc - Virtual Network Computing. Here is my implementaion using dtlogin xdm login feature of broadcast, if any one comes up with a way to use sse, let me know!

1. pkgadd -d . SFWvnc

2. Add to /etc/services

nc1024 stream tcp nowait root /usr/local/bin/Xvnc-1024 Xvnc-1024

3. Add to /etc/inetd.conf

vnc1024 stream tcp nowait root /usr/local/bin/Xvnc-1024 Xvnc-1024

4. Create /usr/local/bin/Xvnc-1024 script , perm 755.

#!/bin/sh
/opt/sfw/bin/Xvnc -inetd -broadcast -once -geometry 1024x768 -depth 8

5. Stop dtlogin

/etc/init.d/dtlogin stop

6. [Restart indentd] kill -HUP inetd-pid

7. Start dtlogin

/etc/init.d/dtlogin start

8. /usr/sbin/shutdown -y -g0 -i6 to test startup.

9. svcs should show:

bash-3.00# svcs |grep vnc

online 16:06:49 svc:/network/vnc1024/tcp:default

Have fun!

[Apr 2, 2007] VNC on Solaris 10 Installing and configuring VNC from the software companion CD

Solaris 10 can be downloaded from sun’s web site. Also available from the download pages is an image of a “Software Companion CD”.

This CD contains extra freeware products which can be added post-installation. One of these is the excellent Virtual Network Computing (VNC) package.

VNC allows remote desktop access to computers over a network. It can be used between Microsoft and Unix/Linux operating systems.

Insert the Solaris Companion CD and allow the volume management daemon to mount it. There are three packages which are required. In this example they are converted into package datastreams on the hard disk:

# cd /cdrom/cdrom0/sparc/Packages

# pkgtrans . /opt/SFWgcmn.pkg SFWgcmn

# pkgtrans . /opt/SFWgcc34l.pkg SFWgcc341

# pkgtrans . /opt/SFWvnc.pkg SFWvnc

[Mar 24, 2007] freshmeat.net Project details for Enhanced TightVNC Viewer

Enhanced TightVNC Viewer 1.0.14 released

The Enhanced TightVNC Viewer package is part of the x11vnc VNC server project. It provides a native VNC viewer that takes advantage of new features in x11vnc, e.g. cursor alpha blending and automatic SSL tunnelling. Some features apply to any VNC server, e.g. automatic SSH tunnelling. Another goal is to provide a package that conveniently bundles everything needed for the user to have the enhanced viewer running quickly. This includes pre-built binaries of the viewer and utility programs for Windows and many Unix variants, and a GUI to configure and launch the viewer. The short name for this project is "ssvnc", for SSL/SSH VNC viewer.
Release focus: Minor bugfixes

Changes:
Using port numbers lower than VNC's default port (5900) now works on Windows (for example, myhost.com:443).

Author:
Karl Runge [contact developer]

[Feb 20, 2007] VNC on Solaris 10

A good explanation of details

Insert the Solaris Companion CD and allow the volume management daemon to mount it. There are three packages which are required. In this example they are converted into package datastreams on the hard disk:

# cd /cdrom/cdrom0/sparc/Packages
# pkgtrans . /opt/SFWgcmn.pkg SFWgcmn
# pkgtrans . /opt/SFWgcc34l.pkg SFWgcc341
# pkgtrans . /opt/SFWvnc.pkg SFWvnc

Add the three packages in the following order:

# cd /opt
# pkgadd -d SFWgcmn.pkg
# pkgadd -d SFWgcc34l.pkg
# pkgadd -d SFWvnc.pkg

Xwindows is configured with text files under /usr/dt but these shouldn’t be modified, make copied of them under /etc/dt and modify the copies.

# mkdir -p /etc/dt/config
# cp /usr/dt/config/Xservers /etc/dt/config/
# cp /usr/dt/config/Xconfig /etc/dt/config/
# cp /usr/dt/config/Xsetup /etc/dt/config/

Edit the /etc/dt/config/Xservers to configure the extra connections to Xwindows. The :0 server is the default one which you log in to locally. In this example one extra connection is being configured:

# vi /etc/dt/config/Xservers

:0 Local local_uid@console root /usr/X11/bin/Xserver :0 -nobanner

:1 Local local_uid@console root /opt/sfw/bin/Xvnc :1 -httpd /opt/sfw/vnc/classes -depth 16 -geometry 1024x768 -r fbwait 120000 -rfbauth /opt/sfw/vnc/.vnc/passwd -rfbport 5901 -httpport 5801 -fp tcp/localhost:7100 -alwaysshared

There are only two lines in my example file, it has just word wrapped on this web page. Here are some of the important options:

-depth 16

This is the colour depth. An 8bit colour depth uses less network bandwidth but looks a bit blocky.

-geometry 1024x768

Choose whichever resolution you want. Yes, that is an “x” between the two numbers not an “*”.

-rfbport 5901

The port number for the client to connect to. Each different server must use a different port number. For example :1 uses 5901 and :2 could use 5902

-httpport 5801

You can point a web browser at the Unix host with this port number and get a java based connection to the Unix host. For example:

http://10.0.0.1:5801

Configure a password, this is prompted for before the user sees a login screen.

# HOME=/opt/sfw/vnc
# export HOME
# mkdir $HOME/.vnc
# /opt/sfw/bin/vncpasswd
Password: password
Verify: password

To enable vnc to start when dtlogin does, uncomment the Dtlogin*grabServer line in /etc/dt/config/Xconfig or do this sed command which will do the same thing:

# sed '/Dtlogin\*grabServer/s/^#//' /etc/dt/config/Xconfig >/tmp/Xconfig
# mv /tmp/Xconfig /etc/dt/config/Xconfig

This would work but the fonts will not be right. Add the font server to the /etc/inetd.conf and convert it into a new SMF managed network service:

# echo "fs stream tcp wait nobody /usr/openwin/lib/fs.auto fs" >>/etc/inetd.conf
# inetconv

That should be it! I rebooted here but I think you should be able to get away with hupping the inetd and stopping and restarting dtlogin:

# pkill –HUP inetd
# /etc/init.d/dtlogin stop
# /etc/init.d/dtlogin start

Any time you change the /etc/dt/config/Xservers file, you will need to stop and restart dtlogin. In previous versions you could just /usr/dt/bin/dtlogin –reset but it doesn’t seem to work in Solaris 10.

Java Desktop

There is a bug which will prevent text being entered in terminal windows in the Java Desktop. This can be solved by modifying the file shown below:

# cd /etc/gconf/gconf.xml.defaults/apps/gnome_settings_daemon/keybindings/
# mv %gconf.xml %gconf.xml.orig
# vi %gconf.xml

<gconf>

<entry name="volume_up" mtime="1110896708" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="volume_mute" mtime="1110896705" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="volume_down" mtime="1110896702" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="help" mtime="1110896698" type="string">

<stringvalue></stringvalue>

</entry>

</gconf>

The next time you log in to a Java Desktop, it will work.

VNC

2011-11-22T14:14:32Z

129.57.167.14:

'''VNC''' stands for ''Virtual Network Computing package'' and is included into both Solaris and Linux installations.

To start ''vncserver'' do following:

'''LINUX RHEL5''': as 'root' do following:

1. Edit ''/etc/sysconfig/vncservers'', two lines must be uncommented (will be running as ''boiarino''):

VNCSERVERS="2:boiarino"
VNCSERVERARGS[2]="-geometry 1600x1024 -nolisten tcp -nohttpd -localhost"

2. Edit ''boiarino/.vnc/xstartup'': change window manager from ''twm'' to ''startkde'' (for KDE) or to ''exec gnome-session'' (for GNOME)

3. As root run ''/etc/init.d/vncserver start''

4. As root run ''vncpasswd'' and set password

5. Make it start automatically at boot time:
/sbin/chkconfig --level 3 vncserver on
/sbin/chkconfig --level 4 vncserver on
/sbin/chkconfig --level 5 vncserver on
/sbin/chkconfig --list | grep vncserver
vncserver 0:off 1:off 2:off 3:on 4:on 5:on 6:off

6. Since 590X ports blocked by computer center, use tunneling; for example to connect to vncserver running on clontrig0, run following on your desktop:
ssh -L 5902:localhost:5902 clontrig0
and keep it running; type:
vncviewer localhost:5902
enter password and VNC window will shows up.

'''LINUX RHEL4''': as 'root' do following:

/etc/init.d/vncserver start
Starting VNC server: [ OK ]
vncpasswd
Password:
Verify:
vncserver
New 'clonweb:1 (boiarino)' desktop is clonweb:1
Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/clonweb:1.log

So what did we do there? First, we started the vncserver service. It may or may not have already been running on your system. Next we set a password to access the VNC desktop. When you set the password, you will not see any characters on the screen, and you must enter the password twice. You will only need to do this the very first time you run vncserver. The password will be saved in the Linux filesystem, and you can change it at any time by invoking the vncpasswd command again. Last, to activate the VNC desktop, we simply invoked the vncserver command. Notice the output; the desktop is named "clonweb:1" which can also be replaced via the machines IP address.

To kill server type:
vncserver -kill :1

NOTE: do we need to do following ??? :
/sbin/chkconfig --level 3 vncserver on
/sbin/chkconfig --level 4 vncserver on
/sbin/chkconfig --level 5 vncserver on
/sbin/chkconfig --list | grep vncserver
vncserver 0:off 1:off 2:off 3:on 4:on 5:on 6:off

'''SOLARIS 10''':

Download, install and use ''x11vnc'' as following (as 'root'):

cd /usr/local/src/x11vnc-0.9.13
./configure <<<--without-ssl>>>
gmake
gmake install

Now create password (we assume that x1vnc server will be running as 'clasrun'; it is important that all clon workstations usually logged in as clasrun):

x11vnc -storepasswd

It will create password which will be used by vncviewer. Now start server(s) (one per screen), for example if two monitors connected to the workstation run two servers:

x11vnc -rfbauth /home/clasrun/.vnc/passwd -display :0.0 -forever -shared
x11vnc -rfbauth /home/clasrun/.vnc/passwd -display :0.1 -forever -shared

Option '-shared' allows more then one client to connect, and '-forever' will prevent server from dying when last client disconnected. Now you can connect from another machine, for example to connect to two 'clon06' screens type follwoing on your machine:

vncviewer clon06:0
vncviewer clon06:1

If running 'vncviewer' from Solaris but from non-solaris terminal it maybe useful to specify flag '-bgr233' to get better colors.

TRIED, DID NOT WORK:

Tell dtlogin to run Xvnc rather than the normal X server:

# mkdir /etc/dt/config
# cat >/etc/dt/config/Xservers <<EOF
:0 Local local_uid@none root /opt/csw/bin/Xvnc :0 -geometry 1024x768 -depth 24
EOF
#

Restart dtlogin:

# /etc/init.d/dtlogin stop
# /etc/init.d/dtlogin start
#

After a couple of seconds, it’s worth checking that Xvnc is actually running:

# ps -ef | grep Xvnc
root 102525 102484 0 15:48:15 ? 0:00 /opt/csw/bin/Xvnc :0 -auth /var/dt/A:0-7Zavqg
#

======================

[Apr 2, 2007] Solaris 10, gcc and VNC server by jalzabrewer

May 28, 2005 (forum.java.sun.com) VNC is on the SUN Supplimental SFW disk for Soalris 10 system SFWvnc vnc - Virtual Network Computing. Here is my implementaion using dtlogin xdm login feature of broadcast, if any one comes up with a way to use sse, let me know!

1. pkgadd -d . SFWvnc

2. Add to /etc/services

nc1024 stream tcp nowait root /usr/local/bin/Xvnc-1024 Xvnc-1024

3. Add to /etc/inetd.conf

vnc1024 stream tcp nowait root /usr/local/bin/Xvnc-1024 Xvnc-1024

4. Create /usr/local/bin/Xvnc-1024 script , perm 755.

#!/bin/sh
/opt/sfw/bin/Xvnc -inetd -broadcast -once -geometry 1024x768 -depth 8

5. Stop dtlogin

/etc/init.d/dtlogin stop

6. [Restart indentd] kill -HUP inetd-pid

7. Start dtlogin

/etc/init.d/dtlogin start

8. /usr/sbin/shutdown -y -g0 -i6 to test startup.

9. svcs should show:

bash-3.00# svcs |grep vnc

online 16:06:49 svc:/network/vnc1024/tcp:default

Have fun!

[Apr 2, 2007] VNC on Solaris 10 Installing and configuring VNC from the software companion CD

Solaris 10 can be downloaded from sun’s web site. Also available from the download pages is an image of a “Software Companion CD”.

This CD contains extra freeware products which can be added post-installation. One of these is the excellent Virtual Network Computing (VNC) package.

VNC allows remote desktop access to computers over a network. It can be used between Microsoft and Unix/Linux operating systems.

Insert the Solaris Companion CD and allow the volume management daemon to mount it. There are three packages which are required. In this example they are converted into package datastreams on the hard disk:

# cd /cdrom/cdrom0/sparc/Packages

# pkgtrans . /opt/SFWgcmn.pkg SFWgcmn

# pkgtrans . /opt/SFWgcc34l.pkg SFWgcc341

# pkgtrans . /opt/SFWvnc.pkg SFWvnc

[Mar 24, 2007] freshmeat.net Project details for Enhanced TightVNC Viewer

Enhanced TightVNC Viewer 1.0.14 released

The Enhanced TightVNC Viewer package is part of the x11vnc VNC server project. It provides a native VNC viewer that takes advantage of new features in x11vnc, e.g. cursor alpha blending and automatic SSL tunnelling. Some features apply to any VNC server, e.g. automatic SSH tunnelling. Another goal is to provide a package that conveniently bundles everything needed for the user to have the enhanced viewer running quickly. This includes pre-built binaries of the viewer and utility programs for Windows and many Unix variants, and a GUI to configure and launch the viewer. The short name for this project is "ssvnc", for SSL/SSH VNC viewer.
Release focus: Minor bugfixes

Changes:
Using port numbers lower than VNC's default port (5900) now works on Windows (for example, myhost.com:443).

Author:
Karl Runge [contact developer]

[Feb 20, 2007] VNC on Solaris 10

A good explanation of details

Insert the Solaris Companion CD and allow the volume management daemon to mount it. There are three packages which are required. In this example they are converted into package datastreams on the hard disk:

# cd /cdrom/cdrom0/sparc/Packages
# pkgtrans . /opt/SFWgcmn.pkg SFWgcmn
# pkgtrans . /opt/SFWgcc34l.pkg SFWgcc341
# pkgtrans . /opt/SFWvnc.pkg SFWvnc

Add the three packages in the following order:

# cd /opt
# pkgadd -d SFWgcmn.pkg
# pkgadd -d SFWgcc34l.pkg
# pkgadd -d SFWvnc.pkg

Xwindows is configured with text files under /usr/dt but these shouldn’t be modified, make copied of them under /etc/dt and modify the copies.

# mkdir -p /etc/dt/config
# cp /usr/dt/config/Xservers /etc/dt/config/
# cp /usr/dt/config/Xconfig /etc/dt/config/
# cp /usr/dt/config/Xsetup /etc/dt/config/

Edit the /etc/dt/config/Xservers to configure the extra connections to Xwindows. The :0 server is the default one which you log in to locally. In this example one extra connection is being configured:

# vi /etc/dt/config/Xservers

:0 Local local_uid@console root /usr/X11/bin/Xserver :0 -nobanner

:1 Local local_uid@console root /opt/sfw/bin/Xvnc :1 -httpd /opt/sfw/vnc/classes -depth 16 -geometry 1024x768 -r fbwait 120000 -rfbauth /opt/sfw/vnc/.vnc/passwd -rfbport 5901 -httpport 5801 -fp tcp/localhost:7100 -alwaysshared

There are only two lines in my example file, it has just word wrapped on this web page. Here are some of the important options:

-depth 16

This is the colour depth. An 8bit colour depth uses less network bandwidth but looks a bit blocky.

-geometry 1024x768

Choose whichever resolution you want. Yes, that is an “x” between the two numbers not an “*”.

-rfbport 5901

The port number for the client to connect to. Each different server must use a different port number. For example :1 uses 5901 and :2 could use 5902

-httpport 5801

You can point a web browser at the Unix host with this port number and get a java based connection to the Unix host. For example:

http://10.0.0.1:5801

Configure a password, this is prompted for before the user sees a login screen.

# HOME=/opt/sfw/vnc
# export HOME
# mkdir $HOME/.vnc
# /opt/sfw/bin/vncpasswd
Password: password
Verify: password

To enable vnc to start when dtlogin does, uncomment the Dtlogin*grabServer line in /etc/dt/config/Xconfig or do this sed command which will do the same thing:

# sed '/Dtlogin\*grabServer/s/^#//' /etc/dt/config/Xconfig >/tmp/Xconfig
# mv /tmp/Xconfig /etc/dt/config/Xconfig

This would work but the fonts will not be right. Add the font server to the /etc/inetd.conf and convert it into a new SMF managed network service:

# echo "fs stream tcp wait nobody /usr/openwin/lib/fs.auto fs" >>/etc/inetd.conf
# inetconv

That should be it! I rebooted here but I think you should be able to get away with hupping the inetd and stopping and restarting dtlogin:

# pkill –HUP inetd
# /etc/init.d/dtlogin stop
# /etc/init.d/dtlogin start

Any time you change the /etc/dt/config/Xservers file, you will need to stop and restart dtlogin. In previous versions you could just /usr/dt/bin/dtlogin –reset but it doesn’t seem to work in Solaris 10.

Java Desktop

There is a bug which will prevent text being entered in terminal windows in the Java Desktop. This can be solved by modifying the file shown below:

# cd /etc/gconf/gconf.xml.defaults/apps/gnome_settings_daemon/keybindings/
# mv %gconf.xml %gconf.xml.orig
# vi %gconf.xml

<gconf>

<entry name="volume_up" mtime="1110896708" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="volume_mute" mtime="1110896705" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="volume_down" mtime="1110896702" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="help" mtime="1110896698" type="string">

<stringvalue></stringvalue>

</entry>

</gconf>

The next time you log in to a Java Desktop, it will work.

VNC

2011-11-22T14:05:23Z

129.57.167.14:

'''VNC''' stands for ''Virtual Network Computing package'' and is included into both Solaris and Linux installations.

To start ''vncserver'' do following:

'''LINUX RHEL5''': as 'root' do following:

1. Edit ''/etc/sysconfig/vncservers'', two lines must be uncommented (will be running as ''boiarino''):

VNCSERVERS="2:boiarino"
VNCSERVERARGS[2]="-geometry 1600x1024 -nolisten tcp -nohttpd -localhost"

2. Edit ''boiarino/.vnc/xstartup'': change window manager from ''twm'' to ''startkde'' (for KDE) or to ''exec gnome-session'' (for GNOME)

3. As root run ''/etc/init.d/vncserver start''

4. As root run ''vncpasswd'' and set password

5. Make it start automatically at boot time:
/sbin/chkconfig --level 3 vncserver on
/sbin/chkconfig --level 4 vncserver on
/sbin/chkconfig --level 5 vncserver on
/sbin/chkconfig --list | grep vncserver
vncserver 0:off 1:off 2:off 3:on 4:on 5:on 6:off

6. Since 590X ports blocked by computer center, use tunneling; for example to connect to vncserver running on clontrig0, run following on your desktop:
ssh -L 5902:localhost:5902 clontrig0
and keep it running; type:
vncviewer localhost:5902
enter password and VNC window will shows up.

'''LINUX RHEL4''': as 'root' do following:

/etc/init.d/vncserver start
Starting VNC server: [ OK ]
vncpasswd
Password:
Verify:
vncserver
New 'clonweb:1 (boiarino)' desktop is clonweb:1
Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/clonweb:1.log

So what did we do there? First, we started the vncserver service. It may or may not have already been running on your system. Next we set a password to access the VNC desktop. When you set the password, you will not see any characters on the screen, and you must enter the password twice. You will only need to do this the very first time you run vncserver. The password will be saved in the Linux filesystem, and you can change it at any time by invoking the vncpasswd command again. Last, to activate the VNC desktop, we simply invoked the vncserver command. Notice the output; the desktop is named "clonweb:1" which can also be replaced via the machines IP address.

To kill server type:
vncserver -kill :1

NOTE: do we need to do following ??? :
/sbin/chkconfig --level 3 vncserver on
/sbin/chkconfig --level 4 vncserver on
/sbin/chkconfig --level 5 vncserver on
/sbin/chkconfig --list | grep vncserver
vncserver 0:off 1:off 2:off 3:on 4:on 5:on 6:off

'''SOLARIS 10''':

Download, install and use ''x11vnc'' as following (as 'root'):

cd /usr/local/src/x11vnc-0.9.13
./configure --without-ssl
gmake
gmake install

NOTE: following line was changed in file 'x11vnc/sslhelper.h' BEFORE running 'gmake':

/*sergey: undefined symbol, define it here until figure it out ...*/
/*extern char *certret_str;*/
static char *certret_str;

Now create password (we assume that x1vnc server will be running as 'clasrun'; it is important that all clon workstations usually logged in as clasrun):

x11vnc -storepasswd

It will create password which will be used by vncviewer. Now start server(s) (one per screen), for example if two monitors connected to the workstation run two servers:

x11vnc -rfbauth /home/clasrun/.vnc/passwd -display :0.0 -forever -shared
x11vnc -rfbauth /home/clasrun/.vnc/passwd -display :0.1 -forever -shared

Option '-shared' allows more then one client to connect, and '-forever' will prevent server from dying when last client disconnected. Now you can connect from another machine, for example to connect to two 'clon06' screens type follwoing on your machine:

vncviewer clon06:0
vncviewer clon06:1

If running 'vncviewer' from Solaris but from non-solaris terminal it maybe useful to specify flag '-bgr233' to get better colors.

TRIED, DID NOT WORK:

Tell dtlogin to run Xvnc rather than the normal X server:

# mkdir /etc/dt/config
# cat >/etc/dt/config/Xservers <<EOF
:0 Local local_uid@none root /opt/csw/bin/Xvnc :0 -geometry 1024x768 -depth 24
EOF
#

Restart dtlogin:

# /etc/init.d/dtlogin stop
# /etc/init.d/dtlogin start
#

After a couple of seconds, it’s worth checking that Xvnc is actually running:

# ps -ef | grep Xvnc
root 102525 102484 0 15:48:15 ? 0:00 /opt/csw/bin/Xvnc :0 -auth /var/dt/A:0-7Zavqg
#

======================

[Apr 2, 2007] Solaris 10, gcc and VNC server by jalzabrewer

May 28, 2005 (forum.java.sun.com) VNC is on the SUN Supplimental SFW disk for Soalris 10 system SFWvnc vnc - Virtual Network Computing. Here is my implementaion using dtlogin xdm login feature of broadcast, if any one comes up with a way to use sse, let me know!

1. pkgadd -d . SFWvnc

2. Add to /etc/services

nc1024 stream tcp nowait root /usr/local/bin/Xvnc-1024 Xvnc-1024

3. Add to /etc/inetd.conf

vnc1024 stream tcp nowait root /usr/local/bin/Xvnc-1024 Xvnc-1024

4. Create /usr/local/bin/Xvnc-1024 script , perm 755.

#!/bin/sh
/opt/sfw/bin/Xvnc -inetd -broadcast -once -geometry 1024x768 -depth 8

5. Stop dtlogin

/etc/init.d/dtlogin stop

6. [Restart indentd] kill -HUP inetd-pid

7. Start dtlogin

/etc/init.d/dtlogin start

8. /usr/sbin/shutdown -y -g0 -i6 to test startup.

9. svcs should show:

bash-3.00# svcs |grep vnc

online 16:06:49 svc:/network/vnc1024/tcp:default

Have fun!

[Apr 2, 2007] VNC on Solaris 10 Installing and configuring VNC from the software companion CD

Solaris 10 can be downloaded from sun’s web site. Also available from the download pages is an image of a “Software Companion CD”.

This CD contains extra freeware products which can be added post-installation. One of these is the excellent Virtual Network Computing (VNC) package.

VNC allows remote desktop access to computers over a network. It can be used between Microsoft and Unix/Linux operating systems.

Insert the Solaris Companion CD and allow the volume management daemon to mount it. There are three packages which are required. In this example they are converted into package datastreams on the hard disk:

# cd /cdrom/cdrom0/sparc/Packages

# pkgtrans . /opt/SFWgcmn.pkg SFWgcmn

# pkgtrans . /opt/SFWgcc34l.pkg SFWgcc341

# pkgtrans . /opt/SFWvnc.pkg SFWvnc

[Mar 24, 2007] freshmeat.net Project details for Enhanced TightVNC Viewer

Enhanced TightVNC Viewer 1.0.14 released

The Enhanced TightVNC Viewer package is part of the x11vnc VNC server project. It provides a native VNC viewer that takes advantage of new features in x11vnc, e.g. cursor alpha blending and automatic SSL tunnelling. Some features apply to any VNC server, e.g. automatic SSH tunnelling. Another goal is to provide a package that conveniently bundles everything needed for the user to have the enhanced viewer running quickly. This includes pre-built binaries of the viewer and utility programs for Windows and many Unix variants, and a GUI to configure and launch the viewer. The short name for this project is "ssvnc", for SSL/SSH VNC viewer.
Release focus: Minor bugfixes

Changes:
Using port numbers lower than VNC's default port (5900) now works on Windows (for example, myhost.com:443).

Author:
Karl Runge [contact developer]

[Feb 20, 2007] VNC on Solaris 10

A good explanation of details

Insert the Solaris Companion CD and allow the volume management daemon to mount it. There are three packages which are required. In this example they are converted into package datastreams on the hard disk:

# cd /cdrom/cdrom0/sparc/Packages
# pkgtrans . /opt/SFWgcmn.pkg SFWgcmn
# pkgtrans . /opt/SFWgcc34l.pkg SFWgcc341
# pkgtrans . /opt/SFWvnc.pkg SFWvnc

Add the three packages in the following order:

# cd /opt
# pkgadd -d SFWgcmn.pkg
# pkgadd -d SFWgcc34l.pkg
# pkgadd -d SFWvnc.pkg

Xwindows is configured with text files under /usr/dt but these shouldn’t be modified, make copied of them under /etc/dt and modify the copies.

# mkdir -p /etc/dt/config
# cp /usr/dt/config/Xservers /etc/dt/config/
# cp /usr/dt/config/Xconfig /etc/dt/config/
# cp /usr/dt/config/Xsetup /etc/dt/config/

Edit the /etc/dt/config/Xservers to configure the extra connections to Xwindows. The :0 server is the default one which you log in to locally. In this example one extra connection is being configured:

# vi /etc/dt/config/Xservers

:0 Local local_uid@console root /usr/X11/bin/Xserver :0 -nobanner

:1 Local local_uid@console root /opt/sfw/bin/Xvnc :1 -httpd /opt/sfw/vnc/classes -depth 16 -geometry 1024x768 -r fbwait 120000 -rfbauth /opt/sfw/vnc/.vnc/passwd -rfbport 5901 -httpport 5801 -fp tcp/localhost:7100 -alwaysshared

There are only two lines in my example file, it has just word wrapped on this web page. Here are some of the important options:

-depth 16

This is the colour depth. An 8bit colour depth uses less network bandwidth but looks a bit blocky.

-geometry 1024x768

Choose whichever resolution you want. Yes, that is an “x” between the two numbers not an “*”.

-rfbport 5901

The port number for the client to connect to. Each different server must use a different port number. For example :1 uses 5901 and :2 could use 5902

-httpport 5801

You can point a web browser at the Unix host with this port number and get a java based connection to the Unix host. For example:

http://10.0.0.1:5801

Configure a password, this is prompted for before the user sees a login screen.

# HOME=/opt/sfw/vnc
# export HOME
# mkdir $HOME/.vnc
# /opt/sfw/bin/vncpasswd
Password: password
Verify: password

To enable vnc to start when dtlogin does, uncomment the Dtlogin*grabServer line in /etc/dt/config/Xconfig or do this sed command which will do the same thing:

# sed '/Dtlogin\*grabServer/s/^#//' /etc/dt/config/Xconfig >/tmp/Xconfig
# mv /tmp/Xconfig /etc/dt/config/Xconfig

This would work but the fonts will not be right. Add the font server to the /etc/inetd.conf and convert it into a new SMF managed network service:

# echo "fs stream tcp wait nobody /usr/openwin/lib/fs.auto fs" >>/etc/inetd.conf
# inetconv

That should be it! I rebooted here but I think you should be able to get away with hupping the inetd and stopping and restarting dtlogin:

# pkill –HUP inetd
# /etc/init.d/dtlogin stop
# /etc/init.d/dtlogin start

Any time you change the /etc/dt/config/Xservers file, you will need to stop and restart dtlogin. In previous versions you could just /usr/dt/bin/dtlogin –reset but it doesn’t seem to work in Solaris 10.

Java Desktop

There is a bug which will prevent text being entered in terminal windows in the Java Desktop. This can be solved by modifying the file shown below:

# cd /etc/gconf/gconf.xml.defaults/apps/gnome_settings_daemon/keybindings/
# mv %gconf.xml %gconf.xml.orig
# vi %gconf.xml

<gconf>

<entry name="volume_up" mtime="1110896708" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="volume_mute" mtime="1110896705" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="volume_down" mtime="1110896702" type="string">

<stringvalue></stringvalue>

</entry>

<entry name="help" mtime="1110896698" type="string">

<stringvalue></stringvalue>

</entry>

</gconf>

The next time you log in to a Java Desktop, it will work.

DAQ/Online Quick Reference

2011-11-21T17:30:02Z

129.57.167.14:

'''DAQ Recovery Procedure'''

In most cases DAQ will crash because of ROC problem. Do following:

* click 'End Run' or 'Abort', whatever button is active

* click 'Cancel'

* run 'roc_status' command from clon10 window; it will show troubled ROC(s) as 'UNDEFINED' or 'active'; most likely clastrig2 or/and polar

* click 'Reset'

* from clon10 window, reboot troubled ROC(s) using command 'roc_reboot <roc>'; if rebooting any ROC, reboot clastrig2 as well

* wait 2 minutes and run 'roc_status' again; rebooted ROC(s) must be in 'booted' stage

* go to 'EB_TERM' window, Ctrl-C and run previous command using arrow up (command is 'coda_eb ...')

* start new run from 'Download'

'''NOTE1''': there is a new crate dc12, roc_status will not show it; to reboot it, use ''roc_reboot dc12''

'''NOTE2''': 'tage' and 'tage2' are Linux processes now, so instead of roc_reboot, goto appropriate window, Ctrl-C and restart previous command (ex: coda_roc -s clasprod -s "tage ROC" -i)

If it does not help, page DAQ pager.

'''FOLLOWING INFORMATION IS FOR DAQ EXPERTS ONLY (DO NOT DO IT IF YOU ARE NOT SURE !!!)'''

'''DAQ procedures'''

First make sure you can open X window from the terminal clon03:0.1: type 'xterm' and make sure xterminal is opened. If it is not, message 'X11 connection rejected because of wrong authentication.' will shows up. In this case kill xwindow, start new one (you will be on clon03) and 'ssh clon10'. Make sure 'xterm' command can open xterminal.

To start all DAQ processes goto clon10 as ''clasrun'' and type following command:

daq_start this

to open all windows on current workspace. If ''daq_start'' executed without parameter, it will open everything on clon03's bottom screen on currently active workspace.

To kill all DAQ processes goto clon10 as ''clasrun'' and type

daq_exit

To start/stop serial connections to ROCs and PMCs, type following on clon00:

roc_xterm_this
pmc_xterms_this

roc_xterms_kill
pmc_xterms_kill

'''Online procedures'''

To start online monitoring processes use following commands:
mon0_start (starts everything on clonmon0 - fast online reconstruction;
NOTE: root presenter must be started separately: ssh to clonmon0 (usually from clon03:0.2) and type 'cmroot clasprod hist_clasprod')
mon1_start (starts everything on clonmon1 - event_monitor)
mon2_start (starts everything on clonmon2 - online recsis, ced; if ced does not started, run it manually by ssh to clonmon2 from clon04:0.2)

To kill online monitoring processes use following commands:
mon0_exit (kills everything on clonmon0)
mon1_exit (kills everything on clonmon1)
mon2_exit (kills everything on clonmon2)

Occupancy plots must be started on clonmon1. Login as ''clasrun'' and type
monb

Stadis to be started on clon05:0.1.

DCHV to be started on clon06:0.1.

Diman to be started on clon03:0.0.

'''Background information'''

* CRON JOBS: on clon10, clon00, clon01 and clondb1 make sure cron jobs are running (they must start automatically after reboot); if have to start cron jobs, login as ''clascron'' to the mentioned machines, goto directory /home/clascron/backup and follow instructions in README file, normally it will be following:

login as 'clascron' to the clon10
goto /home/clascron/backup
type 'crontab -l' to see what is running
type 'crontab -r' to erase all existing cron jobs
type 'crontab clon10'
repeat everything for clon00 and clon01, using file 'clon00' and 'clon01' respectively

'''Process management'''

Sometimes you may want to turn some cron jobs off to avoid alarms during system reconfiguration: do not forget to turn them on again after you are done !

* Process management: SmartSockets-based process management procedures must be started on clon10 as ''clasrun'' using following command:
control_process_manager start clasprod clasprod
control_procmgr_manager start clasprod clasprod
control_ipc_process_manager start clasprod clasprod
and killed by following:
control_process_manager stop clasprod clasprod
control_procmgr_manager stop clasprod clasprod
control_ipc_process_manager stop clasprod clasprod

'''Services on clondb1'''

Clondb1 is running several important servers which suppose to be started automatically on reboot:
SmartSockets
Mysql

'''Logbook and/or run database does not work'''

Contact Marty Wise.

'''Useful commands'''

r_ts_prescale - reads prescales from TS2
s_ts_prescale 0 0 0 0 0 0 100 0 - sets prescales to TS2; 0 means prescale 1, 1 means prescale 2 etc
proconhost, netonhost - run everything on main VME CPU (execute from hosts's serial connection before Download)
proconpmc, netonpmc - run everything on co-processor (execute from hosts's serial connection before Download)

'''Sergey Boyarinov''' (from JLAB local phone):
Moscow cell phone: 9-011-7-916-729-8203
Moscow home phones: 9-011-7-495-392-3569 (otec), 9-011-7-499-189-0189 (teshya)
SKYPE: boiarino

BONUS Readout

2009-10-28T02:34:56Z

129.57.167.14: /* Drivers and system software */

This page contains information related to the BONUS TPC chamber readout system.

== Drivers and system software ==

* Edit the file /boot/grub/menu.lst (as root). Pass the argument 'mem=xxxM' to the kernel to limit LINUX memory to xxxMegaBytes (xxx should be replaced by a number, in our case 1024).

'''DHCP'''

dhcpd server must be running using config file ''/etc/dhcpd.conf'' provided by CERN (copy from bonuspc1 or other configured machines); set port eth1 to static 10.10.10.10 with mask 255.0.0.0

'''RORC driver installation'''

ssh root@bonuspc2
cd /date/rorc
export DATE_MAKEFILES=/date/makefiles/

add at the beginning of GNUmakefile line
DATE_SYS := Linux

make distclean
make # errors must be ignored
make driver
make dev # (as root)
reboot

'''PHYSMEM installation'''

cd /date/physmem
make
make driver
make dev # (as root)
cp physmem.ko /lib/modules/`uname -r`/kernel/drivers/misc/ # (as root)
/sbin/depmod # (as root)

* Edit the file /etc/rc.modules (as root). Add the line '/sbin/modprobe physmem physmemsize0=xxx' to limit the size of the /dev/physmem0 PHYSMEM device to xxxMegaBytes. The size of the /dev/physmem1 device will be the complement of the LINUX memory and the /dev/physmem0 memory. RORC sw test programs use /dev/physmem0, while DATE uses /dev/physmem1 memories.

chmod +x /etc/rc.modules # (as root)

* Switch off autoupdate by the following way (as root).

/etc/rc.d/init.d/yum-autoupdate stop
- Check /var/lock/subsys/yum-autoupdate (have to create if missing).
- Edit /etc/sysconfig/yum-autoupdate :
YUMUPDATE=0
YUMONBOOT=0

* reboot

'''NOTE''': to run test programs as root add following to the ''/root/.bashrc'':
export LD_LIBRARY_PATH=/rcufec/tools:/rcufec/rcu:/rcufec/fec::$LD_LIBRARY_PATH

== History ==

From: Stephen Bueltmann <sbueltma@odu.edu> 11/30/07 14:48
Subject: Fwd: RORC and DDL Information
To:
Sergey Boyarinov <boiarino@jlab.org>, David Abbott <abbottd@jlab.org>,
Stepan Stepanyan <stepanya@jlab.org>, Howard Fenker <hcf@jlab.org>, hafidi@jlab.org
Hello All,
I am attaching a reply I received from the Hungarian company manufacturing the
ALICE (not the TPC) side of the electronics.
As per their web site (http://www.cerntech.hu) the price for the
PCI-X RORC is 1540 Euro and for one
SIU card 615 Euro.
The total would be 2770 Euro for a complete readout w/o RCU and w/o spares.
There seem to be special prices for CERN collaborating institutions.
Wrote to Luciano about the RCU boards but have not heard back.
Cheers,
Stephen
Begin forwarded message:
From: "Kiss Tivadar" <tivadar.kiss@cerntech.hu>
Date: November 30, 2007 2:35:36 PM EST
To: "'Stephen Bueltmann'" <sbueltma@odu.edu>
Subject: RE: RORC and DDL Information
Dear Stephen,
We are pleased to receive your inquiry for DDL hardware. Of course it is possible for you to get such boards from our company. Delivery time would be cca. 1-3
monthes after receiving an order, depending on the quantity. (We have usually only a few cards on stock.)
The DIU (Destination Interface Unit) of a DDL link is now integrated on the RORC (Read-out Receiver Card). A single RORC (that is 100 MHz, 64-bit
PCI-X interaface card) has two DIUs integrated on it. Thus, for two DDL links (that is, to read out two ALICE RCUs) you need 2 SIU cards and 1 RORC card.
Please let me know how many quantities you are interested in, and what is a delivery schedule that would fit to your needs.
Best regards,
Tivadar Kiss
Cerntech Ltd.
Ulloi ut 31.
H-1091 Budapest
Hungary
tel: +36 1 216 8219)
fax: +36 20 919 5132
From: Stephen Bueltmann [mailto:sbueltma@odu.edu]
Sent: Thursday, November 29, 2007 4:05 PM
To: info@cerntech.hu
Subject: RORC and DDL Information
Hello,
I am involved with the experimental programme at Jefferson Lab
in the US. One of the experiments is using a TPC which we are
reading out with the ALICE TPC front end cards FECs and their
USB readout controller (U2F).
We are considering to change to the standard ALICE readout controller
(RCU) used by the ALICE TPC.
This would require to purchase also the RORC and DDL hardware.
Would it be possible and how long would it take to get one RORC, one
SIU and one DIU card from you?
Thanks in advance and
Cheers,
Stephen
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu
----------------------------------------------------------------------
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu

== Sep 2009 NOTES for the system operations ==

* firmware loading:
from croctest5: ssh root@10.1.3.241 (bottm crate, top has .236), pass is 'dcs'
cd /eg6
./program rcu_170809.bit

* documentation

http://www.ge.infn.it/~minutoli/BONUS/SW/

http://www.jlab.org/~devita/eg6/

* event size (spars off, 200 threshold - no noise): 68+4+128+Nfecs

* expected rate about 1.3kHz

== connections ==

* every crates is connected to 4 power supplies, one 4.0V and 3 3.3V

* several crates can be connected to the same power supplies

* + or - labels must be ignored, rad cable always goes to plus

* ethernets from crates goes to switch, switch is connected to the eth1 of the host machine; those connections will be used to load firmware; to get IP address crates must be off, use 'tail -f /var/log/messages' and turn crate on, message will be issued containing IP addresses:
bonuspc1(left): upper crate: 10.0.2.254, bottom crate 10.0.2.252
bonuspc2(right): upper crate: 10.1.3.242, bottom crate 10.1.3.241
bonuspc3: 10.1.3.236

* to download firmware ssh root@10.1.3.241 (etc) with password 'dcs', goto ''/eg6'' and run command ''./program rcu_170809.bit''

* fibers goes from crates to RORCs; to see if RORCs are configured by Linux, use ''/date/rorc/Linux/rorc_find''; to check connection use http://www.jlab.org/~devita/eg6/eg6_electronics.pdf

* to check FECs use command:

bonuspc1(left): /rcufec/altro-loopback/altro-loopback -N -p0 -f0 -a5 -c8 --ddl-revision=4 --ddl-channel=1 --ddl-serial=7165
bonuspc2(right): same, --ddl-serial=6038
bonuspc3(): same, --ddl-serial=9044

see /proc/rorc_map for serial and other parameters

output should looks like this
Switching on FEC 0...done.
Writing data to ALTROs for event: 1...2...3...4...done.
Reading data from ALTROs for event: 1...354 words, 2...355 words, 3...355 words, 4...355 words, done.
Verifying event: 1...2...3...4...done.
Switching off FEC 0...done.

* to check real communications do following:

cd /rcufec/fec
./fecscope
2
1,1 (rorc channel number)
3,0,0
3,0,3(the number of FECs),0,1,2
10,100,0,0,1/6

cd /date/rorc/Linux
./rorc_receive -m0 -X0 -c1 -e1
it will wait for 1 event; to issue software trigger ssh root@10.1.3.241 and type ''rcu-sh c 0x5306''

== useful docs ==

[https://clonwiki.jlab.org/wiki/clondocs/Docs/RCUFirmwar2511.pdf RCU Firmware User Manual (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/rorc_lib_rev10.pdf ALICE Detector Data Link (DDL-RORC Library) (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/UserManual_draft_02.pdf ALICE TPC Readout Chip User Manual (pdf)]

BONUS Readout

2009-10-21T02:57:01Z

129.57.167.14: /* Drivers and system software */

This page contains information related to the BONUS TPC chamber readout system.

== Drivers and system software ==

'''DHCP'''

dhcpd server must be running using config file ''/etc/dhcpd.conf'' provided by CERN (copy from bonuspc1 or other configured machines); set port eth1 to static 10.10.10.10 with mask 255.0.0.0

'''RORC driver installation'''

ssh root@bonuspc2
cd /date/rorc
export DATE_MAKEFILES=/date/makefiles/

add at the beginning of GNUmakefile line
DATE_SYS := Linux

make distclean
make # errors must be ignored
make driver
make dev # (as root)
reboot

'''PHYSMEM installation'''

cd /date/physmem
make
make driver
make dev # (as root)
cp physmem.ko /lib/modules/`uname -r`/kernel/drivers/misc/ # (as root)
/sbin/depmod # (as root)

* Edit the file /boot/grub/menu.lst (as root). Pass the argument 'mem=xxxM' to the kernel to limit LINUX memory to xxxMegaBytes (xxx should be replaced by a number, in our case 1024).

* Edit the file /etc/rc.modules (as root). Add the line '/sbin/modprobe physmem physmemsize0=xxx' to limit the size of the /dev/physmem0 PHYSMEM device to xxxMegaBytes. The size of the /dev/physmem1 device will be the complement of the LINUX memory and the /dev/physmem0 memory. RORC sw test programs use /dev/physmem0, while DATE uses /dev/physmem1 memories.

chmod +x /etc/rc.modules # (as root)

* Switch off autoupdate by the following way (as root).

/etc/rc.d/init.d/yum-autoupdate stop
- Check /var/lock/subsys/yum-autoupdate (have to create if missing).
- Edit /etc/sysconfig/yum-autoupdate :
YUMUPDATE=0
YUMONBOOT=0

* reboot

'''NOTE''': to run test programs as root add following to the ''/root/.bashrc'':
export LD_LIBRARY_PATH=/rcufec/tools:/rcufec/rcu:/rcufec/fec::$LD_LIBRARY_PATH

== History ==

From: Stephen Bueltmann <sbueltma@odu.edu> 11/30/07 14:48
Subject: Fwd: RORC and DDL Information
To:
Sergey Boyarinov <boiarino@jlab.org>, David Abbott <abbottd@jlab.org>,
Stepan Stepanyan <stepanya@jlab.org>, Howard Fenker <hcf@jlab.org>, hafidi@jlab.org
Hello All,
I am attaching a reply I received from the Hungarian company manufacturing the
ALICE (not the TPC) side of the electronics.
As per their web site (http://www.cerntech.hu) the price for the
PCI-X RORC is 1540 Euro and for one
SIU card 615 Euro.
The total would be 2770 Euro for a complete readout w/o RCU and w/o spares.
There seem to be special prices for CERN collaborating institutions.
Wrote to Luciano about the RCU boards but have not heard back.
Cheers,
Stephen
Begin forwarded message:
From: "Kiss Tivadar" <tivadar.kiss@cerntech.hu>
Date: November 30, 2007 2:35:36 PM EST
To: "'Stephen Bueltmann'" <sbueltma@odu.edu>
Subject: RE: RORC and DDL Information
Dear Stephen,
We are pleased to receive your inquiry for DDL hardware. Of course it is possible for you to get such boards from our company. Delivery time would be cca. 1-3
monthes after receiving an order, depending on the quantity. (We have usually only a few cards on stock.)
The DIU (Destination Interface Unit) of a DDL link is now integrated on the RORC (Read-out Receiver Card). A single RORC (that is 100 MHz, 64-bit
PCI-X interaface card) has two DIUs integrated on it. Thus, for two DDL links (that is, to read out two ALICE RCUs) you need 2 SIU cards and 1 RORC card.
Please let me know how many quantities you are interested in, and what is a delivery schedule that would fit to your needs.
Best regards,
Tivadar Kiss
Cerntech Ltd.
Ulloi ut 31.
H-1091 Budapest
Hungary
tel: +36 1 216 8219)
fax: +36 20 919 5132
From: Stephen Bueltmann [mailto:sbueltma@odu.edu]
Sent: Thursday, November 29, 2007 4:05 PM
To: info@cerntech.hu
Subject: RORC and DDL Information
Hello,
I am involved with the experimental programme at Jefferson Lab
in the US. One of the experiments is using a TPC which we are
reading out with the ALICE TPC front end cards FECs and their
USB readout controller (U2F).
We are considering to change to the standard ALICE readout controller
(RCU) used by the ALICE TPC.
This would require to purchase also the RORC and DDL hardware.
Would it be possible and how long would it take to get one RORC, one
SIU and one DIU card from you?
Thanks in advance and
Cheers,
Stephen
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu
----------------------------------------------------------------------
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu

== Sep 2009 NOTES for the system operations ==

* firmware loading:
from croctest5: ssh root@10.1.3.241 (bottm crate, top has .236), pass is 'dcs'
cd /eg6
./program rcu_170809.bit

* documentation

http://www.ge.infn.it/~minutoli/BONUS/SW/

http://www.jlab.org/~devita/eg6/

* event size (spars off, 200 threshold - no noise): 68+4+128+Nfecs

* expected rate about 1.3kHz

== connections ==

* every crates is connected to 4 power supplies, one 4.0V and 3 3.3V

* several crates can be connected to the same power supplies

* + or - labels must be ignored, rad cable always goes to plus

* ethernets from crates goes to switch, switch is connected to the eth1 of the host machine; those connections will be used to load firmware; to get IP address crates must be off, use 'tail -f /var/log/messages' and turn crate on, message will be issued containing IP addresses:
bonuspc1(left): upper crate: 10.0.2.254, bottom crate 10.0.2.252
bonuspc2(right): upper crate: 10.1.3.242, bottom crate 10.1.3.241
bonuspc3: 10.1.3.236

* to download firmware ssh root@10.1.3.241 (etc) with password 'dcs', goto ''/eg6'' and run command ''./program rcu_170809.bit''

* fibers goes from crates to RORCs; to see if RORCs are configured by Linux, use ''/date/rorc/Linux/rorc_find''; to check connection use http://www.jlab.org/~devita/eg6/eg6_electronics.pdf

* to check FECs use command:

bonuspc1(left): /rcufec/altro-loopback/altro-loopback -N -p0 -f0 -a5 -c8 --ddl-revision=4 --ddl-channel=1 --ddl-serial=7165
bonuspc2(right): same, --ddl-serial=6038
bonuspc3(): same, --ddl-serial=9044

see /proc/rorc_map for serial and other parameters

output should looks like this
Switching on FEC 0...done.
Writing data to ALTROs for event: 1...2...3...4...done.
Reading data from ALTROs for event: 1...354 words, 2...355 words, 3...355 words, 4...355 words, done.
Verifying event: 1...2...3...4...done.
Switching off FEC 0...done.

* to check real communications do following:

cd /rcufec/fec
./fecscope
2
1,1 (rorc channel number)
3,0,0
3,0,3(the number of FECs),0,1,2
10,100,0,0,1/6

cd /date/rorc/Linux
./rorc_receive -m0 -X0 -c1 -e1
it will wait for 1 event; to issue software trigger ssh root@10.1.3.241 and type ''rcu-sh c 0x5306''

== useful docs ==

[https://clonwiki.jlab.org/wiki/clondocs/Docs/RCUFirmwar2511.pdf RCU Firmware User Manual (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/rorc_lib_rev10.pdf ALICE Detector Data Link (DDL-RORC Library) (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/UserManual_draft_02.pdf ALICE TPC Readout Chip User Manual (pdf)]

BONUS Readout

2009-10-21T02:49:25Z

129.57.167.14: /* connections */

This page contains information related to the BONUS TPC chamber readout system.

== Drivers and system software ==

'''DHCP'''

dhcpd server must be running using config file ''/etc/dhcpd.conf'' provided by CERN (copy from bonuspc1 or other configured machines); set port eth1 to static 10.10.10.10 with mask 255.0.0.0

'''RORC driver installation'''

ssh root@bonuspc2
cd /date/rorc
export DATE_MAKEFILES=/date/makefiles/

add at the beginning of GNUmakefile line
DATE_SYS := Linux

make distclean
make # errors must be ignored
make driver
make dev # (as root)
reboot

'''PHYSMEM installation'''

cd /date/physmem
make
make driver
make dev # (as root)
cp physmem.ko /lib/modules/`uname -r`/kernel/drivers/misc/ # (as root)
/sbin/depmod # (as root)

* Edit the file /boot/grub/menu.lst (as root). Pass the argument 'mem=xxxM' to the kernel to limit LINUX memory to xxxMegaBytes (xxx should be replaced by a number, in our case 1024).

* Edit the file /etc/rc.modules (as root). Add the line '/sbin/modprobe physmem physmemsize0=xxx' to limit the size of the /dev/physmem0 PHYSMEM device to xxxMegaBytes. The size of the /dev/physmem1 device will be the complement of the LINUX memory and the /dev/physmem0 memory. RORC sw test programs use /dev/physmem0, while DATE uses /dev/physmem1 memories.

chmod +x /etc/rc.modules # (as root)

* Switch off autoupdate by the following way (as root).

/etc/rc.d/init.d/yum-autoupdate stop
- Check /var/lock/subsys/yum-autoupdate (have to create if missing).
- Edit /etc/sysconfig/yum-autoupdate :
YUMUPDATE=0
YUMONBOOT=0

* reboot

== History ==

From: Stephen Bueltmann <sbueltma@odu.edu> 11/30/07 14:48
Subject: Fwd: RORC and DDL Information
To:
Sergey Boyarinov <boiarino@jlab.org>, David Abbott <abbottd@jlab.org>,
Stepan Stepanyan <stepanya@jlab.org>, Howard Fenker <hcf@jlab.org>, hafidi@jlab.org
Hello All,
I am attaching a reply I received from the Hungarian company manufacturing the
ALICE (not the TPC) side of the electronics.
As per their web site (http://www.cerntech.hu) the price for the
PCI-X RORC is 1540 Euro and for one
SIU card 615 Euro.
The total would be 2770 Euro for a complete readout w/o RCU and w/o spares.
There seem to be special prices for CERN collaborating institutions.
Wrote to Luciano about the RCU boards but have not heard back.
Cheers,
Stephen
Begin forwarded message:
From: "Kiss Tivadar" <tivadar.kiss@cerntech.hu>
Date: November 30, 2007 2:35:36 PM EST
To: "'Stephen Bueltmann'" <sbueltma@odu.edu>
Subject: RE: RORC and DDL Information
Dear Stephen,
We are pleased to receive your inquiry for DDL hardware. Of course it is possible for you to get such boards from our company. Delivery time would be cca. 1-3
monthes after receiving an order, depending on the quantity. (We have usually only a few cards on stock.)
The DIU (Destination Interface Unit) of a DDL link is now integrated on the RORC (Read-out Receiver Card). A single RORC (that is 100 MHz, 64-bit
PCI-X interaface card) has two DIUs integrated on it. Thus, for two DDL links (that is, to read out two ALICE RCUs) you need 2 SIU cards and 1 RORC card.
Please let me know how many quantities you are interested in, and what is a delivery schedule that would fit to your needs.
Best regards,
Tivadar Kiss
Cerntech Ltd.
Ulloi ut 31.
H-1091 Budapest
Hungary
tel: +36 1 216 8219)
fax: +36 20 919 5132
From: Stephen Bueltmann [mailto:sbueltma@odu.edu]
Sent: Thursday, November 29, 2007 4:05 PM
To: info@cerntech.hu
Subject: RORC and DDL Information
Hello,
I am involved with the experimental programme at Jefferson Lab
in the US. One of the experiments is using a TPC which we are
reading out with the ALICE TPC front end cards FECs and their
USB readout controller (U2F).
We are considering to change to the standard ALICE readout controller
(RCU) used by the ALICE TPC.
This would require to purchase also the RORC and DDL hardware.
Would it be possible and how long would it take to get one RORC, one
SIU and one DIU card from you?
Thanks in advance and
Cheers,
Stephen
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu
----------------------------------------------------------------------
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu

== Sep 2009 NOTES for the system operations ==

* firmware loading:
from croctest5: ssh root@10.1.3.241 (bottm crate, top has .236), pass is 'dcs'
cd /eg6
./program rcu_170809.bit

* documentation

http://www.ge.infn.it/~minutoli/BONUS/SW/

http://www.jlab.org/~devita/eg6/

* event size (spars off, 200 threshold - no noise): 68+4+128+Nfecs

* expected rate about 1.3kHz

== connections ==

* every crates is connected to 4 power supplies, one 4.0V and 3 3.3V

* several crates can be connected to the same power supplies

* + or - labels must be ignored, rad cable always goes to plus

* ethernets from crates goes to switch, switch is connected to the eth1 of the host machine; those connections will be used to load firmware; to get IP address crates must be off, use 'tail -f /var/log/messages' and turn crate on, message will be issued containing IP addresses:
bonuspc1(left): upper crate: 10.0.2.254, bottom crate 10.0.2.252
bonuspc2(right): upper crate: 10.1.3.242, bottom crate 10.1.3.241
bonuspc3: 10.1.3.236

* to download firmware ssh root@10.1.3.241 (etc) with password 'dcs', goto ''/eg6'' and run command ''./program rcu_170809.bit''

* fibers goes from crates to RORCs; to see if RORCs are configured by Linux, use ''/date/rorc/Linux/rorc_find''; to check connection use http://www.jlab.org/~devita/eg6/eg6_electronics.pdf

* to check FECs use command:

bonuspc1(left): /rcufec/altro-loopback/altro-loopback -N -p0 -f0 -a5 -c8 --ddl-revision=4 --ddl-channel=1 --ddl-serial=7165
bonuspc2(right): same, --ddl-serial=6038
bonuspc3(): same, --ddl-serial=9044

see /proc/rorc_map for serial and other parameters

output should looks like this
Switching on FEC 0...done.
Writing data to ALTROs for event: 1...2...3...4...done.
Reading data from ALTROs for event: 1...354 words, 2...355 words, 3...355 words, 4...355 words, done.
Verifying event: 1...2...3...4...done.
Switching off FEC 0...done.

* to check real communications do following:

cd /rcufec/fec
./fecscope
2
1,1 (rorc channel number)
3,0,0
3,0,3(the number of FECs),0,1,2
10,100,0,0,1/6

cd /date/rorc/Linux
./rorc_receive -m0 -X0 -c1 -e1
it will wait for 1 event; to issue software trigger ssh root@10.1.3.241 and type ''rcu-sh c 0x5306''

== useful docs ==

[https://clonwiki.jlab.org/wiki/clondocs/Docs/RCUFirmwar2511.pdf RCU Firmware User Manual (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/rorc_lib_rev10.pdf ALICE Detector Data Link (DDL-RORC Library) (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/UserManual_draft_02.pdf ALICE TPC Readout Chip User Manual (pdf)]

BONUS Readout

2009-10-21T02:33:05Z

129.57.167.14: /* connections */

This page contains information related to the BONUS TPC chamber readout system.

== Drivers and system software ==

'''DHCP'''

dhcpd server must be running using config file ''/etc/dhcpd.conf'' provided by CERN (copy from bonuspc1 or other configured machines); set port eth1 to static 10.10.10.10 with mask 255.0.0.0

'''RORC driver installation'''

ssh root@bonuspc2
cd /date/rorc
export DATE_MAKEFILES=/date/makefiles/

add at the beginning of GNUmakefile line
DATE_SYS := Linux

make distclean
make # errors must be ignored
make driver
make dev # (as root)
reboot

'''PHYSMEM installation'''

cd /date/physmem
make
make driver
make dev # (as root)
cp physmem.ko /lib/modules/`uname -r`/kernel/drivers/misc/ # (as root)
/sbin/depmod # (as root)

* Edit the file /boot/grub/menu.lst (as root). Pass the argument 'mem=xxxM' to the kernel to limit LINUX memory to xxxMegaBytes (xxx should be replaced by a number, in our case 1024).

* Edit the file /etc/rc.modules (as root). Add the line '/sbin/modprobe physmem physmemsize0=xxx' to limit the size of the /dev/physmem0 PHYSMEM device to xxxMegaBytes. The size of the /dev/physmem1 device will be the complement of the LINUX memory and the /dev/physmem0 memory. RORC sw test programs use /dev/physmem0, while DATE uses /dev/physmem1 memories.

chmod +x /etc/rc.modules # (as root)

* Switch off autoupdate by the following way (as root).

/etc/rc.d/init.d/yum-autoupdate stop
- Check /var/lock/subsys/yum-autoupdate (have to create if missing).
- Edit /etc/sysconfig/yum-autoupdate :
YUMUPDATE=0
YUMONBOOT=0

* reboot

== History ==

From: Stephen Bueltmann <sbueltma@odu.edu> 11/30/07 14:48
Subject: Fwd: RORC and DDL Information
To:
Sergey Boyarinov <boiarino@jlab.org>, David Abbott <abbottd@jlab.org>,
Stepan Stepanyan <stepanya@jlab.org>, Howard Fenker <hcf@jlab.org>, hafidi@jlab.org
Hello All,
I am attaching a reply I received from the Hungarian company manufacturing the
ALICE (not the TPC) side of the electronics.
As per their web site (http://www.cerntech.hu) the price for the
PCI-X RORC is 1540 Euro and for one
SIU card 615 Euro.
The total would be 2770 Euro for a complete readout w/o RCU and w/o spares.
There seem to be special prices for CERN collaborating institutions.
Wrote to Luciano about the RCU boards but have not heard back.
Cheers,
Stephen
Begin forwarded message:
From: "Kiss Tivadar" <tivadar.kiss@cerntech.hu>
Date: November 30, 2007 2:35:36 PM EST
To: "'Stephen Bueltmann'" <sbueltma@odu.edu>
Subject: RE: RORC and DDL Information
Dear Stephen,
We are pleased to receive your inquiry for DDL hardware. Of course it is possible for you to get such boards from our company. Delivery time would be cca. 1-3
monthes after receiving an order, depending on the quantity. (We have usually only a few cards on stock.)
The DIU (Destination Interface Unit) of a DDL link is now integrated on the RORC (Read-out Receiver Card). A single RORC (that is 100 MHz, 64-bit
PCI-X interaface card) has two DIUs integrated on it. Thus, for two DDL links (that is, to read out two ALICE RCUs) you need 2 SIU cards and 1 RORC card.
Please let me know how many quantities you are interested in, and what is a delivery schedule that would fit to your needs.
Best regards,
Tivadar Kiss
Cerntech Ltd.
Ulloi ut 31.
H-1091 Budapest
Hungary
tel: +36 1 216 8219)
fax: +36 20 919 5132
From: Stephen Bueltmann [mailto:sbueltma@odu.edu]
Sent: Thursday, November 29, 2007 4:05 PM
To: info@cerntech.hu
Subject: RORC and DDL Information
Hello,
I am involved with the experimental programme at Jefferson Lab
in the US. One of the experiments is using a TPC which we are
reading out with the ALICE TPC front end cards FECs and their
USB readout controller (U2F).
We are considering to change to the standard ALICE readout controller
(RCU) used by the ALICE TPC.
This would require to purchase also the RORC and DDL hardware.
Would it be possible and how long would it take to get one RORC, one
SIU and one DIU card from you?
Thanks in advance and
Cheers,
Stephen
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu
----------------------------------------------------------------------
---------------------------------------------------------------------
Stephen Bueltmann
Old Dominion University
At ODU: Physics Department
4600 Elkhorn Avenue
Norfolk, VA 23529, USA
Telephone (757) 683-6401
Facsimile (757) 683-5809
sbueltma@odu.edu

== Sep 2009 NOTES for the system operations ==

* firmware loading:
from croctest5: ssh root@10.1.3.241 (bottm crate, top has .236), pass is 'dcs'
cd /eg6
./program rcu_170809.bit

* documentation

http://www.ge.infn.it/~minutoli/BONUS/SW/

http://www.jlab.org/~devita/eg6/

* event size (spars off, 200 threshold - no noise): 68+4+128+Nfecs

* expected rate about 1.3kHz

== connections ==

* every crates is connected to 4 power supplies, one 4.0V and 3 3.3V

* several crates can be connected to the same power supplies

* + or - labels must be ignored, rad cable always goes to plus

* ethernets from crates goes to switch, switch is connected to the eth1 of the host machine; those connections will be used to load firmware; to get IP address crates must be off, use 'tail -f /var/log/messages' and turn crate on, message will be issued containing IP addresses:
bonuspc1(left): upper crate: 10.0.2.254, bottom crate 10.0.2.252
bonuspc2(right): upper crate: 10.1.3.242, bottom crate 10.1.3.241

* to download firmware ssh root@10.1.3.241 (etc) with password 'dcs', goto ''/eg6'' and run command ''./program rcu_170809.bit''

* fibers goes from crates to RORCs; to see if RORCs are configured by Linux, use ''/date/rorc/Linux/rorc_find''; to check connection use http://www.jlab.org/~devita/eg6/eg6_electronics.pdf

* to check FECs use command:

bonuspc1(left): /rcufec/altro-loopback/altro-loopback -N -p0 -f0 -a5 -c8 --ddl-revision=4 --ddl-channel=1 --ddl-serial=7165
bonuspc2(right): same, --ddl-serial=6038
bonuspc3(): same, --ddl-serial=9044

see /proc/rorc_map for serial and other parameters

output should looks like this
Switching on FEC 0...done.
Writing data to ALTROs for event: 1...2...3...4...done.
Reading data from ALTROs for event: 1...354 words, 2...355 words, 3...355 words, 4...355 words, done.
Verifying event: 1...2...3...4...done.
Switching off FEC 0...done.

* to check real communications do following:

cd /rcufec/fec
./fecscope
2
1,1 (rorc channel number)
3,0,0
3,0,3(the number of FECs),0,1,2
10,100,0,0,1/6

cd /date/rorc/Linux
./rorc_receive -m0 -X0 -c1 -e1
it will wait for 1 event; to issue software trigger ssh root@10.1.3.241 and type ''rcu-sh c 0x5306''

== useful docs ==

[https://clonwiki.jlab.org/wiki/clondocs/Docs/RCUFirmwar2511.pdf RCU Firmware User Manual (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/rorc_lib_rev10.pdf ALICE Detector Data Link (DDL-RORC Library) (pdf)]

[https://clonwiki.jlab.org/wiki/clondocs/Docs/UserManual_draft_02.pdf ALICE TPC Readout Chip User Manual (pdf)]

ERROR Book

2009-06-11T16:41:43Z

129.57.167.14:

* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:

..........................
taskDelay(sysClkRateGet()*5)
Args = -session clasprod -objects sc2 ROC -i
CODA_Init reached
CODA_Init 1
CODA_Init 2
sc2
CODA_Init 11
CODA_Init: objectTy >(null)<
CODA_Init 12
CODA_Init: use 'SESSION' as >clasprod<
Tcl_AppInit CALLS !!!!!
11-11
11-12
11-13
11111111111111111
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
value = 0 = 0x0
taskSpawn "TCP_SERVER",250,0,100000,tcpServer
value = 231091088 = 0xdc62b90
proconhost
value = 0 = 0x0
......................

Second reboot shows the same:

...................
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
list >clon10 clasrun 2508 9998<
machine name = clon10
user ID = 2508
group ID = 9998
.................

* Sergey B. around 6-June-2009: ec2pmc1 error message:

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait= 199 send= 8 microsec per event (nev=12755)
CALLING customized :
machine check
SRAM ERROR :
SRAM Error Address : 0x00000000.f2000504
SRAM Error Data Low : 0x006805e0
SRAM Error Data High : 0x00420000
SRAM Error Parity : 0x00000004
SRAM Error Cause : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-06-11T16:37:58Z

129.57.167.14:

* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:

..........................
taskDelay(sysClkRateGet()*5)
Args = -session clasprod -objects sc2 ROC -i
CODA_Init reached
CODA_Init 1
CODA_Init 2
sc2
CODA_Init 11
CODA_Init: objectTy >(null)<
CODA_Init 12
CODA_Init: use 'SESSION' as >clasprod<
Tcl_AppInit CALLS !!!!!
11-11
11-12
11-13
11111111111111111
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
value = 0 = 0x0
taskSpawn "TCP_SERVER",250,0,100000,tcpServer
value = 231091088 = 0xdc62b90
proconhost
value = 0 = 0x0
......................

* Sergey B. around 6-June-2009: ec2pmc1 error message:

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait= 199 send= 8 microsec per event (nev=12755)
CALLING customized :
machine check
SRAM ERROR :
SRAM Error Address : 0x00000000.f2000504
SRAM Error Data Low : 0x006805e0
SRAM Error Data High : 0x00420000
SRAM Error Parity : 0x00000004
SRAM Error Cause : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-06-08T18:57:23Z

129.57.167.14:

* Sergey B. around 6-June-2009: ec2pmc1 error message:

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait= 199 send= 8 microsec per event (nev=12755)
CALLING customized :
machine check
SRAM ERROR :
SRAM Error Address : 0x00000000.f2000504
SRAM Error Data Low : 0x006805e0
SRAM Error Data High : 0x00420000
SRAM Error Parity : 0x00000004
SRAM Error Cause : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr) ByteEnable=0 Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-06-04T18:43:28Z

129.57.167.14:

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry +68 : coda_pmc ()
1e2275c0 coda_pmc +144: mysql_query ()
1e258520 mysql_query +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
->

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-06-04T18:42:40Z

129.57.167.14:

* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:

0x1e42a360 (coda_pmc): bb_new: 'big' buffer
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-05-15T12:59:02Z

129.57.167.14:

* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait= 176 send= 6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait= 176 send= 12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99

interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.

* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-05-14T12:35:41Z

129.57.167.14:

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
Tasks: 92 total, 2 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi, 0.1% si
Mem: 3995356k total, 3968880k used, 26476k free, 88168k buffers
Swap: 8385920k total, 208k used, 8385712k free, 3432056k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4471 root 25 0 5908 4540 1956 R 100 0.1 300:24.06 rtserver.x
1 root 16 0 4756 556 460 S 0 0.0 0:11.25 init
2 root RT 0 0 0 0 S 0 0.0 0:03.85 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:10.23 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:01.79 migration/1
5 root 34 19 0 0 0 S 0 0.0 1:03.72 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.62 migration/2

while normally it uses fractions of percent.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ERROR Book

2009-05-14T12:30:30Z

129.57.167.14:

* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year.

* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.

* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

* Sergey B. 7-may-2008: EB1 crashed, last messages:

EB: WARNING - resyncronization in crate controller number 5 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
EB: WARNING - resyncronization in crate controller number 6 ... fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted

* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine

* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s) : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
# kernel - use the kernel for notification
# poll - use polling for notification
# none - don't use any notification
#
# the poll_limit is the number of seconds
# that must pass before a resource is polled again.
# It is optional, and if it is not present the previous
# value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..

* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait= 135 send= 28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait= 130 send= 31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait= 135 send= 31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait= 135 send= 31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
ERROR: bufout overflow - skip the rest ...
bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait= 128 send= 42 microsec per event (nev=12778)
proc_thread: wait= 371 send= 31 microsec per event (nev=6291)
ROC # 22 Event # 0 : Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................

At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time

* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure

and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

Query test_ts2 table failed: Error reading table'test_ts2' definition
ec3 ec3

If one runcontrol is restarted, then it works, but another one complains !!!

* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie

* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
[1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
[2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
[3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
[4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
[5] 0xfea60020(), at 0xfea60020
(dbx)

* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 : Bad Block Read signature 0x00680BD9 -> resyncronize !!!
end run failed. Reboot sc2.
During end transition ec2 froze with message:
interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181 0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled
Reboot ec2. Started new run 55463, everything looks normal.

* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
data taking continues, but runcontrol printed message:
WARN : ec2 has not reported status for 1516 seconds
ERROR : ec2 is in state disconnected should be active
ec2pmc1 looked fine; end run failed, need to reboot ec2

ZFS

2009-03-26T14:09:56Z

129.57.167.14:

'''RAID partitions on clondaq2'''

Four zfs partitions made on clonfs raid system connected to clondaq2: raid0, raid1, raid2 and raid3.

NOTE: at some point (after machine was renamed from clonxt3 and several reboots) raid3 dissapeared. When I tried to created it said it exist, but mount did not worked. Maybe it happened when I was playing with the same partition from clondaq1. Anyway, I created raid3 again, will watch it ...

To create ''raid0-raid3'' on clondaq2:

zpool create -f raid0 c0t600C0FF00000000009839205A879C400d0s2
zpool create -f raid1 c0t600C0FF0000000000983926ED5760700d0s2
zpool create -f raid2 c0t600C0FF00000000009839223AFDC0A00d0s2
zpool create -f raid3 c0t600C0FF0000000000983920939EB6300d0s2

zfs set mountpoint=/mnt/raid0 raid0
zfs set mountpoint=/mnt/raid1 raid1
zfs set mountpoint=/mnt/raid2 raid2
zfs set mountpoint=/mnt/raid3 raid3

To destroy for example ''raid3'' on clondaq2:

zfs umount /mnt/raid3
zpool destroy raid3

To display the list of all zfs partitions:

zfs list

NOTE: After reboot zfs partitions maybe (will be ?) unmounted, for example on clon10 'zfs list' returns following:

NAME USED AVAIL REFER MOUNTPOINT
data 134G 214G 134G /data
data1 131K 182G 49K /data1

but ''/data'' and ''/data1'' do not exists or empty. To mount those two partitions, do following as 'root':

zfs mount data
zfs mount data1

'''RAIDOLD on clon10'''

mkdir /raidold
zpool create -f raidold \
c1t32d0 c1t33d0 c1t34d0 c1t35d0 c1t36d0 c1t37d0 c1t38d0 c1t39d0 c1t40d0 c1t41d0 c1t42d0 \
c1t48d0 c1t49d0 c1t50d0 c1t51d0 c1t52d0 c1t53d0 c1t54d0 c1t55d0 c1t56d0 c1t57d0 c1t58d0 \
c1t64d0 c1t65d0 c1t66d0 c1t67d0 c1t68d0 c1t69d0 c1t70d0 c1t71d0 c1t72d0 c1t73d0 c1t74d0 \
c1t80d0 c1t81d0 c1t82d0 c1t83d0 c1t84d0 c1t85d0 c1t86d0 c1t87d0 c1t88d0 c1t89d0 c1t90d0
zfs set mountpoint=/raidold raidold

'''extra disks on clons'''

clon01:
zpool create -f space c1t1d0
zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 17.9G 49.0G 94K /rpool
rpool/ROOT 8.94G 49.0G 18K legacy
rpool/ROOT/s10s_u6wos_07b 8.94G 49.0G 6.42G /
rpool/ROOT/s10s_u6wos_07b/var 2.53G 49.0G 2.53G /var
rpool/dump 1.00G 49.0G 1.00G -
rpool/export 38K 49.0G 20K /export
rpool/export/home 18K 49.0G 18K /export/home
rpool/swap 8G 57.0G 24K -
space 89.5K 66.9G 1K /space

To change swap size for example, do following:

zfs set volsize=4G rpool/swap

To add new disk to the existing zfs pool, do following:

iostat -E (shows all disks available)
zpool add rpool ...

=============== from the web =================

Ok, ZFS is now in the tree, what's now? Below you'll find some
instructions how to quickly make it up and running.

First of all you need some disks. Let's assume you have three spare SCSI
disks: da0, da1, da2.

Add a line to your /etc/rc.conf to start ZFS automatically on boot:

# echo 'zfs_enable="YES"' >> /etc/rc.conf

Load ZFS kernel module, for the first time by hand:

# kldload zfs.ko

Now, setup one pool using RAIDZ:

# zpool create tank raidz da0 da1 da2

It should automatically mount /tank/ for you.

Ok, now put /usr/ on ZFS and propose some file systems layout. I know
you probably have some files already, so we will work on /tank/usr
directory and once we ready, we will just change the mountpoint to /usr.

# zfs create tank/usr

Create ports/ file system and enable gzip compression on it, because
most likely we will have only text files there. On the other hand, we
don't want to compress ports/distfiles/, because we keep compressed
stuff already in-there:

# zfs create tank/usr/ports
# zfs set compression=gzip tank/usr/ports
# zfs create tank/usr/ports/distfiles
# zfs set compression=off tank/usr/ports/distfiles

(You do see how your life is changing, don't you?:))

Let's create home file system, my own home/pjd/ file system. I know we
use RAIDZ, but I want to have directory where I put extremly important
stuff, you I'll define that each block has to be stored in tree copies:

# zfs create tank/usr/home
# zfs create tank/usr/home/pjd
# zfs create tank/usr/home/pjd/important
# zfs set copies=3 tank/usr/home/pjd/important

I'd like to have directory with music, etc. that I NFS share. I don't
really care about this stuff and my computer is not very fast, so I'll
just turn off checksumming (this is only for example purposes! please,
benchmark before doing it, because it's most likely not worth it!):

# zfs create tank/music
# zfs set checksum=off tank/music
# zfs set sharenfs=on tank/music

Oh, I almost forget. Who cares about access time updates?

# zfs set atime=off tank

Yes, we set it only on tank and it will be automatically inherited by
others.

Will be also good to be informed if everything is fine with our pool:

# echo 'daily_status_zfs_enable="YES"' >> /etc/periodic.conf

For some reason you still need UFS file system, for example you use ACLs
or extended attributes which are not yet supported by our ZFS. If so,
why not just use ZFS to provide storage? This way we gain cheap UFS
snapshots, UFS clones, etc. by simply using ZVOLs.

# zfs create -V 10g tank/ufs
# newfs /dev/zvol/tank/ufs
# mount /dev/zvol/tank/ufs /ufs

# zfs snapshot tank/ufs at 20070406
# mount -r /dev/zvol/tank/ufs at 20070406 /ufs20070406

# zfs clone tank/ufs at 20070406 tank/ufsok
# fsck_ffs -p /dev/zvol/tank/ufsok
# mount /dev/zvol/tank/ufsok /ufsok

Want to encrypt your swap and still use ZFS? Nothing more trivial:

# zfs create -V 4g tank/swap
# geli onetime -s 4096 /dev/zvol/tank/swap
# swapon /dev/zvol/tank/swap.eli

Trying to do something risky with your home? Snapshot it first!

# zfs snapshot tank/home/pjd at justincase

Turns out it was more stupid than risky? Rollback your snapshot!

# zfs rollback tank/home/pjd at justincase
# zfs destroy tank/home/pjd at justincase

Ok, everything works, we may set tank/usr as our real /usr:

# zfs set mountpoint=/usr tank/usr

Don't forget to read zfs(8) and zpool(8) manual pages and SUN's ZFS
administration guide:

http://www.opensolaris.org/os/community/zfs/docs/zfsadmin.pdf

--

======================================== from the web =============================================

ZFS and Containers: An Example
This section is a step-by-step guide that shows how to perform certain ZFS file system tasks inside of Solaris Containers; for example, taking snapshots and managing data compression. It does this by going through the following steps:

* Creating a zpool
* Creating a Zone
* Allocating a ZFS File System to a Zone
* Creating New File Systems
* Applying Quota to the File Systems
* Changing the Mountpoint of a File System
* Setting the Compression Property
* Taking a Snapshot

Each of these steps is described in detail below.

Creating a zpool
ZFS uses device names or partition names when dealing with pools and devices. For a device, this will be something like c1t0d0 (for a SCSI device) or c1d1 (for an IDE device). For a partition, it will be something like c1t0d0s0 (for a SCSI device) or c1d1s0 (for an IDE device). This example creates a pool that is mirrored using two disks.

1. To create a zpool in the global zone, use the zpool create command. Typically, you use two devices to provide redundancy.

Global# zpool create mypool mirror c2t5d0 c2t6d0

Note that the zpool create command may fail if the devices are in use or contain some types of existing data (e.g. UFS file system). If they are in use, you will need to unmount them or otherwise stop using them. If they contain existing data you can use the -f(force) flag to override the safety check, but be sure that you are not destroying any data you want to retain.
2. Examine the pool properties using the zpool list command.

Global# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT
mypool

199G

164K

199G

0%

online

--
This shows you that there is one zpool, named mypool,with a capacity of 199GBytes.

Creating a Zone
To show ZFS working in an environment that is isolated from other applications on the system, you need to create a zone. To create a zone:

1. Create a directory where the zone file system will reside, using the mkdir command. Be sure to chose a location where the filesystem has at least 80MBytes of available space.

Global# mkdir /zones

Note that in this example, for the sake of brevity, the root file system of the zone is a UFS file system.

2. Configure the zone (myzone), using the zonecfg command, and specify the location of the zone's files (/zones/myzone). Use the following series of commands.

Global# zonecfg -z myzone
myzone: No such zone configured
Use 'create' to begin configuring a new zone
zonecfg:myzone< create
zonecfg:myzone< set zonepath=/zones/myzone
zonecfg:myzone< verify
zonecfg:myzone< commit
zonecfg:myzone< exit

Again, for the purpose of streamlining, this example uses a very minimal zone. For more details on creating zones see the Solaris Containers How To Do Stuff guide at: sun.com/software/solaris/howtoguides/containersLowRes.jsp

3. Install the zone by using the zoneadm.

Global# zoneadm -z myzone install
Preparing to install zone >myzone<
[Output from zoneadm, this may take a few mins]

4. Boot the zone to complete the installation, using the zoneadm command.

Global# zoneadm -z myzone boot

5. Use the zlogin command to connect to the zone console.

Global# zlogin -C myzone
[Connected to zone 'myzone' console]
[Initial zone boot output, service descriptions are loaded etc.]
It may take a short while for the first boot to configure everything, load all the service descriptors, and so on. You will need to answer the system configuration details. Some suggestions are:

Terminal=(12)X Terminal Emulator (xterms)
Not Networked
No Kerberos
Name service = None
Time Zone = your-time-zone
root passwd = (Your choice—remember it though!)

The zone will reboot after you have provided the configuration information.

6. Before you can proceed to the next stage, the configured zone needs to be shutdown (configuration changes are only applied when the zone boots).

Global# zlogin myzone init 5

Allocating a ZFS File System to a Zone
Now that you have a zpool (mypool) and a zone (myzone) you are ready to allocate a ZFS file system to the zone.

1. To create a ZFS file system, use the zfs create command.

Global# zfs create mypool/myzonefs

2. To apply a quota to the file system, use the zfs set quota command.

Global# zfs set quota=5G mypool/myzonefs

The file system and all of its child file systems can be no larger than the designated quota. Note that both these steps must be performed in the global zone. Also notice that creating the file system in ZFS is much simpler than with a traditional file system/volume manager combination.

To illustrate the isolation/security properties of containers with ZFS this example now creates a ZFS file system that will remain outside the container. There is no need to apply a quota to this outside file system.

3. To create this other file system, again use the zfs create command.

Global# zfs create mypool/myfs

4. To show the properties of all the pool and the file systems, use the zfs list command.

Global# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

396G

197G

99.5K

/mypool
mypool/myfs

98.5K

197GK

98.5K

/mypool/myfs
mypool/myzonefs

98.5K

5G

98.5K

/mypool/myzonefs

To make the file system (myzonefs) available in the zone (myzone), the zone configuration needs to be updated.

5. To update the zone configuration, use the zonecfg command.

Global# zonecfg -z myzone
zonecfg:myzone> add dataset
zonecfg:myzone:dataset> set name=mypool/myzonefs
zonecfg:myzone:dataset> end
zonecfg:myzone> commit
zonecfg:myzone> exit

The mypool/myzonefs file system is now added to the zone configuration. Note that these steps must be performed with the zone shut down, otherwise the zone configuration changes would not be visible until the next reboot. To check that the zone is shut down try logging into it using zlogin myzone. If the zone is shut down the login will fail; if the zone is running you will see a login prompt—login as root and shut the zone down with init 5. These steps are performed in the global zone.

6. Now boot the zone.

Global# zoneadm -z myzone boot

7. Log in to the zone. (Leave a few seconds for the zone to boot.)

Global# zlogin -C myzone
[Connected to zone 'myzone' pts/3]
[Usual Solaris login sequence]

8. List the ZFS file systems in the zone.

NAME USED AVAIL REFER MOUNTPOINT
mypool

0M

200B

--

/mypool
mypool/myzonefs

8K

5G

8K

/mypool/myzonefs

Note the 5GByte maximum available from the external quota and that the other file systems in the pool (mypool/myfs) are not visible. This demonstrates the isolation property that Containers provide.

Creating New File Systems
Administering ZFS file systems from the non-global zone is done just like it is in the global zone, although you are limited to operating within the file system that is allocated to the zone (mypool/myzonefs). New ZFS file systems are always created as a child of this file system because this is the only ZFS file system the non-global zone can see. The administrator in the non-global zone can create these file systems. There is no need to involve administrator of the global zone, though the administer could do so if it were necessary.

1. To create a newfile system, use the zfs create command.

MyZone# zfs create mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

594M

197G

99K

/mypool
mypool/myzonefs

197K

5.00G

98.5K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

5.00G

98.5K

/mypool/myzonefs/tim

The non-global zone administrator can create as many child file systems as s/he wants and each child file system can have its own file systems, and in that way form a hierarchy.

As a demonstration that the non-global zone administrator is limited to the assigned file systems, this example demonstrates trying to break security by creating a file system outside the container's "space".

2. Try to create another file system outside of mypool/myzonefs, using the zfs create command.

MyZone# zfs create mypool/myzonefs1
cannot create 'mypool/myzonefs1': permission denied

As you can see, ZFS and zones security denies permission for the non-global zone to access resources it has not been allocated and the operation fails.

Applying Quota to the File Systems
Typically, to prevent the user consuming all of the space, a non-global zone administrator will want to apply a quota to the new file system. Of course, the child's quota can't be more than 5GByte as that's the quota specified by the global zone administrator to all of the file systems below mypool/myzonefs.

1. To set a quota on our new file system, use the zfs set quota command.

MyZone# zfs set quota=1G mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

508M

197G

99K

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

1024M

98.5K

/mypool/myzonefs/tim

The administrator of the non-global zone has set the quota of the child file system to be 1G. They have full authority to do this because they are operating on their delegated resources and do not need to involve the global zone administrator.

The ZFS property inheritance mechanism applies across zone boundaries, so the non-global zone administrator can specify his/her own property values should s/he wish to do so. As with normal ZFS property inheritance, these override inherited values.

Changing the Mountpoint of a File System
Now that the file system is set up and has the correct quota assigned to it, it is ready for use. However, the place where the file system appears (the mountpoint) is partially dictated by what the global zone administrator initially chose as the pool name (in this example, mypool/myzonefs). But typically, a non-global zone administrator would want to change it.

1. To change the mountpoint, use the zfs set mountpoint command.

MyZone# zfs set mountpoint=/export/home/tim mypool/myzonefs/tim
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

508M

197G

99K

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs/tim

98.5K

1024M

98.5K

/export/home/tim

Note that the mountpoint can be changed for any file system independently.

Setting the Compression Property
The next example demonstrates the compression property. If compression is enabled, ZFS will transparently compress all of the data before it is written to disk.

The benefits of compression are both saved disk space and possible write speed improvements.

1. To see what the current compression setting is, use the zfs get command.

MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim

NAME PROPERTY VALUE SOURCE
mypool

compression

off

default
mypool/myzonefs

compression

off

default
mypool/myzonefs/tim

compression

off

default

Beaware that the compression property on the pool is inherited by the file system and child file system. So if the non- global zone administrator sets the compression property for the delegated file system, it will set it for everything below, as well.

2. To set the compression for the file system, use the zfs set command.

MyZone# zfs set compression=on mypool/myzonefs

3. Examine the compression property again in the non-global zone.

MyZone# zfs get compression mypool mypool/myzonefs mypool/myzonefs/tim

NAME PROPERTY VALUE SOURCE
mypool

compression

off

default
mypool/myzonefs

compression

on

local
mypool/myzonefs/tim

compression

off

Inherited from mypool/myzonefs

Note the compression property has been inherited by mypool/myzonefs/tim as with normal ZFS administration.

Taking a Snapshot
One of the major advantages of ZFS is the ability to create an instant snapshot of any file system. By delegating a file system to a non-global zone this feature becomes available as an option for the non-global zone administrator.

1. To take a snapshot named "1st" of the file system, use the zfs snapshot command.

MyZone# zfs snapshot mypool/myzonefs@1st
MyZone# zfs list

NAME USED AVAIL REFER MOUNTPOINT
mypool

512K

99K

default

/mypool
mypool/myzonefs

198K

5.00G

99K

/mypool/myzonefs
mypool/myzonefs@1st

OK

--

99K

--
mypool/myzonefs/tim

98.5K

1024M

98.5K

/export/home/tim

As with ZFS file systems in the global zone, this snapshot is now accessible from the root of the file system in .zfs/snapshot/1st.

Back To Top

Summary
Once a zone has been created and a ZFS file system has been allocated to it, the administrator for that (non-global) zone can create file systems, take snapshots, create clones, and perform all the other functions of an administrator—within that zone. Yet the global zone, and any other zones, are fully isolated from whatever happens in that zone.

The integration of Solaris Containers and Solaris ZFS is just another way that the Solaris 10 OS is providing cost benefits to customers by allowing them to safely consolidate applications and more easily manage the data those applications use.

Foundry Switches

2009-02-20T19:18:24Z

129.57.167.14:

== BigIron RX-8 ==

Main clon cluster switch located in counting room. Inventory number: F423964, p/n: BI-RX-8-AC, s/n: SA29085027.

== general information ==

'''PROBLEM 1:''' make sure broadcast is working between 167,68,69,160 subnets, otherwise EPICS will not work; it must be set in BigIron switch

'''PROBLEM 2:''' switches in the hall must have spanning tree OFF; it must be set in every Foundry switch.

To access switches normally goto clon10/clon00 and do ''ssh boiarino@<switch_ip_name>'', use 'boiarino's pass. Then type 'enable' and use pin and hardkey pin.

To access switches in the hall using tsconnect: the same passwords (?).

'''VLAN change'''

show vlan
configure terminal
vlan 596
no tagged ethernet 13/1
no untagged ethernet 13/2 to 13/8
vlan 569
tagged ethernet 13/1
untagged ethernet 13/2 to 13/8
exit
exit
write memory

'''Who is connected'''

List of all ports currently active:
show int | include is up

Get MAC address(es) of all devices connected to the particular port:
show mac-ad | include 3/17

Get IP address for particular MAC address:
show arp mac 0006.5b49.8ef6

Get full table of IP addresses, MAC addresses and port numbers:
show arp

The same for particular IP address:
show arp | include xxx.xxx.xxx.xxx

X server

2009-01-07T19:52:26Z

129.57.167.14: /* Solaris CDE */

That section contains SOME useful information about X servers.

== Solaris: trying to make dtlogin to use /tmp/.Xauthority.$LOGNAME ==

NOT in /etc/X11 (/etc/X11/gdm/gdm.conf was changed, then /etc/X11 was renamed, no effects)

ERROR FILE /var/dt/Xerrors !!!

some files copied from /usr/dt/config to /etc/dt/config, file Xconfig was modified, effects:

##Dtlogin*environment: XAUTHORITY=/tmp/.Xauthority.$LOGNAME # LOGNAME does not work ??!!
##Dtlogin*userAuthDir: /tmp
Dtlogin*environment: XAUTHORITY=/tmp/.Xauthority.root
Dtlogin*AuthFile: /tmp/.Xauthority.root

But now cannot login from console, and .Xauthority still created in HOME directory. If make link in /tmp:
/tmp/.Xauthority.root to /root/.Xauthority, it works ! Who creates /root/.Xauthority ????

File /etc/dt/config/Xservers calls /usr/X11/bin/Xserver which is script. For arch=sun4 it calls /usr/openwin/bin/Xsun (hide it, error reported).

??? It looks like has configs in /usr/openwin/server/etc, or can be copied to /etc/openwin/server/etc ?

'''clon04: /etc/X11/gdm/gdm.conf set /tmp'''

'''GOOD ~clasrun/.dt/startlog:'''

--- Tue Jan 6 15:25:13 EST 2009
--- /usr/dt/bin/Xsession starting...
--- starting /usr/dt/bin/dthello &
--- starting /usr/openwin/bin/speckeysd
--- Xsession started by dtlogin
--- starting /usr/dt/bin/dtsession_res -load -system
--- sourcing /home/clasrun/.dtprofile...
clon00.jlab.org being added to access control list
clon01.jlab.org being added to access control list
clon02.jlab.org being added to access control list
clon03.jlab.org being added to access control list
clon04.jlab.org being added to access control list
clon05.jlab.org being added to access control list
clon06.jlab.org being added to access control list
clon10.jlab.org being added to access control list
--- sourcing /usr/dt/config/Xsession.d/0010.dtpaths...
--- sourcing /usr/dt/config/Xsession.d/0015.sun.env...
--- sourcing /usr/dt/config/Xsession.d/0020.dtims...
--- sourcing /usr/dt/config/Xsession.d/0030.dttmpdir...
--- sourcing /usr/dt/config/Xsession.d/0040.xmbind...
--- sourcing /usr/dt/config/Xsession.d/1001.swupnot...
--- sourcing /usr/dt/config/Xsession.d/1099.br...
/usr/dt/bin/Xsession[624]: /var/tmp/brlog: cannot create
--- starting /usr/dt/bin/dtsearchpath
--- starting /usr/dt/bin/dtappgather &
--- starting /usr/dt/bin/dsdm &
--- session log file is /dev/null
--- DTSOURCEPROFILE is 'true' (see /home/clasrun/.dtprofile)
--- execing /usr/dt/bin/dtsession with a /bin/tcsh login shell ...
--- starting desktop on /dev/pts/5
- .login starting -

-

- start /usr/local/clas/release/current/clon/common/scripts/clasrun.alias -

- finish /usr/local/clas/release/current/clon/common/scripts/clasrun.alias -

-

-

- start /usr/local/clas/release/current/clon/common/scripts/complete.tcsh -

- finish /usr/local/clas/release/current/clon/common/scripts/complete.tcsh -

-

-

- .login has been executed -

-

- Welcome to the CLAS online run account on Tue Jan 6 15:25:14 EST 2009

-

-

/usr/dt/bin/ttsession[24761]: starting

Notification Application cannot run: User is not running
in the GNOME environment.

== bla ==

xdm ? make /usr/openwin/lib/xdm ???

DisplayManager.DISPLAY.userAuthDir
When xdm is unable to write to the usual user authori-
zation file ($HOME/.Xauthority), it creates a unique
file name in this directory and points the environment
variable XAUTHORITY at the created file. It uses /tmp
by default.

SETUP PROGRAM
The Xsetup file is run after the server is reset, but before
the Login window is offered. The file is typically a shell
script. It is run as root, so should be careful about secu-
rity. This is the place to change the root background or
bring up other windows that should appear on the screen
along with the Login widget.
In addition to any specified by DisplayManager.exportList,
the following environment variables are passed:
DISPLAY the associated display name
PATH the value of DisplayManager.DISPLAY.systemPath
SHELL the value of DisplayManager.DISPLAY.systemShell
XAUTHORITY may be set to an authority file
Note that since xdm grabs the keyboard, any other windows
will not be able to receive keyboard input. They will be
able to interact with the mouse, however; beware of poten-
tial security holes here. If
DisplayManager.DISPLAY.grabServer is set, Xsetup will not be
able to connect to the display at all. Resources for this
program can be put into the file named by
DisplayManager.DISPLAY.resources.

STARTUP PROGRAM
The Xstartup file is typically a shell script. It is run as
root and should be very careful about security. This is the
place to put commands which add entries to /etc/utmp, mount
users' home directories from file servers, display the mes-
sage of the day, or abort the session if logins are not
allowed.
In addition to any specified by DisplayManager.exportList,
the following environment variables are passed:
DISPLAY the associated display name
HOME the initial working directory of the user
USER the user name
PATH the value of DisplayManager.DISPLAY.systemPath
SHELL the value of DisplayManager.DISPLAY.systemShell
XAUTHORITY may be set to an authority file
No arguments are passed to the script. Xdm waits until this
script exits before starting the user session. If the exit
value of this script is non-zero, xdm discontinues the ses-
sion and starts another authentication cycle.

== Solaris CDE ==

We are running it on Sparc-based workstations. When istalling from scratch, copy ''/usr/dt/config/Xservers'' to ''/etc/dt/config/'' (create that directory first) and modify to enforce multiple screen support. See backgroud information at [[copy from Sun site about CDE]].

What was learned during last struggle on October 27, 2007:

* first, create directory ''/etc/dt/config'' and copy there file ''/usr/dt/config/Xservers''; the only uncommented line must look like following (it is important to use ''/usr/openwin/bin/Xsun'', not ''Xserver'' etc):

:0 Local local_uid@console root /usr/openwin/bin/Xsun :0 -dev /dev/pfb0a bottom -dev /dev/pfb0b top

* after new video card was installed, or old card was removed, reboot with command

reboot -- -r

* to see all available graphic cards type command

fbconfig -list
Device-Filename Specific Config Program
--------------- -----------------------
/dev/fbs/pfb0 not configurable
/dev/fbs/pfb0a not configurable
/dev/fbs/pfb0b not configurable
/dev/fbs/pfb1 SUNWpfb_config
/dev/fbs/pfb1a SUNWpfb_config
/dev/fbs/pfb1b SUNWpfb_config
/dev/fbs/m640 SUNWm64_config

('not configurable' cards were installed once but then removed, they are not in a system anymore). In shown list device ''/dev/fbs/pfb1'' is not real device, you should use ''/dev/fbs/pfb1a'' and ''/dev/fbs/pfb1b'' instead. You can use aliases, for example in our case command

ll /dev/fb4

will show

/dev/fb4 -> fbs/pfb1a

which is link to the real device.

* if several displays are connected to the system, adjust ''/etc/dt/config/Xservers'' file, for example for ''clon03'' (3 monitors) it look as following:

:0 Local local_uid@console root /usr/X11/bin/Xserver :0 -nobanner -dev /dev/fb6 left -dev /dev/fb4 right -dev /dev/fb5 top

(the position of next monitor defined relatively to previous one). If you want to learn more about Xserver parameters, command

X -help

will show all possible options.

* to restart X server, type

/etc/init.d/dtlogin stop

then login as root from non-graphic prompt and type

/etc/init.d/dtlogin start
exit

and wait for X server to start. If it did not start, it will store erros in file and tell you where that file is located.

* to see the list of resolution modes available type

fbconfig -res \?
Valid values for -res option are:
VESA_STD_640x480x60 [1][2]
VESA_STD_640x480x72 [1]
VESA_STD_640x480x75 [1][2]
VESA_STD_800x600x75 [1][2]
VESA_STD_1024x768x60 [1]
VESA_STD_1024x768x70 [1][2]
VESA_STD_1024x768x75 [1]
SUNW_STD_1024x768x77
SUNW_STD_1024x800x84
SUNW_STD_1152x900x66 [1]
SUNW_STD_1152x900x76 [1]
SUNW_STD_1280x768x56
SUNW_STD_1280x800x76
VESA_STD_1280x1024x60 [2]
SUNW_STD_1280x1024x67 [1]
VESA_STD_1280x1024x75 [1][2]
SUNW_STD_1280x1024x76 [1][3]
VESA_STD_1280x1024x85
SUNW_STD_1440x900x76
SUNW_STD_1600x1000x66
SUNW_STD_1600x1000x76
SUNW_DIG_1600x1024x60
VESA_STD_1600x1200x60
SUNW_DIG_1600x1200x60
SI-LMT_RFE_1600x1200x73
VESA_STD_1600x1200x75
SUNW_STD_1600x1280x76
VESA_STD_1792x1344x60
VESA_STD_1792x1344x75
SUNW_DIG_1920x1080x60
SUNW_STD_1920x1080x72
SUNW_DIG_1920x1200x60
SUNW_STD_1920x1200x70
SUNW_STD_1920x1200x75
SUNW_FSC_640x480x60
VESA_STD_720x400x85
SUNW_STD_640x480x180
VESA_STD_800x600x56 [1]
VESA_STD_800x600x60 [1][2]
VESA_STD_800x600x72 [1]
VESA_STD_800x600x85
VESA_STD_1024x768x85
VESA_STD_1152x864x75 [2]
SUNW_STD_1280x1024x60
SGI_STD_1600x1024x60
VESA_STD_1600x1200x65
VESA_STD_1600x1200x70
SIEMENS_STD_1600x1200x73
VESA_STD_1600x1200x85
VESA_STD_1856x1392x60
VESA_STD_1856x1392x75
SUNW_DIG_1400x1050x60
Notes:
[1] monitor A supports these resolutions.
[2] monitor B supports these resolutions.
[3] preferred resolution for monitor A
All resolutions which are not marked by above "[x]" are not supported
Use unsupported resolutions at your own risk.
Abbreviations such as "1280x1024x75" may also be used

To get information about particular videocard and monitor, use
fbconfig -dev /dev/fb6 -propt
fbconfig -dev /dev/fb6 -prconf

Based on that information change resolution and depth. If system tells you that resolution you are trying to set may not be supported by particular display, try another setting. Following settings were used for ''clon01'' and ''clon03'':

clon01:clasrun> fbconfig -dev /dev/fbs/pfb0a -res VESA_STD_1600x1200x75 -depth 24
clon01:clasrun> fbconfig -dev /dev/fbs/pfb0b -res SUNW_STD_1280x1024x76 -depth 24

clon03:clasrun> fbconfig -dev /dev/fb4 -res VESA_STD_1600x1200x75 -depth 24
clon03:clasrun> fbconfig -dev /dev/fb5 -res SUNW_STD_1280x1024x76 -depth 24
clon03:clasrun> fbconfig -dev /dev/fb6 -res 1280x1024x75 -depth 24

also for m64* card:
m64config -res \?
m64config -help

FROM WEB:

SUMMARY: How to change resolution for solaris 8?
Subject: SUMMARY: How to change resolution for solaris 8?
List-id: The summary-only version of the Sun Managers mailing list. <summaries.sunmanagers.org>

> I just installed solaris 8 1/01 on a SunBlade 100
> and
> the screen resolution is not properly set. I was
> told
> to use "kdmconfig" to change resolution, but there
> is
> no kdmconfig on my system (I did a "find . -print |
> grep kdmconfig" under /).
>
> Another problem is that colors are not always
> displayed properly. Fore example, Netscape seems ok
> but when launching admintool, I get a lot of errors
> such as 'Warning: Cannot allocate colormap entry for
> "gray"' and 'Warning: Cannot parse default
> background
> color specification.' This happens on other
> software,
> esp. acroread, as well. The effect of this is that
> the
> apps often appear in B&W, sometimes even white on
> white. Once acroread runs at the foreground,
> background becomes fuzzy.
>
The solution is "m64config -depth 24 -res 1024x768x75". Other suggestions are
"fbconfig -depth 24 -res 1024x768x75", and to edit "/etc/dt/config/Xservers". I
haven't tried those, though.

> The third problem is that the printer always prints
> three pages of heading before any printing job. I
> setup my network printer as follows.
> Printer type: PostScript
> File contents: PostScript
> Fault Notification: Write to Superuser
> Destination: name.com
> Protocol: BSD
> Options: Default Printer is checked
> User Access List: all
>
The solution is to telnet to the printer and disable banner page. The number of
banner pages is reduced to one after this. To get rid of banner page, use "lp
-o nobanner".

Thanks to the following gurus:
Dan Barnes <barnesdan@xxxxxxxxx>
John Gibson <jgibson@xxxxxxxxxxx>
Ian Camm <ian.camm@xxxxxxxxxxxxxxx>
Fricot, Daniel <daniel.fricot@xxxxxxxxx>
Koos van den Hout <koos@xxxxxxxx>
Surendar Dinkar <surender@xxxxxxxx>
Jonathan Hays <jhays@xxxxxxxx>
system administration account <sysadmin@xxxxxxxxxxx>
Riddoch, John E SITI-ITDSEP3 <John.E.Riddoch@xxxxxxxxxxxx>
Lars Hecking <lhecking@xxxxxxx>
Jai Khurana <jaikhurana2000@xxxxxxxxx>
Steve B. <bsdfan@xxxxxxxxxxx>
ltiu <ltiu@xxxxxxxxxxxxx>

__________________________________________________
Do You Yahoo!?
Check out Yahoo! Shopping and Yahoo! Auctions for all of
your unique holiday gifts! Buy at http://shopping.yahoo.com
or bid at http://auctions.yahoo.com

== Xorg-style server (clon04, both Linux and Solaris experience) ==

monitors:

clon04:0.0 - SUN GDM 5410
clon04:0.1 - SUN GDM 5510
clon04:0.2 - SUN GDM 20E20

'''WHAT I DID ON Solaris''':

1. Run command

/usr/openwin/bin/kdmconfig

it will offer X server Selection: 'Xorg server' or 'Xsun server', choose 'Xorg server'. On next screen it will provide short instructions for configuring the Xorg Server:

(1) Create an xorg.conf file with one of these commands:
/usr/X11/bin/Xorg -configure, or
/usr/X11/bin/xorgconfig, or
/usr/X11/bin/xorgcfg
(2) Edit the resultant xorg.conf file with any changes needed
(3) Move the xorg.conf file to /etc/X11/xorg.conf

For more details about Xorg configuration, refer to the Xorg man pages
in /usr/X11/man. This directory is not in the default man page path, so
to view these pages, use one of these options:
(1) Add this directory to the MANPATH environment variable setting.
(2) Use the "-M /usr/X11/man" option of the man command.

Note, Xorg command line options may override xorg.conf settings.

2. Run command

/usr/X11/bin/xorgconfig

and give following answers when asked:

Enter a protocol number: 1 <enter>
Do you want to enable Emulate3Buttons? n <enter>
Mouse device: <enter>
Enter a number to choose the keyboard.
Press enter for the next page 8 <enter>
Enter a number to choose the country.
Press enter for the next page 1 <enter>
Please enter a variant name for 'us' layout. Or just press enter
for default variant <enter>
Do you want to select additional XKB options (group switcher,
group indicator, etc.)? n <enter>
You must indicate the horizontal sync range of your monitor.... (clon04:0.0 has 81.1kHz/76Hz 1280x1024 mode)
Enter your choice (1-11): 10 <enter> <--- that was "31.5 - 82.0; Monitor that can do 1280x1024 @ 76 Hz"
You must indicate the vertical sync range of your monitor....
Enter your choice: 2 <enter> <--- that was "50-90"
Enter an identifier for your monitor definition: <enter>
Now we must configure video card specific settings....
Do you want to look at the card database? y <enter>
Enter a number to choose the corresponding card definition.
Press enter for the next page, q to continue configuration. q <enter>
How much video memory do you have on your video card:
Enter your choice: 10 <enter> <--- that was "10 131072K", we have 128MB
Enter an identifier for your video card definition: FX560 <enter>
Please specify which color depth you want to use by default:
Enter a number to choose the default depth. 5 <enter> <--- that was "5 24 bits (16 million colors)"
Shall I write it to /etc/X11/xorg.conf? y <enter>
File has been written. Take a look at it before starting an X server. Note that
the xorg.conf file must be in one of the directories searched by the server
(e.g. /etc/X11) in order to be used. Within the server press
ctrl, alt and '+' simultaneously to cycle video resolutions. Pressing ctrl,
alt and backspace simultaneously immediately exits the server (use if
the monitor doesn't sync for a particular mode).
For further configuration, refer to the xorg.conf(4) manual page.

Produced file is crapp !!!!!!

Found that solaris itseld produces following files during reboot:

/etc/X11/.video.devs
/etc/X11/.xorg.conf

and contents of those files looks reasonable. Graphic works but must specify CDE first.

Installed drivers from "Sun Ultra 20 M2 Workstation Tools and Drivers V1.4" CD.
File /etc/X11/xorg.conf was produced. Rebooted - nothing works. Renamed xorg.conf, graphics came back.

Installed drivers from "Sun Ultra 20 M2 Workstation Tools and Drivers V1.6A" iso image (downloaded from sun web site) by mounting image using 'lofi' and executing /drivers/../install.sh and dual....sh. Same result. Looks like everything installed fine, but graphics does not work. Procedure installs its /etc/X11/xorg.conf file but it does not look reasonable.

Copied xorg.conf from the web (maybe the same as in Ultra 20 M2 instructions ?). After reboot something trying to come up but complains about errors. Look at /var/dt/Xerrors, fount that it cannot find font. Copied font passes from another xorg.conf.
Reboot - works ! Login as root using java (not CDE) desktop, and found menu:

Applications->Utilities->NVIDIA X Server Settings

Set resolution for both displays in that menu, click 'Save', and it saved to /etc/X11/xorg.conf file. This is how that file looks now:

# nvidia-settings: X configuration file generated by nvidia-settings
# nvidia-settings: version 1.0 (@builder29) Mon Feb 26 23:33:03 PST 2007
Section "ServerLayout"
Identifier "Layout0"
Screen 0 "Screen0" 0 0
Screen 1 "Screen1" RightOf "Screen0"
InputDevice "Keyboard0" "CoreKeyboard"
InputDevice "Mouse0" "CorePointer"
EndSection
Section "Files"
RgbPath "/usr/X11R6/lib/X11/rgb"
FontPath "/usr/X11R6/lib/X11/fonts/misc/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/100dpi/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/75dpi/:unscaled"
FontPath "/usr/X11R6/lib/X11/fonts/misc/"
FontPath "/usr/X11R6/lib/X11/fonts/Type1/"
FontPath "/usr/X11R6/lib/X11/fonts/100dpi/"
FontPath "/usr/X11R6/lib/X11/fonts/75dpi/"
FontPath "/usr/X11R6/lib/X11/fonts/TrueType/"
FontPath "/usr/X11R6/lib/X11/fonts/Type1/sun/"
FontPath "/usr/X11R6/lib/X11/fonts/F3bitmaps/"
EndSection
Section "Module"
Load "dbe"
Load "extmod"
Load "type1"
Load "IA"
Load "bitstream"
Load "xtsol"
Load "glx"
EndSection
Section "ServerFlags"
Option "Xinerama" "0"
EndSection
Section "InputDevice"
# generated from default
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "auto"
Option "Device" "/dev/mouse"
Option "Emulate3Buttons" "no"
Option "ZAxisMapping" "4 5"
EndSection
Section "InputDevice"
# generated from default
Identifier "Keyboard0"
Driver "keyboard"
EndSection
Section "Monitor"
# HorizSync source: edid, VertRefresh source: edid
Identifier "Monitor0"
VendorName "Unknown"
ModelName "SUN GDM-5410"
HorizSync 30.0 - 121.0
VertRefresh 48.0 - 160.0
Option "DPMS"
EndSection
Section "Monitor"
# HorizSync source: xconfig, VertRefresh source: xconfig
Identifier "Monitor1"
VendorName "Unknown"
ModelName "CRT-0"
HorizSync 31.5 - 67.0
VertRefresh 50.0 - 75.0
Option "DPMS"
EndSection
Section "Device"
Identifier "Videocard0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "Quadro FX 560"
BusID "PCI:2:0:0"
EndSection
Section "Device"
Identifier "Videocard1"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "Quadro FX 560"
BusID "PCI:7:0:0"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
Option "metamodes" "1280x1024 +0+0"
SubSection "Display"
Depth 24
Modes "1600x1200" "1280x1024" "1024x768" "800x600" "640x480"
EndSubSection
EndSection
Section "Screen"
Identifier "Screen1"
Device "Videocard1"
Monitor "Monitor1"
DefaultDepth 24
Option "TwinView" "1"
Option "metamodes" "CRT-0: 1024x768 +1024+0, CRT-1: 1280x1024 +0+0"
SubSection "Display"
Depth 24
Modes "1600x1200" "1280x1024" "1024x768" "800x600" "640x480"
EndSubSection
EndSection

Still need to activate third display and align them to each other, will do it later.

'''WHAT I DID ON Linix''':

Install drivers from V1.6A in ''single'' mode. During reboot Linux recognized 2 new videocards as ''unknown devices'', I said ''configure''. It is also recognized new ethernet device (probably because nvidia ethernet driver was installed), I said ''ignore''.

I adjusted /etc/X11/xorg.conf file similar to Solaris, and clon04:0.0 booted fine, but clon04:0.1 was dark. Then I found menu:

Applications->System Tools->NVIDIA X Server Settings

and found that second display was disabled. I enabled it, saved config and restarted - second display works !!!

'''NOTE''': you should run GUI as 'root'. To start it from the prompt, type

/usr/bin/nvidia-settings

At that point machine was subscribed so I ran 'up2date' and rebooted. Nothing worked after that. I installed drivers again (xorg.conf was not changed) and everything works again. Should I reinstall drivers after every 'up2date' ? Will see.

Finally:

# nvidia-settings: X configuration file generated by nvidia-settings
# nvidia-settings: version 1.0 (buildmeister@builder3) Mon Feb 26 23:39:38 PST 2007
Section "ServerLayout"
Identifier "Layout0"
Screen 0 "Screen0" 640 1024
Screen 1 "Screen1" Above "Screen0"
Screen 2 "Screen2" LeftOf "Screen1"
InputDevice "Keyboard0" "CoreKeyboard"
InputDevice "Mouse0" "CorePointer"
EndSection
Section "Files"
RgbPath "/usr/X11R6/lib/X11/rgb"
FontPath "unix/:7100"
EndSection
Section "Module"
Load "dbe"
Load "extmod"
Load "type1"
Load "freetype"
Load "glx"
EndSection
Section "ServerFlags"
Option "Xinerama" "0"
EndSection
Section "InputDevice"
# generated from data in "/etc/sysconfig/mouse"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
Option "Emulate3Buttons" "no"
Option "ZAxisMapping" "4 5"
EndSection
Section "InputDevice"
# generated from data in "/etc/sysconfig/keyboard"
Identifier "Keyboard0"
Driver "kbd"
Option "XkbLayout" "us"
Option "XkbModel" "pc105"
EndSection
Section "Monitor"
# HorizSync source: edid, VertRefresh source: edid
Identifier "Monitor0"
VendorName "Unknown"
ModelName "SUN GDM-5410"
HorizSync 30.0 - 121.0
VertRefresh 48.0 - 160.0
Option "DPMS"
EndSection
Section "Monitor"
# HorizSync source: edid, VertRefresh source: edid
Identifier "Monitor1"
VendorName "Unknown"
ModelName "SUN GDM-5510"
HorizSync 30.0 - 130.0
VertRefresh 48.0 - 170.0
Option "DPMS"
EndSection
Section "Monitor"
# HorizSync source: xconfig, VertRefresh source: xconfig
Identifier "Monitor2"
VendorName "Unknown"
ModelName "CRT-0"
HorizSync 28.0 - 33.0
VertRefresh 43.0 - 72.0
Option "DPMS"
EndSection
Section "Device"
Identifier "Videocard0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "Quadro FX 560"
BusID "PCI:2:0:0"
EndSection
Section "Device"
Identifier "Videocard1"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "Quadro FX 560"
BusID "PCI:7:0:0"
Screen 0
EndSection
Section "Device"
Identifier "Videocard2"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "Quadro FX 560"
BusID "PCI:7:0:0"
Screen 1
EndSection
Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
Option "metamodes" "1280x1024 +0+0"
SubSection "Display"
Depth 24
Modes "1600x1200" "1280x1024" "1024x768" "800x600" "640x480"
EndSubSection
EndSection
Section "Screen"
Identifier "Screen1"
Device "Videocard1"
Monitor "Monitor1"
DefaultDepth 24
Option "metamodes" "CRT-1: 1280x1024 +0+0"
SubSection "Display"
Depth 24
Modes "1600x1200" "1280x1024" "1024x768" "800x600" "640x480"
EndSubSection
EndSection
Section "Screen"
Identifier "Screen2"
Device "Videocard2"
Monitor "Monitor2"
DefaultDepth 24
Option "metamodes" "CRT-0: 1280x1024 +0+0"
SubSection "Display"
Depth 24
Modes "1280x1024" "1024x768" "800x600" "640x480"
EndSubSection
EndSection

GENIAL'NO (V SMISLE PROSTO) about observed problem when window opens on :0.0 but does not open on hostname:0.0:

On Fri, 18 Mar 2005 14:14:57 GMT, Paul Coene staggered into the Black
Sun and said:
> I'm relatively new to linux, but I'm an old hand (20+) years with Unix
> in general and I've used alot of X windows. Recently, I've started
>
> %xterm
> it brings up an xterm as one might expect.
>
> If I type:
> %xterm -display "localhost:0"
> xterm Xt error: Can't open display: localhost:0

Yep. X is trying to use TCP/IP to connect to localhost. This may not
work. If you do "ps auxw | grep X", you'll probably see a line like so:

root 9492 0.1 4.6 19652 17976 ? S Mar17 1:00
/usr/X11R6/bin/X -nolisten tcp -auth /var/run/xauth/A:0-ZMuEvo vt7

...note the "-nolisten tcp". That makes it so that X won't listen for
TCP of any type. This makes it impossible to use the "normal" xhost
+foo method, or to use X-over-TCP at all.

> The same occurs with any display other than ":0.0"...
> I have xhost + set, so that should not be the problem.

AAAAH! Don't do that. Use "xhost +machine" or better yet, use ssh -X.

> I have three systems. Two of the 3 behave this way.

Find the ?dm config file where X is started. Mine's in
/usr/kde/3.3/share/config/kdm/Xservers since I'm using kdm on KDE 3.3.
gdm is probably /etc/X11/gdm/gdm.conf . Find the line where X is
started, delete the "-nolisten tcp" part of it. For gdm, make sure
"DisallowTCP=true" is commented out. Restart X and things should work
better.

On a local network, it's OK to use xhost and X-over-TCP. Over the wide
Net, use ssh -X (or ssh -Y) so your X traffic is encrypted. If you take
out the "-nolisten tcp" on the machines on your LAN, you might want to
block TCP 6000 at your gateway, just in case.

On standard RHEL4 I added ''DisallowTCP=false'' line into ''/etc/X11/gdm/gdm.conf'' file, and X restarted without ''-nolisten''. Program ''gdmsetup'' can be used to edit that file. Default ''gdm.conf'' does not contains that line and X server starts with ''-nolisten'', which means it is coded into X server binary file supplied by Red Hat.

Similar problem was observed when Solaris 10 Update 3 was installed from DVD on clon03. Probably similar fix must be applied although config file may be different (or the same - who knows).

== advise ==

To configure Solaris for dual monitors, follow these steps:

1. Install the two video cards you intend to use, attach monitors to the cards, and boot the system.

Note - X-windows may not start, initially, due to the change in configuration.

The system outputs to PCI-E slot 2 (lower slot) until X-windows comes up.

2. When the system boots, insert the Tools and Drivers CD, and change directory to the following location:

/cdrom/<T&D_disk_name>/drivers/solx86

3. Run the install.sh and the dual_monitor.sh scripts.

4. Reboot the system.

The dual monitors should come up.

5. To change from Clone to Xinerama modes, edit the file /etc/X11/xorg.conf and turn the appropriate settings in the file on or off.

Note - The top card is at PCI address (2,0,0) and the bottom is at (7,0,0).

An example xorg.conf file is shown below:

# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig: version 1.0 (buildmeister@builder26) Mon Oct 16 22:13:48 PDT 2006
# XFree86 4 configuration created by pyxf86config
#Section "ServerLayout"
# Identifier "Default Layout"
# Screen 0 "Screen0" 0 0
# InputDevice "Mouse0" "CorePointer"
# InputDevice "Keyboard0" "CoreKeyboard"
#EndSection
Section "ServerLayout"
Identifier "Default Layout"
Screen 0 "Screen 0"
Screen 1 "Screen 1" RightOf "Screen 0"
# Screen 2 "Screen 2" LeftOf "Screen 1"
InputDevice "Mouse0" "CorePointer"
InputDevice "Keyboard0" "CoreKeyboard"
EndSection
Section "Files"
# RgbPath is the location of the RGB database. Note, this is the name of the
# file minus the extension (like ".txt" or ".db"). There is normally
# no need to change the default.
# Multiple FontPath entries are allowed (they are concatenated together)
# By default, Red Hat 6.0 and later now use a font server independent of
# the X server to render fonts.
RgbPath "/usr/X11R6/lib/X11/rgb"
FontPath "unix/:7100"
EndSection
Section "Module"
Load "dbe"
Load "extmod"
Load "fbdevhw"
Load "glx"
Load "record"
Load "freetype"
Load "type1"
EndSection
Section "InputDevice"
# Specify which keyboard LEDs can be user-controlled (eg, with xset(1))
# Option "Xleds" "1 2 3"
# To disable the XKEYBOARD extension, uncomment XkbDisable.
# Option "XkbDisable"
# To customise the XKB settings to suit your keyboard, modify the
# lines below (which are the defaults). For example, for a non-U.S.
# keyboard, you will probably want to use:
# Option "XkbModel" "pc102"
# If you have a US Microsoft Natural keyboard, you can use:
# Option "XkbModel" "microsoft"
#
# Then to change the language, change the Layout setting.
# For example, a german layout can be obtained with:
# Option "XkbLayout" "de"
# or:
# Option "XkbLayout" "de"
# Option "XkbVariant" "nodeadkeys"
#
# If you'd like to switch the positions of your capslock and
# control keys, use:
# Option "XkbOptions" "ctrl:swapcaps"
# Or if you just want both to be control, use:
# Option "XkbOptions" "ctrl:nocaps"
#
Identifier "Keyboard0"
Driver "kbd"
Option "XkbModel" "pc105"
Option "XkbLayout" "us"
EndSection
Section "InputDevice"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
Option "ZAxisMapping" "4 5"
Option "Emulate3Buttons" "yes"
EndSection
Section "Monitor"
Identifier "Monitor0"
VendorName "Monitor Vendor"
ModelName "Unprobed Monitor"
HorizSync 31.5 - 67.0
VertRefresh 50.0 - 75.0
Option "dpms"
EndSection
Section "Monitor"
Identifier "Monitor1"
VendorName "Sun Microsystems"
ModelName "X7198A"
HorizSync 31.5 - 67.0
VertRefresh 50.0 - 75.0
Option "dpms"
EndSection
Section "Device"
Identifier "Videocard0"
Driver "nvidia"
VendorName "Videocard vendor"
BoardName "VESA driver (generic)"
BusID "PCI:2:0:0"
EndSection
Section "Device"
Identifier "Videocard1"
Driver "nvidia"
VendorName "Videocard vendor"
BoardName "VESA driver (generic)"
BusID "PCI:7:0:0"
EndSection
Section "Screen"
Identifier "Screen 0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
Option "TwinView" "True"
Option "TwinViewOrientation" "LeftOf"
Option "UseEdidFreqs" "True"
Option "MetaModes" "800x600,800x600"
SubSection "Display"
Viewport 0 0
Depth 24
Modes "800x600" "640x480"
EndSubSection
EndSection
Section "Screen"
Identifier "Screen 1"
Device "Videocard1"
Monitor "Monitor1"
DefaultDepth 24
Option "TwinView" "True"
Option "TwinViewOrientation" "LeftOf"
Option "UseEdidFreqs" "True"
Option "MetaModes" "1024x768, 1024x768"
SubSection "Display"
Viewport 0 0
Depth 24
Modes "800x600" "640x480"
EndSubSection
EndSection

== advise ==

(8.0) X Windows

(8.1) How do I find a Solaris video driver for my graphics card?

First look in the HCL for your release of Solaris to see if your graphics card is supported. Having a supported graphics card on the HCL is very important--more important than a supported motherboard. If the adapter's not listed, look for a driver from these sources:

* Solaris 10 comes with both X.org Xserver software and the original XSun software. Xorg is open source software available for multiple UNIX/BSD/Linux-based platforms. Xorg supports more video adapters and supports them better than XSun.

* http://developers.Sun.COM/ may have newer drivers for your release.

* Some video drivers may be bundled (hidden) with Solaris Recommended Update Patches.

* Consider 3rd-party drivers. XiGraphics, http://www.xig.com/, is especially recommended if you have a laptop. The nVidia video drivers are at http://www.nvidia.com/object/unix.html

* Finally, if all else fails, see the question below on installing Xorg on Solaris. Personally, it's a lot easier to use a card supported by Xsun than to install Xorg.

During installation, if you have a unsupported video card, be sure to bypass kdmconfig with the F4 key. Use type 4, "Solaris Interactive Text (Console session)," for Solaris 10 and the "Software 1 of 2" CD for Solaris 8 or 9.

[Thanks to Paul Karagianis for NVida info]

(8.2)How do I install the Xorg video drivers?

Beginning with Solaris 10, all Xorg drivers are included with Solaris. The old "Video Drivers Porting Kit" is no longer needed. Xorg drivers are also available with the current Solaris 9 Update release.

(8.3) How do I configure Xorg on Solaris?

First use /usr/openwin/bin/kdmconfig to select Xorg instead of Xsun. Run /usr/X11R6/bin/xorgconfig to configure Xorg, which asks questions and creates file /etc/X11/xorg.conf View errors, if any, in /var/log/Xorg.0.log. Try the generic "VESA driver" (which uses the BIOS) if the driver you picked didn't work. You may also (carefully) edit /etc/X11/xorg.conf directly (save the original file first). To start the X server manually, use /usr/X11R6/bin/xinit

(8.4) How do I configure 64K colors for XSun/CDE?

Solaris' kdmconfig offers only a choice of 256 or 16M colors for your graphics board with XSun. To configure XSun to use 16-bit color, configure using kdmconfig for the desired resolution in 256 color mode. Note the board line from file /etc/openwin/server/etc/OWconfig. Look in the referenced file in directory /usr/openwin/share/etc/devdata/SUNWaccel/boards to see if there is support for 16-bit color mode for your card.

If your board is listed, edit file /etc/openwin/server/etc/OWconfig and change defdepth="8" to defdepth="16" in it. Also, values for MaxPClk for some cards are way too low in 16-bit and 24-bit modes in file /usr/openwin/share/etc/devdata/SUNWaccel/boards. But if you want to change this, you are doing it on your own risk. Some applications hang on 16-bit colors.

The CDE logo that is displayed after one logs in is a bit-mapped grayscale image(!?), but everything else should work just fine.

[Thanks to Aleksandar Milivojevic, by way of John Groenveld]

(8.5) How do I Add KDE, FVWM, or other non-CDE/GNOME Window Mangers to the dtlogin screen?

KDE The easiest way to add substitute KDE for CDE is to add an "exec /path/to/kde" statement in your $HOME/.dtprofile file.

To add a KDE selection to the dtlogin screen (in addition to the usual command-line, OpenWindows, or CDE choices), follow one or both of these links:

* http://www.tiem.utk.edu/~peek/solaris/ written by Michael Peek
* http://twirl.mcc.ac.uk/~zzassgl/wm.html written by Geoff Lane

FVWM The instructions are for FVWM, but apply to any window manager, such as KDE, AfterStep, WindowMaker, or Enlightenment. Be aware that these new WMs usually take a bigger memory footprint.

(8.6) Where can I get GNOME or KDE packages for Solaris/x86?

GNOME is available with Solaris 9 and above. KDE is available with the Solaris Companion CD. They are also available on the web for download.

For information about GNOME on Solaris or for a free download, see http://www.sun.com/gnome/ For information on GNOME, see http://www.gnome.org/

For general KDE information, see http://www.kde.org/ When installing KDE, make sure to install the QT library first, which is required by KDE.

Which is better? "Religious" wars could be fought over this question. KDE has a more familiar M$ windows-like interface, and I used to use it. I use GNOME now since it's most easily available on both Solaris and Linux. KDE is more mature than GNOME and more tightly integrated, but some say GNOME is catching up. GNOME and Mozilla both use the C-based GTK library. The C++-based QT library, required by KDE, used to have stricter licensing restrictions, but now it's licensed under GPL (not LGPL). KDE is endorsed and supported by IBM, Caldera, and SuSE. GNOME is endorsed and supported by Sun (and RedHat and HP). I recommend installing and trying out both GNOME and KDE for a test drive. You can have both installed at the same time and select which one to use with dtlogin. You can also run KDE apps under GNOME and vice versa.

Sun picked GNOME over KDE because of Qt licensing issues, CORBRA in GNOME, and because GNOME uses C and KDE uses C++ (the latter causes name mangling problems which prevents using different C++ compilers).

(8.7) After upgrading to Solaris 9 or installing GNOME, GNOME does not appear in the dtlogin menu. How can I fix this?

Check what your LANG variable is set to (AFTER logging in to CDE). If it's not "C", you need to copy the GNOME dtlogin resource files to your current locale. Type this command, as root (1 line):
cp /usr/dt/config/C/Xresources.d/Xresource.Sun-gnome-2.0* /usr/dt/config/$LANG/Xresouces.d
(where $LANG is you current locale) For example:
cp /usr/dt/config/C/Xresources.d/Xresource.Sun-gnome-2.0* /usr/dt/config/en_US.ISO8859-15/Xresouces.d

If you have trouble AFTER logging into a GNOME session, check these:

* If you define $LD_LIBRARY_PATH, make sure /usr/lib and /opt/sfw/lib are at the beginning. E.g., /usr/lib:/opt/sfw/lib:/usr/local/lib
* Rename your ~/.xinitrc file to something else (e.g., ~/.xinitrc.old).
* Erase your GNOME configuration files: /usr/bin/gnome-cleanup
* For more tips, see the GNOME 2.0 Desktop for the Solaris O. E. Troubleshooting Guide at http://docs.sun.com/

(8.8) Are TrueType fonts supported in Solaris?

Yes, Xsun supports them. Use the Font Administrator GUI, /usr/dt/bin/sdtfontadm, to add them to the server's list of fonts. See also http://www.bolthole.com/solaris/verdana.font.txt

[Thanks to Tran Tran]

(8.9) After installing and booting Solaris I get this message: "The X-server can not be started on display :0"

Your video or mouse settings are probably incorrect or a video card or mouse was not detected at all. Check the log files to be sure (/var/log/Xorg.0.log if using Xorg or /var/log/Xsun.0.log if using Xsun with a XF86/Xorg driver). First login in text mode and check /var/dt/Xerrors for error messages. If nothing obvious is there, run kdmconfig and try a simpler video setting (plain VGA, low resolution). Once that works, try higher-resolution settings and other drivers. Also try using the Xorg video drivers instead of XSun (see a previous question on configuring with Xorg drivers).

(8.10) How do I disable CDE auto-start upon booting multi-user?

Boot single-user and run dtconfig(1). That is, at the booting Solaris prompt, type "b -s" After entering your root password, mount /usr and run "/usr/dt/bin/dtconfig -d"

[Thanks to John Groenveld]

(8.11) How do I su(1) to another user and run an X application?

You need to set your $DISPLAY and allow others to use your console. If you don't, you get a message like "Can't open display". By default, only the user who logged in on console can connect to the display.

To allow another user to connect there are two ways - one easy and one secure (unfortunately, you can't have both):

Easy: As user logged in on console run "xhost +LOCAL:" (This gives permission to connect to the display to anyone logged in on the machine—this includes being able to open windows, read your keystrokes, send keystrokes, etc. Obviously not a good idea if you have other users on the system you don't trust completely.) Starting with Solaris 10, you can also use this safer, more-restrictive format: xhost xhost +si:localuser:username For example: xhost +si:localuser:root

As any other user logged in on the box, run "setenv DISPLAY :0" or "DISPLAY=:0; export DISPLAY", depending on your shell type. This all assumes you're connecting from the same host, for a remote host, change to "xhost +other-hostname" and "setenv DISPLAY PutYourHostnameHere:0"

Safe: As user logged in on console run "xauth list" Look for the line for your hostname followed by ":0" and copy it. As the user you want to grant access to run "xauth" and at the xauth> prompt type "add " and paste the line you copied. Now "setenv DISPLAY hostname:0", making sure you match the way it was listed in the line you copied and pasted.

Easier variation of safe method for special cases: If the other user you want to access your screen is root, and your home directory is either local or on an NFS filesystem exported with root permissions, just have root do this:
setenv XAUTHORITY /home/myuserid/.Xauthority ; setenv DISPLAY :0
(replace "/home/myuserid" with your actual home directory).

[Thanks to Alan Coopersmith]

(8.12) Does Solaris x86 support multiple heads?

Yes, with Xorg. Xsun supports multiple heads only for Sun Ray workstations. A commercial X server from Xi graphics, http://www.xig.com/ also supports multiple heads.

[Thanks to John Groenveld, Alan Coopersmith]

(8.13) How do I get my 2-button mouse to emulate 3 buttons?

In Solaris 8, "kdmconfig" will autodetect a two-button PS/2 mouse, so you may never look in the "change pointing device" section. You need to go in there, and change it from "PS/2 Mouse (2 Button)" to "PS/2 Mouse (2 Button+100ms 3 Button Emulation)"

[Thanks to Philip Brown]

(8.14)How do I get some older Solaris GUI's to run with Xorg?

Some applications are configured to use Sun's proprietary F3 fonts. The work-around is to run Sun font server, xfs(1), and to prefix Xorg's FontPath with it.

(8.15) Can I use an old Sun monitor with a 13W3 connector with a VGA adaptor card?

Yes. You need an adaptor. Get a 13W3 female to DB-15HD (VGA) male adaptor to connect a Sun monitor to a PC VGA adaptor port. Make sure to get the 13W3 female adaptor (the more-common male adaptor is for connecting PC-style VGA monitors to Sun video ports).

== advise ==

What does :0.0 mean?

The :0.0 part of the DISPLAY variable denote the display and the screen of an X server.

The display is the first number and equals to a running instance of an X server. If it is using TCP/IP for communication it listens on port 6000 + display-number for connections. With local (UNIX domain sockets) it uses the socket /tmp/.X11-unix/Xdisplay-number

The screen denotes different output devices of the X server. You could start XWin.exe with two -screen options and would end up with two X11 windows. Each of them is a different screen. Other X servers open different screens for different monitors connected to the computer.

Cygwin/X supports different formats of the DISPLAY variable

:0.0 or unix:0.0
This names a local X server and the communication uses the UNIX domain sockets.

hostname:0.0
This names a remote X server and the communication uses the TCP/IP network.

== advise ==

Cygwin:

Xlib: connection to "local_host_name_or_ip_address:0.0" refused by server Xlib: Maximum number of clients reached

Cygwin/X queries getdtablesize () for the maximum number of client connections allowed; by default Cygwin returns 32 from getdtablesize (). Cygwin/X Server Test Series release Test44, released on 2001-08-15, changes the maximum number of clients from 32 to 1024 by passing the square of getdtablesize () to setdtablesize ().

Solaris:

It's in the FAQ, naturally:
3.57) How can I have more than 128 X windows clients?
When you get the following errors, you've run out of X sockets.
Xlib: connection to :0.0 refused by server
Xlib: maximum number of clients reached
By default, the X server has a limit of just 128. In order to
increase this limit, you need to run at least Solaris 8 or an
earlier release with the Xserver patch applies that fixes bug:
4185418 the X server should support more connections
Then change the Xservers configuration file and add the "-clients 1024"
option to the X commandline.
Since I'm running Solaris 8, I can change max clients.

Raidold

2008-11-17T15:56:24Z

129.57.167.14:

A5200 sun raid system from about 2000, currently mounted as /raidold on clon10. It has [[ZFS]] file system on it and in use mostly by test setups. It had two units: with 9GB disks and with 18GB disks, total volume id 544GB.

There is no maintenance agreement on that unit, repaires must be done by purchaising parts. Disk and rear fan were replaced already. During rear fan replacement second rear fan shows error. Rear fan which was removed was installed and shows no errors - maybe bad contact ?

Raidold

2008-11-17T15:52:44Z

129.57.167.14:

A5200 sun raid system from about 2000, currently mounted as /raidold on clon10. It has [[ZFS]] file system on it and in use mostly by test setups.

Raidold

2008-11-17T15:52:31Z

129.57.167.14:

A5200 sun raid system from about 2000, currently mounted as /raidold on clon10. It has [[zfs]] file system on it and in use mostly by test setups.

CLON Index

2008-11-17T15:50:56Z

129.57.167.14:

[[Apache]]

[[Bootp]]

[[CAEN SY527 Mainframe]]

[[Capfast]]

[[Cisco Switches]]

[[Clasweb]]

[[Clon00]]

[[Clon01]]

[[Clon02]]

[[Clon03]]

[[Clon04]]

[[Clon05]]

[[Clon06]]

[[Clon10]]

[[Clondaq1]]

[[Clondaq2]]

[[Clondb1]]

[[Clonfs]] Main RAID System

[[Clonfs2]]

[[Clonmon]]

[[Clonmon1]]

[[Clonpc2]]

[[Clonweb]]

[[Clonwiki]]

[[clonhp]]

[[clonhp2]]

[[CycladesTS1000]]

[[DNS server]]

[[Electronics Documentation]]

[[EtherLite32]]

[[EPICS Base Installation]]

[[EPICS: Software IOC]]

[[EPICS: vxWorks IOC]]

[[EPICS: Software IOC for CAEN HV]]

[[EPICS: generation of DBs and GUIs for HV]]

[[EPICS: CVS usage examples]]

[[EPICS: Gamma Profiler]]

[[Fiber Channel Switches]]

[[Foundry BigIron 15000 Switch]]

[[Foundry Switches]]

[[JAVA installation]]

[[JLAB Discriminators]]

[[Labeling Machines]]

[[Linux Installation Procedure]]

[[Linux Customization on CLON Cluster]]

[[Lnf cluster]]

[[Mac OS X Installation Procedure]]

[[MRTG]]

[[mSQL]]

[[MySQL Installation]]

[[MySQL Usage]]

[[Nagios]]

[[Nagios Plugins]]

[[NFS]]

[[NIS server]]

[[Nrpe]]

[[Ntp]]

[[OpenSSL]]

[[PERL]]

[[Procmail]]

[[raidold]]

[[Routing]]

[[Rrdtool]]

[[Serial Connections from Computer Ports]]

[[Slow Controls Time History]]

[[SmartSockets]]

[[SNMP]]

[[Solaris Installation Procedure]]

[[Solaris Customization on CLON Cluster]]

[[Sudo]]

[[SUN Microsystems]]

[[Tcl/Tk]]

[[Tftp]]

[[Troubleshooting printing]]

[[UNIX Automount on CLON Cluster]]

[[UPS (Uninterruptible Power Supplies)]]

[[UPS CHB1]]

[[UPS CHB2]]

[[UPS Troubleshooting]]

[[VI editor]]

[[VNC]]

[[WIENER]]

[[Wiki]]

[[Wiring]]

[[X server]]

[[ZFS]]

Troubleshooting printing

2006-11-02T02:58:58Z

129.57.167.14:

There is no printing server currently on CLON Cluster, every machine sends job directly to printers.

WAS: To RESET printing queue use "lpc" command on CLON01 (as root):

# lpc
lpc> clean clonhp
removed clonhp-789
lpc> status
clonhp:
queueing is enabled
printing is enabled
no entries

Troubleshooting

2006-11-02T02:56:44Z

129.57.167.14:

[[Printing]]

Software

2006-11-02T02:56:15Z

129.57.167.14:

[[Basic Operating System Installation]]

[[Additional System Components Installation]]

[[Applications Installation]]

[[Troubleshooting]]

Software

2006-11-02T02:54:00Z

129.57.167.14:

[[Basic Operating System Installation]]

[[Additional System Components Installation]]

[[Applications Installation]]

[[Troubleshouting]]

Clonhp2

2006-11-02T02:51:13Z

129.57.167.14:

''clonhp2'' is color Xerox Phaser 860 printer located in the Counting Room, it can be accessed at http://clonhp2.jlab.org.

Printer Supplies can be purchased from JLAB WEBSTOCK http://misportal.jlab.org/webstock/:

Part Num: CUYIS8635 (InkStiks for Xerox Phaser 860, 5 Cyan + 2 Black) ($163.40 on June 2006)
Part Num: CUYIS8645 (InkStiks for Xerox Phaser 860, 5 Magenta + 2 Black) ($163.40 on June 2006)
Part Num: CUYIS8655 (InkStiks for Xerox Phaser 860, 5 Yellow + 2 Black) ($163.40 on June 2006)

Hardware problems must be emailed to helpdesk@jlab.org.

Clonhp

2006-11-02T02:47:58Z

129.57.167.14:

''clonhp'' is black/white HP Laser Jet 4250 printer located in Counting Room. It can be accessed at http://clonhp.jlab.org.

Supplies can be purchased from JLAB WEBSTOCK http://misportal.jlab.org/webstock/:

Part Num: Q5942A (Smart Toner Cartridge for HP LaserJet 4250n) ($149.24 on June 2006)

Hardware problems must be emailed to helpdesk@jlab.org

Clonhp2

2006-11-02T02:43:13Z

129.57.167.14:

Printers in the Counting Room

CLON01 is configured as Print Server for two printers in the Counting Room: clonhp http://clonhp.jlab.org - B/W - HP Laser Jet 4250 and clonhp2 http://clonhp2.jlab.org - color - Xerox Phaser 860.

To RESET printing queue use "lpc" command on CLON01 (as root):

<pre>
# lpc
lpc> clean clonhp
removed clonhp-789
lpc> status
clonhp:
queueing is enabled
printing is enabled
no entries
</pre>

++ Printer Supplies

can be purchased from JLAB WEBSTOCK http://misportal.jlab.org/webstock/:
<pre>
clonhp2:

Part Num: CUYIS8635 (InkStiks for Xerox Phaser 860, 5 Cyan + 2 Black) ($163.40 on June 2006)
Part Num: CUYIS8645 (InkStiks for Xerox Phaser 860, 5 Magenta + 2 Black) ($163.40 on June 2006)
Part Num: CUYIS8655 (InkStiks for Xerox Phaser 860, 5 Yellow + 2 Black) ($163.40 on June 2006)

clonhp:

Part Num: Q5942A (Smart Toner Cartridge for HP LaserJet 4250n) ($149.24 on June 2006)
</pre>

++ Hardware problems ? Email to helpdesk@jlab.org

Clonhp

2006-11-02T02:42:24Z

129.57.167.14: