ERROR Book: Difference between revisions

From CLONWiki
Jump to navigation Jump to search
No edit summary
Boiarino (talk | contribs)
 
(52 intermediate revisions by 3 users not shown)
Line 1: Line 1:
* Sergey B. 10-feb-2011: test setup coda_net on pmc dies:
'''CLAS12'''
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
 
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
== Sergey B. Dec 2017 run: RESOLVED in library by always using even number ==
  proc_thread: waiting=    127 processing=      3 microsec per event (nev=6559)
 
  net_threadwaiting=   131    sending=      2 microsec per event (nev=12708)
  TI VME (Local) address = 0x00a80000 (0xabf81000)
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
  tiReload:
  proc_thread: waiting=    139 processing=      3 microsec per event (nev=6474)
  FPGA Re-Load !
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
CODAtcpServer: wait: coda request >download PROD< in progress
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
  ID: 0x7101    Firmware (type - revision): 0x3 - 0x07A (prodID = 1)
proc_thread: waiting=   129 processing=      3 microsec per event (nev=6452)
  ... Configure as TI Slave...
  net_thread: waiting=    133    sending=      2 microsec per event (nev=12598)
tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1)
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
tiSetTriggerSource: ERROR: Invalid Trigger Source (129).  Must be between 0 and 11.
  proc_thread: waiting=    127 processing=     3 microsec per event (nev=6553)
  FiberMeas: syncDelay_write=0x82828200
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=16)
  FiberMeas: syncDelay=0x8200820b
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=12)
  net_threadwaiting=    131    sending=      2 microsec per event (nev=12714)
  FiberMeas: ERROR: TI Fiber Measurement failed!
  proc_thread: waiting=    139 processing=      3 microsec per event (nev=6489)
        First Measurement != Second Measurement (60 != 61)
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=8)
   
  proc_thread: waiting=    128 processing=      3 microsec per event (nev=6478)
tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
  cannot find TI, ret=-1 - exit
net_thread:  waiting=   134   sending=      2 microsec per event (nev=12377)
coda_roc is exiting, clear dma memory
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=0)
 
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6554)
== Sergey B. Dec 2017 run: ==
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=3)
 
  proc_thread: waiting=    139 processing=      3 microsec per event (nev=6470)
  tiInit: Scanning for TI...
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
tiFind: Found TI at 0x00a80000
proc_thread: waiting=    128 processing=      3 microsec per event (nev=6516)
TI VME (Local) address = 0x00a80000 (0xabf5f000)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
  ID: 0x7101   Firmware (type - revision): 0x3 - 0x07A (prodID = 1)
  0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
  ... Configure as TI Slave...
ERROR2: LINK_sized_write() returns errno=32
tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1)
Broken pipe
  tiSetTriggerSource: ERROR: Invalid Trigger Source (129).  Must be between 0 and 11.
ERROR: net_thread failed (in LINK_sized_write).
 
->
  FiberMeas: ERROR: TI Fiber Measurement failed!
-> i
        First Measurement != Second Measurement (90 != 58)
   NAME        ENTRY      TID    PRI  STATUS      PC      SP    ERRNO  DELAY
 
---------- ------------ -------- --- ---------- -------- -------- ------- -----
  tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
tExcTask   excTask      1effe6c0   0 PEND        14e508 1effe5a0  3d0001    0
cannot find TI, ret=-1 - exit
  tLogTask  logTask      1effbb20   0 PEND        14e508 1effba10      0    0
  coda_roc is exiting, clear dma memory
tShell    shell        1eb8cb70   1 READY        145290 1eb8c750      d     0
 
tRlogind   rlogind      1ebb3850   2 PEND        13f934 1ebb33f0      0    0
== Sergey B. Dec 2017 run: possible interference with DiagGuiServer ??? ==
tWdbTask   wdbTask      1ebaa530  3 PEND        13f934 1ebaa400      0    0
 
  tNetTask   netTask      1ed0f280  50 PEND        13f934 1ed0f190      0    0
tiInit: ERROR: Invalid Board ID: 0x0 (rval = 0x00000000)
  tPortmapd  portmapd     1ebb1690  54 PEND        13f934 1ebb1430  3d0002    0
  tiInit: Scanning for TI...
tFtpdTask b6064        1ebaebc0 55 PEND        13f934 1ebaea10      0     0
  jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x100000
coda_proc  proc_thread  1aacca00 115 DELAY        144bb4 1aacc900      0   10
  jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000
coda_pmc   coda_pmc    1e3f9720 200 DELAY        144bb4 1e3f9480      0    5
  jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x600000
  value = 0 = 0x0
tiInit: ERROR: Unable to find TI
-> printErrno 32
cannot find TI, ret=-1 - exit
  S_errno_EPIPE
coda_roc is exiting, clear dma memory
  value = 14 = 0xe
 
  -> tt
== Sergey B. Dec 2017 run: ==
  Sorry, traces of my own stack begin at tt ().
 
  value = -1 = 0xffffffff
  clasrun@clondaq6:stage_in> tcpClient trig1 tdGStatus
-> memShow
 
  invalid block at 0x1ec60000 deleted
  TD Module Status Summary
value = -1 = 0xffffffff
              Ports          SyncSrc   BusySrc   Busy Status  Triggers TrgAck
-> memShow
  Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678Received  Wait
status    bytes    blocks  avg block  max block
  --------------------------------------------------------------------------------
------ ---------- --------- ---------- ----------
  3    12345678    12345678  V----  ---12345678  --1-3----- 0x0e1d0000  N
current
  4    12345678    12345678  V----  ---12345678  --1-----7- 0x21bd0000  Y
  free  421348656        56    7524083  411333840
  5    12345678    12345678  V----   ---12345678  ----3---7- 0xe71d0000  Y
  alloc  79807344      1986      40184          -
  6    123456-8    123456--   V----   ---123456--   --1------- 0x4fdd0000  N
cumulative
  7    12345678    123456--   V----   ---123456--   --1-3----- 0x363d0000  Y
  alloc  87490320      7392      11835          -
  8    12345678    123456--  V----  ---123456--   --1----6-- 0xbf7d0000   Y
  value = 0 = 0x0
   9     123456--    123456--   V----   ---123456--   ---------- 0x753d0000   N
  -> usrNetStackSysPoolStatus
  10     123456--    123456--  V----  ---123456--  ---------- 0xe8fd0000  N
value = 0 = 0x0
  13     1234567-   123456--  V----   ---123456--  ---------- 0xa1dd0000  N
  -> usrNetStackSysPoolStatus("CODAtcpServer",0)
 
value = 0 = 0x0
  TD Port STATUS Summary
-> usrNetStackDataPoolStatus("CODAtcpServer",0)
                                                      Block Status      Block Buffer
  value = 454823200 = 0x1b1c0d20
  Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status Ready / NeedAck    Level  Level
  -> usrNetStackDataPoolStatus("bla",0)
  --------------------------------------------------------------------------------
  value = 454823168 = 0x1b1c0d00
  3-1 adcecal1 UP      1    ENABLED  BUSY-BlkAck      1 /  0          1      0
  -> usrNetStackDataPoolStatus("bla",0)
  3-2  tdcecal1 UP      2    ENABLED                    0 /  0           1      0
  value = 454823040 = 0x1b1c0c80
  3-3 adcpcal1 UP      3    ENABLED  BUSY-BlkAck      1 /  0          1      0
  -> memShow
  3-4 tdcpcal1 UP      4    ENABLED                    0 /  0          1      0
status    bytes     blocks   avg block  max block
  3-5 adcftof1 UP      5     ENABLED                    0 /   0          1      0
------ ---------- --------- ---------- ----------
  3-6  tdcftof1  UP      6    ENABLED                    0 /  0          1      0
current
  3-7    trig2  UP    38    ENABLED                    0 /  0          1      0
  free 421348528        55    7660882  411333840
  3-8  adccnd1  UP    73    ENABLED                    0 /  0          1      0
  alloc  79807472      1990      40104          -
  4-1  adcecal2  UP      7    ENABLED  BUSY-BlkAck      1 /  0          1      0
  cumulative
  4-2  tdcecal2  UP      8    ENABLED                    0 /  0          1      0
  alloc  87496528      7401      11822          -
  4-3  adcpcal2  UP      9    ENABLED                    0 /  0          1      0
value = 0 = 0x0
  4-4  tdcpcal2  UP    10    ENABLED                    0 /  0          1      0
->
  4-5  adcftof2  UP    11    ENABLED                    0 /  0          1      0
  4-6  tdcftof2  UP    12    ENABLED                    0 /  0          1      0
  4-7  adcctof1  UP    59    ENABLED  BUSY-BlkAck      1 /  1          74      74
  4-8  tdcctof1  UP    60    ENABLED                    0 /  0          1      0
  5-1  adcecal3  UP    13    ENABLED                    0 /  0          1      0
  5-2  tdcecal3  UP    14    ENABLED                    0 /  0          1      0
  5-3  adcpcal3  UP    15    ENABLED  BUSY-BlkAck      1 /  0          1      0
  5-4  tdcpcal3  UP    16    ENABLED                    0 /  0          1      0
  5-5  adcftof3  UP    17    ENABLED                    0 /  0          1      0
  5-6  tdcftof3  UP    18    ENABLED                    0 /  0          1      0
  5-7      svt1  UP    61    ENABLED  BUSY-BlkAck      1 /  1          0      0
  5-8      svt2  UP    62    ENABLED                    0 /  0          1      0
  6-1  adcecal4  UP    19    ENABLED  BUSY-BlkAck      1 /  0          1      0
  6-2  tdcecal4  UP    20    ENABLED                    0 /  0          1      0
  6-3  adcpcal4  UP    21    ENABLED                    0 /  0          1      0
  6-4  tdcpcal4  UP    22    ENABLED                    0 /  0          1      0
  6-5  adcftof4  UP    23    ENABLED                    0 /  0          1      0
  6-6  tdcftof4  UP    24    ENABLED                    0 /  0          1      0
  7-1  adcecal5  UP    25    ENABLED  BUSY-BlkAck      1 /  1          0      0
  7-2  tdcecal5  UP    26    ENABLED                    0 /  0          1      0
  7-3  adcpcal5  UP    27    ENABLED  BUSY-BlkAck      1 /  1          0      0
  7-4  tdcpcal5  UP    28    ENABLED                    0 /  0          1      0
  7-5  adcftof5  UP    30    ENABLED                    0 /  0          1      0
  7-6  tdcftof5  UP    29    ENABLED                    0 /  0          1      0
  8-1  adcecal6  UP    31    ENABLED  BUSY-BlkAck      1 /  1          0      0
  8-2  tdcecal6  UP    32    ENABLED                    0 /  0          1      0
  8-3  adcpcal6  UP    33    ENABLED                    0 /  0          1      0
  8-4  tdcpcal6  UP    34    ENABLED                    0 /  0          1      0
  8-5  adcftof6  UP    35    ENABLED                    0 /  0          1      0
  8-6  tdcftof6  UP    36    ENABLED  BUSY-BlkAck      1 /  1          0      0
  9-1      dc11  UP    41    ENABLED                    0 /  0          1      0
  9-2      dc21  UP    44    ENABLED                    0 /  0          1      0
  9-3      dc31  UP    47    ENABLED                    0 /  0          1      0
  9-4      dc41  UP    50    ENABLED                    0 /  0          1      0
  9-5      dc51  UP    53    ENABLED                    0 /  0          1      0
  9-6      dc61  UP    56    ENABLED                    0 /  0          1      0
10-1      dc12  UP    42    ENABLED                    0 /  0          1      0
10-2      dc22  UP    45    ENABLED                    0 /  0          1      0
10-3      dc32  UP    48    ENABLED                    0 /  0          1      0
10-4      dc42  UP    51    ENABLED                    0 /  0          1      0
10-5      dc52  UP    54    ENABLED                    0 /  0          1      0
10-6      dc62  UP    57    ENABLED                    0 /  0          1      0
13-1      dc13  UP    43    ENABLED                    0 /  0          1      0
13-2      dc23  UP    46    ENABLED                    0 /  0          1      0
13-3      dc33  UP    49    ENABLED                    0 /  0          1      0
13-4      dc43  UP    52    ENABLED                    0 /  0          1      0
13-5      dc53  UP    55    ENABLED                    0 /  0          1      0
13-6      dc63  UP    58    ENABLED                    0 /  0          1      0
 
Total Slaves Added = 60
--------------------------------------------------------------------------------
 
== Sergey B. Dec 2017 run: handle it better, send error and continue ==
 
net_thread:  waiting=    881    sending=      1 microsec per event (nev=28)
proc_thread: waiting=    35 processing=    818 microsec per event (nev=28)
net_thread:  waiting=    856    sending=      1 microsec per event (nev=28)
usrVme2MemDmaDone: ERROR: DMA terminated on master byte count,    however DCNT (1997952) != 0
faReadBlock: WARN: DMA transfer terminated by word count 0x7a120
Segmentation fault
clasrun@adccnd1:clasrun>
 
== Sergey B. Dec 2017 run: ==
 
dabufp[0] = 0x00002676 (9846)
dabufp[1] = 0x00010e01 (69121)
TI UNKNOWN data: [  0] 0x08010003
 
  Previous stuff
          [-20][  0] 0x84000bf2
          [-19][  0] 0x900003f2
          [-18][  0] 0x98000074
          [-17][  0] 0x00a432d5
          [-16][  0] 0xa026001f
          [-15][  0] 0x8c000006
          [-14][  0] 0xf8000000
          [-13][  0] 0xf8000000
          [-12][  0] 0x84400bf2
          [-11][  0] 0x900003f2
          [-10][  0] 0x98000074
          [ -9][  0] 0x00a432d5
          [ -8][  0] 0xa026001f
          [ -7][  0] 0x8c400006
          [ -6][  0] 0xf8000000
          [ -5][  0] 0xf8000000
          [ -4][  0] 0x0000006d
          [ -3][  0] 0x00010ef4
          [ -2][  0] 0x00000009
          [ -1][  0] 0xe10a013e
          [  0][  0] 0x08010003
          [  1][  0] 0x000003f4
          [  2][  0] 0xe94a05fb
          [  3][  0] 0x00000000
          [  4][  0] 0x8d400007
          [  5][  0] 0xfd4003f4
          [  6][  0] 0xfd4f1110
          [  7][  0] 0xfd4f1110
          [  8][  0] 0x00000061
          [  9][  0] 0xe104013e
  End Of Previous stuff
coda_roc is exiting, clear dma memory
 
runcontrol complains about adcft3 first, then about svt2
restarted svt2, download-prestart-go works
 
== Sergey B. Dec 2017 run: check vtp readout in general ==
 
proc_thread: waiting=    164 processing=    937 microsec per event (nev=30)
net_thread:  waiting=    54    sending=      1 microsec per event (nev=606)
VTP UNKNOWN data2: [353] 0x000000d0
VTP UNKNOWN data2: [354] 0x00010ee5
VTP UNKNOWN data2: [355] 0x00000055
VTP UNKNOWN data2: [356] 0xe10a016c
VTP UNKNOWN data2: [357] 0x8540e514
VTP UNKNOWN data2: [359] 0x09010003
VTP UNKNOWN data2: [360] 0x000a11d1
VTP UNKNOWN data2: [361] 0xbbffc6ed
VTP UNKNOWN data2: [362] 0x00000002
VTP UNKNOWN data2: [363] 0x09010003
VTP UNKNOWN data2: [364] 0x000a11d2
VTP UNKNOWN data2: [365] 0xbbffe48f
VTP UNKNOWN data2: [366] 0x00000002
VTP UNKNOWN data2: [367] 0x0a010003
VTP UNKNOWN data2: [368] 0x000a11d3
VTP UNKNOWN data2: [369] 0xbc004d73
VTP UNKNOWN data2: [370] 0x00000002
VTP UNKNOWN data2: [371] 0x0b010003
VTP UNKNOWN data2: [372] 0x000a11d4
VTP UNKNOWN data2: [373] 0xbc008dec
VTP UNKNOWN data2: [374] 0x00000002
VTP UNKNOWN data2: [310] 0x00d0006a
VTP UNKNOWN data2: [339] 0x00d0009b
VTP UNKNOWN data2: [353] 0x000000d0
VTP UNKNOWN data2: [354] 0x00010ee5
VTP UNKNOWN data2: [355] 0x00000055
VTP UNKNOWN data2: [356] 0xe10a016c
VTP UNKNOWN data2: [357] 0x8540e514
VTP UNKNOWN data2: [359] 0x09010003
VTP UNKNOWN data2: [360] 0x000a11d1
VTP UNKNOWN data2: [361] 0xbbffc6ed
VTP UNKNOWN data2: [362] 0x00000002
VTP UNKNOWN data2: [363] 0x09010003
VTP UNKNOWN data2: [364] 0x000a11d2
VTP UNKNOWN data2: [365] 0xbbffe48f
VTP UNKNOWN data2: [366] 0x00000002
VTP UNKNOWN data2: [367] 0x0a010003
VTP UNKNOWN data2: [368] 0x000a11d3
VTP UNKNOWN data2: [369] 0xbc004d73
VTP UNKNOWN data2: [370] 0x00000002
VTP UNKNOWN data2: [371] 0x0b010003
VTP UNKNOWN data2: [372] 0x000a11d4
VTP UNKNOWN data2: [373] 0xbc008dec
VTP UNKNOWN data2: [374] 0x00000002
VTP UNKNOWN data2: [375] 0x20010003
VTP UNKNOWN data2: [376] 0x000a11d5
VTP UNKNOWN data2: [377] 0xbc00a9c1
 
Segmentation fault (core dumped)
clasrun@adcpcal2vtp:clasrun>
 
== Sergey B. Dec 2017 run: ==
 
proc_thread: waiting=  1245 processing=    925 microsec per event (nev=23)
net_thread:  waiting=    53    sending=      1 microsec per event (nev=932)
ERROR in TI event header word: event type = 0
ERROR in TI event header word (0x45)
ERROR: TS has more then 7 data words - exit
clasrun@adcecal3vtp:clasrun>
 
== Sergey B. Dec 2017 run: after reboot trig1, have to do clock setup on ts and td, ask Bryan ??? ==
 
trig1 rebooted, after prestart (adcecal5 missing, wrong block_level):
 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    DISABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    DISABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    DISABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    DISABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    DISABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    DISABLED                    0 /  0          20      0
  3-7    trig2  UP    38    DISABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    DISABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    DISABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    DISABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    DISABLED                    0 /  0          0      0
  4-4  tdcpcal2  UP    10    DISABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    DISABLED                    0 /  0          0      0
  4-6  tdcftof2  UP    12    DISABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    DISABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    DISABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    DISABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    DISABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    DISABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    DISABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    DISABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    DISABLED                    0 /  0          20      0
  5-7      svt1  UP    61    DISABLED                    0 /  0          0      0
  5-8      svt2  UP    62    DISABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    DISABLED                    0 /  0          0      0
  6-2  tdcecal4  UP    20    DISABLED                    0 /  0          0      0
  6-3  adcpcal4  UP    21    DISABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    DISABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    DISABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    DISABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    DISABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    DISABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    DISABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    DISABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    DISABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    DISABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    DISABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    DISABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    DISABLED                    0 /  0          0      0
  8-3  adcpcal6  UP    33    DISABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    DISABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    DISABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    DISABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    DISABLED                    0 /  0          20      0
  9-1      dc11  UP    41    DISABLED                    0 /  0          20      0
  9-2      dc21  UP    44    DISABLED                    0 /  0          20      0
  9-3      dc31  UP    47    DISABLED                    0 /  0          20      0
  9-4      dc41  UP    50    DISABLED                    0 /  0          20      0
  9-5      dc51  UP    53    DISABLED                    0 /  0          20      0
  9-6      dc61  UP    56    DISABLED                    0 /  0          20      0
10-1      dc12  UP    42    DISABLED                    0 /  0          20      0
10-2      dc22  UP    45    DISABLED                    0 /  0          0      0
10-3      dc32  UP    48    DISABLED                    0 /  0          20      0
10-4      dc42  UP    51    DISABLED                    0 /  0          20      0
10-5      dc52  UP    54    DISABLED                    0 /  0          20      0
10-6      dc62  UP    57    DISABLED                    0 /  0          20      0
13-1      dc13  UP    43    DISABLED                    0 /  0          0      0
13-2      dc23  UP    46    DISABLED                    0 /  0          20      0
13-3      dc33  UP    49    DISABLED                    0 /  0          20      0
13-4      dc43  UP    52    DISABLED                    0 /  0          20      0
13-5      dc53  UP    55    DISABLED                    0 /  0          20      0
13-6      dc63  UP    58    DISABLED                    0 /  0          20      0
 
Total Slaves Added = 62
--------------------------------------------------------------------------------
 
 
 
AFTER SECOND RESET_DOWNLOAD_PRESTART:
 
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    --------  V----  ---12345678  ----------  0x41a56800  N
  4    12345678    --------  V----  ---12345678  ----------  0xbc456800  N
  5    12345678    --------  V----  ---12345678  ----------  0x8cc56800  N
  6    123456-8    --------  V----  ---123456--  ----------  0x6b856800  N
  7    12345678    --------  V----  ---12345678  ----------  0x63856800  N
  8    12345678    --------  V----  ---1234567-  ----------  0xd9456800  N
  9    123456--    --------  V----  ---123456--  ----------  0x79a56800  N
10    123456--    --------  V----  ---123456--  ----------  0xe0e56800  N
13    1234567-    --------  V----  ---123456--  ----------  0x3c856800  N
 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    DISABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    DISABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    DISABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    DISABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    DISABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    DISABLED                    0 /  0          20      0
  3-7    trig2  UP    38    DISABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    DISABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    DISABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    DISABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    DISABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    DISABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    DISABLED                    0 /  0          20      0
  4-6  tdcftof2  UP    12    DISABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    DISABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    DISABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    DISABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    DISABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    DISABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    DISABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    DISABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    DISABLED                    0 /  0          20      0
  5-7      svt1  UP    61    DISABLED                    0 /  0          20      0
  5-8      svt2  UP    62    DISABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    DISABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    DISABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    DISABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    DISABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    DISABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    DISABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    DISABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    DISABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    DISABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    DISABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    DISABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    DISABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    DISABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    DISABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    DISABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    DISABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    DISABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    DISABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    DISABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    DISABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    DISABLED                    0 /  0          20      0
  9-1      dc11  UP    41    DISABLED                    0 /  0          20      0
  9-2      dc21  UP    44    DISABLED                    0 /  0          20      0
  9-3      dc31  UP    47    DISABLED                    0 /  0          20      0
  9-4      dc41  UP    50    DISABLED                    0 /  0          20      0
  9-5      dc51  UP    53    DISABLED                    0 /  0          20      0
  9-6      dc61  UP    56    DISABLED                    0 /  0          20      0
10-1      dc12  UP    42    DISABLED                    0 /  0          20      0
10-2      dc22  UP    45    DISABLED                    0 /  0          20      0
10-3      dc32  UP    48    DISABLED                    0 /  0          20      0
10-4      dc42  UP    51    DISABLED                    0 /  0          20      0
10-5      dc52  UP    54    DISABLED                    0 /  0          20      0
10-6      dc62  UP    57    DISABLED                    0 /  0          20      0
13-1      dc13  UP    43    DISABLED                    0 /  0          20      0
13-2      dc23  UP    46    DISABLED                    0 /  0          20      0
13-3      dc33  UP    49    DISABLED                    0 /  0          20      0
13-4      dc43  UP    52    DISABLED                    0 /  0          20      0
13-5      dc53  UP    55    DISABLED                    0 /  0          20      0
13-6      dc63  UP    58    DISABLED                    0 /  0          20      0
 
Total Slaves Added = 63
--------------------------------------------------------------------------------
 
 
AFTER GO:
 
 
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    12345678  V----  ---12345678  ----------  0xfeb56400  N
  4    12345678    12345678  V----  ---12345678  ----------  0x69f56400  N
  5    12345678    12345678  V----  ---12345678  ----------  0x2ad56400  N
  6    123456-8    123456--  V----  ---123456--  ----------  0xfa156400  N
  7    12345678    12345678  V----  ---12345678  ----------  0xe2956400  N
  8    12345678    1234567-  V----  ---1234567-  ----3-----  0x48f56400  N
  9    123456--    123456--  V----  ---123456--  ----------  0xd9b56400  N
10    123456--    123456--  V----  ---123456--  ----------  0x31756400  N
13    1234567-    123456--  V----  ---123456--  ----------  0x7db56400  N
 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    ENABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    ENABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    ENABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    ENABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    ENABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    ENABLED                    0 /  0          20      0
  3-7    trig2  UP    38    ENABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    ENABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    ENABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    ENABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    ENABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    ENABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    ENABLED                    0 /  0          20      0
  4-6  tdcftof2  UP    12    ENABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    ENABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    ENABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    ENABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    ENABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    ENABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    ENABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    ENABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    ENABLED                    0 /  0          20      0
  5-7      svt1  UP    61    ENABLED                    0 /  0          20      0
  5-8      svt2  UP    62    ENABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    ENABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    ENABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    ENABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    ENABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    ENABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    ENABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    ENABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    ENABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    ENABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    ENABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    ENABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    ENABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    ENABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    ENABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    ENABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    ENABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    ENABLED  BUSY-BlkAck      4 /  0          20      0
  8-4  tdcpcal6  UP    34    ENABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    ENABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    ENABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    ENABLED                    0 /  0          20      0
  9-1      dc11  UP    41    ENABLED                    0 /  0          20      0
  9-2      dc21  UP    44    ENABLED                    0 /  0          20      0
  9-3      dc31  UP    47    ENABLED                    0 /  0          20      0
  9-4      dc41  UP    50    ENABLED                    0 /  0          20      0
  9-5      dc51  UP    53    ENABLED                    0 /  0          20      0
  9-6      dc61  UP    56    ENABLED                    0 /  0          20      0
10-1      dc12  UP    42    ENABLED                    0 /  0          20      0
10-2      dc22  UP    45    ENABLED                    0 /  0          20      0
10-3      dc32  UP    48    ENABLED                    0 /  0          20      0
10-4      dc42  UP    51    ENABLED                    0 /  0          20      0
10-5      dc52  UP    54    ENABLED                    0 /  0          20      0
10-6      dc62  UP    57    ENABLED                    0 /  0          20      0
13-1      dc13  UP    43    ENABLED                    0 /  0          20      0
13-2      dc23  UP    46    ENABLED                    0 /  0          20      0
13-3      dc33  UP    49    ENABLED                    0 /  0          20      0
13-4      dc43  UP    52    ENABLED                    0 /  0          20      0
13-5      dc53  UP    55    ENABLED                    0 /  0          20      0
13-6      dc63  UP    58    ENABLED                    0 /  0          20      0
 
Total Slaves Added = 63
--------------------------------------------------------------------------------
 
RESET-DOWNLOAD-PRESTART
 
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    --------  V----  ---12345678  ----------  0x03857000  N
  4    12345678    --------  V----  ---12345678  ----------  0x6ec57000  N
  5    12345678    --------  V----  ---12345678  ----------  0x2fa57000  N
  6    123456-8    --------  V----  ---123456--  ----------  0xfee57000  N
  7    12345678    --------  V----  ---12345678  ----------  0xe7657000  N
  8    12345678    --------  V----  ---1234567-  ----------  0x4dc57000  N
  9    123456--    --------  V----  ---123456--  ----------  0xde857000  N
10    123456--    --------  V----  ---123456--  ----------  0x36457000  N
13    1234567-    --------  V----  ---123456--  ----------  0x82857000  N
 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    DISABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    DISABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    DISABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    DISABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    DISABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    DISABLED                    0 /  0          20      0
  3-7    trig2  UP    38    DISABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    DISABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    DISABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    DISABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    DISABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    DISABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    DISABLED                    0 /  0          20      0
  4-6  tdcftof2  UP    12    DISABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    DISABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    DISABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    DISABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    DISABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    DISABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    DISABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    DISABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    DISABLED                    0 /  0          20      0
  5-7      svt1  UP    61    DISABLED                    0 /  0          20      0
  5-8      svt2  UP    62    DISABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    DISABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    DISABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    DISABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    DISABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    DISABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    DISABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    DISABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    DISABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    DISABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    DISABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    DISABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    DISABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    DISABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    DISABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    DISABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    DISABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    DISABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    DISABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    DISABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    DISABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    DISABLED                    0 /  0          20      0
  9-1      dc11  UP    41    DISABLED                    0 /  0          20      0
  9-2      dc21  UP    44    DISABLED                    0 /  0          20      0
  9-3      dc31  UP    47    DISABLED                    0 /  0          20      0
  9-4      dc41  UP    50    DISABLED                    0 /  0          20      0
  9-5      dc51  UP    53    DISABLED                    0 /  0          20      0
  9-6      dc61  UP    56    DISABLED                    0 /  0          20      0
10-1      dc12  UP    42    DISABLED                    0 /  0          20      0
10-2      dc22  UP    45    DISABLED                    0 /  0          20      0
10-3      dc32  UP    48    DISABLED                    0 /  0          20      0
10-4      dc42  UP    51    DISABLED                    0 /  0          20      0
10-5      dc52  UP    54    DISABLED                    0 /  0          20      0
10-6      dc62  UP    57    DISABLED                    0 /  0          20      0
13-1      dc13  UP    43    DISABLED                    0 /  0          20      0
13-2      dc23  UP    46    DISABLED                    0 /  0          20      0
13-3      dc33  UP    49    DISABLED                    0 /  0          20      0
13-4      dc43  UP    52    DISABLED                    0 /  0          20      0
13-5      dc53  UP    55    DISABLED                    0 /  0          20      0
13-6      dc63  UP    58    DISABLED                    0 /  0          20      0
 
Total Slaves Added = 63
--------------------------------------------------------------------------------
 
 
GO
 
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    12345678  V----  ---12345678  ----------  0xd1505c00  N
  4    12345678    12345678  V----  ---12345678  ------5---  0x3cb05c00  N
  5    12345678    12345678  V----  ---12345678  ----------  0xfdb05c00  N
  6    123456-8    123456--  V----  ---123456--  ----------  0x81806c00  N
  7    12345678    12345678  V----  ---12345678  ----------  0xb5505c00  N
  8    12345678    1234567-  V----  ---1234567-  ----------  0x1bb05c00  N
  9    123456--    123456--  V----  ---123456--  ----------  0xac905c00  N
10    123456--    123456--  V----  ---123456--  ----------  0x04505c00  N
13    1234567-    123456--  V----  ---123456--  ----------  0x50705c00  N
 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    ENABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    ENABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    ENABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    ENABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    ENABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    ENABLED                    0 /  0          20      0
  3-7    trig2  UP    38    ENABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    ENABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    ENABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    ENABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    ENABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    ENABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    ENABLED  BUSY-BlkAck      4 /  0          20      0
  4-6  tdcftof2  UP    12    ENABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    ENABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    ENABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    ENABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    ENABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    ENABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    ENABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    ENABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    ENABLED                    0 /  0          20      0
  5-7      svt1  UP    61    ENABLED                    0 /  0          20      0
  5-8      svt2  UP    62    ENABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    ENABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    ENABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    ENABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    ENABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    ENABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    ENABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    ENABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    ENABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    ENABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    ENABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    ENABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    ENABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    ENABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    ENABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    ENABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    ENABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    ENABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    ENABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    ENABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    ENABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    ENABLED                    0 /  0          20      0
  9-1      dc11  UP    41    ENABLED                    0 /  0          20      0
  9-2      dc21  UP    44    ENABLED                    0 /  0          20      0
  9-3      dc31  UP    47    ENABLED                    0 /  0          20      0
  9-4      dc41  UP    50    ENABLED                    0 /  0          20      0
  9-5      dc51  UP    53    ENABLED                    0 /  0          20      0
  9-6      dc61  UP    56    ENABLED                    0 /  0          20      0
10-1      dc12  UP    42    ENABLED                    0 /  0          20      0
10-2      dc22  UP    45    ENABLED                    0 /  0          20      0
10-3      dc32  UP    48    ENABLED                    0 /  0          20      0
10-4      dc42  UP    51    ENABLED                    0 /  0          20      0
10-5      dc52  UP    54    ENABLED                    0 /  0          20      0
10-6      dc62  UP    57    ENABLED                    0 /  0          20      0
13-1      dc13  UP    43    ENABLED                    0 /  0          20      0
13-2      dc23  UP    46    ENABLED                    0 /  0          20      0
13-3      dc33  UP    49    ENABLED                    0 /  0          20      0
13-4      dc43  UP    52    ENABLED                    0 /  0          20      0
13-5      dc53  UP    55    ENABLED                    0 /  0          20      0
13-6      dc63  UP    58    ENABLED                    0 /  0          20      0
 
Total Slaves Added = 63
--------------------------------------------------------------------------------
 
  roc_reboot adcftof2
 
  DOWNLOAD-PRESTART-GO:


EB looks fine, message was:
-- Got all fragments of go
BOS_encode_spec reached
BOS_encode_spec done: len=116
FFFFFFFFFFFFFFFFFF
PUT: wait while icb=8 == cbp->read=8 -------
clastest::EB5>


TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    12345678  V----  ---12345678  ----------  0x747d1400  N
  4    12345678    12345678  V----  ---12345678  ----------  0xef5d1400  N
  5    12345678    12345678  V----  ---12345678  --------7-  0xbfbd1400  N
  6    123456-8    123456--  V----  ---123456--  ----------  0xf2205c00  N
  7    12345678    12345678  V----  ---12345678  ----------  0x965d1400  N
  8    12345678    1234567-  V----  ---1234567-  ----------  0x0c3d1400  N
  9    123456--    123456--  V----  ---123456--  ----------  0xac9d1400  N
10    123456--    123456--  V----  ---123456--  ----------  0x13dd1400  N
13    1234567-    123456--  V----  ---123456--  ----------  0x6f7d1400  N 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    ENABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    ENABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    ENABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    ENABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    ENABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    ENABLED                    0 /  0          20      0
  3-7    trig2  UP    38    ENABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    ENABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    ENABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    ENABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    ENABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    ENABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    ENABLED                    0 /  0          20      0
  4-6  tdcftof2  UP    12    ENABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    ENABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    ENABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    ENABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    ENABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    ENABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    ENABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    ENABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    ENABLED                    0 /  0          20      0
  5-7      svt1  UP    61    ENABLED  BUSY-BlkAck    16 /  0          20      0
  5-8      svt2  UP    62    ENABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    ENABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    ENABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    ENABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    ENABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    ENABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    ENABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    ENABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    ENABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    ENABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    ENABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    ENABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    ENABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    ENABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    ENABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    ENABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    ENABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    ENABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    ENABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    ENABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    ENABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    ENABLED                    0 /  0          20      0
  9-1      dc11  UP    41    ENABLED                    0 /  0          20      0
  9-2      dc21  UP    44    ENABLED                    0 /  0          20      0
  9-3      dc31  UP    47    ENABLED                    0 /  0          20      0
  9-4      dc41  UP    50    ENABLED                    0 /  0          20      0
  9-5      dc51  UP    53    ENABLED                    0 /  0          20      0
  9-6      dc61  UP    56    ENABLED                    0 /  0          20      0
10-1      dc12  UP    42    ENABLED                    0 /  0          20      0
10-2      dc22  UP    45    ENABLED                    0 /  0          20      0
10-3      dc32  UP    48    ENABLED                    0 /  0          20      0
10-4      dc42  UP    51    ENABLED                    0 /  0          20      0
10-5      dc52  UP    54    ENABLED                    0 /  0          20      0
10-6      dc62  UP    57    ENABLED                    0 /  0          20      0
13-1      dc13  UP    43    ENABLED                    0 /  0          20      0
13-2      dc23  UP    46    ENABLED                    0 /  0          20      0
13-3      dc33  UP    49    ENABLED                    0 /  0          20      0
13-4      dc43  UP    52    ENABLED                    0 /  0          20      0
13-5      dc53  UP    55    ENABLED                    0 /  0          20      0
13-6      dc63  UP    58    ENABLED                    0 /  0          20      0
Total Slaves Added = 63
--------------------------------------------------------------------------------
reboot svt1
GO
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    12345678  V----  ---12345678  ----------  0x76d04800  N
  4    12345678    12345678  V----  ---12345678  ------5---  0xe2504800  N
  5    12345678    12345678  V----  ---12345678  ----------  0xa4e06c00  N
  6    123456-8    123456--  V----  ---123456--  ----------  0x34d07800  N
  7    12345678    12345678  V----  ---12345678  ----------  0x5c806c00  N
  8    12345678    1234567-  V----  ---1234567-  ----------  0xc1304800  N
  9    123456--    123456--  V----  ---123456--  ----------  0x52104800  N
10    123456--    123456--  V----  ---123456--  ----------  0xa9d04800  N
13    1234567-    123456--  V----  ---123456--  ----------  0xf6104800  N 
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    ENABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    ENABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    ENABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    ENABLED                    0 /  0          20      0
  3-5  adcftof1  UP      5    ENABLED                    0 /  0          20      0
  3-6  tdcftof1  UP      6    ENABLED                    0 /  0          20      0
  3-7    trig2  UP    38    ENABLED                    0 /  0          20      0
  3-8  adccnd1  UP    73    ENABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    ENABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    ENABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    ENABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    ENABLED                    0 /  0          20      0
  4-5  adcftof2  UP    11    ENABLED  BUSY-BlkAck      4 /  0          20      0
  4-6  tdcftof2  UP    12    ENABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    ENABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    ENABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    ENABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    ENABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    ENABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    ENABLED                    0 /  0          20      0
  5-5  adcftof3  UP    17    ENABLED                    0 /  0          20      0
  5-6  tdcftof3  UP    18    ENABLED                    0 /  0          20      0
  5-7      svt1  UP    61    ENABLED                    0 /  0          20      0
  5-8      svt2  UP    62    ENABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    ENABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    ENABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    ENABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    ENABLED                    0 /  0          20      0
  6-5  adcftof4  UP    23    ENABLED                    0 /  0          20      0
  6-6  tdcftof4  UP    24    ENABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    ENABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    ENABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    ENABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    ENABLED                    0 /  0          20      0
  7-5  adcftof5  UP    30    ENABLED                    0 /  0          20      0
  7-6  tdcftof5  UP    29    ENABLED                    0 /  0          20      0
  7-7    adcft1  UP    70    ENABLED                    0 /  0          20      0
  7-8    adcft2  UP    71    ENABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    ENABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    ENABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    ENABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    ENABLED                    0 /  0          20      0
  8-5  adcftof6  UP    35    ENABLED                    0 /  0          20      0
  8-6  tdcftof6  UP    36    ENABLED                    0 /  0          20      0
  8-7    adcft3  UP    72    ENABLED                    0 /  0          20      0
  9-1      dc11  UP    41    ENABLED                    0 /  0          20      0
  9-2      dc21  UP    44    ENABLED                    0 /  0          20      0
  9-3      dc31  UP    47    ENABLED                    0 /  0          20      0
  9-4      dc41  UP    50    ENABLED                    0 /  0          20      0
  9-5      dc51  UP    53    ENABLED                    0 /  0          20      0
  9-6      dc61  UP    56    ENABLED                    0 /  0          20      0
10-1      dc12  UP    42    ENABLED                    0 /  0          20      0
10-2      dc22  UP    45    ENABLED                    0 /  0          20      0
10-3      dc32  UP    48    ENABLED                    0 /  0          20      0
10-4      dc42  UP    51    ENABLED                    0 /  0          20      0
10-5      dc52  UP    54    ENABLED                    0 /  0          20      0
10-6      dc62  UP    57    ENABLED                    0 /  0          20      0
13-1      dc13  UP    43    ENABLED                    0 /  0          20      0
13-2      dc23  UP    46    ENABLED                    0 /  0          20      0
13-3      dc33  UP    49    ENABLED                    0 /  0          20      0
13-4      dc43  UP    52    ENABLED                    0 /  0          20      0
13-5      dc53  UP    55    ENABLED                    0 /  0          20      0
13-6      dc63  UP    58    ENABLED                    0 /  0          20      0
Total Slaves Added = 63
--------------------------------------------------------------------------------
roc_reboot all rocs
have to reboot adcecal2 twice because of vtp did not come up
load FC_HTCC_VTP
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    --------  V----  ---1234----  ----------  0xcde56800  N
  4    12345678    --------  V----  ---1234--78  ----------  0xcee56800  N
  5    12345678    --------  V----  ---1234----  ----------  0x25a56800  N
  6    123456-8    --------  V----  ---1234----  ----------  0x89c56800  N
  7    12345678    --------  V----  ---1234----  ----------  0x08c56800  N
  8    12345678    --------  V----  ---12-4----  ----------  0x02c56800  N
  9    123456--    --------  V----  -----------  ----------  0x26a56800  N
10    123456--    --------  V----  -----------  ----------  0x16656800  N
13    1234567-    --------  V----  -----------  ----------  0xfa856800  N
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    DISABLED                    0 /  0          0      0
  3-2  tdcecal1  UP      2    DISABLED                    0 /  0          0      0
  3-3  adcpcal1  UP      3    DISABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    DISABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    DISABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    DISABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    DISABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    DISABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    DISABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    DISABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    DISABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    DISABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    DISABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    DISABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    DISABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    DISABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    DISABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    DISABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    DISABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    DISABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    DISABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    DISABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    DISABLED                    0 /  0          0      0
  8-2  tdcecal6  UP    32    DISABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    DISABLED                    0 /  0          20      0 
Total Slaves Added = 25
--------------------------------------------------------------------------------
do not try GO, DOWNLOAD again
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status  Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
  3    12345678    --------  V----  ---1234--7-  ----------  0xc0f56400  N
  4    12345678    --------  V----  ---1234--78  ----------  0x2c956400  N
  5    12345678    --------  V----  ---1234----  ----------  0xedf56400  N
  6    123456-8    --------  V----  ---1234----  ----------  0xbc956400  N
  7    12345678    --------  V----  ---1234----  ----------  0xa6156400  N
  8    12345678    --------  V----  ---1234----  ----------  0x0ab56400  N
  9    123456--    --------  V----  -----------  ----------  0x99356400  N
10    123456--    --------  V----  -----------  ----------  0xf3956400  N
13    1234567-    --------  V----  -----------  ----------  0x42556400  N
TD Port STATUS Summary
                                                      Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
--------------------------------------------------------------------------------
  3-1  adcecal1  UP      1    DISABLED                    0 /  0          20      0
  3-2  tdcecal1  UP      2    DISABLED                    0 /  0          20      0
  3-3  adcpcal1  UP      3    DISABLED                    0 /  0          20      0
  3-4  tdcpcal1  UP      4    DISABLED                    0 /  0          20      0
  3-7    trig2  UP    38    DISABLED                    0 /  0          20      0
  4-1  adcecal2  UP      7    DISABLED                    0 /  0          20      0
  4-2  tdcecal2  UP      8    DISABLED                    0 /  0          20      0
  4-3  adcpcal2  UP      9    DISABLED                    0 /  0          20      0
  4-4  tdcpcal2  UP    10    DISABLED                    0 /  0          20      0
  4-7  adcctof1  UP    59    DISABLED                    0 /  0          20      0
  4-8  tdcctof1  UP    60    DISABLED                    0 /  0          20      0
  5-1  adcecal3  UP    13    DISABLED                    0 /  0          20      0
  5-2  tdcecal3  UP    14    DISABLED                    0 /  0          20      0
  5-3  adcpcal3  UP    15    DISABLED                    0 /  0          20      0
  5-4  tdcpcal3  UP    16    DISABLED                    0 /  0          20      0
  6-1  adcecal4  UP    19    DISABLED                    0 /  0          20      0
  6-2  tdcecal4  UP    20    DISABLED                    0 /  0          20      0
  6-3  adcpcal4  UP    21    DISABLED                    0 /  0          20      0
  6-4  tdcpcal4  UP    22    DISABLED                    0 /  0          20      0
  7-1  adcecal5  UP    25    DISABLED                    0 /  0          20      0
  7-2  tdcecal5  UP    26    DISABLED                    0 /  0          20      0
  7-3  adcpcal5  UP    27    DISABLED                    0 /  0          20      0
  7-4  tdcpcal5  UP    28    DISABLED                    0 /  0          20      0
  8-1  adcecal6  UP    31    DISABLED                    0 /  0          20      0
  8-2  tdcecal6  UP    32    DISABLED                    0 /  0          20      0
  8-3  adcpcal6  UP    33    DISABLED                    0 /  0          20      0
  8-4  tdcpcal6  UP    34    DISABLED                    0 /  0          20      0 
Total Slaves Added = 27
--------------------------------------------------------------------------------
RUNNING !!!!!!!!!!!!!!!!!!!!! PIECE OF SSSSSSSSSSSSSSSSS...........


* Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!???:
clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod  -m eb -v
et_nodename >eb<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station LEVEL3
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_station_create_at, created station ET2ER
unique name >et_start_eb_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_clasprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
ipc_init called at Fri Apr 16 11:45:20 2010
et_start: pthread_create(0x0000000e,...) done
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et INFO: et_sys_heartmonitor, kill bad process (2,22078)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments


svt1 still gives busy shortly after go:
clasrun@clonpc19:clasrun> tcpClient svt1 'tiStatus(1)'
STATUS for TI at VME (Local) base address 0x00a80000 (0xac02c000)
--------------------------------------------------------------------------------
A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5ac000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 1715
    L1A Count: 17150
  Block Limit: 0 
  Block Count: 1715
Registers (offset):
  boardID        (0x0000) = 0x7101153d    fiber          (0x0004) = 0x020311ff
  intsetup      (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
  adr32          (0x0010) = 0x08000000    blocklevel    (0x0014) = 0x0a0a0001
  dataFormat    (0x0018) = 0x08080006    vmeControl    (0x001c) = 0x00000011
  trigger        (0x0020) = 0x01000012    sync          (0x0024) = 0x00000c82
  busy          (0x0028) = 0x00000002    clock          (0x002c) = 0x00000002
  blockBuffer    (0x0034) = 0x00000008    output        (0x004c) = 0x00000000
  fiberSyncDelay (0x0050) = 0x4f004f13    nblocks        (0x0094) = 0x000006b3
  GTPStatusA    (0x00b0) = 0x02801fff    GTPStatusB    (0x00b4) = 0xff07ff07
  livetime      (0x00a8) = 0x01a7f1ad    busytime      (0x00ac) = 0x00000000
  GTPTrgBufLen  (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x00000007
  triggerRule    (0x0038) = 0x8a010182    triggerRuleMin (0x0138) = 0x00000000
Block Level        = 10
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
Fiber Status        1    2    3    4    5    6    7    8
                    ----- ----- ----- ----- ----- ----- ----- -----
  Connected          YES  YES                                     
Clock Source (2) =
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source =
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source =
  Switch Slot B   
Interrupts DISABLED
  Level = 5  Vector = 0xc8
Bus Errors Enabled
Blocks ready for readout: 0
Input counter 0
--------------------------------------------------------------------------------
clasrun@clonpc19:clasrun> tcpClient svt1 'tiStatus(2)'
STATUS for TI at VME (Local) base address 0x00a80000 (0xac02c000)
--------------------------------------------------------------------------------
A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5ac000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 1715
    L1A Count: 17150
  Block Limit: 0 
  Block Count: 1715
Registers (offset):
  boardID        (0x0000) = 0x7101153d    fiber          (0x0004) = 0x020311ff
  intsetup      (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
  adr32          (0x0010) = 0x08000000    blocklevel    (0x0014) = 0x0a0a0001
  dataFormat    (0x0018) = 0x08080006    vmeControl    (0x001c) = 0x00000011
  trigger        (0x0020) = 0x01000012    sync          (0x0024) = 0x00000c82
  busy          (0x0028) = 0x00000002    clock          (0x002c) = 0x00000002
  blockBuffer    (0x0034) = 0x00000008    output        (0x004c) = 0x00000000
  fiberSyncDelay (0x0050) = 0x4f004f13    nblocks        (0x0094) = 0x000006b3
  GTPStatusA    (0x00b0) = 0x02801fff    GTPStatusB    (0x00b4) = 0xff07ff07
  livetime      (0x00a8) = 0x01b0e3ad    busytime      (0x00ac) = 0x00000000
  GTPTrgBufLen  (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x00000007
  triggerRule    (0x0038) = 0x8a010182    triggerRuleMin (0x0138) = 0x00000000
Block Level        = 10
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
Fiber Status        1    2    3    4    5    6    7    8
                    ----- ----- ----- ----- ----- ----- ----- -----
  Connected          YES  YES                                     
Clock Source (2) =
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source =
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source =
  Switch Slot B   
Interrupts DISABLED
  Level = 5  Vector = 0xc8
Bus Errors Enabled
Blocks ready for readout: 0
Input counter 0
--------------------------------------------------------------------------------




clondaq2:clasrun> cd et2_start
clondaq2:et2_start>
clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod
-m er -v
et_nodename >er<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station TAPE
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur
rent/coda/SunOS_i86pc/lib/libet_user.so
et INFO: et_conductor 2, loaded function et2et_function
et INFO: et_station_create_at, created station ET2ET10
unique name >et_start_er_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R
Tserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve
r
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established
.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl
asprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab
.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>


ipc_init called at Fri Apr 16 11:45:20 2010
clasrun@clonpc19:clasrun> tcpClient svt1 'tiStatus(1)' >svt1
clasrun@clonpc19:clasrun> tcpClient svt2 'tiStatus(1)' > svt2
clasrun@clonpc19:clasrun> diff svt1 svt2
2c2
< STATUS for TI at VME (Local) base address 0x00a80000 (0xac02c000)
---
> STATUS for TI at VME (Local) base address 0x00a80000 (0xac020000)
4c4
<  A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5ac000)
---
>  A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5a0000)
13c13
<  boardID        (0x0000) = 0x7101153d          fiber          (0x0004) = 0x020311ff
---
>  boardID        (0x0000) = 0x7101153e          fiber          (0x0004) = 0x101111ff
18,22c18,22
<  busy          (0x0028) = 0x00000002          clock          (0x002c) = 0x00000002
<  blockBuffer    (0x0034) = 0x00000008          output        (0x004c) = 0x00000000
<  fiberSyncDelay (0x0050) = 0x4f004f13          nblocks        (0x0094) = 0x000006b3
<  GTPStatusA    (0x00b0) = 0x02801fff          GTPStatusB    (0x00b4) = 0xff07ff07
<  livetime      (0x00a8) = 0x01dfb7ea          busytime      (0x00ac) = 0x00000000
---
>  busy          (0x0028) = 0x00010002          clock          (0x002c) = 0x00000002
>  blockBuffer    (0x0034) = 0x00000001          output        (0x004c) = 0x00000000
>  fiberSyncDelay (0x0050) = 0x4f004f15          nblocks        (0x0094) = 0x000006b3
>  GTPStatusA    (0x00b0) = 0x10801fff          GTPStatusB    (0x00b4) = 0xff11ff11
>  livetime      (0x00a8) = 0x01e80db3          busytime      (0x00ac) = 0x00000000
24c24
<  triggerRule    (0x0038) = 0x8a010182          triggerRuleMin (0x0138) = 0x00000000
---
>  triggerRule    (0x0038) = 0x0101010a          triggerRuleMin (0x0138) = 0x00000000
32c32
<  Connected          YES  YES                                     
---
>  Connected          YES                    YES                   
 


et_start: pthread_create(0x0000000e,...) done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_restore_events, no events found belonging to attachment 4
et INFO: et_station_detach, done
et WARN: et_command_loop: remote client connection broken
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et INFO: et_sys_heartmonitor, kill bad process (1,8898)
et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1
et INFO: et_flush_events, read 4000 events
et INFO: et_flush_events, wrote 4000 events
et INFO: et_station_detach, make station "TAPE" (cec1d440) idle
et INFO: et_restore_events, no events found belonging to attachment 3
et INFO: et_station_detach, done
et INFO: et_sys_heartmonitor, cleanup process 1
et INFO: set_fix_nprocs, change # of ET processes from 1 to 1
et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments
et INFO: set_fix_natts, station ET2ET10 has 1 attachments
et INFO: set_fix_natts, # total attachments 3 -> 3
et INFO: set_fix_natts, proc 0 has 1 attachments






if it possible that svt1 was the only problem from the beginning,
and TD info about other crates misleading ????????????


clondaq1:dac>
== Sergey B. Dec 2017 run: ==
clondaq1:dac>
clondaq1:dac>
clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 22276) (lang C) (local) (period = 5 sec)


  STATIC INFO - maximum of:
adccnd1:
    events(4000), event size(200000), temps(4000)
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,


   DYNAMIC INFO - currently there are:
proc_thread: waiting=   8924 processing=    819 microsec per event (nev=10)
    processes(4), attachments(4), temps(0)
net_thread:  waiting=    973    sending=      0 microsec per event (nev=102)
    stations(3), hearbeat(1)
usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00700000 < 0x08000000)
 
tiReadBlock: ERROR: vmeDmaDone returned an Error
ERROR in tiReadBlock : No data or error, len = -1
FADCs: no events  stat=0  intcount = 0  gbready = 0x00000000  fadcSlotMask = 0x000027f8
Missing slots:  3  4  5  6  7  8  9 10 13
FADCs: no events  stat=0  intcount = 0  gbready = 0x00000000  fadcSlotMask = 0x000027f8
Missing slots:  3  4  5  6  7  8  9 10 13
FADCs: no events  stat=0  intcount = 0  gbready = 0x00000000  fadcSlotMask = 0x000027f8
Missing slots: 3  4  5  6  7  8  9 10 13
ERROR1: LINK_sized_write() returns errno=32 (cc=-1, sizeof(nbytes)=4(424), netlong=2818637824)
ERROR: net_thread failed (in LINK_sized_write).
FADCs: no events  stat=0  intcount = 0   gbready = 0x00000000  fadcSlotMask = 0x000027f8
Missing slots:  3 4  5  6  7  8  9 10 13
ERROR: bigproc.failure=0, bignet.failure=1
 
Calls bb_cleanup.
Calls bb_cleanup.
Calls bb_cleanup.


  STATIONS:
    "GRAND_CENTRAL" (id = 0)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(2,3,)
        input  list: cnt =  4000, events in  = 0
        output list: cnt =    0, events out = 0


    "LEVEL3" (id = 1)
== Sergey B. Dec 2017 run: send to Bryan ==
      static info
adcpcal5:
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(1),  ids(1,)
        input  list: cnt =     0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0


    "ET2ER" (id = 2)
tiReload:
      static info
FPGA Re-Load !
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
CODAtcpServer: wait: coda request >download PROD< in progress
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
  ID: 0x7101    Firmware (type - revision): 0x3 - 0x076 (prodID = 1)
      dynamic info
... Configure as TI Slave...
        attachments: total#(1),  ids(0,)
tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1)
        input  list: cnt =    0, events in  = 0, events try = 0
tiSetClockSource: ERROR Setting Clock Source (clkset = 0x2, clkread = 0x0)
        output list: cnt =    0, events out = 0
tiSetTriggerSource: ERROR: Invalid Trigger Source (129).  Must be between 0 and 11.
FiberMeas: ERROR: Unable to measure fiber latency without HFBR1/5 as Clock Source
        Using default Fiber Sync Delay = 522133248 (0x1f1f1f00)tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
tiInit: Scanning for TI...
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0xa00000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x100000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x600000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x880000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x900000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x980000
tiInit: ERROR: Unable to find TI
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory


  LOCAL USERS:
    process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
    process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
    process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)


  ATTACHMENTS:
try again without roc_reboot:
    att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
 
    proc(0), blocked(NO)
LAST COMPILED: Dec 10 2017 12:24:00
      eventsmake(0), get(0), put(0), dump(0)
USRSTRING >usr<
    att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
 
    proc(1), blocked(YES)
vmeCheckMutexHealth: Checking health of vmeBus shared mutex...
      events: make(0), get(0), put(0), dump(0)
* vmeBusTryLock: ERROR: EBUSY
    att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
* vmeBusUnlock: ERROR: EPERM
    proc(2), blocked(NO)
* vmeBusTimedLock: ERROR: ETIMEDOUT
      events: make(0), get(0), put(0), dump(0)
  * vmeBusMutexInit: Initializing vmeBus mutex
    att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
vmeCheckMutexHealth: Mutex Clean and Unlocked
    proc(3), blocked(NO)
usrVmeDmaInit: userMemBase = 0xa07df000, physMemBase = 0x1ba00000, memSize = 1048576 bytes
      events: make(0), get(0), put(0), dump(0)
tiprimarytinit: i2_from_rol1 = 0xa07df000
tiprimarytinit: i2_from_rol1 = 0xa07df000
tiprimarytinit: i2_from_rol1 = 0xa07df010
TI VME (Local) address = 0x00a80000 (0xac014000)
tiInit: ERROR: Invalid Board ID: 0x0 (rval = 0x00000000)
tiInit: Scanning for TI...
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0xa00000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x100000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x600000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x880000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x900000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x980000
tiInit: ERROR: Unable to find TI
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory
 
roc_reboot helped


  EVENTS OWNED BY:
== Sergey B. Dec 2017 run: ==
    system (4000),  att0 (0),  att1 (0),  att2 (0),  att3 (0),
tdcpcal4 during the run:


  EVENT RATE of GC = 0 events/sec
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=480, res=400 =>
[ 0] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=400, res=64 =>
[ 0] ERROR: tdc1190ReadEvent[Dma] returns -2
ERROR: word counters: 13 != 11 (subtract=9)
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=416, res=16 =>
[ 0] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=416, res=400 =>
[ 0] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=416, res=400 =>
[ 0] ERROR: tdc1190ReadEvent[Dma] returns -2


  CREATING STATIONS:
== Sergey B. Dec 2017 run: ==
  IDLE STATIONS:     
  STATION CHAIN:      GRAND_CENTRAL, LEVEL3, ET2ER,
  LOCKED MUTEXES:   


*****************************************
adcft1 on Go:


  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
clasrun@clonpc18:clasrun> tcpClient adcft1 'tiStatus(1)'
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 22276) (lang C) (local) (period = 5 sec)


  STATIC INFO - maximum of:
STATUS for TI at VME (Local) base address 0x00a80000 (0xabfd1000)  
    events(4000), event size(200000), temps(4000)
--------------------------------------------------------------------------------
    stations(10), attaches(20), procs(10), concurrency added(16)
A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac551000)
    network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,
Configured as a TI Slave
Readout Count: 0
    Ack Count: 634
    L1A Count: 6420
  Block Limit: 0 
  Block Count: 642


   DYNAMIC INFO - currently there are:
Registers (offset):
     processes(4), attachments(4), temps(0)
   boardID        (0x0000) = 0x71011546    fiber          (0x0004) = 0x0e0f11ff
     stations(3), hearbeat(1)
  intsetup      (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
  adr32          (0x0010) = 0x08000000    blocklevel    (0x0014) = 0x0a0a0001
  dataFormat     (0x0018) = 0x08080006    vmeControl    (0x001c) = 0x00000011
  trigger        (0x0020) = 0x01000012    sync          (0x0024) = 0x00000c82
  busy          (0x0028) = 0x00000003    clock          (0x002c) = 0x00000002
  blockBuffer    (0x0034) = 0x08000808    output        (0x004c) = 0x00080800
  fiberSyncDelay (0x0050) = 0x4f004f13    nblocks        (0x0094) = 0x00000282
  GTPStatusA    (0x00b0) = 0x0e801fff    GTPStatusB     (0x00b4) = 0xff0fff0f
  livetime      (0x00a8) = 0x009aaadc    busytime      (0x00ac) = 0x00000000
  GTPTrgBufLen  (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x08000807
  triggerRule    (0x0038) = 0x0101010a    triggerRuleMin (0x0138) = 0x00000000


  STATIONS:
    "GRAND_CENTRAL" (id = 0)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(2,3,)
        input  list: cnt =  4000, events in  = 0
        output list: cnt =    0, events out = 0


    "LEVEL3" (id = 1)
Block Level        = 10
      static info
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
         status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
 
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
Fiber Status         1     2    3    4    5    6    7    8
      dynamic info
                    ----- ----- ----- ----- ----- ----- ----- -----
        attachments: total#(1), ids(1,)
  Connected          YES  YES  YES  YES                         
        input list: cnt =     0, events in = 0, events try = 0
 
        output list: cnt =     0, events out = 0
 
Clock Source (2) =
  HFBR #1 Input
  Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
  Sync source =  
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
  BUSY input source =  
  Switch Slot A   
  Switch Slot B   
 
Interrupts DISABLED
  Level = 5  Vector = 0xc8
Bus Errors Enabled


    "ET2ER" (id = 2)
  Blocks ready for readout: 8
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(1),  ids(0,)
        input  list: cnt =    0, events in = 0, events try = 0
        output list: cnt =    0, events out = 0


  LOCAL USERS:
Input counter 1
    process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
--------------------------------------------------------------------------------
    process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
    process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
    process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)


  ATTACHMENTS:
    att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
    proc(0), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
    proc(1), blocked(YES)
      events:  make(0), get(0), put(0), dump(0)
    att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
    proc(2), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
    proc(3), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)


  EVENTS OWNED BY:
clasrun@clonpc18:clasrun> tcpClient trig1 'tdGStatus'
    system (4000),  att0 (0),  att1 (0),  att2 (0),  att3 (0),


   EVENT RATE of GC = 0 events/sec
TD Module Status Summary
              Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
3    12345678    12345678  V----  ---12345678  ----------  0xba304800  N
4    12345678    12345678  V----  ---12345678  ----------  0x32f04800  N
5    12345678    12345678  V----  ---12345678  ----------  0x42b05c00  N
6    123456-8    123456--  V----  ---123456--  ----------  0x0ff05c00  N
7    12345678    12345678  V----  ---12345678  --------7-  0xfb305c00  N
8    12345678    12345678  V----  ---12345678  ----------  0x11504800  N
9    123456--    123456--  V----  ---123456--  ----------  0xe9905c00  N
10    123456--    123456--  V----  ---123456--  ----------  0x47505c00  N
13    1234567-    1234567-  V----  ---1234567-  ----------  0x82705c00  N


   CREATING STATIONS:  
TD Port STATUS Summary
   IDLE STATIONS:      
                                                      Block Status      Block  Buffer
   STATION CHAIN:     GRAND_CENTRAL, LEVEL3, ET2ER,
Sl-P  RocName  COMM  ROCID  TrigSrcEn  Busy Status  Ready / NeedAck    Level  Level
   LOCKED MUTEXES:      
--------------------------------------------------------------------------------
3-1  adcecal1  UP      1    ENABLED                    0 /  0          10      0
3-2  tdcecal1  UP      2    ENABLED                    0 /  0          10      0
3-3  adcpcal1  UP      3    ENABLED                    0 /  0          10      0
3-4  tdcpcal1  UP      4    ENABLED                    0 /  0          10      0
3-5  adcftof1  UP      5    ENABLED                    0 /  0          10      0
3-6  tdcftof1  UP      6    ENABLED                    0 /  0          10      0
3-7    trig2  UP    38    ENABLED                    0 /  0          10      0
3-8  adccnd1  UP    73    ENABLED                    0 /  0          10      0
4-1  adcecal2  UP      7    ENABLED                    0 /  0          10      0
4-2  tdcecal2  UP      8    ENABLED                    0 /  0          10      0
4-3  adcpcal2  UP      9    ENABLED                    0 /  0          10      0
4-4  tdcpcal2  UP    10    ENABLED                    0 /  0          10      0
4-5  adcftof2  UP    11    ENABLED                    0 /  0          10      0
4-6  tdcftof2  UP    12    ENABLED                    0 /  0          10      0
4-7  adcctof1  UP    59    ENABLED                    0 /  0          10      0
4-8  tdcctof1  UP    60    ENABLED                    0 /  0          10      0
5-1  adcecal3  UP    13    ENABLED                    0 /  0          10      0
5-2  tdcecal3  UP    14    ENABLED                    0 /  0          10      0
5-3  adcpcal3  UP    15    ENABLED                    0 /  0          10      0
5-4  tdcpcal3  UP    16    ENABLED                    0 /  0          10      0
5-5  adcftof3  UP    17    ENABLED                    0 /  0          10      0
5-6  tdcftof3  UP    18    ENABLED                    0 /  0          10      0
5-7      svt1  UP    61    ENABLED                    0 /  0          10      0
5-8      svt2  UP    62    ENABLED                    0 /  0          10      0
6-1  adcecal4  UP    19    ENABLED                    0 /  0          10      0
6-2  tdcecal4  UP    20    ENABLED                    0 /  0          10      0
6-3  adcpcal4  UP    21    ENABLED                    0 /  0          10      0
6-4  tdcpcal4  UP    22    ENABLED                    0 /  0          10      0
6-5  adcftof4  UP    23    ENABLED                    0 /  0          10      0
6-6  tdcftof4  UP    24    ENABLED                    0 /  0          10      0
7-1  adcecal5  UP    25    ENABLED                    0 /  0          10      0
7-2  tdcecal5  UP    26    ENABLED                    0 /  0          10      0
7-3  adcpcal5  UP    27    ENABLED                    0 /  0          10      0
7-4  tdcpcal5  UP    28    ENABLED                    0 /  0          10      0
7-5  adcftof5  UP    30    ENABLED                    0 /   0          10      0
7-6  tdcftof5 UP    29    ENABLED                    0 /  0          10      0
7-7    adcft1  UP    70    ENABLED   BUSY-BlkAck     8 /  0          10      0
7-8    adcft2  UP    71    ENABLED                    0 /   0          10      0
8-1  adcecal6  UP    31    ENABLED                    0 /  0          10      0
8-2  tdcecal6  UP    32    ENABLED                    0 /  0          10      0
8-3  adcpcal6  UP    33    ENABLED                    0 /  0          10      0
8-4  tdcpcal6  UP    34    ENABLED                    0 /  0          10      0
8-5  adcftof6  UP    35    ENABLED                    0 /  0          10      0
8-6  tdcftof6  UP    36    ENABLED                    0 /  0          10      0
8-7    adcft3  UP    72    ENABLED                    0 /  0          10      0
8-8    mmft1  UP    75    ENABLED                    0 /  0          10      0
9-1     dc11  UP    41    ENABLED                    0 /  0          10      0
9-2      dc21  UP    44    ENABLED                    0 /  0          10      0
9-3      dc31  UP    47    ENABLED                    0 /  0          10      0
9-4      dc41  UP    50    ENABLED                    0 /  0          10      0
9-5      dc51  UP    53    ENABLED                    0 /  0          10      0
9-6      dc61  UP    56    ENABLED                    0 /  0          10      0
10-1      dc12  UP    42    ENABLED                    0 /  0          10      0
10-2      dc22  UP    45    ENABLED                    0 /  0          10      0
10-3      dc32  UP    48    ENABLED                    0 /  0          10      0
10-4      dc42  UP    51    ENABLED                    0 /  0          10      0
10-5      dc52  UP    54    ENABLED                    0 /  0          10      0
10-6      dc62  UP    57    ENABLED                    0 /   0          10      0
13-1      dc13  UP    43    ENABLED                    0 /  0          10      0
13-2      dc23  UP    46     ENABLED                    0 /  0          10      0
13-3      dc33  UP    49    ENABLED                    0 /  0          10      0
13-4      dc43  UP    52    ENABLED                    0 /  0          10      0
13-5      dc53  UP    55    ENABLED                    0 /  0          10      0
13-6      dc63  UP    58    ENABLED                    0 /  0          10      0
13-7      mvt1  UP    69    ENABLED                    0 /  0          10      0


*****************************************
Total Slaves Added = 65
--------------------------------------------------------------------------------


ET SYSTEM is DEAD!
*****************************************


^C
== Sergey B. Dec 2017 run: ==
clondaq1:dac>


tdcftof3 during download:


FiberMeas: ERROR: Unable to measure fiber latency without HFBR1/5 as Clock Source
        Using default Fiber Sync Delay = 522133248 (0x1f1f1f00)
tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
tiInit: Scanning for TI...
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x980000
tiInit: ERROR: Unable to find TI
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory




coda_roc: ======= Close the default VME windows =========


== Sergey B. Dec 2017 run: ==
tiReload:
FPGA Re-Load !
CODAtcpServer: wait: coda request >download PROD< in progress
  ID: 0x7101    Firmware (type - revision): 0x3 - 0x076 (prodID = 1)
... Configure as TI Slave...
tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1)
tiSetClockSource: ERROR Setting Clock Source (clkset = 0x2, clkread = 0x0)
tiSetTriggerSource: ERROR: Invalid Trigger Source (129).  Must be between 0 and 11.
FiberMeas: ERROR: Unable to measure fiber latency without HFBR1/5 as Clock Source
        Using default Fiber Sync Delay = 522133248 (0x1f1f1f00)tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
tiInit: Scanning for TI...
tiFind: Found TI at 0x00a80000
TI VME (Local) address = 0x00a80000 (0xabfd8000)
tiInit: ERROR: Invalid Board ID: 0x0 (rval = 0x00000000)
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory


clondaq2:stage_in>
clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 16127) (lang C) (local) (period = 5 sec)


  STATIC INFO - maximum of:
== Sergey B. Dec 2017 run: DiagGuiServer ... ==
    events(4000), event size(200000), temps(4000)
adcctof1: DiagGuiServer was restarted
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,


   DYNAMIC INFO - currently there are:
proc_thread: waiting=   7610 processing=    203 microsec per event (nev=12)
    processes(3), attachments(3), temps(0)
net_thread:  waiting=    777    sending=      0 microsec per event (nev=128)
    stations(3), hearbeat(1)
jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x500404
usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00800404 < 0x0e000000)
faReadBlock: ERROR: vmeDmaDone returned an Error
FADCs: No data or error.  dCnt = -1 (slots from 3)
jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x880404
jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x900404
usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00b00404 < 0x0e000000)
faReadBlock: ERROR: vmeDmaDone returned an Error
FADCs: No data or error.  dCnt = -1 (slots from 3)
jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x580000
usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00880000 < 0x0e000000)
faReadBlock: ERROR: vmeDmaDone returned an Error
FADCs: No data or error.  dCnt = -1 (slots from 3)
jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x980000
usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00100000 < 0x08000000)


  STATIONS:
tiReadBlock: ERROR: vmeDmaDone returned an Error
    "GRAND_CENTRAL" (id = 0)
ERROR in tiReadBlock : No data or error, len = -1
      static info
ERROR1: LINK_sized_write() returns errno=32 (cc=-1, sizeof(nbytes)=4(162600)
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
 
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
== Sergey B. Dec 2017 run: ==
      dynamic info
 
        attachments: total#(0), ids()
adcpcal3vtp:
        input list: cnt = 4000, events in = 0
VTP UNKNOWN data2: [569] 0x00000008
        output list: cnt =     0, events out = 0
VTP UNKNOWN data2: [570] 0x8d40002b
VTP UNKNOWN data2: [572] 0x000001e3
VTP UNKNOWN data1C: [573] 0xe122016d
VTP UNKNOWN data2: [574] 0x82fc060a
[70667][ 56] ERROR1 in VTP data: blockheader slot 21 != blocktrailer slot 11
VTP UNKNOWN data2: [  4] 0x000001d7
VTP UNKNOWN data2: [  5] 0x00010e1c
VTP UNKNOWN data2: [  6] 0x0000002d
VTP UNKNOWN data1C: [ 7] 0xe10a016d
VTP UNKNOWN data2: [ 8] 0x85401c0a
VTP UNKNOWN data2: [ 11] 0x000ac90f
VTP UNKNOWN data2: [ 12] 0x6f13541b
VTP UNKNOWN data2: [ 13] 0x00000008
VTP UNKNOWN data2: [ 14] 0x0c010003
VTP UNKNOWN data2: [ 15] 0x000ac910
VTP UNKNOWN data2: [ 16] 0x6f1435d1
VTP UNKNOWN data2: [ 17] 0x00000008
VTP UNKNOWN data2: [ 19] 0x000ac911
VTP UNKNOWN data2: [ 20] 0x6f1451e5
VTP UNKNOWN data2: [ 21] 0x00000008
VTP UNKNOWN data2: [ 23] 0x000ac912
VTP UNKNOWN data2: [ 24] 0x6f159615
VTP UNKNOWN data2: [ 25] 0x00000008
VTP UNKNOWN data2: [ 26] 0x20010003
VTP UNKNOWN data2: [ 27] 0x000ac913
VTP UNKNOWN data2: [ 28] 0x6f15b188
VTP UNKNOWN data2: [ 29] 0x00000008
VTP UNKNOWN data2: [ 31] 0x000ac914
VTP UNKNOWN data2: [ 32] 0x6f1739ea
VTP UNKNOWN data2: [ 33] 0x00000008
VTP UNKNOWN data2: [ 35] 0x000ac915
VTP UNKNOWN data2: [ 36] 0x6f176682
VTP UNKNOWN data2: [ 37] 0x00000008
VTP UNKNOWN data2: [ 39] 0x000ac916
VTP UNKNOWN data2: [ 40] 0x6f17b3e0
VTP UNKNOWN data2: [ 41] 0x00000008
VTP UNKNOWN data2: [ 43] 0x000ac917
VTP UNKNOWN data2: [ 44] 0x6f185ceb
VTP UNKNOWN data2: [ 45] 0x00000008
VTP UNKNOWN data2: [ 47] 0x000ac918
VTP UNKNOWN data2: [ 48] 0x6f186d5e
VTP UNKNOWN data2: [ 49] 0x00000008
VTP UNKNOWN data2: [ 50] 0x8d40002b
VTP UNKNOWN data2: [ 52] 0x000001a7
VTP UNKNOWN data1C: [ 53] 0xe122016d
VTP UNKNOWN data2: [ 87] 0x00560014
VTP UNKNOWN data2: [115] 0x00b100c9
VTP UNKNOWN data2: [239] 0x008e0016
[70684][421] ERROR1 in VTP data: blockheader slot 21 != blocktrailer slot 11
VTP UNKNOWN data2: [381] 0x00000042
VTP UNKNOWN data2: [382] 0x00010e1f
VTP UNKNOWN data2: [383] 0x0000002d
VTP UNKNOWN data1C: [384] 0xe10a016d
VTP UNKNOWN data2: [385] 0x85401f0a
VTP UNKNOWN data2: [388] 0x000ac92d
VTP UNKNOWN data2: [389] 0x6f22f0b4
VTP UNKNOWN data2: [390] 0x00000008
VTP UNKNOWN data2: [392] 0x000ac92e
VTP UNKNOWN data2: [393] 0x6f2335c2
VTP UNKNOWN data2: [394] 0x00000008
VTP UNKNOWN data2: [396] 0x000ac92f
[70687][ 16] ERROR1 in VTP data: blockheader slot 21 != blocktrailer slot 11
 
== Sergey B. Dec 2017 run: ==
adcpcal5vtp:


     "TAPE" (id = 1)
proc_thread: waiting=    977 processing=    898 microsec per event (nev=26)
      static info
net_thread:  waiting=    165    sending=     18 microsec per event (nev=266)
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
VTP UNKNOWN data2: [131] 0x000000dc
        restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
VTP UNKNOWN data2: [132] 0x00010e22
      dynamic info
VTP UNKNOWN data2: [133] 0x0000002d
        attachments: total#(2),  ids(1,2,)
VTP UNKNOWN data1C: [134] 0xe10a016f
        input  list: cnt =    0, events in  = 0, events try = 0
VTP UNKNOWN data2: [135] 0x8541220a
        output list: cnt =    0, events out = 0
VTP UNKNOWN data2: [138] 0x00046b4b
VTP UNKNOWN data1A: [139] 0xd7a62fbd
VTP UNKNOWN data2: [140] 0x00000003
VTP UNKNOWN data2: [142] 0x00046b4c
VTP UNKNOWN data1A: [143] 0xd7a7146f
VTP UNKNOWN data2: [144] 0x00000003
VTP UNKNOWN data2: [221] 0x00bd001c
VTP UNKNOWN data2: [326] 0x09333469
Segmentation fault (core dumped)
clasrun@adcpcal5vtp:clasrun>


    "ET2ET10" (id = 2)
== Sergey B. Dec 2017 run: ==
      static info
dc52, on next 'go' dc23, on next 'go' dc42, dc51, dc61
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
        lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so,  function = et2et_function, class =  
      dynamic info
        attachments: total#(1), ids(0,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0


  LOCAL USERS:
    process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
    process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)


  ATTACHMENTS:
    att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
    proc(0), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
    proc(1), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
    proc(2), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)


  EVENTS OWNED BY:
    system (4000),  att0 (0),  att1 (0),  att2 (0),


  EVENT RATE of GC = 0 events/sec
......


  CREATING STATIONS:
CODAtcpServerWorkTask exit ?
  IDLE STATIONS:    
CODAtcpServerWorkTask exit !
  STATION CHAIN:     GRAND_CENTRAL, TAPE, ET2ET10,  
usrVme2MemDmaStart ERROR: DMA Physical Memory has an invalid base address (0x00000000)
  LOCKED MUTEXES:    
tiReadBlock: ERROR in DMA transfer Initialization 0xffffffff
ERROR in tiReadBlock : No data or error, len = -1
usrVme2MemDmaStart ERROR: DMA Physical Memory has an invalid base address (0x00000000)dcrbReadBlock: ERROR in DMA transfer Initialization 0xffffffff
Segmentation fault


*****************************************
.......


  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
== Sergey B. Dec 2017 run: ==
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 16127) (lang C) (local) (period = 5 sec)


  STATIC INFO - maximum of:
adcpcal5vtp on holdoffs:
    events(4000), event size(200000), temps(4000)
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,


  DYNAMIC INFO - currently there are:
#TS_HOLDOFF 1 10 1
    processes(3), attachments(3), temps(0)
#TS_HOLDOFF 2 10 1
    stations(3), hearbeat(1)
#TS_HOLDOFF 3 7 1
#TS_HOLDOFF 4 5 1


  STATIONS:
dabufp[0] = 0x00000133 (307)
    "GRAND_CENTRAL" (id = 0)
dabufp[1] = 0x00010e01 (69121)
      static info
1][137] ERROR2 in VTP data: trailer #words=137 != actual #words=138 - ndnv=0)
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
proc_thread: waiting= 31339 processing=    906 microsec per event (nev=22)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
net_thread: waiting=   774    sending=     0 microsec per event (nev=912)
      dynamic info
ERROR in TI event header word: event type = 0
        attachments: total#(0), ids()
ERROR in TI event header word (0x02)
        input list: cnt = 4000, events in  = 0
ERROR: TS has more then 7 data words - exit
        output list: cnt =    0, events out = 0
clasrun@adcpcal5vtp:clasrun>


    "TAPE" (id = 1)
== Sergey B. Dec 2017 run: sync to tdc ??? do not read if almost full or full ?? ==
      static info
 
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
NEED ALARM BROWSER AND SEND THIS KIND OF ERRORS THERE !!! OR AT LEAST TO RUNCONTROL
        restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(1,2,)
        input  list: cnt =     0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0


    "ET2ET10" (id = 2)
tdcftof4:
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
        lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so,  function = et2et_function, class =
      dynamic info
        attachments: total#(1),  ids(0,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0


  LOCAL USERS:
it complains constantly:
    process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
    process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)


  ATTACHMENTS:
ERR: event#  925554, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000
    att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
ERR: event# 925594, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000
    proc(0), blocked(NO)
ERR: event#  925654, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000
      eventsmake(0), get(0), put(0), dump(0)
ERR: event# 925694, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000
    att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
 
    proc(1), blocked(NO)
 
      eventsmake(0), get(0), put(0), dump(0)
 
    att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
but on high rate happens this:
    proc(2), blocked(NO)
 
      events: make(0), get(0), put(0), dump(0)
 
[ 4] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2048, res=176 => mbytes=1872
[ 4] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2144, res=2064 => mbytes=80
[ 4] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2288, res=160 => mbytes=2128
[ 4] ERROR: tdc1190ReadEvent[Dma] returns -2
tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2704, res=2272 => mbytes=432
[ 4] ERROR: tdc1190ReadEvent[Dma] returns -2


  EVENTS OWNED BY:
    system (4000),  att0 (0),  att1 (0),  att2 (0),


  EVENT RATE of GC = 0 events/sec


  CREATING STATIONS: 
  IDLE STATIONS:     
  STATION CHAIN:      GRAND_CENTRAL, TAPE, ET2ET10,
  LOCKED MUTEXES:   


*****************************************
6-3  adcpcal4  UP    21    DISABLED                    0 /  0          20      0
6-4  tdcpcal4  UP    22    DISABLED                    0 /  0          20      0
6-5  adcftof4  UP    23    DISABLED                    0 /  0          20      0
6-6  tdcftof4  UP    24    ENABLED  BUSY-Fiber      0 /  0          20      0
6-8  scaler1  UP    64    DISABLED                    0 /  0          20      0
7-1  adcecal5  UP    25    DISABLED                    0 /  0          20      0
7-2  tdcecal5  UP    26    DISABLED                    0 /  0          20      0
7-3  adcpcal5  UP    27    DISABLED                    0 /  0          20      0
7-4  tdcpcal5  UP    28    DISABLED                    0 /  0          20      0
7-5  adcftof5  UP    30    DISABLED                    0 /  0          20      0
7-6  tdcftof5  UP    29    DISABLED                    0 /  0          20      0
7-7    adcft1  UP    70    DISABLED                    0 /  0          20      0
7-8    adcft2  UP    71    DISABLED                    0 /  0          20      0
8-1  adcecal6  UP    31    DISABLED                    0 /  0          20      0
8-2  tdcecal6  UP    32    DISABLED                    0 /  0          20      0
8-3  adcpcal6  UP    33    DISABLED                    0 /  0          20      0
8-4  tdcpcal6  UP    34    DISABLED                    0 /  0          20      0
8-5  adcftof6  UP    35    DISABLED                    0 /  0          20      0
8-6  tdcftof6  UP    36    DISABLED                    0 /  0          20      0
8-7    adcft3  UP    72    DISABLED                    0 /  0          20      0
9-1      dc11  UP    41    DISABLED                    0 /  0          20      0
9-2      dc21  UP    44    DISABLED                    0 /  0          20      0
9-3      dc31  UP    47    DISABLED                    0 /  0          20      0
9-4      dc41  UP    50    DISABLED                    0 /  0          20      0
9-5      dc51  UP    53    DISABLED                    0 /  0          20      0
9-6      dc61  UP    56    DISABLED                    0 /  0          20      0
10-1      dc12  UP    42    DISABLED                    0 /  0          20      0
10-2      dc22  UP    45    DISABLED                    0 /  0          20      0
10-3      dc32  UP    48    DISABLED                    0 /  0          20      0
10-4      dc42  UP    51    DISABLED                    0 /  0          20      0
10-5      dc52  UP    54    DISABLED                    0 /  0          20      0
10-6      dc62  UP    57    DISABLED                    0 /  0          20      0
13-1      dc13  UP    43    DISABLED                    0 /  0          20      0
13-2      dc23  UP    46    DISABLED                    0 /  0          20      0
13-3      dc33  UP    49    DISABLED                    0 /  0          20      0
13-4      dc43  UP    52    DISABLED                    0 /  0          20      0
13-5      dc53  UP    55    DISABLED                    0 /  0          20      0
13-6      dc63  UP    58    DISABLED                    0 /  0          20      0


ET SYSTEM is DEAD!
Total Slaves Added = 62
*****************************************
--------------------------------------------------------------------------------


^C
clondaq2:stage_in>


clasrun@clondaq6:clasrun> tcpClient tdcftof4 tiStatus


* Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem:
STATUS for TI at VME (Local) base address 0x00a80000 (0xabfae000)
--------------------------------------------------------------------------------
A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac52e000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 88880
    L1A Count: 1777608
  Block Limit:
  Block Count: 88880


  wait: coda request >end< in progress
  Registers (offset):
0x916e80 (twork0028): TS GO bit cleared
  boardID        (0x0000) = 0x71011518    fiber          (0x0004) = 0x001111ff
0x916e80 (twork0028): TS csr:  0xfc070000
  intsetup      (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
0x916e80 (twork0028): TS csr2:  0xffff1c1f
  adr32          (0x0010) = 0x08000000    blocklevel    (0x0014) = 0x14140001
interrupt: TRIG ERROR: no pool buffer available
  dataFormat    (0x0018) = 0x08080006    vmeControl    (0x001c) = 0x00000011
0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0
  trigger        (0x0020) = 0x01000000    sync          (0x0024) = 0x00000c82
0x916e80 (twork0028): INFO: User End Executed
  busy          (0x0028) = 0x00850004    clock          (0x002c) = 0x00840006
codaUpdateStatus: dbConnecting ..
  blockBuffer    (0x0034) = 0x00080001    output        (0x004c) = 0x00000000
codaUpdateStatus: dbConnect done
  fiberSyncDelay (0x0050) = 0x4f004f11    nblocks        (0x0094) = 0x08015b30
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<
  GTPStatusA    (0x00b0) = 0x10801fff    GTPStatusB    (0x00b4) = 0xff11ff11
codaUpdateStatus: dbDisconnecting ..
  livetime      (0x00a8) = 0x01112a8e    busytime      (0x00ac) = 0x003e21ce
codaUpdateStatus: dbDisconnect done
  GTPTrgBufLen  (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x00000007
  triggerRule    (0x0038) = 0x0101010a    triggerRuleMin (0x0138) = 0x00000000




Block Level        = 20
Block Buffer Level = 8 -Broadcast- (Busy not enabled)


Fiber Status        1    2    3    4    5    6    7    8
                    ----- ----- ----- ----- ----- ----- ----- -----
  Connected          YES                    YES                   


* Sergey B. 14-apr-2010: message from runcontrol during cancel-reset:
Get all components failed: Lost connection to MySQL server during query


* Sergey B. 6-apr-2010: EB waiting for following ROCs:
Clock Source (2) =
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source =
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source =
  P2 Input        ** BUSY **


  [0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
  Interrupts DISABLED
  7 9 10 11 17 20 21 22 25 26 28 29 30
  Level = 5  Vector = 0xc8
  Bus Errors Enabled


Files produced:
Blocks ready for readout: 0


  clon10:ccscans> ll | grep Apr
  Input counter 1
drwxrwxrwx  2 clasrun  onliners    4096 Apr  2 16:23 ./
--------------------------------------------------------------------------------
drwxr-xr-x 163 clasrun  100        73728 Apr  6  2010 ../
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_cc1*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:09 good_clastrig2*
-rwxrwxrwx  1 boiarino clas          34 Apr  2 16:25 good_croctest10*
-rwxrwxrwx  1 boiarino clas          34 Apr  2 16:25 good_croctest2*
-rwxrwxrwx  1 boiarino clas          34 Apr  2 16:21 good_croctest3*
-rwxrwxrwx  1 baturin  clas          34 Apr  2 13:26 good_ctoftest1*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc1*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc10*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc11*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc2*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc3*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc4*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc5*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc6*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc7*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc8*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc9*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_ec1*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_ec2*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_ec3*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_ec4*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_polar*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_sc1*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_sc2*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_scaler1*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_scaler2*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_scaler3*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_scaler4*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_tage*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_tage2*
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_tage3*
clon10:ccscans> more good_dc7
accepted link from >129.57.68.80<
clon10:ccscans> more good_dc8
accepted link from >129.57.68.74<
clon10:ccscans>


On 'Reset' EB crashed, outputs:


[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
  7  9 10 11 17 20 21 22 25 26 28 29 30


  [0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
clasrun@clondaq6:clasrun
   7 9 10 11 17 20 21 22 25 26 28 29 30
 
 
 
 
 
clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(0)'
STATUS for v1290 TDC at VME (USER) base address 0x11180000 (0xb56ae000)
----------------------------------------------
Interrupts Disabled
 
  Data Status
    Events to transfer via BLT = 0x0014
    Status        = 0x0039 (Data Ready )
    Events Stored = 8
    Total Events  = 8136
    Almost Full  = 16384
 
  TDC Mode/Status
    Resolution      : 25 ps
    Trigger Matching: On
    TDC Headers/EOB : On
    Bus Errors      : Off
    TDC Errors      : 0:0   1:0  2:0  3:0
 
 
clasrun@clondaq6:clasrun>
clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(1)'
STATUS for v1290 TDC at VME (USER) base address 0x11300000 (0xb582e000)
----------------------------------------------
Interrupts Disabled
 
  Data Status
    Events to transfer via BLT  = 0x0014
    Status        = 0x0039 (Data Ready )
    Events Stored = 8
    Total Events  = 8136
    Almost Full  = 16384
 
  TDC Mode/Status
    Resolution      : 25 ps
    Trigger Matching: On
    TDC Headers/EOB : On
    Bus Errors      : Off
    TDC Errors      : 0:0   1:0  2:0  3:0
 
 
clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(2)'
STATUS for v1290 TDC at VME (USER) base address 0x11480000 (0xb59ae000)
----------------------------------------------
  Interrupts Disabled
 
  Data Status
    Events to transfer via BLT  = 0x0014
    Status        = 0x0039 (Data Ready )
    Events Stored = 8
    Total Events  = 8136
    Almost Full  = 16384
 
  TDC Mode/Status
    Resolution      : 25 ps
    Trigger Matching: On
    TDC Headers/EOB : On
    Bus Errors      : Off
    TDC Errors      : 0:0  1:0  2:0  3:0
 


CODAtcpServer: start work thread
clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(3)'
befor: socket=5 address>129.57.167.14< port=34468
STATUS for v1290 TDC at VME (USER) base address 0x11680000 (0xb5bae000)
wait: coda request >prestart< in progress
----------------------------------------------
codaExecute reached, message >exit<, len=4
  Interrupts Disabled
codaExecute: 'exit' transition
 
  debShutdownBuild 1
  Data Status
  cb_delete roc=0
    Events to transfer via BLT = 0x0014
cb_delete roc=1
    Status        = 0x0079 (Data Ready )
cb_delete roc=2
    Events Stored = 8
  cb_delete roc=3
    Total Events = 8136
cb_delete roc=4
    Almost Full  = 16384
cb_delete roc=5
 
cb_delete roc=6
  TDC Mode/Status
cb_delete roc=7
    Resolution      : 25 ps
cb_delete roc=8
    Trigger Matching: On
cb_delete roc=9
    TDC Headers/EOB : On
cb_delete roc=10
    Bus Errors      : Off
cb_delete roc=11
    TDC Errors      : 0:1  1:0  2:0  3:0
cb_delete roc=12
 
cb_delete roc=13
 
cb_delete roc=14
clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(4)'
cb_delete roc=15
STATUS for v1190 TDC at VME (USER) base address 0x11800000 (0xb5d2e000)
cb_delete roc=16
----------------------------------------------
cb_delete roc=17
  Interrupts Disabled
cb_delete roc=18
 
  cb_delete roc=19
  Data Status
cb_delete roc=20
    Events to transfer via BLT = 0x0014
  cb_delete roc=21
    Status        = 0x003b (Almost Full)
cb_delete roc=22
    Events Stored = 368
cb_delete roc=23
    Total Events = 8136
  cb_delete roc=24
    Almost Full  = 16384
cb_delete roc=25
 
cb_delete roc=26
  TDC Mode/Status
cb_delete roc=27
    Resolution      : 100 ps
cb_delete roc=28
    Trigger Matching: On
cb_delete roc=29
    TDC Headers/EOB : On
cb_delete roc=30
    Bus Errors      : Off
cb_delete roc=31
    TDC Errors      : 0:0  1:0  2:0   3:0
debShutdownBuild 2
 
Stop waiting for the rocs because of force_end condition
== Sergey B. 11-jan-2016: pcal0 test setup coda_roc error messages ==
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0
 
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!
  ..............
[0] et_event_new() 1
  net_thread:  waiting=      0    sending=      1 microsec per event (nev=120)
  [0] EVENT_ENCODE_SPEC 2 ..
  proc_thread: waiting=      0 processing=  1568 microsec per event (nev=30)
  BOS_encode_spec reached
  proc_thread: waiting=      0 processing=  1539 microsec per event (nev=30)
  BOS_encode_spec done: len=116
  ERROR in TI event header word (0x1d)
  [0] .. done
  ERROR in TI event header word (0x1d)
  [0] handle_build_cleanup
  ERROR in TI event header word (0x1d)
  [0] flush ET system
  ERROR in TI event header word (0x1d)
  [0] detached from ET
  ERROR in TI event header word (0x1d)
  [0] remove mutex locks and shutdown fifos
  ERROR in TI event header word (0x1d)
  [0] flush input streams
  ERROR in TI event header word (0x1e)
  CCCCCCCCCC: 3
  ERROR in TI event header word (0x1e)
  [0] count for roc[00] = 3
  ERROR in TI event header word (0x1e)
  CCCCCCCCCC: 3
  ERROR in TI event header word (0x1e)
  CCCCCCCCCC: 1
  ERROR in TI event header word (0x1e)
  [0] count for roc[00] = 1
  ERROR in TI event header word (0x1e)
  CCCCCCCCCC: 1
  ERROR in TI event header word (0x1e)
  CCCCCCCCCC: 0
  ERROR in TI event header word (0x1e)
  [0] count for roc[00] = 0
  ERROR in TI event header word (0x1f)
  CCCCCCCCCC: 0
  ERROR in TI event header word (0x1f)
  [0] flush input streams
  ERROR in TI event header word (0x1f)
  CCCCCCCCCC: 3
  ERROR in TI event header word (0x1f)
  [0] count for roc[01] = 3
  ERROR in TI event header word (0x1f)
  CCCCCCCCCC: 3
  ERROR in TI event header word (0x1f)
  CCCCCCCCCC: 1
  ERROR in TI event header word (0x1f)
  [0] count for roc[01] = 1
  ERROR in TI event header word (0x1f)
  CCCCCCCCCC: 1
  ERROR in TI event header word (0x20)
  CCCCCCCCCC: 0
  ERROR in TI event header word (0x20)
  [0] count for roc[01] = 0
  ERROR in TI event header word (0x20)
  CCCCCCCCCC: 0
  TI UNKNOWN data: [162] 0x00000000
  [0] flush input streams
 
  CCCCCCCCCC: 3
  Previous stuff
[0] count for roc[02] = 3
          [142][162] 0xa41fc94d
CCCCCCCCCC: 3
          [143][162] 0x00000000
CCCCCCCCCC: 1
          [144][162] 0x01010003
[0] count for roc[02] = 1
          [145][162] 0x000052cd
CCCCCCCCCC: 1
          [146][162] 0xa41fe9db
CCCCCCCCCC: 0
          [147][162] 0x00000000
[0] count for roc[02] = 0
          [148][162] 0x01010003
CCCCCCCCCC: 0
          [149][162] 0x000052ce
[0] flush input streams
          [150][162] 0xa4200a68
CCCCCCCCCC: 3
          [151][162] 0x00000000
[0] count for roc[03] = 3
          [152][162] 0x01010003
CCCCCCCCCC: 3
          [153][162] 0x000052cf
CCCCCCCCCC: 1
          [154][162] 0xa4202af5
[0] count for roc[03] = 1
          [155][162] 0x00000000
CCCCCCCCCC: 1
          [156][162] 0x01010003
CCCCCCCCCC: 0
          [157][162] 0x000052d0
[0] count for roc[03] = 0
          [158][162] 0xa4204b82
CCCCCCCCCC: 0
          [159][162] 0x00000000
[0] flush input streams
          [160][162] 0x0000fafe
CCCCCCCCCC: 3
          [161][162] 0x0101d97e
[0] count for roc[04] = 3
          [162][162] 0x00000000
CCCCCCCCCC: 3
          [163][162] 0xb9820003
CCCCCCCCCC: 1
          [164][162] 0x0000faff
[0] count for roc[04] = 1
          [165][162] 0x0101fa0b
CCCCCCCCCC: 1
          [166][162] 0x00000000
CCCCCCCCCC: 0
          [167][162] 0x01010003
[0] count for roc[04] = 0
          [168][162] 0x0000fb00
CCCCCCCCCC: 0
          [169][162] 0xb9830003
[0] flush input streams
          [170][162] 0x0000fb01
CCCCCCCCCC: 3
          [171][162] 0x01013b25
[0] count for roc[05] = 3
  End Of Previous stuff
CCCCCCCCCC: 3
  net_threadwaiting=    104    sending=      1 microsec per event (nev=1080)
CCCCCCCCCC: 1
  coda_roc is exiting, clear dma memory
[0] count for roc[05] = 1
  net_threadwaiting=      0   sending=      1 microsec per event (nev=120)
CCCCCCCCCC: 1
  bb_delete1: gefVmeFreeDmaBuf[ 0] has been called
CCCCCCCCCC: 0
  bb_delete1: gefVmeFreeDmaBuf[ 1] has been called
[0] count for roc[05] = 0
  Segmentation fault
CCCCCCCCCC: 0
  clasrun@pcal0:clasrun>
[0] flush input streams
 
CCCCCCCCCC: 3
AFTER roc_reboot:
[0] count for roc[06] = 3
 
CCCCCCCCCC: 3
  .......
CCCCCCCCCC: 1
  TI UNKNOWN data: [ 0] 0x00000000
[0] count for roc[06] = 1
 
CCCCCCCCCC: 1
  Previous stuff
CCCCCCCCCC: 0
          [-20]0] 0x00000000
[0] count for roc[06] = 0
          [-19][ 0] 0x900052f7
CCCCCCCCCC: 0
          [-18][ 0] 0xa0000005
[0] flush input streams
          [-17][ 0] 0x00000000
CCCCCCCCCC: 3
          [-16][ 0] 0x00001466
[0] count for roc[07] = 3
          [-15][ 0] 0x000052f7
CCCCCCCCCC: 3
          [-14][ 0] 0x00000000
  CCCCCCCCCC: 1
          [-13][  0] 0x00000000
  [0] count for roc[07] = 1
          [-12][  0] 0x900052f8
  CCCCCCCCCC: 1
          [-11][ 0] 0xa0000005
  CCCCCCCCCC: 0
          [-10][ 0] 0x00000000
  [0] count for roc[07] = 0
          [ -9][ 0] 0x00001466
  CCCCCCCCCC: 0
          [ -8][ 0] 0x000052f8
[0] flush input streams
          [ -7][ 0] 0x5693d19e
  CCCCCCCCCC: 3
          [ -6][ 0] 0x00000001
  [0] count for roc[08] = 3
          [ -5][ 0] 0x8800011a
  CCCCCCCCCC: 3
          [ -4][ 0] 0x000069de
CCCCCCCCCC: 1
          [ -3][ 0] 0x00010e14
[0] count for roc[08] = 1
          [ -2][ 0] 0x000000e2
  CCCCCCCCCC: 1
          [ -1][ 0] 0xe10a0106
  CCCCCCCCCC: 0
          [ 0][  0] 0x00000000
  [0] count for roc[08] = 0
          [ 1][ 0] 0x01013cf3
  CCCCCCCCCC: 0
          [ 2][ 0] 0x00006cbe
  [0] flush input streams
          [ 3][ 0] 0x6cbe0003
  CCCCCCCCCC: 3
          [ 4][ 0] 0x00006cbf
  [0] count for roc[09] = 3
          [ 5][ 0] 0x01010003
  CCCCCCCCCC: 3
          [ 6][ 0] 0xfd400000
  CCCCCCCCCC: 1
          [ 7][ 0] 0x8d407e0e
[0] count for roc[09] = 1
          [ 8][ 0] 0xff116cc0
CCCCCCCCCC: 1
          [ 9][ 0] 0x85420003
  CCCCCCCCCC: 0
  End Of Previous stuff
[0] count for roc[09] = 0
  coda_roc is exiting, clear dma memory
  CCCCCCCCCC: 0
  bb_delete1: gefVmeFreeDmaBuf[ 0] has been called
[0] flush input streams
  bb_delete1: gefVmeFreeDmaBuf[ 1] has been called
  CCCCCCCCCC: 3
  bb_delete1: gefVmeFreeDmaBuf[ 2] has been called
[0] count for roc[10] = 3
  bb_delete1: gefVmeFreeDmaBuf[ 3] has been called
  CCCCCCCCCC: 3
  bb_delete1: gefVmeFreeDmaBuf[ 4] has been called
  CCCCCCCCCC: 1
  bb_delete1: gefVmeFreeDmaBuf[ 5] has been called
[0] count for roc[10] = 1
  bb_delete1: gefVmeFreeDmaBuf[ 6] has been called
  CCCCCCCCCC: 1
  bb_delete1: gefVmeFreeDmaBuf[ 7] has been called
  CCCCCCCCCC: 0
  bb_delete1: gefVmeFreeDmaBuf[ 8] has been called
  [0] count for roc[10] = 0
  bb_delete1: gefVmeFreeDmaBuf[ 9] has been called
  CCCCCCCCCC: 0
  bb_delete1: gefVmeFreeDmaBuf[10] has been called
  [0] flush input streams
  bb_delete1: gefVmeFreeDmaBuf[11] has been called
CCCCCCCCCC: 2
  bb_delete1: gefVmeFreeDmaBuf[12] has been called
  [0] count for roc[11] = 2
  bb_delete1: gefVmeFreeDmaBuf[13] has been called
  CCCCCCCCCC: 2
  bb_delete1: gefVmeFreeDmaBuf[14] has been called
CCCCCCCCCC: 0
  bb_delete1: gefVmeFreeDmaBuf[15] has been called
  [0] count for roc[11] = 0
  clasrun@pcal0:clasrun>
  CCCCCCCCCC: 0
 
  [0] flush input streams
roc_reboot did not helped. Flashed firmwares in all boards (TI, SD, FADC250s), roc_reboot again - now it works. Will watch it ...
  CCCCCCCCCC: 2
 
  [0] count for roc[12] = 2
Happened again. Replaced TI. Took 300M events, happened again:
CCCCCCCCCC: 2
 
  CCCCCCCCCC: 0
  ..........................
  [0] count for roc[12] = 0
  net_thread: waiting=      0   sending=      1 microsec per event (nev=120)
  CCCCCCCCCC: 0
  TI UNKNOWN data: [162] 0x1519028c
  [0] flush input streams
 
  CCCCCCCCCC: 2
  Previous stuff
  [0] count for roc[13] = 2
          [142][162] 0x01010003
CCCCCCCCCC: 2
          [143][162] 0x1518991c
  CCCCCCCCCC: 0
          [144][162] 0xed570324
  [0] count for roc[13] = 0
          [145][162] 0x0000028b
  CCCCCCCCCC: 0
          [146][162] 0x01010003
  [0] flush input streams
          [147][162] 0x1518991d
  CCCCCCCCCC: 0
          [148][162] 0xed5723b1
  [0] count for roc[14] = 0
          [149][162] 0x0000028b
  CCCCCCCCCC: 0
          [150][162] 0x01010003
[0] flush input streams
          [151][162] 0x1518991e
  CCCCCCCCCC: 0
          [152][162] 0xed57443e
[0] count for roc[15] = 0
          [153][162] 0x0000028b
  CCCCCCCCCC: 0
          [154][162] 0x01010003
[0] flush input streams
          [155][162] 0x1518991f
  CCCCCCCCCC: 0
          [156][162] 0xed5764da
  [0] count for roc[16] = 0
          [157][162] 0x0000028b
  CCCCCCCCCC: 0
          [158][162] 0x01010003
[0] flush input streams
          [159][162] 0x15189920
  CCCCCCCCCC: 0
          [160][162] 0x00009bab
[0] count for roc[17] = 0
          [161][162] 0x01019815
  CCCCCCCCCC: 0
          [162][162] 0x1519028c
[0] flush input streams
          [163][162] 0x01010003
  CCCCCCCCCC: 0
          [164][162] 0x15199bac
  [0] count for roc[18] = 0
          [165][162] 0x0e0db894
  CCCCCCCCCC: 0
          [166][162] 0x0000028c
[0] flush input streams
          [167][162] 0x01010003
  CCCCCCCCCC: 0
          [168][162] 0x15199bad
[0] count for roc[19] = 0
          [169][162] 0x0e0ed921
  CCCCCCCCCC: 0
          [170][162] 0x0000028c
[0] flush input streams
          [171][162] 0x01010003
  CCCCCCCCCC: 0
  End Of Previous stuff
[0] count for roc[20] = 0
 
CCCCCCCCCC: 0
Notice that shortly before that error following condition occur:
[0] flush input streams
 
CCCCCCCCCC: 0
............
  [0] count for roc[21] = 0
proc_thread: waiting=     0 processing=   975 microsec per event (nev=30)
  CCCCCCCCCC: 0
proc_thread: waiting=     0 processing=   983 microsec per event (nev=30)
  [0] flush input streams
proc_thread: waiting=     0 processing=   995 microsec per event (nev=30)
CCCCCCCCCC: 0
  bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[22] = 0
  bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
CCCCCCCCCC: 0
timer:    763 microsec (min=   722 max= 14613 rms**2=6321261)
[0] flush input streams
  bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
CCCCCCCCCC: 0
  bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
[0] count for roc[23] = 0
  bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
  bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[24] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[25] = 0
  bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[26] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[27] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[28] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
  bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[29] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[30] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] flush input streams
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] count for roc[31] = 0
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
CCCCCCCCCC: 0
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
[0] ============= Build threads cleaned
net_thread:  waiting=     0    sending=   2001 microsec per event (nev=1080)
[0] ============= Build threads cleaned
net_thread:  waiting=     0    sending=     2 microsec per event (nev=120)
[0] ============= Build threads cleaned
  net_thread:  waiting=     0    sending=     2 microsec per event (nev=1080)
  [0] ============= Build threads cleaned
................
  [0] ============= Build threads cleaned
 
  [0] build thread exiting: 1 1
 
  [0] The implementation has detected that the value  specified by threa
=== CLAS era ===
  d does not refer to a joinable thread.
 
  wait: coda request >exit< in progress
== Sergey B. 10-dec-2011: nothing bad happened, security scan got through, need to block it better ==
polling_routine ================!!!!!!!!!!!!!!!!!!=====
 
cancel building threads
.....
  =c=============================================
  FFFFFFFFFFFFFFFFFF
  =c=============================================
  WARN: ignore request from 129.57.71.28
  =c=============================================
  CODAtcpServer: start work thread
  =c====
  befor: socket=29 address>129.57.71.28< port=59401
  >>>>> close link from >dc1< link=0x081c2af0
  WARN: ignore request from 129.57.71.28
  debCloseLink: theLink=0x081c2af0 -> closing
  WARN: ignore request from 129.57.71.28
  debCloseLink: link is down
  WARN: ignore request from 129.57.71.28
  debCloseLink: free memory
  WARN: ignore request from 129.57.71.28
  debCloseLink: done.
  WARN: ignore request from 129.57.71.28
  =c====
  WARN: ignore request from 129.57.71.28
>>>>> close link from >dc2< link=0x081c4a90
  WARN: ignore request from 129.57.71.28
  debCloseLink: theLink=0x081c4a90 -> closing
  WARN: ignore request from 129.57.71.28
  debCloseLink: link is down
  WARN: ignore request from 129.57.71.28
  debCloseLink: free memory
  WARN: ignore request from 129.57.71.28
  debCloseLink: done.
  WARN: ignore request from 129.57.71.28
  =c====
  WARN: ignore request from 129.57.71.28
  >>>>> close link from >dc3< link=0x081c4b28
  WARN: ignore request from 129.57.71.28
  debCloseLink: theLink=0x081c4b28 -> closing
  WARN: ignore request from 129.57.71.28
  debCloseLink: link is down
  WARN: ignore request from 129.57.71.28
debCloseLink: free memory
  WARN: ignore request from 129.57.71.28
  debCloseLink: done.
  WARN: ignore request from 129.57.71.28
  =c====
  WARN: ignore request from 129.57.71.28
>>>>> close link from >dc4< link=0x081c14f8
  WARN: ignore request from 129.57.71.28
  debCloseLink: theLink=0x081c14f8 -> closing
  WARN: ignore request from 129.57.71.28
  debCloseLink: link is down
  accept: Too many open files
  debCloseLink: free memory
  .....
  debCloseLink: done.
 
  =c====
 
  >>>>> close link from >dc5< link=0x081c1590
== Sergey B. 23-nov-2011: EB1 hung, probably because if security scan ==
  debCloseLink: theLink=0x081c1590 -> closing
 
  debCloseLink: link is down
  .....
  debCloseLink: free memory
  WARN: ignore request from 129.57.71.28
debCloseLink: done.
  WARN: ignore request from 129.57.71.28
=c====
  WARN: ignore request from 129.57.71.28
>>>>> close link from >dc6< link=0x081c1628
  tcl7.4(init.tcl): invalid command name "GET"
  debCloseLink: theLink=0x081c1628 -> closing
  clasprod::EB1> WARN: ignore request from 129.57.71.28
  debCloseLink: link is down
  CODAtcpServer: start work thread
  debCloseLink: free memory
  befor: socket=6 address>129.57.71.28< port=47514
  debCloseLink: done.
  WARN: ignore request from 129.57.71.28
  =c====
  WARN: ignore request from 129.57.71.28
  >>>>> close link from >dc8< link=0x081c16c0
  <HUNG>
  debCloseLink: theLink=0x081c16c0 -> closing
 
  debCloseLink: link is down
 
  debCloseLink: free memory
== Sergey B. 7-nov-2011: dc12, error message from from rol.h (10kHz pulser) ==
  debCloseLink: done.
 
  =c====
  codaUpdateStatus: updating request done
  >>>>> close link from >cc1< link=0x081c1a08
active, events so far 0
  debCloseLink: theLink=0x081c1a08 -> closing
wait: coda request >go< in progress
  debCloseLink: link is down
  0x3e55cd0 (twork0047): INIT_NAME: INFO: rolp->daproc = 5
debCloseLink: free memory
  0x3e55cd0 (twork0047): INFO: User Go ...
  debCloseLink: done.
  codaExecute done
  =c====
  SEND_BUFFER_ROC 4 (0x017f4050 0) (conditions 1 13)
  >>>>> close link from >sc1< link=0x081c1aa0
  FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
debCloseLink: theLink=0x081c1aa0 -> closing
  SSS1 4 (0x017f4050 11)
  debCloseLink: link is down
  SSS2 4 (0x017f4050 11)
  debCloseLink: free memory
  SEND_BUFFER_ROC 4 (0x016bd8e0 0) (conditions 1 13)
  debCloseLink: done.
  FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
=c====
  SSS1 4 (0x016bd8e0 11)
  >>>>> close link from >ec1< link=0x081c1b38
  SSS2 4 (0x016bd8e0 11)
debCloseLink: theLink=0x081c1b38 -> closing
  SEND_BUFFER_ROC 4 (0x01587170 0) (conditions 1 13)
  debCloseLink: link is down
  FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
  debCloseLink: free memory
  SSS1 4 (0x01587170 11)
  debCloseLink: done.
  SSS2 4 (0x01587170 11)
=c====
  SEND_BUFFER_ROC 4 (0x01450a00 0) (conditions 1 13)
  >>>>> close link from >ec2< link=0x081c1bd0
  FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
debCloseLink: theLink=0x081c1bd0 -> closing
  SSS1 4 (0x01450a00 11)
  debCloseLink: link is down
  SSS2 4 (0x01450a00 11)
  debCloseLink: free memory
  SEND_BUFFER_ROC 4 (0x0131a290 0) (conditions 1 13)
  debCloseLink: done.
  FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
  =c====
  SSS1 4 (0x0131a290 11)
  >>>>> close link from >scaler2< link=0x681d95c8
  SSS2 4 (0x0131a290 11)
  debCloseLink: theLink=0x681d95c8 -> closing
  bb_write: no bufs, back end too slow ..)
  debCloseLink: link is down
  bb_write: no bufs, back end too slow ..)
  debCloseLink: free memory
  interrupt: INFO from bb_write: NO BUFS1
  debCloseLink: done.
  bb_write: no bufs, back end too slow ..)
  =c====
  bb_write: no bufs, back end too slow ..)
  >>>>> close link from >scaler3< link=0x681d9660
  bb_write: no bufs, back end too slow ..)
  debCloseLink: theLink=0x681d9660 -> closing
  bb_write: no bufs, back end too slow ..)
  debCloseLink: link is down
  bb_write: no bufs, back end too slow ..)
  debCloseLink: free memory
  bb_write: no bufs, back end too slow ..)
  debCloseLink: done.
  setHeartError: 0 >sys 0, mask 4<
  =c====
  WARN: HeartBeat[0]: heartbeat=21263(21263) heartmask=4
  >>>>> close link from >scaler4< link=0x681d96f8
  bb_write: no bufs, back end too slow ..)
debCloseLink: theLink=0x681d96f8 -> closing
  bb_write: no bufs, back end too slow ..)
  debCloseLink: link is down
  bb_write: no bufs, back end too slow ..)
  debCloseLink: free memory
bb_write: no bufs, back end too slow ..)
  debCloseLink: done.
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
  =c====
  UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
  >>>>> close link from >ec4< link=0x00000001
bb_write: no bufs, back end too slow ..)
  debCloseLink: theLink=0x00000001 -> closing
bb_write: no bufs, back end too slow ..)
  Segmentation fault
bb_write: no bufs, back end too slow ..)
  clondaq1:coda_eb>  
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
...........
 
Click end - ended fine (probably network was stuck ?? or proc ???).
 
 
== Sergey B. 8-jun-2011: test setup croctest2: error on croctest2 after 99M events, DMAing from single v1720 (NOTE: first test of ROL1 writing directly to the big buffer) ==
proc_thread: waiting=    846 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    192 processing=    460 microsec per event (nev=189)
rols_thread: waiting=    194 processing=    457 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
proc_thread: waiting=    846 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    185 processing=    469 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    200 processing=    449 microsec per event (nev=188)
proc_thread: waiting=    847 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
net_thread:  waiting=    720    sending=      0 microsec per event (nev=2218)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
proc_thread: waiting=    847 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    205 processing=    450 microsec per event (nev=188)
machine check
Exception next instruction address: 0x0946e298
Machine Status Register: 0x0010b030
Condition Register: 0x80000088
Task: 0x946c670 "ROLS_LOOP"
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=54077(54077) heartmask=4
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
-> tt
1668c8 vxTaskEntry    +68 : rols_loop ()
d2e5290 rols_loop      +b4 : output_proc_network ()
d2e5574 output_proc_network+298: vmenrol1__init ()
946ccb8 vmenrol1__init +19c: 946cafc ()
946cb0c end            +91cb68c: 946ca1c ()
946cabc end            +91cb63c: 946c890 ()
946c920 end            +91cb4a0: 946e298 ()
value = 0 = 0x0
->
 
== Sergey B. 10-feb-2011: test setup croctest2: coda_net on pmc dies ==
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6559)
net_thread:  waiting=    131    sending=      2 microsec per event (nev=12708)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
proc_thread: waiting=    139 processing=      3 microsec per event (nev=6474)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting=    129 processing=      3 microsec per event (nev=6452)
net_thread:  waiting=    133    sending=      2 microsec per event (nev=12598)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6553)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=16)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=12)
net_thread:  waiting=    131    sending=      2 microsec per event (nev=12714)
proc_thread: waiting=    139 processing=      3 microsec per event (nev=6489)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=8)
proc_thread: waiting=    128 processing=      3 microsec per event (nev=6478)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
net_thread:  waiting=    134    sending=      2 microsec per event (nev=12377)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=0)
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6554)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=3)
proc_thread: waiting=    139 processing=      3 microsec per event (nev=6470)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting=    128 processing=      3 microsec per event (nev=6516)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
ERROR2: LINK_sized_write() returns errno=32
Broken pipe
ERROR: net_thread failed (in LINK_sized_write).
->
-> i
  NAME        ENTRY      TID    PRI  STATUS      PC      SP    ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask  excTask      1effe6c0  0 PEND        14e508 1effe5a0  3d0001    0
tLogTask  logTask      1effbb20  0 PEND        14e508 1effba10      0    0
tShell    shell        1eb8cb70  1 READY        145290 1eb8c750      d    0
tRlogind  rlogind      1ebb3850  2 PEND        13f934 1ebb33f0      0    0
tWdbTask  wdbTask      1ebaa530  3 PEND        13f934 1ebaa400      0    0
tNetTask  netTask      1ed0f280  50 PEND        13f934 1ed0f190      0    0
tPortmapd  portmapd    1ebb1690  54 PEND        13f934 1ebb1430  3d0002    0
tFtpdTask  b6064        1ebaebc0  55 PEND        13f934 1ebaea10      0    0
coda_proc  proc_thread  1aacca00 115 DELAY        144bb4 1aacc900      0    10
coda_pmc  coda_pmc    1e3f9720 200 DELAY        144bb4 1e3f9480      0    5
value = 0 = 0x0
-> printErrno 32
S_errno_EPIPE
value = 14 = 0xe
-> tt
Sorry, traces of my own stack begin at tt ().
value = -1 = 0xffffffff
-> memShow
  invalid block at 0x1ec60000 deleted
value = -1 = 0xffffffff
-> memShow
status    bytes    blocks  avg block  max block
------ ---------- --------- ---------- ----------
current
  free  421348656        56    7524083  411333840
  alloc  79807344      1986      40184          -
cumulative
  alloc  87490320      7392      11835          -
value = 0 = 0x0
-> usrNetStackSysPoolStatus
value = 0 = 0x0
-> usrNetStackSysPoolStatus("CODAtcpServer",0)
value = 0 = 0x0
-> usrNetStackDataPoolStatus("CODAtcpServer",0)
value = 454823200 = 0x1b1c0d20
-> usrNetStackDataPoolStatus("bla",0)
value = 454823168 = 0x1b1c0d00
-> usrNetStackDataPoolStatus("bla",0)
value = 454823040 = 0x1b1c0c80
-> memShow
status    bytes    blocks  avg block  max block
------ ---------- --------- ---------- ----------
current
  free  421348528        55    7660882  411333840
  alloc  79807472      1990      40104          -
cumulative
  alloc  87496528      7401      11822          -
value = 0 = 0x0
->
 
EB looks fine, message was:
-- Got all fragments of go
BOS_encode_spec reached
BOS_encode_spec done: len=116
FFFFFFFFFFFFFFFFFF
PUT: wait while icb=8 == cbp->read=8 -------
clastest::EB5>
 
Tried to reset-download-prestart and got error on pmc at Download:
-> i
  NAME        ENTRY      TID    PRI  STATUS      PC      SP    ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask  excTask      1effe6c0  0 PEND        14e508 1effe5a0  3d0001    0
tLogTask  logTask      1effbb20  0 PEND        14e508 1effba10      0    0
tShell    shell        1eb8cb70  1 READY        145290 1eb8c750      d    0
tRlogind  rlogind      1ebb3850  2 PEND        13f934 1ebb33f0      0    0
tWdbTask  wdbTask      1ebaa530  3 PEND        13f934 1ebaa400      0    0
tNetTask  netTask      1ed0f280  50 PEND        13f934 1ed0f190      0    0
tPortmapd  portmapd    1ebb1690  54 PEND        13f934 1ebb1430  3d0002    0
tFtpdTask  b6064        1ebaebc0  55 PEND        13f934 1ebaea10      0    0
coda_pmc  coda_pmc    1e3f9720 200 DELAY        144bb4 1e3f9480  30065    1
value = 0 = 0x0
-> Download command received
-> bignet at 0x0d3a6e68
<- bignet at 0xcd3a6e68
bignet at 0xcd3a6e68, bignet.gbigBuffer at 0xcd3a6e78 -> 0xcd3a6e78
proc_on_pmc=1, net_on_pmc=1
rocID=0
ObjInitName >_rol2_tt_testsetup__init<
data access
Exception current instruction address: 0x000d2090
Machine Status Register: 0x00003030
Data Access Register: 0x3d40001c
Condition Register: 0x40002083
Data storage interrupt Register: 0x40000000
Task: 0x1ed0f280 "tNetTask"
-> tt
14cad8 vxTaskEntry    +68 : netTask ()
c3e88 netTask        +70 : 209c0 ()
20a14 mgiEndLoad    +cc8: 20a88 ()
20bd8 mgiEndLoad    +e8c: muxReceive ()
bc638 muxReceive    +13c: b002c ()
b02c4 random        +358: do_protocol_with_type ()
9c920 do_protocol_with_type+2c : ipintr (1ec60e04)
9fb28 ipintr        +c8c: dbbb4 ()
dbbb4 tcp_input      +2968: tcp_output ()
dca24 tcp_output    +5c8: mHdrClGet ()
aa940 mHdrClGet      +8c : d1eb4 ()
value = 0 = 0x0
->
-> i
  NAME        ENTRY      TID    PRI  STATUS      PC      SP    ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask  excTask      1effe6c0  0 PEND        14e508 1effe5a0  3d0001    0
tLogTask  logTask      1effbb20  0 PEND        14e508 1effba10      0    0
tShell    shell        1eb8cb70  1 READY        145290 1eb8c750      d    0
tRlogind  rlogind      1ebb3850  2 PEND        13f934 1ebb33f0      0    0
tWdbTask  wdbTask      1ebaa530  3 PEND        13f934 1ebaa400      0    0
tNetTask  netTask      1ed0f280  50 SUSPEND      d2090 1ed0efd0      0    0
tPortmapd  portmapd    1ebb1690  54 PEND        13f934 1ebb1430  3d0002    0
tFtpdTask  b6064        1ebaebc0  55 PEND        13f934 1ebaea10      0    0
coda_pmc  coda_pmc    1e3f9720 200 PEND        1415a0 1e3f8ab0  3d0004    0
value = 0 = 0x0
->
 
 
 
== Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!??? ==
clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod  -m eb -v
et_nodename >eb<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station LEVEL3
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_station_create_at, created station ET2ER
unique name >et_start_eb_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_clasprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
ipc_init called at Fri Apr 16 11:45:20 2010
et_start: pthread_create(0x0000000e,...) done
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et INFO: et_sys_heartmonitor, kill bad process (2,22078)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments
 
 
 
clondaq2:clasrun> cd et2_start
clondaq2:et2_start>
clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod
-m er -v
et_nodename >er<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station TAPE
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur
rent/coda/SunOS_i86pc/lib/libet_user.so
et INFO: et_conductor 2, loaded function et2et_function
et INFO: et_station_create_at, created station ET2ET10
unique name >et_start_er_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R
Tserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve
r
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>
RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established
.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl
asprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab
.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
 
ipc_init called at Fri Apr 16 11:45:20 2010
 
et_start: pthread_create(0x0000000e,...) done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_netserver: magic numbers do match, accept ET client
et INFO: et_station_attach, done
et INFO: et_restore_events, no events found belonging to attachment 4
et INFO: et_station_detach, done
et WARN: et_command_loop: remote client connection broken
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et ERROR: et_netserver: ET server being probed by non-ET client or rea
d failure
et INFO: et_sys_heartmonitor, kill bad process (1,8898)
et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1
et INFO: et_flush_events, read 4000 events
et INFO: et_flush_events, wrote 4000 events
et INFO: et_station_detach, make station "TAPE" (cec1d440) idle
et INFO: et_restore_events, no events found belonging to attachment 3
et INFO: et_station_detach, done
et INFO: et_sys_heartmonitor, cleanup process 1
et INFO: set_fix_nprocs, change # of ET processes from 1 to 1
et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments
et INFO: set_fix_natts, station ET2ET10 has 1 attachments
et INFO: set_fix_natts, # total attachments 3 -> 3
et INFO: set_fix_natts, proc 0 has 1 attachments
 
 
 
 
clondaq1:dac>
clondaq1:dac>
clondaq1:dac>
clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 22276) (lang C) (local) (period = 5 sec)
 
  STATIC INFO - maximum of:
    events(4000), event size(200000), temps(4000)
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,
 
  DYNAMIC INFO - currently there are:
    processes(4), attachments(4), temps(0)
    stations(3), hearbeat(1)
 
  STATIONS:
    "GRAND_CENTRAL" (id = 0)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(2,3,)
        input  list: cnt =  4000, events in  = 0
        output list: cnt =    0, events out = 0
 
    "LEVEL3" (id = 1)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(1),  ids(1,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
    "ET2ER" (id = 2)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(1),  ids(0,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
  LOCAL USERS:
    process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
    process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
    process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)


DC7:
  ATTACHMENTS:
    att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
    proc(0), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
    proc(1), blocked(YES)
      events:  make(0), get(0), put(0), dump(0)
    att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
    proc(2), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
    proc(3), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
 
  EVENTS OWNED BY:
    system (4000),  att0 (0),  att1 (0),  att2 (0),  att3 (0),
 
  EVENT RATE of GC = 0 events/sec
 
  CREATING STATIONS: 
  IDLE STATIONS:     
  STATION CHAIN:      GRAND_CENTRAL, LEVEL3, ET2ER,
  LOCKED MUTEXES:   
 
*****************************************
 
  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 22276) (lang C) (local) (period = 5 sec)
 
  STATIC INFO - maximum of:
    events(4000), event size(200000), temps(4000)
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,
 
  DYNAMIC INFO - currently there are:
    processes(4), attachments(4), temps(0)
    stations(3), hearbeat(1)
 
  STATIONS:
    "GRAND_CENTRAL" (id = 0)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(2,3,)
        input  list: cnt =  4000, events in  = 0
        output list: cnt =    0, events out = 0
 
    "LEVEL3" (id = 1)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(1),  ids(1,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
    "ET2ER" (id = 2)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(1),  ids(0,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
  LOCAL USERS:
    process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
    process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
    process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)
 
  ATTACHMENTS:
    att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
    proc(0), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
    proc(1), blocked(YES)
      events:  make(0), get(0), put(0), dump(0)
    att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
    proc(2), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
    proc(3), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
 
  EVENTS OWNED BY:
    system (4000),  att0 (0),  att1 (0),  att2 (0),  att3 (0),
 
  EVENT RATE of GC = 0 events/sec
 
  CREATING STATIONS: 
  IDLE STATIONS:     
  STATION CHAIN:      GRAND_CENTRAL, LEVEL3, ET2ER,
  LOCKED MUTEXES:   
 
*****************************************
 
ET SYSTEM is DEAD!
*****************************************
 
^C
clondaq1:dac>
 
 
 
 
 
 
clondaq2:stage_in>
clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod
LOCALITY = 1
  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 16127) (lang C) (local) (period = 5 sec)
 
  STATIC INFO - maximum of:
    events(4000), event size(200000), temps(4000)
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,
 
  DYNAMIC INFO - currently there are:
    processes(3), attachments(3), temps(0)
    stations(3), hearbeat(1)
 
  STATIONS:
    "GRAND_CENTRAL" (id = 0)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(0),  ids()
        input  list: cnt =  4000, events in  = 0
        output list: cnt =    0, events out = 0
 
    "TAPE" (id = 1)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(1,2,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
    "ET2ET10" (id = 2)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
        lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so,  function = et2et_function, class =
      dynamic info
        attachments: total#(1),  ids(0,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
  LOCAL USERS:
    process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
    process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)
 
  ATTACHMENTS:
    att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
    proc(0), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
    proc(1), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
    proc(2), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
 
  EVENTS OWNED BY:
    system (4000),  att0 (0),  att1 (0),  att2 (0),
 
  EVENT RATE of GC = 0 events/sec
 
  CREATING STATIONS: 
  IDLE STATIONS:     
  STATION CHAIN:      GRAND_CENTRAL, TAPE, ET2ET10,
  LOCKED MUTEXES:   
 
*****************************************
 
  ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
              (tcp port 0) (udp port 11111) (multicast port 11111)
              (pid 16127) (lang C) (local) (period = 5 sec)
 
  STATIC INFO - maximum of:
    events(4000), event size(200000), temps(4000)
    stations(10), attaches(20), procs(10), concurrency added(16)
    network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,
 
  DYNAMIC INFO - currently there are:
    processes(3), attachments(3), temps(0)
    stations(3), hearbeat(1)
 
  STATIONS:
    "GRAND_CENTRAL" (id = 0)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(0),  ids()
        input  list: cnt =  4000, events in  = 0
        output list: cnt =    0, events out = 0
 
    "TAPE" (id = 1)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
        restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
      dynamic info
        attachments: total#(2),  ids(1,2,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
    "ET2ET10" (id = 2)
      static info
        status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
        restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
        lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so,  function = et2et_function, class =
      dynamic info
        attachments: total#(1),  ids(0,)
        input  list: cnt =    0, events in  = 0, events try = 0
        output list: cnt =    0, events out = 0
 
  LOCAL USERS:
    process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
    process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
    process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)
 
  ATTACHMENTS:
    att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
    proc(0), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
    proc(1), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
    att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
    proc(2), blocked(NO)
      events:  make(0), get(0), put(0), dump(0)
 
  EVENTS OWNED BY:
    system (4000),  att0 (0),  att1 (0),  att2 (0),
 
  EVENT RATE of GC = 0 events/sec
 
  CREATING STATIONS: 
  IDLE STATIONS:     
  STATION CHAIN:      GRAND_CENTRAL, TAPE, ET2ET10,
  LOCKED MUTEXES:   
 
*****************************************
 
ET SYSTEM is DEAD!
*****************************************
 
^C
clondaq2:stage_in>


codaExecute done
wait: coda request >download prod< in progress
codaExecute reached, message >prestart<, len=8
codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410567520
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc7<
LINK_constructor_C: set name to >dc7->EB1<
3121: dbsock=410567520
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda req<<<
uest >fields [pr2estart] >>>< in prodowngress
<<<
fields [3] >>>37683<<<
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x18772230 (twork0028INFO: Entering Us): er
Prestart 2
INIT_NAME: rolp->daproc =
trans_rol2:2
adr(nddl)=1db60750 NWBOS=16384
uthbook1: WARN id=1047 already exist - will be replaced
uthbook1: free histogram buffer ... done.
RAW=0  PROFILE=1  NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x18772230INFO: User Prest (art 2 executed
twork0028): CTRIGRSS: set handler and done,
code=1
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4130) returns 490106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a580) returns 490106296
0x18772230 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x18772230 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc
0x1882d864 0x00000001 0x1882b0c8
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1  = FFFFF000
Fastbus Status Register2  = FFFF0000
Fastbus Last Primary Addr = 00000019
spds_mask1: 0xfdfbf0
0x18772230_tdc1877 ===> (4twork0028wait: co)
:  Use 32-bit Ram List
da request >firsprestartt< in progre csr0=ss
100 (1100) csr18=1364 csr1=6400001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0=100 (900) csr18=13
bignetptr=0x641d366fe0 csr1= offset=0x00bigp
rocptr=0x000001d366db80 offset=0x6400000000
00001b
bigproc at 0x
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =
==>1d366dc818 -> 0x1d366dc8
proc_thread reacheUDP_cancel: cancel >inf:d
c7 sys 0, mask 1d
<
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached
<
wait: cproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot#  1872  1875  1881(2)  1877(2)  1877S(2)
[ 0]      0    0      0 0bignetptr=0x     
1d366fe00 offset=0x0000000 0
0      0 0
[ 1]      0    0      0 0      0 0      0
0
[ 2]      0    0      0 0      0 0      0
0
[ 3]      0    0      0 0      0 0      0
0
[ 4]      0    0      0 0      1 0      0
0
[ 5]      0    0      0 0      1 0      0
0
[ 6]      0    0      0 0      1 0      0
0
[ 7]      0    0      0 0      1 0      0
0
[ 8]      0    0      0 0      1 0      0
0
[ 9]      0    0      0 0      1 2      0
0
[10]      wait: coda reques0t >    prestart
0<      in progress
0 0      0 0      0 0
[11]      0    0      0 0      1 0      0
0
[12]      0    0      0 0      1 0      0
0
[13]      0    0      0 0      1 0      0
0
[14]      0    0      0 0      1 0      0
0
[15]      0    0      0 0      1 0      0
0
[16]      0    0      0 0      1 2      0
0
[17]      0    0      0 0      0 0      0
0
[18]      0    0      0 0      1 0      0
0
[19]      0    0      0 0      1 0      0
0
[20] bignet at 0x    1d366fe00, bignet.gbig
Buffer at 0x    1d366ff00 -> 0x1d366ff0
      0 0      1 0      0 0
[21]      0    0      0 0      1 0      0
0
[22]      0    0      0 0      1 0      0
0
[23]      0    0      0 0      1 2      0
0
[24]      0    0      0 0      0 0      0
0
[25]      0    0      0 0      0 0      0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC07<
0x18772230 (twork0028111): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <
<<<<<<<<<<<<<<<<
s: >UPDATE process SET state='paused' WHERE
name='dc7'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_cancel: cancel >sta:dc7 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
->
->


DC8:
== Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem ==


  codaExecute: 'prestart' transition
  wait: coda request >end< in progress
roc_prestart reached
  0x916e80 (twork0028): TS GO bit cleared
  partReInitAll() reached ?
  0x916e80 (twork0028): TS csr:  0xfc070000
  partReInitAll() reached !
  0x916e80 (twork0028): TS csr2: 0xffff1c1f
  partReInitAll() done
  interrupt: TRIG ERROR: no pool buffer available
  rocCleanup reached
  0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0
  partReInitAll() reached ?
  0x916e80 (twork0028): INFO: User End Executed
wait: coda request >prestart< in progress
  codaUpdateStatus: dbConnecting ..
  partReInitAll() reached !
  codaUpdateStatus: dbConnect done
  partReInitAll() done
  codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<
  315: dbsock=410505056
  codaUpdateStatus: dbDisconnecting ..
  >>> prestarting, run 62496, type 47
codaUpdateStatus: dbDisconnect done
  rocMask=0x76fefffe
 
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
 
  WHERE name='dc8'<
 
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
 
  WHERE name='dc8'<
== Sergey B. 14-apr-2010: message from runcontrol during cancel-reset ==
  !!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
  Get all components failed: Lost connection to MySQL server during query
  WHERE name='dc8'<
 
  ++++> output to network
== Sergey B. 6-apr-2010: EB waiting for following ROCs ==
  !!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
 
  3
  [0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
  7 9 10 11 17 20 21 22 25 26 28 29 30
  !!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
 
  !!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
Files produced:
  rocCloselink reached
 
  rocOpenLinks reached
  clon10:ccscans> ll | grep Apr
  LINK_constructor_C: set host to >dc8<
  drwxrwxrwx  2 clasrun  onliners    4096 Apr  2 16:23 ./
  LINK_constructor_C: set name to >dc8->EB1<
  drwxr-xr-x 163 clasrun  100        73728 Apr  6  2010 ../
  3121: dbsock=410505056
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_cc1*
  name selected
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:09 good_clastrig2*
  nrow=1
  -rwxrwxrwx  1 boiarino clas          34 Apr  2 16:25 good_croctest10*
  fields [0] >>>TCP<<<
-rwxrwxrwx  1 boiarino clas          34 Apr  2 16:25 good_croctest2*
  fields [1] >>>clondaq1-daq1wait: coda re<<<
  -rwxrwxrwx  1 boiarino clas          34 Apr  2 16:21 good_croctest3*
  quest >fields [p2restart] >>>< in prupogress
  -rwxrwxrwx  1 baturin  clas          34 Apr  2 13:26 good_ctoftest1*
  <<<
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc1*
  fields [3] >>>37772<<<
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc10*
  parsing results: type=>TCP< host=>clondaq1-d
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc11*
  aq1< state=>up< port=>37772< -> 37772
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc2*
  LINK_establish: socket # 17
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc3*
  LINK_establish: socket buffer size is 48000(
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc4*
  0x0000bb80) bytes
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc5*
  LINK_establish: keepAlive is 8
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc6*
  LINK_establish: socket 17 is ready: host 129
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc7*
  .57.68.22 port 37772
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_dc8*
  DB command >UPDATE links SET state='up' WHER
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_dc9*
  E name='dc8->EB1'<
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_ec1*
  UPDATE process success
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_ec2*
  Spawn proc/net threads
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_ec3*
  Spawn proc/net threads
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_ec4*
  Spawn proc/net threads
-rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_polar*
  0x187ef280INFO: Entering U (ser Prestart 2
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_sc1*
  twork0028
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_sc2*
  trans_rol2:): adr(nddl)=INIT_NAME: rolp->da
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:08 good_scaler1*
  proc = 1d2b60750
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_scaler2*
   NWBOS=16384
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_scaler3*
  uthbook1: WARN id=1048 already exist - will
  -rwxrwxrwx  1 clasrun  onliners      34 Apr  6 20:26 good_scaler4*
  be replaced
  -rwxrwxrwx  1 clasrun  onliners      34 Apr 6 20:08 good_tage*
  uthbook1: free histogram buffer ... done.
  -rwxrwxrwx  1 clasrun onliners      34 Apr 6 20:08 good_tage2*
  RAW=0  PROFILE=1  NOTRANS=0
  -rwxrwxrwx   1 clasrun  onliners      34 Apr 6 20:08 good_tage3*
  MAX_EVENT_LENGTH = 65536 NWBOS = 16384
  clon10:ccscans> more good_dc7
  0x187ef280 (twork0028INFO: User Pr): estart
  accepted link from >129.57.68.80<
  2 executedCTRIGRSS: set handler and done, co
  clon10:ccscans> more good_dc8
  de=
  accepted link from >129.57.68.74<
  1
  clon10:ccscans>
  0x187ef280 (twork0028): CTRIGRSS: 0x187705ac
 
  0x187704d0 0x1876fdf0 0x00000001 0x1876fde8
On 'Reset' EB crashed, outputs:
 
  [0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
  7  9 10 11 17 20 21 22 25 26 28 29 30
 
  [0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
  7 9 10 11 17 20 21 22 25 26 28 29 30
 
  CODAtcpServer: start work thread
befor: socket=5 address>129.57.167.14< port=34468
  wait: coda request >prestart< in progress
  wait: coda request >prestart< in progress
  bignetptr=0x1d366fe0 offset=0x00000000
  codaExecute reached, message >exit<, len=4
  taskSpawn("coda_net",0x187d4110) returns 490
  codaExecute: 'exit' transition
  106848
  debShutdownBuild 1
  wait: coda request >prestart< in progress
  cb_delete roc=0
  setHeartError: 0 >sys 0, mask 1<
  cb_delete roc=1
  WARN: HeartBeat[0]: heartbeat=19424(19424) h
  cb_delete roc=2
  eartmask=1
  cb_delete roc=3
  bignetptr=0x1d366fe0 offset=0x00000000
  cb_delete roc=4
  bigprocptr=0x1d366db8 offset=0x00000000
cb_delete roc=5
  taskSpawn("coda_proc",0x1874a560) returns 49
  cb_delete roc=6
  0106296
cb_delete roc=7
  0x187ef280 (twork0028): INIT_NAME: rolp->dap
  cb_delete roc=8
  roc = 2
  cb_delete roc=9
  0x187ef280 (twork0028): INFO: Entering User
  cb_delete roc=10
  Prestart
  cb_delete roc=11
  disconnecting vector 236
  cb_delete roc=12
  0x187ef280 (twork0028): CTRIGRSA: set handle
  cb_delete roc=13
  r and done, code=1
  cb_delete roc=14
  0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c
  cb_delete roc=15
  0x1882d1e4 0x00000001 0x1882aa48
  cb_delete roc=16
  Sequencer Status Register = FFFFA000
  cb_delete roc=17
  Last Sequencer KeyAddress = FFFF0003
  cb_delete roc=18
  Fastbus Status Register1  = FFFFF000
  cb_delete roc=19
  Fastbus Status Register2  = FFFF0000
  cb_delete roc=20
  Fastbus Last Primary Addr = wait: coda req00
  cb_delete roc=21
  0000uest >19presta
  cb_delete roc=22
  rtspds_mask1: 0x< ifdfbf0n progress
  cb_delete roc=23
  0x187ef280_tdc1877 === (>twork0028): Use 32-
  cb_delete roc=24
  bit Ram List
  cb_delete roc=25
  4 first csr0=100 (1100) csr18=1364 csr1=6400
cb_delete roc=26
  001b
cb_delete roc=27
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
  cb_delete roc=28
  1364 csr1=6400001b
  cb_delete roc=29
  _tdc1877 ===>6 middle csr0=100 (1900) csr18=
  cb_delete roc=30
  1364 csr1=6400001b
  cb_delete roc=31
  _tdc1877 ===>7 middle csr0=100 (1900) csr18=
  debShutdownBuild 2
  1364 csr1=6400001b
  Stop waiting for the rocs because of force_end condition
  _tdc1877 ===>8 middle csr0=100 (1900) csr18=
  [0] ROC mask 0x76fefffe force_end 1 end_event_done 0
  1364 csr1=6400001b
  [0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!
  _tdc1877 ===>9 last csr0=100 (900) csr18=136
  [0] et_event_new() 1
  4 csr1=6400001b
  [0] EVENT_ENCODE_SPEC 2 ..
  _tdc1877 ===>11 first csr0=100 (1100) csr18=
  BOS_encode_spec reached
  1364 csr1=6400001b
  BOS_encode_spec done: len=116
  _tdc1877 ===>12 middle csr0=100 (1900) csr18
  [0] .. done
  =1364 csr1=6400001b
  [0] handle_build_cleanup
  _tdc1877 ===>13 middle csr0=100 (1900) csr18
  [0] flush ET system
  =1364 csr1=6400001b
  [0] detached from ET
  _tdc1877 ===>14 middle csr0=100 (1900) csr18
  [0] remove mutex locks and shutdown fifos
  =1364 csr1=6400001b
  [0] flush input streams
  _tdc1877 ===>15 middle csr0=100 (1900) csr18
  CCCCCCCCCC: 3
  =1364 csr1=6400001b
  [0] count for roc[00] = 3
  _tdc1877 ===>16 last csr0bignetptr=0x=1d366f
  CCCCCCCCCC: 3
  e0100 offset=0x0000000 (0
  CCCCCCCCCC: 1
  900) csr18=1364bigprocptr=0x csr1=1d366db864
  [0] count for roc[00] = 1
  00001b offset=0x
  CCCCCCCCCC: 1
  00000000
  CCCCCCCCCC: 0
  bigproc at 0x_tdc1877 ===1d366db8>, bigproc.
  [0] count for roc[00] = 0
  gbigBuffer at 0x181d3UDP_cancel: cancel >inf
  CCCCCCCCCC: 0
  :dc8 sys 0, mask 166dc8<
  [0] flush input streams
  wait: coda rUDP_cancel: cancel >inf:dc8 sys
  CCCCCCCCCC: 3
  0, mask 1 -> 0x<
  [0] count for roc[01] = 3
  1d366dc8eq
  CCCCCCCCCC: 3
  firstproc_thread reached
CCCCCCCCCC: 1
  uest >proc_thread reached
  [0] count for roc[01] = 1
  csr0=proc_thread reached
  CCCCCCCCCC: 1
  prproc_thread reached
  CCCCCCCCCC: 0
  proc_thread reached
  [0] count for roc[01] = 0
  proc_thread reached
  CCCCCCCCCC: 0
  proc_thread reached
  [0] flush input streams
proc_thread reached
  CCCCCCCCCC: 3
  proc_thread reached
  [0] count for roc[02] = 3
  proc_thread reached
  CCCCCCCCCC: 3
  proc_thread reached
  CCCCCCCCCC: 1
  proc_thread reached
  [0] count for roc[02] = 1
  100estart (< in prog1100ress
  CCCCCCCCCC: 1
  ) csr18=1364 csr1=6400001b
  CCCCCCCCCC: 0
  _tdc1877 ===>19 middle csr0=100 (1900) csr18
  [0] count for roc[02] = 0
  =1364 csr1=6400001b
  CCCCCCCCCC: 0
  _tdc1877 ===>20 middle csr0=100 (1900) csr18
  [0] flush input streams
  =1364 csr1=6400001b
  CCCCCCCCCC: 3
  _tdc1877 ===>21 middle csr0=100 (1900) csr18
  [0] count for roc[03] = 3
  =1364 csr1=6400001b
  CCCCCCCCCC: 3
  _tdc1877 ===>22 middle csr0=100 (1900) csr18
  CCCCCCCCCC: 1
  =1364 csr1=6400001b
  [0] count for roc[03] = 1
  _tdc1877 ===>23 last csr0=100 (900) csr18=13
  CCCCCCCCCC: 1
  64 csr1=6400001b
  CCCCCCCCCC: 0
DEBUG in SFIRamLoad => 0 1 0 0 0
  [0] count for roc[03] = 0
  slot#  1872 bignetptr=0x1875 181d366fe081
CCCCCCCCCC: 0
  (2) 1877( offset=0x2) 1877S(00000002)
[0] flush input streams
  0
  CCCCCCCCCC: 3
  [ 0]     0     0      0 0     0 0      0
  [0] count for roc[04] = 3
  0
  CCCCCCCCCC: 3
  [ 1]     0    0      0 0      0 0      0
  CCCCCCCCCC: 1
  0
  [0] count for roc[04] = 1
  [ 2]     0     0     0 0      0 0      0  
  CCCCCCCCCC: 1
  0
  CCCCCCCCCC: 0
  [ 3]     0    0      0 0      0 0      0
  [0] count for roc[04] = 0
  0
CCCCCCCCCC: 0
  [ 4]     0    0      0 0      1 0      0
[0] flush input streams
  0
  CCCCCCCCCC: 3
  [ 5]     0    0      0 0      1 0      0
  [0] count for roc[05] = 3
  0
CCCCCCCCCC: 3
  [ 6]     0    0      0 0      1 0      0  
  CCCCCCCCCC: 1
  0
  [0] count for roc[05] = 1
  [ 7]     0    0      0 0      1 0      0
CCCCCCCCCC: 1
  0
CCCCCCCCCC: 0
  [ 8]     0    0      0 0      1 0      0
[0] count for roc[05] = 0
  0
  CCCCCCCCCC: 0
  [ 9]      0    0      0 0      1 2      0
  [0] flush input streams
  0
  CCCCCCCCCC: 3
  [10]      wait: coda request0 >    prestart
  [0] count for roc[06] = 3
  0< i      n progress
CCCCCCCCCC: 3
  0 0      0 0      0 0
  CCCCCCCCCC: 1
[11]     0    0      0 0      1 0      0  
  [0] count for roc[06] = 1
  0
CCCCCCCCCC: 1
  [12]     0    0      0 0      1 0      0
  CCCCCCCCCC: 0
  0
  [0] count for roc[06] = 0
  [13]     0    0      0 0      1 0      0
  CCCCCCCCCC: 0
  0
  [0] flush input streams
  [14]      0    0      0 0      1 0      0
  CCCCCCCCCC: 3
  0
  [0] count for roc[07] = 3
[15]     0    0      0 0      1 0      0
  CCCCCCCCCC: 3
  0
  CCCCCCCCCC: 1
  [16]     0    0      0 0      1 2      0  
  [0] count for roc[07] = 1
  0
  CCCCCCCCCC: 1
  [17]     0    0      0 0      0 0      0
  CCCCCCCCCC: 0
  0
  [0] count for roc[07] = 0
  [18]      0    0      0 0      1 0      0
  CCCCCCCCCC: 0
  0
  [0] flush input streams
  [19]     0 bignet at 0x    1d366fe00, bigne
  CCCCCCCCCC: 3
  t.gbigBuffer at 0x      1d366ff00 -> 0x1d366
  [0] count for roc[08] = 3
  ff0
  CCCCCCCCCC: 3
  0     1 0      0 0
  CCCCCCCCCC: 1
  [20]     0    0      0 0      1 0      0
  [0] count for roc[08] = 1
  0
CCCCCCCCCC: 1
  [21]     0    0      0 0      1 0      0
  CCCCCCCCCC: 0
  0
  [0] count for roc[08] = 0
  [22]     0    0      0 0      1 0      0
  CCCCCCCCCC: 0
  0
  [0] flush input streams
[23]      0    0      0 0      1 2      0
CCCCCCCCCC: 3
  0
  [0] count for roc[09] = 3
  [24]     0    0      0 0      0 0      0
  CCCCCCCCCC: 3
0
  CCCCCCCCCC: 1
[25]     0    0      0 0      0 0      0  
  [0] count for roc[09] = 1
  0
  CCCCCCCCCC: 1
  DEBUG 0 1 0 0  
  CCCCCCCCCC: 0
  INFO: datascan = 0x00000000
[0] count for roc[09] = 0
  INFO: datascan&spds_mask1 = 0x00000000
CCCCCCCCCC: 0
  rcname >RC08<
  [0] flush input streams
  0x187ef280 (twork0028111
  CCCCCCCCCC: 3
  ): INFO: User Prestart Executed
  [0] count for roc[10] = 3
  333: rocp->primwait: coda request >efd=prest
CCCCCCCCCC: 3
  art17< in p
  CCCCCCCCCC: 1
  rogress
  [0] count for roc[10] = 1
  informEB reached
  CCCCCCCCCC: 1
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011
  CCCCCCCCCC: 0
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000
  [0] count for roc[10] = 0
  0000
  CCCCCCCCCC: 0
  codaUpdateStatus: dbConnecting ..
  [0] flush input streams
  codaUpdateStatus: dbConnect done
  CCCCCCCCCC: 2
  codaUpdateStatus: >UPDATE process SET state=
  [0] count for roc[11] = 2
  'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<
CCCCCCCCCC: 2
  <<<<<<<<<<<<
  CCCCCCCCCC: 0
ERE name='dc8'<
  [0] count for roc[11] = 0
codaUpdateStatus: dbDisconnecting ..
  CCCCCCCCCC: 0
  codaUpdateStatus: dbDisconnect done
  [0] flush input streams
  codaUpdateStatus: updating request ..
CCCCCCCCCC: 2
  UDP_standard_request >sta:dc8 paused<
  [0] count for roc[12] = 2
  UDP_standard_request >sta:dc8 paused<
  CCCCCCCCCC: 2
  UDP_standard_request >sta:dc8 paused<
  CCCCCCCCCC: 0
  UDP_standard_request >sta:dc8 paused<
  [0] count for roc[12] = 0
  UDP_standard_request >sta:dc8 paused<
  CCCCCCCCCC: 0
  UDP_standard_request >sta:dc8 paused<
  [0] flush input streams
  UDP_cancel: cancel >sta:dc8 downloaded<
  CCCCCCCCCC: 2
  codaUpdateStatus: updating request done
  [0] count for roc[13] = 2
prestarted
  CCCCCCCCCC: 2
  POLLS: 0 0
  CCCCCCCCCC: 0
  codaExecute done
  [0] count for roc[13] = 0
  .....
  CCCCCCCCCC: 0
 
  [0] flush input streams
 
  CCCCCCCCCC: 0
* Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download:
  [0] count for roc[14] = 0
........
  CCCCCCCCCC: 0
  wait: coda request >download prod< in progress
  [0] flush input streams
  wait: coda request >download prod< in progress
  CCCCCCCCCC: 0
  wait: coda request >download prod< in progress
  [0] count for roc[15] = 0
  3123: dbsock=59553504
  CCCCCCCCCC: 0
  3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<
[0] flush input streams
  ====1==== !!! object->runType=47
  CCCCCCCCCC: 0
  token_interval=64
  [0] count for roc[16] = 0
  object->name >polar<
CCCCCCCCCC: 0
  big0.proc_on_pmc=0, big0.net_on_pmc=0
[0] flush input streams
  big1.proc_on_pmc=0, big1.net_on_pmc=0
  CCCCCCCCCC: 0
  name selected
  [0] count for roc[17] = 0
  nrow=1
  CCCCCCCCCC: 0
  download: name >polar<
  [0] flush input streams
  code selected
  CCCCCCCCCC: 0
nrow=1
  [0] count for roc[18] = 0
  download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <
  CCCCCCCCCC: 0
  nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<
  [0] flush input streams
  nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
  CCCCCCCCCC: 0
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
  [0] count for roc[19] = 0
  set this_roc_id = 25, rocId() can be called from now on
  CCCCCCCCCC: 0
  ObjInitName >_moller__init<
  [0] flush input streams
  wait: coda request >download prod< in progress
  CCCCCCCCCC: 0
  wait: coda request >download prod< in progress
  [0] count for roc[20] = 0
  wait: coda request >download prod< in progress
  CCCCCCCCCC: 0
  wait: coda request >download prod< in progress
  [0] flush input streams
  wait: coda request >download prod< in progress
CCCCCCCCCC: 0
  INFO: >_moller__init()< routine found
  [0] count for roc[21] = 0
  readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded
  CCCCCCCCCC: 0
  allocate 200 buffers with length 65536 bytes
  [0] flush input streams
  partCreate1() reached
  CCCCCCCCCC: 0
  partCreate1(): pPart=0x008e7570
[0] count for roc[22] = 0
  listInit: cleanup 24 bytes starting from address 0x008e75a0
  CCCCCCCCCC: 0
  partCreate1() done
  [0] flush input streams
  0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0
  CCCCCCCCCC: 0
  0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1
[0] count for roc[23] = 0
  0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
  CCCCCCCCCC: 0
  0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<
[0] flush input streams
  0x81c7e0 (twork0004): Init - Done
  CCCCCCCCCC: 0
  0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...
  [0] count for roc[24] = 0
  wait: coda request >download prod< in progress
  CCCCCCCCCC: 0
  wait: coda request >download prod< in progress
  [0] flush input streams
  Exception at interrupt level:
  CCCCCCCCCC: 0
  Exception next instruction address: 0xfff901f0
[0] count for roc[25] = 0
  Fixed Point Register: 0x00000012
  CCCCCCCCCC: 0
  Condition Register: 0x40482044
  [0] flush input streams
  Regs at 0x64e498
  CCCCCCCCCC: 0
Press any key to stop auto-boot...
[0] count for roc[26] = 0
2
  CCCCCCCCCC: 0
  [VxWorks Boot]:
[0] flush input streams
 
  CCCCCCCCCC: 0
It suppose to be as following:
  [0] count for roc[27] = 0
  .....
CCCCCCCCCC: 0
  0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0
[0] flush input streams
  0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1
  CCCCCCCCCC: 0
  0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
  [0] count for roc[28] = 0
  rol1: downloading DDL table ...
  CCCCCCCCCC: 0
  0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<
  [0] flush input streams
0x91ae30 (twork0001): Init - Done
  CCCCCCCCCC: 0
  0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1
  [0] count for roc[29] = 0
  wait: coda request >download prod< in progress
  CCCCCCCCCC: 0
adr1(nddl_)=0x037b69c8
  [0] flush input streams
  N  name (nname)    fmt (nfmt)    ncol
  CCCCCCCCCC: 0
  [ 1]   >PTRN< (4)        >B32< (3)    1
  [0] count for roc[30] = 0
  [ 2]  >PSYN< (4)        >B32< (3)    1
CCCCCCCCCC: 0
  [ 3]  >RC00< (4)        >B32< (3)    1
  [0] flush input streams
[ 4]  >RC01< (4)        >B32< (3)    1
  CCCCCCCCCC: 0
  .....
  [0] count for roc[31] = 0
 
  CCCCCCCCCC: 0
 
[0] ============= Build threads cleaned
* Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output:
  [0] ============= Build threads cleaned
-> ts2status
  [0] ============= Build threads cleaned
  CSR 1 (0x0):
  [0] ============= Build threads cleaned
                                      Go : 1
  [0] ============= Build threads cleaned
            Pause on Next scheduled Sync : 0
  [0] build thread exiting: 1 1
                          Sync and Pause : 0
  [0] The implementation has detected that the value  specified by threa
                    Initiate Sync Event : 0
d does not refer to a joinable thread.
                Initiate Program 1 Event : 0
  wait: coda request >exit< in progress
                Initiate Program 2 Event : 1
  polling_routine ================!!!!!!!!!!!!!!!!!!=====
        Enable Level 1 (drives outputs) : 1
  cancel building threads
                        Override Inhibit : 0
=c=============================================
                              Test Mode : 0
  =c=============================================
                                Reserved : 0
=c=============================================
                                  Reset : 0
=c====
                              Initialize : 0
>>>>> close link from >dc1< link=0x081c2af0
                    Sync Event occurred : 0
debCloseLink: theLink=0x081c2af0 -> closing
                Program 1 Event occurred : 1
debCloseLink: link is down
                Program 2 Event occurred : 0
debCloseLink: free memory
                      Late Fail occurred : 0
debCloseLink: done.
                        Inhibit occurred : 1
=c====
              Write FIFO error occurred : 0
>>>>> close link from >dc2< link=0x081c4a90
              Read FIFO error occurreds : 1
debCloseLink: theLink=0x081c4a90 -> closing
  CSR 2 (0x4):
debCloseLink: link is down
                      Enable Scheduled Sync : 1
  debCloseLink: free memory
                    Use Clear Permit Timer : 1
debCloseLink: done.
                      Use Front Busy Timer : 1
=c====
                      Use Clear Hold Timer : 1
>>>>> close link from >dc3< link=0x081c4b28
                    Use External Front Busy : 1
debCloseLink: theLink=0x081c4b28 -> closing
                          Lock ROC Branch 1 : 0
debCloseLink: link is down
                          Lock ROC Branch 2 : 0
debCloseLink: free memory
                          Lock ROC Branch 3 : 0
debCloseLink: done.
                          Lock ROC Branch 4 : 0
=c====
                          Lock ROC Branch 5 : 0
>>>>> close link from >dc4< link=0x081c14f8
        Enable Program 1 front panel input : 1
debCloseLink: theLink=0x081c14f8 -> closing
                          Enable Interrupt : 1
debCloseLink: link is down
                Enable local ROC (branch 5) : 1
debCloseLink: free memory
  Trigger Control Register (0x8): 0x00001fff
debCloseLink: done.
  ROC Enable Register (0xc) val=0xf07fff7f:
=c====
  Branch 1: 0x7f  bits: 01111111
>>>>> close link from >dc5< link=0x081c1590
  Branch 2: 0xff  bits: 11111111
  debCloseLink: theLink=0x081c1590 -> closing
  Branch 3: 0x7f bits: 01111111
  debCloseLink: link is down
  Branch 4: 0xf0  bits: 11110000
  debCloseLink: free memory
  Synchronization Interval Register (0x10): 1000
  debCloseLink: done.
  Trigger Word Count Register (0x14): 0
  =c====
  Trigger Data Register (0x18): 256
  >>>>> close link from >dc6< link=0x081c1628
  Local ROC (Branch 5) Data Register (0x1c): 60
  debCloseLink: theLink=0x081c1628 -> closing
                      Synchronization Flag : 0
  debCloseLink: link is down
                            Late Fail Flag : 0
  debCloseLink: free memory
                                  ROC code : 15
  debCloseLink: done.
  Input Trigger Prescale Registers:
  =c====
                Input 1 Prescale Factor : 0
>>>>> close link from >dc8< link=0x081c16c0
                Input 2 Prescale Factor : 0
debCloseLink: theLink=0x081c16c0 -> closing
                Input 3 Prescale Factor : 0
debCloseLink: link is down
                Input 4 Prescale Factor : 0
  debCloseLink: free memory
                Input 5 Prescale Factor : 0
debCloseLink: done.
                Input 6 Prescale Factor : 0
=c====
                Input 7 Prescale Factor : 0
>>>>> close link from >cc1< link=0x081c1a08
                Input 8 Prescale Factor : 0
debCloseLink: theLink=0x081c1a08 -> closing
Clear Permit Timer Register = 0
debCloseLink: link is down
Level2 Accept Timer Register = 83
debCloseLink: free memory
Level3 Accept Timer Register = 0
debCloseLink: done.
Front Busy Timer Register    = 325
  =c====
Clear Hold Timer Register    = 50
  >>>>> close link from >sc1< link=0x081c1aa0
  Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040
debCloseLink: theLink=0x081c1aa0 -> closing
  Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
debCloseLink: link is down
  Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
  debCloseLink: free memory
  Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
  debCloseLink: done.
  Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
=c====
  Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086
  >>>>> close link from >ec1< link=0x081c1b38
  Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0
  debCloseLink: theLink=0x081c1b38 -> closing
ROC Acknowledge Status Register (0x60): val=0x0
debCloseLink: link is down
  Branch 1: 0x 0 bits: 00000000 (enabled: 01111111)
  debCloseLink: free memory
  Branch 2: 0x 0 bits: 00000000 (enabled: 11111111)
  debCloseLink: done.
  Branch 3: 0x 0 bits: 00000000 (enabled: 01111111)
=c====
  Branch 4: 0x 0 bits: 00000000 (enabled: 11110000)
  >>>>> close link from >ec2< link=0x081c1bd0
  State Register (0x6C):
  debCloseLink: theLink=0x081c1bd0 -> closing
                      Level 1 Accept    : 0
  debCloseLink: link is down
                    Start Level 2 Trigger : 0
  debCloseLink: free memory
                    Level 2 Pass Latched : 0
  debCloseLink: done.
                    Level 2 Fail Latched : 0
  =c====
                          Level 2 Accept : 0
>>>>> close link from >scaler2< link=0x681d95c8
                    Start Level 3 Trigger : 0
  debCloseLink: theLink=0x681d95c8 -> closing
                    Level 3 Pass Latched : 0
  debCloseLink: link is down
                    Level 3 Fail Latched : 0
  debCloseLink: free memory
                          Level 3 Accept : 0
debCloseLink: done.
                                    Clear : 0
=c====
                Front End Busy (external) : 0
>>>>> close link from >scaler3< link=0x681d9660
                        External Inhibit : 0
debCloseLink: theLink=0x681d9660 -> closing
                          Latched Trigger : 0
debCloseLink: link is down
                                  TS Busy : 1
debCloseLink: free memory
                                TS Active : 1
debCloseLink: done.
                                TS Ready : 0
=c====
                    Main Sequencer Active : 0
>>>>> close link from >scaler4< link=0x681d96f8
        Synchronization Sequencer Active : 0
debCloseLink: theLink=0x681d96f8 -> closing
        Program 1 Event Sequencer Active : 1
debCloseLink: link is down
        Program 2 Event Sequencer Active : 0
debCloseLink: free memory
Event Count (0xc8): 135
debCloseLink: done.
  Live 1 Count (0xcc): 54866
=c====
  Live 2 Count (0xd0): 9290375
>>>>> close link from >ec4< link=0x00000001
value = 1 = 0x1
debCloseLink: theLink=0x00000001 -> closing
-> exit
  Segmentation fault
  clondaq1:coda_eb>  


clasprod::clastrig2> clastrig2 statistics
DC7:
200 0 0 0


codaExecute done
wait: coda request >download prod< in progress
codaExecute reached, message >prestart<, len=8
codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410567520
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
3
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc7<
LINK_constructor_C: set name to >dc7->EB1<
3121: dbsock=410567520
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda req<<<
uest >fields [pr2estart] >>>< in prodowngress
<<<
fields [3] >>>37683<<<
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x18772230 (twork0028INFO: Entering Us): er
Prestart 2
INIT_NAME: rolp->daproc =
trans_rol2:2
adr(nddl)=1db60750 NWBOS=16384
uthbook1: WARN id=1047 already exist - will be replaced
uthbook1: free histogram buffer ... done.
RAW=0  PROFILE=1  NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x18772230INFO: User Prest (art 2 executed
twork0028): CTRIGRSS: set handler and done,
code=1
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4130) returns 490106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a580) returns 490106296
0x18772230 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x18772230 (twork0028): INFO: Entering User
Prestart
disconnecting vector 236
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc
0x1882d864 0x00000001 0x1882b0c8
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1  = FFFFF000
Fastbus Status Register2  = FFFF0000
Fastbus Last Primary Addr = 00000019
spds_mask1: 0xfdfbf0
0x18772230_tdc1877 ===> (4twork0028wait: co)
:  Use 32-bit Ram List
da request >firsprestartt< in progre csr0=ss
100 (1100) csr18=1364 csr1=6400001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0=100 (900) csr18=13
bignetptr=0x641d366fe0 csr1= offset=0x00bigp
rocptr=0x000001d366db80 offset=0x6400000000
00001b
bigproc at 0x
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =
==>1d366dc818 -> 0x1d366dc8
proc_thread reacheUDP_cancel: cancel >inf:d
c7 sys 0, mask 1d
<
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached
<
wait: cproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot#  1872  1875  1881(2)  1877(2)  1877S(2)
[ 0]      0    0      0 0bignetptr=0x     
1d366fe00 offset=0x0000000 0
0      0 0
[ 1]      0    0      0 0      0 0      0
0
[ 2]      0    0      0 0      0 0      0
0
[ 3]      0    0      0 0      0 0      0
0
[ 4]      0    0      0 0      1 0      0
0
[ 5]      0    0      0 0      1 0      0
0
[ 6]      0    0      0 0      1 0      0
0
[ 7]      0    0      0 0      1 0      0
0
[ 8]      0    0      0 0      1 0      0
0
[ 9]      0    0      0 0      1 2      0
0
[10]      wait: coda reques0t >    prestart
0<      in progress
0 0      0 0      0 0
[11]      0    0      0 0      1 0      0
0
[12]      0    0      0 0      1 0      0
0
[13]      0    0      0 0      1 0      0
0
[14]      0    0      0 0      1 0      0
0
[15]      0    0      0 0      1 0      0
0
[16]      0    0      0 0      1 2      0
0
[17]      0    0      0 0      0 0      0
0
[18]      0    0      0 0      1 0      0
0
[19]      0    0      0 0      1 0      0
0
[20] bignet at 0x    1d366fe00, bignet.gbig
Buffer at 0x    1d366ff00 -> 0x1d366ff0
      0 0      1 0      0 0
[21]      0    0      0 0      1 0      0
0
[22]      0    0      0 0      1 0      0
0
[23]      0    0      0 0      1 2      0
0
[24]      0    0      0 0      0 0      0
0
[25]      0    0      0 0      0 0      0
0
DEBUG 0 1 0 0
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC07<
0x18772230 (twork0028111): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <
<<<<<<<<<<<<<<<<
s: >UPDATE process SET state='paused' WHERE
name='dc7'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_cancel: cancel >sta:dc7 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
->
->


* Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages:
DC8:
 
interrupt: Program Events 1-2:      247        0
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)
proc_thread: waiting=  3743 processing=    30 microsec per event (nev=404)
rols_thread: waiting=    754 processing=    21 microsec per event (nev=454)
wait: coda request >go< in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
roc_end reached ??
roc_end reached !!
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <
interrupt: inputs  1-6:        0        0        0        0        0        0
interrupt: inputs 7-12:        0        0      339        0        0        0
interrupt: Program Events 1-2:      172        0
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)
wait: coda request >end< in progress
0x18c6f5c0 (twork0003): TS GO bit cleared
0x18c6f5c0 (twork0003): TS csr:  0xfc000000
interrupt: TRIG ERROR: no pool buffer available
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0
0x18c6f5c0 (twork0003): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_cancel: cancel >sta:clastrig2 active<
codaUpdateStatus: updating request done
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
roc_end done
codaExecute done
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
 
 
 
* Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine):
.....
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
.....
 
* Sergey B. 2-apr-2010: event recorder gives error during Prestart:
file[ 0]->/raid/stage_in/clas_062414.A00<-
bosopen.c: splitted files handling ..
bosopen.c: >/raid/stage_in/clas_062414.A00<
bosopen.c: len1=30
bosopen.c: len2=30
bosopen.c: len3=14
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14
bosopen.c: checking how much disk space left in directory /raid/stage_in
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available
1100988411 blocks (1 block = 512 bytes)
bosopen.c: so we have 550493696 kbytes available space
bosopen.c: we can continue to write file.
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-
bosopen.c: system call: >checkdisk  4094<
[1] 9253
bosopen.c: system call completed
codaUpdateStatus: dbConnecting ..
ER: Not attached to ET system
er_write_thread loop ended (0 145452664)
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:ER3 paused<
UDP_standard_request >sta:ER3 paused<
 
Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.
 
* Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB:
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:
7  9 10 11 17 20 22 25 26 28 29
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???
 
* Sergey B. 14-mar-2010: old problem, can be posted already: if EB was not restarted afre the ROC crash, it sometimes waiting for  several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash
 
* Sergey B. 14-mar-2010: during reboot dc9 gives following:
.................
ppc/bootscripts/boot_dc9
taskSpawn("TCPSERVER") returns 475458320
bind on port 5001
myname >dc9<
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded
INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
333333333333333
program
Exception current instruction address: 0x1c5
e8734
Machine Status Register: 0x00081000
Condition Register: 0x40000085
Task: 0x1c5e8e50 "ROC"
interrupt: Unconnected main interrupt 0
logTask: 3222 log messages lost.
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
..............
 
probably mysql connection problem
 
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up.
 
* Sergey B. 14-mar-2010: if in prestart stage hit abort and reset, all ROCs printing
wait: coda request >exit< in progress
indefinitely, must check
 
* Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were:
 
codaEnd 10
codaEnd 11
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
codaExecute done
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59319
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
codaExecute: ERROR: len=0 - do nothing
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59342
wait: coda request >< in progress
Executing >TP/1.0
 
Probably last string contains something, will remove printing
 
* Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages:
.....
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
  .. done.
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30)
sent -1 (type 18)
[0] ERROR: Discard data until next control event
clondaq1:coda_eb>
 
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer)  - does not helped. Did daq_exit, daq_start,
reboot all rocs again - not it works.


In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...
codaExecute: 'prestart' transition
 
roc_prestart reached
* Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er:
partReInitAll() reached ?
 
  partReInitAll() reached !
  .....
  partReInitAll() done
  UDP_standard_request >sta:EB1 booted<
  rocCleanup reached
  UDP_standard_request >sta:EB1 booted<
  partReInitAll() reached ?
  UDP_standard_request >sta:EB1 booted<
  wait: coda request >prestart< in progress
  UDP_standard_request >sta:EB1 booted<
  partReInitAll() reached !
UDP_cancel: cancel >sta:EB1 booted<
partReInitAll() done
  codaUpdateStatus: updating request done
  315: dbsock=410505056
  CODA_Init 14
  >>> prestarting, run 62496, type 47
  2
  rocMask=0x76fefffe
  clasprod::EB1> bind on port 5001
  !!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
  DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<
  WHERE name='dc8'<
  CODAtcpServer: start work thread
  !!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
befor: socket=5 address>129.57.71.28< port=37258
  WHERE name='dc8'<
wait: coda request >< in progress
  !!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
  wait: coda request >< in progress
  WHERE name='dc8'<
  wait: coda request >< in progress
  ++++> output to network
  wait: coda request >< in progress
  !!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<  
wait: coda request >< in progress
  3
  wait: coda request >< in progress
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
Executing >< (len=0)
  !!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
  Segmentation fault (core dumped)
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
clondaq1:coda_eb>  
rocCloselink reached
 
rocOpenLinks reached
In the same time ''et_start'' complains (we have now ET system protected against CC scans):
  LINK_constructor_C: set host to >dc8<
 
LINK_constructor_C: set name to >dc8->EB1<
  et ERROR: et_netserver: ET server being probed by non-ET client or read failure
  3121: dbsock=410505056
  et ERROR: et_netserver: ET server being probed by non-ET client or read failure
name selected
 
nrow=1
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already
fields [0] >>>TCP<<<
 
fields [1] >>>clondaq1-daq1wait: coda re<<<
* Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access:
  quest >fields [p2restart] >>>< in prupogress
 
  <<<
  Done executing startup script $CODA/VXWORKS_
  fields [3] >>>37772<<<
  ppc/bootscripts/boot_pmc1
  parsing results: type=>TCP< host=>clondaq1-d
  -> INFO(mysql_real_connect9): errno=0
  aq1< state=>up< port=>37772< -> 37772
  INFO(mysql_real_connect9): OK
  LINK_establish: socket # 17
  mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
  LINK_establish: socket buffer size is 48000(
  dbConnect ERROR: mysql == NULL
  0x0000bb80) bytes
  program
  LINK_establish: keepAlive is 8
  Exception current instruction address: 0x00000000
  LINK_establish: socket 17 is ready: host 129
  Machine Status Register: 0x0008b030
  .57.68.22 port 37772
  Condition Register: 0x40000085
  DB command >UPDATE links SET state='up' WHER
  Task: 0x1e404190 "coda_pmc"
  E name='dc8->EB1'<
 
  UPDATE process success
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.
Spawn proc/net threads
 
  Spawn proc/net threads
* Sergey B. 1-sep-2009: during croctest1 reboot:
Spawn proc/net threads
 
0x187ef280INFO: Entering U (ser Prestart 2
  INFO(mysql_real_connect9): errno=0
twork0028
  INFO(mysql_real_connect9): OK
trans_rol2:): adr(nddl)=INIT_NAME: rolp->da
  mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
proc = 1d2b60750
  dbConnect ERROR: mysql == NULL
  NWBOS=16384
 
uthbook1: WARN id=1048 already exist - will
second reboot went fine
be replaced
 
  uthbook1: free histogram buffer ... done.
* Sergey B. 15-june-2009: just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?
  RAW=0  PROFILE=1  NOTRANS=0
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
 
0x187ef280 (twork0028INFO: User Pr): estart
* Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host:
2 executedCTRIGRSS: set handler and done, co
 
  de=
  ..........................
  1
  taskDelay(sysClkRateGet()*5)
  0x187ef280 (twork0028): CTRIGRSS: 0x187705ac
  Args = -session clasprod -objects sc2 ROC -i
  0x187704d0 0x1876fdf0 0x00000001 0x1876fde8
  CODA_Init reached
  wait: coda request >prestart< in progress
  CODA_Init 1
  bignetptr=0x1d366fe0 offset=0x00000000
  CODA_Init 2
taskSpawn("coda_net",0x187d4110) returns 490
sc2
  106848
  CODA_Init 11
  wait: coda request >prestart< in progress
  CODA_Init: objectTy >(null)<
  setHeartError: 0 >sys 0, mask 1<
  CODA_Init 12
  WARN: HeartBeat[0]: heartbeat=19424(19424) h
  CODA_Init: use 'SESSION' as >clasprod<
  eartmask=1
  Tcl_AppInit CALLS !!!!!
  bignetptr=0x1d366fe0 offset=0x00000000
  11-11
  bigprocptr=0x1d366db8 offset=0x00000000
11-12
  taskSpawn("coda_proc",0x1874a560) returns 49
  11-13
  0106296
  11111111111111111
  0x187ef280 (twork0028): INIT_NAME: rolp->dap
  22222222222222222
  roc = 2
  0xdd0e880 (tNetTask): arptnew failed on 8139a743
  0x187ef280 (twork0028): INFO: Entering User
  value = 0 = 0x0
  Prestart
  taskSpawn "TCP_SERVER",250,0,100000,tcpServer
disconnecting vector 236
  value = 231091088 = 0xdc62b90
0x187ef280 (twork0028): CTRIGRSA: set handle
  proconhost
r and done, code=1
  value = 0 = 0x0
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c
  ......................
  0x1882d1e4 0x00000001 0x1882aa48
 
Sequencer Status Register = FFFFA000
Second reboot shows the same:
  Last Sequencer KeyAddress = FFFF0003
 
Fastbus Status Register1  = FFFFF000
  ...................
Fastbus Status Register2  = FFFF0000
  22222222222222222
Fastbus Last Primary Addr = wait: coda req00
  0xdd0e880 (tNetTask): arptnew failed on 8139a743
0000uest >19presta
  >>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
  rtspds_mask1: 0x< ifdfbf0n progress
  list >clon10 clasrun 2508 9998<
0x187ef280_tdc1877 === (>twork0028): Use 32-
machine name = clon10
  bit Ram List
  user ID      = 2508
  4 first csr0=100 (1100) csr18=1364 csr1=6400
  group ID    = 9998
001b
  .................
  _tdc1877 ===>5 middle csr0=100 (1900) csr18=
 
  1364 csr1=6400001b
 
  _tdc1877 ===>6 middle csr0=100 (1900) csr18=
* Sergey B. around 6-June-2009: ec2pmc1 error message:
  1364 csr1=6400001b
 
  _tdc1877 ===>7 middle csr0=100 (1900) csr18=
  write_thread: about to print
  1364 csr1=6400001b
  write_thread: about to print
  _tdc1877 ===>8 middle csr0=100 (1900) csr18=
  write_thread: about to print
  1364 csr1=6400001b
  write_thread: about to print: 255115 20
  _tdc1877 ===>9 last csr0=100 (900) csr18=136
  write_thread: about to print: 255115 20
  4 csr1=6400001b
  write_thread: about to print: 255115 20
  _tdc1877 ===>11 first csr0=100 (1100) csr18=
write_thread: wait=   199 send=     8 microsec per event (nev=12755)
  1364 csr1=6400001b
  CALLING customized :
  _tdc1877 ===>12 middle csr0=100 (1900) csr18
  machine check
  =1364 csr1=6400001b
SRAM ERROR :
  _tdc1877 ===>13 middle csr0=100 (1900) csr18
  SRAM Error Address    : 0x00000000.f2000504
  =1364 csr1=6400001b
  SRAM Error Data Low  : 0x006805e0
  _tdc1877 ===>14 middle csr0=100 (1900) csr18
  SRAM Error Data High  : 0x00420000
  =1364 csr1=6400001b
  SRAM Error Parity    : 0x00000004
  _tdc1877 ===>15 middle csr0=100 (1900) csr18
  SRAM Error Cause      : 0x00000001
  =1364 csr1=6400001b
  PCI_0 DEVICE ERROR :
  _tdc1877 ===>16 last csr0bignetptr=0x=1d366f
  PCI_0 Error: Status=0x00000100
  e0100 offset=0x0000000 (0
  Master abort
900) csr18=1364bigprocptr=0x csr1=1d366db864
  PCI Cmd=7 (MemWr) ByteEnable=0  Par=0
  00001b offset=0x
  Error Address High: 0x00000000
  00000000
  Error Address Low : 0xf0d10128
  bigproc at 0x_tdc1877 ===1d366db8>, bigproc.
  CALLING generic    :
  gbigBuffer at 0x181d3UDP_cancel: cancel >inf
  machine check
  :dc8 sys 0, mask 166dc8<
  Exception next instruction address: 0x0001f058
  wait: coda rUDP_cancel: cancel >inf:dc8 sys
  Machine Status Register: 0x0012b030
  0, mask 1 -> 0x<
Condition Register: 0x40222042
1d366dc8eq
  Task: 0x1e3b0020 "coda_net"
firstproc_thread reached
 
uest >proc_thread reached
reboot did not helped, have to turn pmc off and run on host only
  csr0=proc_thread reached
 
prproc_thread reached
 
proc_thread reached
* Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following:
  proc_thread reached
 
  proc_thread reached
  0x1e42a360 (coda_pmc): bb_new: 'big' buffer
proc_thread reached
  created (addr=0x1eba3590, 16 bufs, 3145728 size)
  proc_thread reached
  coda_pmc: big buffer1 allocated
  proc_thread reached
  myname >ec1pmc1<
  proc_thread reached
  Done executing startup script $CODA/VXWORKS_
  proc_thread reached
  ppc/bootscripts/boot_pmc1
  100estart (< in prog1100ress
  -> INFO(mysql_real_connect9): errno=0
) csr18=1364 csr1=6400001b
  INFO(mysql_real_connect9): OK
  _tdc1877 ===>19 middle csr0=100 (1900) csr18
  program
  =1364 csr1=6400001b
  Exception current instruction address: 0x00000000
  _tdc1877 ===>20 middle csr0=100 (1900) csr18
  Machine Status Register: 0x0008b030
  =1364 csr1=6400001b
  Condition Register: 0x40000085
  _tdc1877 ===>21 middle csr0=100 (1900) csr18
  Task: 0x1e42a360 "coda_pmc"
  =1364 csr1=6400001b
  -> tt
  _tdc1877 ===>22 middle csr0=100 (1900) csr18
  14cad8 vxTaskEntry    +68 : coda_pmc ()
  =1364 csr1=6400001b
  1e2275c0 coda_pmc      +144: mysql_query ()
  _tdc1877 ===>23 last csr0=100 (900) csr18=13
  1e258520 mysql_query    +54 : mysql_real_query ()
  64 csr1=6400001b
  1e249850 mysql_real_query+110: mysql_send_query ()
DEBUG in SFIRamLoad => 0 1 0 0 0
  1e249718 mysql_send_query+1d0: 0 ()
slot#  1872  bignetptr=0x1875  181d366fe081
  value = 0 = 0x0
  (2)  1877( offset=0x21877S(00000002)
  ->
  0
 
  [ 0]      0    0      0 0      0 0      0
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.
0
 
[ 1]      0    0      0 0      0 0      0
 
0
* Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift):
[ 2]      0    0      0 0      0 0      0
 
0
  write_thread: about to print: 394603 20
  [ 3]      0    0      0 0      0 0      0
  write_thread: about to print: 394603 20
  0
  write_thread: wait=    176 send=     6 microsec per event (nev=19730)
  [ 4]      0    0      0 0     1 0      0
  0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
0
  proc_thread: wait=    176 send=     12 microsec per event (nev=14534)
  [ 5]      0    0      0 0      1 0      0
  0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0
  0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
  [ 6]      0     0      0 0      1 0      0
  bosMgetid: ERROR: lenn=0
0
  ???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
  [ 7]      0    0      0 0      1 0      0
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
  0
  bosMgetid: lenn=4, len=4, nddl1=116
  [ 8]      0    0      0 0      1 0      0  
  bosMgetid: name=>RC-286331154<
  0
  try again ..
  [ 9]      0    0      0 0      1 2      0
  bosMgetid: ERROR: lenn=0
  0
  !!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
  [10]      wait: coda request0 >    prestart
  bosMgetid: lenn=4, len=4
  0< i      n progress
  bosMgetid: ERROR: name >RC-2< does not descr
  0 0      0 0      0 0
  ibed, update clonbanks.ddl file !!!
  [11]      0    0      0 0      1 0      0
  bosMgetid: lenn=4, len=4, nddl1=116
  0
  bosMgetid: name=>RC-286331154<
  [12]      0    0      0 0      1 0      0
no way !!
  0
  bosMlink: ERROR: bosMgetid returns -99
  [13]      0    0      0 0      1 0      0
 
  0
  interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
  [14]      0    0      0 0      1 0      0
  interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
  0
  interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
  [15]      0    0      0 0      1 0      0
  interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
0
  setHeartError: 0 >sys 0, mask 14<
  [16]      0    0      0 0      1 2       0
  WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
0
  UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
[17]      0    0      0 0      0 0      0
  UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
  0
 
  [18]     0    0      0 0      1 0      0
 
  0
 
  [19]      0 bignet at 0x    1d366fe00, bigne
 
t.gbigBuffer at 0x      1d366ff00 -> 0x1d366
* Sergey B. 14-May-2009 5:54am: got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:
ff0
 
  0      1 0      0 0
  clon00:clasrun> ipc_info -a clasprod
[20]      0    0      0 0      1 0      0
  08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
  0
  08:08:31: TAL-SS-00089-I Using local protocol
  [21]      0    0      0 0      1 0      0
  08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
  0
  08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
  [22]      0    0      0 0      1 0      0
  08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
  0
  08:08:31: TAL-SS-00089-I Using local protocol
  [23]      0    0      0 0      1 2      0
  08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
  0
  08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
  [24]      0    0      0 0      0 0      0
08:08:31: TAL-SS-00089-I Using tcp protocol
0
^C
  [25]      0    0      0 0      0 0      0
 
0
and it waits here. After restarting server on clondb1 by commands
DEBUG 0 1 0 0
 
INFO: datascan = 0x00000000
/etc/init.d/smartsockets stop
INFO: datascan&spds_mask1 = 0x00000000
/etc/init.d/smartsockets start
rcname >RC08<
0x187ef280 (twork0028111
): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000
0000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state=
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<
<<<<<<<<<<<<
ERE name='dc8'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_cancel: cancel >sta:dc8 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
.....


everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:


  top - 08:06:16 up 130 days, 8:59, 1 user, load average: 2.13, 2.03, 1.64
== Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download ==
  Tasks: 92 total,  2 running90 sleeping,  0 stopped,   0 zombie
........
  Cpu(s): 25.0% us, 0.1% sy, 0.0% ni, 73.0% id, 1.9% wa, 0.0% hi,  0.1% si
  wait: coda request >download prod< in progress
  Mem:  3995356k total3968880k used,    26476k free,    88168k buffers
wait: coda request >download prod< in progress
  Swap8385920k total,      208k used, 8385712k free, 3432056k cached
  wait: coda request >download prod< in progress
  PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND                                                                                                     
  3123: dbsock=59553504
  4471 root      25  0 5908 4540 1956 R 100  0.1 300:24.06 rtserver.x                                                                                                   
  3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<
    1 root      16  0 4756 556  460 S    0.0  0:11.25 init                                                                                                         
====1==== !!! object->runType=47
    2 root      RT  0    0    0    0 S    0 0.0  0:03.85 migration/0                                                                                                   
token_interval=64
    3 root      34 19    0    0    0 S    0 0.0  0:10.23 ksoftirqd/0                                                                                                   
  object->name >polar<
    4 root      RT  0    0    0    0 S    0 0.0  0:01.79 migration/1                                                                                                   
  big0.proc_on_pmc=0, big0.net_on_pmc=0
    5 root      34 19    0    0    0 S    0 0.0  1:03.72 ksoftirqd/1                                                                                                   
  big1.proc_on_pmc=0, big1.net_on_pmc=0
    6 root      RT  0    0    0    0 S    0 0.0  0:01.62 migration/2
  name selected
 
nrow=1
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.
download: name >polar<
 
code selected
   
  nrow=1
* Sergey B. 12-may-2009: looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.
  download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <
 
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<
 
  nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
* Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure':
  listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
  rcConfigure::loadRcDbaseCbk: Loading database failed !!!
  set this_roc_id = 25, rocId() can be called from now on
On second 'Configure' click it worked.
  ObjInitName >_moller__init<
  wait: coda request >download prod< in progress
  wait: coda request >download prod< in progress
  wait: coda request >download prod< in progress
  wait: coda request >download prod< in progress
  wait: coda request >download prod< in progress
  INFO: >_moller__init()< routine found
  readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded
  allocate 200 buffers with length 65536 bytes
  partCreate1() reached
  partCreate1(): pPart=0x008e7570
  listInit: cleanup 24 bytes starting from address 0x008e75a0
  partCreate1() done
  0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0
  0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1
  0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
  0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<
  0x81c7e0 (twork0004): Init - Done
  0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...
  wait: coda request >download prod< in progress
  wait: coda request >download prod< in progress
  Exception at interrupt level:
Exception next instruction address: 0xfff901f0
Fixed Point Register: 0x00000012
Condition Register: 0x40482044
  Regs at 0x64e498
Press any key to stop auto-boot...
2
  [VxWorks Boot]:  


* Sergey B. June-2008: it looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!
It suppose to be as following:
 
.....
* Sergey B. 7-may-2008: EB1 crashed, last messages:
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0
 
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1
  EB: WARNING - resyncronization in crate controller number  5 ...  fixed
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
  .
  rol1: downloading DDL table ...
  EB: WARNING - resyncronization in crate controller number 6 ... fixed
  0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<
  .
  0x91ae30 (twork0001): Init - Done
  EB: WARNING - resyncronization in crate controller number 6 ... fixed
  0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1
.
wait: coda request >download prod< in progress
ERROR: lfmt=0 bankid=0
  adr1(nddl_)=0x037b69c8
N  name (nname)    fmt (nfmt)    ncol
[ 1]  >PTRN< (4)        >B32< (3)    1
  [ 2]  >PSYN< (4)        >B32< (3)    1
  [ 3]  >RC00< (4)        >B32< (3)    1
  [ 4]  >RC01< (4)        >B32< (3)    1
  .....


Found that tage2 and tage3 must be rebooted
* Sergey B. 5-may-2008: polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine
* Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:
#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s)  : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
#                                  kernel - use the kernel for notification
#                                  poll - use polling for notification
#                                  none - don't use any notification
#
#                                  the poll_limit is the number of seconds
#                                  that must pass before a resource is polled again.
#                                  It is optional, and if it is not present the previous
#                                  value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away
Created file, will see if it helped ..
* Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)
.................................................
write_thread: wait=    135 send=    28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait=    130 send=    31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait=    135 send=    31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait=    135 send=    31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
  ERROR: bufout overflow - skip the rest ...
        bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait=    128 send=    42 microsec per event (nev=12778)
proc_thread: wait=    371 send=    31 microsec per event (nev=6291)
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................


== Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output ==
-> ts2status
CSR 1 (0x0):
                                      Go : 1
            Pause on Next scheduled Sync : 0
                          Sync and Pause : 0
                    Initiate Sync Event : 0
                Initiate Program 1 Event : 0
                Initiate Program 2 Event : 1
        Enable Level 1 (drives outputs) : 1
                        Override Inhibit : 0
                              Test Mode : 0
                                Reserved : 0
                                  Reset : 0
                              Initialize : 0
                    Sync Event occurred : 0
                Program 1 Event occurred : 1
                Program 2 Event occurred : 0
                      Late Fail occurred : 0
                        Inhibit occurred : 1
              Write FIFO error occurred : 0
              Read FIFO error occurreds : 1
CSR 2 (0x4):
                      Enable Scheduled Sync : 1
                    Use Clear Permit Timer : 1
                      Use Front Busy Timer : 1
                      Use Clear Hold Timer : 1
                    Use External Front Busy : 1
                          Lock ROC Branch 1 : 0
                          Lock ROC Branch 2 : 0
                          Lock ROC Branch 3 : 0
                          Lock ROC Branch 4 : 0
                          Lock ROC Branch 5 : 0
        Enable Program 1 front panel input : 1
                          Enable Interrupt : 1
                Enable local ROC (branch 5) : 1
Trigger Control Register (0x8): 0x00001fff
ROC Enable Register (0xc) val=0xf07fff7f:
Branch 1: 0x7f  bits: 01111111
Branch 2: 0xff  bits: 11111111
Branch 3: 0x7f  bits: 01111111
Branch 4: 0xf0  bits: 11110000
Synchronization Interval Register (0x10): 1000
Trigger Word Count Register (0x14): 0
Trigger Data Register (0x18): 256
Local ROC (Branch 5) Data Register (0x1c): 60
                      Synchronization Flag : 0
                            Late Fail Flag : 0
                                  ROC code : 15
Input Trigger Prescale Registers:
                Input 1 Prescale Factor : 0
                Input 2 Prescale Factor : 0
                Input 3 Prescale Factor : 0
                Input 4 Prescale Factor : 0
                Input 5 Prescale Factor : 0
                Input 6 Prescale Factor : 0
                Input 7 Prescale Factor : 0
                Input 8 Prescale Factor : 0
Clear Permit Timer Register  = 0
Level2 Accept Timer Register = 83
Level3 Accept Timer Register = 0
Front Busy Timer Register    = 325
Clear Hold Timer Register    = 50
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0
ROC Acknowledge Status Register (0x60): val=0x0
Branch 1: 0x 0  bits: 00000000 (enabled: 01111111)
Branch 2: 0x 0  bits: 00000000 (enabled: 11111111)
Branch 3: 0x 0  bits: 00000000 (enabled: 01111111)
Branch 4: 0x 0  bits: 00000000 (enabled: 11110000)
State Register (0x6C):
                      Level 1 Accept    : 0
                    Start Level 2 Trigger : 0
                    Level 2 Pass Latched : 0
                    Level 2 Fail Latched : 0
                          Level 2 Accept : 0
                    Start Level 3 Trigger : 0
                    Level 3 Pass Latched : 0
                    Level 3 Fail Latched : 0
                          Level 3 Accept : 0
                                    Clear : 0
                Front End Busy (external) : 0
                        External Inhibit : 0
                          Latched Trigger : 0
                                  TS Busy : 1
                                TS Active : 1
                                TS Ready : 0
                    Main Sequencer Active : 0
        Synchronization Sequencer Active : 0
        Program 1 Event Sequencer Active : 1
        Program 2 Event Sequencer Active : 0
Event Count (0xc8): 135
Live 1 Count (0xcc): 54866
Live 2 Count (0xd0): 9290375
value = 1 = 0x1
-> exit


At the same time at sc2:
clasprod::clastrig2> clastrig2 statistics
200 0 0 0
 
 
== Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages ==


......................................
  interrupt: Program Events 1-2:     247        0
  interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
  interrupt: live time = 93 percent (gated=1956740 ungated=2094143)
  interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
  interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)
  interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
  proc_thread: waiting=  3743 processing=    30 microsec per event (nev=404)
  interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
  rols_thread: waiting=   754 processing=     21 microsec per event (nev=454)
  interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
  wait: coda request >go< in progress
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
  codaExecute reached, message >end<, len=3
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  codaExecute: 'end' transition
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
roc_end reached ??
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
roc_end reached !!
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  interrupt: inputs  1-6:       0        0        0        0        0        0
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  interrupt: inputs 7-12:       0        0      339        0        0        0
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  interrupt: Program Events 1-2:     172        0
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  interrupt: live time = 97 percent (gated=1358086 ungated=1400034)
................................
  interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  wait: coda request >end< in progress
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  0x18c6f5c0 (twork0003): TS GO bit cleared
  logTask: 220 log messages lost.
  0x18c6f5c0 (twork0003): TS csr: 0xfc000000
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  interrupt: TRIG ERROR: no pool buffer available
  logTask: 692 log messages lost.
  0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
  0x18c6f5c0 (twork0003): INFO: User End Executed
  logTask: 544 log messages lost.
  codaUpdateStatus: dbConnecting ..
  ..................................
  codaUpdateStatus: dbConnect done
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<
  logTask: 562 log messages lost.
  codaUpdateStatus: dbDisconnecting ..
  interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
  codaUpdateStatus: dbDisconnect done
  logTask: 707 log messages lost.
  codaUpdateStatus: updating request ..
  interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
  UDP_standard_request >sta:clastrig2 ending<
  logTask: 517 log messages lost.
  UDP_standard_request >sta:clastrig2 ending<
  interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
  UDP_standard_request >sta:clastrig2 ending<
  logTask: 620 log messages lost.
  UDP_standard_request >sta:clastrig2 ending<
  interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
  UDP_standard_request >sta:clastrig2 ending<
  logTask: 559 log messages lost.
  UDP_standard_request >sta:clastrig2 ending<
  interrupt: SYNC: scan_flag=0x00390000
  UDP_cancel: cancel >sta:clastrig2 active<
  logTask: 512 log messages lost.
  codaUpdateStatus: updating request done
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
NOW !!!!!!!!!!!!!!!!!!!
  logTask: 671 log messages lost.
  NOW !!!!!!!!!!!!!!!!!!!
  ......................................
  NOW !!!!!!!!!!!!!!!!!!!
  logTask: 563 log messages lost.
  NOW !!!!!!!!!!!!!!!!!!!
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  NOW !!!!!!!!!!!!!!!!!!!
  logTask: 385 log messages lost.
  roc_end done
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  codaExecute done
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: rocp->state == ENDING and input list is empty
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: process rol1 buffer by rol2 completed
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: break loop
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: rocp->state == ENDING and input list is empty
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: process rol1 buffer by rol2 completed
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: break loop
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: in DA_ENDING last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: rocp->state == ENDING and input list is empty
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: process rol1 buffer by rol2 completed
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: break loop
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: in DA_ENDING last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: rocp->state == ENDING and input list is empty
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: process rol1 buffer by rol2 completed
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: break loop
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: in DA_ENDING last event=12313 nevents=12314
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: rocp->state == ENDING and input list is empty
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
 
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
 
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
 
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
== Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine) ==
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  .....
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  event=2405 nevents=2405
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  coda_roc: processing case 'DA_ENDING': last
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  event=2405 nevents=2405
  wait: coda request in progress
  coda_roc: processing case 'DA_ENDING'
  codaExecute reached, message >end<, len=3
  coda_roc: processing case 'DA_ENDING'
  codaExecute: 'end' transition
  coda_roc: processing case 'DA_ENDING'
  wait: coda request in progress
  coda_roc: processing case 'DA_ENDING': last
  wait: coda request in progress
  event=2405 nevents=2405
  wait: coda request in progress
  coda_roc: processing case 'DA_ENDING'
wait: coda request in progress
  coda_roc: processing case 'DA_ENDING'
  wait: coda request in progress
  coda_roc: processing case 'DA_ENDING'
  .......................................
  coda_roc: processing case 'DA_ENDING': last
wait: coda request in progress
  event=2405 nevents=2405
  wait: coda request in progress
  coda_roc: processing case 'DA_ENDING'
  interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
  .....
  interrupt: TRIGGER ERROR: no pool buffer available
  wait: coda request in progress
  wait: coda request in progress
  .....................................


Note: sc2 never warn about wrong slot number before that moment.
== Sergey B. 2-apr-2010: event recorder gives error during Prestart ==
 
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time
file[ 0]->/raid/stage_in/clas_062414.A00<-
bosopen.c: splitted files handling ..
bosopen.c: >/raid/stage_in/clas_062414.A00<
bosopen.c: len1=30
bosopen.c: len2=30
bosopen.c: len3=14
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14
bosopen.c: checking how much disk space left in directory /raid/stage_in
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available
1100988411 blocks (1 block = 512 bytes)
bosopen.c: so we have 550493696 kbytes available space
bosopen.c: we can continue to write file.
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-
bosopen.c: system call: >checkdisk  4094<
[1] 9253
bosopen.c: system call completed
codaUpdateStatus: dbConnecting ..
ER: Not attached to ET system
er_write_thread loop ended (0 145452664)
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:ER3 paused<
UDP_standard_request >sta:ER3 paused<


Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.


* Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB:
== Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB ==
[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:
7  9 10 11 17 20 22 25 26 28 29
Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???
 
== Sergey B. 14-mar-2010: old problem, can be posted already ==
 
If EB was not restarted afre the ROC crash, it sometimes waiting for  several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash


  wait: coda request in progress
== Sergey B. 14-mar-2010: during reboot dc9 gives following ==
  wait: coda request in progress
  .................
  wait: coda request in progress
  ppc/bootscripts/boot_dc9
  wait: coda request in progress
  taskSpawn("TCPSERVER") returns 475458320
  wait: coda request in progress
  bind on port 5001
  wait: coda request in progress
  myname >dc9<
  Error(old rc): nRead=0, must be 1032
  -> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded
  CODAtcpServer: start work thread
  INFO(mysql_real_connect9): errno=0
  befor: socket=5 address>129.57.71.38< port=59063
  INFO(mysql_real_connect9): OK
  wait: coda request in progress
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
  Error(old rc): nRead=0, must be 1032
  dbConnect ERROR: mysql == NULL
  CODAtcpServer: start work thread
333333333333333
  befor: socket=5 address>129.57.71.38< port=60187
program
  wait: coda request in progress
Exception current instruction address: 0x1c5
  Error(old rc): nRead=0, must be 1032
e8734
  CODAtcpServer: start work thread
Machine Status Register: 0x00081000
  befor: socket=5 address>129.57.71.38< port=60295
  Condition Register: 0x40000085
  wait: coda request in progress
  Task: 0x1c5e8e50 "ROC"
  Error(old rc): nRead=0, must be 1032
  interrupt: Unconnected main interrupt 0
  CODAtcpServer: start work thread
  logTask: 3222 log messages lost.
  befor: socket=5 address>129.57.71.38< port=60418
  interrupt: Unconnected main interrupt 1
  wait: coda request in progress
  interrupt: Unconnected main interrupt 0
Error(old rc): nRead=0, must be 1032
  interrupt: Unconnected main interrupt 1
  interrupt: Unconnected main interrupt 0
  interrupt: Unconnected main interrupt 1
  interrupt: Unconnected main interrupt 0
  interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
  ..............
 
probably mysql connection problem
 
'''RESOLVING''': will make 5 attempts to to connect waiting 3 seconds in between, then will give up.  
 
== Sergey B. 14-mar-2010 ==
 
If in prestart stage hit abort and reset, all ROCs printing
  wait: coda request >exit< in progress
indefinitely, must check
 
== Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were ==
 
codaEnd 10
codaEnd 11
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
codaExecute done
  CODAtcpServer: start work thread
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=60823
  befor: socket=5 address>129.57.71.28< port=59319
  wait: coda request in progress
  wait: coda request >< in progress
  Error(old rc): nRead=0, must be 1032
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
  Executing >< (len=0)
codaExecute: ERROR: len=0 - do nothing
  CODAtcpServer: start work thread
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=32961
  befor: socket=5 address>129.57.71.28< port=59342
  wait: coda request in progress
  wait: coda request >< in progress
  Error(old rc): nRead=0, must be 1032
  Executing >TP/1.0
  CODAtcpServer: start work thread
 
  befor: socket=5 address>129.57.71.38< port=33122
Probably last string contains something, will remove printing
  wait: coda request in progress
 
  Error(old rc): nRead=0, must be 1032
== Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages ==
  CODAtcpServer: start work thread
.....
  befor: socket=5 address>129.57.71.38< port=33795
case 0: no swap
  wait: coda request in progress
case 0: no swap
  Error(old rc): nRead=0, must be 1032
case 0: no swap
  CODAtcpServer: start work thread
case 0: no swap
  befor: socket=5 address>129.57.71.38< port=34251
case 0: no swap
  wait: coda request in progress
case 0: no swap
  Error(old rc): nRead=0, must be 1032
case 0: no swap
case 0: no swap
  case 0: no swap
  case 0: no swap
  .. done.
  [0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30)
  sent -1 (type 18)
[0] ERROR: Discard data until next control event
  clondaq1:coda_eb>
 
After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start,
reboot all rocs again - not it works.
 
In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...
 
== Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er ==
 
.....
UDP_standard_request >sta:EB1 booted<
  UDP_standard_request >sta:EB1 booted<
  UDP_standard_request >sta:EB1 booted<
  UDP_standard_request >sta:EB1 booted<
  UDP_cancel: cancel >sta:EB1 booted<
  codaUpdateStatus: updating request done
CODA_Init 14
2
clasprod::EB1> bind on port 5001
  DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<
  CODAtcpServer: start work thread
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=34529
  befor: socket=5 address>129.57.71.28< port=37258
  wait: coda request in progress
  wait: coda request >< in progress
  Error(old rc): nRead=0, must be 1032
  wait: coda request >< in progress
  CODAtcpServer: start work thread
  wait: coda request >< in progress
befor: socket=5 address>129.57.71.38< port=35811
  wait: coda request >< in progress
  wait: coda request in progress
  wait: coda request >< in progress
  Error(old rc): nRead=0, must be 1032
  wait: coda request >< in progress
CODAtcpServer: start work thread
  Executing >< (len=0)
befor: socket=5 address>129.57.71.38< port=36134
  Segmentation fault (core dumped)
  wait: coda request in progress
  clondaq1:coda_eb>  
  Error(old rc): nRead=0, must be 1032
 
  CODAtcpServer: start work thread
In the same time ''et_start'' complains (we have now ET system protected against CC scans):
  befor: socket=5 address>129.57.71.38< port=41569
 
wait: coda request in progress
  et ERROR: et_netserver: ET server being probed by non-ET client or read failure
Error(old rc): nRead=0, must be 1032
  et ERROR: et_netserver: ET server being probed by non-ET client or read failure
  CODAtcpServer: start work thread
 
  befor: socket=5 address>129.57.71.38< port=53913
'''FIXED''' by checking for the len=0 in the message string, all other possible checks were implemented already
wait: coda request in progress
 
Error(old rc): nRead=0, must be 1032
== Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access ==
CODAtcpServer: start work thread
 
befor: socket=5 address>129.57.71.38< port=57701
Done executing startup script $CODA/VXWORKS_
  wait: coda request in progress
  ppc/bootscripts/boot_pmc1
  Error(old rc): nRead=0, must be 1032
  -> INFO(mysql_real_connect9): errno=0
  CODAtcpServer: start work thread
  INFO(mysql_real_connect9): OK
  befor: socket=5 address>129.57.71.38< port=33357
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
  wait: coda request in progress
  dbConnect ERROR: mysql == NULL
  Error(old rc): nRead=0, must be 1032
program
  CODAtcpServer: start work thread
  Exception current instruction address: 0x00000000
  befor: socket=5 address>129.57.71.38< port=36136
  Machine Status Register: 0x0008b030
wait: coda request in progress
  Condition Register: 0x40000085
  Error(old rc): nRead=0, must be 1032
  Task: 0x1e404190 "coda_pmc"
  CODAtcpServer: start work thread
 
  befor: socket=5 address>129.57.71.38< port=39509
It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.
wait: coda request in progress
 
  Error(old rc): nRead=0, must be 1032
== Sergey B. 1-sep-2009: during croctest1 reboot ==
CODAtcpServer: start work thread
 
befor: socket=5 address>129.57.71.38< port=39900
  INFO(mysql_real_connect9): errno=0
wait: coda request in progress
  INFO(mysql_real_connect9): OK
Error(old rc): nRead=0, must be 1032
  mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
CODAtcpServer: start work thread
  dbConnect ERROR: mysql == NULL
  befor: socket=5 address>129.57.71.38< port=45174
 
  wait: coda request in progress
second reboot went fine
Error(old rc): nRead=0, must be 1032
 
  CODAtcpServer: start work thread
== Sergey B. 15-june-2009 ==
  befor: socket=5 address>129.57.71.38< port=45909
 
  wait: coda request in progress
Just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ?
  Error(old rc): nRead=0, must be 1032
It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.
  CODAtcpServer: start work thread
 
  befor: socket=5 address>129.57.71.38< port=34521
== Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host ==
wait: coda request in progress
 
Error(old rc): nRead=0, must be 1032
  ..........................
  CODAtcpServer: start work thread
  taskDelay(sysClkRateGet()*5)
  befor: socket=5 address>129.57.71.38< port=49023
  Args = -session clasprod -objects sc2 ROC -i
  wait: coda request in progress
  CODA_Init reached
  Error(old rc): nRead=0, must be 1032
  CODA_Init 1
  CODAtcpServer: start work thread
  CODA_Init 2
  befor: socket=5 address>129.57.71.38< port=49526
  sc2
  wait: coda request in progress
  CODA_Init 11
  Error(old rc): nRead=0, must be 1032
  CODA_Init: objectTy >(null)<
  CODAtcpServer: start work thread
  CODA_Init 12
  befor: socket=5 address>129.57.71.38< port=53555
  CODA_Init: use 'SESSION' as >clasprod<
  wait: coda request in progress
  Tcl_AppInit CALLS !!!!!
  Error(old rc): nRead=0, must be 1032
11-11
  CODAtcpServer: start work thread
  11-12
befor: socket=5 address>129.57.71.38< port=55694
  11-13
wait: coda request in progress
  11111111111111111
Error(old rc): nRead=0, must be 1032
  22222222222222222
CODAtcpServer: start work thread
  0xdd0e880 (tNetTask): arptnew failed on 8139a743
  befor: socket=5 address>129.57.71.38< port=56833
value = 0 = 0x0
  wait: coda request in progress
  taskSpawn "TCP_SERVER",250,0,100000,tcpServer
  Error(old rc): nRead=0, must be 1032
  value = 231091088 = 0xdc62b90
  CODAtcpServer: start work thread
  proconhost
  befor: socket=5 address>129.57.71.38< port=58251
  value = 0 = 0x0
  wait: coda request in progress
  ......................
  Error(old rc): nRead=0, must be 1032
 
  CODAtcpServer: start work thread
Second reboot shows the same:
  befor: socket=5 address>129.57.71.38< port=59023
 
  wait: coda request in progress
  ...................
  Error(old rc): nRead=0, must be 1032
  22222222222222222
  CODAtcpServer: start work thread
  0xdd0e880 (tNetTask): arptnew failed on 8139a743
  befor: socket=5 address>129.57.71.38< port=33687
  >>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
  wait: coda request in progress
  list >clon10 clasrun 2508 9998<
  Error(old rc): nRead=0, must be 1032
  machine name = clon10
CODAtcpServer: start work thread
  user ID      = 2508
  befor: socket=5 address>129.57.71.38< port=35731
  group ID    = 9998
  wait: coda request in progress
  .................
  Error(old rc): nRead=0, must be 1032
 
  CODAtcpServer: start work thread
 
  befor: socket=5 address>129.57.71.38< port=51581
== Sergey B. around 6-June-2009: ec2pmc1 error message ==
  wait: coda request in progress
 
  wait: coda request in progress
  write_thread: about to print
  Error(old rc): nRead=18, must be 1032
  write_thread: about to print
  CODAtcpServer: start work thread
  write_thread: about to print
  befor: socket=5 address>129.57.71.38< port=41371
  write_thread: about to print: 255115 20
wait: coda request in progress
  write_thread: about to print: 255115 20
  Error(old rc): nRead=6, must be 1032
  write_thread: about to print: 255115 20
  CODAtcpServer: start work thread
  write_thread: wait=    199 send=     8 microsec per event (nev=12755)
  befor: socket=5 address>129.57.71.38< port=43198
  CALLING customized :
  wait: coda request in progress
  machine check
  Error(old rc): nRead=10, must be 1032
  SRAM ERROR :
  CODAtcpServer: start work thread
  SRAM Error Address    : 0x00000000.f2000504
  befor: socket=5 address>129.57.71.38< port=55260
  SRAM Error Data Low  : 0x006805e0
  wait: coda request in progress
  SRAM Error Data High  : 0x00420000
  Error(old rc): nRead=0, must be 1032
  SRAM Error Parity    : 0x00000004
  clasprod::EB1> ^C
  SRAM Error Cause      : 0x00000001
  PCI_0 DEVICE ERROR :
  PCI_0 Error: Status=0x00000100
  Master abort
  PCI Cmd=7 (MemWr)  ByteEnable=0  Par=0
  Error Address High: 0x00000000
  Error Address Low : 0xf0d10128
  CALLING generic    :
  machine check
Exception next instruction address: 0x0001f058
  Machine Status Register: 0x0012b030
  Condition Register: 0x40222042
  Task: 0x1e3b0020 "coda_net"


Similar messages were coming from ER.
reboot did not helped, have to turn pmc off and run on host only


ET reported following:


TCP server got a connection so spawn thread
== Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following ==
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure 


and another ET:
0x1e42a360 (coda_pmc): bb_new: 'big' buffer
  .......
created (addr=0x1eba3590, 16 bufs, 3145728 size)
  et_start: pthread_create(0x0000000d,...) done
coda_pmc: big buffer1 allocated
  TCP server got a connection so spawn thread
myname >ec1pmc1<
  et ERROR: et_client_thread: read failure
Done executing startup script $CODA/VXWORKS_
  TCP server got a connection so spawn thread
ppc/bootscripts/boot_pmc1
  et ERROR: et_client_thread: read failure
-> INFO(mysql_real_connect9): errno=0
  TCP server got a connection so spawn thread
INFO(mysql_real_connect9): OK
  et ERROR: et_client_thread: read failure
program
  TCP server got a connection so spawn thread
Exception current instruction address: 0x00000000
  et ERROR: et_client_thread: read failure
Machine Status Register: 0x0008b030
  TCP server got a connection so spawn thread
Condition Register: 0x40000085
  et ERROR: et_client_thread: read failure
Task: 0x1e42a360 "coda_pmc"
  TCP server got a connection so spawn thread
-> tt
  et ERROR: et_client_thread: read failure
14cad8 vxTaskEntry    +68 : coda_pmc ()
  TCP server got a connection so spawn thread
1e2275c0 coda_pmc      +144: mysql_query ()
  et ERROR: et_client_thread: read failure
1e258520 mysql_query    +54 : mysql_real_query ()
  TCP server got a connection so spawn thread
1e249850 mysql_real_query+110: mysql_send_query ()
  et ERROR: et_client_thread: read failure
1e249718 mysql_send_query+1d0: 0 ()
  TCP server got a connection so spawn thread
value = 0 = 0x0
  et ERROR: et_client_thread: read failure
->
  TCP server got a connection so spawn thread
 
  et ERROR: et_client_thread: read failure
for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.
  TCP server got a connection so spawn thread
 
  et ERROR: et_client_thread: read failure
 
== Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift) ==
 
write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait=    176 send=      6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait=    176 send=    12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99
 
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
 
 
 
 
== Sergey B. 14-May-2009 5:54am ==
 
Got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:
 
clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C
 
and it waits here. After restarting server on clondb1 by commands
 
/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start
 
everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:
 
top - 08:06:16 up 130 days,  8:59,  1 user,  load average: 2.13, 2.03, 1.64
Tasks:  92 total,  2 running,  90 sleeping,  0 stopped,  0 zombie
Cpu(s): 25.0% us,  0.1% sy,  0.0% ni, 73.0% id,  1.9% wa,  0.0% hi,  0.1% si
Mem:  3995356k total,  3968880k used,    26476k free,    88168k buffers
Swap:  8385920k total,      208k used,  8385712k free,  3432056k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                     
  4471 root      25  0  5908 4540 1956 R  100  0.1 300:24.06 rtserver.x                                                                                                   
    1 root      16  0  4756  556  460 S    0  0.0  0:11.25 init                                                                                                         
    2 root      RT  0    0    0    0 S    0  0.0  0:03.85 migration/0                                                                                                   
    3 root      34  19    0    0    0 S    0  0.0  0:10.23 ksoftirqd/0                                                                                                   
    4 root      RT  0    0    0    0 S    0  0.0  0:01.79 migration/1                                                                                                   
    5 root      34  19    0    0    0 S    0  0.0  1:03.72 ksoftirqd/1                                                                                                   
    6 root      RT  0    0    0    0 S    0  0.0  0:01.62 migration/2
 
while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.
 
== Sergey B. 12-may-2009 ==
 
Looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.
 
 
== Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure' ==
 
rcConfigure::loadRcDbaseCbk: Loading database failed !!!
On second 'Configure' click it worked.
 
== Sergey B. June-2008 ==
 
It looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!
 
== Sergey B. 7-may-2008: EB1 crashed, last messages ==
 
EB: WARNING - resyncronization in crate controller number  5 ...  fixed
.
EB: WARNING - resyncronization in crate controller number  6 ...  fixed
  .
EB: WARNING - resyncronization in crate controller number  6 ... fixed
.
ERROR: lfmt=0 bankid=0
 
Found that tage2 and tage3 must be rebooted
 
 
== Sergey B. 5-may-2008 ==
 
polar crate turned off; turn it back on, but notice that
one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything
works fine
 
 
== Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU ==
 
Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:
 
#configuration for gamin
# Can be used to override the default behaviour.
  # notify filepath(s) : indicate to use kernel notification
# poll filepath(s)  : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
#                                  kernel - use the kernel for notification
#                                  poll - use polling for notification
#                                  none - don't use any notification
#
#                                  the poll_limit is the number of seconds
#                                  that must pass before a resource is polled again.
#                                  It is optional, and if it is not present the previous
#                                  value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away
 
Created file, will see if it helped ..
 
 
== Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???) ==
 
.................................................
write_thread: wait=    135 send=    28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait=    130 send=    31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait=    135 send=    31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait=    135 send=    31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
  ERROR: bufout overflow - skip the rest ...
        bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait=    128 send=    42 microsec per event (nev=12778)
proc_thread: wait=    371 send=    31 microsec per event (nev=6291)
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
..................................................
 
 
At the same time at sc2:
 
......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
  interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................
 
Note: sc2 never warn about wrong slot number before that moment.
 
FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time
 
 
== Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB ==
 
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
  Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
  CODAtcpServer: start work thread
  befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C
 
Similar messages were coming from ER.
 
ET reported following:
 
  TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
  TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
et ERROR: et_client_thread: read failure
  et INFO: et_sys_heartmonitor, kill bad process (2,3063)
TCP server got a connection so spawn thread
  et INFO: et_sys_heartmonitor, cleanup process 2
et ERROR: et_client_thread: read failure
  et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
TCP server got a connection so spawn thread
  et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et ERROR: et_client_thread: read failure
  et INFO: set_fix_natts, station LEVEL3 has 1 attachments
TCP server got a connection so spawn thread
  et INFO: set_fix_natts, station ET2ER has 1 attachments
et ERROR: et_client_thread: read failure
  et INFO: set_fix_natts, # total attachments 2 -> 2
TCP server got a connection so spawn thread
  et INFO: set_fix_natts, proc 0 has 1 attachments
et ERROR: et_client_thread: read failure
  et INFO: set_fix_natts, proc 1 has 1 attachments  
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure 
 
and another ET:
.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
  TCP server got a connection so spawn thread
  et ERROR: et_client_thread: read failure
  et INFO: et_sys_heartmonitor, kill bad process (2,3063)
  et INFO: et_sys_heartmonitor, cleanup process 2
  et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
  et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
  et INFO: set_fix_natts, station LEVEL3 has 1 attachments
  et INFO: set_fix_natts, station ET2ER has 1 attachments
  et INFO: set_fix_natts, # total attachments 2 -> 2
  et INFO: set_fix_natts, proc 0 has 1 attachments
  et INFO: set_fix_natts, proc 1 has 1 attachments  
 
Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':
 
  Query test_ts2 table failed: Error reading table'test_ts2' definition
        ec3                                        ec3
 
If one runcontrol is restarted, then it works, but another one complains !!!
 
== Sergey B. 31-mar-2008 ==
 
'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.
 
== Sergey B. 10-mar-2008 ==
 
seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie
 
== 23-jan-2008 ==
ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
  [1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
  [2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
  [3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
  [4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
  [5] 0xfea60020(), at 0xfea60020
(dbx)


Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':
== 14-nov-2007 ==


  Query test_ts2 table failed: Error reading table'test_ts2' definition
first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far
        ec3                                        ec3
 
If one runcontrol is restarted, then it works, but another one complains !!!
 
* Sergey B. 31-mar-2008: 'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "<!-- xxx -->" comment sign.
 
* Sergey B. 10-mar-2008: seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters
XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window
was not big enough; set to 100 and 900 respectively, will ask Jie
 
* 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:
(dbx) where
  [1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20
  [2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d
  [3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a
  [4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36
  [5] 0xfea60020(), at 0xfea60020
(dbx)


* 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far
== Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following ==


* Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:
  ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
  ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
  ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
  ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
Line 2,589: Line 4,847:
Reboot ec2. Started new run 55463, everything looks normal.
Reboot ec2. Started new run 55463, everything looks normal.


* Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:
== Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen ==
  Unknown error errno=65
  Unknown error errno=65
  Unknown error errno=65
  Unknown error errno=65

Latest revision as of 12:31, 3 January 2018

CLAS12

Sergey B. Dec 2017 run: RESOLVED in library by always using even number

TI VME (Local) address = 0x00a80000 (0xabf81000)
tiReload: 
 FPGA Re-Load ! 
CODAtcpServer: wait: coda request >download PROD< in progress
  ID: 0x7101    Firmware (type - revision): 0x3 - 0x07A (prodID = 1)
... Configure as TI Slave...
tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1)
tiSetTriggerSource: ERROR: Invalid Trigger Source (129).  Must be between 0 and 11.
FiberMeas: syncDelay_write=0x82828200
FiberMeas: syncDelay=0x8200820b

FiberMeas: ERROR: TI Fiber Measurement failed!
        First Measurement != Second Measurement (60 != 61)

tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory

Sergey B. Dec 2017 run:

tiInit: Scanning for TI...
tiFind: Found TI at 0x00a80000
TI VME (Local) address = 0x00a80000 (0xabf5f000)
 ID: 0x7101    Firmware (type - revision): 0x3 - 0x07A (prodID = 1)
... Configure as TI Slave...
tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1)
tiSetTriggerSource: ERROR: Invalid Trigger Source (129).  Must be between 0 and 11.
FiberMeas: ERROR: TI Fiber Measurement failed!
       First Measurement != Second Measurement (90 != 58)
tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory

Sergey B. Dec 2017 run: possible interference with DiagGuiServer ???

tiInit: ERROR: Invalid Board ID: 0x0 (rval = 0x00000000)
tiInit: Scanning for TI...
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x100000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x600000
tiInit: ERROR: Unable to find TI
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory

Sergey B. Dec 2017 run:

clasrun@clondaq6:stage_in> tcpClient trig1 tdGStatus
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    12345678   V----   ---12345678   --1-3-----  0x0e1d0000   N
 4     12345678    12345678   V----   ---12345678   --1-----7-  0x21bd0000   Y
 5     12345678    12345678   V----   ---12345678   ----3---7-  0xe71d0000   Y
 6     123456-8    123456--   V----   ---123456--   --1-------  0x4fdd0000   N
 7     12345678    123456--   V----   ---123456--   --1-3-----  0x363d0000   Y
 8     12345678    123456--   V----   ---123456--   --1----6--  0xbf7d0000   Y
 9     123456--    123456--   V----   ---123456--   ----------  0x753d0000   N
10     123456--    123456--   V----   ---123456--   ----------  0xe8fd0000   N
13     1234567-    123456--   V----   ---123456--   ----------  0xa1dd0000   N
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1     ENABLED   BUSY-BlkAck      1 /   0           1       0
 3-2  tdcecal1  UP      2     ENABLED                    0 /   0           1       0
 3-3  adcpcal1  UP      3     ENABLED   BUSY-BlkAck      1 /   0           1       0
 3-4  tdcpcal1  UP      4     ENABLED                    0 /   0           1       0
 3-5  adcftof1  UP      5     ENABLED                    0 /   0           1       0
 3-6  tdcftof1  UP      6     ENABLED                    0 /   0           1       0
 3-7     trig2  UP     38     ENABLED                    0 /   0           1       0
 3-8   adccnd1  UP     73     ENABLED                    0 /   0           1       0
 4-1  adcecal2  UP      7     ENABLED   BUSY-BlkAck      1 /   0           1       0
 4-2  tdcecal2  UP      8     ENABLED                    0 /   0           1       0
 4-3  adcpcal2  UP      9     ENABLED                    0 /   0           1       0
 4-4  tdcpcal2  UP     10     ENABLED                    0 /   0           1       0
 4-5  adcftof2  UP     11     ENABLED                    0 /   0           1       0
 4-6  tdcftof2  UP     12     ENABLED                    0 /   0           1       0
 4-7  adcctof1  UP     59     ENABLED   BUSY-BlkAck      1 /   1          74      74
 4-8  tdcctof1  UP     60     ENABLED                    0 /   0           1       0
 5-1  adcecal3  UP     13     ENABLED                    0 /   0           1       0
 5-2  tdcecal3  UP     14     ENABLED                    0 /   0           1       0
 5-3  adcpcal3  UP     15     ENABLED   BUSY-BlkAck      1 /   0           1       0
 5-4  tdcpcal3  UP     16     ENABLED                    0 /   0           1       0
 5-5  adcftof3  UP     17     ENABLED                    0 /   0           1       0
 5-6  tdcftof3  UP     18     ENABLED                    0 /   0           1       0
 5-7      svt1  UP     61     ENABLED   BUSY-BlkAck      1 /   1           0       0
 5-8      svt2  UP     62     ENABLED                    0 /   0           1       0
 6-1  adcecal4  UP     19     ENABLED   BUSY-BlkAck      1 /   0           1       0
 6-2  tdcecal4  UP     20     ENABLED                    0 /   0           1       0
 6-3  adcpcal4  UP     21     ENABLED                    0 /   0           1       0
 6-4  tdcpcal4  UP     22     ENABLED                    0 /   0           1       0
 6-5  adcftof4  UP     23     ENABLED                    0 /   0           1       0
 6-6  tdcftof4  UP     24     ENABLED                    0 /   0           1       0
 7-1  adcecal5  UP     25     ENABLED   BUSY-BlkAck      1 /   1           0       0
 7-2  tdcecal5  UP     26     ENABLED                    0 /   0           1       0
 7-3  adcpcal5  UP     27     ENABLED   BUSY-BlkAck      1 /   1           0       0
 7-4  tdcpcal5  UP     28     ENABLED                    0 /   0           1       0
 7-5  adcftof5  UP     30     ENABLED                    0 /   0           1       0
 7-6  tdcftof5  UP     29     ENABLED                    0 /   0           1       0
 8-1  adcecal6  UP     31     ENABLED   BUSY-BlkAck      1 /   1           0       0
 8-2  tdcecal6  UP     32     ENABLED                    0 /   0           1       0
 8-3  adcpcal6  UP     33     ENABLED                    0 /   0           1       0
 8-4  tdcpcal6  UP     34     ENABLED                    0 /   0           1       0
 8-5  adcftof6  UP     35     ENABLED                    0 /   0           1       0
 8-6  tdcftof6  UP     36     ENABLED   BUSY-BlkAck      1 /   1           0       0
 9-1      dc11  UP     41     ENABLED                    0 /   0           1       0
 9-2      dc21  UP     44     ENABLED                    0 /   0           1       0
 9-3      dc31  UP     47     ENABLED                    0 /   0           1       0
 9-4      dc41  UP     50     ENABLED                    0 /   0           1       0
 9-5      dc51  UP     53     ENABLED                    0 /   0           1       0
 9-6      dc61  UP     56     ENABLED                    0 /   0           1       0
10-1      dc12  UP     42     ENABLED                    0 /   0           1       0
10-2      dc22  UP     45     ENABLED                    0 /   0           1       0
10-3      dc32  UP     48     ENABLED                    0 /   0           1       0
10-4      dc42  UP     51     ENABLED                    0 /   0           1       0
10-5      dc52  UP     54     ENABLED                    0 /   0           1       0
10-6      dc62  UP     57     ENABLED                    0 /   0           1       0
13-1      dc13  UP     43     ENABLED                    0 /   0           1       0
13-2      dc23  UP     46     ENABLED                    0 /   0           1       0
13-3      dc33  UP     49     ENABLED                    0 /   0           1       0
13-4      dc43  UP     52     ENABLED                    0 /   0           1       0
13-5      dc53  UP     55     ENABLED                    0 /   0           1       0
13-6      dc63  UP     58     ENABLED                    0 /   0           1       0
Total Slaves Added = 60
--------------------------------------------------------------------------------

Sergey B. Dec 2017 run: handle it better, send error and continue

net_thread:  waiting=    881    sending=      1 microsec per event (nev=28)
proc_thread: waiting=     35 processing=    818 microsec per event (nev=28)
net_thread:  waiting=    856    sending=      1 microsec per event (nev=28)
usrVme2MemDmaDone: ERROR: DMA terminated on master byte count,    however DCNT (1997952) != 0 
faReadBlock: WARN: DMA transfer terminated by word count 0x7a120
Segmentation fault
clasrun@adccnd1:clasrun>

Sergey B. Dec 2017 run:

dabufp[0] = 0x00002676 (9846)
dabufp[1] = 0x00010e01 (69121)
TI UNKNOWN data: [  0] 0x08010003
  Previous stuff
         [-20][  0] 0x84000bf2
         [-19][  0] 0x900003f2
         [-18][  0] 0x98000074
         [-17][  0] 0x00a432d5
         [-16][  0] 0xa026001f
         [-15][  0] 0x8c000006
         [-14][  0] 0xf8000000
         [-13][  0] 0xf8000000
         [-12][  0] 0x84400bf2
         [-11][  0] 0x900003f2
         [-10][  0] 0x98000074
         [ -9][  0] 0x00a432d5
         [ -8][  0] 0xa026001f
         [ -7][  0] 0x8c400006
         [ -6][  0] 0xf8000000
         [ -5][  0] 0xf8000000
         [ -4][  0] 0x0000006d
         [ -3][  0] 0x00010ef4
         [ -2][  0] 0x00000009
         [ -1][  0] 0xe10a013e
          [  0][  0] 0x08010003
          [  1][  0] 0x000003f4
          [  2][  0] 0xe94a05fb
          [  3][  0] 0x00000000
          [  4][  0] 0x8d400007
          [  5][  0] 0xfd4003f4
          [  6][  0] 0xfd4f1110
          [  7][  0] 0xfd4f1110
          [  8][  0] 0x00000061
          [  9][  0] 0xe104013e
  End Of Previous stuff
coda_roc is exiting, clear dma memory
runcontrol complains about adcft3 first, then about svt2
restarted svt2, download-prestart-go works

Sergey B. Dec 2017 run: check vtp readout in general

proc_thread: waiting=    164 processing=    937 microsec per event (nev=30)
net_thread:  waiting=     54    sending=      1 microsec per event (nev=606)
VTP UNKNOWN data2: [353] 0x000000d0
VTP UNKNOWN data2: [354] 0x00010ee5
VTP UNKNOWN data2: [355] 0x00000055
VTP UNKNOWN data2: [356] 0xe10a016c
VTP UNKNOWN data2: [357] 0x8540e514
VTP UNKNOWN data2: [359] 0x09010003
VTP UNKNOWN data2: [360] 0x000a11d1
VTP UNKNOWN data2: [361] 0xbbffc6ed
VTP UNKNOWN data2: [362] 0x00000002
VTP UNKNOWN data2: [363] 0x09010003
VTP UNKNOWN data2: [364] 0x000a11d2
VTP UNKNOWN data2: [365] 0xbbffe48f
VTP UNKNOWN data2: [366] 0x00000002
VTP UNKNOWN data2: [367] 0x0a010003
VTP UNKNOWN data2: [368] 0x000a11d3
VTP UNKNOWN data2: [369] 0xbc004d73
VTP UNKNOWN data2: [370] 0x00000002
VTP UNKNOWN data2: [371] 0x0b010003
VTP UNKNOWN data2: [372] 0x000a11d4
VTP UNKNOWN data2: [373] 0xbc008dec
VTP UNKNOWN data2: [374] 0x00000002
VTP UNKNOWN data2: [310] 0x00d0006a
VTP UNKNOWN data2: [339] 0x00d0009b
VTP UNKNOWN data2: [353] 0x000000d0
VTP UNKNOWN data2: [354] 0x00010ee5
VTP UNKNOWN data2: [355] 0x00000055
VTP UNKNOWN data2: [356] 0xe10a016c
VTP UNKNOWN data2: [357] 0x8540e514
VTP UNKNOWN data2: [359] 0x09010003
VTP UNKNOWN data2: [360] 0x000a11d1
VTP UNKNOWN data2: [361] 0xbbffc6ed
VTP UNKNOWN data2: [362] 0x00000002
VTP UNKNOWN data2: [363] 0x09010003
VTP UNKNOWN data2: [364] 0x000a11d2
VTP UNKNOWN data2: [365] 0xbbffe48f
VTP UNKNOWN data2: [366] 0x00000002
VTP UNKNOWN data2: [367] 0x0a010003
VTP UNKNOWN data2: [368] 0x000a11d3
VTP UNKNOWN data2: [369] 0xbc004d73
VTP UNKNOWN data2: [370] 0x00000002
VTP UNKNOWN data2: [371] 0x0b010003
VTP UNKNOWN data2: [372] 0x000a11d4
VTP UNKNOWN data2: [373] 0xbc008dec
VTP UNKNOWN data2: [374] 0x00000002
VTP UNKNOWN data2: [375] 0x20010003
VTP UNKNOWN data2: [376] 0x000a11d5
VTP UNKNOWN data2: [377] 0xbc00a9c1
Segmentation fault (core dumped)
clasrun@adcpcal2vtp:clasrun>

Sergey B. Dec 2017 run:

proc_thread: waiting=   1245 processing=    925 microsec per event (nev=23)
net_thread:  waiting=     53    sending=      1 microsec per event (nev=932)
ERROR in TI event header word: event type = 0
ERROR in TI event header word (0x45)
ERROR: TS has more then 7 data words - exit
clasrun@adcecal3vtp:clasrun>

Sergey B. Dec 2017 run: after reboot trig1, have to do clock setup on ts and td, ask Bryan ???

trig1 rebooted, after prestart (adcecal5 missing, wrong block_level):
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1    DISABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2    DISABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3    DISABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4    DISABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5    DISABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6    DISABLED                    0 /   0          20       0
 3-7     trig2  UP     38    DISABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73    DISABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7    DISABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8    DISABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9    DISABLED                    0 /   0           0       0
 4-4  tdcpcal2  UP     10    DISABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11    DISABLED                    0 /   0           0       0
 4-6  tdcftof2  UP     12    DISABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59    DISABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60    DISABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13    DISABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14    DISABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15    DISABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16    DISABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17    DISABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18    DISABLED                    0 /   0          20       0
 5-7      svt1  UP     61    DISABLED                    0 /   0           0       0
 5-8      svt2  UP     62    DISABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19    DISABLED                    0 /   0           0       0
 6-2  tdcecal4  UP     20    DISABLED                    0 /   0           0       0
 6-3  adcpcal4  UP     21    DISABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22    DISABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23    DISABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24    DISABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26    DISABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27    DISABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28    DISABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30    DISABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29    DISABLED                    0 /   0          20       0
 7-7    adcft1  UP     70    DISABLED                    0 /   0          20       0
 7-8    adcft2  UP     71    DISABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31    DISABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32    DISABLED                    0 /   0           0       0
 8-3  adcpcal6  UP     33    DISABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34    DISABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35    DISABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36    DISABLED                    0 /   0          20       0
 8-7    adcft3  UP     72    DISABLED                    0 /   0          20       0
 9-1      dc11  UP     41    DISABLED                    0 /   0          20       0
 9-2      dc21  UP     44    DISABLED                    0 /   0          20       0
 9-3      dc31  UP     47    DISABLED                    0 /   0          20       0
 9-4      dc41  UP     50    DISABLED                    0 /   0          20       0
 9-5      dc51  UP     53    DISABLED                    0 /   0          20       0
 9-6      dc61  UP     56    DISABLED                    0 /   0          20       0
10-1      dc12  UP     42    DISABLED                    0 /   0          20       0
10-2      dc22  UP     45    DISABLED                    0 /   0           0       0
10-3      dc32  UP     48    DISABLED                    0 /   0          20       0
10-4      dc42  UP     51    DISABLED                    0 /   0          20       0
10-5      dc52  UP     54    DISABLED                    0 /   0          20       0
10-6      dc62  UP     57    DISABLED                    0 /   0          20       0
13-1      dc13  UP     43    DISABLED                    0 /   0           0       0
13-2      dc23  UP     46    DISABLED                    0 /   0          20       0
13-3      dc33  UP     49    DISABLED                    0 /   0          20       0
13-4      dc43  UP     52    DISABLED                    0 /   0          20       0
13-5      dc53  UP     55    DISABLED                    0 /   0          20       0
13-6      dc63  UP     58    DISABLED                    0 /   0          20       0
Total Slaves Added = 62
--------------------------------------------------------------------------------


AFTER SECOND RESET_DOWNLOAD_PRESTART:
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    --------   V----   ---12345678   ----------  0x41a56800   N
 4     12345678    --------   V----   ---12345678   ----------  0xbc456800   N
 5     12345678    --------   V----   ---12345678   ----------  0x8cc56800   N
 6     123456-8    --------   V----   ---123456--   ----------  0x6b856800   N
 7     12345678    --------   V----   ---12345678   ----------  0x63856800   N
 8     12345678    --------   V----   ---1234567-   ----------  0xd9456800   N
 9     123456--    --------   V----   ---123456--   ----------  0x79a56800   N
10     123456--    --------   V----   ---123456--   ----------  0xe0e56800   N
13     1234567-    --------   V----   ---123456--   ----------  0x3c856800   N
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1    DISABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2    DISABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3    DISABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4    DISABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5    DISABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6    DISABLED                    0 /   0          20       0
 3-7     trig2  UP     38    DISABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73    DISABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7    DISABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8    DISABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9    DISABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10    DISABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11    DISABLED                    0 /   0          20       0
 4-6  tdcftof2  UP     12    DISABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59    DISABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60    DISABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13    DISABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14    DISABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15    DISABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16    DISABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17    DISABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18    DISABLED                    0 /   0          20       0
 5-7      svt1  UP     61    DISABLED                    0 /   0          20       0
 5-8      svt2  UP     62    DISABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19    DISABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20    DISABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21    DISABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22    DISABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23    DISABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24    DISABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25    DISABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26    DISABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27    DISABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28    DISABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30    DISABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29    DISABLED                    0 /   0          20       0
 7-7    adcft1  UP     70    DISABLED                    0 /   0          20       0
 7-8    adcft2  UP     71    DISABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31    DISABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32    DISABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33    DISABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34    DISABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35    DISABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36    DISABLED                    0 /   0          20       0
 8-7    adcft3  UP     72    DISABLED                    0 /   0          20       0
 9-1      dc11  UP     41    DISABLED                    0 /   0          20       0
 9-2      dc21  UP     44    DISABLED                    0 /   0          20       0
 9-3      dc31  UP     47    DISABLED                    0 /   0          20       0
 9-4      dc41  UP     50    DISABLED                    0 /   0          20       0
 9-5      dc51  UP     53    DISABLED                    0 /   0          20       0
 9-6      dc61  UP     56    DISABLED                    0 /   0          20       0
10-1      dc12  UP     42    DISABLED                    0 /   0          20       0
10-2      dc22  UP     45    DISABLED                    0 /   0          20       0
10-3      dc32  UP     48    DISABLED                    0 /   0          20       0
10-4      dc42  UP     51    DISABLED                    0 /   0          20       0
10-5      dc52  UP     54    DISABLED                    0 /   0          20       0
10-6      dc62  UP     57    DISABLED                    0 /   0          20       0
13-1      dc13  UP     43    DISABLED                    0 /   0          20       0
13-2      dc23  UP     46    DISABLED                    0 /   0          20       0
13-3      dc33  UP     49    DISABLED                    0 /   0          20       0
13-4      dc43  UP     52    DISABLED                    0 /   0          20       0
13-5      dc53  UP     55    DISABLED                    0 /   0          20       0
13-6      dc63  UP     58    DISABLED                    0 /   0          20       0
Total Slaves Added = 63
--------------------------------------------------------------------------------


AFTER GO:


TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    12345678   V----   ---12345678   ----------  0xfeb56400   N
 4     12345678    12345678   V----   ---12345678   ----------  0x69f56400   N
 5     12345678    12345678   V----   ---12345678   ----------  0x2ad56400   N
 6     123456-8    123456--   V----   ---123456--   ----------  0xfa156400   N
 7     12345678    12345678   V----   ---12345678   ----------  0xe2956400   N
 8     12345678    1234567-   V----   ---1234567-   ----3-----  0x48f56400   N
 9     123456--    123456--   V----   ---123456--   ----------  0xd9b56400   N
10     123456--    123456--   V----   ---123456--   ----------  0x31756400   N
13     1234567-    123456--   V----   ---123456--   ----------  0x7db56400   N
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1     ENABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2     ENABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3     ENABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4     ENABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5     ENABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6     ENABLED                    0 /   0          20       0
 3-7     trig2  UP     38     ENABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73     ENABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7     ENABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8     ENABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9     ENABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10     ENABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11     ENABLED                    0 /   0          20       0
 4-6  tdcftof2  UP     12     ENABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59     ENABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60     ENABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13     ENABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14     ENABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15     ENABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16     ENABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17     ENABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18     ENABLED                    0 /   0          20       0
 5-7      svt1  UP     61     ENABLED                    0 /   0          20       0
 5-8      svt2  UP     62     ENABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19     ENABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20     ENABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21     ENABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22     ENABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23     ENABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24     ENABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25     ENABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26     ENABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27     ENABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28     ENABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30     ENABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29     ENABLED                    0 /   0          20       0
 7-7    adcft1  UP     70     ENABLED                    0 /   0          20       0
 7-8    adcft2  UP     71     ENABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31     ENABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32     ENABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33     ENABLED   BUSY-BlkAck      4 /   0          20       0
 8-4  tdcpcal6  UP     34     ENABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35     ENABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36     ENABLED                    0 /   0          20       0
 8-7    adcft3  UP     72     ENABLED                    0 /   0          20       0
 9-1      dc11  UP     41     ENABLED                    0 /   0          20       0
 9-2      dc21  UP     44     ENABLED                    0 /   0          20       0
 9-3      dc31  UP     47     ENABLED                    0 /   0          20       0
 9-4      dc41  UP     50     ENABLED                    0 /   0          20       0
 9-5      dc51  UP     53     ENABLED                    0 /   0          20       0
 9-6      dc61  UP     56     ENABLED                    0 /   0          20       0
10-1      dc12  UP     42     ENABLED                    0 /   0          20       0
10-2      dc22  UP     45     ENABLED                    0 /   0          20       0
10-3      dc32  UP     48     ENABLED                    0 /   0          20       0
10-4      dc42  UP     51     ENABLED                    0 /   0          20       0
10-5      dc52  UP     54     ENABLED                    0 /   0          20       0
10-6      dc62  UP     57     ENABLED                    0 /   0          20       0
13-1      dc13  UP     43     ENABLED                    0 /   0          20       0
13-2      dc23  UP     46     ENABLED                    0 /   0          20       0
13-3      dc33  UP     49     ENABLED                    0 /   0          20       0
13-4      dc43  UP     52     ENABLED                    0 /   0          20       0
13-5      dc53  UP     55     ENABLED                    0 /   0          20       0
13-6      dc63  UP     58     ENABLED                    0 /   0          20       0
Total Slaves Added = 63
--------------------------------------------------------------------------------
RESET-DOWNLOAD-PRESTART
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    --------   V----   ---12345678   ----------  0x03857000   N
 4     12345678    --------   V----   ---12345678   ----------  0x6ec57000   N
 5     12345678    --------   V----   ---12345678   ----------  0x2fa57000   N
 6     123456-8    --------   V----   ---123456--   ----------  0xfee57000   N
 7     12345678    --------   V----   ---12345678   ----------  0xe7657000   N
 8     12345678    --------   V----   ---1234567-   ----------  0x4dc57000   N
 9     123456--    --------   V----   ---123456--   ----------  0xde857000   N
10     123456--    --------   V----   ---123456--   ----------  0x36457000   N
13     1234567-    --------   V----   ---123456--   ----------  0x82857000   N
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1    DISABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2    DISABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3    DISABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4    DISABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5    DISABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6    DISABLED                    0 /   0          20       0
 3-7     trig2  UP     38    DISABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73    DISABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7    DISABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8    DISABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9    DISABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10    DISABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11    DISABLED                    0 /   0          20       0
 4-6  tdcftof2  UP     12    DISABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59    DISABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60    DISABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13    DISABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14    DISABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15    DISABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16    DISABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17    DISABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18    DISABLED                    0 /   0          20       0
 5-7      svt1  UP     61    DISABLED                    0 /   0          20       0
 5-8      svt2  UP     62    DISABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19    DISABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20    DISABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21    DISABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22    DISABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23    DISABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24    DISABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25    DISABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26    DISABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27    DISABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28    DISABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30    DISABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29    DISABLED                    0 /   0          20       0
 7-7    adcft1  UP     70    DISABLED                    0 /   0          20       0
 7-8    adcft2  UP     71    DISABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31    DISABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32    DISABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33    DISABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34    DISABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35    DISABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36    DISABLED                    0 /   0          20       0
 8-7    adcft3  UP     72    DISABLED                    0 /   0          20       0
 9-1      dc11  UP     41    DISABLED                    0 /   0          20       0
 9-2      dc21  UP     44    DISABLED                    0 /   0          20       0
 9-3      dc31  UP     47    DISABLED                    0 /   0          20       0
 9-4      dc41  UP     50    DISABLED                    0 /   0          20       0
 9-5      dc51  UP     53    DISABLED                    0 /   0          20       0
 9-6      dc61  UP     56    DISABLED                    0 /   0          20       0
10-1      dc12  UP     42    DISABLED                    0 /   0          20       0
10-2      dc22  UP     45    DISABLED                    0 /   0          20       0
10-3      dc32  UP     48    DISABLED                    0 /   0          20       0
10-4      dc42  UP     51    DISABLED                    0 /   0          20       0
10-5      dc52  UP     54    DISABLED                    0 /   0          20       0
10-6      dc62  UP     57    DISABLED                    0 /   0          20       0
13-1      dc13  UP     43    DISABLED                    0 /   0          20       0
13-2      dc23  UP     46    DISABLED                    0 /   0          20       0
13-3      dc33  UP     49    DISABLED                    0 /   0          20       0
13-4      dc43  UP     52    DISABLED                    0 /   0          20       0
13-5      dc53  UP     55    DISABLED                    0 /   0          20       0
13-6      dc63  UP     58    DISABLED                    0 /   0          20       0
Total Slaves Added = 63
--------------------------------------------------------------------------------


GO
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    12345678   V----   ---12345678   ----------  0xd1505c00   N
 4     12345678    12345678   V----   ---12345678   ------5---  0x3cb05c00   N
 5     12345678    12345678   V----   ---12345678   ----------  0xfdb05c00   N
 6     123456-8    123456--   V----   ---123456--   ----------  0x81806c00   N
 7     12345678    12345678   V----   ---12345678   ----------  0xb5505c00   N
 8     12345678    1234567-   V----   ---1234567-   ----------  0x1bb05c00   N
 9     123456--    123456--   V----   ---123456--   ----------  0xac905c00   N
10     123456--    123456--   V----   ---123456--   ----------  0x04505c00   N
13     1234567-    123456--   V----   ---123456--   ----------  0x50705c00   N
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1     ENABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2     ENABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3     ENABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4     ENABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5     ENABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6     ENABLED                    0 /   0          20       0
 3-7     trig2  UP     38     ENABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73     ENABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7     ENABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8     ENABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9     ENABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10     ENABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11     ENABLED   BUSY-BlkAck      4 /   0          20       0
 4-6  tdcftof2  UP     12     ENABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59     ENABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60     ENABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13     ENABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14     ENABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15     ENABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16     ENABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17     ENABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18     ENABLED                    0 /   0          20       0
 5-7      svt1  UP     61     ENABLED                    0 /   0          20       0
 5-8      svt2  UP     62     ENABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19     ENABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20     ENABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21     ENABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22     ENABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23     ENABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24     ENABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25     ENABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26     ENABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27     ENABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28     ENABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30     ENABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29     ENABLED                    0 /   0          20       0
 7-7    adcft1  UP     70     ENABLED                    0 /   0          20       0
 7-8    adcft2  UP     71     ENABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31     ENABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32     ENABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33     ENABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34     ENABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35     ENABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36     ENABLED                    0 /   0          20       0
 8-7    adcft3  UP     72     ENABLED                    0 /   0          20       0
 9-1      dc11  UP     41     ENABLED                    0 /   0          20       0
 9-2      dc21  UP     44     ENABLED                    0 /   0          20       0
 9-3      dc31  UP     47     ENABLED                    0 /   0          20       0
 9-4      dc41  UP     50     ENABLED                    0 /   0          20       0
 9-5      dc51  UP     53     ENABLED                    0 /   0          20       0
 9-6      dc61  UP     56     ENABLED                    0 /   0          20       0
10-1      dc12  UP     42     ENABLED                    0 /   0          20       0
10-2      dc22  UP     45     ENABLED                    0 /   0          20       0
10-3      dc32  UP     48     ENABLED                    0 /   0          20       0
10-4      dc42  UP     51     ENABLED                    0 /   0          20       0
10-5      dc52  UP     54     ENABLED                    0 /   0          20       0
10-6      dc62  UP     57     ENABLED                    0 /   0          20       0
13-1      dc13  UP     43     ENABLED                    0 /   0          20       0
13-2      dc23  UP     46     ENABLED                    0 /   0          20       0
13-3      dc33  UP     49     ENABLED                    0 /   0          20       0
13-4      dc43  UP     52     ENABLED                    0 /   0          20       0
13-5      dc53  UP     55     ENABLED                    0 /   0          20       0
13-6      dc63  UP     58     ENABLED                    0 /   0          20       0
Total Slaves Added = 63
--------------------------------------------------------------------------------
roc_reboot adcftof2
DOWNLOAD-PRESTART-GO:


TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    12345678   V----   ---12345678   ----------  0x747d1400   N
 4     12345678    12345678   V----   ---12345678   ----------  0xef5d1400   N
 5     12345678    12345678   V----   ---12345678   --------7-  0xbfbd1400   N
 6     123456-8    123456--   V----   ---123456--   ----------  0xf2205c00   N
 7     12345678    12345678   V----   ---12345678   ----------  0x965d1400   N
 8     12345678    1234567-   V----   ---1234567-   ----------  0x0c3d1400   N
 9     123456--    123456--   V----   ---123456--   ----------  0xac9d1400   N
10     123456--    123456--   V----   ---123456--   ----------  0x13dd1400   N
13     1234567-    123456--   V----   ---123456--   ----------  0x6f7d1400   N  
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1     ENABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2     ENABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3     ENABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4     ENABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5     ENABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6     ENABLED                    0 /   0          20       0
 3-7     trig2  UP     38     ENABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73     ENABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7     ENABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8     ENABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9     ENABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10     ENABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11     ENABLED                    0 /   0          20       0
 4-6  tdcftof2  UP     12     ENABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59     ENABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60     ENABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13     ENABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14     ENABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15     ENABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16     ENABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17     ENABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18     ENABLED                    0 /   0          20       0
 5-7      svt1  UP     61     ENABLED   BUSY-BlkAck     16 /   0          20       0
 5-8      svt2  UP     62     ENABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19     ENABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20     ENABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21     ENABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22     ENABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23     ENABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24     ENABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25     ENABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26     ENABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27     ENABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28     ENABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30     ENABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29     ENABLED                    0 /   0          20       0
 7-7    adcft1  UP     70     ENABLED                    0 /   0          20       0
 7-8    adcft2  UP     71     ENABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31     ENABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32     ENABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33     ENABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34     ENABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35     ENABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36     ENABLED                    0 /   0          20       0
 8-7    adcft3  UP     72     ENABLED                    0 /   0          20       0
 9-1      dc11  UP     41     ENABLED                    0 /   0          20       0
 9-2      dc21  UP     44     ENABLED                    0 /   0          20       0
 9-3      dc31  UP     47     ENABLED                    0 /   0          20       0
 9-4      dc41  UP     50     ENABLED                    0 /   0          20       0
 9-5      dc51  UP     53     ENABLED                    0 /   0          20       0
 9-6      dc61  UP     56     ENABLED                    0 /   0          20       0
10-1      dc12  UP     42     ENABLED                    0 /   0          20       0
10-2      dc22  UP     45     ENABLED                    0 /   0          20       0
10-3      dc32  UP     48     ENABLED                    0 /   0          20       0
10-4      dc42  UP     51     ENABLED                    0 /   0          20       0
10-5      dc52  UP     54     ENABLED                    0 /   0          20       0
10-6      dc62  UP     57     ENABLED                    0 /   0          20       0
13-1      dc13  UP     43     ENABLED                    0 /   0          20       0
13-2      dc23  UP     46     ENABLED                    0 /   0          20       0
13-3      dc33  UP     49     ENABLED                    0 /   0          20       0
13-4      dc43  UP     52     ENABLED                    0 /   0          20       0
13-5      dc53  UP     55     ENABLED                    0 /   0          20       0
13-6      dc63  UP     58     ENABLED                    0 /   0          20       0
Total Slaves Added = 63
--------------------------------------------------------------------------------
reboot svt1
GO
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    12345678   V----   ---12345678   ----------  0x76d04800   N
 4     12345678    12345678   V----   ---12345678   ------5---  0xe2504800   N
 5     12345678    12345678   V----   ---12345678   ----------  0xa4e06c00   N
 6     123456-8    123456--   V----   ---123456--   ----------  0x34d07800   N
 7     12345678    12345678   V----   ---12345678   ----------  0x5c806c00   N
 8     12345678    1234567-   V----   ---1234567-   ----------  0xc1304800   N
 9     123456--    123456--   V----   ---123456--   ----------  0x52104800   N
10     123456--    123456--   V----   ---123456--   ----------  0xa9d04800   N
13     1234567-    123456--   V----   ---123456--   ----------  0xf6104800   N   
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1     ENABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2     ENABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3     ENABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4     ENABLED                    0 /   0          20       0
 3-5  adcftof1  UP      5     ENABLED                    0 /   0          20       0
 3-6  tdcftof1  UP      6     ENABLED                    0 /   0          20       0
 3-7     trig2  UP     38     ENABLED                    0 /   0          20       0
 3-8   adccnd1  UP     73     ENABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7     ENABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8     ENABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9     ENABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10     ENABLED                    0 /   0          20       0
 4-5  adcftof2  UP     11     ENABLED   BUSY-BlkAck      4 /   0          20       0
 4-6  tdcftof2  UP     12     ENABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59     ENABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60     ENABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13     ENABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14     ENABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15     ENABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16     ENABLED                    0 /   0          20       0
 5-5  adcftof3  UP     17     ENABLED                    0 /   0          20       0
 5-6  tdcftof3  UP     18     ENABLED                    0 /   0          20       0
 5-7      svt1  UP     61     ENABLED                    0 /   0          20       0
 5-8      svt2  UP     62     ENABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19     ENABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20     ENABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21     ENABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22     ENABLED                    0 /   0          20       0
 6-5  adcftof4  UP     23     ENABLED                    0 /   0          20       0
 6-6  tdcftof4  UP     24     ENABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25     ENABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26     ENABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27     ENABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28     ENABLED                    0 /   0          20       0
 7-5  adcftof5  UP     30     ENABLED                    0 /   0          20       0
 7-6  tdcftof5  UP     29     ENABLED                    0 /   0          20       0
 7-7    adcft1  UP     70     ENABLED                    0 /   0          20       0
 7-8    adcft2  UP     71     ENABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31     ENABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32     ENABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33     ENABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34     ENABLED                    0 /   0          20       0
 8-5  adcftof6  UP     35     ENABLED                    0 /   0          20       0
 8-6  tdcftof6  UP     36     ENABLED                    0 /   0          20       0
 8-7    adcft3  UP     72     ENABLED                    0 /   0          20       0
 9-1      dc11  UP     41     ENABLED                    0 /   0          20       0
 9-2      dc21  UP     44     ENABLED                    0 /   0          20       0
 9-3      dc31  UP     47     ENABLED                    0 /   0          20       0
 9-4      dc41  UP     50     ENABLED                    0 /   0          20       0
 9-5      dc51  UP     53     ENABLED                    0 /   0          20       0
 9-6      dc61  UP     56     ENABLED                    0 /   0          20       0
10-1      dc12  UP     42     ENABLED                    0 /   0          20       0
10-2      dc22  UP     45     ENABLED                    0 /   0          20       0
10-3      dc32  UP     48     ENABLED                    0 /   0          20       0
10-4      dc42  UP     51     ENABLED                    0 /   0          20       0
10-5      dc52  UP     54     ENABLED                    0 /   0          20       0
10-6      dc62  UP     57     ENABLED                    0 /   0          20       0
13-1      dc13  UP     43     ENABLED                    0 /   0          20       0
13-2      dc23  UP     46     ENABLED                    0 /   0          20       0
13-3      dc33  UP     49     ENABLED                    0 /   0          20       0
13-4      dc43  UP     52     ENABLED                    0 /   0          20       0
13-5      dc53  UP     55     ENABLED                    0 /   0          20       0
13-6      dc63  UP     58     ENABLED                    0 /   0          20       0
Total Slaves Added = 63
--------------------------------------------------------------------------------


roc_reboot all rocs
have to reboot adcecal2 twice because of vtp did not come up
load FC_HTCC_VTP
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    --------   V----   ---1234----   ----------  0xcde56800   N
 4     12345678    --------   V----   ---1234--78   ----------  0xcee56800   N
 5     12345678    --------   V----   ---1234----   ----------  0x25a56800   N
 6     123456-8    --------   V----   ---1234----   ----------  0x89c56800   N
 7     12345678    --------   V----   ---1234----   ----------  0x08c56800   N
 8     12345678    --------   V----   ---12-4----   ----------  0x02c56800   N
 9     123456--    --------   V----   -----------   ----------  0x26a56800   N
10     123456--    --------   V----   -----------   ----------  0x16656800   N
13     1234567-    --------   V----   -----------   ----------  0xfa856800   N 
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1    DISABLED                    0 /   0           0       0
 3-2  tdcecal1  UP      2    DISABLED                    0 /   0           0       0
 3-3  adcpcal1  UP      3    DISABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4    DISABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7    DISABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8    DISABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9    DISABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10    DISABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59    DISABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60    DISABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13    DISABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14    DISABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15    DISABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16    DISABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19    DISABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20    DISABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21    DISABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22    DISABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25    DISABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26    DISABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27    DISABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28    DISABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31    DISABLED                    0 /   0           0       0
 8-2  tdcecal6  UP     32    DISABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34    DISABLED                    0 /   0          20       0   
Total Slaves Added = 25
--------------------------------------------------------------------------------
do not try GO, DOWNLOAD again
TD Module Status Summary
             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck
Slot  (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678)  Received  Wait
--------------------------------------------------------------------------------
 3     12345678    --------   V----   ---1234--7-   ----------  0xc0f56400   N
 4     12345678    --------   V----   ---1234--78   ----------  0x2c956400   N
 5     12345678    --------   V----   ---1234----   ----------  0xedf56400   N
 6     123456-8    --------   V----   ---1234----   ----------  0xbc956400   N
 7     12345678    --------   V----   ---1234----   ----------  0xa6156400   N
 8     12345678    --------   V----   ---1234----   ----------  0x0ab56400   N
 9     123456--    --------   V----   -----------   ----------  0x99356400   N
10     123456--    --------   V----   -----------   ----------  0xf3956400   N
13     1234567-    --------   V----   -----------   ----------  0x42556400   N 
TD Port STATUS Summary
                                                     Block Status      Block  Buffer
Sl-P  RocName  COMM  ROCID  TrigSrcEn   Busy Status  Ready / NeedAck    Level   Level
--------------------------------------------------------------------------------
 3-1  adcecal1  UP      1    DISABLED                    0 /   0          20       0
 3-2  tdcecal1  UP      2    DISABLED                    0 /   0          20       0
 3-3  adcpcal1  UP      3    DISABLED                    0 /   0          20       0
 3-4  tdcpcal1  UP      4    DISABLED                    0 /   0          20       0
 3-7     trig2  UP     38    DISABLED                    0 /   0          20       0
 4-1  adcecal2  UP      7    DISABLED                    0 /   0          20       0
 4-2  tdcecal2  UP      8    DISABLED                    0 /   0          20       0
 4-3  adcpcal2  UP      9    DISABLED                    0 /   0          20       0
 4-4  tdcpcal2  UP     10    DISABLED                    0 /   0          20       0
 4-7  adcctof1  UP     59    DISABLED                    0 /   0          20       0
 4-8  tdcctof1  UP     60    DISABLED                    0 /   0          20       0
 5-1  adcecal3  UP     13    DISABLED                    0 /   0          20       0
 5-2  tdcecal3  UP     14    DISABLED                    0 /   0          20       0
 5-3  adcpcal3  UP     15    DISABLED                    0 /   0          20       0
 5-4  tdcpcal3  UP     16    DISABLED                    0 /   0          20       0
 6-1  adcecal4  UP     19    DISABLED                    0 /   0          20       0
 6-2  tdcecal4  UP     20    DISABLED                    0 /   0          20       0
 6-3  adcpcal4  UP     21    DISABLED                    0 /   0          20       0
 6-4  tdcpcal4  UP     22    DISABLED                    0 /   0          20       0
 7-1  adcecal5  UP     25    DISABLED                    0 /   0          20       0
 7-2  tdcecal5  UP     26    DISABLED                    0 /   0          20       0
 7-3  adcpcal5  UP     27    DISABLED                    0 /   0          20       0
 7-4  tdcpcal5  UP     28    DISABLED                    0 /   0          20       0
 8-1  adcecal6  UP     31    DISABLED                    0 /   0          20       0
 8-2  tdcecal6  UP     32    DISABLED                    0 /   0          20       0
 8-3  adcpcal6  UP     33    DISABLED                    0 /   0          20       0
 8-4  tdcpcal6  UP     34    DISABLED                    0 /   0          20       0  
Total Slaves Added = 27
--------------------------------------------------------------------------------
RUNNING !!!!!!!!!!!!!!!!!!!!! PIECE OF SSSSSSSSSSSSSSSSS...........


svt1 still gives busy shortly after go:
clasrun@clonpc19:clasrun> tcpClient svt1 'tiStatus(1)'
STATUS for TI at VME (Local) base address 0x00a80000 (0xac02c000) 
--------------------------------------------------------------------------------
A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5ac000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 1715
    L1A Count: 17150
  Block Limit: 0   
  Block Count: 1715
Registers (offset):
 boardID        (0x0000) = 0x7101153d    fiber          (0x0004) = 0x020311ff
 intsetup       (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
 adr32          (0x0010) = 0x08000000    blocklevel     (0x0014) = 0x0a0a0001
 dataFormat     (0x0018) = 0x08080006    vmeControl     (0x001c) = 0x00000011
 trigger        (0x0020) = 0x01000012    sync           (0x0024) = 0x00000c82
 busy           (0x0028) = 0x00000002    clock          (0x002c) = 0x00000002
 blockBuffer    (0x0034) = 0x00000008    output         (0x004c) = 0x00000000
 fiberSyncDelay (0x0050) = 0x4f004f13    nblocks        (0x0094) = 0x000006b3
 GTPStatusA     (0x00b0) = 0x02801fff    GTPStatusB     (0x00b4) = 0xff07ff07
 livetime       (0x00a8) = 0x01a7f1ad    busytime       (0x00ac) = 0x00000000
 GTPTrgBufLen   (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x00000007
 triggerRule    (0x0038) = 0x8a010182    triggerRuleMin (0x0138) = 0x00000000


Block Level        = 10 
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
Fiber Status         1     2     3     4     5     6     7     8
                   ----- ----- ----- ----- ----- ----- ----- -----
 Connected          YES   YES                                       


Clock Source (2) = 
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source = 
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source = 
  Switch Slot B    
Interrupts DISABLED
  Level = 5   Vector = 0xc8
Bus Errors Enabled
Blocks ready for readout: 0
Input counter 0
--------------------------------------------------------------------------------


clasrun@clonpc19:clasrun> tcpClient svt1 'tiStatus(2)'
STATUS for TI at VME (Local) base address 0x00a80000 (0xac02c000) 
--------------------------------------------------------------------------------
A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5ac000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 1715
    L1A Count: 17150
  Block Limit: 0   
  Block Count: 1715
Registers (offset):
 boardID        (0x0000) = 0x7101153d    fiber          (0x0004) = 0x020311ff
 intsetup       (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
 adr32          (0x0010) = 0x08000000    blocklevel     (0x0014) = 0x0a0a0001
 dataFormat     (0x0018) = 0x08080006    vmeControl     (0x001c) = 0x00000011
 trigger        (0x0020) = 0x01000012    sync           (0x0024) = 0x00000c82
 busy           (0x0028) = 0x00000002    clock          (0x002c) = 0x00000002
 blockBuffer    (0x0034) = 0x00000008    output         (0x004c) = 0x00000000
 fiberSyncDelay (0x0050) = 0x4f004f13    nblocks        (0x0094) = 0x000006b3
 GTPStatusA     (0x00b0) = 0x02801fff    GTPStatusB     (0x00b4) = 0xff07ff07
 livetime       (0x00a8) = 0x01b0e3ad    busytime       (0x00ac) = 0x00000000
 GTPTrgBufLen   (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x00000007
 triggerRule    (0x0038) = 0x8a010182    triggerRuleMin (0x0138) = 0x00000000


Block Level        = 10 
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
Fiber Status         1     2     3     4     5     6     7     8
                   ----- ----- ----- ----- ----- ----- ----- -----
 Connected          YES   YES                                       


Clock Source (2) = 
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source = 
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source = 
  Switch Slot B    
Interrupts DISABLED
  Level = 5   Vector = 0xc8
Bus Errors Enabled
Blocks ready for readout: 0
Input counter 0
--------------------------------------------------------------------------------


clasrun@clonpc19:clasrun> tcpClient svt1 'tiStatus(1)' >svt1
clasrun@clonpc19:clasrun> tcpClient svt2 'tiStatus(1)' > svt2
clasrun@clonpc19:clasrun> diff svt1 svt2
2c2
< STATUS for TI at VME (Local) base address 0x00a80000 (0xac02c000) 
---
> STATUS for TI at VME (Local) base address 0x00a80000 (0xac020000) 
4c4
<  A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5ac000)
---
>  A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac5a0000)
13c13
<   boardID        (0x0000) = 0x7101153d          fiber          (0x0004) = 0x020311ff
---
>   boardID        (0x0000) = 0x7101153e          fiber          (0x0004) = 0x101111ff
18,22c18,22
<   busy           (0x0028) = 0x00000002          clock          (0x002c) = 0x00000002
<   blockBuffer    (0x0034) = 0x00000008          output         (0x004c) = 0x00000000
<   fiberSyncDelay (0x0050) = 0x4f004f13          nblocks        (0x0094) = 0x000006b3
<   GTPStatusA     (0x00b0) = 0x02801fff          GTPStatusB     (0x00b4) = 0xff07ff07
<   livetime       (0x00a8) = 0x01dfb7ea          busytime       (0x00ac) = 0x00000000
---
>   busy           (0x0028) = 0x00010002          clock          (0x002c) = 0x00000002
>   blockBuffer    (0x0034) = 0x00000001          output         (0x004c) = 0x00000000
>   fiberSyncDelay (0x0050) = 0x4f004f15          nblocks        (0x0094) = 0x000006b3
>   GTPStatusA     (0x00b0) = 0x10801fff          GTPStatusB     (0x00b4) = 0xff11ff11
>   livetime       (0x00a8) = 0x01e80db3          busytime       (0x00ac) = 0x00000000
24c24
<   triggerRule    (0x0038) = 0x8a010182          triggerRuleMin (0x0138) = 0x00000000
---
>   triggerRule    (0x0038) = 0x0101010a          triggerRuleMin (0x0138) = 0x00000000
32c32
<   Connected          YES   YES                                       
---
>   Connected          YES                     YES                     



if it possible that svt1 was the only problem from the beginning, and TD info about other crates misleading ????????????

Sergey B. Dec 2017 run:

adccnd1:

proc_thread: waiting= 8924 processing= 819 microsec per event (nev=10) net_thread: waiting= 973 sending= 0 microsec per event (nev=102) usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00700000 < 0x08000000)

tiReadBlock: ERROR: vmeDmaDone returned an Error ERROR in tiReadBlock : No data or error, len = -1 FADCs: no events stat=0 intcount = 0 gbready = 0x00000000 fadcSlotMask = 0x000027f8 Missing slots: 3 4 5 6 7 8 9 10 13 FADCs: no events stat=0 intcount = 0 gbready = 0x00000000 fadcSlotMask = 0x000027f8 Missing slots: 3 4 5 6 7 8 9 10 13 FADCs: no events stat=0 intcount = 0 gbready = 0x00000000 fadcSlotMask = 0x000027f8 Missing slots: 3 4 5 6 7 8 9 10 13 ERROR1: LINK_sized_write() returns errno=32 (cc=-1, sizeof(nbytes)=4(424), netlong=2818637824) ERROR: net_thread failed (in LINK_sized_write). FADCs: no events stat=0 intcount = 0 gbready = 0x00000000 fadcSlotMask = 0x000027f8 Missing slots: 3 4 5 6 7 8 9 10 13 ERROR: bigproc.failure=0, bignet.failure=1

Calls bb_cleanup. Calls bb_cleanup. Calls bb_cleanup.


Sergey B. Dec 2017 run: send to Bryan

adcpcal5:

tiReload:

FPGA Re-Load ! 

CODAtcpServer: wait: coda request >download PROD< in progress

 ID: 0x7101    Firmware (type - revision): 0x3 - 0x076 (prodID = 1)

... Configure as TI Slave... tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1) tiSetClockSource: ERROR Setting Clock Source (clkset = 0x2, clkread = 0x0) tiSetTriggerSource: ERROR: Invalid Trigger Source (129). Must be between 0 and 11. FiberMeas: ERROR: Unable to measure fiber latency without HFBR1/5 as Clock Source

        Using default Fiber Sync Delay = 522133248 (0x1f1f1f00)tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,

tiInit: Scanning for TI... jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0xa00000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x100000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x600000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x880000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x900000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x980000 tiInit: ERROR: Unable to find TI cannot find TI, ret=-1 - exit coda_roc is exiting, clear dma memory


try again without roc_reboot:

LAST COMPILED: Dec 10 2017 12:24:00 USRSTRING >usr<

vmeCheckMutexHealth: Checking health of vmeBus shared mutex...

* vmeBusTryLock: ERROR: EBUSY
* vmeBusUnlock: ERROR: EPERM 
* vmeBusTimedLock: ERROR: ETIMEDOUT
* vmeBusMutexInit: Initializing vmeBus mutex

vmeCheckMutexHealth: Mutex Clean and Unlocked usrVmeDmaInit: userMemBase = 0xa07df000, physMemBase = 0x1ba00000, memSize = 1048576 bytes tiprimarytinit: i2_from_rol1 = 0xa07df000 tiprimarytinit: i2_from_rol1 = 0xa07df000 tiprimarytinit: i2_from_rol1 = 0xa07df010 TI VME (Local) address = 0x00a80000 (0xac014000) tiInit: ERROR: Invalid Board ID: 0x0 (rval = 0x00000000) tiInit: Scanning for TI... jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0xa00000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x100000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x600000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x880000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x900000 jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x980000 tiInit: ERROR: Unable to find TI cannot find TI, ret=-1 - exit coda_roc is exiting, clear dma memory

roc_reboot helped

Sergey B. Dec 2017 run:

tdcpcal4 during the run:

tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=480, res=400 => [ 0] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=400, res=64 => [ 0] ERROR: tdc1190ReadEvent[Dma] returns -2 ERROR: word counters: 13 != 11 (subtract=9) tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=416, res=16 => [ 0] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=416, res=400 => [ 0] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[0]=416, res=400 => [ 0] ERROR: tdc1190ReadEvent[Dma] returns -2

Sergey B. Dec 2017 run:

adcft1 on Go:

clasrun@clonpc18:clasrun> tcpClient adcft1 'tiStatus(1)'

STATUS for TI at VME (Local) base address 0x00a80000 (0xabfd1000)


A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac551000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 634
    L1A Count: 6420
  Block Limit: 0   
  Block Count: 642
Registers (offset):
 boardID        (0x0000) = 0x71011546    fiber          (0x0004) = 0x0e0f11ff
 intsetup       (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
 adr32          (0x0010) = 0x08000000    blocklevel     (0x0014) = 0x0a0a0001
 dataFormat     (0x0018) = 0x08080006    vmeControl     (0x001c) = 0x00000011
 trigger        (0x0020) = 0x01000012    sync           (0x0024) = 0x00000c82
 busy           (0x0028) = 0x00000003    clock          (0x002c) = 0x00000002
 blockBuffer    (0x0034) = 0x08000808    output         (0x004c) = 0x00080800
 fiberSyncDelay (0x0050) = 0x4f004f13    nblocks        (0x0094) = 0x00000282
 GTPStatusA     (0x00b0) = 0x0e801fff    GTPStatusB     (0x00b4) = 0xff0fff0f
 livetime       (0x00a8) = 0x009aaadc    busytime       (0x00ac) = 0x00000000
 GTPTrgBufLen   (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x08000807
 triggerRule    (0x0038) = 0x0101010a    triggerRuleMin (0x0138) = 0x00000000


Block Level        = 10 
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
Fiber Status         1     2     3     4     5     6     7     8
                   ----- ----- ----- ----- ----- ----- ----- -----
 Connected          YES   YES   YES   YES                           


Clock Source (2) = 
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source = 
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source = 
  Switch Slot A    
  Switch Slot B    
Interrupts DISABLED
  Level = 5   Vector = 0xc8
Bus Errors Enabled
Blocks ready for readout: 8
Input counter 1


clasrun@clonpc18:clasrun> tcpClient trig1 'tdGStatus'

TD Module Status Summary

             Ports          SyncSrc    BusySrc    Busy Status   Triggers TrgAck

Slot (Connected, TrigSrcEn) (V15FL) (PLT12345678) (PL12345678) Received Wait


3     12345678    12345678   V----   ---12345678   ----------  0xba304800   N
4     12345678    12345678   V----   ---12345678   ----------  0x32f04800   N
5     12345678    12345678   V----   ---12345678   ----------  0x42b05c00   N
6     123456-8    123456--   V----   ---123456--   ----------  0x0ff05c00   N
7     12345678    12345678   V----   ---12345678   --------7-  0xfb305c00   N
8     12345678    12345678   V----   ---12345678   ----------  0x11504800   N
9     123456--    123456--   V----   ---123456--   ----------  0xe9905c00   N

10 123456-- 123456-- V---- ---123456-- ---------- 0x47505c00 N 13 1234567- 1234567- V---- ---1234567- ---------- 0x82705c00 N

TD Port STATUS Summary

                                                     Block Status      Block  Buffer

Sl-P RocName COMM ROCID TrigSrcEn Busy Status Ready / NeedAck Level Level


3-1  adcecal1  UP      1     ENABLED                    0 /   0          10       0
3-2  tdcecal1  UP      2     ENABLED                    0 /   0          10       0
3-3  adcpcal1  UP      3     ENABLED                    0 /   0          10       0
3-4  tdcpcal1  UP      4     ENABLED                    0 /   0          10       0
3-5  adcftof1  UP      5     ENABLED                    0 /   0          10       0
3-6  tdcftof1  UP      6     ENABLED                    0 /   0          10       0
3-7     trig2  UP     38     ENABLED                    0 /   0          10       0
3-8   adccnd1  UP     73     ENABLED                    0 /   0          10       0
4-1  adcecal2  UP      7     ENABLED                    0 /   0          10       0
4-2  tdcecal2  UP      8     ENABLED                    0 /   0          10       0
4-3  adcpcal2  UP      9     ENABLED                    0 /   0          10       0
4-4  tdcpcal2  UP     10     ENABLED                    0 /   0          10       0
4-5  adcftof2  UP     11     ENABLED                    0 /   0          10       0
4-6  tdcftof2  UP     12     ENABLED                    0 /   0          10       0
4-7  adcctof1  UP     59     ENABLED                    0 /   0          10       0
4-8  tdcctof1  UP     60     ENABLED                    0 /   0          10       0
5-1  adcecal3  UP     13     ENABLED                    0 /   0          10       0
5-2  tdcecal3  UP     14     ENABLED                    0 /   0          10       0
5-3  adcpcal3  UP     15     ENABLED                    0 /   0          10       0
5-4  tdcpcal3  UP     16     ENABLED                    0 /   0          10       0
5-5  adcftof3  UP     17     ENABLED                    0 /   0          10       0
5-6  tdcftof3  UP     18     ENABLED                    0 /   0          10       0
5-7      svt1  UP     61     ENABLED                    0 /   0          10       0
5-8      svt2  UP     62     ENABLED                    0 /   0          10       0
6-1  adcecal4  UP     19     ENABLED                    0 /   0          10       0
6-2  tdcecal4  UP     20     ENABLED                    0 /   0          10       0
6-3  adcpcal4  UP     21     ENABLED                    0 /   0          10       0
6-4  tdcpcal4  UP     22     ENABLED                    0 /   0          10       0
6-5  adcftof4  UP     23     ENABLED                    0 /   0          10       0
6-6  tdcftof4  UP     24     ENABLED                    0 /   0          10       0
7-1  adcecal5  UP     25     ENABLED                    0 /   0          10       0
7-2  tdcecal5  UP     26     ENABLED                    0 /   0          10       0
7-3  adcpcal5  UP     27     ENABLED                    0 /   0          10       0
7-4  tdcpcal5  UP     28     ENABLED                    0 /   0          10       0
7-5  adcftof5  UP     30     ENABLED                    0 /   0          10       0
7-6  tdcftof5  UP     29     ENABLED                    0 /   0          10       0
7-7    adcft1  UP     70     ENABLED   BUSY-BlkAck      8 /   0          10       0
7-8    adcft2  UP     71     ENABLED                    0 /   0          10       0
8-1  adcecal6  UP     31     ENABLED                    0 /   0          10       0
8-2  tdcecal6  UP     32     ENABLED                    0 /   0          10       0
8-3  adcpcal6  UP     33     ENABLED                    0 /   0          10       0
8-4  tdcpcal6  UP     34     ENABLED                    0 /   0          10       0
8-5  adcftof6  UP     35     ENABLED                    0 /   0          10       0
8-6  tdcftof6  UP     36     ENABLED                    0 /   0          10       0
8-7    adcft3  UP     72     ENABLED                    0 /   0          10       0
8-8     mmft1  UP     75     ENABLED                    0 /   0          10       0
9-1      dc11  UP     41     ENABLED                    0 /   0          10       0
9-2      dc21  UP     44     ENABLED                    0 /   0          10       0
9-3      dc31  UP     47     ENABLED                    0 /   0          10       0
9-4      dc41  UP     50     ENABLED                    0 /   0          10       0
9-5      dc51  UP     53     ENABLED                    0 /   0          10       0
9-6      dc61  UP     56     ENABLED                    0 /   0          10       0

10-1 dc12 UP 42 ENABLED 0 / 0 10 0 10-2 dc22 UP 45 ENABLED 0 / 0 10 0 10-3 dc32 UP 48 ENABLED 0 / 0 10 0 10-4 dc42 UP 51 ENABLED 0 / 0 10 0 10-5 dc52 UP 54 ENABLED 0 / 0 10 0 10-6 dc62 UP 57 ENABLED 0 / 0 10 0 13-1 dc13 UP 43 ENABLED 0 / 0 10 0 13-2 dc23 UP 46 ENABLED 0 / 0 10 0 13-3 dc33 UP 49 ENABLED 0 / 0 10 0 13-4 dc43 UP 52 ENABLED 0 / 0 10 0 13-5 dc53 UP 55 ENABLED 0 / 0 10 0 13-6 dc63 UP 58 ENABLED 0 / 0 10 0 13-7 mvt1 UP 69 ENABLED 0 / 0 10 0

Total Slaves Added = 65



Sergey B. Dec 2017 run:

tdcftof3 during download:
FiberMeas: ERROR: Unable to measure fiber latency without HFBR1/5 as Clock Source
        Using default Fiber Sync Delay = 522133248 (0x1f1f1f00)
tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,
tiInit: Scanning for TI...
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x580000
jlabgefMemProbe: Clearing VME BERR/2eST (0x800cf900) at VME address 0x980000
tiInit: ERROR: Unable to find TI
cannot find TI, ret=-1 - exit
coda_roc is exiting, clear dma memory 


coda_roc: ======= Close the default VME windows =========

Sergey B. Dec 2017 run:

tiReload:

FPGA Re-Load ! 

CODAtcpServer: wait: coda request >download PROD< in progress

 ID: 0x7101    Firmware (type - revision): 0x3 - 0x076 (prodID = 1)

... Configure as TI Slave... tiSetClockSource: Setting clock source to EXTERNAL-HFBR1 (1) tiSetClockSource: ERROR Setting Clock Source (clkset = 0x2, clkread = 0x0) tiSetTriggerSource: ERROR: Invalid Trigger Source (129). Must be between 0 and 11. FiberMeas: ERROR: Unable to measure fiber latency without HFBR1/5 as Clock Source

        Using default Fiber Sync Delay = 522133248 (0x1f1f1f00)tiInit: Fiber Measurement failure.  Check fiber and/or fiber port,

tiInit: Scanning for TI... tiFind: Found TI at 0x00a80000 TI VME (Local) address = 0x00a80000 (0xabfd8000) tiInit: ERROR: Invalid Board ID: 0x0 (rval = 0x00000000) cannot find TI, ret=-1 - exit coda_roc is exiting, clear dma memory


Sergey B. Dec 2017 run: DiagGuiServer ...

adcctof1: DiagGuiServer was restarted

proc_thread: waiting= 7610 processing= 203 microsec per event (nev=12) net_thread: waiting= 777 sending= 0 microsec per event (nev=128) jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x500404 usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00800404 < 0x0e000000) faReadBlock: ERROR: vmeDmaDone returned an Error FADCs: No data or error. dCnt = -1 (slots from 3) jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x880404 jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x900404 usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00b00404 < 0x0e000000) faReadBlock: ERROR: vmeDmaDone returned an Error FADCs: No data or error. dCnt = -1 (slots from 3) jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x580000 usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00880000 < 0x0e000000) faReadBlock: ERROR: vmeDmaDone returned an Error FADCs: No data or error. dCnt = -1 (slots from 3) jlabgefClearException: Clearing VME Exception (0x800cf900) at VME address 0x980000 usrVme2MemDmaDone: ERROR: VME Exception Address < DMA Source Address (0x00100000 < 0x08000000)

tiReadBlock: ERROR: vmeDmaDone returned an Error ERROR in tiReadBlock : No data or error, len = -1 ERROR1: LINK_sized_write() returns errno=32 (cc=-1, sizeof(nbytes)=4(162600)

Sergey B. Dec 2017 run:

adcpcal3vtp: VTP UNKNOWN data2: [569] 0x00000008 VTP UNKNOWN data2: [570] 0x8d40002b VTP UNKNOWN data2: [572] 0x000001e3 VTP UNKNOWN data1C: [573] 0xe122016d VTP UNKNOWN data2: [574] 0x82fc060a [70667][ 56] ERROR1 in VTP data: blockheader slot 21 != blocktrailer slot 11 VTP UNKNOWN data2: [ 4] 0x000001d7 VTP UNKNOWN data2: [ 5] 0x00010e1c VTP UNKNOWN data2: [ 6] 0x0000002d VTP UNKNOWN data1C: [ 7] 0xe10a016d VTP UNKNOWN data2: [ 8] 0x85401c0a VTP UNKNOWN data2: [ 11] 0x000ac90f VTP UNKNOWN data2: [ 12] 0x6f13541b VTP UNKNOWN data2: [ 13] 0x00000008 VTP UNKNOWN data2: [ 14] 0x0c010003 VTP UNKNOWN data2: [ 15] 0x000ac910 VTP UNKNOWN data2: [ 16] 0x6f1435d1 VTP UNKNOWN data2: [ 17] 0x00000008 VTP UNKNOWN data2: [ 19] 0x000ac911 VTP UNKNOWN data2: [ 20] 0x6f1451e5 VTP UNKNOWN data2: [ 21] 0x00000008 VTP UNKNOWN data2: [ 23] 0x000ac912 VTP UNKNOWN data2: [ 24] 0x6f159615 VTP UNKNOWN data2: [ 25] 0x00000008 VTP UNKNOWN data2: [ 26] 0x20010003 VTP UNKNOWN data2: [ 27] 0x000ac913 VTP UNKNOWN data2: [ 28] 0x6f15b188 VTP UNKNOWN data2: [ 29] 0x00000008 VTP UNKNOWN data2: [ 31] 0x000ac914 VTP UNKNOWN data2: [ 32] 0x6f1739ea VTP UNKNOWN data2: [ 33] 0x00000008 VTP UNKNOWN data2: [ 35] 0x000ac915 VTP UNKNOWN data2: [ 36] 0x6f176682 VTP UNKNOWN data2: [ 37] 0x00000008 VTP UNKNOWN data2: [ 39] 0x000ac916 VTP UNKNOWN data2: [ 40] 0x6f17b3e0 VTP UNKNOWN data2: [ 41] 0x00000008 VTP UNKNOWN data2: [ 43] 0x000ac917 VTP UNKNOWN data2: [ 44] 0x6f185ceb VTP UNKNOWN data2: [ 45] 0x00000008 VTP UNKNOWN data2: [ 47] 0x000ac918 VTP UNKNOWN data2: [ 48] 0x6f186d5e VTP UNKNOWN data2: [ 49] 0x00000008 VTP UNKNOWN data2: [ 50] 0x8d40002b VTP UNKNOWN data2: [ 52] 0x000001a7 VTP UNKNOWN data1C: [ 53] 0xe122016d VTP UNKNOWN data2: [ 87] 0x00560014 VTP UNKNOWN data2: [115] 0x00b100c9 VTP UNKNOWN data2: [239] 0x008e0016 [70684][421] ERROR1 in VTP data: blockheader slot 21 != blocktrailer slot 11 VTP UNKNOWN data2: [381] 0x00000042 VTP UNKNOWN data2: [382] 0x00010e1f VTP UNKNOWN data2: [383] 0x0000002d VTP UNKNOWN data1C: [384] 0xe10a016d VTP UNKNOWN data2: [385] 0x85401f0a VTP UNKNOWN data2: [388] 0x000ac92d VTP UNKNOWN data2: [389] 0x6f22f0b4 VTP UNKNOWN data2: [390] 0x00000008 VTP UNKNOWN data2: [392] 0x000ac92e VTP UNKNOWN data2: [393] 0x6f2335c2 VTP UNKNOWN data2: [394] 0x00000008 VTP UNKNOWN data2: [396] 0x000ac92f [70687][ 16] ERROR1 in VTP data: blockheader slot 21 != blocktrailer slot 11

Sergey B. Dec 2017 run:

adcpcal5vtp:

proc_thread: waiting= 977 processing= 898 microsec per event (nev=26) net_thread: waiting= 165 sending= 18 microsec per event (nev=266) VTP UNKNOWN data2: [131] 0x000000dc VTP UNKNOWN data2: [132] 0x00010e22 VTP UNKNOWN data2: [133] 0x0000002d VTP UNKNOWN data1C: [134] 0xe10a016f VTP UNKNOWN data2: [135] 0x8541220a VTP UNKNOWN data2: [138] 0x00046b4b VTP UNKNOWN data1A: [139] 0xd7a62fbd VTP UNKNOWN data2: [140] 0x00000003 VTP UNKNOWN data2: [142] 0x00046b4c VTP UNKNOWN data1A: [143] 0xd7a7146f VTP UNKNOWN data2: [144] 0x00000003 VTP UNKNOWN data2: [221] 0x00bd001c VTP UNKNOWN data2: [326] 0x09333469 Segmentation fault (core dumped) clasrun@adcpcal5vtp:clasrun>

Sergey B. Dec 2017 run:

dc52, on next 'go' dc23, on next 'go' dc42, dc51, dc61



......

CODAtcpServerWorkTask exit ? CODAtcpServerWorkTask exit ! usrVme2MemDmaStart ERROR: DMA Physical Memory has an invalid base address (0x00000000) tiReadBlock: ERROR in DMA transfer Initialization 0xffffffff ERROR in tiReadBlock : No data or error, len = -1 usrVme2MemDmaStart ERROR: DMA Physical Memory has an invalid base address (0x00000000)dcrbReadBlock: ERROR in DMA transfer Initialization 0xffffffff Segmentation fault

.......

Sergey B. Dec 2017 run:

adcpcal5vtp on holdoffs:

  1. TS_HOLDOFF 1 10 1
  2. TS_HOLDOFF 2 10 1
  3. TS_HOLDOFF 3 7 1
  4. TS_HOLDOFF 4 5 1

dabufp[0] = 0x00000133 (307) dabufp[1] = 0x00010e01 (69121) [ 1][137] ERROR2 in VTP data: trailer #words=137 != actual #words=138 - ndnv=0) proc_thread: waiting= 31339 processing= 906 microsec per event (nev=22) net_thread: waiting= 774 sending= 0 microsec per event (nev=912) ERROR in TI event header word: event type = 0 ERROR in TI event header word (0x02) ERROR: TS has more then 7 data words - exit clasrun@adcpcal5vtp:clasrun>

Sergey B. Dec 2017 run: sync to tdc ??? do not read if almost full or full ??

NEED ALARM BROWSER AND SEND THIS KIND OF ERRORS THERE !!! OR AT LEAST TO RUNCONTROL

tdcftof4:

it complains constantly:

ERR: event# 925554, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000

ERR: event#  925594, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000
ERR: event#  925654, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000
ERR: event#  925694, slot# 13, tdc# 00, error_flags=0x00004000, err=0x0000, lock=0x0000


but on high rate happens this:


[ 4] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2048, res=176 => mbytes=1872 [ 4] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2144, res=2064 => mbytes=80 [ 4] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2288, res=160 => mbytes=2128 [ 4] ERROR: tdc1190ReadEvent[Dma] returns -2 tdc1190ReadBoardDmaDone: WRONG: nbytes_save[4]=2704, res=2272 => mbytes=432 [ 4] ERROR: tdc1190ReadEvent[Dma] returns -2



6-3  adcpcal4  UP     21    DISABLED                    0 /   0          20       0
6-4  tdcpcal4  UP     22    DISABLED                    0 /   0          20       0
6-5  adcftof4  UP     23    DISABLED                    0 /   0          20       0
6-6  tdcftof4  UP     24     ENABLED   BUSY-Fiber      0 /   0          20       0
6-8   scaler1  UP     64    DISABLED                    0 /   0          20       0
7-1  adcecal5  UP     25    DISABLED                    0 /   0          20       0
7-2  tdcecal5  UP     26    DISABLED                    0 /   0          20       0
7-3  adcpcal5  UP     27    DISABLED                    0 /   0          20       0
7-4  tdcpcal5  UP     28    DISABLED                    0 /   0          20       0
7-5  adcftof5  UP     30    DISABLED                    0 /   0          20       0
7-6  tdcftof5  UP     29    DISABLED                    0 /   0          20       0
7-7    adcft1  UP     70    DISABLED                    0 /   0          20       0
7-8    adcft2  UP     71    DISABLED                    0 /   0          20       0
8-1  adcecal6  UP     31    DISABLED                    0 /   0          20       0
8-2  tdcecal6  UP     32    DISABLED                    0 /   0          20       0
8-3  adcpcal6  UP     33    DISABLED                    0 /   0          20       0
8-4  tdcpcal6  UP     34    DISABLED                    0 /   0          20       0
8-5  adcftof6  UP     35    DISABLED                    0 /   0          20       0
8-6  tdcftof6  UP     36    DISABLED                    0 /   0          20       0
8-7    adcft3  UP     72    DISABLED                    0 /   0          20       0
9-1      dc11  UP     41    DISABLED                    0 /   0          20       0
9-2      dc21  UP     44    DISABLED                    0 /   0          20       0
9-3      dc31  UP     47    DISABLED                    0 /   0          20       0
9-4      dc41  UP     50    DISABLED                    0 /   0          20       0
9-5      dc51  UP     53    DISABLED                    0 /   0          20       0
9-6      dc61  UP     56    DISABLED                    0 /   0          20       0

10-1 dc12 UP 42 DISABLED 0 / 0 20 0 10-2 dc22 UP 45 DISABLED 0 / 0 20 0 10-3 dc32 UP 48 DISABLED 0 / 0 20 0 10-4 dc42 UP 51 DISABLED 0 / 0 20 0 10-5 dc52 UP 54 DISABLED 0 / 0 20 0 10-6 dc62 UP 57 DISABLED 0 / 0 20 0 13-1 dc13 UP 43 DISABLED 0 / 0 20 0 13-2 dc23 UP 46 DISABLED 0 / 0 20 0 13-3 dc33 UP 49 DISABLED 0 / 0 20 0 13-4 dc43 UP 52 DISABLED 0 / 0 20 0 13-5 dc53 UP 55 DISABLED 0 / 0 20 0 13-6 dc63 UP 58 DISABLED 0 / 0 20 0

Total Slaves Added = 62



clasrun@clondaq6:clasrun> tcpClient tdcftof4 tiStatus

STATUS for TI at VME (Local) base address 0x00a80000 (0xabfae000)


A32 Data buffer ENABLED at VME (Local) base address 0x08000000 (0xac52e000)
Configured as a TI Slave
Readout Count: 0
    Ack Count: 88880
    L1A Count: 1777608
  Block Limit: 0   
  Block Count: 88880
Registers (offset):
 boardID        (0x0000) = 0x71011518    fiber          (0x0004) = 0x001111ff
 intsetup       (0x0008) = 0x000005c8    trigDelay      (0x000c) = 0x0f000f00
 adr32          (0x0010) = 0x08000000    blocklevel     (0x0014) = 0x14140001
 dataFormat     (0x0018) = 0x08080006    vmeControl     (0x001c) = 0x00000011
 trigger        (0x0020) = 0x01000000    sync           (0x0024) = 0x00000c82
 busy           (0x0028) = 0x00850004    clock          (0x002c) = 0x00840006
 blockBuffer    (0x0034) = 0x00080001    output         (0x004c) = 0x00000000
 fiberSyncDelay (0x0050) = 0x4f004f11    nblocks        (0x0094) = 0x08015b30
 GTPStatusA     (0x00b0) = 0x10801fff    GTPStatusB     (0x00b4) = 0xff11ff11
 livetime       (0x00a8) = 0x01112a8e    busytime       (0x00ac) = 0x003e21ce
 GTPTrgBufLen   (0x00b8) = 0xe000fc18    syncDelay      (0x007c) = 0x00000007
 triggerRule    (0x0038) = 0x0101010a    triggerRuleMin (0x0138) = 0x00000000


Block Level        = 20 
Block Buffer Level = 8 -Broadcast- (Busy not enabled)
Fiber Status         1     2     3     4     5     6     7     8
                   ----- ----- ----- ----- ----- ----- ----- -----
 Connected          YES                     YES                     


Clock Source (2) = 
  HFBR #1 Input
Trigger input source (ENABLED) =
  HFBR #1 Input
  VME Command
Sync source = 
  HFBR #1 Input
  User SYNCRESET Receieve Enabled
BUSY input source = 
  P2 Input         ** BUSY **
Interrupts DISABLED
  Level = 5   Vector = 0xc8
Bus Errors Enabled
Blocks ready for readout: 0
Input counter 1


clasrun@clondaq6:clasrun



clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(0)' STATUS for v1290 TDC at VME (USER) base address 0x11180000 (0xb56ae000)


Interrupts Disabled
 Data Status 
   Events to transfer via BLT  = 0x0014
   Status        = 0x0039 (Data Ready )
   Events Stored = 8
   Total Events  = 8136
   Almost Full   = 16384
 TDC Mode/Status 
   Resolution      : 25 ps
   Trigger Matching: On
   TDC Headers/EOB : On
   Bus Errors      : Off
   TDC Errors      : 0:0   1:0   2:0   3:0


clasrun@clondaq6:clasrun> clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(1)' STATUS for v1290 TDC at VME (USER) base address 0x11300000 (0xb582e000)


Interrupts Disabled
 Data Status 
   Events to transfer via BLT  = 0x0014
   Status        = 0x0039 (Data Ready )
   Events Stored = 8
   Total Events  = 8136
   Almost Full   = 16384
 TDC Mode/Status 
   Resolution      : 25 ps
   Trigger Matching: On
   TDC Headers/EOB : On
   Bus Errors      : Off
   TDC Errors      : 0:0   1:0   2:0   3:0


clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(2)' STATUS for v1290 TDC at VME (USER) base address 0x11480000 (0xb59ae000)


Interrupts Disabled
 Data Status 
   Events to transfer via BLT  = 0x0014
   Status        = 0x0039 (Data Ready )
   Events Stored = 8
   Total Events  = 8136
   Almost Full   = 16384
 TDC Mode/Status 
   Resolution      : 25 ps
   Trigger Matching: On
   TDC Headers/EOB : On
   Bus Errors      : Off
   TDC Errors      : 0:0   1:0   2:0   3:0


clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(3)' STATUS for v1290 TDC at VME (USER) base address 0x11680000 (0xb5bae000)


Interrupts Disabled
 Data Status 
   Events to transfer via BLT  = 0x0014
   Status        = 0x0079 (Data Ready )
   Events Stored = 8
   Total Events  = 8136
   Almost Full   = 16384
 TDC Mode/Status 
   Resolution      : 25 ps
   Trigger Matching: On
   TDC Headers/EOB : On
   Bus Errors      : Off
   TDC Errors      : 0:1   1:0   2:0   3:0


clasrun@clondaq6:clasrun> tcpClient tdcftof4 'tdc1190Status(4)' STATUS for v1190 TDC at VME (USER) base address 0x11800000 (0xb5d2e000)


Interrupts Disabled
 Data Status 
   Events to transfer via BLT  = 0x0014
   Status        = 0x003b (Almost Full)
   Events Stored = 368
   Total Events  = 8136
   Almost Full   = 16384
 TDC Mode/Status 
   Resolution      : 100 ps
   Trigger Matching: On
   TDC Headers/EOB : On
   Bus Errors      : Off
   TDC Errors      : 0:0   1:0   2:0   3:0

Sergey B. 11-jan-2016: pcal0 test setup coda_roc error messages

..............
net_thread:  waiting=      0    sending=      1 microsec per event (nev=120)
proc_thread: waiting=      0 processing=   1568 microsec per event (nev=30)
proc_thread: waiting=      0 processing=   1539 microsec per event (nev=30)
ERROR in TI event header word (0x1d)
ERROR in TI event header word (0x1d)
ERROR in TI event header word (0x1d)
ERROR in TI event header word (0x1d)
ERROR in TI event header word (0x1d)
ERROR in TI event header word (0x1d)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1e)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x1f)
ERROR in TI event header word (0x20)
ERROR in TI event header word (0x20)
ERROR in TI event header word (0x20)
TI UNKNOWN data: [162] 0x00000000
  Previous stuff
         [142][162] 0xa41fc94d
         [143][162] 0x00000000
         [144][162] 0x01010003
         [145][162] 0x000052cd
         [146][162] 0xa41fe9db
         [147][162] 0x00000000
         [148][162] 0x01010003
         [149][162] 0x000052ce
         [150][162] 0xa4200a68
         [151][162] 0x00000000
         [152][162] 0x01010003
         [153][162] 0x000052cf
         [154][162] 0xa4202af5
         [155][162] 0x00000000
         [156][162] 0x01010003
         [157][162] 0x000052d0
         [158][162] 0xa4204b82
         [159][162] 0x00000000
         [160][162] 0x0000fafe
         [161][162] 0x0101d97e
          [162][162] 0x00000000
          [163][162] 0xb9820003
          [164][162] 0x0000faff
          [165][162] 0x0101fa0b
          [166][162] 0x00000000
          [167][162] 0x01010003
          [168][162] 0x0000fb00
          [169][162] 0xb9830003
          [170][162] 0x0000fb01
          [171][162] 0x01013b25
  End Of Previous stuff
net_thread:  waiting=    104    sending=      1 microsec per event (nev=1080)
coda_roc is exiting, clear dma memory
net_thread:  waiting=      0    sending=      1 microsec per event (nev=120)
bb_delete1: gefVmeFreeDmaBuf[ 0] has been called
bb_delete1: gefVmeFreeDmaBuf[ 1] has been called
Segmentation fault
clasrun@pcal0:clasrun>

AFTER roc_reboot:

.......
TI UNKNOWN data: [  0] 0x00000000
  Previous stuff
         [-20][  0] 0x00000000
         [-19][  0] 0x900052f7
         [-18][  0] 0xa0000005
         [-17][  0] 0x00000000
         [-16][  0] 0x00001466
         [-15][  0] 0x000052f7
         [-14][  0] 0x00000000
         [-13][  0] 0x00000000
         [-12][  0] 0x900052f8
         [-11][  0] 0xa0000005
         [-10][  0] 0x00000000
         [ -9][  0] 0x00001466
         [ -8][  0] 0x000052f8
         [ -7][  0] 0x5693d19e
         [ -6][  0] 0x00000001
         [ -5][  0] 0x8800011a
         [ -4][  0] 0x000069de
         [ -3][  0] 0x00010e14
         [ -2][  0] 0x000000e2
         [ -1][  0] 0xe10a0106
          [  0][  0] 0x00000000
          [  1][  0] 0x01013cf3
          [  2][  0] 0x00006cbe
          [  3][  0] 0x6cbe0003
          [  4][  0] 0x00006cbf
          [  5][  0] 0x01010003
          [  6][  0] 0xfd400000
          [  7][  0] 0x8d407e0e
          [  8][  0] 0xff116cc0
          [  9][  0] 0x85420003
  End Of Previous stuff
coda_roc is exiting, clear dma memory
bb_delete1: gefVmeFreeDmaBuf[ 0] has been called
bb_delete1: gefVmeFreeDmaBuf[ 1] has been called
bb_delete1: gefVmeFreeDmaBuf[ 2] has been called
bb_delete1: gefVmeFreeDmaBuf[ 3] has been called
bb_delete1: gefVmeFreeDmaBuf[ 4] has been called
bb_delete1: gefVmeFreeDmaBuf[ 5] has been called
bb_delete1: gefVmeFreeDmaBuf[ 6] has been called
bb_delete1: gefVmeFreeDmaBuf[ 7] has been called
bb_delete1: gefVmeFreeDmaBuf[ 8] has been called
bb_delete1: gefVmeFreeDmaBuf[ 9] has been called
bb_delete1: gefVmeFreeDmaBuf[10] has been called
bb_delete1: gefVmeFreeDmaBuf[11] has been called
bb_delete1: gefVmeFreeDmaBuf[12] has been called
bb_delete1: gefVmeFreeDmaBuf[13] has been called
bb_delete1: gefVmeFreeDmaBuf[14] has been called
bb_delete1: gefVmeFreeDmaBuf[15] has been called
clasrun@pcal0:clasrun>

roc_reboot did not helped. Flashed firmwares in all boards (TI, SD, FADC250s), roc_reboot again - now it works. Will watch it ...

Happened again. Replaced TI. Took 300M events, happened again:

..........................
net_thread:  waiting=      0    sending=      1 microsec per event (nev=120)
TI UNKNOWN data: [162] 0x1519028c
  Previous stuff
         [142][162] 0x01010003
         [143][162] 0x1518991c
         [144][162] 0xed570324
         [145][162] 0x0000028b
         [146][162] 0x01010003
         [147][162] 0x1518991d
         [148][162] 0xed5723b1
         [149][162] 0x0000028b
         [150][162] 0x01010003
         [151][162] 0x1518991e
         [152][162] 0xed57443e
         [153][162] 0x0000028b
         [154][162] 0x01010003
         [155][162] 0x1518991f
         [156][162] 0xed5764da
         [157][162] 0x0000028b
         [158][162] 0x01010003
         [159][162] 0x15189920
         [160][162] 0x00009bab
         [161][162] 0x01019815
          [162][162] 0x1519028c
          [163][162] 0x01010003
          [164][162] 0x15199bac
          [165][162] 0x0e0db894
          [166][162] 0x0000028c
          [167][162] 0x01010003
          [168][162] 0x15199bad
          [169][162] 0x0e0ed921
          [170][162] 0x0000028c
          [171][162] 0x01010003
  End Of Previous stuff

Notice that shortly before that error following condition occur:

............
proc_thread: waiting=      0 processing=    975 microsec per event (nev=30)
proc_thread: waiting=      0 processing=    983 microsec per event (nev=30)
proc_thread: waiting=      0 processing=    995 microsec per event (nev=30)
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
timer:     763 microsec (min=    722 max=  14613 rms**2=6321261)
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
bb_write[0x08a24418]: waiting for buffer (read=5 write=4) ...
bb_write[0x08a25280]: waiting for buffer (read=9 write=8) ...
net_thread:  waiting=      0    sending=   2001 microsec per event (nev=1080)
net_thread:  waiting=      0    sending=      2 microsec per event (nev=120)
net_thread:  waiting=      0    sending=      2 microsec per event (nev=1080)
................


CLAS era

Sergey B. 10-dec-2011: nothing bad happened, security scan got through, need to block it better

.....
FFFFFFFFFFFFFFFFFF
WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=29 address>129.57.71.28< port=59401
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
accept: Too many open files
.....


Sergey B. 23-nov-2011: EB1 hung, probably because if security scan

.....
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
tcl7.4(init.tcl): invalid command name "GET"
clasprod::EB1> WARN: ignore request from 129.57.71.28
CODAtcpServer: start work thread
befor: socket=6 address>129.57.71.28< port=47514
WARN: ignore request from 129.57.71.28
WARN: ignore request from 129.57.71.28
<HUNG>


Sergey B. 7-nov-2011: dc12, error message from from rol.h (10kHz pulser)

codaUpdateStatus: updating request done
active, events so far 0
wait: coda request >go< in progress
0x3e55cd0 (twork0047): INIT_NAME: INFO: rolp->daproc = 5
0x3e55cd0 (twork0047): INFO: User Go ...
codaExecute done
SEND_BUFFER_ROC 4 (0x017f4050 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x017f4050 11)
SSS2 4 (0x017f4050 11)
SEND_BUFFER_ROC 4 (0x016bd8e0 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x016bd8e0 11)
SSS2 4 (0x016bd8e0 11)
SEND_BUFFER_ROC 4 (0x01587170 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01587170 11)
SSS2 4 (0x01587170 11)
SEND_BUFFER_ROC 4 (0x01450a00 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x01450a00 11)
SSS2 4 (0x01450a00 11)
SEND_BUFFER_ROC 4 (0x0131a290 0) (conditions 1 13)
FILLING HEADER: dataInBuf>>2=11, g_events_in_buffer=1
SSS1 4 (0x0131a290 11)
SSS2 4 (0x0131a290 11)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
interrupt: INFO from bb_write: NO BUFS1
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=21263(21263) heartmask=4
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
UDP_cancel: cancel >inf:dc12 sys 0, mask 4<
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..)
bb_write: no bufs, back end too slow ..) 
...........

Click end - ended fine (probably network was stuck ?? or proc ???).


Sergey B. 8-jun-2011: test setup croctest2: error on croctest2 after 99M events, DMAing from single v1720 (NOTE: first test of ROL1 writing directly to the big buffer)

proc_thread: waiting=    846 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    192 processing=    460 microsec per event (nev=189)
rols_thread: waiting=    194 processing=    457 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
proc_thread: waiting=    846 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    185 processing=    469 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    200 processing=    449 microsec per event (nev=188)
proc_thread: waiting=    847 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
net_thread:  waiting=    720    sending=      0 microsec per event (nev=2218)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    451 microsec per event (nev=188)
proc_thread: waiting=    635 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
rols_thread: waiting=    199 processing=    450 microsec per event (nev=188)
proc_thread: waiting=    847 processing=      3 microsec per event (nev=188)
rols_thread: waiting=    205 processing=    450 microsec per event (nev=188)
machine check
Exception next instruction address: 0x0946e298
Machine Status Register: 0x0010b030
Condition Register: 0x80000088
Task: 0x946c670 "ROLS_LOOP"
setHeartError: 0 >sys 0, mask 4<
WARN: HeartBeat[0]: heartbeat=54077(54077) heartmask=4
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
UDP_cancel: cancel >inf:croctest2 sys 0, mask 4<
-> tt
1668c8 vxTaskEntry    +68 : rols_loop ()
d2e5290 rols_loop      +b4 : output_proc_network ()
d2e5574 output_proc_network+298: vmenrol1__init ()
946ccb8 vmenrol1__init +19c: 946cafc ()
946cb0c end            +91cb68c: 946ca1c ()
946cabc end            +91cb63c: 946c890 ()
946c920 end            +91cb4a0: 946e298 ()
value = 0 = 0x0
-> 

Sergey B. 10-feb-2011: test setup croctest2: coda_net on pmc dies

0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6559)
net_thread:  waiting=    131    sending=      2 microsec per event (nev=12708)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
proc_thread: waiting=    139 processing=      3 microsec per event (nev=6474)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting=    129 processing=      3 microsec per event (nev=6452)
net_thread:  waiting=    133    sending=      2 microsec per event (nev=12598)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=19)
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6553)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=16)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=12)
net_thread:  waiting=    131    sending=      2 microsec per event (nev=12714)
proc_thread: waiting=    139 processing=      3 microsec per event (nev=6489)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=8)
proc_thread: waiting=    128 processing=      3 microsec per event (nev=6478)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=4)
net_thread:  waiting=    134    sending=      2 microsec per event (nev=12377)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=0)
proc_thread: waiting=    127 processing=      3 microsec per event (nev=6554)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=3)
proc_thread: waiting=    139 processing=      3 microsec per event (nev=6470)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=7)
proc_thread: waiting=    128 processing=      3 microsec per event (nev=6516)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=11)
0x1aacca00 (coda_proc): timer: 1 microsec (min=1 max=173 rms**2=15)
ERROR2: LINK_sized_write() returns errno=32
Broken pipe
ERROR: net_thread failed (in LINK_sized_write).
-> 
-> i
 NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask   excTask      1effe6c0   0 PEND         14e508 1effe5a0  3d0001     0
tLogTask   logTask      1effbb20   0 PEND         14e508 1effba10       0     0
tShell     shell        1eb8cb70   1 READY        145290 1eb8c750       d     0
tRlogind   rlogind      1ebb3850   2 PEND         13f934 1ebb33f0       0     0
tWdbTask   wdbTask      1ebaa530   3 PEND         13f934 1ebaa400       0     0
tNetTask   netTask      1ed0f280  50 PEND         13f934 1ed0f190       0     0
tPortmapd  portmapd     1ebb1690  54 PEND         13f934 1ebb1430  3d0002     0
tFtpdTask  b6064        1ebaebc0  55 PEND         13f934 1ebaea10       0     0
coda_proc  proc_thread  1aacca00 115 DELAY        144bb4 1aacc900       0    10
coda_pmc   coda_pmc     1e3f9720 200 DELAY        144bb4 1e3f9480       0     5
value = 0 = 0x0
-> printErrno 32
S_errno_EPIPE
value = 14 = 0xe
-> tt
Sorry, traces of my own stack begin at tt ().
value = -1 = 0xffffffff
-> memShow
 invalid block at 0x1ec60000 deleted
value = -1 = 0xffffffff
-> memShow
status    bytes     blocks   avg block  max block
------ ---------- --------- ---------- ----------
current
  free  421348656        56    7524083  411333840
 alloc   79807344      1986      40184          -
cumulative
 alloc   87490320      7392      11835          -
value = 0 = 0x0
-> usrNetStackSysPoolStatus
value = 0 = 0x0
-> usrNetStackSysPoolStatus("CODAtcpServer",0)
value = 0 = 0x0
-> usrNetStackDataPoolStatus("CODAtcpServer",0)
value = 454823200 = 0x1b1c0d20
-> usrNetStackDataPoolStatus("bla",0)
value = 454823168 = 0x1b1c0d00
-> usrNetStackDataPoolStatus("bla",0)
value = 454823040 = 0x1b1c0c80
-> memShow
status    bytes     blocks   avg block  max block
------ ---------- --------- ---------- ----------
current
  free  421348528        55    7660882  411333840
 alloc   79807472      1990      40104          -
cumulative
 alloc   87496528      7401      11822          -
value = 0 = 0x0
-> 

EB looks fine, message was:

-- Got all fragments of go
BOS_encode_spec reached
BOS_encode_spec done: len=116
FFFFFFFFFFFFFFFFFF
PUT: wait while icb=8 == cbp->read=8 -------
clastest::EB5>

Tried to reset-download-prestart and got error on pmc at Download:

-> i
 NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask   excTask      1effe6c0   0 PEND         14e508 1effe5a0  3d0001     0
tLogTask   logTask      1effbb20   0 PEND         14e508 1effba10       0     0
tShell     shell        1eb8cb70   1 READY        145290 1eb8c750       d     0
tRlogind   rlogind      1ebb3850   2 PEND         13f934 1ebb33f0       0     0
tWdbTask   wdbTask      1ebaa530   3 PEND         13f934 1ebaa400       0     0
tNetTask   netTask      1ed0f280  50 PEND         13f934 1ed0f190       0     0
tPortmapd  portmapd     1ebb1690  54 PEND         13f934 1ebb1430  3d0002     0
tFtpdTask  b6064        1ebaebc0  55 PEND         13f934 1ebaea10       0     0
coda_pmc   coda_pmc     1e3f9720 200 DELAY        144bb4 1e3f9480   30065     1
value = 0 = 0x0
-> Download command received
-> bignet at 0x0d3a6e68
<- bignet at 0xcd3a6e68
bignet at 0xcd3a6e68, bignet.gbigBuffer at 0xcd3a6e78 -> 0xcd3a6e78
proc_on_pmc=1, net_on_pmc=1
rocID=0
ObjInitName >_rol2_tt_testsetup__init< 
data access
Exception current instruction address: 0x000d2090
Machine Status Register: 0x00003030
Data Access Register: 0x3d40001c
Condition Register: 0x40002083
Data storage interrupt Register: 0x40000000
Task: 0x1ed0f280 "tNetTask"
-> tt
14cad8 vxTaskEntry    +68 : netTask ()
c3e88 netTask        +70 : 209c0 ()
20a14 mgiEndLoad     +cc8: 20a88 ()
20bd8 mgiEndLoad     +e8c: muxReceive ()
bc638 muxReceive     +13c: b002c ()
b02c4 random         +358: do_protocol_with_type ()
9c920 do_protocol_with_type+2c : ipintr (1ec60e04)
9fb28 ipintr         +c8c: dbbb4 ()
dbbb4 tcp_input      +2968: tcp_output ()
dca24 tcp_output     +5c8: mHdrClGet ()
aa940 mHdrClGet      +8c : d1eb4 ()
value = 0 = 0x0
-> 
-> i
 NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask   excTask      1effe6c0   0 PEND         14e508 1effe5a0  3d0001     0
tLogTask   logTask      1effbb20   0 PEND         14e508 1effba10       0     0
tShell     shell        1eb8cb70   1 READY        145290 1eb8c750       d     0
tRlogind   rlogind      1ebb3850   2 PEND         13f934 1ebb33f0       0     0
tWdbTask   wdbTask      1ebaa530   3 PEND         13f934 1ebaa400       0     0
tNetTask   netTask      1ed0f280  50 SUSPEND       d2090 1ed0efd0       0     0
tPortmapd  portmapd     1ebb1690  54 PEND         13f934 1ebb1430  3d0002     0
tFtpdTask  b6064        1ebaebc0  55 PEND         13f934 1ebaea10       0     0
coda_pmc   coda_pmc     1e3f9720 200 PEND         1415a0 1e3f8ab0  3d0004     0
value = 0 = 0x0
-> 


Sergey B. 16-apr-2010: end run failed, both ET systems are dead !!!???

clondaq1:et1_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod  -m eb -v
et_nodename >eb<
et_start: asking for 200000 byte events.
et_start: asking for 4000 events.
removing file >/tmp/et_sys_clasprod<
file >/tmp/et_sys_clasprod< removed
et_start: starting ET system /tmp/et_sys_clasprod
ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/
libet_user.so< will be used
et INFO: et_add_stations, nstations = 2, stats_max = 10
et INFO: et_conductor 1, found station cec1d440
et INFO: et_station_create_at, created station LEVEL3
et INFO: et_add_stations, nstations = 3, stats_max = 10
et INFO: et_conductor 2, found station cec1d860
et INFO: et_station_create_at, created station ET2ER
unique name >et_start_eb_clasprod<
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver
11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using local protocol
11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver
11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
11:45:20: TAL-SS-00089-I Using tcp protocol
11:45:20: TAL-SS-00091-I Message from RTserver: Connection established.
11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_eb_clasprod>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq1.jlab.org>
11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>
ipc_init called at Fri Apr 16 11:45:20 2010
et_start: pthread_create(0x0000000e,...) done
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et INFO: et_sys_heartmonitor, kill bad process (2,22078)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments


clondaq2:clasrun> cd et2_start clondaq2:et2_start> clondaq2:et2_start> et_start -n 4000 -s 200000 -f /tmp/et_sys_clasprod

-m er -v

et_nodename >er< et_start: asking for 200000 byte events. et_start: asking for 4000 events. removing file >/tmp/et_sys_clasprod< file >/tmp/et_sys_clasprod< removed et_start: starting ET system /tmp/et_sys_clasprod ET user library >/usr/local/clas/release/current/coda/SunOS_i86pc/lib/ libet_user.so< will be used et INFO: et_add_stations, nstations = 2, stats_max = 10 et INFO: et_conductor 1, found station cec1d440 et INFO: et_station_create_at, created station TAPE et INFO: et_add_stations, nstations = 3, stats_max = 10 et INFO: et_conductor 2, found station cec1d860 et INFO: et_conductor 2, loaded shared lib /usr/local/clas/release/cur rent/coda/SunOS_i86pc/lib/libet_user.so et INFO: et_conductor 2, loaded function et2et_function et INFO: et_station_create_at, created station ET2ET10 unique name >et_start_er_clasprod< 11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <local> R Tserver 11:45:20: TAL-SS-00089-I Using local protocol 11:45:20: TAL-SS-00090-I Could not connect to <local> RTserver 11:45:20: TAL-SS-00093-I Skipping starting <start_never:local> RTserve r 11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>

RTserver

11:45:20: TAL-SS-00089-I Using local protocol 11:45:20: TAL-SS-00090-I Could not connect to <clondb1> RTserver 11:45:20: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1>

RTserver

11:45:20: TAL-SS-00089-I Using tcp protocol 11:45:20: TAL-SS-00091-I Message from RTserver: Connection established . 11:45:20: TAL-SS-00096-I Start subscribing to subject </et_start_er_cl asprod> 11:45:20: TAL-SS-00111-I Start subscribing to subject </_clondaq2.jlab .org> 11:45:20: TAL-SS-00111-I Start subscribing to subject </_all>

ipc_init called at Fri Apr 16 11:45:20 2010

et_start: pthread_create(0x0000000e,...) done et INFO: et_netserver: magic numbers do match, accept ET client et INFO: et_station_attach, done et INFO: et_netserver: magic numbers do match, accept ET client et INFO: et_station_attach, done et INFO: et_netserver: magic numbers do match, accept ET client et INFO: et_station_attach, done et INFO: et_restore_events, no events found belonging to attachment 4 et INFO: et_station_detach, done et WARN: et_command_loop: remote client connection broken et ERROR: et_netserver: ET server being probed by non-ET client or rea d failure et ERROR: et_netserver: ET server being probed by non-ET client or rea d failure et INFO: et_sys_heartmonitor, kill bad process (1,8898) et INFO: et_sys_heartmonitor, detach attachment 3 from stat 1 et INFO: et_flush_events, read 4000 events et INFO: et_flush_events, wrote 4000 events et INFO: et_station_detach, make station "TAPE" (cec1d440) idle et INFO: et_restore_events, no events found belonging to attachment 3 et INFO: et_station_detach, done et INFO: et_sys_heartmonitor, cleanup process 1 et INFO: set_fix_nprocs, change # of ET processes from 1 to 1 et INFO: set_fix_natts, station GRAND_CENTRAL has 2 attachments et INFO: set_fix_natts, station ET2ET10 has 1 attachments et INFO: set_fix_natts, # total attachments 3 -> 3 et INFO: set_fix_natts, proc 0 has 1 attachments



clondaq1:dac> clondaq1:dac> clondaq1:dac> clondaq1:dac> et_monitor -f /tmp/et_sys_clasprod LOCALITY = 1

 ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
             (tcp port 0) (udp port 11111) (multicast port 11111)
             (pid 22276) (lang C) (local) (period = 5 sec)
 STATIC INFO - maximum of:
   events(4000), event size(200000), temps(4000)
   stations(10), attaches(20), procs(10), concurrency added(16)
   network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,
 DYNAMIC INFO - currently there are:
   processes(4), attachments(4), temps(0)
   stations(3), hearbeat(1)
 STATIONS:
   "GRAND_CENTRAL" (id = 0)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
       restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(2),  ids(2,3,)
       input  list: cnt =  4000, events in  = 0
       output list: cnt =     0, events out = 0
   "LEVEL3" (id = 1)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
       restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(1),  ids(1,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
   "ET2ER" (id = 2)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
       restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(1),  ids(0,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
 LOCAL USERS:
   process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
   process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
   process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
   process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)
 ATTACHMENTS:
   att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
   proc(0), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
   proc(1), blocked(YES)
     events:  make(0), get(0), put(0), dump(0)
   att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
   proc(2), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
   proc(3), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
 EVENTS OWNED BY:
   system (4000),  att0 (0),  att1 (0),  att2 (0),  att3 (0),
 EVENT RATE of GC = 0 events/sec
 CREATING STATIONS:  
 IDLE STATIONS:      
 STATION CHAIN:      GRAND_CENTRAL, LEVEL3, ET2ER, 
 LOCKED MUTEXES:     
 ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq1.jlab.org) (bits 32)
             (tcp port 0) (udp port 11111) (multicast port 11111)
             (pid 22276) (lang C) (local) (period = 5 sec)
 STATIC INFO - maximum of:
   events(4000), event size(200000), temps(4000)
   stations(10), attaches(20), procs(10), concurrency added(16)
   network interfaces(3): 129.57.167.107, 129.57.69.200, 129.57.68.22,
 DYNAMIC INFO - currently there are:
   processes(4), attachments(4), temps(0)
   stations(3), hearbeat(1)
 STATIONS:
   "GRAND_CENTRAL" (id = 0)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
       restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(2),  ids(2,3,)
       input  list: cnt =  4000, events in  = 0
       output list: cnt =     0, events out = 0
   "LEVEL3" (id = 1)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
       restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(1),  ids(1,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
   "ET2ER" (id = 2)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(ALL)
       restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(1),  ids(0,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
 LOCAL USERS:
   process #0, # attachments(1), attach ids(0,), pid(22408), hbeat(1)
   process #1, # attachments(1), attach ids(1,), pid(22491), hbeat(1)
   process #2, # attachments(1), attach ids(2,), pid(23984), hbeat(70)
   process #3, # attachments(1), attach ids(3,), pid(24082), hbeat(68)
 ATTACHMENTS:
   att #0, is at station(ET2ER) on host(clondaq1.jlab.org) at pid(22408)
   proc(0), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #1, is at station(LEVEL3) on host(clondaq1.jlab.org) at pid(22491)
   proc(1), blocked(YES)
     events:  make(0), get(0), put(0), dump(0)
   att #2, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(23984)
   proc(2), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #3, is at station(GRAND_CENTRAL) on host(clondaq1.jlab.org) at pid(24082)
   proc(3), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
 EVENTS OWNED BY:
   system (4000),  att0 (0),  att1 (0),  att2 (0),  att3 (0),
 EVENT RATE of GC = 0 events/sec
 CREATING STATIONS:  
 IDLE STATIONS:      
 STATION CHAIN:      GRAND_CENTRAL, LEVEL3, ET2ER, 
 LOCKED MUTEXES:     

ET SYSTEM is DEAD!

^C clondaq1:dac>




clondaq2:stage_in> clondaq2:stage_in> et_monitor -f /tmp/et_sys_clasprod LOCALITY = 1

 ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
             (tcp port 0) (udp port 11111) (multicast port 11111)
             (pid 16127) (lang C) (local) (period = 5 sec)
 STATIC INFO - maximum of:
   events(4000), event size(200000), temps(4000)
   stations(10), attaches(20), procs(10), concurrency added(16)
   network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,
 DYNAMIC INFO - currently there are:
   processes(3), attachments(3), temps(0)
   stations(3), hearbeat(1)
 STATIONS:
   "GRAND_CENTRAL" (id = 0)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
       restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(0),  ids()
       input  list: cnt =  4000, events in  = 0
       output list: cnt =     0, events out = 0
   "TAPE" (id = 1)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
       restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(2),  ids(1,2,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
   "ET2ET10" (id = 2)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
       restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
       lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so,  function = et2et_function, class = 
     dynamic info
       attachments: total#(1),  ids(0,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
 LOCAL USERS:
   process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
   process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
   process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1662)
 ATTACHMENTS:
   att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
   proc(0), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
   proc(1), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
   proc(2), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
 EVENTS OWNED BY:
   system (4000),  att0 (0),  att1 (0),  att2 (0),
 EVENT RATE of GC = 0 events/sec
 CREATING STATIONS:  
 IDLE STATIONS:      
 STATION CHAIN:      GRAND_CENTRAL, TAPE, ET2ET10, 
 LOCKED MUTEXES:     
 ET SYSTEM - (/tmp/et_sys_clasprod) (host clondaq2) (bits 32)
             (tcp port 0) (udp port 11111) (multicast port 11111)
             (pid 16127) (lang C) (local) (period = 5 sec)
 STATIC INFO - maximum of:
   events(4000), event size(200000), temps(4000)
   stations(10), attaches(20), procs(10), concurrency added(16)
   network interfaces(3): 129.57.167.109, 129.57.69.202, 129.57.68.24,
 DYNAMIC INFO - currently there are:
   processes(3), attachments(3), temps(0)
   stations(3), hearbeat(1)
 STATIONS:
   "GRAND_CENTRAL" (id = 0)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
       restore(GC), prescale(1), cue(10), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(0),  ids()
       input  list: cnt =  4000, events in  = 0
       output list: cnt =     0, events out = 0
   "TAPE" (id = 1)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(MULTI), select(ALL)
       restore(OUT), prescale(1), cue(10), select words(-1,-1,-1,-1,)
     dynamic info
       attachments: total#(2),  ids(1,2,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
   "ET2ET10" (id = 2)
     static info
       status(ACTIVE), flow(SERIAL), blocking(YES), user(1), select(USER)
       restore(OUT), prescale(1), cue(100), select words(-1,-1,-1,-1,)
       lib = /usr/local/clas/release/current/coda/SunOS_i86pc/lib/libet_user.so,  function = et2et_function, class = 
     dynamic info
       attachments: total#(1),  ids(0,)
       input  list: cnt =     0, events in  = 0, events try = 0
       output list: cnt =     0, events out = 0
 LOCAL USERS:
   process #0, # attachments(1), attach ids(0,), pid(16189), hbeat(1)
   process #1, # attachments(1), attach ids(1,), pid(8432), hbeat(625)
   process #2, # attachments(1), attach ids(2,), pid(10990), hbeat(1672)
 ATTACHMENTS:
   att #0, is at station(ET2ET10) on host(clondaq2) at pid(16189)
   proc(0), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #1, is at station(TAPE) on host(clondaq2) at pid(8432)
   proc(1), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
   att #2, is at station(TAPE) on host(clondaq2) at pid(10990)
   proc(2), blocked(NO)
     events:  make(0), get(0), put(0), dump(0)
 EVENTS OWNED BY:
   system (4000),  att0 (0),  att1 (0),  att2 (0),
 EVENT RATE of GC = 0 events/sec
 CREATING STATIONS:  
 IDLE STATIONS:      
 STATION CHAIN:      GRAND_CENTRAL, TAPE, ET2ET10, 
 LOCKED MUTEXES:     

ET SYSTEM is DEAD!

^C clondaq2:stage_in>


Sergey B. 14-apr-2010: debugging clastrig2 endofrun problem

wait: coda request >end< in progress
0x916e80 (twork0028): TS GO bit cleared
0x916e80 (twork0028): TS csr:  0xfc070000
0x916e80 (twork0028): TS csr2:  0xffff1c1f
interrupt: TRIG ERROR: no pool buffer available
0x916e80 (twork0028): setForceSyncInterval: forceSyncInterval set to 0
0x916e80 (twork0028): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='croctest10'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done



Sergey B. 14-apr-2010: message from runcontrol during cancel-reset

Get all components failed: Lost connection to MySQL server during query

Sergey B. 6-apr-2010: EB waiting for following ROCs

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
 7  9 10 11 17 20 21 22 25 26 28 29 30

Files produced:

clon10:ccscans> ll | grep Apr
drwxrwxrwx   2 clasrun  onliners    4096 Apr  2 16:23 ./
drwxr-xr-x 163 clasrun  100        73728 Apr  6  2010 ../
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_cc1*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:09 good_clastrig2*
-rwxrwxrwx   1 boiarino clas          34 Apr  2 16:25 good_croctest10*
-rwxrwxrwx   1 boiarino clas          34 Apr  2 16:25 good_croctest2*
-rwxrwxrwx   1 boiarino clas          34 Apr  2 16:21 good_croctest3*
-rwxrwxrwx   1 baturin  clas          34 Apr  2 13:26 good_ctoftest1*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_dc1*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_dc10*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_dc11*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_dc2*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_dc3*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_dc4*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_dc5*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_dc6*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_dc7*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_dc8*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_dc9*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_ec1*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_ec2*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_ec3*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_ec4*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_polar*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_sc1*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_sc2*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_scaler1*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_scaler2*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_scaler3*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:26 good_scaler4*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_tage*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_tage2*
-rwxrwxrwx   1 clasrun  onliners      34 Apr  6 20:08 good_tage3* 
clon10:ccscans> more good_dc7
accepted link from >129.57.68.80<
clon10:ccscans> more good_dc8
accepted link from >129.57.68.74<
clon10:ccscans>

On 'Reset' EB crashed, outputs:

[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
 7  9 10 11 17 20 21 22 25 26 28 29 30
[0] 0x76fefffe != 0x008cf17e, waiting for the following ROC IDs:
 7  9 10 11 17 20 21 22 25 26 28 29 30
CODAtcpServer: start work thread
befor: socket=5 address>129.57.167.14< port=34468
wait: coda request >prestart< in progress
codaExecute reached, message >exit<, len=4
codaExecute: 'exit' transition
debShutdownBuild 1
cb_delete roc=0
cb_delete roc=1
cb_delete roc=2
cb_delete roc=3
cb_delete roc=4
cb_delete roc=5
cb_delete roc=6
cb_delete roc=7
cb_delete roc=8
cb_delete roc=9
cb_delete roc=10
cb_delete roc=11
cb_delete roc=12
cb_delete roc=13
cb_delete roc=14
cb_delete roc=15
cb_delete roc=16
cb_delete roc=17
cb_delete roc=18
cb_delete roc=19
cb_delete roc=20
cb_delete roc=21
cb_delete roc=22
cb_delete roc=23
cb_delete roc=24
cb_delete roc=25
cb_delete roc=26
cb_delete roc=27
cb_delete roc=28
cb_delete roc=29
cb_delete roc=30
cb_delete roc=31
debShutdownBuild 2
Stop waiting for the rocs because of force_end condition
[0] ROC mask 0x76fefffe force_end 1 end_event_done 0
[0] NEED TO TEST AND PROBABLY REDESIGN THAT PART !!!
[0] et_event_new() 1
[0] EVENT_ENCODE_SPEC 2 ..
BOS_encode_spec reached
BOS_encode_spec done: len=116
[0] .. done
[0] handle_build_cleanup
[0] flush ET system
[0] detached from ET
[0] remove mutex locks and shutdown fifos
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[00] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[00] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[00] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[01] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[01] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[01] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[02] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[02] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[02] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[03] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[03] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[03] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[04] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[04] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[04] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[05] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[05] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[05] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[06] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[06] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[06] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[07] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[07] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[07] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[08] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[08] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[08] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[09] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[09] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[09] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 3
[0] count for roc[10] = 3
CCCCCCCCCC: 3
CCCCCCCCCC: 1
[0] count for roc[10] = 1
CCCCCCCCCC: 1
CCCCCCCCCC: 0
[0] count for roc[10] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[11] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[11] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[12] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[12] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 2
[0] count for roc[13] = 2
CCCCCCCCCC: 2
CCCCCCCCCC: 0
[0] count for roc[13] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[14] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[15] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[16] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[17] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[18] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[19] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[20] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[21] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[22] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[23] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[24] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[25] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[26] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[27] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[28] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[29] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[30] = 0
CCCCCCCCCC: 0
[0] flush input streams
CCCCCCCCCC: 0
[0] count for roc[31] = 0
CCCCCCCCCC: 0
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] ============= Build threads cleaned
[0] build thread exiting: 1 1
[0] The implementation has detected that the value  specified by threa
d does not refer to a joinable thread.
wait: coda request >exit< in progress
polling_routine ================!!!!!!!!!!!!!!!!!!=====
cancel building threads
=c=============================================
=c=============================================
=c=============================================
=c====
>>>>> close link from >dc1< link=0x081c2af0
debCloseLink: theLink=0x081c2af0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc2< link=0x081c4a90
debCloseLink: theLink=0x081c4a90 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc3< link=0x081c4b28
debCloseLink: theLink=0x081c4b28 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc4< link=0x081c14f8
debCloseLink: theLink=0x081c14f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc5< link=0x081c1590
debCloseLink: theLink=0x081c1590 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc6< link=0x081c1628
debCloseLink: theLink=0x081c1628 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >dc8< link=0x081c16c0
debCloseLink: theLink=0x081c16c0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >cc1< link=0x081c1a08
debCloseLink: theLink=0x081c1a08 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >sc1< link=0x081c1aa0
debCloseLink: theLink=0x081c1aa0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec1< link=0x081c1b38
debCloseLink: theLink=0x081c1b38 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec2< link=0x081c1bd0
debCloseLink: theLink=0x081c1bd0 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler2< link=0x681d95c8
debCloseLink: theLink=0x681d95c8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler3< link=0x681d9660
debCloseLink: theLink=0x681d9660 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >scaler4< link=0x681d96f8
debCloseLink: theLink=0x681d96f8 -> closing
debCloseLink: link is down
debCloseLink: free memory
debCloseLink: done.
=c====
>>>>> close link from >ec4< link=0x00000001
debCloseLink: theLink=0x00000001 -> closing
Segmentation fault
clondaq1:coda_eb> 

DC7:

codaExecute done
wait: coda request >download prod< in progress
codaExecute reached, message >prestart<, len=8
codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ? 
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410567520
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod WHERE name='dc7'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1< 
3
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc7< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc7<
LINK_constructor_C: set name to >dc7->EB1< 
3121: dbsock=410567520
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda req<<<
uest >fields [pr2estart] >>>< in prodowngress
<<<
fields [3] >>>37683<<<
parsing results: type=>TCP< host=>clondaq1-daq1< state=>down< port=>37683< -> 37683
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129.57.68.22 port 37683
DB command >UPDATE links SET state='up' WHERE name='dc7->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x18772230 (twork0028INFO: Entering Us): er 
Prestart 2
INIT_NAME: rolp->daproc = 
trans_rol2:2
adr(nddl)=1db60750 NWBOS=16384
uthbook1: WARN id=1047 already exist - will be replaced
uthbook1: free histogram buffer ... done.
RAW=0  PROFILE=1  NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x18772230INFO: User Prest (art 2 executed
twork0028): CTRIGRSS: set handler and done, 
code=1
0x18772230 (twork0028): CTRIGRSS: 0x1877da9c
0x1877d9c0 0x1877d2e0 0x00000001 0x1877d2d8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4130) returns 490106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a580) returns 490106296
0x18772230 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x18772230 (twork0028): INFO: Entering User 
Prestart
disconnecting vector 236 
0x18772230 (twork0028): CTRIGRSA: set handler and done, code=1
0x18772230 (twork0028): CTRIGRSA: 0x1882e7bc
0x1882d864 0x00000001 0x1882b0c8
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1  = FFFFF000
Fastbus Status Register2  = FFFF0000
Fastbus Last Primary Addr = 00000019
spds_mask1: 0xfdfbf0
0x18772230_tdc1877 ===> (4twork0028wait: co)
:  Use 32-bit Ram List
da request >firsprestartt< in progre csr0=ss
100 (1100) csr18=1364 csr1=6400001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=1364 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0=100 (900) csr18=13
bignetptr=0x641d366fe0 csr1= offset=0x00bigp
rocptr=0x000001d366db80 offset=0x6400000000
00001b
bigproc at 0x
1d366db8, bigproc.gbigBuffer at 0x_tdc1877 =
==>1d366dc818 -> 0x1d366dc8
proc_thread reacheUDP_cancel: cancel >inf:d
c7 sys 0, mask 1d
<
firstUDP_cancel: cancel >proc_threainf:dc7 sys 0, mask 1d reached
<
wait: cproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
csr0=oda request >1prestart00< in progr (ess1100) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot#   1872  1875  1881(2)  1877(2)  1877S(2)
[ 0]      0     0      0 0bignetptr=0x      
1d366fe00 offset=0x0000000 0
0       0 0
[ 1]      0     0      0 0      0 0       0 
0
[ 2]      0     0      0 0      0 0       0 
0
[ 3]      0     0      0 0      0 0       0 
0
[ 4]      0     0      0 0      1 0       0 
0
[ 5]      0     0      0 0      1 0       0 
0
[ 6]      0     0      0 0      1 0       0 
0
[ 7]      0     0      0 0      1 0       0 
0
[ 8]      0     0      0 0      1 0       0 
0
[ 9]      0     0      0 0      1 2       0 
0
[10]      wait: coda reques0t >     prestart
0<       in progress
0 0      0 0       0 0
[11]      0     0      0 0      1 0       0 
0
[12]      0     0      0 0      1 0       0 
0
[13]      0     0      0 0      1 0       0 
0
[14]      0     0      0 0      1 0       0 
0
[15]      0     0      0 0      1 0       0 
0
[16]      0     0      0 0      1 2       0 
0
[17]      0     0      0 0      0 0       0 
0
[18]      0     0      0 0      1 0       0 
0
[19]      0     0      0 0      1 0       0 
0
[20] bignet at 0x     1d366fe00, bignet.gbig
Buffer at 0x     1d366ff00 -> 0x1d366ff0
     0 0      1 0       0 0
[21]      0     0      0 0      1 0       0 
0
[22]      0     0      0 0      1 0       0 
0
[23]      0     0      0 0      1 2       0 
0
[24]      0     0      0 0      0 0       0 
0
[25]      0     0      0 0      0 0       0 
0
DEBUG 0 1 0 0 
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC07<
0x18772230 (twork0028111): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 7 1 17 0 - 0x00000004 0x001101cc 0x000004b0 0x0000f420 0x0000002f 0x00000000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatu>>>>>>>>>>>>>>>> use pid=-1 <
<<<<<<<<<<<<<<<<
s: >UPDATE process SET state='paused' WHERE 
name='dc7'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request .. 
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_standard_request >sta:dc7 paused<
UDP_cancel: cancel >sta:dc7 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
-> 
-> 

DC8:

codaExecute: 'prestart' transition
roc_prestart reached
partReInitAll() reached ?
partReInitAll() reached !
partReInitAll() done
rocCleanup reached
partReInitAll() reached ?
wait: coda request >prestart< in progress
partReInitAll() reached !
partReInitAll() done
315: dbsock=410505056
>>> prestarting, run 62496, type 47
rocMask=0x76fefffe
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
 WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
 WHERE name='dc8'<
!!!!!!!!!!!!! tmpp >SELECT outputs FROM prod
 WHERE name='dc8'<
++++> output to network
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1< 
3
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
!!!!!!!!!!!!! for rocOpenLinks: >dc8< >EB1<
rocCloselink reached
rocOpenLinks reached
LINK_constructor_C: set host to >dc8<
LINK_constructor_C: set name to >dc8->EB1<
3121: dbsock=410505056
name selected
nrow=1
fields [0] >>>TCP<<<
fields [1] >>>clondaq1-daq1wait: coda re<<<
quest >fields [p2restart] >>>< in prupogress
<<<
fields [3] >>>37772<<<
parsing results: type=>TCP< host=>clondaq1-d
aq1< state=>up< port=>37772< -> 37772
LINK_establish: socket # 17
LINK_establish: socket buffer size is 48000(
0x0000bb80) bytes
LINK_establish: keepAlive is 8
LINK_establish: socket 17 is ready: host 129
.57.68.22 port 37772
DB command >UPDATE links SET state='up' WHER
E name='dc8->EB1'<
UPDATE process success
Spawn proc/net threads
Spawn proc/net threads
Spawn proc/net threads
0x187ef280INFO: Entering U (ser Prestart 2
twork0028
trans_rol2:):  adr(nddl)=INIT_NAME: rolp->da
proc = 1d2b60750
 NWBOS=16384
uthbook1: WARN id=1048 already exist - will 
be replaced
uthbook1: free histogram buffer ... done.
RAW=0  PROFILE=1  NOTRANS=0
MAX_EVENT_LENGTH = 65536 NWBOS = 16384
0x187ef280 (twork0028INFO: User Pr): estart 
2 executedCTRIGRSS: set handler and done, co
de=
1
0x187ef280 (twork0028): CTRIGRSS: 0x187705ac
 0x187704d0 0x1876fdf0 0x00000001 0x1876fde8
wait: coda request >prestart< in progress
bignetptr=0x1d366fe0 offset=0x00000000
taskSpawn("coda_net",0x187d4110) returns 490
106848
wait: coda request >prestart< in progress
setHeartError: 0 >sys 0, mask 1<
WARN: HeartBeat[0]: heartbeat=19424(19424) h
eartmask=1
bignetptr=0x1d366fe0 offset=0x00000000
bigprocptr=0x1d366db8 offset=0x00000000
taskSpawn("coda_proc",0x1874a560) returns 49
0106296
0x187ef280 (twork0028): INIT_NAME: rolp->dap
roc = 2
0x187ef280 (twork0028): INFO: Entering User 
Prestart
disconnecting vector 236 
0x187ef280 (twork0028): CTRIGRSA: set handle
r and done, code=1
0x187ef280 (twork0028): CTRIGRSA: 0x1882e13c
 0x1882d1e4 0x00000001 0x1882aa48
Sequencer Status Register = FFFFA000
Last Sequencer KeyAddress = FFFF0003
Fastbus Status Register1  = FFFFF000
Fastbus Status Register2  = FFFF0000
Fastbus Last Primary Addr = wait: coda req00
0000uest >19presta
rtspds_mask1: 0x< ifdfbf0n progress 
0x187ef280_tdc1877 === (>twork0028): Use 32-
bit Ram List
4 first csr0=100 (1100) csr18=1364 csr1=6400
001b
_tdc1877 ===>5 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>6 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>7 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>8 middle csr0=100 (1900) csr18=
1364 csr1=6400001b
_tdc1877 ===>9 last csr0=100 (900) csr18=136
4 csr1=6400001b
_tdc1877 ===>11 first csr0=100 (1100) csr18=
1364 csr1=6400001b
_tdc1877 ===>12 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>13 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>14 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>15 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>16 last csr0bignetptr=0x=1d366f
e0100 offset=0x0000000 (0
900) csr18=1364bigprocptr=0x csr1=1d366db864
00001b offset=0x
00000000
bigproc at 0x_tdc1877 ===1d366db8>, bigproc.
gbigBuffer at 0x181d3UDP_cancel: cancel >inf
:dc8 sys 0, mask 166dc8<
wait: coda rUDP_cancel: cancel >inf:dc8 sys 
0, mask 1 -> 0x<
1d366dc8eq
firstproc_thread reached
uest >proc_thread reached
 csr0=proc_thread reached
prproc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
proc_thread reached
100estart (< in prog1100ress
) csr18=1364 csr1=6400001b
_tdc1877 ===>19 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>20 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>21 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>22 middle csr0=100 (1900) csr18
=1364 csr1=6400001b
_tdc1877 ===>23 last csr0=100 (900) csr18=13
64 csr1=6400001b
DEBUG in SFIRamLoad => 0 1 0 0 0
slot#   1872  bignetptr=0x1875  181d366fe081
(2)  1877( offset=0x2)  1877S(00000002)
0
[ 0]      0     0      0 0      0 0       0 
0
[ 1]      0     0      0 0      0 0       0 
0
[ 2]      0     0      0 0      0 0       0 
0
[ 3]      0     0      0 0      0 0       0 
0
[ 4]      0     0      0 0      1 0       0 
0
[ 5]      0     0      0 0      1 0       0 
0
[ 6]      0     0      0 0      1 0       0 
0
[ 7]      0     0      0 0      1 0       0 
0
[ 8]      0     0      0 0      1 0       0 
0
[ 9]      0     0      0 0      1 2       0 
0
[10]      wait: coda request0 >     prestart
0< i      n progress
0 0      0 0       0 0
[11]      0     0      0 0      1 0       0 
0
[12]      0     0      0 0      1 0       0 
0
[13]      0     0      0 0      1 0       0 
0
[14]      0     0      0 0      1 0       0 
0
[15]      0     0      0 0      1 0       0 
0
[16]      0     0      0 0      1 2       0 
0
[17]      0     0      0 0      0 0       0 
0
[18]      0     0      0 0      1 0       0 
0
[19]      0 bignet at 0x    1d366fe00, bigne
t.gbigBuffer at 0x      1d366ff00 -> 0x1d366
ff0
 0      1 0       0 0
[20]      0     0      0 0      1 0       0 
0
[21]      0     0      0 0      1 0       0 
0
[22]      0     0      0 0      1 0       0 
0
[23]      0     0      0 0      1 2       0 
0
[24]      0     0      0 0      0 0       0 
0
[25]      0     0      0 0      0 0       0 
0
DEBUG 0 1 0 0 
INFO: datascan = 0x00000000
INFO: datascan&spds_mask1 = 0x00000000
rcname >RC08<
0x187ef280 (twork0028111
): INFO: User Prestart Executed
333: rocp->primwait: coda request >efd=prest
art17< in p
rogress
informEB reached
informEB: 11 -1 8 1 17 0 - 0x00000004 0x0011
01cc 0x000004b0 0x0000f420 0x0000002f 0x0000
0000
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state=
'paused' WH>>>>>>>>>>>>>>>> use pid=-1 <<<<<
<<<<<<<<<<<<
ERE name='dc8'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_standard_request >sta:dc8 paused<
UDP_cancel: cancel >sta:dc8 downloaded<
codaUpdateStatus: updating request done
prestarted
POLLS: 0 0
codaExecute done
.....


Sergey B. 6-apr-2010: after TS problem described below, polar self rebooted on next download

........

wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
3123: dbsock=59553504
3123: tmpp>SELECT id FROM runTypes WHERE name='prod'<
====1==== !!! object->runType=47
token_interval=64
object->name >polar<
big0.proc_on_pmc=0, big0.net_on_pmc=0
big1.proc_on_pmc=0, big1.net_on_pmc=0
name selected
nrow=1
download: name >polar<
code selected
nrow=1
download: code >{$CODA/VXWORKS_ppc/rol/moller.o usr} {$CODA/VXWORKS_ppc/rol/rol2_tt.o usr} <
nrols [0] >$CODA/VXWORKS_ppc/rol/moller.o usr<
nrols [1] >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
listArgc=2 listArgv >$CODA/VXWORKS_ppc/rol/moller.o usr< >$CODA/VXWORKS_ppc/rol/rol2_tt.o usr<
set this_roc_id = 25, rocId() can be called from now on
ObjInitName >_moller__init< 
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
INFO: >_moller__init()< routine found
readout list >$CODA/VXWORKS_ppc/rol/moller.o< loaded
allocate 200 buffers with length 65536 bytes
partCreate1() reached
partCreate1(): pPart=0x008e7570
listInit: cleanup 24 bytes starting from address 0x008e75a0
partCreate1() done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = 0
0x81c7e0 (twork0004): INIT_NAME: Initializing new rol structures for SCALROL1
0x81c7e0 (twork0004): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
0x81c7e0 (twork0004): INIT_NAME: name >SCALROL1:pool<
0x81c7e0 (twork0004): Init - Done
0x81c7e0 (twork0004): INIT_NAME: rolp->daproc = rol11: downloading DDL table ...
wait: coda request >download prod< in progress
wait: coda request >download prod< in progress
Exception at interrupt level:
Exception next instruction address: 0xfff901f0
Fixed Point Register: 0x00000012
Condition Register: 0x40482044
Regs at 0x64e498
Press any key to stop auto-boot...
2
[VxWorks Boot]: 

It suppose to be as following:

.....
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 0
0x91ae30 (twork0001): INIT_NAME: Initializing new rol structures for SCALROL1
0x91ae30 (twork0001): INIT_NAME: MAX_EVENT_LENGTH = 65536 bytes, MAX_EVENT_POOL = 200
rol1: downloading DDL table ...
0x91ae30 (twork0001): INIT_NAME: name >SCALROL1:pool<
0x91ae30 (twork0001): Init - Done
0x91ae30 (twork0001): INIT_NAME: rolp->daproc = 1
wait: coda request >download prod< in progress
adr1(nddl_)=0x037b69c8
N   name (nname)     fmt (nfmt)     ncol
[ 1]   >PTRN< (4)         >B32< (3)     1
[ 2]   >PSYN< (4)         >B32< (3)     1
[ 3]   >RC00< (4)         >B32< (3)     1
[ 4]   >RC01< (4)         >B32< (3)     1
.....


Sergey B. 6-apr-2010: TS busy condition at the beginning of the run, all rocs have 205 events, clastrig2 has 200 events, ts2status output

-> ts2status
CSR 1 (0x0):
                                     Go : 1
           Pause on Next scheduled Sync : 0
                         Sync and Pause : 0
                    Initiate Sync Event : 0
               Initiate Program 1 Event : 0
               Initiate Program 2 Event : 1
        Enable Level 1 (drives outputs) : 1
                       Override Inhibit : 0
                              Test Mode : 0
                               Reserved : 0
                                  Reset : 0
                             Initialize : 0
                    Sync Event occurred : 0
               Program 1 Event occurred : 1
               Program 2 Event occurred : 0
                     Late Fail occurred : 0
                       Inhibit occurred : 1
              Write FIFO error occurred : 0
              Read FIFO error occurreds : 1
CSR 2 (0x4):
                     Enable Scheduled Sync : 1
                    Use Clear Permit Timer : 1
                      Use Front Busy Timer : 1
                      Use Clear Hold Timer : 1
                   Use External Front Busy : 1
                         Lock ROC Branch 1 : 0
                         Lock ROC Branch 2 : 0
                         Lock ROC Branch 3 : 0
                         Lock ROC Branch 4 : 0
                         Lock ROC Branch 5 : 0
        Enable Program 1 front panel input : 1
                          Enable Interrupt : 1
               Enable local ROC (branch 5) : 1
Trigger Control Register (0x8): 0x00001fff
ROC Enable Register (0xc) val=0xf07fff7f:
Branch 1: 0x7f  bits: 01111111
Branch 2: 0xff  bits: 11111111
Branch 3: 0x7f  bits: 01111111
Branch 4: 0xf0  bits: 11110000
Synchronization Interval Register (0x10): 1000
Trigger Word Count Register (0x14): 0
Trigger Data Register (0x18): 256
Local ROC (Branch 5) Data Register (0x1c): 60
                      Synchronization Flag : 0
                            Late Fail Flag : 0
                                  ROC code : 15
Input Trigger Prescale Registers:
                Input 1 Prescale Factor : 0
                Input 2 Prescale Factor : 0
                Input 3 Prescale Factor : 0
                Input 4 Prescale Factor : 0
                Input 5 Prescale Factor : 0
                Input 6 Prescale Factor : 0
                Input 7 Prescale Factor : 0
                Input 8 Prescale Factor : 0
Clear Permit Timer Register  = 0
Level2 Accept Timer Register = 83
Level3 Accept Timer Register = 0
Front Busy Timer Register    = 325
Clear Hold Timer Register    = 50
Branch (1-4) ROC Buffer Status Register (0x58): 0x40404040
Branch 1: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 2: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 3: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Branch 4: Buffer Count = 0, Empty Flag = 1, Full Flag = 0
Local ROC (Branch 5) Buffer Status Register (0x5c): 0xffff8086
Branch 5: Buffer Count = 6, Empty Flag = 0, Full Flag = 1,Local Acknowledge = 0
ROC Acknowledge Status Register (0x60): val=0x0
Branch 1: 0x 0  bits: 00000000 (enabled: 01111111)
Branch 2: 0x 0  bits: 00000000 (enabled: 11111111)
Branch 3: 0x 0  bits: 00000000 (enabled: 01111111)
Branch 4: 0x 0  bits: 00000000 (enabled: 11110000)
State Register (0x6C):
                      Level 1 Accept     : 0
                   Start Level 2 Trigger : 0
                    Level 2 Pass Latched : 0
                    Level 2 Fail Latched : 0
                          Level 2 Accept : 0
                   Start Level 3 Trigger : 0
                    Level 3 Pass Latched : 0
                    Level 3 Fail Latched : 0
                          Level 3 Accept : 0
                                   Clear : 0
               Front End Busy (external) : 0
                        External Inhibit : 0
                         Latched Trigger : 0
                                 TS Busy : 1
                               TS Active : 1
                                TS Ready : 0
                   Main Sequencer Active : 0
        Synchronization Sequencer Active : 0
        Program 1 Event Sequencer Active : 1
        Program 2 Event Sequencer Active : 0
Event Count (0xc8): 135
Live 1 Count (0xcc): 54866
Live 2 Count (0xd0): 9290375
value = 1 = 0x1
-> exit
clasprod::clastrig2> clastrig2 statistics
200 0 0 0


Sergey B. 6-apr-2010: end failed, only one ROC was in use (clastrig2), messages

interrupt: Program Events 1-2:      247        0
interrupt: live time = 93 percent (gated=1956740 ungated=2094143)
interrupt: live_corr = 93 percent (gated=1956740 ungated=2094143)
proc_thread: waiting=   3743 processing=     30 microsec per event (nev=404)
rols_thread: waiting=    754 processing=     21 microsec per event (nev=454)
wait: coda request >go< in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
roc_end reached ??
roc_end reached !!
0x18c6f5c0 (twork0003): INIT_NAME: rolp->daproc = 3
UDP_cancel: cancel >ts2: 93 0 0 0 0 0 0 0 0 507 0 0 0 <
interrupt: inputs  1-6:        0        0        0        0        0        0
interrupt: inputs 7-12:        0        0      339        0        0        0
interrupt: Program Events 1-2:      172        0
interrupt: live time = 97 percent (gated=1358086 ungated=1400034)
interrupt: live_corr = 97 percent (gated=1358086 ungated=1400034)
wait: coda request >end< in progress
0x18c6f5c0 (twork0003): TS GO bit cleared
0x18c6f5c0 (twork0003): TS csr:  0xfc000000
interrupt: TRIG ERROR: no pool buffer available
0x18c6f5c0 (twork0003): setForceSyncInterval: forceSyncInterval set to 0
0x18c6f5c0 (twork0003): INFO: User End Executed
codaUpdateStatus: dbConnecting ..
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='ending' WHERE name='clastrig2'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_standard_request >sta:clastrig2 ending<
UDP_cancel: cancel >sta:clastrig2 active<
codaUpdateStatus: updating request done
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
NOW !!!!!!!!!!!!!!!!!!!
roc_end done
codaExecute done
coda_roc: processing case 'DA_ENDING': last event=12113 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=20212 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING dataInBuf=7224 BBHEAD_BYTES=24
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty
coda_roc: process rol1 buffer by rol2 completed
coda_roc: break loop
coda_roc: processing case 'DA_ENDING': last event=12313 nevents=12314
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: in DA_ENDING last event=12313 nevents=12314
coda_roc: rocp->state == ENDING and input list is empty


Sergey B. 5-apr-2010: scrolling messages from all ROCs (start from Download - works fine)

.....
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last 
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last 
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING'
coda_roc: processing case 'DA_ENDING': last 
event=2405 nevents=2405
coda_roc: processing case 'DA_ENDING'
.....

Sergey B. 2-apr-2010: event recorder gives error during Prestart

file[ 0]->/raid/stage_in/clas_062414.A00<-
bosopen.c: splitted files handling ..
bosopen.c: >/raid/stage_in/clas_062414.A00<
bosopen.c: len1=30
bosopen.c: len2=30
bosopen.c: len3=14
bosopen.c: >/raid/stage_in/clas_062414.A00< len=14
bosopen.c: checking how much disk space left in directory /raid/stage_in
bosopen.c: total 1403778969 blocks, free 1100988411 blocks, available 
1100988411 blocks (1 block = 512 bytes)
bosopen.c: so we have 550493696 kbytes available space
bosopen.c: we can continue to write file.
bosopen.c: check for partition swaping: file->/raid/stage_in/clas_062414.A00<-
bosopen.c: system call: >checkdisk  4094<
[1] 9253
bosopen.c: system call completed
codaUpdateStatus: dbConnecting ..
ER: Not attached to ET system
er_write_thread loop ended (0 145452664)
codaUpdateStatus: dbConnect done
codaUpdateStatus: >UPDATE process SET state='paused' WHERE name='ER3'<
codaUpdateStatus: dbDisconnecting ..
codaUpdateStatus: dbDisconnect done
codaUpdateStatus: updating request ..
UDP_standard_request >sta:ER3 paused<
UDP_standard_request >sta:ER3 paused<

Station TAPE was idle, but et system seems operational; end run (successful), restart coda_er, start from Download - everything works fine.

Sergey B. 18-mar-2010: previous run ended fine, click 'prestart' and got from EB

[0] 0x76fefffe != 0x40acf17e, waiting for the following ROC IDs:
7  9 10 11 17 20 22 25 26 28 29

Similar problem was observed before. Kill rcServer, next run started fine, ROCs were NOT rebooted. CC scans again ???

Sergey B. 14-mar-2010: old problem, can be posted already

If EB was not restarted afre the ROC crash, it sometimes waiting for several ROcs in the end of prestart, however runcontrol allows 'Go' button become active; something wrong with the logic, must check; will be good of course to avoid that situation by making EB reset itself properly on 'reset' transition after the ROC crash

Sergey B. 14-mar-2010: during reboot dc9 gives following

.................
ppc/bootscripts/boot_dc9
taskSpawn("TCPSERVER") returns 475458320
bind on port 5001
myname >dc9<
-> Query >UPDATE Ports SET Host='dc9',tcpClient_tcp=5001 WHERE Name='dc9'< succeeded
INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
333333333333333
program
Exception current instruction address: 0x1c5
e8734
Machine Status Register: 0x00081000
Condition Register: 0x40000085
Task: 0x1c5e8e50 "ROC"
interrupt: Unconnected main interrupt 0
logTask: 3222 log messages lost.
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
interrupt: Unconnected main interrupt 1
interrupt: Unconnected main interrupt 0
..............

probably mysql connection problem

RESOLVING: will make 5 attempts to to connect waiting 3 seconds in between, then will give up.

Sergey B. 14-mar-2010

If in prestart stage hit abort and reset, all ROCs printing

wait: coda request >exit< in progress

indefinitely, must check

Sergey B. 14-mar-2010: coda_eb x-term hung completely, kill coda_eb but yellow window still stuck; last messages were

codaEnd 10
codaEnd 11
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
codaExecute done
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59319
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
codaExecute: ERROR: len=0 - do nothing
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=59342
wait: coda request >< in progress
Executing >TP/1.0

Probably last string contains something, will remove printing

Sergey B. 13-mar-2010: sometimes after bad crash all rocs (even scalers) must be rebooted, otherwise EB is crashing with messages

.....
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
case 0: no swap
 .. done.
[0] FATAL: Event (Num 1 type 1) NUMBER mismatch -- roc[11] (rocid 30) 
sent -1 (type 18)
[0] ERROR: Discard data until next control event
clondaq1:coda_eb>

After that problem stays, restarted every daq component (EB, ER, ETs, runcontrol, rcServer) - does not helped. Did daq_exit, daq_start, reboot all rocs again - not it works.

In the same time ROCs were printing messages 'no data', may be it was the source of EB troubles ...

Sergey B.12-mar-2010 around 10:50am: something is breaking coda_eb/coda_er

.....
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_standard_request >sta:EB1 booted<
UDP_cancel: cancel >sta:EB1 booted<
codaUpdateStatus: updating request done
CODA_Init 14
2
clasprod::EB1> bind on port 5001
DB update: >UPDATE process SET inuse='5001' WHERE name='EB1'<
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.28< port=37258
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
wait: coda request >< in progress
Executing >< (len=0)
Segmentation fault (core dumped)
clondaq1:coda_eb> 

In the same time et_start complains (we have now ET system protected against CC scans):

et ERROR: et_netserver: ET server being probed by non-ET client or read failure
et ERROR: et_netserver: ET server being probed by non-ET client or read failure

FIXED by checking for the len=0 in the message string, all other possible checks were implemented already

Sergey B. 12-mar-2010: global reboot gives almost all PMCs (except first two) failing on MySQL access

Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e404190 "coda_pmc"

It should be mentioned that we are doing something right now; day before it was no activity and all rebooted fine.

Sergey B. 1-sep-2009: during croctest1 reboot

INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
mysql_real_connect9: error message: 2013/HY000 (Lost connection to MySQL server during query)
dbConnect ERROR: mysql == NULL

second reboot went fine

Sergey B. 15-june-2009

Just after run was ended noticed that all runs starting from 60073 has 'No configuration!' instead of 'PROD' etc. Correcting manually. Must check if other information is correct, and understand the reason if possible .. Is it related to the database update procedure inplemented around that time ? It appeares that begin_clasprod_xxxxxx.txt log files has that wrong starting from run 60073.

Sergey B. 11-June-2009: swaped ec4 and sc2 and during sc2 boot see following from host

..........................
taskDelay(sysClkRateGet()*5)
 Args = -session clasprod -objects sc2 ROC -i
CODA_Init reached
CODA_Init 1
CODA_Init 2
sc2
CODA_Init 11
CODA_Init: objectTy >(null)<
CODA_Init 12
CODA_Init: use 'SESSION' as >clasprod<
Tcl_AppInit CALLS !!!!!
11-11
11-12
11-13
11111111111111111
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
value = 0 = 0x0
taskSpawn "TCP_SERVER",250,0,100000,tcpServer
value = 231091088 = 0xdc62b90
proconhost
value = 0 = 0x0
......................

Second reboot shows the same:

...................
22222222222222222
0xdd0e880 (tNetTask): arptnew failed on 8139a743
>>>>>>>clon10 clasrun 2508 9998<<<<<<<<< 24
list >clon10 clasrun 2508 9998<
machine name = clon10
user ID      = 2508
group ID     = 9998
.................


Sergey B. around 6-June-2009: ec2pmc1 error message

write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: about to print: 255115 20
write_thread: wait=    199 send=      8 microsec per event (nev=12755)
CALLING customized :
machine check 
SRAM ERROR :
SRAM Error Address    : 0x00000000.f2000504
SRAM Error Data Low   : 0x006805e0
SRAM Error Data High  : 0x00420000
SRAM Error Parity     : 0x00000004
SRAM Error Cause      : 0x00000001
PCI_0 DEVICE ERROR :
PCI_0 Error: Status=0x00000100
Master abort
PCI Cmd=7 (MemWr)  ByteEnable=0  Par=0
Error Address High: 0x00000000
Error Address Low : 0xf0d10128
CALLING generic    :
machine check
Exception next instruction address: 0x0001f058
Machine Status Register: 0x0012b030
Condition Register: 0x40222042
Task: 0x1e3b0020 "coda_net"

reboot did not helped, have to turn pmc off and run on host only


Sergey B. 4-June-2009: when rebooting all rocs, some cannot connect to DB, as result have following

0x1e42a360 (coda_pmc): bb_new: 'big' buffer 
created (addr=0x1eba3590, 16 bufs, 3145728 size)
coda_pmc: big buffer1 allocated
myname >ec1pmc1<
Done executing startup script $CODA/VXWORKS_
ppc/bootscripts/boot_pmc1
-> INFO(mysql_real_connect9): errno=0
INFO(mysql_real_connect9): OK
program
Exception current instruction address: 0x00000000
Machine Status Register: 0x0008b030
Condition Register: 0x40000085
Task: 0x1e42a360 "coda_pmc"
-> tt
14cad8 vxTaskEntry    +68 : coda_pmc ()
1e2275c0 coda_pmc       +144: mysql_query ()
1e258520 mysql_query    +54 : mysql_real_query ()
1e249850 mysql_real_query+110: mysql_send_query ()
1e249718 mysql_send_query+1d0: 0 ()
value = 0 = 0x0
-> 

for some reason it efefcts mostly PMCs. First 10 or so PMCs booted fine, and then few shows that error, and few following booted fine again, and so on. It looks like DB does not respond right the way, and PMC exits on timeout and do not try again. Should fix that place.


Sergey B. 15-may-2009: last night following error messages from sc2 were reportedly associated with 2 crashes (I was not on shift)

write_thread: about to print: 394603 20
write_thread: about to print: 394603 20
write_thread: wait=    176 send=      6 microsec per event (nev=19730)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=11)
proc_thread: wait=    176 send=     12 microsec per event (nev=14534)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=19)
0x1b1c7e00 (coda_proc): timer: 9 microsec (min=3 max=909 rms**2=17)
bosMgetid: ERROR: lenn=0
???: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: ERROR: name >RC-2< does not described, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
try again ..
bosMgetid: ERROR: lenn=0
!!!: 0x52432d32 0x38363333 0x31313534 0x00696d3d 0x25640a00
bosMgetid: lenn=4, len=4
bosMgetid: ERROR: name >RC-2< does not descr
ibed, update clonbanks.ddl file !!!
bosMgetid: lenn=4, len=4, nddl1=116
bosMgetid: name=>RC-286331154<
no way !!
bosMlink: ERROR: bosMgetid returns -99
interrupt: timer: 48 microsec (min=9 max=195 rms**2=9)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=18)
interrupt: timer: 45 microsec (min=9 max=195 rms**2=10)
interrupt: timer: 48 microsec (min=9 max=195 rms**2=14)
setHeartError: 0 >sys 0, mask 14<
WARN: HeartBeat[0]: heartbeat=6658(6658) heartmask=14
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<
UDP_cancel: cancel >inf:sc2 sys 0, mask 14<



Sergey B. 14-May-2009 5:54am

Got page from clascron@clon10: 'Process monitor: missing process: alarm_server - Online problem: alarm_server did not respond to status pool request'. Found later that stadis was not updating etc. It appeares that smartsockets server was hund somehow on clondb1, for example command 'ipc_info -a clasprod' from clon10 or clon00 did not worked, messages were:

clon00:clasrun> ipc_info -a clasprod
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <local> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <local> RTserver
08:08:31: TAL-SS-00093-I Skipping starting <start_never:local> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using local protocol
08:08:31: TAL-SS-00090-I Could not connect to <clondb1> RTserver
08:08:31: TAL-SS-00088-I Connecting to project <clasprod> on <clondb1> RTserver
08:08:31: TAL-SS-00089-I Using tcp protocol
^C

and it waits here. After restarting server on clondb1 by commands

/etc/init.d/smartsockets stop
/etc/init.d/smartsockets start

everything came back to normal. We should think about detecting that problem and restarting server automatically, also it happens very rearly, few time a year. Probably it can be done by monitoring cpu usage: when server was hung it was using 100% of cpu:

top - 08:06:16 up 130 days,  8:59,  1 user,  load average: 2.13, 2.03, 1.64
Tasks:  92 total,   2 running,  90 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.0% us,  0.1% sy,  0.0% ni, 73.0% id,  1.9% wa,  0.0% hi,  0.1% si
Mem:   3995356k total,  3968880k used,    26476k free,    88168k buffers
Swap:  8385920k total,      208k used,  8385712k free,  3432056k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                       
 4471 root      25   0  5908 4540 1956 R  100  0.1 300:24.06 rtserver.x                                                                                                     
    1 root      16   0  4756  556  460 S    0  0.0   0:11.25 init                                                                                                           
    2 root      RT   0     0    0    0 S    0  0.0   0:03.85 migration/0                                                                                                    
    3 root      34  19     0    0    0 S    0  0.0   0:10.23 ksoftirqd/0                                                                                                    
    4 root      RT   0     0    0    0 S    0  0.0   0:01.79 migration/1                                                                                                    
    5 root      34  19     0    0    0 S    0  0.0   1:03.72 ksoftirqd/1                                                                                                    
    6 root      RT   0     0    0    0 S    0  0.0   0:01.62 migration/2

while normally it uses fractions of percent. It is also will be useful to monitor cpu usage in general, few days ago clasrun's sshd on clon10 was taking almost 50% of cpu.


Sergey B. 12-may-2009

Looking for the reason of missing gate messages in all DC crates we found that front end busy cable from SF to FC was disconnected on the floor. For the future we should remember that disconnecting that cable may create missing gate messages. It is also important before every run to use 'dc_all' configuration and look at scope in the counting house to make sure front end busy signal is there.


Sergey B. 16-feb-2009: runcontrol pop-up message while pressing 'Configure'

rcConfigure::loadRcDbaseCbk: Loading database failed !!! On second 'Configure' click it worked.

Sergey B. June-2008

It looks like when one SILO drive goes bad, something is happening in script's logic and it never tries to run 2 streams, only one ! One .temp link can be observed, aling with one presilo1 link which is NOT becoming another .temp. Need check !!!

Sergey B. 7-may-2008: EB1 crashed, last messages

EB: WARNING - resyncronization in crate controller number  5 ...  fixed
.
EB: WARNING - resyncronization in crate controller number  6 ...  fixed
.
EB: WARNING - resyncronization in crate controller number  6 ...  fixed
.
ERROR: lfmt=0 bankid=0

Found that tage2 and tage3 must be rebooted


Sergey B. 5-may-2008

polar crate turned off; turn it back on, but notice that one of str7201 scalers has all lights on; scaler was replaced in 2 days, everything works fine


Sergey B. 30-apr-2008: gam_server on clon04 takes 100% CPU

Found on the web that creating file '/etc/gamin/gaminrc' with following contents will help:

#configuration for gamin
# Can be used to override the default behaviour.
# notify filepath(s) : indicate to use kernel notification
# poll filepath(s)   : indicate to use polling instead
# fsset fsname method poll_limit : indicate what method of notification for the filesystem
#                                  kernel - use the kernel for notification
#                                  poll - use polling for notification
#                                  none - don't use any notification
#
#                                  the poll_limit is the number of seconds
#                                  that must pass before a resource is polled again.
#                                  It is optional, and if it is not present the previous
#                                  value will be used or the default.
fsset nfs poll 10
# use polling on nfs mounts and poll once every 10 seconds
# This will limit polling to every 10 seconds and seams to prevent it from running away

Created file, will see if it helped ..


Sergey B. 20-apr-2008: sc2pmc1 error message (during "end run' ???)

.................................................
write_thread: wait=    135 send=     28 microsec per event (nev=12351)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=4)
proc_thread: wait=    130 send=     31 microsec per event (nev=14709)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=10)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=15)
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: about to print: 252148 20
write_thread: wait=    135 send=     31 microsec per event (nev=12607)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
proc_thread: wait=    135 send=     31 microsec per event (nev=14739)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=19)
0x19c18050 (coda_proc): timer: 23 microsec (min=8 max=2161 rms**2=14)
 ERROR: bufout overflow - skip the rest ...
        bufout=485322144 hit=485387636 endofbufout=485387680
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: about to print: 255564 20
write_thread: wait=    128 send=     42 microsec per event (nev=12778)
proc_thread: wait=    371 send=     31 microsec per event (nev=6291)
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000140 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x0150107F -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x015810DC -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x02B80FD6 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x02A010BE -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
.................................................. 


At the same time at sc2:

......................................
interrupt: timer: 33 microsec (min=6 max=401 rms**2=14)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=4)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=1)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=17)
interrupt: timer: 33 microsec (min=6 max=401 rms**2=0)
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 220 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 692 log messages lost.
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 544 log messages lost.
..................................
interrupt: WARN: [ 0] slotnums=1, tdcslot=16 -> use slotnums
logTask: 562 log messages lost.
interrupt: SYNC: ERROR: [ 0] slot=16 error_flag=1 - clear
logTask: 707 log messages lost.
interrupt: SYNC: ERROR: [ 3] slot=19 error_flag=1 - clear
logTask: 517 log messages lost.
interrupt: SYNC: ERROR: [ 4] slot=20 error_flag=1 - clear
logTask: 620 log messages lost.
interrupt: SYNC: ERROR: [ 5] slot=21 error_flag=1 - clear
logTask: 559 log messages lost.
interrupt: SYNC: scan_flag=0x00390000
logTask: 512 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 671 log messages lost.
......................................
logTask: 563 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
logTask: 385 log messages lost.
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
wait: coda request in progress
codaExecute reached, message >end<, len=3
codaExecute: 'end' transition
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
.......................................
wait: coda request in progress
wait: coda request in progress
interrupt: WARN: [ 0] slotnums=0, tdcslot=16 -> use slotnums
interrupt: TRIGGER ERROR: no pool buffer available
wait: coda request in progress
wait: coda request in progress
.....................................

Note: sc2 never warn about wrong slot number before that moment.

FIX: it seems there is a bug in 1190/1290-related rols and library: NBOARDS set to 21 and several arrays allocated with that length, but actual board maximum number is 21; NBOARDS was set to 22 everywhere, will test next time


Sergey B. 31-mar-2008: CC scans CODA !!! messages from EB

wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59063
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60187
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60295
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60418
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=60823
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=32961
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33122
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33795
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34529
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35811
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36134
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41569
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53913
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=57701
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33357
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=36136
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39509
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=39900
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45174
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=45909
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=34521
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=49526
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=53555
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55694
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=56833
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=58251
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=59023
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=33687
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=35731
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=51581
wait: coda request in progress
wait: coda request in progress
Error(old rc): nRead=18, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=41371
wait: coda request in progress
Error(old rc): nRead=6, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=43198
wait: coda request in progress
Error(old rc): nRead=10, must be 1032
CODAtcpServer: start work thread
befor: socket=5 address>129.57.71.38< port=55260
wait: coda request in progress
Error(old rc): nRead=0, must be 1032
clasprod::EB1> ^C

Similar messages were coming from ER.

ET reported following:

TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure  

and another ET:

.......
et_start: pthread_create(0x0000000d,...) done
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
TCP server got a connection so spawn thread
et ERROR: et_client_thread: read failure
et INFO: et_sys_heartmonitor, kill bad process (2,3063)
et INFO: et_sys_heartmonitor, cleanup process 2
et INFO: set_fix_nprocs, change # of ET processes from 2 to 2
et INFO: set_fix_natts, station GRAND_CENTRAL has 0 attachments
et INFO: set_fix_natts, station LEVEL3 has 1 attachments
et INFO: set_fix_natts, station ET2ER has 1 attachments
et INFO: set_fix_natts, # total attachments 2 -> 2
et INFO: set_fix_natts, proc 0 has 1 attachments
et INFO: set_fix_natts, proc 1 has 1 attachments 

Also noticed that CODAs (production and test setup) do not like each other: reported following during 'configure':

 Query test_ts2 table failed: Error reading table'test_ts2' definition
        ec3                                        ec3

If one runcontrol is restarted, then it works, but another one complains !!!

Sergey B. 31-mar-2008

'run_log_comment.tcl' and 'rlComment' cannot recognize xml comments when extraction level1 trigger file name from <l1trig> tag, it seems using first line after <l1trig> tag without looking into "" comment sign.

Sergey B. 10-mar-2008

seems found error in run control: Xui/src.s/rcMenuWindow.cc parameters XmNpaneMinimum and XmNpaneMaximum were both set to 480, as result run control gui area above log messages window was not big enough; set to 100 and 900 respectively, will ask Jie

23-jan-2008

ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:

(dbx) where
  [1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 
  [2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d 
  [3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a 
  [4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 
  [5] 0xfea60020(), at 0xfea60020 
(dbx)

14-nov-2007

first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following

ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00680BD9 -> resyncronize !!!

end run failed. Reboot sc2. During end transition ec2 froze with message:

interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181  0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled

Reboot ec2. Started new run 55463, everything looks normal.

Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen

Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65

data taking continues, but runcontrol printed message:

WARN   : ec2 has not reported status for 1516 seconds
ERROR  : ec2 is in state disconnected should be active

ec2pmc1 looked fine; end run failed, need to reboot ec2