ERROR Book

• Sergey B. 2-nov-2007: during the run ec2 started to print on tsconnect screen:

Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65
Unknown error errno=65

data taking continues, but runcontrol printed message:

WARN   : ec2 has not reported status for 1516 seconds
ERROR  : ec2 is in state disconnected should be active

ec2pmc1 looked fine; end run failed, need to reboot ec2

• Sergey B. 3-nov-2007: after about 26Mevents during the run sc2pmc1 started to print following:

ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000040 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000080 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x03C80ADD -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x8A90097F -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00000060 -> resyncronize !!!
ROC # 22 Event # 0 :  Bad Block Read signature 0x00680BD9 -> resyncronize !!!

end run failed. Reboot sc2. During end transition ec2 froze with message:

interrupt: timer: 32 microsec (min=19 max=86 rms**2=18)
0x1a05fdf0 (twork0005): sfiUserEnd: INFO: Last Event 26723663, status=0 (0x1ca648c8 0x1ca648c0)
0x1a05fdf0 (twork0005): data: 0x00000003 0x0007014f 0x00120000 0x00000000 0xc8009181 0xc0001181
0x1a05fdf0 (twork0005): jw1 : 0x00000000 0x0197c54f 0x00000003 0x0007014f 0x00120000 0x00000000
0x1a05fdf0 (twork0005): Last DMA status = 0x200000b count=11 blen=11
0x1a05fdf0 (twork0006): sfiUserEnd: ERROR: Last Transfer Event NUMBER 26723663, status = 0x1a000 (0x90001181 0x88001181 0x80009181  0x78001181)
0x1a05fdf0 (twork0006): SFI_SEQ_ERR: Sequencer not Enabled

Reboot ec2. Started new run 55463, everything looks normal.

• 14-nov-2007: first week of running G9A: crashes observed in ec1 (twice), tage3, scaler1, clastrig2, EB; no feather details were obtained so far

• 23-jan-2008: ER3 crashed several times during last 3 weeks, mostly (only ?) during 'End' transition; todays core file:

(dbx) where
  [1] 0xfe9e4c20(0x8068f9d), at 0xfe9e4c20 
  [2] codaExecute(0xce4fdbc0, 0xce4fdbc0, 0x1, 0x8068c39), at 0x8068f9d 
  [3] CODAtcpServerWorkTask(0x811fa00, 0x0, 0x0, 0xce4fdff8, 0xfea60020, 0xfe591400), at 0x8068d3a 
  [4] 0xfea5fd36(0xfe591400, 0x0, 0x0, ), at 0xfea5fd36 
  [5] 0xfea60020(), at 0xfea60020 
(dbx)