GE XVB601: Difference between revisions

From CLONWiki
Jump to navigation Jump to search
Boiarino (talk | contribs)
 
(133 intermediate revisions by 11 users not shown)
Line 1: Line 1:
Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]
Company site: [http://www.ge-ip.com/products/family/embedded-systems GE Intelligent Platforms]
== Module manufacturer information ==


Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]
Board page on company site: [http://www.ge-ip.com/products/3448 XVB601]


== Customization for the CLON cluster ==
Documentation:
[https://clonwiki0.jlab.org/wiki/clondocs/Docs/ge_xvb601_ds_gfa1299.pdf Datasheet]
[https://clonwiki0.jlab.org/wiki/clondocs/Docs/ge_xvb601_hr_500-930007815-000_E.pdf Hardware Reference]
[https://clonwiki0.jlab.org/wiki/clondocs/Docs/ge_522-9300800627-000_A.book.pdf Transition Module Installation Guide]
[https://clonwiki0.jlab.org/wiki/clondocs/Docs/ge_Configuration_Guide_XVB601.pdf Product Configurations]
[https://clonwiki0.jlab.org/wiki/clondocs/Docs/ge_Ruggedization_Levels_GFA-926a.pdf Ruggedization]
 
== BIOS settings ==
 
Bryan Moffit's instractions for [https://coda.jlab.org/drupal/content/concurrent-technologies-vx915 Concurrent Boards]
 
== DiagGuiServer ==
 
Used to start by cron job:
 
@reboot  /bin/csh -c "(source /home/clasioc/.cshrc; diagguiserver_init_cron 1) > ~/.crontab_diagguiserver_init "
0-59/1 * * * *  /bin/csh -c "(source /home/clasioc/.cshrc; diagguiserver_cron 1) > ~/.crontab_diagguiserver "
 
Now part of startup script ''/etc/rc.local'':
 
rm -f /var/log/DiagGuiServer.log
touch /var/log/DiagGuiServer.log
chmod 777 /var/log/DiagGuiServer.log
su clasrun -c "DiagGuiServer init >> /var/log/DiagGuiServer.log &"
 
 
== yum update ==
 
https://coda.jlab.org/drupal/content/maintaining-your-diskless-setup
 
old: https://codaold.jlab.org/wiki/index.php/Maintaining_your_Diskless_Setup
 
For example ssh root@clon00 and do following:
 
mount -o bind /proc /diskless/Centos5/root/proc
mount -o bind /dev /diskless/Centos5/root/dev
mount -o bind /dev/pts /diskless/Centos5/root/dev/pts
mount -o bind /sys /diskless/Centos5/root/sys
chroot /diskless/Centos5/root
 
After that ''yum'' can be used. Sometimes it is needed to cleanup database(s):


* fix ''/etc/nsswitch.conf''
rm /var/lib/rpm/__db.*
 
== List of boards ==
 
* [01] 00:20:38:04:23:5E (pcal0)
 
* [02] 00:20:38:04:23:60 (svt2)
 
* [03] 00:20:38:04:29:2E (dcrb1)
 
* [04] 00:20:38:04:23:64 (ftof1)
 
* [05] Model: 14110110 SN: 63089370 MAC (eth1): 00:20:ce:f6:04:0e MAC (eth2): 00:20:ce:f6:04:0f (adcecal1)
 
* [06] Model: 14110110 SN: 63089318 MAC (eth1): 00:20:ce:f6:03:fe MAC (eth2): 00:20:ce:f6:03:ff (ftof0)
 
* [07] Model: 14110110 SN: 63089363 MAC (eth1): 00:20:ce:f6:03:fa MAC (eth2): 00:20:ce:f6:03:fb (ltcc0)
 
* Model: 14110110 SN: 63089332 MAC (eth1): 00:20:ce:f6:04:0a MAC (eth2): 00:20:ce:f6:04:0b (given to Ben 9-apr-2013 to be sent to Saclay)
 
== Initial settings ==
 
Install new board into VME crate. Connect ethernet to the upper LAN port, rs232 9600N1 serial line (optional), VGA monitor and USB keyboard. Turn on power, setup page should appear (for future reboots hold '''DEL''' key to enter setup). Goto '''Advanced->Onboard Device''', set memory size to '''512MB''' and enable '''Onboard Gigabit LAN 1'''. Goto '''Advanced->Serial Port Console Redirection''', enable '''COM1 Console Redirection''' and in '''Console Redirection Settings''' set '''Bits per second=9600'''.  Goto '''Chipset->South Bridge Configuration->SATA Configuration''' and set '''SATA Mode Selection''' to '''AHCI'''. Do '''Save & Exit'''.
 
Sergey Boyarinov only: open web browser from '''clonweb''' and goto '''jnet.jlab.org'''. Login using CUE login and password. Click on '''Machine Registration''' button. Click '''Add Device'''. Fill in a form, for example:
Connection Type: wired
MAC address: 00:20:38:04:23:64
VLAN: VLAN 568 : Hall B Data Acquisition <choose from the list>
Hostname: tage2.jlab.org
IP address: 129.57.68.86 <unselect 'Automatically Assign Statis IP>
JLAB Username: boiarino
Has PII: no
Machine Type: other
Property Tag: F
Click '''Submit'''. Following message must shows up:
The device (00:20:38:04:23:64) has been successfully added to the registration table.
-----------------
The device (00:20:38:04:23:64) has been successfully added to the assignment table.
-----------------
(DHCP) This entry will be added to DHCP the next time the server restarts (about 10 seconds).
-----------------
hallb-rtr.jlab.org ethernet8/9 is not an access port, blip failed...
 
Add IP name to the [[clonfs1]] export list, if not done yet.
 
Recycle power. Messages on monitor must show successful DHCP connection. Go to the following section to setup TFTP etc.
 
== Software structure description ==
 
* following is installed in ''/tftpboot'' area on clon10 server:
 
linux-install/Centos5PAE_new/initrd.img
                            /initrd.img.backup
                            /vmlinuz
              /msgs/boot.msg
                  /expert.msg
                  /general.msg
                  /param.msg
                  /rescue.msg
                  /snake.msg
              /pxelinux.0
              /pxelinux.cfg/81394458 -> croctest1
                          /81394459 -> croctest2
                          /croctest1
                          /croctest2
                          /default
                          /pxeos.xml
         
Every client must has a file in ''/pxelinux.cfg/'' area with the name containing IP address in HEX format, for example croctest1's IP is 129.57.68.88, so file name must be 81394458. It can be a symbolic link. The contents of the file for croctest1:
 
default Centos5PAE_new
label Centos5PAE_new
    kernel Centos5PAE_new/vmlinuz
    append  initrd=Centos5PAE_new/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=129.57.167.16:/vol/diskless/Centos5 ramdisk_size=31809 ETHERNET=eth0 SNAPSHOT=croctest1.jlab.org NISDOMAIN=CCCHP acpi=force vga=0x305 vmalloc=256MB console=ttyS0,9600


* fix ''/etc/yp.conf''
All settings in that file are more or less generic. To customize it for another client only ''SNAPSHOT=croctest1.jlab.org'' must be changed. Statement ''NFSROOT=129.57.167.16:/vol/diskless/Centos5'' defines the fileserver IP and directory where diskless Linux is located, in our case it is ''clonfs1''. Statement ''console=ttyS0,9600'' force Centos5 to send boot messages to the com port so they can be monitored using ''tsconnect''.


* fix ''/etc/sysconfig/network''
* diskless Linux is installed on ''clonfs1'' fileserver in ''/vol/diskless/Centos5'' volume. That volume is mounted on the clonpc3 RHEL machine as well, and all Centos5 installation was performed on clonpc3, as it described in EXPERT PART section below. Directory ''/diskless/Centos5/root'' will be mounted as ''/''  level on all diskless clients, so that part is generic for all clients. Custom areas in ''/diskless/Centos5/snapshot'' will be client-specific, for every new client new directory with appropriate name must be creates there using templete tar ball, for example:
tar xvf templete.jlab.org.tar
mv templete.jlab.org croctest1.jlab.org
* File ''files'' contains the list of all files/directories to be used from snapshot area rather then from root area, that file was created during Centos5 installation. File ''files.custom'' can be used to specify additional files/directories, not in use right now.


* ''/etc/init.d/ypbind start''
== Customizing kernel installed in ''/diskless/Centos5/root'' ==


* ''chkconfig ypbind on''
After expert installation is done, following changes must be applied. They are similar to the RHEL installation on clon cluster PCs and servers:


* ''/usr/bin/system-config-network''


* set RHEL-style [[automounts]]
* fix ''/etc/passwd'', ''/etc/shadow'', ''/etc/group''


* ''/etc/init.d/autofs restart''
'''NOTE''': to add new used you may use command ''useradd'', for example
  useradd -u 6246 -g nagios -d /home/nagios -c "Nagios" -s /bin/tcsh nagios
    (add flag '-m' if want to force home directory creation).


== Network-based booting ==
* fix ''/etc/nsswitch.conf''


The page is geared towards the setup of Diskless nodes for Intel-based VME Controllers, but may be useful for similar other applications.  Much of what is here was derived from the [http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/ch-diskless.html Redhat Enterprise Linux 3 Documentation].
* fix ''/etc/yp.conf'' (add lines ''domain CCCHP server clon00.jlab.org'' and ''ypserver  clon10.jlab.org'')


This examples assumes:
* fix ''/etc/sysconfig/network'' (add line ''NISDOMAIN=CCCHP'')
* The DHCP server (phecda: 129.57.29.104) has been installed and works.  Here we're using the dhcp 3.0.1 package distributed by RHEL4.
* The TFTP and NFS server (mizar: 129.57.29.100) is running RHEL5.5 x86_64.
* The Diskless Client (dafarm44: 129.57.29.54) will run Centos 5.5 x86_64.


There's nothing stopping you from having the DHCP server on the same machine as that servering TFTP and NFS.  You can obviously have more than one diskless client.
* ''/etc/init.d/ypbind start''


Once the first four steps have been completed, new clients can be configured individually with step 5.
* ''chkconfig ypbind on''


= Client Setup of OS in VirtualBox =
* set RHEL-style [[automounts]]
Here, we've downloaded the latest and greatest Centos 5.5 x86_64 ISO and have set up a VirtualBox Client to use it to boot.  We will not attempt to describe the VirtualBox installation or Setup of the ISO to boot as the VirtualBox Guest.


The Centos 5.5 installation proceeds as normal installations go, but we opt to do a minimal installation (excluding KDE and/or GNOME components) since we're running on a VME controller.  Other packages will be installed after first boot.
* ''/etc/init.d/autofs restart''


=== First Boot ===
* fix symbolic links to /apps, /home, /usr/local etc
At first boot (after installation), a UI will appear and provide you with some options.:
* Disable firewall and SELinux.  SELinux setup can be double checked in the file:
/etc/selinux/config
* Disable sendmail, and other services not required for a VME controller


=== Update packages and transfer the OS to Server ===
To install updates, ssh to clonpc3 as ''root'' and execute following commands:
Login as root
  mount -o bind /misc /diskless/Centos5/root
* Update all installed packages:
mount -o bind /misc/apps /diskless/Centos5/root/apps
  yum update
  mount -o bind /misc/local /diskless/Centos5/root/local
* Install the following packages, with:
mount -o bind /misc/home /diskless/Centos5/root/home
  yum install <i>package</i>
mount -o bind /misc/work /diskless/Centos5/root/work
:: busybox-anaconda
mount -o bind /misc/clas12 /diskless/Centos5/root/clas12
:: gcc
mount -o bind /misc/downloads /diskless/Centos5/root/downloads
:: kernel-devel
:: gdb
:: valgrind
:: cvs
:: subversion
:: compat-libstdc++-33
:: redhat-rpm-config


* Copy the Client OS to the tftp/NFS Server.
Now you can start chroot environment: ''chroot /diskless/Centos5/root''.
** On The Server:
mkdir -p /diskless/x86_64/Centos5.5/root
: In The VirtualBox Guest:
rsync -Pav -e ssh --exclude='/proc/*' --exclude='/sys/*' / mizar:/diskless/x86_64/Centos5.5/root/


You are now done with the Centos5.5 Guest installed in the VirtualBox. It can be closed down now.
Fix yum proxy: add line ''proxy=http://jprox.jlab.org:8082'' to the file ''/etc/yum.conf'', and fix file ''/etc/yum.repos.d/CentOS-Base.repo'' by commenting out all ''mirrorlist=http:...'' lines, uncommenting all ''baseurl=http:...'' lines, current settings (Sep 2018) is following:


= tftp and NFS Server Setup =
[base]
Follow these steps on the machine serving out tftp and NFS:
baseurl=http://vault.centos.org/5.11/os/$basearch/


===Install <code>system-config-netboot</code> ===
[updates]
(This should automatically install the tftp server)
baseurl=http://vault.centos.org/5.11/updates/$basearch/
yum install system-config-netboot


=== Enable the tftp server to start at boot ===
* Change line (<code>/etc/xinetd.d/tftp</code>):
disable                = yes
: To
disable                = no


* Restart <code>xinetd</code>
service xinetd restart


=== Configure NFS ===
After that you can use ''yum'', for example:
* Add these entries to <code>/etc/exports</code>
/diskless/x86_64/Centos5.5/root/    129.57.29.0/255.255.255.0(ro,sync,no_root_squash)
/diskless/x86_64/Centos5.5/snapshot/ 129.57.29.0/255.255.255.0(rw,sync,no_root_squash)
: Where the <code>129.57.29.0/255.255.255.0</code> only allows the 29 subnet to access these mounts.
           
* Reload the NFS configuration or start NFS, if it's not started
service nfs reload


* Make sure NFS is started at boot
yum install gcc-c++
  /sbin/chkconfig --level 345 nfs on
yum install mysql-devel
  yum install compat-gcc-34-g77
yum install libXpm-devel
yum install libXtst
yum install openmotif-devel
    yum install java


=== Enable remote syslog (optional) ===
If need local mysql server, do following (see [[MySQL Installation]] as well):
* Change syslog argument in <code>/etc/sysconfig/syslog</code> from:
SYSLOGD_OPTIONS="-m 0"
:: To
SYSLOGD_OPTIONS="-m 0 -r"


* Restart syslogd
  yum install mysql-server
  service syslog restart


= DHCP Server Setup =
In case of yum, rpm etc problems usually you have to rebuild dictionary, google the error message. Following may help:
On the machine hosting the DHCP server.
mv /var/lib/rpm/__db* ~/
rpm --rebuilddb
and check using command:
rpm -qa


* Add an entry in <code>/etc/dhcpd.conf</code>:
<pre>
host dafarm44
              {
              filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/
              next-server 129.57.29.38;            # TFTP server
              hardware ethernet 00:20:38:03:D8:52; # Client MAC
              fixed-address 129.57.29.54;          # Client IP
              }
</pre>


* Restart dhcpd
'''IMPORTANT''': to prevent controllers to hung, do following change in files ''/etc/sysconfig/network-scripts/ifcfg-eth0'', ''...eth1'' etc: instead of
service dhcpd restart


= Configure Client OS =
  ONBOOT=yes


Here, we configure the OS that each and every Diskless Client will enjoy.
it must be


=== Setup of Diskless Kernel ===
  ONBOOT=no
Run system-config-netboot (on TFTP server):
system-config-netboot


* In "First Time Druid" window.  Click on "Diskless"
== CompactFlash ==
# Click "Forward" on first page
# Enter Name and Description.
# Enter IP address of TFTP server and the "root" directory exported via NFS
# Select a kernel
# Click on Apply.
: This will take a minute or two to compile a boot kernel for the diskless client.


=== Setup of chroot environment ===
Local MySQL server startup:
There are lots of specifics here for setup on the JLab CUE and Data Acquisition Network (daqfs).
This enviroment will give you the capability to do system updates/upgrades for all diskless clients at the same time.
chroot /diskless/x86_64/Centos5.5/root
mount /proc /proc -t proc
mkdir /u
mkdir /group
mkdir /daqfs
mv /home /home.old
ln -s /u/apps /apps
ln -s /u/site /site
ln -s /daqfs/home /home


=== Setup of diskless system configurations ===
ssh clondb1 as root
''' Which NFS directories to mount '''
mysqldump -u root -p daq_daq > daq_daq.sql
* Add entry to <code>/etc/fstab</code>
  mysqldump -u root -p daq_classvt > daq_classvt.sql
  phecda:/raid    /daqfs  nfs    rw,soft,intr 0 0


* Mount daqfs
ssh as root
  mount /daqfs
/sbin/chkconfig mysqld on
  /etc/init.d/mysqld start
scp root@clondb1:/root/daq_classvt.sql .
scp root@clondb1:/root/daq_daq.sql .


''' Which users/groups to allow access '''
mysql
* Add entries to /etc/passwd
mysql> CREATE USER 'root'@'clonweb0.jlab.org' IDENTIFIED BY '*********';  use mysql clon root password
+@da::::::
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'clonweb0.jlab.org' WITH GRANT OPTION;
+@ccc::::::
+::::::/bin/false


''' A proxy for yum ''' (may be optional)
mysql> CREATE USER 'clasrun'@'localhost';
* Add entry to /etc/yum.conf
mysql> GRANT ALL PRIVILEGES ON *.* TO 'clasrun'@'localhost';
proxy=http://jprox:8080


''' domain servers for ypbind '''
mysql> flush privileges;
* Add entries to /etc/yp.conf
mysql> create database daq_daq;
  domain CCCHP server nis2.jlab.org
  mysql> create database daq_classvt;
  domain CCCHP server nis3.jlab.org
  mysql> exit


* Make sure ypbind started at boot
mysql daq_daq < daq_daq.sql
  /sbin/chkconfig --level 345 nfs on
  mysql daq_classvt < daq_classvt.sql


''' NTP setup ''' (optional... ntp must be installed)
* Add entries to <code>/etc/ntp.conf</code>
server 127.0.0.1 # local clock
fudge 127.0.0.1 stratum 10
server ntp1.jlab.org
server ntp2.jlab.org
driftfile /etc/ntp/drift
broadcastdelay 0.008
authenticate no


''' Some additional JLab CUE directories to mount '''
* Add entries to <code>/etc/auto.master</code>
/group          /etc/auto.group --timeout=0
/u              auto.u.bb      --DOSNAME=Linux-RHEL5


''' Additional files, not included in the standard diskless filesystem '''
Original CentOS installation on CompactFlash was done by Bryan Moffit. To clone flash card, insert it to the reader on macbook pro and do following:
* Add a new file <code>/diskless/x86_64/Centos5.5/snapshot/files.custom</code>.  Entry:
/tmp/
/etc/minirc.dfl


= Configure a NEW diskless client =
dd if=/dev/rdisk1 conv=sync,noerror | gzip -c > disk1.img.gz
This operation must be peformed to add a new Diskless Client.


=== Configuration on the tftp/NFS server ===
Replace it with empty one and do following:
* In <code>system-config-netboot</code> "Network Installation and Diskless Environment" window:
# Click on "New"
# Enter hostname
# Select OS
# Enter "CCCHP" in Enable NISDOMAIN box.
# Enter options in "Edit Extra Kernel Boot Options" <br /> <code>vga=0x305 acpi=force</code>
# Enter hostname as snapshot name.  Make sure "generate" is check-marked
# Click "OK"


=== Configuration on the Diskless Client ===
gunzip -c disk1.img.gz | dd of=/dev/rdisk1
* PXEboot must be enabled and should be the primary boot option. This is set in the BIOS.


= Other useful packages to install =
Repeat last step to all empty flash cards you want to program.
Later on... I installed some other useful packages using the chroot environment. Here they are:


* <code>xorg-x11-xauth</code> - Required to run X-applications remotely
== EXPERT PART: setting everything from scratch using clonpc3 ==
* <code>ntp</code> - Synchronize system clock with a central server
: To Execute <code>ntpdate</code> on server startup:
:* Add <code><b>-x</b></code> to <code>OPTIONS</code> in
/etc/sysconfig/ntpd
:* Make sure it starts on boot
/sbin/chkconfig --level 345 nfs on

Latest revision as of 09:55, 18 July 2019

Company site: GE Intelligent Platforms

Module manufacturer information

Board page on company site: XVB601

Documentation: Datasheet Hardware Reference Transition Module Installation Guide Product Configurations Ruggedization

BIOS settings

Bryan Moffit's instractions for Concurrent Boards

DiagGuiServer

Used to start by cron job:

@reboot  /bin/csh -c "(source /home/clasioc/.cshrc; diagguiserver_init_cron 1) > ~/.crontab_diagguiserver_init "
0-59/1 * * * *  /bin/csh -c "(source /home/clasioc/.cshrc; diagguiserver_cron 1) > ~/.crontab_diagguiserver "

Now part of startup script /etc/rc.local:

rm -f /var/log/DiagGuiServer.log
touch /var/log/DiagGuiServer.log
chmod 777 /var/log/DiagGuiServer.log
su clasrun -c "DiagGuiServer init >> /var/log/DiagGuiServer.log &"


yum update

https://coda.jlab.org/drupal/content/maintaining-your-diskless-setup

old: https://codaold.jlab.org/wiki/index.php/Maintaining_your_Diskless_Setup

For example ssh root@clon00 and do following:

mount -o bind /proc /diskless/Centos5/root/proc
mount -o bind /dev /diskless/Centos5/root/dev
mount -o bind /dev/pts /diskless/Centos5/root/dev/pts
mount -o bind /sys /diskless/Centos5/root/sys
chroot /diskless/Centos5/root

After that yum can be used. Sometimes it is needed to cleanup database(s):

rm /var/lib/rpm/__db.*

List of boards

  • [01] 00:20:38:04:23:5E (pcal0)
  • [02] 00:20:38:04:23:60 (svt2)
  • [03] 00:20:38:04:29:2E (dcrb1)
  • [04] 00:20:38:04:23:64 (ftof1)
  • [05] Model: 14110110 SN: 63089370 MAC (eth1): 00:20:ce:f6:04:0e MAC (eth2): 00:20:ce:f6:04:0f (adcecal1)
  • [06] Model: 14110110 SN: 63089318 MAC (eth1): 00:20:ce:f6:03:fe MAC (eth2): 00:20:ce:f6:03:ff (ftof0)
  • [07] Model: 14110110 SN: 63089363 MAC (eth1): 00:20:ce:f6:03:fa MAC (eth2): 00:20:ce:f6:03:fb (ltcc0)
  • Model: 14110110 SN: 63089332 MAC (eth1): 00:20:ce:f6:04:0a MAC (eth2): 00:20:ce:f6:04:0b (given to Ben 9-apr-2013 to be sent to Saclay)

Initial settings

Install new board into VME crate. Connect ethernet to the upper LAN port, rs232 9600N1 serial line (optional), VGA monitor and USB keyboard. Turn on power, setup page should appear (for future reboots hold DEL key to enter setup). Goto Advanced->Onboard Device, set memory size to 512MB and enable Onboard Gigabit LAN 1. Goto Advanced->Serial Port Console Redirection, enable COM1 Console Redirection and in Console Redirection Settings set Bits per second=9600. Goto Chipset->South Bridge Configuration->SATA Configuration and set SATA Mode Selection to AHCI. Do Save & Exit.

Sergey Boyarinov only: open web browser from clonweb and goto jnet.jlab.org. Login using CUE login and password. Click on Machine Registration button. Click Add Device. Fill in a form, for example:

Connection Type: wired
MAC address: 00:20:38:04:23:64
VLAN: VLAN 568 : Hall B Data Acquisition <choose from the list>
Hostname: tage2.jlab.org
IP address: 129.57.68.86 <unselect 'Automatically Assign Statis IP>
JLAB Username: boiarino
Has PII: no
Machine Type: other
Property Tag: F

Click Submit. Following message must shows up:

The device (00:20:38:04:23:64) has been successfully added to the registration table.
-----------------
The device (00:20:38:04:23:64) has been successfully added to the assignment table.
-----------------
(DHCP) This entry will be added to DHCP the next time the server restarts (about 10 seconds).
-----------------
hallb-rtr.jlab.org ethernet8/9 is not an access port, blip failed...

Add IP name to the clonfs1 export list, if not done yet.

Recycle power. Messages on monitor must show successful DHCP connection. Go to the following section to setup TFTP etc.

Software structure description

  • following is installed in /tftpboot area on clon10 server:
linux-install/Centos5PAE_new/initrd.img
                            /initrd.img.backup
                            /vmlinuz
             /msgs/boot.msg
                  /expert.msg
                  /general.msg
                  /param.msg
                  /rescue.msg
                  /snake.msg
             /pxelinux.0
             /pxelinux.cfg/81394458 -> croctest1
                          /81394459 -> croctest2
                          /croctest1
                          /croctest2
                          /default
                          /pxeos.xml
         

Every client must has a file in /pxelinux.cfg/ area with the name containing IP address in HEX format, for example croctest1's IP is 129.57.68.88, so file name must be 81394458. It can be a symbolic link. The contents of the file for croctest1:

default Centos5PAE_new
label Centos5PAE_new
    kernel Centos5PAE_new/vmlinuz
    append  initrd=Centos5PAE_new/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=129.57.167.16:/vol/diskless/Centos5 ramdisk_size=31809 ETHERNET=eth0 SNAPSHOT=croctest1.jlab.org NISDOMAIN=CCCHP acpi=force vga=0x305 vmalloc=256MB console=ttyS0,9600

All settings in that file are more or less generic. To customize it for another client only SNAPSHOT=croctest1.jlab.org must be changed. Statement NFSROOT=129.57.167.16:/vol/diskless/Centos5 defines the fileserver IP and directory where diskless Linux is located, in our case it is clonfs1. Statement console=ttyS0,9600 force Centos5 to send boot messages to the com port so they can be monitored using tsconnect.

  • diskless Linux is installed on clonfs1 fileserver in /vol/diskless/Centos5 volume. That volume is mounted on the clonpc3 RHEL machine as well, and all Centos5 installation was performed on clonpc3, as it described in EXPERT PART section below. Directory /diskless/Centos5/root will be mounted as / level on all diskless clients, so that part is generic for all clients. Custom areas in /diskless/Centos5/snapshot will be client-specific, for every new client new directory with appropriate name must be creates there using templete tar ball, for example:
tar xvf templete.jlab.org.tar
mv templete.jlab.org croctest1.jlab.org
  • File files contains the list of all files/directories to be used from snapshot area rather then from root area, that file was created during Centos5 installation. File files.custom can be used to specify additional files/directories, not in use right now.

Customizing kernel installed in /diskless/Centos5/root

After expert installation is done, following changes must be applied. They are similar to the RHEL installation on clon cluster PCs and servers:

  • /usr/bin/system-config-network
  • fix /etc/passwd, /etc/shadow, /etc/group

NOTE: to add new used you may use command useradd, for example

 useradd -u 6246 -g nagios -d /home/nagios -c "Nagios" -s /bin/tcsh nagios
   (add flag '-m' if want to force home directory creation).
  • fix /etc/nsswitch.conf
  • fix /etc/yp.conf (add lines domain CCCHP server clon00.jlab.org and ypserver clon10.jlab.org)
  • fix /etc/sysconfig/network (add line NISDOMAIN=CCCHP)
  • /etc/init.d/ypbind start
  • chkconfig ypbind on
  • /etc/init.d/autofs restart
  • fix symbolic links to /apps, /home, /usr/local etc

To install updates, ssh to clonpc3 as root and execute following commands:

mount -o bind /misc /diskless/Centos5/root
mount -o bind /misc/apps /diskless/Centos5/root/apps
mount -o bind /misc/local /diskless/Centos5/root/local
mount -o bind /misc/home /diskless/Centos5/root/home
mount -o bind /misc/work /diskless/Centos5/root/work
mount -o bind /misc/clas12 /diskless/Centos5/root/clas12
mount -o bind /misc/downloads /diskless/Centos5/root/downloads

Now you can start chroot environment: chroot /diskless/Centos5/root.

Fix yum proxy: add line proxy=http://jprox.jlab.org:8082 to the file /etc/yum.conf, and fix file /etc/yum.repos.d/CentOS-Base.repo by commenting out all mirrorlist=http:... lines, uncommenting all baseurl=http:... lines, current settings (Sep 2018) is following:

[base]
baseurl=http://vault.centos.org/5.11/os/$basearch/
[updates]
baseurl=http://vault.centos.org/5.11/updates/$basearch/


After that you can use yum, for example:

yum install gcc-c++
yum install mysql-devel
yum install compat-gcc-34-g77
yum install libXpm-devel
yum install libXtst
yum install openmotif-devel
    yum install java

If need local mysql server, do following (see MySQL Installation as well):

yum install mysql-server

In case of yum, rpm etc problems usually you have to rebuild dictionary, google the error message. Following may help:

mv /var/lib/rpm/__db* ~/
rpm --rebuilddb

and check using command:

rpm -qa


IMPORTANT: to prevent controllers to hung, do following change in files /etc/sysconfig/network-scripts/ifcfg-eth0, ...eth1 etc: instead of

  ONBOOT=yes

it must be

  ONBOOT=no

CompactFlash

Local MySQL server startup:

ssh clondb1 as root
mysqldump -u root -p daq_daq > daq_daq.sql
mysqldump -u root -p daq_classvt > daq_classvt.sql
ssh as root
/sbin/chkconfig mysqld on
/etc/init.d/mysqld start
scp root@clondb1:/root/daq_classvt.sql .
scp root@clondb1:/root/daq_daq.sql .
mysql
mysql> CREATE USER 'root'@'clonweb0.jlab.org' IDENTIFIED BY '*********';   use mysql clon root password
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'clonweb0.jlab.org' WITH GRANT OPTION;
mysql> CREATE USER 'clasrun'@'localhost';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'clasrun'@'localhost';
mysql> flush privileges;
mysql> create database daq_daq;
mysql> create database daq_classvt;
mysql> exit
mysql daq_daq < daq_daq.sql
mysql daq_classvt < daq_classvt.sql


Original CentOS installation on CompactFlash was done by Bryan Moffit. To clone flash card, insert it to the reader on macbook pro and do following:

dd if=/dev/rdisk1 conv=sync,noerror | gzip -c > disk1.img.gz

Replace it with empty one and do following:

gunzip -c disk1.img.gz | dd of=/dev/rdisk1

Repeat last step to all empty flash cards you want to program.

EXPERT PART: setting everything from scratch using clonpc3