Linux Boot Server: Difference between revisions

From CLONWiki
Jump to navigation Jump to search
← Older edit
Boiarino (talk | contribs)
jpeople, Administrators
4,356 edits
No edit summary
Boiarino (talk | contribs)
jpeople, Administrators
4,356 edits
No edit summary
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''Setting of the Linux server to be used for PXE and EFI boot of the VME controllers'''
== '''Setting of the Linux server to be used for PXE and EFI boot of the VME controllers''' ==


Login to the server as root. Copy two files to the root directory and untar them:
Login to the server as root. Copy two files to the root directory and untar them:
Line 54: Line 54:
  sysctl -p
  sysctl -p


Configure [[NFS]]


Configure [[DHCP server]]
Configure [[DHCP server]]
Line 59: Line 60:
Configure [[DNS server]]
Configure [[DNS server]]


Configure [[Tftp]] and [[Tftpboot]]


Configure [[Iptables]]


 
Do not forget (is it for controllers ?):
 
 
 
 
 
 
'''IPTABLES'''
 
Do not forget:
  chattr +i /etc/resolv.conf
  chattr +i /etc/resolv.conf


Show tables:
iptables -vL -t filter
iptables -vL -t nat
iptables -vL -t mangle
iptables -vL -t raw
iptables -vL -t security
Only first two seems relevant. Clear them:
iptables -t filter -F
iptables -t nat -F
 
Set needed settings (we assumes loval network port is 'enp_bond', and uplink port is 'em1'):
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i enp_bond -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -o em1 -j MASQUERADE
iptables -A FORWARD -i em1 -o enp_bond -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i enp_bond -o em1 -j ACCEPT


Save rules and restart service:
  yum install ypbind (????)
  iptables-save > /etc/sysconfig/iptables
systemctl enable iptables
systemctl start iptables




'''Configuring snapshot area'''


The list of files and directories which suppose to be in snapshot area (which has RW permissions) should be set in config file ''/diskless/.../root/etc/statetab''. In particular, ''/et'' has to be added there, and also directory ''/diskless/.../root/et'' should be created.


'''Reboot server, make sure everything is running as expected.'''


'''old gefvme removal'''


yum remove kmod-gefvme gefvme-library


If failed on 'gefvme-library', do
rpm -e --noscripts gefvme-library


If file ''/etc/modules-load.d/gefvme.conf'' was not removed by above commands, remove it manually (if working from ''chroot ...'' then everything is done in ''root'' area, but ''gefvme.conf'' may survive in ''snapshot'' area, in that case remove it on controller and not in ''chroot'').


File ''/etc/modules-load.d/vme.conf'' with contents
cmem_rcc
jvme
vme_vivo
vme_ca91cx42
vme_tsi148
vme
may not be needed (if modules already in kernel ?).


'''new jvme (not sure if module installation is needed, maybe in kernel already ?) '''
== '''Configuring in chroot''' ==


'''On VME controller in BIOS, make sure VME memry size is set to 512M'''
Since root area in VME controllers will be read-only, you cannot ssh to controllers as root and change anything there, only snapshot area can be modified. To make changes in root area, in particular to configure and run ''yum'', ''chroot' command should be used. Login to the boot server as root and do following:
 
Project git link: https://code.jlab.org/fedaq/drivers/jvme/-/tree/release-3.0
 
On vme controller as ''boiarino'', copy ''jvme-release-3.0.tar'' to ''$CODA/src'' and untar it.
 
Create two environment scripts:
 
jvme_bash:
#!/bin/bash
export LINUXVME=${CODA}/src/jvme-release-3.0/linuxvme
export LINUXVME_INC=${LINUXVME}/include
export LINUXVME_LIB=${LINUXVME}/Linux_`uname -m`_vme/lib
export LINUXVME_BIN=${LINUXVME}/Linux_`uname -m`_vme/bin
export LD_LIBRARY_PATH=${LINUXVME_LIB}:${LD_LIBRARY_PATH}
export KERNELRELEASE=3.10.0-1062.9.1.el7.x86_64
 
jvme_tcsh:
#!/bin/tcsh
setenv LINUXVME ${CODA}/src/jvme-release-3.0/linuxvme
setenv LINUXVME_INC ${LINUXVME}/include
setenv LINUXVME_LIB ${LINUXVME}/Linux_`uname -m`_vme/lib
setenv LINUXVME_BIN ${LINUXVME}/Linux_`uname -m`_vme/bin
setenv LD_LIBRARY_PATH ${LINUXVME_LIB}:${LD_LIBRARY_PATH}
setenv KERNELRELEASE 3.10.0-1062.9.1.el7.x86_64
 
Run ''source jvme_tcsh''.
Go inside ''jvme-release-3.0''.
In two files ''CMakeLists.txt'' and ''src/CMakeLists.txt'', change
set(libpath Linux-${CMAKE_SYSTEM_PROCESSOR}/lib)
set(libpath Linux-${CMAKE_SYSTEM_PROCESSOR}/bin)
to
set(libpath Linux_${CMAKE_SYSTEM_PROCESSOR}_vme/lib)
set(libpath Linux_${CMAKE_SYSTEM_PROCESSOR}_vme/bin)
 
Type ''cmake -B build -S . -DCMAKE_INSTALL_PREFIX=$LINUXVME''
 
Fix Makefile in ''kernel_driver'' and three it's subdirectories, it must have following in the beginning:
KVERSION := $(KERNELRELEASE)
ifeq ($(origin KERNELRELEASE), undefined)
KVERSION := $(shell uname -r)
endif
 
In directory ''jvme-release-3.0'', type ''make'' and ''make install'',  
 
Do ''cd kernel_driver'' and ''make'' (do NOT do ''make install'').
 
On the server, do


mount -o bind /home /diskless/CentOS7/x86_64/root/home
mount -o bind /usr/local /diskless/CentOS7/x86_64/root/usr/local
  mount -o bind /usr/clas12 /diskless/CentOS7/x86_64/root/usr/clas12
  mount -o bind /usr/clas12 /diskless/CentOS7/x86_64/root/usr/clas12
mount -o bind /usr/local /diskless/CentOS7/x86_64/root/usr/local
mount -o bind /home /diskless/CentOS7/x86_64/root/home
  chroot /diskless/CentOS7/x86_64/root
  chroot /diskless/CentOS7/x86_64/root
cd /usr/clas12/release/2.0.0/coda/src
source jvme_bash
cd jvme-release-3.0/kernel_driver
make install


Still on server, add two files to ''/etc/udev/rules.d'' directory:
After that you will see all root directories as if you login to VME controller, and modifications can be made.


99-cmem.rules:
'''Configure yum'''
KERNEL=="cmem_rcc", MODE="0666"
 
99-vme.rules:
KERNEL=="bus/vme/ctl", MODE="0666"
KERNEL=="bus/vme/m_a16", MODE="0666"
KERNEL=="bus/vme/m_a24", MODE="0666"
KERNEL=="bus/vme/m_a32", MODE="0666"
KERNEL=="bus/vme/m_crcsr", MODE="0666"
KERNEL=="bus/vme/s_a32", MODE="0666"
KERNEL=="bus/vme/s_rsvd1", MODE="0666"
KERNEL=="bus/vme/s_rsvd2", MODE="0666"
KERNEL=="bus/vme/s_rsvd3", MODE="0666"
 
Reboot controller, check if everything is good.
 
'''NOTE''': if changing something in kernel module(s), it is not needed to reboot every time after make/make install is done in ''jvme-release-3.0/kernel_driver'' directory. Just run ''./load_driver.sh'' as root on controller, and all modules will be reloaded.
 
 
'''NOTE''': to add ''/et'' to snapshot area: on server, create /et directories in both root and snapshot areas, then add line '/et' to /etc/statetab file in root area, then reboot controller.
 
 
 
 
'''Enable systemd log persistency (remember all reboots, not only last one)'''
 
Run ''emacs /etc/systemd/journald.conf'', set
 
Storage=persistent
 
Do following:
 
mkdir /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald
 
'''yum'''
 
On server where vme is loading from (as root; ''/zzz'' will be needed for mysqltcl installation below):
 
mkdir /diskless/CentOS7/x86_64/root/zzz
#mount -o bind /usr/local/src /diskless/CentOS7/x86_64/root/zzz
mount -o bind /usr/local /diskless/CentOS7/x86_64/root/usr/local
mount -o bind /usr/clas12 /diskless/CentOS7/x86_64/root/usr/clas12
chroot /diskless/CentOS7/x86_64/root


Add ''multilib_policy=all'' to ''/etc/yum.conf''.
Add ''multilib_policy=all'' to ''/etc/yum.conf''.
Line 255: Line 123:




'''NOTE''': Login from console may not work because of file ''/etc/securetty'' permissions, it must be 644.


 
'''NOTE''': THere is a service PAM, it may prevent login from console if some required services did not start. To work around, comment out some lines in ''/etc/pam.d/system-auth-ac'' file:
 
 
 
 
 
 
'''Setting TFTP server (RHEL7)'''
 
yum install tftp tftp-server
 
Bryan:
 
yum install nfs-utils tftp-server syslinux-tftpboot syslinux
 
yum install dnsmasq
 
mkdir /tftpboot
 
grub2-mknetdir --net-directory=/tftpboot/efi
 
 
Create file ''/etc/systemd/system/tftp.service'':
 
[Unit]
Description=Tftp Server
Requires=tftp.socket
Documentation=man:in.tftpd
 
[Service]
#ExecStart=/usr/sbin/in.tftpd -s /var/lib/tftpboot
ExecStart=/usr/sbin/in.tftpd -s /tftpboot
StandardInput=socket
 
[Install]
Also=tftp.socket
 
 
Start tftp:
 
systemctl start tftp
#systemctl start tftp.socket ??
 
Create file ''/tftpboot/efi/boot/grub2/grub.cfg'' with following contents:
 
 
function load_video {
insmod efi_gop
insmod efi_uga
insmod video_bochs
insmod video_cirrus
insmod all_video
}
 
load_video
set gfxpayload=keep
insmod gzio
set timeout=2
menuentry 'Diskless CentOS7 x86_64, any network device'  --class redhat --class gnu-linux --class gnu --class os {
linuxefi linux-install/CentOS7-x86_64-Diskless/vmlinuz-3.10.0-1062.9.1.el7.x86_64 zram=1 ip=::::::dhcp root=nfs:192.168.10.1:/diskless/CentOS7-devel/x86_64/root ro vga=0x305 module_blacklist=ipmi_si,ipmi_msghandler,ipmi_devintf,w83977f_wdt
initrdefi linux-install/CentOS7-x86_64-Diskless/initramfs-jvme-3.10.0-1062.9.1.el7.x86_64.img
}
 
Edit file ''/etc/dhcp/dhcpd.conf'':
 
subnet 192.168.10.0 netmask 255.255.255.0 {
        option domain-name "jlab.org";
        option domain-name-servers 129.57.32.100, 129.57.32.101;
        option routers 192.168.10.1;
        use-host-decl-names true;
pool {
    range 192.168.10.2 192.168.10.20;
    deny dynamic bootp clients;
    allow unknown clients;
    }
}
 
set vendorclass = option vendor-class-identifier;
option pxe-system-type code 93 = unsigned integer 16;
set pxetype = option pxe-system-type;
 
# DISKLESS Clients in here
group
{
if substring(vendorclass, 0, 9)="PXEClient" {
  if pxetype=00:06 or pxetype=00:07 {
      filename  "efi/boot/grub2/x86_64-efi/core.efi";
  } else {
      filename "linux-install/pxelinux.0";
  }
}
  next-server 192.168.10.1;
host test1 {
hardware ethernet 00:20:38:03:10:34;
fixed-address 192.168.10.4;
}
host test4 {
hardware ethernet 00:20:38:10:14:f7;
fixed-address 192.168.10.5;
}
} # Diskless clients group
 
Start dhcp:
 
systemctl start dhcpd
 
 
Install nfs:
 
yum install nfs-utils
 
Configure file ''/etc/exports'':
 
/diskless 192.168.10.0/24(rw,no_root_squash,sync)
 
Start NFS server:
 
systemctl status nfs-server
 
Check that NFS is exporting. Command
 
showmount -e
 
have to show following:
 
Export list for clondaq15.jlab.org:
/diskless 192.168.10.0/24
 
 
To use local name server, install bind:
 
yum install bind
 
 
 
Login from console may not work because of file ''/etc/securetty'' permissions, it must be 644.
 
 
THere is a service PAM, it may prevent login from console if some required services did not start. To work around, comment out some lines in ''/etc/pam.d/system-auth-ac'' file:


  #%PAM-1.0
  #%PAM-1.0
Line 419: Line 149:
  session    [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
  session    [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
  #session    required      pam_unix.so
  #session    required      pam_unix.so
'''After Linux boot server is ready and operational, VME controllers can be booted. On the first boot, individual ''/diskless/.../snapshot/<controller name>'' directory will be created for each controller. For VME controller specific information see [[VME]]'''

Latest revision as of 10:38, 16 May 2025

Setting of the Linux server to be used for PXE and EFI boot of the VME controllers

Login to the server as root. Copy two files to the root directory and untar them: 

cd /
cp /usr/downloads/tftpboot.tar.gz_clondaq15 tftpboot.tar.gz
cp /usr/downloads/diskless.tar.gz_clondaq15 diskless.tar.gz
gunzip tftpboot.tar.gz
gunzip diskless.tar.gz
tar xvf tftpboot.tar
tar xvf diskless.tar
rm tftpboot.tar diskless.tar


Make sure correct snapshot location in root area /diskless/CentOS7/x86_64/root/etc/sysconfig/readonly-root: 

CLIENTSTATE=192.168.10.1:/diskless/CentOS7/x86_64/snapshot

In /diskless/CentOS7/x86_64/root area, /etc/ssh/sshd_config must have following setting (to allow remote ssh): 

#UsePAM yes

it will be propagated to the controller's snapshot area on the first boot when corresponding snapshot/<vme_controller_name> directory will be created. Not sure about other settings: 

HostbasedAuthentication yes
#HostbasedAuthentication no

IgnoreRhosts no
#IgnoreRhosts yes

PasswordAuthentication yes
#PasswordAuthentication yes
PasswordAuthentication yes

#UsePrivilegeSeparation sandbox		# Default for new installations.
UsePrivilegeSeparation sandbox		# Default for new installations.


To add or remove VME controllers, modify following files: 

/var/named/10.168.192.in-addr.arpa.db
/var/named/clontest.com
/etc/dhcp/dhcpd.conf

restart corresponding services: 

systemctl restart dhcpd
systemctl restart named

and check services status: 

systemctl status dhcpd
systemctl status named

To set gateway, add 

net.ipv4.ip_forward=1

to /etc/sysctl.conf and execute 

sysctl -p

Configure NFS

Configure DHCP server

Configure DNS server

Configure Tftp and Tftpboot

Configure Iptables

Do not forget (is it for controllers ?): 

chattr +i /etc/resolv.conf



yum install ypbind (????)


Configuring snapshot area

The list of files and directories which suppose to be in snapshot area (which has RW permissions) should be set in config file /diskless/.../root/etc/statetab. In particular, /et has to be added there, and also directory /diskless/.../root/et should be created.

Reboot server, make sure everything is running as expected.




Configuring in chroot

Since root area in VME controllers will be read-only, you cannot ssh to controllers as root and change anything there, only snapshot area can be modified. To make changes in root area, in particular to configure and run yum, chroot' command should be used. Login to the boot server as root and do following: 

mount -o bind /home /diskless/CentOS7/x86_64/root/home
mount -o bind /usr/local /diskless/CentOS7/x86_64/root/usr/local
mount -o bind /usr/clas12 /diskless/CentOS7/x86_64/root/usr/clas12
chroot /diskless/CentOS7/x86_64/root

After that you will see all root directories as if you login to VME controller, and modifications can be made.

Configure yum

Add multilib_policy=all to /etc/yum.conf.

Add (and remove the rest ?) following to /etc/yum.repos.d/CentOS-Base.repo: 

[base]
name=CentOS-$releasever - Base
baseurl=http://archive.kernel.org/centos-vault/centos/$releasever/os/$basearch/

Clean yum database: 

rm /var/lib/rpm/__db.*

Install following using yum: 

yum install motif-devel tcl-devel tk-devel libXpm-devel apr-devel libXaw-devel ncurses-devel

Install following for dbedit: 

yum install tix itcl itk

Install remaining tcl stuff from /usr/local/src, mounted as /zzz above: 

cd /zzz/mysqltcl-3.052
make install
ln -s /usr/lib/mysqltcl-3.052 /usr/lib64/tcl8.5/mysqltcl-3.052


NOTE: Login from console may not work because of file /etc/securetty permissions, it must be 644.

NOTE: THere is a service PAM, it may prevent login from console if some required services did not start. To work around, comment out some lines in /etc/pam.d/system-auth-ac file: 

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
#auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
#auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
#auth        required      pam_deny.so

#account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
#account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
#password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
#session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
#session     required      pam_unix.so




After Linux boot server is ready and operational, VME controllers can be booted. On the first boot, individual /diskless/.../snapshot/<controller name> directory will be created for each controller. For VME controller specific information see VME

Retrieved from "https://clonwiki0.jlab.org/wiki/index.php?title=Linux_Boot_Server&oldid=8296"