Mellanox

RHEL7.5

Driver already included into OS, Mellanox card will be identified and configured by default. Still, have to install Mellanox driver to have a tool for example to switch from ib to eth.

yum install numactl

cd /root
cp /usr/downloads/MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz .
tar xvf MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz
rm MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz
cd MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64
./mlnxofedinstall --force --add-kernel-support

/etc/init.d/openibd restart   <- if complains about some modules, unload those or just reboot machine

Run command

/sbin/connectx_port_config --show

to display port settings, change it with

/sbin/connectx_port_config 

setting all ports to 'eth'.

Check (and modify if necessary) following files:

/etc/hostname: contains something like

clonfarm3.jlab.org

/etc/sysconfig/network:

NISDOMAIN=CCCHP
NETWORKDELAY=30
GATEWAY=129.57.167.99

/etc/resolv.conf:

search jlab.org acc.jlab.org
nameserver 129.57.90.255
nameserver 129.57.32.101

run nmtui, set port name like p3p1. Set 'IPv4 as Manual', IP address with mask (like 129.57.167.103/24) and gateway (like 129.57.167.99) only, ignore the rest; file /etc/sysconfig/network-scripts/ifcfg-p6p1 should looks like following:

HWADDR=F4:52:14:41:07:71
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=129.57.167.103
PREFIX=24
GATEWAY=129.57.167.99
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=p3p1
UUID=c9f979fb-c7ca-3237-bacc-4118389d03af
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999

In file 'ifcfg-p6p1' set 'ONBOOT=yes', in file 'ifcfg-em1' set 'ONBOOT=no'.

Reboot machine. When it started to go back, unplug copper wire and plug mellanox in.

After machine is booted, check /etc/resolv.conf, sometimes it is loosing contents, restore it if needed.

Useful command: netstat -rn shows routing.

RHEL7.4

Run

yum install numactl

cd /root
cp /usr/downloads/MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64.tgz .
gunzip MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64.tgz
tar xvf MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64.tar
rm MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64.tar
cd MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64
./mlnxofedinstall --force

It will install everything and print following:

Please reboot your system for the changes to take effect.
To load the new driver, run:
/etc/init.d/openibd restart

Change following files:

/etc/hostname: contains something like

clondaq3.jlab.org

/etc/sysconfig/network:

NISDOMAIN=CCCHP
GATEWAY=129.57.167.99

/etc/resolv.conf:

search jlab.org acc.jlab.org
nameserver 129.57.90.255
nameserver 129.57.32.101

run nmtui, set p6p1. Set 'Manual', IP address with mask (like 129.57.167.226/24) and gateway (like 129.57.167.99) only, ignore the rest; file /etc/sysconfig/network-scripts/ifcfg-p6p1 should looks like following:

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=p6p1
UUID=ec3cd5dc-8f9e-4e13-aa0f-a9c1ec1996c2
DEVICE=p6p1
ONBOOT=no
PROXY_METHOD=none
BROWSER_ONLY=no
IPADDR=129.57.167.226
PREFIX=24
GATEWAY=129.57.167.99

In file 'ifcfg-p6p1' set 'ONBOOT=yes', in file 'ifcfg-em1' set 'ONBOOT=no'.

Reboot machine. When it started to go back, unplug copper wire and plug mellanox in.

After machine is booted, check /etc/resolv.conf and restore it if needed.

Useful command: netstat -rn shows routing.

RHEL7

Run

yum install numactl

cd /root
cp /usr/downloads/mlnx-en-3.1-1.0.4.tgz .
gunzip mlnx-en-3.1-1.0.4.tgz
tar xvf mlnx-en-3.1-1.0.4.tar
rm mlnx-en-3.1-1.0.4.tar
cd mlnx-en-3.1-1.0.4
./install.sh

#Run command
# /sbin/connectx_port_config --show
#to display port settings, change it with
# /sbin/connectx_port_config 
#setting all ports to 'eth'.

Configure new 'eth' port as desired (left port seems p6p2 now ...). Use for example text interface to NetworkManager called 'nmtui'. Set 'Manual', IP address with mask (like 129.57.167.41/24) and gateway (like 129.57.167.99) only, ignore the rest. Reboot machine, unplug copper ethernet when starts booting, it should come back using fiber.

RHEL6

Run

yum install numactl

cd /root
cp /usr/downloads/MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.iso .
mkdir tmp
mount -o loop MLNX_OFED_LINUX-3.1-1.0.3-rhel6.7-x86_64.iso tmp
cd tmp
./mlnxofedinstall

Run command (as instructed by installation process):

/etc/init.d/openibd restart

Run command

/sbin/connectx_port_config --show

to display port settings, change it with

/sbin/connectx_port_config 

setting all ports to 'eth'.

After it installed, run /usr/bin/system-config-network and add new device eth0 as manual network with appropriate settings (do NOT specify DNS servers !). Shutdown computer, unplug copper cable and start computer, 10G fiber link should become default one. OR, if fiber already plugged in, set BOOT=no for copper port currently in use, set BOOT=yes for fiber port, and reboot.