How to set IP over Infiniband.

With Infiniband be could gain more speed for data transfer between 2 nodes. There are a lot of ways to use Infiniband. In this particular case we will use it for TCP/IP connection. It’s not the fastest method for data transfer but it’s just a start. We could user iSCSI over Infiniband as well. We will work with QLE7342 Card on Debian Jessie.

  • First we will configure some kernel options on grub. We just need to modify this line in /etc/default/grub
    • GRUB_CMDLINE_LINUX_DEFAULT=”quiet”
  • for
    • GRUB_CMDLINE_LINUX_DEFAULT=”quiet options ib_qib singleport=1 krcvqs=2″
  • (Here you could find the configuration for your processor)

 

  • We update grub’s configuration:
    • update-grub
  • And reboot the server.

 

  • Once again on our server, we have to install the Infiniband tools and some dependencies:
    • apt-get install -y build-essential infiniband-diags opensm
  • Next we need to add the modules to /etc/modules:

# Protocol modules
# Common modules
rdma_ucm
ib_umad
ib_uverbs

# IP over IB
ib_ipoib

  • And you need to add them for first time:
    • modprobe rdma_ucm
    • modprobe ib_umad
    • modprobe ib_uverbs
    • modprobe ib_ipoib
  • With this card I had a problem that leave the card as “Disable” and no leds on. To fix that we have to enable it running this command:
    • ibportstate -D 0 1 enable

 

  • Now we have to configure the Network interface. Add to the /etc/network/interfaces the block for the card:

auto ib0
iface ib0 inet static
address 172.31.130.14
netmask 255.255.255.0
broadcast 172.31.130.255

  • We up the card:
    • ifup ib0
  • And we are ready to use IPoIB.
Print Friendly, PDF & Email

Pablo Javier Furnari

Linux System Administrator at La Plata Linux
I'm a Linux Sysadmin with 8 years of experience. I work with several clients as a consulter here in Argentina and oversea (I have clients in the United States, Mexico, Pakistan and Germany).

I know my strengths and weaknesses. I'm a quick learner, I know how to work with small and big teams. I'm hard worker, proactive and I achieve everything I propose.

Leave a Reply

Your email address will not be published. Required fields are marked *


CAPTCHA Image
Reload Image