Solaris 11: Configure IP Over Infiniband Devices

Solaris 11: Configure IP Over Infiniband Devices

In this article we will be going over the configuration of an infiniband interface with the IPoIB protocol on Solaris 11, specifically Solaris 11.2 (previous versions of Solaris 11 should work the same, however there have been changes in the ipadm and dladm commands).

Identify Infiniband Datalinks

First we need to identify the underlying interfaces of the infiniband interfaces.  In my case net2 and net3.

# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net0 Ethernet up 1000 full ixgbe0
net2 Infiniband up 32000 unknown ibp0
net3 Infiniband up 32000 unknown ibp1
net5 Ethernet up 1000 full vsw0

Another way to confirm the infiniband interfaces is to use the show-ib command.

# dladm show-ib
LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS
net2 10E0000128EBC8 10E0000128EBC9 1 up kel01-gw01 0a-eth-1 7FFF,FFFF
 kel01-gw02 0a-eth-1
net3 10E0000128EBC8 10E0000128EBCA 2 up kel01-gw01 0a-eth-1 7FFF,FFFF
 kel01-gw02 0a-eth-1

Rename Infiniband Datalinks

I like to rename the datalinks to ib0 and ib1, it makes it easier to keep everything nice and tidy.

# dladm rename-link net2 ib0
# dladm rename-link net3 ib1

Now to show the updated datalinks.

# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net0 Ethernet up 1000 full ixgbe0
ib0 Infiniband up 32000 unknown ibp0
ib1 Infiniband up 32000 unknown ibp1
net5 Ethernet up 1000 full vsw0

Now in subsequent actions we will use ib0 and ib1 as our datalinks.

Create Infiniband Partition

First lets talk about partitions, partitions are most closely related to VLANs.  However the purpose of partitions is to provide isolated segments, so there is no concept of a “router” on IB.  So your use case might be for isolating storage or database services, or even isolating customers from one another (which you definitely should do if you have a multitenant environment where customers have access to the operating system.  So what we want to do is identify the partition to be created, if you do not use IB partitioning, then you will need to use the “default” partition of ffff.

# dladm create-part -l ib0 -P 0xffff pffff.ib0

If you do use partitioning, then you will need to define the partition that you wish to use, for this example 7fff.  Which partition to use is determined by the dladm show-ib output, it lists the PKEY that are available, these are partitions.

# dladm create-part -l ib0 -P 0x7fff p7fff.ib0

Now lets review the partitions.

# dladm show-part
LINK PKEY OVER STATE FLAGS
pffff.ib0 FFFF ib0 unknown ----
p7fff.ib0 7FFF ib0 unknown ----

We now have our two partitions defined.

Create IP Interfaces

Now that we have the Infiniband pieces configured, we simply create the IP interfaces, so that we can subsequently assign an IP address, the IP interfaces are named as follows (ibpartition.interfacename).  Below is for the “default” partition.

# ipadm create-ip pffff.ib0

And for our named partition for 7fff we create an interface as well.

# ipadm create-ip p7fff.ib0

Now we have our interfaces configured correctly.

Create IP Address

Now the easy part this is exactly the same as we would do with a standard ethernet interface.  Assign a static IP address for the default partition.

# ipadm create-addr -T static -a 10.1.10.11/24 pffff.ib0/v4

Also for our named partition.

# ipadm create-addr -T static -a 10.2.10.11/24 p7fff.ib0/v4

Now a few ping tests and we are in business.  Remember you will not be able to ping from one partition to another, so you will need to identify a few endpoints on your existing Infiniband networks to test your configuration.

One thought on “Solaris 11: Configure IP Over Infiniband Devices

  1. chris

    This is great, thanks so much! Couldn’t find this information anywhere. But there is a typo:

    ipadm create-ip pffff.ib0 , should be ipadm create-if pffff.ib0

    Best,
    Chris