In this article we will be going over the configuration of an infiniband interface with the IPoIB protocol on Solaris 11, specifically Solaris 11.2 (previous versions of Solaris 11 should work the same, however there have been changes in the ipadm and dladm commands).
Identify Infiniband Datalinks
First we need to identify the underlying interfaces of the infiniband interfaces. In my case net2 and net3.
# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet unknown 0 unknown ixgbe1 net0 Ethernet up 1000 full ixgbe0 net2 Infiniband up 32000 unknown ibp0 net3 Infiniband up 32000 unknown ibp1 net5 Ethernet up 1000 full vsw0
Another way to confirm the infiniband interfaces is to use the show-ib command.
# dladm show-ib LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS net2 10E0000128EBC8 10E0000128EBC9 1 up kel01-gw01 0a-eth-1 7FFF,FFFF kel01-gw02 0a-eth-1 net3 10E0000128EBC8 10E0000128EBCA 2 up kel01-gw01 0a-eth-1 7FFF,FFFF kel01-gw02 0a-eth-1
Rename Infiniband Datalinks
I like to rename the datalinks to ib0 and ib1, it makes it easier to keep everything nice and tidy.
# dladm rename-link net2 ib0 # dladm rename-link net3 ib1
Now to show the updated datalinks.
# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet unknown 0 unknown ixgbe1 net0 Ethernet up 1000 full ixgbe0 ib0 Infiniband up 32000 unknown ibp0 ib1 Infiniband up 32000 unknown ibp1 net5 Ethernet up 1000 full vsw0
Now in subsequent actions we will use ib0 and ib1 as our datalinks.
Create Infiniband Partition
First lets talk about partitions, partitions are most closely related to VLANs. However the purpose of partitions is to provide isolated segments, so there is no concept of a “router” on IB. So your use case might be for isolating storage or database services, or even isolating customers from one another (which you definitely should do if you have a multitenant environment where customers have access to the operating system. So what we want to do is identify the partition to be created, if you do not use IB partitioning, then you will need to use the “default” partition of ffff.
# dladm create-part -l ib0 -P 0xffff pffff.ib0
If you do use partitioning, then you will need to define the partition that you wish to use, for this example 7fff. Which partition to use is determined by the dladm show-ib output, it lists the PKEY that are available, these are partitions.
# dladm create-part -l ib0 -P 0x7fff p7fff.ib0
Now lets review the partitions.
# dladm show-part LINK PKEY OVER STATE FLAGS pffff.ib0 FFFF ib0 unknown ---- p7fff.ib0 7FFF ib0 unknown ----
We now have our two partitions defined.
Create IP Interfaces
Now that we have the Infiniband pieces configured, we simply create the IP interfaces, so that we can subsequently assign an IP address, the IP interfaces are named as follows (ibpartition.interfacename). Below is for the “default” partition.
# ipadm create-ip pffff.ib0
And for our named partition for 7fff we create an interface as well.
# ipadm create-ip p7fff.ib0
Now we have our interfaces configured correctly.
Create IP Address
Now the easy part this is exactly the same as we would do with a standard ethernet interface. Assign a static IP address for the default partition.
# ipadm create-addr -T static -a 10.1.10.11/24 pffff.ib0/v4
Also for our named partition.
# ipadm create-addr -T static -a 10.2.10.11/24 p7fff.ib0/v4
Now a few ping tests and we are in business. Remember you will not be able to ping from one partition to another, so you will need to identify a few endpoints on your existing Infiniband networks to test your configuration.