Linux-KVM: Ubuntu 12.04 with Openvswitch

Linux-KVM: Ubuntu 12.04 with Openvswitch

UPDATE: October 29, 2012

This article doesn’t work on Ubuntu 12.10, to use Ubuntu 12.10 please refer to my new article here.

Openvswitch is a better way of managing your virtual networking stacks for KVM and Xen.  This can be used instead of the bridge-utils package, and has the ability to use VLANs, LACP, QoS, sFlow, and others.  In this article we will be using Ubuntu 12.04 (beta1) to configure Openvswitch to hand out networking interfaces to KVM guests.

Please keep in mind that at the time of this writing Ubuntu 12.04 is beta, which means it is not fully baked for production.  Though it is looking very stable.

Install Updates and Prerequisite Software

Check for updates, and install some commonly used hypervisor utilities along with the openvswitch and kvm stacks.

# apt-get update && apt-get dist-upgrade
# apt-get install aptitude apt-show-versions ntp ntpdate vim kvm libvirt-bin vlan virtinst virt-manager virt-viewer openssh-server iperf pv openvswitch-controller openvswitch-brcompat openvswitch-switch openvswitch-datapath-source

Destroy the Default Libvirt Bridge (virbr0)

We want to delete the default libvirt interface which is created by default to be used for guests (I think it is NAT – I never use it).

# virsh net-destroy default
# virsh net-autostart --disable default

Stop Libvirt and Qemu

This will prevent libvirt from bringing up the legacy bridge.

# service libvirt-bin stop
# service qemu-kvm stop

Enable Openvswitch for Brcompat

# vi /etc/default/openvswitch-switch
BRCOMPAT=yes

Purge Ebtables from System

Ebtables was needed when we were using VLANs on bridge-utils, we no longer need this for openvswitch.

# aptitude purge ebtables

Fix DKMS Module Build

When we originally installed openvswitch, you might have noticed an error talking about the module not available for your kernel, if you didn’t notice it you can see the same error by trying to start the openvswitch-switch service.

# service openvswitch-switch start

If you receive the module error, then we will need to build the openvswitch-datapath module.

# module-assistant auto-install openvswitch-datapath

Note that after installing new kernels you will most likely need to repeat this step, to rebuild the module.

Restart the Openvswitch Services

# service openvswitch-switch restart
# service openvswitch-controller restart

Reboot

# reboot

Check Configuration

Based on your reboot one of two things happened…  Your modules came back up loaded and ready for business, though more likely they didn’t come up.

Here is an example of a working configuration.  If you get this then move on to configuration of the switch itself.

# lsmod | grep brcom
brcompat_mod 13512 0
openvswitch_mod 83993 2 brcompat_mod
# service openvswitch-switch status
ovsdb-server is running with pid 1885
ovs-vswitchd is running with pid 1908
ovs-brcompatd is running with pid 2096
Here is an example of a non-working configuration.  If your modules are not loaded then you will need to get that sorted before we blow a configuration on there and try to pass traffic.
# lsmod | grep brcom
# service openvswitch-switch status
ovsdb-server is running with pid 1885
ovs-vswitchd is running with pid 1908
ovs-brcompatd is running with pid 2096

Ensuring the Modules Load at Boot

The easy way to fix this is to restart the openvswitch-switch service whenever you reboot your hypervisor.

# service openvswitch-switch restart
* Killing ovs-brcompatd (2096)
* Killing ovs-vswitchd (1908)
* Killing ovsdb-server (1885)
* Starting ovsdb-server
* Configuring Open vSwitch system IDs
* Starting ovs-vswitchd
* Starting ovs-brcompatd
* iptables already has a rule for gre, not explicitly enabling

But that might not be ideal for your situation.  As an alternative we can try and address the timing issue which is causing the whole issue in the first place.  Basically what is happening is that upstart is starting openvswitch too late, and it is failing.  If we adjust the waits in /etc/init/failsafe.conf

We want to change this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 40
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 59
$PLYMOUTH message --text="Booting system without full network configuration..." || :

To this…

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :

Now after a reboot you shouldn’t have this problem with the modules not loading.

Reboot

# reboot

Check Modules

# lsmod | grep brcom
brcompat_mod           13512  0
openvswitch_mod        83993  1 brcompat_mod

Design our Network Layout

In our example we will use 2 physical interfaces, one of which (eth0) we will consider our management interface, it will have a static IP address and will be “the hypervisor” the other one (eth1) will be the uplink for our Openvswitch.  Now eth1 will have an untagged fake bridge (br0) which will be the parent for our other fake bridge (br10) which will be tagged 10.  If we had more to add they would still use br0 as the parent and then be tagged on the subordinate fake bridge.  Another important thing to keep in mind is that your uplinked switch port must support and be configured to allow your devices to tag for these other VLANs.

Configure Openvswitch Fake Bridges

# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 eth1
# ovs-vsctl add-br br10 br0 10
# ovs-vsctl list-br
br0
br10

Create Dummy Interface Entries

We need to edit /etc/network/interfaces to add some entries

# cat /etc/network/interfaces
auto eth0
iface eth0 inet static
address 10.0.0.141
netmask 255.255.255.0
gateway 10.0.0.1

auto eth1
iface eth1 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto br0
iface br0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

auto br10
iface br10 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down

These bring up the interfaces with No IP address, now if you run ifconfig -a they should be visible.

Once you have a guest up on br0 or br10 you will see a  “vnet0” interface.

35 thoughts on “Linux-KVM: Ubuntu 12.04 with Openvswitch

  1. TooMeeK

    Very interesting thing, I was looking some info about KVM + Open vSwitch implementation and Uncle Google redirected me here..
    Yep definitely gonna try it!

  2. TooMeeK

    I have server running with 2 x Intel PRO/1000 NICs in round-robing aggregated link, and a second runnig with 5 x Intel/PRO NICs. I see the limitation on bridge-tools where I can only hit 1Gbit/s for single NIC in VM, so what about Open vSwitch..?
    Do You know something about performance, is it better or not?

    1. matthew.mattoon Post author

      Hi TooMeeK,

      Openvswitch is a soft-switch, and it supports LACP, so that said if your physical switch also supports LACP, then they should be able to create an actual LACP configuration between groups of interfaces.

      I personally haven’t had the need to aggregate ports on hosts, I have some low capacity hosts that only need 1Gbps and some high capacity hosts which have 10Gbps. So I haven’t had the need to use multiple NICs. But it is most definitely possible.

      -matt

  3. TooMeeK

    Heh, I was asking about BRIDGE capacity.
    Example:
    I create bond0 in round-robin aggregation. I can reach about 1,9 Gbit/s between servers with iperf easily (saturating both links).
    Now, same thing, but using br0 which is bridged to bond0 (standard bridge). 1Gbit/s speed only.
    Now I’m wondering is it possible to reach more with Open vSwitch.
    Yep, I have do some test. We do not have 10Gbit-based network.. 🙁 too expensive.
    All my switches support LACP, however static aggregation is used. LACP sometimes confuses different model of switches. We use up to 4Gbit/s static aggregation links, but I’ve been able to create 8Gbit/s LACP. Again, not reachable for single connection, only for multiple connections between multiple hosts.

    1. matthew.mattoon Post author

      Sorry about the delay, you fell off of my radar. I am not sure that I understand your question… Basically bridges inherit the capacity of the underlying interface. If I have a bridge on a 1Gb connection then the bridge is 1Gb. If I have the bridge on a bonded interface of 2Gb then the bridge is 2Gb. However with bonds the connection is not really aggregated, in that you can’t have a single connection using all 2Gb, but you can have 2 connections each using a 1Gb. This is due to the nature of switching. The switch must be aware of that sort of configuration through LACP, in order to provide true link aggregation.

      Now Openvswitch changes this proposition in that you are connecting all VMs via this virtual switch, which its only real speed limitation is in software and the underlying memory bus. So guest to guest is very fast. Meanwhile, the outbound connection can be true LACP to the core switches of your network, allowing an uplink of whatever you can aggregate, and then have a single guest be able to utilize the whole thing.

      Architecturally it is a much better solution, that and add in the other switch type features that we have with Openvswitch, such as QoS, sFlow, etc. and you have a very compelling stack.

      -matt

      1. TooMeeK

        Well.. U missed something 🙂 I’m getting something like:

        root@vswitch1:~# service openvswitch-switch restart
        [ ok ] ovs-brcompatd is not running.
        [ ok ] ovs-vswitchd is not running.
        [ ok ] ovsdb-server is not running.
        ERROR: could not insert ‘brcompat_mod’: Exec format error
        [FAIL] Inserting brcompat module … failed!
        Module has probably not been built for this kernel.
        For instructions, read
        /usr/share/doc/openvswitch-datapath-source/README.Debian
        ERROR: could not insert ‘brcompat_mod’: Exec format error
        [FAIL] Inserting brcompat module … failed!

        I’ve installed “aptitude install linux-headers-`uname -r`” before “module-assistant auto-install openvswitch-datapath” step and still doesn’t work..

          1. TooMeeK

            seems it just requires 2.6.x kernel..
            locate brcompat_mod
            /lib/modules/2.6.32-5-amd64/updates/dkms/brcompat_mod.ko
            /var/lib/dkms/openvswitch/1.4.0+git20120426/2.6.32-5-amd64/x86_64/module/brcompat_mod.ko
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.brcompat_mod.ko.cmd
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.brcompat_mod.mod.o.cmd
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.brcompat_mod.o.cmd
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.ko
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.mod.c
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.mod.o
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.o
            /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.tmp_versions/brcompat_mod.mod

        1. matthew.mattoon Post author

          It sounds to me like the problems you are having is because you are not using the same environment as I was. I was using Ubuntu 12.04 Beta1 and my instructions were written with that in mind, however I have been using the release version of 12.04, and have updated frequently as well, so on 12.04 the instructions are solid…

          Based on some of your comments it looks like you are using an older version of Ubuntu (or perhaps a different distro), you mention 2.6.32 which would have been either Lucid or Maverick. It also looks like you might be using a daily build of openvswitch. So lets start with the basics on your environment…

          -matt

  4. matthew.mattoon Post author

    Hi Kris,

    If you don’t enable openvswitch for brcompat, then you will be unable to create fake bridges. If you are unable to create the fake bridges then libvirt (which currently requires bridges) will be unable to utilize the openvswitch for connectivity. So it entirely depends on your use case. I believe (though I am not positive and it will be dependent on versions and what not) that you can utilize openvswitch for connectivity via the command line directly, in other words if you manually execute your VM processes by executing a qemu/qemu-kvm/kvm command. This removes libvirt from the picture (virt-manager uses libvirt under the covers) which removes the requirement for bridges.

    Again I have not tested this and I am not positive. The way I have documented is the only way I am aware of it working. There is another way to utilize a single network adapter (if you had 10Gb or had low enough load that a single 1Gb would be sufficient) however this way will result in losing all connectivity to the host (outside of drac/ilom) if you lost the openvswitch (like when doing kernel upgrades) which is why I haven’t yet documented it here.

    -matt

  5. Kris

    Hi Matt,

    You are right,today i encountered an error when create a VM by virt-manager, and i selected “Specify shared device name”, and input br0 (which created in OVS). after click Finish button, an error display: ” Unable to complete install: ‘Cannot get interface MTU on ‘br0′: No such device’.
    Please send me a email at zhang.kris@gmail. when you write a document about creating VM by kvm/qemu command.

    Thanks a lot,
    Kris

    1. matthew.mattoon Post author

      Kris,

      I will most likely not create a document with regards to that. I prefer to manage my VMs using libvirt. It shouldn’t be too long before libvirt has this support inbuilt. I believe its current status is that it is in the testing versions, however hasn’t been exposed via virtual machine manager. More information can be found about the kvm/qemu command by reading the man page. Keep in mind that the downside of using this command is you remove the management layer from the equation and you will always need to start the VM using that command, it will not have some sort of persisent settings for a VM. It is almost like you are not “creating” a VM but “executing” one. Best to stick with brcompat for now until they roll support fully through the libvirt stack.

      -matt

  6. Victor

    Hi Matt,

    Thanks for the HOWTO, it works fine for me even if I have one question about it.
    Can I skip the step “Destroy the default libvirt bridge” ? Personnaly, I would say yes because virbr0 is for NAT and vnet0, vnet1, … are for the bridged mode. However, I am not sure because I don’t know well LibVirt stack, so what is your thought about this ?

    1. matthew.mattoon Post author

      Victor,

      The steps I documented are the steps I recommend. I have not done any functional testing with the libvirt bridge still intact, however the steps I have documented above have been thoroughly tested and further more I use them everyday on a subset of my production machines (I haven’t had a chance to migrate everything over to openvswitch). Feel free to test outside of the lines I have drawn, just make sure that, you are familiar enough with what you are doing to understand what and why you are doing it, and that you can undo whatever you do.

      -matt

  7. Alexander

    TooMeeK :seems it just requires 2.6.x kernel..
    locate brcompat_mod
    /lib/modules/2.6.32-5-amd64/updates/dkms/brcompat_mod.ko
    /var/lib/dkms/openvswitch/1.4.0+git20120426/2.6.32-5-amd64/x86_64/module/brcompat_mod.ko
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.brcompat_mod.ko.cmd
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.brcompat_mod.mod.o.cmd
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.brcompat_mod.o.cmd
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.ko
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.mod.c
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.mod.o
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/brcompat_mod.o
    /var/lib/dkms/openvswitch/1.4.0+git20120426/build/datapath/linux/.tmp_versions/brcompat_mod.mod

    The problem is that you have the bridge module loaded. You need to destroy your bridge, then unload the module and load the brcompat one. After that you should be ready to go.

  8. tweene

    is there any difference between installing OVS using openswitch-switch right away from terminal and installing using package from openvswitch.org?
    i’m actually installed openvswitch-1.6.1 on linux 12.04.
    and how can i make sure within my OVS is really running or not?

    1. matthew.mattoon Post author

      Yes. When you install openvswitch using the apt-get command, you are installing packages from a repository. If you install from openvswitch.org/download then you are downloading source files and must compile them yourself. From source will give you a more recent version as it will have the latest and greatest, but you really can’t beat the simplicity of packages.

      To check to see if openvswitch is running you can check the status of the service (as is outlined in the article above)…

      # service openvswitch-switch status

      Though more importantly you should be able to pass traffic over it. In this article I am assuming you want to use it to hand out networking to KVM guests, so if they can connect to the fake bridge and get to the larger network then it is working. If you have a different use case you will need to come up with your own use case.

      -matt

  9. Heinz-Bastian

    Hi!
    Has someone tried to run openvswitch with Ubuntu 12.10.
    I keep getting the error that the brcompat module cannot be loaded.
    Any help appreciated.

    1. matthew.mattoon Post author

      I have not verified these instructions on 12.10. But this sounds like a dkms error… See the section in the article called Fix DKMS Module Build.

      -matt

  10. Scott Lowe

    It appears that the datapath module will not compile correctly under Ubuntu 12.10. Matthew, when you try to use the instructions in the “Fix DKMS Module Build” section of your article for 12.10, the compile process fails with an error and will never build a module that actually loads. I have not been able to find any workaround or locate any additional information on what is causing the failure.

    1. matthew.mattoon Post author

      Scott,

      I will take a look at this. Look for either a comment or a full article if it warrants that. I am assuming you goal is to use Openvswitch in combination with KVM?

      -matt

      1. matthew.mattoon Post author

        I have an article which addresses this issue it will be published tomorrow morning 6AM Central.

  11. Scott Lowe

    Matthew, did I miss the blog post that addresses the issue with the DKMS module build under Ubuntu 12.10? (And yes, my goal is to use OVS in combination with KVM and libvirt.)

    1. matthew.mattoon Post author

      The article was published on Wednesday morning. I updated this article with a link.

  12. Arun Sharma

    I have followed the above notes but facing one problem.

    Problem – whenever we add an additional port in ovsbr0 then the IP address of ovsbr0 is unreachable. Any idea?

    Basically, i am just trying to boot KVM with “eth0, and ovsbr0 associated with eth0”

    root@kvmserver:~# ifconfig
    eth0 Link encap:Ethernet HWaddr fa:16:3e:66:10:02
    inet6 addr: fe80::f816:3eff:fe66:1002/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:2798 errors:0 dropped:0 overruns:0 frame:0
    TX packets:1193 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:200242 (200.2 KB) TX bytes:139438 (139.4 KB)
    Interrupt:10

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:559 errors:0 dropped:0 overruns:0 frame:0
    TX packets:559 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:45648 (45.6 KB) TX bytes:45648 (45.6 KB)

    ovsbr0 Link encap:Ethernet HWaddr fa:16:3e:66:10:02
    inet addr:10.34.102.253 Bcast:10.34.102.255 Mask:255.255.255.0
    inet6 addr: fe80::f816:3eff:fe66:1002/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:1718 errors:0 dropped:0 overruns:0 frame:0
    TX packets:1166 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:167278 (167.2 KB) TX bytes:137368 (137.3 KB)

    root@kvmserver:~# cat /etc/network/interfaces
    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).

    # The loopback network interface
    auto lo
    iface lo inet loopback

    auto eth0
    iface eth0 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    down ifconfig $IFACE down

    auto ovsbr0
    iface ovsbr0 inet dhcp
    hwaddress ether fa:16:3e:66:10:02

    root@kvmserver:~# ovs-vsctl add-port ovsbr0 p1
    <<<<>>>>

    1. matthew.mattoon Post author

      Hi Arun,

      Your issue is that you are trying to use the same interface as an interface and as a bridge uplink. This is not possible. What you need to do instead is have a virtual interface hanging off of the bridge, which the hypervisor can use as if it were the physical interface. I covered this in my Ubuntu 12.10 with Openvswitch article. Basically I create ovsbr0p1 which serves this purpose.

      http://blog.allanglesit.com/2012/10/linux-kvm-ubuntu-12-10-with-openvswitch/

      -matt

    2. matthew.mattoon Post author

      My previous response was partially incorrect. I was assuming that you were trying to assign the IP address to the eth0 as well as your ovsbr0p1. However the end result is the same…

      Let me try to explain.

      1) ovsbr0 == the openvswitch
      2) eth0 == uplink on the switch
      3) ovsbr0p1 == valid interface for the hypervisor to use.

      So bottom line is that neither ovsbr0 or eth0 is a traditional addressable interface, ovsbr0p1 is… Your configuration is trying to use ovsbr0 as if it were ovsbr0p1. But when you do that it breaks the ability for ovsbr0 to support ovsbr0p1.

      So to fix your problem. Remove the references to ovsbr0 in the /etc/network/interfaces, add a reference to ovsbr0p1 with the dhcp option, and restart networking.

      Example:

      # cat /etc/network/interfaces
      auto lo
      iface lo inet loopback

      # The primary network interface
      auto eth0
      iface eth0 inet manual
      up ifconfig $IFACE 0.0.0.0 up
      down ifconfig $IFACE down

      auto ovsbr0p1
      iface ovsbr0p1 inet dhcp

      Also since you have ovsbr0 already defined you might need to reboot the box to make all of its artifacts disappear so that it can support ports (ovsbr0p1).

      -matt

  13. Arun Sharma

    Thanks Matt, for your reply. Basically i tried to configure the KVM server to be quite identical to Xenserver 6.0. In that it has similar kind of configuration the way i am trying to do. Not sure how it works there.

    I tried the alternate soln. which you have shared in above link on 12.04 and it works. But still need to know how it works in Xenserver 6.0

    1. matthew.mattoon Post author

      Hi Arun,

      Openvswitch should be pretty similar from platform to platform. However I have not used XenServer 6.0, though the command APIs are probably different. So I am afraid I can’t help with that department. Though I would expect that it would be simpler to enable ovs for XenServer than for KVM.

      -matt

  14. Arun Sharma

    I am just surprised/willing to know, why “ovsbr0” is not an traditional addressable interface. As “ovsbr0” and “ovsbr0p1” both are internal interface as per below output. We can see in ovsdb, that we have interface record for both “ovsbr0” and “ovsbr0p1”

    # ovs-vsctl show
    ed1986fc-830e-4f66-9ce7-92fad0787bb8
    Bridge “ovsbr0”
    Port “ovsbr0”
    Interface “ovsbr0”
    type: internal
    Port “ovsbr0p1”
    Interface “ovsbr0p1”
    type: internal
    Port “eth0”
    Interface “eth0”
    ovs_version: “1.4.3”

    1. matthew.mattoon Post author

      Hi Arun,

      As you see in your output ovsbr0 is also the bridge device. In order for it to function as a bridge device (read: switch) it has to not be operable as an interface, otherwise it doesn’t know which traffic to pass and which to own. As to if it is addressable or not, techincally it is. However by addressing it you break any ports which are attached to the openvswitch (which you discovered in your experimentation). Thus I simply consider it non-addressable, since there is no reason to have openvswitch if you don’t plan on using it as a switch.

      -matt