Linux-KVM Management: Live Migration Without Shared Storage

Linux-KVM Management: Live Migration Without Shared Storage

Everyday I learn something new about Linux-KVM, this is why I love this platform.  Today we are going to talk about Live Migration.  The kicker?  Shared storage not required.  Now please keep in mind that just because shared storage is not required, doesn’t mean there is not a benefit to it.  Shared storage will accelerate this process, since the data will not have to move.  However virtualization is all about flexibility and so is Linux for that matter, so why shouldn’t we be able to make the choice?  That said this process is not documented by the libvirt team, as such it sounds like an “undocumented” feature.  Please use this at your own risk, and for heaven’s sake please test and understand thoroughly before using this in production.

Migrating from kvmnode01 to kvmnode02, both with LVs with the same name.

On the receiving node we must create the Logical Volume which will receive the VM disk

# lvcreate -L8G kvmnode02 -n vm_vmname_boot

Now assuming that on kvmnode01 (where the VM currently runs) it is hosted on LVM, we can simply create a link from the name of the sending Volume Group to the name of the receiving Volume Group.  You can additionally link from a file name if the source VM is using a file backed VM.

# ln -s /dev/kvmnode02 /dev/kvmnode01

Now we can begin the process of migrating the VM, which theoretically could take a very long time depending on the size of data and the amount of change that takes place.  Also please keep in mind the progress which is displayed here is only the memory move progress.  The disk will be moved prior to the memory move, and the progress will sit at zero for that time.

# virsh migrate --live vmname qemu+ssh://kvmnode02.allanglesit.net/system --copy-storage-all --verbose --persistent --undefinesource

Finally once the move has completed you MUST edit the XML so that it references the Logical Volumes on the receiving host, instead of traversing the symlink.  This is because the /dev tree will be repopulated on the next host reboot, so the symbolic link will no longer be there.  If you edit the XML while the VM is running it won’t implement that change until a reboot of the guest, so it will continue to use the symlink until the guest and host are rebooted, but once the reboot happens you will be using the local LV natively, which is a good thing considering the symlink would be gone.

I am currently working on a wrapper script to roll this functionality into a simple live migration script, so watch for that.

7 thoughts on “Linux-KVM Management: Live Migration Without Shared Storage

  1. baraka

    If VG names are the same on both kvm nodes, there is no need of link arragement.

    Any progress with the script?

    thanks for your great posts!

  2. matthew.mattoon Post author

    Hi Baraka,

    You are correct if the VG names are the same then the linking is not necessary, however this was written from the perspective of using the local storage on the machine, my distribution of choice (Ubuntu) will create the VG for you on install and name it to match the host name. If you are manually creating your VG or using shared storage then the linking is not necessary.

    I have not completed the code yet, I hope to get it done this month. I will post it as a new article when it is done.

    -matt

  3. Warhammer

    Let’s assume I’m running VMs from disk file rather than from LV. Is it possible to do live migration in this case?

    1. matthew.mattoon Post author

      That is really a loaded question, frankly there are too many variables to be able to answer authoritatively. That said, I do remember doing some testing around this, however in my tests we used raw disk images, not qcow or any other disk format. I seem to remember some issue when going file > file, it might have been around having to have the file in the same path location, through symlinking or actual path, but I feel like it was more than this. I do remember that when going from file > lv it worked great, you just had to symlink the filename into the lv, which was a bit more confusing since you had (potentially) the file extension to include in the symlink (file.img).

      So long story short do some testing, file to file is not a configuration that I use at all. As soon as something goes into production I use LVM, so while I _might_ use image files on test machine once something goes into production (and thus needs live migration) then I use LVM since the snapshots, expansion, and what not make my life easier.

      -matt

    2. cj

      Well, I migrated vm using disk file rather than from LV. Its not strictly speaking a live migration, but it worked for me when I tested.

      I made a scp of the qcow2 file of the VM (while the VM is still running) from host A to host B. I also made sure the location of the qcow2 file is same in both locations.

      Later, I just migrated VM using virsh and it worked!!

      1. cj

        But, using this approach can cause serious problems in the live environment. It is advised to pause the VM before the transfer of qcow2 file from SERVER A to B.

  4. Andy M

    I’m about to do a bunch of these transfers, did you ever get 5 mins to script this? If not, no worries, thanks very much for the post.

    Andy M