Experience migrating from VMware ESXi to KVM in a production environment

Peter Technical

My notes from setting up a production KVM environment, after migrating from VMware ESXi 5.1 to Ubuntu 12.04.2 64-bit with Linux Kernel 3.2 and QEMU 1.4.2, and open vSwitch.

General

Disk I/O throughput and performance characteristics

  • Always use LVM backed storage (which is aligned), with cache='none' and io='native' (aio) for guests. Disabling cache allows the host system to properly schedule disk reads and writes
  • Use deadline I/O scheduler for host systems, and vm.swappiness = 0 in on host or equivalent to reduce pressure on I/O resources and make use of host memory
  • Use virtio for bus type to allow direct access to storage instead of going through QEMU, if supported by guest operating system drivers

Processor

  • Pass through CPU flags to guest to take advantage of newer instruction sets, assuming host hardware is the same or migration is not going to be used (-cpu host)

Network

  • Use virtio Network adapters (except with Windows) to realise full throughput and lower latency on guest operating systems, where support is available (Linux 2.6+, FreeBSD)
  • Load vhost_net kernel module on host, which permits direct access to network devices skipping QEMU (libvirt will detect if vhost_net is enabled, and add vhost=on to qemu command line by default)

Software

  • Linux Kernel 3.5, distributed with Ubuntu 12.04.2 does not support building open-vswitch – you must install Kernel 3.5 for the DKMS to properly build
  • Build QEMU from source to include new functionality and Hyper-V enhancements for Windows guests, using the 1.4 stable branch – 1.5 does not work with libvirt due to the way QEMU help parameters are parsed by the library

Guest

Linux

  • vm.swappiness = 0

FreeBSD

  • vm.defer_swapspace_pageouts = 1
  • kern.timecounter.hardware=ACPI-fast
  • kern.timecounter.smp_tsc="1"
  • kern.timecounter.invariant_tsc="1"
  • Use prebuilt virtio drivers, or compile from ports under emulators/virtio-kmod after each system upgrade

Windows

  • Disable HPET via qemu (-no-hpet) or libvirt configuration, force use of TSC to reduce time drift in guest
  • If you are using qemu 1.4 or greater, enable CPU flags (Hyper-V shims) hv_vapic,hv_relaxed,hv_spinlocks=0xffff on top of disabling HPET
  • Install Memory Ballooning service and drivers, SCSI (virtio) drivers using the stable branch from Redhat
  • Use e1000 Ethernet for Windows 2008 R2 to avoid high latency/freezing of guest operating system

libvirt

  • Add xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' to domain type, if you are going to add custom qemu command args
  • If you are using Ubuntu, and want to change the version of qemu you are going to use – you will either need to disable AppArmor, or update the profile to include the directory you've installed the alternative qemu version to

Migration

  • Make sure to uninstall VMware Tools in the guest environment after you have migrated, and install acpid on the guest to allow graceful shutdowns (if Linux)
  • Take a snapshot of the virtual machine – transfer the VMDK, and use a utility such as vmdksync to merge the deltas after shutting down the VM for the file migration to reduce downtime
  • For Windows VMs, make sure to add a dummy virtio SCSI and Ethernet device so you can install drivers and then switch the root drive to virtio