Month: May 2013

Experience migrating from VMware ESXi to KVM in a production environment

Peter

My notes from setting up a production KVM environment, after migrating from VMware ESXi 5.1 to Ubuntu 12.04.2 64-bit with Linux Kernel 3.2 and QEMU 1.4.2, and open vSwitch.

General

Disk I/O throughput and performance characteristics

  • Always use LVM backed storage (which is aligned), with cache='none' and io='native' (aio) for guests. Disabling cache allows the host system to properly schedule disk reads and writes
  • Use deadline I/O scheduler for host systems, and vm.swappiness = 0 in on host or equivalent to reduce pressure on I/O resources and make use of host memory
  • Use virtio for bus type to allow direct access to storage instead of going through QEMU, if supported by guest operating system drivers

Processor

  • Pass through CPU flags to guest to take advantage of newer instruction sets, assuming host hardware is the same or migration is not going to be used (-cpu host)

Network

  • Use virtio Network adapters (except with Windows) to realise full throughput and lower latency on guest operating systems, where support is available (Linux 2.6+, FreeBSD)
  • Load vhost_net kernel module on host, which permits direct access to network devices skipping QEMU (libvirt will detect if vhost_net is enabled, and add vhost=on to qemu command line by default)

Software

  • Linux Kernel 3.5, distributed with Ubuntu 12.04.2 does not support building open-vswitch – you must install Kernel 3.5 for the DKMS to properly build
  • Build QEMU from source to include new functionality and Hyper-V enhancements for Windows guests, using the 1.4 stable branch – 1.5 does not work with libvirt due to the way QEMU help parameters are parsed by the library

Guest

Linux

  • vm.swappiness = 0

FreeBSD

  • vm.defer_swapspace_pageouts = 1
  • kern.timecounter.hardware=ACPI-fast
  • kern.timecounter.smp_tsc="1"
  • kern.timecounter.invariant_tsc="1"
  • Use prebuilt virtio drivers, or compile from ports under emulators/virtio-kmod after each system upgrade

Windows

  • Disable HPET via qemu (-no-hpet) or libvirt configuration, force use of TSC to reduce time drift in guest
  • If you are using qemu 1.4 or greater, enable CPU flags (Hyper-V shims) hv_vapic,hv_relaxed,hv_spinlocks=0xffff on top of disabling HPET
  • Install Memory Ballooning service and drivers, SCSI (virtio) drivers using the stable branch from Redhat
  • Use e1000 Ethernet for Windows 2008 R2 to avoid high latency/freezing of guest operating system

libvirt

  • Add xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' to domain type, if you are going to add custom qemu command args
  • If you are using Ubuntu, and want to change the version of qemu you are going to use – you will either need to disable AppArmor, or update the profile to include the directory you've installed the alternative qemu version to

Migration

  • Make sure to uninstall VMware Tools in the guest environment after you have migrated, and install acpid on the guest to allow graceful shutdowns (if Linux)
  • Take a snapshot of the virtual machine – transfer the VMDK, and use a utility such as vmdksync to merge the deltas after shutting down the VM for the file migration to reduce downtime
  • For Windows VMs, make sure to add a dummy virtio SCSI and Ethernet device so you can install drivers and then switch the root drive to virtio

Sophos AntiVirus (SAVDI) and amavisd-new for AntiVirus on email

Peter

Update: Based on an email I received, I've updated this post with more relevant information regarding setting up SAVDI with amavisd-new.

I recently migrated to using Postfix with amavisd-new on Ubuntu Linux, and was looking at integrating Sophos AntiVirus with amavisd-new. amavisd-new shipping with the LTS release of Ubuntu is 2.6.5, which does not include SPPP functionality for communicating with savdi so you must use Sophie protocol.

The following components were used for setting up this functionality with amavisd-new from MySophos Download & Updates:

This post is assuming that you have setup amavisd-new on your system, and have it integrated with Postfix or equivalent MTA already.

savdid.conf:

channel {
commprotocol {
type: UNIX
socket: /var/run/savdid/savdid.sock
user: amavis
group: amavis
requesttimeout: 120
sendtimeout: 2
recvtimeout: 5
}

scanprotocol {
type: SOPHIE
allowscandir: SUBDIR
maxscandata: 500000
maxmemorysize: 250000
tmpfilestub: /tmp/savid_tmp
}

scanner {
type: SAVI
inprocess: YES
maxscantime: 3
maxrequesttime: 10
deny: /dev
deny: /home
savigrp: GrpArchiveUnpack 0
savigrp: GrpInternet 1
savists: Xml 1
}
}

This should permit amavisd-new to communicate with the SAVDID interface. Don't forget to create the appropriate init.d script to start savdid on boot and make sure to create the /var/run/savdid directory – as Debian/Ubuntu clean /var/run on system startup. Please download the init script from here and place it in /etc/init.d/savdid with an executable bit.

amavis communicates with SAVI using the Sophie protocol, to enable this support in amavis-new edit /etc/amavis/conf.d/15-av_scanners and add the following lines:

  ['Sophie',
    \&ask_daemon, ["{}/\n", '/var/run/savdid/savdid.sock'],
    qr/(?x)^ 0+ ( : | [\000\r\n]* $)/m,  qr/(?x)^ 1 ( : | [\000\r\n]* $)/m,
    qr/(?x)^ [-+]? \d+ : (.*?) [\000\r\n]* $/m ],

Please be aware that the socket line is being pointed at the place where we have SAVI listening for connections, based on our previous post.