Defcon:Blog Keklkakl blog blag


Xen on Debian Stretch (with VLAN networking).

This is an update to my old post about Xen on Debian Wheezy. The process of getting my configuration of Xen running on Debian 9 is very close to the old approach, so I'll only cover the differences.

I've recently been "forced" to update my Xen setup. The setup has been working great for me, and is still working great. But recently I really need to get some Stretch, CentOS and Windows VMs up. The old versions of Xen and pygrub in Wheezy refuses to boot the modern kernels for Linux, and the machine I use as host is unable to du HVM for Windows. And Wheezy is getting really old, and is soon getting dropped out of support.

Installing Xen

The primary install command is still exactly the same as it was before:

apt-get install xen-linux-system

In the old guide, I next described how one would get Grub to automatically boot into the Xen kernel. This is not needed on Stretch, the Xen hypervisor kernel is automatically set as the defaul boot option when it gets installed.
The points about memory allocation and balooning still apply, but the memory requirement is a little bit higher. I recommended reserving 768MB to dom0 on Wheezy. This will work and let you boot and stuff with Stretch, but it will be a bit tight. So upping the reserved allocated memory to a full GB makes sense.

Adding the kernel-option to 'fix' the memory size:

cat <<END >>/etc/default/grub
# Xen boot parameters for all Xen boots

Edit /etc/xen/xend-config.sxp and change the dom0-min-mem and dom0-ballooning lines:

(dom0-min-mem 1024)
(enable-dom0-ballooning no)

Network configuration

There are a few changes to the network setup. One of the more visible changes in configuration files is that Debian is no longer using "legacy names" for network interfaces, instead presenting systemd predictable names. And with this switch to systemd, the familiar command "ifconfig" has been dropped completely, forcing us to move to the much more useful "ip" command.

Next, having dummyX network interfaces now require that you set them up before you try to use them in /etc/network/interfaces. You can use a dummy-interface manually after boot by simply creating it, but to have it available on boot, you need to load the module explicitly. Add the following to /etc/modules:

# /etc/modules: kernel modules to load at boot time.

Next, we need to specify how many dummy-devices we want as a module load option. Create the file /etc/modprobe.d/dummy.conf with a single line:

options dummy numdummies=1

Of course, if you want multiple vm-to-vm local network bridges available, increase the numdummies as desired.

My /etc/network/interfaces is fairly verbosely commented, so again, I'm including it directly, and letting it speak for it self. Remember to install bridge-utils to get working network bridges.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface, used for management and in the future, iSCSI
# In my setup I'm actually using DHCP for address, with the address locked down
# on the DHCP-server side..
allow-hotplug enp3s0
iface enp3s0 inet dhcp
iface enp3s0 inet6 auto

# Create a dummy interface to use with a "local-only" 
# bridge to use for domU-to-domU pure local traffic
auto dummy0
iface dummy0 inet manual
	pre-up ip link set dev $IFACE up
	post-down ip link set dev $IFACE down

# Bring up the NIC for VM networks, but with no address configuration
# The interface will be used later as a bridge-port interface.
# It is also connected to a switchport that does VLAN
# trunking with native (untagged) VLAN 2 and VLANs 3,4 and 5 tagged.
auto ens2f0
iface ens2f0 inet manual
	pre-up ip link set dev $IFACE up
	post-down ip link set dev $IFACE down

# Creating VLANs has become a lot better in Wheezy.
# All that is needed to create a VLAN subinterface now,
# is to add a stanza for a eintfX.Y subinterface. Y becomes
# the VLAN ID.

# Adding VLANs tagged on eth1. These are simply
# brought up, they will be added as bridge-ports later.

auto ens2f0.2
iface ens2f0.2 inet manual
	pre-up ip link set dev $IFACE up
	post-down ip link set dev $IFACE down

auto ens2f0.3
iface ens2f0.3 inet manual
	pre-up ip link set dev $IFACE up
	post-down ip link set dev $IFACE down

auto ens2f0.4
iface ens2f0.4 inet manual
	pre-up ip link set dev $IFACE up
	post-down ip link set dev $IFACE down

auto ens2f0.5
iface ens2f0.5 inet manual
	pre-up ip link set dev $IFACE up
	post-down ip link set dev $IFACE down

# Add the bridge for pure local-only communication between
# domU's. Just for fun, I'm setting an IP on this interface..
auto xenlocal
iface xenlocal inet static
        bridge_ports dummy0
	bridge_stp off

# None of the interfaces below have IP addresses assigned. Any domU hosts using
# these are not supposed to use the bridge for direct communication with dom0.

# The configuration for the VLANs is very repetitive: An 'auto' line to make
# sure the interface comes up automagically, an 'iface' line setting the bridge
# name and defining it to 'manual' configuration, definition of the VLAN
# subinterface, and finally setting Spanning Tree to 'on'.
# If even more control over the bridge interfaces is needed,
# read 'man 5 bridge-utils-interfaces'

# For the VLANs, I used 'vlanX' as bridge-name
# ens2f0 is used as physical-interface for both untagged and for tagged VLANs,
# please review the separate ens2f0 and ens2f0.X definitions above, and separate
# vlanX bridges below. 

auto vlan2
iface vlan2 inet manual
        bridge_ports ens2f0.2
        bridge_stp on
	bridge_maxwait 0

auto vlan3
iface vlan3 inet manual
	bridge_ports ens2f0.3
	bridge_stp on
	bridge_maxwait 0

auto vlan4
iface vlan4 inet manual
	bridge_ports ens2f0.4
	bridge_stp on
	bridge_maxwait 0

auto vlan5
iface vlan5 inet manual
	bridge_ports ens2f0.5
	bridge_stp on
	bridge_maxwait 0

Getting Debian Xen Netinstall data

I've updated my "update-debian" script to make adding and removing releases easier. The new version does the same as the old one: It pulls down the Xen kernel and initramfs images for Debian netinstall from the Debian servers.

DISTROS="wheezy jessie stretch"
for DIST in ${DISTROS}
 for ARCH in i386 amd64
 mkdir -p /srv/xen/boot/debian/${DIST}/${ARCH}
 for FILE in vmlinuz initrd.gz
 wget ${DEBINSTROOT}/${DIST}/main/installer-${ARCH}/current/images/netboot/xen/${FILE} \
 -nc -O /srv/xen/boot/debian/${DIST}/${ARCH}/${FILE}

Creating virtual machines (domU)

In the old post I described how to use some pre-built templates for creating virtual machines. I have generally not used that approach, so I'm not updating that doc. The "official" way of creating new virtual machines seems to be using xen-tools for deployment, or some libvirt magic. Sounds great, I haven't tested it 😀

My way of creating machines is simply to copy a template configuration, and run the installer. My updated template is more or less identical to the old one:

name = "hostname"
memory = 512

# Let xm/xl allocate MAC-address on first boot:
vif = ['bridge=vlan2']
# Get MAC-address from first boot, and update:
#vif = ['mac=xx:xx:xx:xx:xx:xx,bridge=vlan2']

disk = ['file:/srv/xen/hostname_root.img,xvda,w']

# During install, use the erknel/ramdisk/bootloader lines
kernel = "/srv/xen/boot/debian/stretch/amd64/vmlinuz"
ramdisk = "/srv/xen/boot/debian/stretch/amd64/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"

# When install is done, remove those, and switch to pygrub or pvgrub

# Select what to do on a reboot: 
# stop the VM as if a HALT was done, or actually restart.
on_reboot = 'destroy'
#on_reboot = 'restart'

# You can select to auto-restart a VM that crashes.
# I prefer to manyally handle crashes.
on_crash = 'destroy'

The absolute minimum amount of RAM needed to boot the Debian 9 installer is 450MB. I discovered this the hard way, simply because my old template used 384MB RAM as the initial allocation.

This template uses qemu-diskimages, not LVM volumes or such magic. Creating the disk image is as before:

qemu-img create -f raw /srv/xen/hostname_root.img 10G

The toolstack has been changed from xm to xl

Along the upgrade-path from Xen 4.0 to the current 4.9 version, the older "xm" toolstack got replaced by "xl". This really does not make much difference in simple use, as the commands are more-or-less interchangable. My cheat-sheet for commands generally only substitute "xl" where it used to be "xm". Discovering what NIC and MAC-address has been allocated to a VM is now done easily with xl, so poking around in xenstore-ls is no longer needed:

# I use the machine's FQDN as it's domU domainname.
# So, in examples, replace hostname w/Xen domU domainname

# Start VM, don't attach to console
xl create /etc/xen/guests/hostname.cfg

# Start VM, attach to console immediately
xl create -c /etc/xen/guests/hostname.cfg

# Attach console to a running domU/VM
xl console hostname

# To get out of the console again, CTRL+5 is still used.

# Kill a running VM, unclean brutal shutdown
xl destroy hostname

# List running virtual machines/domU's
xl list

# List network adapters (inc MAC) of a given domU
xl network-list hostname

# Real-time status, TOP for Xen
xl top

References and pointers.

This is an update to my old post about Xen on Debian Wheezy. There is more than a little information there that I haven't covered here, because the old info is still correct.

A bit of info on the new-style network interface names can be read at

General documentation links

For more references, simply refer to the old post 🙂

Filed under: Systems administration Tagged as: Leave a comment
Comments (0) Trackbacks (0)

No comments yet.

Leave a comment

Trackbacks are disabled.