ERIKA3 on the Xen hypervisor

Introduction

Xen is an open-source type-1 or bare-metal hypervisor, that runs directly on the hardware and is responsible for handling CPU, memory, timers and interrupts.

The hypervisor is started by the boot loader and, once it is loaded, it starts the privileged domain Dom0 (short for "domain 0") containing the drivers of the system devices. Once Dom0 has started, one or more user domains, referred to as DomU, can be started and controlled in the Dom0. Xen supports both para-virtualized (PV) and fully hardware virtualized (HVM) user domains.

This tutorial will explain how to build and run ERIKA3 as Xen hardware-virtualized DomU on the x86-64 platform.

Xen Installation on x86-64 platform

The section contains the instruction to install XEN (version RELEASE4.11) on Ubuntu 16.04.

Build Dependencies

Install the following packages:

 # apt-get update && apt-get dist-upgrade #update/upgrade the kernel
 # apt-get install build-essential
 # apt-get install bcc bin86 gawk bridge-utils iproute 
 # apt-get install libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif
 # apt-get install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended
 # apt-get install pciutils-dev mercurial
 # apt-get install make gcc libc6-dev zlib1g-dev python python-dev python-twisted 
 # apt-get install libncurses5-dev patch libsdl-dev libjpeg-dev
 # apt-get install libvncserver-dev
 # apt-get install iasl libbz2-dev e2fslibs-dev git-core uuid-dev 
 # apt-get install ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev
 # apt-get install gettext libpixman-1-dev libaio-dev markdown pandoc
 # apt-get install libc6-dev-i386
 # apt-get install lzma lzma-dev liblzma-dev #for rombios
 # apt-get install libsystemd-dev 

Configure and build

Clone the Xen repository and switch to the chosen version (i.e., RELEASE4.11):

 $ git clone https://github.com/xen-project/xen.git
 $ git checkout RELEASE-4.11.0 -b RELEASE-4.11.0

Configure and build:

 $ cd xen
 $ ./configure -enable-systemd
 $ make dist

Install

The following operations has to be performed with root privilege.

Install:

 # make install

Reload dynamic libraries:

 # /sbin/ldconfig

Enable systemd Xen services:

 # systemctl enable xen-qemu-dom0-disk-backend.service
 # systemctl enable xen-init-dom0.service
 # systemctl enable xenconsoled.service
 # systemctl enable xenstored.service
 # systemctl enable xendomains.service

Note that in case of error for enabling xendomains service, remove /etc/init.d/xendomains.

Update grub and reboot:

 # udate-grub
 #reboot

ERIKA3 Xen configuration

This section defines the configuration parameters that are used for running ERIKA3 as Xen DomU. Note that according to Xen configuration syntax, comment lines starts with the '#' symbol.

The configuration file, referred to as erika3-xen.cfg, has to include the following basic parameters, such as domain name, memory requirements, virtualization mode, serial interfaces definition and so on:

- Name of domain (must be unique):

 name = "erika"

- Memory allocation (MB) for ERIKA3 (should be at least 2M):

 memory = 2048
 maxmem = 2048

- Boot disk (where $ERIKA3_PATH is the path of the ERIKA3 bare-metal image):

 disk 	= [ 'file:$ERIKA3_PATH,hdc:cdrom,r' ]
 boot 	= "c"

- Virtualization mode:

 type = "hvm"

- Enable Xen console:

 serial = "pty"

- Domain exit behavior settings:

 on_poweroff = "preserve"
 on_reboot = "destroy"
 on_crash = "destroy"

A configuration file can be downloaded here.[TODO]

Build ERIKA3 for Xen

Building ERIKA3 for Xen requires building an ERIKA3 image for bare-metal as described here.

The path of the generated ERIKA3 bare-metal image has to be included into the Xen configuration file. More in detail, such path, referred to as $ERIKA3_PATH, has to used to configure the boot disk configuration parameter specified in the previous section.

Run ERIKA3 as Xen HVM DomU

The Xen command to start the ERIKA3 DomU is the following:

 # xl create erika3-xen.cfg

where erika3-xen.cfg is the ERIKA3 configuration file. In order to connect to console directly when starting the ERIKA3 domU, launch the following command:

 # xl create -c erika3-xen.cfg

In case of console connection to the ERIKA3 DomU that is already running, the Xen command is the following:

 # xl console erika

where 'erika' is the domain name defined in the configuration file.

In order to poweroff the ERIKA3 DomU, the Xen command is the following:

 # xl destroy erika

where 'erika' is the domain name defined in the configuration file.

ERIKA3 CPUs configuration

In order to tune ERIKA3 DomU, Xen configuration could include CPUs affinity in order to pin ERIKA on a given CPU. Note that current implementation of ERIKA3 on x86-64 is single-core. Thus, the number of virtual CPUs is 1. Such virtual CPU can be pinned to a given physical CPU. The ERIKA3 configuration file has to include the following items in order to set the CPUs affinity:

- Number of virtual CPUs to use (default is 1):

 vcpus	= 1

- List of which CPUs this domain is allowed to use (in the example, it is the CPU with identifier 3):

 cpus = ['3']

Xen hypervisor can divide the physical cpus into distinct groups, called cpupools, so that each pool can have its entirely separate scheduler (e.g., null-scheduler). The "default pool" is named Pool-0 and physical CPUs can be removed from one cpupool and added to another. It follows the Xen basic operations to setup and modify a cpupool:

Show the cpupool list:

 # xl cpupool-list

Show the cpupool list with the corresponding physical CPUs:

 # xl cpupool-list -c

Setup a new empty pool using, for example, the null scheduler:

 # xl cpupool-create name="NullPool" sched="null"

Move a physical CPU from default pool to the new pool:

 # xl cpupool-cpu-remove Pool-0 3
 # xl cpupool-cpu-add NullPool 3

Domains are assigned to pools on creation, and can be moved from one pool to another. In order to assign ERIKA3 to a given cpupool at DomU creation, the Xen configuration has to include the following item specifying the cpupool name:

- CpuPool assignment (in the example, the Cpupool has name 'NullPool'):

 pool="NullPool"

ERIKA3 Xen Passthrough

[TODO]

In order to give control of physical devices to ERIKA3 domU, Xen configuration should include the configuration for PCI passthrough. Note that the chosen PCI device has to be "assignable".

For example, the Xen configuration to allow the PCI passthrough for the Ethernet device with '00:1f.6' as BDF:

 ## ETH1 Passthrough
 pci_permissive=1
 pci = ['00:1f.6']