Get your Greenhost VPS prepared for upgrading to Debian 12 / Ubuntu 22.04

Planning to upgrade your Greenhost VPS? Great! Keeping a VPS's operating system up to date is important to make sure it keeps running smoothly and safely. OS updates bring new features, as well as security improvements, bug fixes, and resource-saving optimisations.

The latest versions of Debian and Ubuntu made a significant change in how some system files are stored. In some situations, you will need to do a quick and simple intervention before upgrading to Debian 12 Bookworm, releasing today (June 6th, 2023), or Ubuntu 22.04 Jammy Jellyfish, as the upgrade will not work out of the box on a Greenhost VPS.

This article will tell you how to check if you need to do the intervention, and will guide you through the required steps.

A bit of context

History of the Linux system files

Historically, Linux distributions placed some system files in the /bin, /sbin and /lib directories, while other system files were placed in the /usr directory, in similarly named sub-directories /usr/bin, /usr/sbin or /usr/lib. The first three directories would usually be mounted on a smaller but faster disk, as they contain information necessary to mount the slower but bigger /usr partition. With disks becoming bigger, faster and cheaper, this is not something that is done any more: all those directories are now typically on the same disk ¹.

For that reason, the separation between a directory at the root of the disk (/) and its equivalent in /usr is not needed anymore. To make things easier and reduce confusion, distributions started to merge directories in what is called the merged /usr directories scheme. From Debian 10 ² and Ubuntu 19.04 ³, new installations are made with this scheme, while upgrades from previous versions would keep the scheme they already had. However, starting from Debian 12 and Ubuntu 22.04, the new scheme is now enforced during the upgrade process, which now requires installing and using the usrmerge package, whose purpose is to move the files around according to the new scheme.

How Greenhost VPSs are started

In order for our customers to get the best of their VPSs and ensure regular security updates, these are by default started with our own build of the Linux kernel. For this to work, we are mounting an in-memory file system containing the kernel modules on /lib/modules.

This approach works with both the old and the new directory scheme. On the new scheme there will be a symbolic link from /lib to /usr/lib, making the kernel modules mounted on /usr/lib/modules instead.

However, upgrading a running VPS from the old scheme to the new one is more complicated: usrmerge does not support this way of loading kernel modules, which means an upgrade will fail with the following message:

1
2
3
4
mv: cannot move '/lib/modules' to '/usr/lib/modules': Device or resource busy

FATAL ERROR:
mv --no-clobber /lib/modules /usr/lib/modules: rc=1

To allow the tool to run, /lib/modules needs to be unmounted first. To unmount it safely, it first has to be freed from handles, so we need to stop the systemd-udevd service. Then, usrmerge can be installed without issue.

However, if we were to only do that, the VPS would lose access to the kernel modules until the next reboot. So we need to move the mount containing the modules to a temporary location, and then move it to /usr/lib/modules after the upgrade.

Make sure your VPS is ready for the upgrade

You can find out whether you need to take action on your VPS by logging in and running the command: readlink /lib. If you get usr/lib as output, you don't have to do anything for this VPS. If instead you get no output, please keep reading.

First you need to make sure that your VPS is up-to-date: run the command apt-cache policy usrmerge. If you get the message N: Unable to locate package usrmerge, you have to upgrade your operating system to Debian 10 Buster or Ubuntu 16.04 Xenial Xerus. Please refer to your distribution's documentation on how to do that.

Then, acquire root rights and run the following commands in succession:

1
2
3
4
5
6
7
8
9
10
11
12
13
mkdir /tmp/temp_modules
mount --bind /lib/modules /tmp/temp_modules

systemctl stop systemd-udevd
# You can safely ignore the warnings about the service being still able to be activated by sockets.
umount /lib/modules
apt install usrmerge
# Answer Yes if prompted to convert to the merged /usr directories scheme.
mount --bind /tmp/temp_modules /usr/lib/modules
systemctl start systemd-udevd

umount /tmp/temp_modules
rmdir /tmp/temp_modules

If you didn't get any error messages, you are now successfully able to upgrade to Debian 12 Bookworm or Ubuntu 22.04 Jammy Jellyfish!

Otherwise, please contact our support, we will be happy to help you.