Planning to upgrade your Greenhost VPS? Great! Keeping a VPS's operating system up to date is important to make sure it keeps running smoothly and safely. OS updates bring new features, as well as security improvements, bug fixes, and resource-saving optimisations.
The latest versions of Debian and Ubuntu made a significant change in how some system files are stored. In some situations, you will need to do a quick and simple intervention before upgrading to Debian 12 Bookworm, releasing today (June 6th, 2023), or Ubuntu 22.04 Jammy Jellyfish, as the upgrade will not work out of the box on a Greenhost VPS.
This article will tell you how to check if you need to do the intervention, and will guide you through the required steps.
A bit of context
History of the Linux system files
Historically, Linux distributions placed some system files in the
/lib directories, while other system files were placed in the
/usr directory, in similarly named sub-directories
The first three directories would usually be mounted on a smaller but faster disk, as they contain information necessary to mount the slower but bigger
With disks becoming bigger, faster and cheaper, this is not something that is done any more: all those directories are now typically on the same disk 1.
For that reason, the separation between a directory at the root of the disk (
/) and its equivalent in
/usr is not needed anymore.
To make things easier and reduce confusion, distributions started to merge directories in what is called the merged
/usr directories scheme.
From Debian 10 2 and Ubuntu 19.04 3, new installations are made with this scheme,
while upgrades from previous versions would keep the scheme they already had.
However, starting from Debian 12 and Ubuntu 22.04, the new scheme is now enforced during the upgrade process,
which now requires installing and using the
usrmerge package, whose purpose is to move the files around according to the new scheme.
How Greenhost VPSs are started
In order for our customers to get the best of their VPSs and ensure regular security updates,
these are by default started with our own build of the Linux kernel.
For this to work, we are mounting an in-memory file system containing the kernel modules on
This approach works with both the old and the new directory scheme.
On the new scheme there will be a symbolic link from
making the kernel modules mounted on
However, upgrading a running VPS from the old scheme to the new one is more complicated:
usrmerge does not support this way of loading kernel modules, which means an upgrade will fail with the following message:
1 2 3 4 mv: cannot move '/lib/modules' to '/usr/lib/modules': Device or resource busy FATAL ERROR: mv --no-clobber /lib/modules /usr/lib/modules: rc=1
To allow the tool to run,
/lib/modules needs to be unmounted first.
To unmount it safely, it first has to be freed from handles, so we need to stop the
usrmerge can be installed without issue.
However, if we were to only do that,
the VPS would lose access to the kernel modules until the next reboot.
So we need to move the mount containing the modules to a temporary location,
and then move it to
/usr/lib/modules after the upgrade.
Make sure your VPS is ready for the upgrade
You can find out whether you need to take action on your VPS
by logging in and running the command:
If you get
usr/lib as output, you don't have to do anything for this VPS.
If instead you get no output, please keep reading.
First you need to make sure that your VPS is up-to-date: run the command
apt-cache policy usrmerge.
If you get the message
N: Unable to locate package usrmerge,
you have to upgrade your operating system to Debian 10 Buster or Ubuntu 16.04 Xenial Xerus.
Please refer to your distribution's documentation on how to do that.
Then, acquire root rights and run the following commands in succession:
1 2 3 4 5 6 7 8 9 10 11 12 13 mkdir /tmp/temp_modules mount --bind /lib/modules /tmp/temp_modules systemctl stop systemd-udevd # You can safely ignore the warnings about the service being still able to be activated by sockets. umount /lib/modules apt install usrmerge # Answer Yes if prompted to convert to the merged /usr directories scheme. mount --bind /tmp/temp_modules /usr/lib/modules systemctl start systemd-udevd umount /tmp/temp_modules rmdir /tmp/temp_modules
If you didn't get any error messages, you are now successfully able to upgrade to Debian 12 Bookworm or Ubuntu 22.04 Jammy Jellyfish!
Otherwise, please contact our support, we will be happy to help you.
This is true for official installation media, but whether the scheme was enabled or not depends on the version of
debootrapused for the install. On our infrastructure, newer installations of Debian 9 came with this scheme. https://wiki.debian.org/UsrMerge ↩