Time line update severe CPU Intel bugs

As you may have read or heard, the Register announced several severe security problems with Intel processors, which are affecting nearly all computers with modern Intel processors (link below). This means an emergency maintenance window is scheduled from tomorrow onwards (January 4th), which can lead to downtime for your website or Virtual Private Server(s) outside the normal maintenance window. Please see below for more details.

Update 19th of January 2018 15:22

All our systems are patched an updated. From now on we will work on the long term consequences of the CPU bugs.

Update 12th of January 2018 17:30

Excluding some exceptions all our customer systems has been updated. Next week we will continue updating our internal systems. During these updates small interruptions to our services could occur. Therefore the updates will take place outside office hours. We will keep you posted.

Update 9th January 2018 18:18

We have sent our VPS users the following email:

We thank you for hosting your services on a VPS at Greenhost.

Further to our email of last week regarding the Intel bugs Meltdown and Spectre we would like to inform our customers who run a Debian or Ubuntu stock kernel.

Due to the Meltdown patches and the way we host your VPS after the Meltdown incident, stock kernels are no longer compatible with our platform.

Because of this, it is necessary to reboot your VPS at your own behalf until Friday. This can be done by using the 'shutdown' and 'start' command from the service centre (a normal reboot is not sufficient).

On Friday, we will force the reboot if you have not executed the reboot yourself by then.

For our customers who use Docker together with the aufs storage driver, we're phasing aufs out. However, we will include overlayfs as an option, which is recommended by Docker.

If you have any questions or concerns please do not hesitate to contact us.

Update 5th January 2018 17:00

We made a diverse set of adjustments to cope with the CPU bugs that are currently plaguing the world. The bugs, called Meltdown and Spectre, are not necessarily fixable. The only permanent solution will probably be replacing the CPU hardware, currently there are no CPU's available that are not affected by this bug. If it is not fixable, what did we do?

Meltdown: This bug is relatively easy to mitigate. We patched our Linux kernels to the latest version. Because we are using Xen virtualisation on our servers the expected performance impact of this patch is limited for the end users. To offer higher security against Meltdown we also decided to switch to a different type of virtualisation inside the Xen hypervisor. We are currently busy with this migration. After this migration (theoretically) all systems are protected on all levels against Meltdown.

Spectre: This is a harder problem. Our CPU's are equipped with a extra security layer (SMEP), this makes usage of the Spectre bug harder, but not impossible. Currently research is being done by multiple suppliers to see if it is possible to protect against this bug in software.

The risks: As of now there is no known method for actually exploiting these bugs in order to execute malicious code on systems outside those you're allowed access to. It is however, theoretically possible, to get access to information from adjacent systems (usually other customers). Right now it is not clear how complex it would be to achieve this and how fast it would work. With the current knowledge it does not seem like a simple exploit. For a better solution we will have to wait on further developments in this field.

Update: 4th of January 2018 19:10

Shared hosting has been updated by now. It could be that we have to reboot the Shared hosting server one more time for performance reasons. Concerning the VPS: for the necessary update we need to update the VPS kernel to 4.14.11 or higher. This means that the VPSes needs to be rebooted once. We expect that most of the VPS will be updated tomorrow. Furthermore the first tests show us that the performance reduction of the update for both Shared hosting and VPS is limited.

Update: 4th of January 2018 17:45

At this moment we're finalizing patching our hosting platform. This means that the bugs that can be fixed, are fixed as much as possible. We looked into using another way to virtualise your system in the Cloud to isolate the bug as much as possible. This is now in a test phase and VPSes will be restarted in the following day(s).

Update: 4th of January 2018 14:55

Our team has been busy searching for ways to secure our hosting nodes even more. In the meantime we received updates from a part of our vendors. We still haven't received all the necessary information but where we could , we have been updating our systems to prevent the bug. Furthermore we're investigating what is the best way to secure the rest of our platform against this bug, where possible.

Update: 3rd of January 2018 18:47

As you may have read or heard, yesterday the Register announced several severe security problems with Intel processors, which are affecting nearly all computers with modern Intel processors (link below). This means an emergency maintenance window is scheduled from tomorrow onwards (January 4th), which can lead to downtime for your website or Virtual Private Server(s) outside the normal maintenance window. Please see below for more details.

Like most hosting providers, Greenhost uses Intel processors for our hosting and cloud infrastructure and this means our servers are also vulnerable for these security issues. The current security issue is severe and cannot be isolated with other methods. This requires us to act as soon as possible to protect against possible exploitation of this bug by attackers. This problem is not just limited to the Greenhost infrastructure, several other (major) cloud providers are scheduling forced reboots of their platform in the coming week(s) to solve this issue when a fix becomes available.

The exact severity of the flaw has not yet been publicly disclosed, but it seems that the different operating systems view it as a serious problem that apparently cannot be patched with a small update. Our customers can be ensured that we are closely monitoring the situation. Where possible we already fixed some parts of our infrastructure, while we are awaiting for more information from our vendors.

As soon we received a fix from our vendors, we will update our hosting and cloud infrastructure. That might mean a temporary loss of service, but of course we will try to keep downtime to a minimum.

Due to the nature of the fix, you might experience performance degradation after the fix is deployed. Unfortunately at this point we can not estimate how big the impact is and we will evaluate if extra resources needs to be added after the fix.

If you want to know more about the technical details of this severe security problem, we like to reference you to the article of the The Register (link below)

We will push out any updates on this on the Greenhost website or Twitter to keep our customers informed.

The Greenhost Team