Kernel Vulnerability

As announced, we are now beginning maintenance. First, we will select a Canary node, and then apply the patch to the rest of the fleet in parallel. We will start by updating the Skylink EGH01 site.

We are now also starting to patch the NTT FRA1 cluster at the same time.

All systems in the Skylink have been fully updated, and all KVM servers are back to operating as usual.

All RYZEN-9 nodes have been updated.

Maintenance Complete: Security Update for All Virtualization Hosts (Linux Kernel, “PwnEscape”)

Summary: During the night of October 15–16, 2026, we applied a critical security update to the Linux kernel on all virtualization hosts at our Frankfurt and Eygelshoven locations. This required each host to be restarted once. Each VM was unavailable only while its host was rebooting, typically for 5 to 15 minutes. All VMs that were running before the maintenance are up and running again. No data was lost during this process.

What was the issue? Security researchers disclosed a vulnerability in the Linux kernel’s network stack, known as “PwnEscape.” A virtual machine could use manipulated checksum information from its virtio network card to trick the host system’s kernel into overwriting packet headers that had already been processed. In the researchers’ lab environment, this allowed them to read the host’s memory and ultimately gain access to the host from within the VM. KVM-based virtualization platforms in general are affected. The flaw had been present in the Linux kernel for many years.

The fix was incorporated into the official Linux kernel on October 14, 2026. Proxmox ships it with kernel 7.0.14-24, which we tested before deployment. Since it is not possible to replace the kernel while the system is running, the hosts had to be rebooted.

What did we do?

We installed the patched kernel on all virtualization hosts and set it as the default. First, we updated a single host as a test run and monitored it, then updated the remaining hosts in waves. We migrated internal services in advance to hosts that had already been updated, without any downtime. We restarted the hosts of our distributed storage system (Ceph) one by one. As a result, the storage remained available and fully redundant at all times. After each restart, we automatically verified that the new kernel was running and that all previously running VMs had restarted.

Timeline (CEST)

Time Period, Region, Downtime per VM (approx.) Oct. 15, 11:15 PM–11:30 PM Eygelshoven, test run on one host 10–15 min Oct. 15, 11:40 p.m.–Oct. 16, 12:30 a.m. Eygelshoven, all remaining hosts in three waves 10–15 min Oct. 16, 12:15 a.m.–1:20 a.m. Frankfurt, compute hosts in several waves 5–15 min Oct. 16, 1:15 a.m.–2:45 a.m. Frankfurt, storage hosts one after another 10–20 min Oct. 16, 2:45 a.m. Maintenance complete

What do you need to do?

Generally, nothing. Your VM is up and running again. We intentionally did not start any VMs that were powered off prior to maintenance. If your VM is unavailable, please restart it once via the customer panel or contact our support team. If you are running virtual machines within your VM (e.g., KVM or Proxmox), please also update the kernel there as soon as your distribution releases the fix.

Thank you for your understanding.