The Myth of Invisible Virtualization: What Fly.io’s Fleet Patch Reminds Us About Hypervisors

On November 6, 2026, the edge-cloud platform Fly.io published an incident post-mortem with a refreshingly candid title: “Well, That Wasn’t Fun: October 16 ‘26”.

The report detailed what happened three weeks earlier when an emergency Linux kernel zero-day landed on their desks. Fly runs bare-metal physical servers across dozens of colocation facilities worldwide, hosting customer workloads as lightweight microVMs on top of standard Linux hosts. They could not paper over this bug with a rolling configuration reload or a managed-cloud toggle.

In a single afternoon, they had to patch and reboot every physical hypervisor in their global fleet.

For a platform whose promise is effortless global deployment—type fly deploy and your code spins up in Tokyo, Amsterdam, and Virginia in seconds—watching bare-metal motherboards power-cycle across twenty time zones was a stark reminder of an uncomfortable reality:

Software cannot abstract away the physical machine.

Underneath every serverless worker and ephemeral microVM sits a physical motherboard with fans, memory DIMMsDIMMDual In-Line Memory ModuleHardwareA modular printed circuit board populated with DRAM integrated circuits that installs into server and desktop motherboard memory channels across a 64-bit data bus., a power supply, and a monolithic Linux host kernel. When ring 0 bleeds, the marketing illusion of weightless compute vanishes.

---

The Illusion of Live Migration

In enterprise IT, virtualization vendors spent twenty years conditioning architects to believe that physical host reboots should never disrupt running workloads. That promise had a name: live migration.

The pitch sounds magical: an active virtual machine, complete with open TCPTCPTransmission Control ProtocolNetworkingA connection-oriented transport layer protocol providing reliable, ordered, congestion-controlled, and error-checked delivery of byte streams over IP networks. connections and gigabytes of RAMRAMRandom Access MemoryHardwareHigh-speed volatile physical memory providing near-instantaneous read and write access for applications and operating system processes running on the CPU., floats gracefully from Server A to Server B with zero downtime.

For enterprise clusters with deep pockets, shared SANSANStorage Area NetworkStorageA dedicated high-speed network that provides block-level storage access to hypervisor clusters and enterprise servers via Fibre Channel, iSCSI, or NVMe-oF. arrays, and homogeneous CPUCPUCentral Processing UnitHardwareThe primary processor in a computer system that fetches, decodes, and executes machine instructions, arithmetic logic, and I/O control. families, live migration worked reasonably well for planned maintenance—swapping a failing power supply, upgrading host RAM, or flashing motherboard firmware on one node while the cluster hummed along. In the face of an emergency host kernel zero-day, however, live migration is a conceptual dead end.

First, modern edge platforms pair microVMs with fast local NVMeNVMeNon-Volatile Memory ExpressStorageAn open logical-device interface specification engineered specifically for solid-state storage over PCIe buses, maximizing parallel I/O queues. drives to deliver microsecond I/O that centralized SANs could never match. Copying tens of gigabytes of raw disk state per VMVMVirtual MachineVirtualizationA software-emulated or hardware-virtualized computer system that runs an isolated guest operating system and application stack atop a physical hypervisor host. across top-of-rack switches takes hours, completely defeating the purpose.

More fundamentally: where do you migrate to?

When a critical vulnerability strikes the host Linux kernel, every hypervisor in the fleet is equally exposed. You cannot migrate workloads from one compromised machine to another and call it security. To make the destination host safe, you have to patch and reboot it first.

Attempting to juggle running workloads across unpatched hosts during a critical security event only drags an acute maintenance window into an agonizing, multi-day circus. A cold reboot is not an engineering failure; it is the only clean remediation.

---

The Litmus Test of an Infrastructure Engineer Is reboot

Years ago, my homelab ran a cluster of three 4U single-socket servers tied together with quad-port Intel 10GbE NICsNICNetwork Interface CardHardwareA hardware expansion card or onboard controller that connects a server or workstation to an Ethernet or fiber-optic computer network. and VMware vSANvSANVMware Virtual SANStorageVMware software-defined enterprise storage that aggregates server-attached flash and magnetic storage across cluster nodes into a shared resilient datastore., before I eventually consolidated down to a single Proxmox node to cut idle rack power.

Anyone who has operated a vSAN cluster recognizes the physical reality of that iron: the sharp whine of chassis fans spinning up, the heat rolling off the processor socket and banks of ECCECCError-Correcting Code MemoryHardwareServer-grade system memory (RAM) that detects and corrects single-bit memory corruption on the fly, preventing silent data corruption in filesystems like ZFS. RAM, amber drive LEDs chattering as storage pools balance, and the glacial wait while server UEFIUEFIUnified Extensible Firmware InterfaceHardwareThe modern modular firmware specification that replaces legacy 16-bit BIOS, supporting GPT boot drives larger than 2 TB, Secure Boot cryptographic verification, and pre-OS networking. firmware trains memory before the bootloader even displays on the IPMIIPMIIntelligent Platform Management InterfaceHardwareA standardized autonomous hardware management specification that lets administrators monitor server health and reboot hosts out-of-band independent of the CPU or OS. console.

Rebooting a vSAN node was a choreographed, high-tension ritual:

# Evacuating storage before maintenance
esxcli system maintenanceMode set --enable true --vsan-mode ensureObjectAccessibility

You would wait for data to evacuate across the 10GbE links, watch surviving nodes absorb rebuild traffic, and pray an adjacent drive wouldn’t throw an I/O fault while the first host was offline. The entire enterprise stack was engineered to avoid a reboot at all costs—which perversely turned every inevitable restart into an operational emergency.

Later, moving to native KVMKVMKernel-based Virtual MachineVirtualizationA full virtualization module built into the Linux kernel that turns the host operating system kernel into a hardware-assisted Type-1 hypervisor. on Debian stripped away the enterprise mysticism:

# Checking physical virtualization primitives directly on Linux
$ ls -lh /dev/kvm
crw-rw---- 1 root kvm 10, 232 Nov 13 10:15 /dev/kvm

A virtual machine is just an ordinary Linux process with a PIDPIDProcess IdentifierArchitectureA unique numerical handle assigned by an operating system kernel to each active running process for scheduling, signal delivery, and resource tracking.. A virtual disk is a block device. A network interface is a bridge. And that brings you face-to-face with an essential truth:

The ultimate litmus test of an infrastructure engineer is typing reboot.

If typing sudo reboot into a hypervisor shell induces panic, your infrastructure is not resilient. It is fragile state pretending to be high availability.

Hypervisors are mortal machines. Power supplies trip, memory modules throw uncorrectable ECC errors, and operating system kernels require zero-day patches. If your architecture relies on hypervisors running uninterrupted for five hundred days without a restart, you haven’t engineered resilience—you have just been lucky.

---

Designing for Hypervisor Mortality

The reason Fly.io rebooted its entire global fleet in an afternoon without catastrophic downtime comes down to modern systems design. Resilience does not mean building an indestructible hypervisor; it means building workloads that treat host mortality as a routine, non-critical event.

1. Sub-second microVM boot times: Legacy virtual machines took minutes to initialize SCSISCSISmall Computer System InterfaceStorageA foundational set of ANSI command protocols and standards for physically connecting and transferring block data between computers and enterprise storage peripherals. drivers, ACPIACPIAdvanced Configuration and Power InterfaceHardwareAn open hardware firmware specification that enables the operating system kernel to discover motherboard devices and manage CPU power states (C-states/P-states), thermal zones, and sleep modes. tables, and enterprise init scripts. By stripping emulated hardware down to bare essentials, modern KVM and Firecracker microVMs boot in 150 to 300 milliseconds. An orchestrator can spin up replacement compute on an adjacent node before an edge proxy’s connection timeout has even elapsed.
2. Decoupling compute from durable state: Local NVMe should only ever be high-speed scratch space for ephemeral caches and temporary buffers. Real state belongs in durable, replicated tiers—object storage (S3S3Simple Storage ServiceCloudAmazon Web Services scalable object storage offering 99.999999999% (11 9s) of data durability with fine-grained IAM policies and versioning.) or replicated databases (like LiteFS or Raft WAL streams) distributed across independent physical machines. If a compute node carries zero irreplaceable state, power-cycling it carries zero emotional weight.
3. Disciplined SIGTERM handlers: A host reboot only drops traffic when software treats shutdown as an emergency. When a hypervisor drains traffic at the edge proxy, microVMs receive SIGTERM, complete in-flight HTTPHTTPHypertext Transfer ProtocolNetworkingThe foundational stateless request-response application protocol of the World Wide Web used by browsers and APIs to fetch hypermedia documents and exchange data. requests, flush pending database transactions, and exit cleanly within seconds before the host reboots.

---

Respecting the Physical Machine

Fly.io’s incident report was a masterclass in operational honesty.

It stripped away the marketing veneer and reminded us of the physical reality underneath the modern cloud. You can wrap your systems in declarative YAMLYAMLYAML Ain't Markup LanguageDevOpsA human-friendly, indentation-sensitive data serialization language widely used for configuration files, cloud manifests, and CI/CD pipelines., invoke them through slick CLIsCLICommand-Line InterfaceDevOpsA text-based user interface mechanism that processes terminal commands to interact directly with programs, operating systems, and cloud APIs., and hide them behind global AnycastAnycastIP Anycast RoutingNetworkingA network addressing and routing methodology in which a single destination IP address is announced from multiple geographic locations via BGP, routing clients to the topologically nearest node. routing. But at the bottom of the stack, your code runs on silicon seated in sockets, powered by copper, and governed by an operating system kernel.

True engineering maturity is not about searching for an immortal hypervisor that never requires maintenance. It is about engineering humility:

• Expect the host to reboot.
• Make guest boot times instantaneous.
• Decouple your state from your compute.
• Handle shutdown signals gracefully.
• And above all, never be afraid to type reboot.