Abdelhalim El Jarzini
Professional Platform

Talos Kubernetes on KVM: a cluster runbook verified three times

A parameterized runbook that takes a bare Rocky Linux host to a healthy three-node Talos cluster, rebuilt from scratch until every silent failure was written down.

My role
Built the clusters, debugged the failures and wrote the runbook.
Context
Internal platform work at ITGS · 2026
Status
Runbook verified end to end three times
Stack
  • Talos Linux
  • Kubernetes
  • talosctl
  • KVM / libvirt
  • Rocky Linux 9
  • Proxmox VE
KVM host · Rocky Linux 9 + libvirt
Bridge: Path A create it and NAT it, or Path B use an existing bridge behind an external router
control planeTalos VM · static IP
worker 1Talos VM · static IP
worker 2Talos VM · static IP
Second physical host · scale-out
worker 3joins the existing cluster

Reachability to the cluster is checked before anything is provisioned

Management machinetalosctl · kubectl

Generates configs, applies them in maintenance mode, keeps the kubeconfig

  1. Prepare the host libvirt/KVM, Talos ISO, firewall ports · Talos and Kubernetes versions pinned, never “latest”
  2. Network and bootstrap DHCP DHCP only until each node has its static config
  3. Provision the VMs 1 control plane + 2 workers
  4. Generate and patch each node’s config real interface name read from the node · hostname set the supported way · heredoc patches, no interactive editor
  5. Apply, then bootstrap etcd talosctl apply-config in maintenance mode · disk-first boot order
  6. Verify health · 3/3 nodes Ready · cross-node pod traffic · service DNS · test deployment · power cycle
DiagramThe build order the runbook follows, with versions pinned and every value set once as a variable. Sanitized: bridge names, subnets and host names removed.

The problem

A cluster that works once is not the goal. The second and third clusters have to come out the same way. Talos Linux has no SSH and no shell: every node is configured through its API from a machine config. That makes it reproducible, but a wrong value often fails quietly instead of loudly.

My role

I built an earlier Talos cluster on Proxmox, then wrote a runbook for plain KVM hosts and rebuilt from it three times, folding each new failure back into the document.

What the runbook covers

  • Values set once. Bridge, subnet, gateway, DNS, node addresses and hostnames, and the exact Talos and Kubernetes versions (Talos 1.13, Kubernetes 1.36) are shell variables, so the same commands build a second or third cluster.
  • Two network paths. Path A creates the bridge and NATs it. Path B uses an existing bridge behind an external router.
  • Bootstrap-only DHCP, then a static address per node.
  • Per-node patches written as heredocs, so the exact change lives in the document and the shell history, not in an editor buffer.
  • Bootstrap, health check, kubeconfig and a test deployment.
  • Scale-out to a second physical host, starting with a reachability check from the new host to the cluster.

Five failures that do not announce themselves

The interface is not eth0

On the VMs the NIC was ens18. With eth0 in the config, the static address applied but the default route never did, so the installer image pull hung with no clear error. The runbook reads the real interface name from the node in maintenance mode before writing its config.

A hostname patch that breaks the config

Setting the hostname through a merged patch produced a second YAML document (HostnameConfig) with an invalid block. The runbook sets the hostname the supported way and keeps each machine config valid.

Overlapping subnets hang etcd forever

If the VM subnet falls inside the default pod (10.244.0.0/16) or service (10.96.0.0/12) range, etcd bootstrap hangs with no error message: Talos leaves overlapping node addresses out of etcd’s advertise list. The runbook checks the subnet before anything is generated.

The installer ISO is still attached

Resizing a VM requires a real power cycle. With the CD-ROM still first in the boot order, the node silently boots the installer again instead of its disk. Disk-first boot order is now part of the install step.

Config injection that is ignored

On the Proxmox build, passing the machine config through QEMU fw_cfg only works on the qemu platform, not metal, and talos.config=file:// is not a supported scheme. Configs are applied over the API in maintenance mode with talosctl apply-config --insecure.

Verification

  • Three end-to-end runs between 28 and 30 July 2026. The first two ran on the lab host, once while debugging the original build and once from a clean slate following the document as written. The third ran on a second host’s existing bridge (Path B).
  • Every run reached 3/3 nodes Ready with correct hostnames, cross-node pod networking and working service DNS, plus a test deployment.
  • The third run survived a real power cycle without losing cluster state.

Status

The runbook is verified three times. Access for multiple users is a separate, proposed design: Kubernetes cluster access.