← notes on building k7d

Notes on building k7d · Part 3

The 45-second fork: qemu, Longhorn, and a bind-mount keystone

The disk-only prequel to k7d. It worked, it shipped, and its ceiling is the reason the rest of this series exists.

August 2026 · ~9 min read

Before k7d there was k7: self-hosted VM sandboxes on Kubernetes, built for AI agents running arbitrary code. This is the story of one piece of its plumbing — forking a running sandbox that has Docker inside, where the fork boots with the parent's Docker image cache already there — told mostly as I told it at the time. The docker-in-VM part itself was easy: a sidecar, done quickly. The hard part was the storage layer underneath, and the order in which I figured things out went roughly like this.

The journey (boring on paper, painful in practice)

I built k7 originally on top of k3s + Kata + Firecracker + devmapper-snapshotter (LVM thin pool). Reason was simple: tiny attack surface, and the devmapper thin pool gives you cheap copy-on-write on a single node. Perfect for spawning dozens of micro-VMs per node without exploding disk usage.

That shipped, hit #1 on Show HN, all good.

Then people came back asking for pause / resume / fork. Not just "scale the pod to zero" — actual disk snapshots. I want to pause this sandbox, come back tomorrow, resume exactly where I left off. And I want to clone it into N branches and run them in parallel.

Ok, so: snapshots and clones. Technically possible on devmapper LVM (lvcreate --snapshot). But as soon as I started thinking about HA / multi-node — fork onto a different node, or replicate the snapshot somewhere safe — devmapper started being a problem. It's a per-node block-device thing. Cross-node replication of LVM thin snapshots is not something you want to be the person inventing.

So I went looking for a CSI-native distributed block store that already solved this. Longhorn was the obvious fit: per-PVC replicated volumes, native VolumeSnapshot + dataSource clone, runs on k3s, no external deps.

But Longhorn does not play nice with Firecracker. Firecracker's microVM expects a particular block-device layout, and Kata's Firecracker integration assumes specific storage paths that Longhorn's block-mode CSI doesn't expose cleanly. I tried various routes and concluded: not worth the fight. I'd be reverse-engineering the Kata Firecracker shim.

So I switched the heavy backend to qemu instead of Firecracker. Kata-qemu happily mounts a Longhorn PVC as the VM disk via virtio-blk. k7 kept firecracker-devmapper for ephemeral workloads (super fast boot, no persistence) and added qemu-longhorn for stateful ones (slower boot, persistent disk, snapshot + fork).

Docker inside, without Docker-in-Docker

The docker-in-VM pattern I'd already shipped: a sidecar container runs dockerd, the user container runs your image with the docker CLI, and they share /var/run/docker.sock via an emptyDir. Both containers live inside the same Kata VM, so the VM is the security boundary, not the container — the sidecar doesn't need --privileged on the host, it just needs root inside its own VM, which is fine. That's the whole point of running on Kata in the first place.

People asked "isn't this just Docker-in-Docker with extra steps?" Partly, and it's worth being precise about which problem the sidecar does and doesn't solve. It doesn't reduce nesting: dockerd still runs inside a container, and the containers the user spawns still live inside the sidecar container. What it changes is the separation of concerns. The daemon moves out of the user's container, so the user container stays unprivileged and runs a stock image, and a docker run creates a sibling of the user container rather than a child inside it. The bigger difference is where all of this runs: both containers sit inside the same Kata VM, so the VM is the security boundary. The sidecar needs root only inside its own VM — not --privileged on the host — and the host kernel never sees any of it; it sees one Kata-managed qemu process.

This pattern transferred cleanly to qemu-longhorn. The harder problem wasn't the sidecar. It was making sure all the state worth forking actually lands on the PVC in the first place.

The keystone: bind-mount the container's root onto the PVC

Here's the problem. In Kata, a container is a process tree inside a microVM. The container's writable overlay — where apt install, pip install, config edits, dotfiles all land — lives inside the VM, not on the PVC. Snapshot the PVC, fork, and the new sandbox boots with a fresh image and none of the state. Fork on such a sandbox is theatre: you fork an empty rootfs.

The fix, without modifying any user image:

• An init container seeds the PVC with the directory structure a wrapper script expects, then drops a .ready marker.
• The user container's command is replaced with a small shell wrapper mounted in via ConfigMap; the image's original Entrypoint + Cmd are passed to it as args. (k7 recovers the original argv by fetching the image's OCI config from the registry.)
• The wrapper mount --binds each of /etc, /var, /usr, /home, /root, /opt, /bin, /sbin, /lib, /lib64 from the PVC onto the corresponding directory in the VM — tar-seeding each one from the image on first boot, a no-op on forks.
• Then exec "$@": the user's original entrypoint takes over, now writing into directories that physically live on the Longhorn PVC.

The docker sidecar plugs into the same PVC via subPath: docker, so /var/lib/docker maps to the same volume. One snapshot atomically captures the user container's OS state and the docker daemon state. Fork the PVC, attach the clone, and dockerd in the fork boots and finds the parent's full image cache. It sounds boring. It is the keystone.

What it looked like

k7 create --sidecar docker --backend ql my-builder docker:27.5-cli
k7 shell my-builder
# inside: docker pull pytorch/..., docker build -t myapp .

for i in $(seq 0 7); do
  k7 fork my-builder exp-$i &
done
wait

k7 shell exp-3
docker images   # the pytorch image is right there

Each fork boots with the same OS state, the same installed packages, the same Docker images already pulled. No re-install, no re-pull, no re-build. This is the workflow that pushed me to ship it: agents that prepare one heavy environment and explore N branches in parallel are forever paying the setup cost N times. With fork, they pay it once.

The numbers, and where the time went

Measured on a single Hetzner dedicated node (3× NVMe, Longhorn replicas=1). For comparison: 8 cold creates would be 8 × 15.2 s ≈ 2 minutes, plus 8 × your before_script — which for a torch + transformers environment is easily 5+ minutes. Fork-8-in-parallel: ~53 s total, no re-installs. Still a real win. But look at the breakdown: the snapshot machinery is ~6 seconds. The other ~39 seconds is Longhorn replaying cloned data on first attach, plus scheduling a fresh pod and booting a new Kata VM. Cold-path work, every single fork.

Warm pools, and the ceiling

The obvious optimization was warm pools: keep a fleet of pre-booted, idle Kata VMs with an empty PVC attached, and on fork, swap the PVC for the clone instead of booting from zero. I expected that to bring the end-to-end comfortably below 10 seconds without changing the fork semantics. It's a good optimization, and it's honest work.

But writing it down is what made the ceiling visible. However warm the pool, this fork is disk-only. It clones the filesystem — packages, images, daemon data — and nothing else. No memory, no CPU registers, no process state. The forked sandbox boots: every process restarts, every cache is cold, every TCP connection is a new TCP connection, and your before_script re-runs. The best case for this architecture is "a fast reboot into the parent's disk", and for the workloads in Part 1 — byte-identical GRPO groups, forking a running Kubernetes cluster mid-flight — a reboot is precisely the thing that's not allowed. Memory clone on this stack meant CRIU on Kata, which was research, not something to ship.

So the warm pool went on the roadmap, and a different question took over: if the VMM itself owned the guest's memory as one file it could copy on write — the way fork(2) treats a process — what would the fork cost then? The answer turned out to be about three orders of magnitude less, and it's the rest of this series. The 45-second fork taught me exactly which problem I actually had: not a slow disk path, but the wrong unit of copy.

Part 4 opens the VMM.