← notes on building k7d

Notes on building k7d · Part 2

Why not Firecracker, Kata, QEMU, or an E2B-style sandbox?

The design space of forkable environments, and the gap none of the existing tools filled.

August 2026 · ~10 min read

Part 1 stated the requirement: RL and agent evals on infrastructure need byte-identical copies of a running Kubernetes cluster, dozens at a time, in ~100 ms. Before writing a VMM from scratch I spent real time trying not to. This post is the map of that design space, with a fair take on each corner of it, because every tool below is excellent at the job it was actually built for. The gap is specific, and it's worth being precise about where it is.

What the workload actually demands

Write down what GRPO-style training and tree search over infrastructure environments require, and the list is short but strict:

• Byte-identical starts. Group-relative methods compare rewards within a group. If member A starts from a different etcd revision or a colder cache than member B, the reward gap is noise. "Reset" is not "identical". (Part 1 covers why.)
• Fork latency ≪ episode length. If producing a copy takes as long as the rollout itself, half your compute budget is environment bring-up. Cold-booting a Kubernetes cluster is ~30 s; an eval step might be shorter than that.
• Density. Fifty concurrent copies must not cost fifty full RAM bills. You want to pay for divergence, not for the base.
• The unit is a cluster, not a process. Control plane, kubelets, pods, the network between them, established TLS. Copying one VM out of a three-VM cluster gives you a corrupted cluster, not a smaller one.
• Tree-shaped lifecycles under a budget. Agents fork, abandon, roll back, fork again. Someone has to enforce RAM and disk caps, because the agent is exactly the party you can't trust to clean up.

No single existing system was designed against this list. Which is not a criticism — it's a young workload. Here's how each candidate scores, honestly.

The map

Two axes. What gets copied (a process, a VM, a whole cluster with its network), and how (reboot from scratch, snapshot then restore, live copy-on-write fork of a running thing). The requirements above put the target squarely in the top-right corner. Let's walk the occupied squares.

Firecracker: snapshot and restore, not live fork

Firecracker is the closest thing this space has to a gold standard, and I mean that. Minimal device model, boots fast, and the security architecture — one jailed process per microVM, seccomp'd to the floor — is exactly what you want for hostile multi-tenant workloads. If your problem is "run a million strangers' functions", Firecracker is the answer and nothing below changes that.

But Firecracker's copy verb is snapshot → restore. You serialize a paused VM to files, then construct a new VM from those files. That's a materially different thing from a fork in three ways. First, the source stops being a live, diverging thing at the snapshot point — you branch from a frozen image, not from "now". Second, restore is a fresh VM that shares no memory with its siblings, so fifty copies cost fifty memory footprints, not fifty sets of dirty pages. Third — and this is the deal-breaker for me — there is no story for a set of VMs with a network between them. A Kubernetes cluster is several machines plus established TCP and TLS sessions binding them together. Snapshotting each node separately and restoring them onto a shared network gets you IP conflicts or broken cluster identity; Part 1's preserved-network-identity mechanism doesn't exist anywhere in the Firecracker ecosystem, because it was never Firecracker's problem.

And the isolation model that makes Firecracker so trustworthy is precisely what forbids live CoW forking: parent and child memory as views of the same pages requires them to live in one address space, and one-jailed-process-per-VM exists to prevent exactly that. You can't have both. Firecracker chose correctly for its threat model. I needed the other choice.

Kata: strong isolation, no fork

Kata Containers puts each Kubernetes pod in its own lightweight VM behind a containerd shim, so runtimeClassName gives you hardware isolation with a kubectl-shaped interface. I know it well — the first version of my sandboxing project, k7, was built on k3s + Kata + Firecracker, and it shipped and worked. Kata is a great answer to "make this pod a VM".

It just doesn't have a fork verb at all. Kata manages VM lifecycles: create, run, delete. Snapshot/fork/rollback are not in the API. The usual suggestion is CRIU — checkpoint the processes inside — and my honest assessment from having looked hard at it: CRIU on Kata is research, not something you ship. And even granting a working CRIU, you're back to the process axis of the map: checkpointing processes doesn't capture the machine (page cache, kernel state, TCP stacks, the clock), and the whole point of Part 1 is that Kubernetes notices the difference.

QEMU: the machinery exists, but it's built for migration

QEMU deserves its own square because it's the maximalist answer: live migration, savevm, dirty-page tracking — the machinery for copying running machines genuinely exists in there, battle-tested by two decades of production migration. If anyone "already solved" VM copying, it's QEMU.

But the machinery is aimed at moving one VM from host A to host B, not at fanning out fifty budget-managed CoW children on one box. There's no snapshot tree, no shared-until-dirty memory between siblings, no cluster-level orchestration of a coordinated multi-VM pause. You'd be building all of that around QEMU anyway — against a device model of maximal generality, every subsystem of which participates in migration serialization. I did in fact build my previous system on QEMU (via Kata), with Longhorn disk snapshots doing the copying. The ~45 seconds per fork came from that storage path, not from QEMU — QEMU alone could likely have forked much faster. But that stack could not copy memory at all, and the reasons above are why I didn't build the memory story on QEMU either. That experience is Part 3 of this series, and it's the single biggest reason k7d exists.

CubeSandbox / E2B-style services: sandbox-level fork, not cluster-level

The sandbox-service generation — E2B, Tencent's CubeSandbox, and friends — is the closest in spirit to what I wanted, and they deserve credit for making "agents need forkable environments" a mainstream idea. CubeSandbox in particular does serious systems work: XFS-reflink disk CoW, incremental memory snapshots, an SDK with snapshot/clone/rollback, auto-pause. Their published clone path is a snapshot plus N restores at around ~220 ms per sandbox. That's a genuinely good number.

The unit, though, is one sandbox — a single microVM or process environment running your code. If your agent's world is a REPL or a browser, that's exactly right, and you should use them. If your agent's world is a Kubernetes cluster — several machines, a control plane, a CNI, in-cluster TLS — a sandbox service gives you no way to say "copy all of this, coherently, with the network identity preserved". You'd fork three sandboxes and get three consistent machines in an inconsistent set: the same problem as Firecracker, one level up the stack. k7d is deliberately not E2B-API compatible for this reason. It's a different unit of isolation, not a competing implementation of the same one.

Comparison

The isolation row is the one to read carefully — it's the trade-off the rest of this post is about.

The trade-off I accepted: one address space

Live copy-on-write fork means the child's memory is a MAP_PRIVATE view of the parent's. Views of the same mapping have to live in the same process. So k7d is one daemon owning every VM's memory in a single address space — the exact architecture Firecracker's jailer exists to prevent.

What this costs: isolation between sibling forks is weaker than one-jailed-process-per-VM. Guest-to-host isolation is unchanged — that's still KVM, the same hardware boundary everyone relies on — but a hypothetical bug in the k7d daemon itself is a bug in the process that maps every guest's RAM. For hostile multi-tenant workloads, where sandbox A's owner must never touch sandbox B, that's disqualifying, and I'll say so plainly: don't use k7d for that. Use Firecracker.

What it buys: the fork. Fifty copies of a running cluster sharing memory until they diverge, in ~100 ms. For fleets of your own environments — RL rollouts, agent evals, CI — the tenant on both sides of the boundary is you, and the weaker sibling isolation is a price worth paying for a verb nobody else has. The mitigation for "one big trusted process" is keeping it small enough to actually audit (≤25k lines of Rust for the VMM plus shim) and machine-checking the scariest parts, which is Part 6.

Every system on the map made its trade correctly for its workload. This one is correct for mine. That's the whole argument.

Where next

Before k7d, I tried to get fork semantics out of the existing stack anyway: qemu + Kata + Longhorn PVC snapshots, forking sandboxes at the disk level in about 45 seconds. It worked, it shipped, and its ceiling taught me exactly which problem I actually had. That story is Part 3 — an explicitly skippable detour. If you only care about how the 100 ms fork works, jump straight to Part 4.

Next: The 45-second fork (skippable), or Anatomy of a 100 ms cluster fork.