Skip to main content

Agents can run as pods in a namespace of their own

ADR 0019 put every agent in an ephemeral Docker container and named Kubernetes as the next sandbox. Mission Control is moving into the homelab's single-node K3s cluster, where it cannot start Docker containers, and where the Docker answers to "how does the container reach the server" (a Unix socket, or the Docker host) do not exist. We decided to add a third launcher beside docker and process: kubernetes, which starts each run's agent runtime as a pod of the agent image in one namespace set aside for agents, and to keep everything else of ADR 0019: the same image and bridge, ACP over stdio, the same paths, the same home that outlives the pod, the same sponsor and runtime configurations.

The decisions, each worked out in docs/spec/kubernetes-runner.md:

  • One pod per run, bare and named for it (restartPolicy: Never), created by the server with Kubernetes' API. Its credentials go into a Secret created with the pod as its owner, so they are named by kubectl get pod but never shown by it, and go when the pod goes.
  • ACP rides the pod's attached stdio (pods/attach, stdin kept open), so the bridge and the agent image do not change. The server never needs a pod's address; nothing may connect into an agent pod.
  • The data lives on one PVC that the server and the pods mount at the same path. A pod mounts only its working directory and its run's directory, as subPaths of the claim, so it sees neither the database nor anyone else's work.
  • Agents reach Mission Control at its in-cluster Service, through the bridge's existing forwarder (MC_API_HOST), on the run's token as before.
  • An agent pod can reach almost nothing. It runs without a service account token, as a non-root user with every capability dropped, in a namespace that enforces the restricted pod security standard and denies all network traffic except DNS, Mission Control's Service, and the internet outside every private range. The cluster, the mesh, the LAN and the forge are not reachable.
  • The server's own access is one namespace: create and delete pods, attach to them and read them, and create Secrets. It cannot read or delete a Secret, exec into a pod or touch another namespace.
  • A pod is cleaned up by the server: deleted when its agent stops, and every agent pod found at startup or later with no run behind it is deleted, since a pod whose server is gone cannot be driven again.
  • No role has cluster access from inside its pod unless an operator grants it, as a Kubernetes service account an operator maps to a role, read-only and without Secrets. Which roles, if any, is left to Morten.

Considered Options​

  • Kubernetes Jobs instead of bare pods: rejected; a run is driven interactively, a Job's retries would start a second agent on the same home, and the one thing a Job adds, deleting itself, is a sweep of a few lines here anyway.
  • The bridge dialling out to Mission Control instead of the server attaching: deferred, not rejected. It needs no pods/attach and keeps ACP out of the node's container log, but it changes the bridge, adds an endpoint that must authenticate pods, and gives up the one contract every launcher shares (stdio). Open decision 3.
  • Exec'ing kubectl attach as the child process: kept as the fallback if the Kubernetes client library does not run in the server's Bun build; it leaves the launcher contract unchanged but puts kubectl in the server image.
  • Credentials as pod environment variables: rejected; kubectl get pod -o yaml and every tool that lists pods would show them.
  • Mounting the whole PVC at the data path: rejected, for the reason ADR 0019 gave against mounting git/repositories, and more: it would put the database, the sealing material and every other run's directory in an agent's reach.
  • A PVC per run, or a checkout on an emptyDir: rejected; the server prepares and fetches clones on the host's path before the pod exists, and mount paths must equal the server's.
  • An agent namespace per organization or per run: rejected; a single-tenant home has one sponsor of risk, and quotas per namespace would only fragment a node with little to spare.
  • Pod-to-pod isolation by the mesh: rejected; agents run outside the mesh so they cannot borrow a workload identity, and NetworkPolicy does the work.
  • Egress allowed only to named model APIs: not available with NetworkPolicy, which filters addresses, not names; agents also need package registries to build. A name-based allow list through a proxy is the stricter option left open (decision 1).

Consequences​

  • The host kernel is shared with the cluster's own workloads and datastore, so an escape reaches more than it did on a laptop. The spec lists the mitigations (user namespaces, a sandboxed runtime class later, Falco rules, an admission backstop, kernel patching) and requires the kubelet's podPidsLimit (a k3s restart); whether to separate the database's disk from agents' is open.
  • Docker is no longer required where the launcher is kubernetes; a Kubernetes cluster with a ReadWriteOnce claim on the same node as the server is.
  • Server and pods must share a node, since the claim is local-path. A multi-node cluster would need a ReadWriteMany claim, which this design does not provide.
  • What treats launcher === 'docker' as "nothing comes from the host's environment" must treat the kubernetes launcher the same, or the server's own environment would be copied into agent pods. That rule lives in one place, launchCommand in runtime-kinds.ts, which runs and sign-ins both start through.
  • ACP traffic is written to the pod's stdout, so the node's container log holds it for as long as the kubelet keeps it. pods/log is therefore granted to nobody by default.
  • A pod being evicted, preempted or reaching its hard deadline interrupts the run the way a server restart does; it resumes with the same home.
  • Stays outside this record: the Helm chart, the Mission Control server image and its backup, which are separate tasks that start only once this design is approved.