Skip to content

Handover to milestone 3

Status: end of milestone 2c, the Paper agent (2026-08-10).

This document is not a spec. It says where 2c stopped and what 3 already finds in place. The design decisions live in superpowers/specs/2026-08-09-paper-agent-design.md, the open points in known-issues.md.

Where we are

A Server reaches phase Ready. Both halves of the two-stage ready gate close: the readiness probe speaks a real server list ping to a real Paper process, and the agent inside that process opens an authenticated ServerSession and reports its readiness and its player counts. Measured against a local kind cluster on 2026-08-10, a Server created at 22:37:16 was Ready at 22:37:36 with status.slots: 100, status.players: 0 and status.readinessLosses: 0.

Nothing else about that server is useful yet, and that is the whole of milestone 3. There is no proxy, so nothing routes a client to a backend; ProxySession answers Unimplemented; no proxy pod has a ServiceAccount; and Paper still runs with online-mode=true and no forwarding secret. A player cannot connect to a network that is otherwise fully ready.

What 3 covers

  • The Velocity image, the second image this repository builds.
  • A Velocity agent, the second consumer of the channel from 2a.
  • ProxySession on the operator side, and the identity that goes with it.
  • Forwarding: online-mode=false on the backends, the forwarding secret, and the configuration rendering design §5.4 already commits to.

The four things that must land, and why they must land together

These are already in known-issues.md under "Preconditions for milestone 3". They are repeated here because three of the four are not features — they are ways the milestone breaks if it is built in the obvious order.

The orphan sweep discards proxy agents. OrphanReconciler.Sweep lists pods with spawnery.cloud/role=server and forgets every registry entry not in that list. The first Velocity agent to open a session is removed from the registry within one sweep interval. Widening the filter — list by spawnery.cloud/managed-by, restrict the server existence check to role=server — belongs in the same change as the proxy podspec, not after it. This is an acceptance criterion, not a note.

ProxySession answers Unimplemented, and no bootstrap creates the spawnery-proxy ServiceAccount. The contract from 2a covers both sessions completely (design §5) but implements and authenticates only ServerSession. internal/controller.Bootstrapper knows only spawnery-server, so a proxy pod would have no identity to present at all. The handler, the bootstrap entry and the widened sweep filter are three parts of one change; none of them is a working proxy agent on its own.

Factor nix/oci-common.nix before the second image exists. nix/paper-image.nix holds four things the Velocity image needs verbatim: the passwd/group pair for the numeric user, the entrypoint's shebang rewrite through substitute --replace-fail, the copy into /usr/local/bin at the literal path a pod spec names, and the layered-image configuration around them. Extracting them while there is exactly one consumer is much cheaper than reconciling two copies later, and the drift is the kind nobody notices — an image that starts fine while its user or its paths quietly differ from the other one's.

Do not extend set_property for the forwarding mode. It is a .properties helper and it does not generalise: the forwarding secret and online-mode live in config/paper-global.yml, which is YAML, and design §5.4 already commits the entrypoint to merging rendered configuration into that file. Editing YAML from shell is the wrong tool. A small Go program baked into the image is the right one — it reuses the buildGoModule path spawnery-slp already establishes, it is testable the way internal/slp is, and it is the natural home for §5.4's per-group ConfigMap rendering. It is also where the /data/config and /data/plugins collisions have to be resolved, since that is the directory the rendered configuration lands in.

What 2c built that a Velocity agent gets almost for free

The Gradle-in-Nix shape. agent/paper/ is a Gradle project built through the nixpkgs Gradle setup hook with a checked-in per-artifact lockfile, deps.json, regenerated by make agent-deps outside the sandbox. It is bit-reproducible: nix build .#paper-agent --rebuild compares two builds and make image-repro covers the jar inside the image. A second agent copies the shape rather than rediscovering it. Three things about it are not obvious and cost real time in 2c:

  • nix build filters source through the git index, so a new file that has not been git added does not exist for the build. A suspiciously fast, quiet build right after adding files is a stale tree, not success.
  • The generated stubs must live in their own source set (src/proto/java) whose compile classpath carries gRPC and protobuf only. javac 21 cannot read the major-69 Paper jars if it ever has to resolve a class out of them.
  • :jar is an implicit dependency of :test. Once shadowJar shares its filename, the check phase silently replaces the shaded jar with an unrelocated one. 2c fixed it with a plain classifier on :jar and an exact install filename instead of a glob.

The relocation set. Everything is relocated under cloud.spawnery.agent.shaded., including the four gRPC service files under META-INF/services. Paper carries its own protobuf-java and netty — both verifiable in the image under /opt/paper/repo/libraries/. Enumerate Velocity's own bundled libraries the same way, out of the real jar, rather than assuming the conflict set is the same one. Two decisions transfer directly regardless of what that list says: use grpc-okhttp rather than grpc-netty, so there is no Netty to collide with at all — and know that grpc-okhttp on a JDK platform picks a TLS-1.2-only legacy ConnectionSpec by default while internal/agentserver serves MinVersion: tls.VersionTLS13. 2c's agent could not have connected to a real operator until that was overridden. It is the first thing to check when a new agent's handshake fails before a byte of HTTP/2.

The stub-operator harness. cmd/spawnery-stubop plus hack/agent-test.sh run the real image against a real gRPC operator with real TLS and a real token, and emit a JSON event trace the script asserts on. It has three phases today: a passive operator (proves the overlap), a superseding one that cancels the displaced stream exactly where sessions.enter() does (proves there is no reconnect storm), and a muting one that accepts a stream and says nothing (proves an unanswered session is bounded). A Velocity agent needs a fourth personality, not a new harness.

The one thing worth saying plainly

Milestone 2c's agent produced five defects in a row, each uncovered by fixing the one before it, and not one of them was in the code the tests were checking. Each was in an assumption about what the tests measured.

  • The in-process transport delivers synchronously, so the agent's local program order and the wire order coincided. Every ordering test through task 6, including a purpose-built tracking channel, carefully measured the wrong quantity — and the renewal did not overlap on a real connection at all.
  • The stub operator accepted streams and never cancelled them, so the real operator's retirement order was invisible, and with it a reconnect storm that would have run at roughly 1 Hz per server, fleet-wide.
  • The stub always answered, so "the operator accepted and said nothing" was not a state that existed in the test space. The agent had no client-side liveness bound at all, and the operator's own rescue is armed too late to be one.
  • Fixing that bound introduced a graceful shutdown on the one call the operator never finishes, parking a transport per stall.
  • And the sharpest one, found only by the final whole-branch review: every reconnect test asserted operator.streams.size and the harness counted stream_opened — both on the operator's side. So how many channels the agent left behind was measured nowhere, and a stream that broke never shut its channel down. An operator outage retained one live ManagedChannel per attempt, each still retrying underneath, at the operator that was trying to come back.

The harness is where the assumptions hide, not the code. A Velocity agent will be built against this same harness. The useful habit is to ask, of every green assertion, which side of the wire is this measured on, and what does the stub do that the operator does not.

The environment

nix develop        # Go, controller-gen, protoc, envtest assets, kubectl, kind, k3d, JDK 21, Gradle
make test          # Go only, ~25 s; must be green before anything is touched
make agent         # the plugin and its JUnit suite
make agent-test    # the plugin against the stub operator, in the real image
make image-test    # the image offline, under the pod spec's constraints
make image-repro   # two image builds, compared byte for byte

make agent-deps regenerates deps.json and is in no other target because it reaches Maven Central.

JVM target is 21, everywhere, and Gradle's toolchain auto-provisioning stays off. The image ships a JDK 25 for Paper; the plugin is compiled against 21 and runs on 25.

A container runtime is required for every target above except make test, and the image targets only work on x86_64-linux. k3d does not work against a rootless Podman socket at all; kind does, wrapped in systemd-run --scope --user --property=Delegate=yes. The README documents the working flow, known-issues.md records why.

Questions worth settling before code

  • Does the Velocity agent share code with the Paper agent, or copy it? The session loop, the token source, the channel construction and the credentials are identical in shape and differ only in which stub they call. A shared Gradle subproject is the obvious answer and the lockfile makes it cheap; the cost is that the two agents can no longer be versioned apart. Decide it before the second build.gradle.kts exists, not after.
  • What is a proxy's readiness, and can it be lowered? A Paper server latches readiness once and never lowers it, which is why the gap in the contract — Hello{ready:false} cannot lower a readiness the registry has recorded — never bit. A proxy that drains wants exactly that expression. If milestone 3 needs it, the contract in internal/agent/registry.go has to change, and that is a 2a change, not a 3 change.
  • Where does the forwarding secret reach the backend? The Network already references a Secret and the podspec does not mount it. Whether it arrives as an env var, a file, or a rendered paper-global.yml decides how much of §5.4 milestone 3 has to build.
  • Does the operator run in the cluster for the E2E flow? Today it runs outside it through go run, and the local flow has to hand-build the Service and Endpoints its own pods dial. That is workable for one person at a terminal and is a wall for milestone 6's CI. An operator image is not in 3's scope, but 3 is when the absence starts to cost.