Skip to content

Build history

This is the record of how Spawnery was built, milestone by milestone, moved here out of README.md where it had grown to two thirds of the front page. Nothing in it has been rewritten: every paragraph is the one that milestone wrote when it closed, kept in the tense it was written in, so a claim that has since been overtaken reads as what was true then rather than as what is true now. Where a later milestone overtook one, the later entry says so.

It is kept rather than compressed into a changelog because most of it is not a list of what shipped. It is the record of what was measured against what was assumed — which claims a cluster confirmed, which ones a mutation showed were narrower than their design document said, and which defects were found by something other than the test written to catch them. That is the part a changelog throws away.

For what is open right now, read known-issues.md. For what the current release does, read the README.

Contents

Milestone 1

Milestone 1 is done: the four CRDs, the operator with the Network, ServerGroup and Server controllers, the state machine including readiness loss, and the orphan sweep.

Milestone 2

2a — the agent channel

Milestone 2a is done: the agent channel. A gRPC service inside the operator process accepts TLS connections from game server pods, identifies them through a pod-bound ServiceAccount token, and the registry behind it feeds the two-stage ready gate and status.players that milestone 1 had left unwired. The channel is proven end to end in envtest — a test agent brings a Server with green pod readiness all the way to phase Ready.

2b — the Paper base image

Milestone 2b is done: the Paper base image. nix build .#paper-image produces a reproducible image holding Paper 26.2, a headless JDK 25 and spawnery-slp, the tool the readiness probe calls to speak a real server list ping. Paper is patched at build time, so a pod downloads nothing at startup; make image-test runs the image offline to keep that true.

2c — the Paper agent

Milestone 2c is done: the Paper agent. nix build .#paper-agent produces a shaded, bit-reproducible Kotlin plugin that is baked into the image; the entrypoint copies it into the pod's writable plugins directory, and from there it opens an authenticated ServerSession to the operator, reports its readiness and its player counts, and renews the session with overlap so a renewal never costs a server its Ready.

Both halves of the ready gate now close. Measured in a local kind cluster, a Server reaches phase Ready about twenty seconds after its pod is created, and status.players and status.slots carry what the agent read off the running server rather than a placeholder.

Milestone 3

3a — the operator's proxy side

Milestone 3a is done: the operator's proxy side. ProxySession joins ServerSession in the fan-out instead of answering Unimplemented, the bootstrap creates a spawnery-proxy ServiceAccount in every namespace it touches alongside spawnery-server, and the orphan sweep no longer discards a connected proxy's registry entry. None of it has an agent to drive it yet, so this closes the operator's half of the contract without anything reaching phase Ready for a proxy.

3b — the Velocity image and configuration rendering

Milestone 3b is done: the Velocity image and configuration rendering. nix build .#velocity-image produces a reproducible image holding Velocity 3.5.1 on the same JDK 25 and non-root base nix/oci-common.nix now shares with the Paper image. cmd/spawnery-config, baked into both images, replaces what the Paper entrypoint used to rewrite by hand: it resolves the operator's rendered ConfigMap, a user overlay and a handful of fields neither may move into server.properties, config/paper-global.yml and velocity.toml, and refuses to start — naming the file and the key — rather than guess at anything missing or malformed. online-mode is now false on the backends and true on the proxy, which is what modern player-info forwarding actually requires.

3c — the Velocity agent

Milestone 3c is done: the Velocity agent. nix build .#agents produces both shaded plugin jars — Paper's and Velocity's, sharing one session loop in agent/common since this milestone's Gradle split. The Velocity plugin opens an authenticated ProxySession, mirrors the operator's server list into Velocity's own registry, binds its readiness port only once a server list has arrived so a proxy pod cannot turn ready before it can route anyone, routes a joining player through the group's fallbackGroups try-list, and moves players off a backend the operator is draining onto whatever that try-list offers next. A ProxyGroup now reaches phase Ready the way a Server already did, and milestone 3's whole point — a player can join — has code behind it.

Getting there found the milestone's one Critical defect, and it is worth stating plainly rather than passed over: the paragraph above about milestone 3b claimed the rendered configuration was correct on disk, and that was not true. internal/render wrote Paper's forwarding secret under the key secret-key; Paper reads secret, silently ignores the one nobody asked for, and disables forwarding in its own log — a line nothing had ever read. Every backend this operator rendered came up healthy and refused every forwarded join. It was found by the first real end-to-end join attempt, ten tasks into this milestone, and no test until then could have caught it: every render test asserts what the renderer wrote, none of them asked what Paper read back. Both are now checked. docs/known-issues.md's "From milestone 3c" section has the rest of what this milestone leaves open, in particular that a NetworkPolicy restricting backends to proxies-only is now overdue rather than deferred, and that proxy drain needs a readiness internal/agent/registry.go cannot yet lower — milestone 4c-1 below is what gave it one, and the NetworkPolicy is still owed, by milestone 6b.

make agent-test runs both plugins, in the real images, against a real operator-shaped gRPC server — including, for the proxy, that its ready port stays closed until a server list arrives and opens once one does. The cluster-level proof beyond that harness is written down as docs/runbook-milestone-3-evidence.md, and it has now been run once, against a real kind cluster (2026-08-12): an automated join through cmd/spawnery-join reached a backend, and Paper's own log and Velocity's own log confirmed it — the first time the forwarding chain built in 3b and 3c was observed working end to end rather than merely rendered correctly on disk. That automated run's drain proof surfaced a real finding instead of a clean result: deleting a Server under a player held open by the evidence tool disconnected them rather than moving them, traced to the tool stopping short of the point Paper counts a player as online rather than to the drain logic itself — full diagnosis in docs/known-issues.md, "From the milestone 3c evidence run", which is to be read as a finding about cmd/spawnery-join and not as an open criterion.

The two things that run could not settle — a join with a real Microsoft account, and a drain moving a real player rather than the tool's stand-in — were both proven by hand the next day (2026-08-13), and docs/handover-milestone-4.md carries the logs. A licensed Minecraft client joined through the proxy; the artifact is the UUID, because Mojang minted a version-4 one only after the client proved its session, where the automated probe could only ever produce the version-3 offline form. Deleting that player's Server while they were in the game moved them onto a fallback inside the same second, with the proxy's log and the destination's both timestamping it. So milestone 3's whole point is proven against a real client, not only against a tool.

Carry-overs and preconditions for later milestones — CA rotation, the NetworkPolicy restricting backends to proxies-only that online-mode=false now makes a real invariant rather than a deferred one, and what earlier milestones leave open — are in docs/known-issues.md.

The design lives under docs/superpowers/specs/, the plans under docs/superpowers/plans/.

Milestone 4

4a — slot-based scaling

Milestone 4a is done: slot-based scaling. An ephemeral ServerGroup no longer sits at its floor. It creates servers as soon as its free player slots fall below spec.scaling.spareSlots, bounded by maxReplicas, and removes them again — one per pass, and only while the group's free slots would still cover the spare — once a server has been empty for scaleDownStabilizationSeconds. That is the rule for a group short of demand; a lowered maxReplicas is a different rule, and a stronger one — it removes the whole surplus in a single pass and waits for neither the stabilization window nor the spare-slot check, because a ceiling is an instruction, not a suggestion. The rule is DecideSize in internal/controller/scaling.go, a pure function beside phase.Decide and SelectDeletionCandidates, and the invariant those already carried holds unchanged: a server that may be carrying players is never nominated.

The one thing worth naming is what the scale-up rule reads. A server created now is not Ready for tens of seconds and adds nothing to status.freeSlots, so a scaler reading that figure would see the same shortfall on every five-second pass and order the same replacement again, until maxReplicas stopped it. It reads a second figure instead, one that credits capacity that has been ordered and has not arrived. The two are deliberately not the same number, and the envtest that carries this milestone is the one that keeps reconciling for ten more passes and asserts the count has not moved — a single decision cannot show that failure.

4b — rolling updates of ephemeral groups

Milestone 4b is done: rolling updates of ephemeral groups. A spec edit bumps a ServerGroup's metadata.generation, which makes every server created before it stale, and 4b is what carries out the changeover instead of a person deleting pods. selectRetirement (internal/controller/scaling.go) nominates one stale server per pass — empty ones first, then the oldest — the group patches spec.retire: true onto it, and phase.Decide moves it into the new phase Retiring: deregistered, so it takes no new joins, while the players it already has finish in their own time. Nobody is kicked. spec.update.maxUnavailable bounds how many may be out at once, and spec.update.maxStaleSeconds is what escalates a server that never empties into a real drain.

The one thing worth naming is the deadlock this had to break twice. A group already at maxReplicas has no room to build the first server of the new generation, so the changeover cannot start; the rule that resolves it sheds an idle stale server first, and when there is nothing to shed the group says which kind of stuck it is — ScalingLimited names the blocked cold start rather than an ordinary capacity shortfall. What 4b leaves open is that any spec change starts a changeover: retuning spareSlots replaces a whole group of functionally identical servers.

4d — per-group backoff and the Degraded condition

Milestone 4d is done: per-group backoff and the Degraded condition. It appears here, out of alphabetical order, because it shipped here: it was cut out of 4b during that milestone's design, on the measurement that it shares no code with the rolling update, and 4c was already named. A group whose servers cannot start no longer rebuilds them as fast as it can notice them dying. CountFailures and DecideBackoff (internal/controller/backoff.go) turn a running failure streak — kept on the CR as status.consecutiveFailures and status.lastFailureAt — into permission to create, waiting ten seconds and doubling to a five-minute cap, and giving up after six. Waiting publishes BackingOff; giving up publishes Degraded with reason CrashLoopBackoff, and the phase follows. The one thing worth naming is what breaks the streak: a success since the last counted failure, not any server being Ready. The weaker rule reads well and is wrong — a group with one healthy server and one that crash-loops would hold its count at zero forever and hammer indefinitely. It leaves open that the threshold counts failed servers rather than failed rounds, so a group with minReplicas of six or more can give up on its very first pass, and giving up is terminal until somebody edits the spec.

4c-1 — the proxy readiness contract

Milestone 4c-1 is done: the proxy readiness contract, and the first drain that uses it. Before a proxy pod is removed, the operator tells its agent to stop being ready — a SetReady message on the proxy channel, which the Velocity plugin's ReadyGate answers by closing the port the kubelet's readiness probe completes against — so the endpoint disappears and no new player is routed there. It then waits for the pod to empty, bounded by spec.drain.timeoutSeconds. The one thing worth naming is that the wait is for empty, not for NotReady, and that nobody is moved: a proxy drain has no elsewhere to put a connection that terminates at the proxy being removed. The deadline is therefore the only path in the milestone that disconnects anyone, and it says so out loud, with a Warning event naming how many people it just disconnected. What it leaves open matters on upgrade: a proxy image predating SetReady ignores the message, never lowers its readiness, and keeps taking new players for the whole drain window. Upgrade proxy images before the operator. Its two cluster claims were driven twice against a real cluster with a licensed client on 2026-08-14 (docs/runbook-milestone-4c1-evidence.md).

4c-2 — proxy rolling updates

Milestone 4c-2 is done: proxy rolling updates. A proxy pod is stale when its spawnery.cloud/pod-hash label differs from a digest of the pod the operator would render for that group right now; DecideRollout (internal/controller/rollout.go) surges by one, takes one pod at a time, and hands each marked pod to 4c-1's drain unchanged rather than inventing a second deadline. The one thing worth naming is what hashing the whole rendered pod costs, rather than a chosen list of spec fields: upgrading the operator can roll every proxy in the cluster with nobody having edited anything — a new default in internal/podspec, an added environment variable, even a different --operator-namespace, all move the digest. The group's own status hides it, because the surge pod arrives before any withdrawal; two distinct pod-hash values in one group is the tell. Its own cluster claims are §11 of the same runbook, added for this milestone and driven the same night against merged master, with a real client (docs/runbook-milestone-4c1-evidence.md).

4c-3 — node drain

Milestone 4c-3 is done: node drain. IsDeparting (internal/controller/nodes.go) has two ways in — spec.unschedulable, hardwired, and any taint whose key was passed to the repeatable -drain-taint flag and whose effect is NoSchedule or NoExecute — and from there the two group kinds answer differently on purpose. A condemned server attaches to DecideSize's decision outside the capacity, ceiling and demand rules, because a node drain answers to none of the three: the node is leaving with or without the group's consent. A proxy on a departing node is simply a second kind of staleness, so 4c-2's rollout drains it. Both kinds get a PodDisruptionBudget sized from the spawnery.cloud/occupied label. The one thing worth naming is what this milestone's own closing review found and fixed: the live budget selector carried no role term, so in a namespace where a ProxyGroup and a ServerGroup share a name, ready occupied proxies inflated currentHealthy against a desiredHealthy counted only from occupied servers — and the eviction API could have spent the difference disconnecting players. Adding the role term closed it in the reconciler, but one copy is out of reach: a ServerGroup last reconciled by pre-4c-3 code left a budget at its own bare name, which nothing renames or deletes, carrying a frozen minAvailable and a frozen copy of the broken selector. Delete it by hand — docs/upgrading.md has the kubectl to find it. docs/known-issues.md also leaves open that an operator running cluster-autoscaler must pass -drain-taint ToBeDeletedByClusterAutoscaler by hand: that autoscaler taints without cordoning, and an unset flag looks exactly like a quiet node.

Milestone 5

5a — persistent groups exist

Milestone 5a is done: persistent groups exist. A ServerGroup of type Persistent used to accept spec.replicas and spec.storage and build nothing. Now DecidePersistentSize (internal/controller/persistent.go) sizes it by ordinal — <group>-0, <group>-1, created lowest-first and removed highest-first — BuildDataClaim (internal/podspec/claim.go) renders one PersistentVolumeClaim per ordinal from spec.storage, and the Server controller creates the claim before the pod that mounts it. The same ordinal always addresses the same claim, so an ordinal that comes back finds its world where it left it. The one thing worth naming is what the claim deliberately lacks: no owner reference, so a world outlives its server, its group, and an operator who deletes the wrong object — and the ClusterRole grants neither delete nor update on claims, with the omission enforced by internal/rbacaudit's table rather than merely written down, so a delete marker added anywhere later turns make test red before it can ship. The consequence is the open item: claims accumulate, and reclaiming a world is a deliberate human act with kubectl. The acceptance test was driven against a real kind cluster on 2026-08-16 — blocks placed, the pod deleted, the client rejoined, the blocks still there (docs/runbook-milestone-5a-evidence.md).

5b — ordered shutdown, Recreate updates and storage growth

Milestone 5b is done: ordered shutdown, Recreate updates and storage growth. An image change now moves a persistent server, a lowered replicas takes one ordinal down at a time, and spec.storage.size growth reaches the claim (growClaim, with patch and deliberately not update, so one field moves rather than the whole object). It all lands in the same DecidePersistentSize, which nominates missing, surplus, stale-spec and resize-pending ordinals in that priority order behind two gates: at most one takedown in flight, and no stale ordinal touched until every required ordinal is Ready. The one thing worth naming is not the feature but the 5a defect it uncovered: a persistent group's failure counter froze on any spec edit, because the counting call site filtered every view through a generation stamped once at creation while DecidePersistentSize is generation-blind by design — invisible until 5b gave persistent failures somewhere to accumulate toward. It leaves open that a permanently broken ordinal stalls the whole group's update, with nothing timing that wait out. Driven on 2026-08-16: two worlds survived an update that recreated both, one ordinal at a time (docs/runbook-milestone-5b-evidence.md). The positive half of storage growth was deliberately not driven there — kind's local-path storage class cannot expand a volume at all.

5c — detecting forwarding secret rotation

Milestone 5c is done: detecting forwarding secret rotation. The Network controller reads the forwarding secret on each resync, records a salted eight-byte digest of it in status.forwardingSecretHash, stamps every pod it creates with that digest in spawnery.cloud/forwarding-hash, and reports the comparison as two conditions and two events. It is detection and reporting only: it restarts nothing and takes no ordinal down, because the restart order is a decision for a person with a maintenance window, working through docs/runbook-milestone-5c-secret-rotation.md. The one thing worth naming is the negative that makes it safe: a rotation must move no pod hash, so both DesiredServerHash and DesiredProxyHash strip the forwarding label before digesting. Had that digest reached spec.podHash, rotating a secret would have restarted every pod on that network by itself, with nobody having asked for one: 5b's takedown rule would have walked every ordinal of every persistent group, and 4b's retirement rule every server of every ephemeral one. It leaves open that the stamp records the last digest the operator read, not the bytes the pod actually mounted — under a refused or failed read a pod can be running the new secret while being reported stale. Driven on 2026-08-16, against the standing procedure rather than around it (docs/runbook-milestone-5c-evidence.md).

Milestone 6

6a — the operator runs inside a cluster

Milestone 6a is done: the operator runs inside a cluster. nix build .#operator-image produces a reproducible image for the operator itself, hack/publish.sh (make publish) is one path that copies all three images from their Nix archives to ghcr.io/spawnery/, and make e2e builds a kind cluster, installs config/deploy/, and drives the operator through twelve ordered scenarios under its own ServiceAccount — scaling, the ceiling, the orphan sweep, the finalizer, the startup deadline, a world outliving its server, the proxy's Service, the permission table against the real authorizer — before reading the operator's whole log and failing on is forbidden:. That last check is the point of the exercise: internal/rbacaudit compares the generated ClusterRole against a hand-maintained table in both directions, so it catches drift, but a permission missing from both leaves make test green while the operator walks into a denial the first time it runs for real.

The one thing worth naming is what that check turned out to cover, because it was measured rather than assumed and the answer is narrower than it looks. Removing a write verb makes it fire: taking create on pods out of the markers produced a quoted is forbidden: ... cannot create resource "pods" on the first attempt. Removing a cache-backed list does not. With list on pods revoked and confirmed revoked at the API server, seven and three-quarter minutes of continuous watching produced no denial in the log, no 403 in the operator's own client metrics, no restart and no drop in Available; list on networks behaved the same way. Those two lists are what was measured, and reads as a class are not: no uncached read was ever revoked and watched. The hypothesis that would license the wider claim — that such a read goes through the manager's cache, whose initial sync is a watch rather than a list, so the revoked verb never reaches a request anyone could deny — is a hypothesis, not something this milestone established. docs/known-issues.md carries both halves of the measurement, the anomaly the hypothesis does not explain, and a second and unrelated way a denied read escapes the check.

What 6a leaves open is recorded as open. No real make publish has been driven — it needs a token nobody in that milestone had — so the digest reference in config/deploy/deployment.yaml has never been resolved by anything, and until someone pushes, every consumer still loads the images into their cluster by hand. The run is single-node, so node drain, HostPort and CIS pod security wait for the RKE2 rollout at the end of milestone 6.

6b — the traffic rules, and the first bounds on the agent channel

Milestone 6b is done: the traffic rules, and the first bounds the agent channel ever had. An accepted Network now writes a NetworkPolicy into its own namespace, owned by that Network so the garbage collector takes it away again, selecting the network's own server pods: ingress on 25565 from that same network's proxies in that same namespace, egress to cluster DNS and to the operator's agent port, and nothing else either way. config/deploy/ gains a second policy, which selects the operator pod and admits 9443 only from pods carrying spawnery.cloud/managed-by in any namespace — cross-namespace by construction, because every managed pod in every game namespace dials the one operator. Behind it the agent channel finally got the bounds it never had (MaxConcurrentStreams, ConnectionTimeout, an idle reaper and a keepalive enforcement policy), a TokenReview cache that deliberately does not cache the pod lookup — so deleting a pod, the revocation an operator actually performs, still takes effect on the next connection attempt — and a per-peer token bucket consulted only when that cache misses.

The one thing worth naming is the asymmetry: the policy selects backends and not proxies. A server's readiness probe is an exec inside its own container against 127.0.0.1, which no NetworkPolicy governs; a proxy's is a dial from the kubelet, which one might, and whether kubelet traffic is subject to policy at all depends on the CNI. The invariant at stake is entirely a backend's — a Paper server runs online-mode=false, authenticates nobody, and trusts whatever completes the modern-forwarding handshake with the right secret — so 6b selects backends and puts no game pod's readiness at a CNI's mercy. The one place it could not avoid the question is the operator's own policy, which selects the operator pod and therefore has to admit the kubelet's probe from a source no selector can name; that rule has no from at all, which is the only formulation correct on every CNI. The price of leaving proxies alone is stated rather than hidden: nothing restricts who may open a connection to a proxy's 25565 from inside the cluster, and the proxy is the front door that has to accept the world anyway.

What it leaves open is why none of the above should be read as protection: 6b has not observed a single connection being refused, anywhere. kindnet, the CNI make e2e runs on, was measured to enforce nothing: with the operator policy's kubelet-probe rule deleted, so that the object in force denied the probe outright, the run stayed green and the rollout succeeded on its usual timeline. Both alternative explanations were closed, because the readiness probe is an httpGet over the real network path that kubectl rollout status cannot succeed without, and hack/e2e.sh recreates the cluster every run with the apply log reading created rather than unchanged. What that measures is one ingress rule; that kindnet implements no NetworkPolicy controller at all, in either direction, is its documentation rather than anything observed here. So 6b ships objects, asserts them as objects, and says so in the tests' own comments. Whether they enforce anything is a property of a cluster's CNI that no run here has tested on any CNI — including the design's one portability trap, whether a pod-selector egress rule still matches after kube-proxy has DNATed a Service ClusterIP to a pod IP. The RKE2 rollout at the end of milestone 6 is the first thing that can turn these objects into a guarantee.

6c — the expose strategies

Milestone 6c is done: the three expose strategies ProxyGroup.spec.expose has named since milestone 1 all reconcile, not one, and a fourth — ClusterIP, added after the milestone-6 rollout needed it — reconciles beside them. NodePort and LoadBalancer each get a Service built through one CreateOrUpdate, branching on type, external traffic policy and whether a node port is named; HostPort gets none, and a group switching into or out of it makes every proxy pod stale, so the drain-aware rollout milestone 4 built replaces them one at a time. ClusterIP is for a network something else fronts — an ingress controller's TCP entry point, a gateway, a tunnel — and gets only the Service that thing routes to: no routing object and no check that the address it publishes resolves or leads anywhere. The operator now owns only the annotations it itself writes onto a LoadBalancer Service, releasing them the moment a group leaves the strategy and never touching a key MetalLB or kube-vip put there. And ProxyGroup gained its first writer of Degraded: a proxy pod the API server refuses to create, or one the scheduler cannot place, now says why on the object instead of only in the operator's log.

The one thing worth naming is what was actually observed being enforced, because it is exactly one thing and everything else 6c ships is an object. A HostPort pod in a namespace enforcing Pod Security baseline is refused by the API server, and ProxyGroupReconciler reports it — Degraded=True/ReasonProxyPodRejected, carrying the API server's own message — confirmed at two levels: envtest, where Kubernetes 1.36.3 runs the PodSecurity admission plugin for real, and a real kind cluster, where commenting out the enforcing namespace label made the same scenario fail twice, by two different mechanisms, before the label was restored. That measurement forced a real, if narrow, trade elsewhere: theOperatorWasNeverDenied, the check the whole E2E package exists to pass, now excludes any log line containing violates PodSecurity from counting as a denial — without the exclusion, the reconciler's own retries of the refused create would fail that check on every green run, for a rejection this milestone causes on purpose.

How many retries that is, this branch recorded as 3,940 in one 139-second run and filed as the normal cost of a refused create. It was neither normal nor a cost of refusal: it was a hot loop of the milestone's own making, found by the final whole-branch review and by none of the seven task reviews before it. The API server's refusal names the pod it refused, a fresh random suffix is drawn for every attempt, so the stored message differed on every pass, so the status write always bumped resourceVersion, and the ProxyGroup watch turned each of those writes straight back into an enqueue ahead of the rate-limited retry — about 28 reconciles a second for as long as the refusal stood, which for a Pod Security label is forever. The condition is now left byte-for-byte alone while it is saying the same thing, and the same E2E run logs 15 refusals in 143 seconds, which is what exponential backoff alone predicts. What the message carries is unchanged: the API server's own words, verbatim, because the remedy is in them.

What it leaves open is that nothing in 6c demonstrates that a client can reach a proxy. No LoadBalancer controller runs anywhere in this repository, no image in the E2E manifest resolves so no container process ever runs, and the one thing observed being enforced is the API server's refusal of a HostPort pod, not anything about a connection succeeding. The LoadBalancer address path — an assigned ingress plus a ready pod producing a non-empty status.address — is now driven through a live reconcile in envtest, which is where the same final review found it had never been driven at all: the E2E's own LoadBalancer scenario writes the ingress entry itself and asserts only that status.address stays empty while no proxy is ever ready, and until that envtest was written the whole package stayed green with the Service severed from the status it is read out of. What that envtest proves is the wiring, in an API server with no load balancer controller and no kubelet: it writes both the ingress entry and the pod's readiness itself. The honest verb for what 6c ships is that the operator publishes an address once conditions hold, or that an object exists — not that anything reaches or works. The RKE2 rollout at the end of milestone 6 is the first thing that can test any of the four strategies against a client that actually tries to connect.

6d — the operator installs by a Helm chart

Milestone 6d is done: the operator installs by a Helm chart, and config/deploy/ — the seven flat manifests it replaces, six of which hard-coded the operator's namespace by hand, which every handover since 6a has called the single most likely way this project ships something that works on the author's machine and nowhere else — no longer exists. charts/spawnery/ renders every object that directory used to hold, plus a ClusterRole and namespaced Role generated from config/rbac/role.yaml and four CRDs generated from config/crd/bases/, both by a second half make manifests gained, hack/chart-templates.sh. internal/rbacaudit's whole suite now audits what helm template actually renders instead of an intermediate on disk, and make e2e proves the chart moves by installing it into platform-system, a namespace chosen to share nothing with the chart's own documented default, spawnery-system — eighteen scenarios still pass, theOperatorWasNeverDenied still last. One file stays outside the chart on purpose: config/rbac/forwarding-secret-reader.yaml, the per-game-namespace grant milestone 5c kept out of config/deploy/ for the same reason a chart installed once still cannot know a namespace a user creates later, and it carries a manual edit an operator installed anywhere but the default namespace has to make by hand — charts/spawnery/README.md now says so in its installation steps.

The one thing worth naming is what theOperatorWasNeverDenied turned out to guard, because a mutation showed it narrower than this milestone's own design document claims. A spawnery-system literal leaked into the chart's own RoleBinding namespace was caught, but by Kubernetes' own admission check refusing a namespace that does not exist — before the Go test suite ever ran, not by the check the design credited. The equivalent leak in a RoleBinding's subject namespace, the shape that actually matches the hazard in forwarding-secret-reader.yaml, was never mutated, so the claim that theOperatorWasNeverDenied catches it stands as reasoning, not measurement. docs/handover-milestone-6d.md has the full account. And no helm upgrade has ever run: the four CRDs carry helm.sh/resource-policy: keep precisely so that a future upgrade would carry a CRD schema change through and an uninstall would not destroy every Network in the cluster — helm uninstall leaving them standing was driven once, against a real cluster, and observed; upgrade was not driven at all.

6e — CI

Milestone 6e is done: GitHub Actions now blocks a pull request on four jobs, watches one more on its own schedule, and holds a third ready for a tag. .github/workflows/ci.yml's test, lint, deps and e2e jobs run on every pull request and on push to master; .golangci.yml pins errcheck and staticcheck with max-issues-per-linter: 0 and max-same-issues: 0, closing a default output cap that otherwise shows a sample and lets it be trusted as a total; deps regenerates agent/deps.json against a real Maven Central and fails on a non-empty diff, paying a debt docs/known-issues.md has carried since milestone 2c and shown, by a deliberate corruption and revert, to actually fire; e2e runs hack/e2e.sh completely unmodified on a hosted runner's Docker daemon, eighteen scenarios green in 7m00s on the first attempt. .github/workflows/nightly.yml re-runs milestone 6a's bit-identical-rebuild acceptance criterion on a schedule — driven once, by a temporary trigger, 9m21s green. .github/workflows/release.yml waits for a v* tag to publish for real; no tag has been pushed by this milestone, and none should be — that is the repository owner's decision, not this one's.

Two scheduled workflows have been added since, each closing a gap this milestone left and each driven before it was believed rather than after. .github/workflows/paper-watch.yml asks daily whether PaperMC has published a build newer than nix/paper.nix names, and turns a newer build into an issue rather than a red run, because it persists until somebody acts; the first thing it found was that the pin was eight builds behind. .github/workflows/e2e-podman.yml runs hack/e2e.sh under rootless podman — the invocation a person uses, which e2e above does not exercise, since a hosted runner has a Docker daemon — and most of it is teaching a runner to accept that invocation at all: dispatched 2026-08-27, green in seven minutes against podman 5.8.4. On the same day the if: failure() half of nightly.yml, which had never run because this project's only red night predated it, was driven for real on a throwaway branch carrying one wrong character in a fixed-output hash: the step opened its issue, the release gate refused with exit 1 against the live repository, and the green run dispatched afterwards closed it.

The one thing worth naming is what this milestone's own first cold-cache run found, because it is the best evidence in this repository for what CI is worth. Three local runs and two independent reviewers had all reported make lint clean going into this milestone's own fix round. CI's first lint run, on a runner with no cache to answer from, found five real SA1019 findings in internal/controller/setup.go every one of those five checks had missed — a stale golangci-lint cache, not a stale tree, had been answering "clean" for all of them. The count this milestone had already written down as final, 33, was measured against that same stale cache and was short by exactly those five: the true figure, from a cleared cache, is 38. Fixing them forced a real migration — internal/controller's five event recorders off the deprecated record.EventRecorder onto events.EventRecorder, twenty-three call sites, twenty-one fake recorders — which needed its own RBAC grant and surfaced a length limit, events.k8s.io/v1's 1024-byte note cap, the old API never enforced; five call sites needed it and were fixed, a sixth is recorded open in docs/known-issues.md.

What it leaves open is that running a check more often does not make the check itself see more. chart-lint still catches neither half of a typo'd {{ .Release.Namespace }}, measured by milestone 6d, and putting every check in this repository behind a schedule instead of a memory changes nothing about what any one of them can see. CI introduces its own version of that limit on day one: make e2e on a hosted runner proves hack/e2e.sh under Docker, and this milestone left nothing automatic proving it under the rootless Podman the author's machine has always run it under — CI proved Docker, the author's machine proved podman, and neither proved the other. It also left three paths that existed only on paper: nightly.yml's if: failure() reporting, release.yml in full, and the podman run itself.

All three have since been driven, and the paragraph above is kept as what the milestone left rather than as what is true now. .github/workflows/e2e-podman.yml runs the podman path nightly, dispatched green on 2026-08-27; release.yml has published three releases; and the if: failure() step was fired on 2026-08-27 by a throwaway branch carrying one wrong character in a fixed-output hash, which opened its issue and made hack/require-no-red-nightly.sh refuse a release for real. Each of those workflows now carries what it was driven with, in its own header.

The RKE2 rollout

Milestone 6 is done: the RKE2 rollout has been driven, from docs/runbook-milestone-6-rollout.md, against a three-node cluster running RKE2 v1.34.3 and Cilium v1.18.4. Spawnery installs through Flux from the chart at tag v0.1.0 — two Kustomizations, the second gated on the first — and runs in spawnery-system under Pod Security restricted, from a digest, with no pull secret anywhere in the chain. A Network, a lobby of ephemeral Paper servers and two Velocity proxies run in a second restricted namespace: the first time either image has run in any cluster, because test/e2e/manifests/e2e.yaml names unresolvable tags on purpose. Milestone 6b's NetworkPolicy was enforced for the first time in this project's history and holds in both directions, measured with two probes that differ in one label. HostPort binds under Cilium's portmap chaining, measured by a TCP connect with a control on an unbound port.

Three of the run's results are worth naming because the cluster found what reading had not. No git tag can carry its own operator digest — the digest comes from skopeo copy --digestfile and so cannot exist until after the tag is published, which makes charts/spawnery/values.yaml's value permanently one release behind and made the design's own acceptance criterion unachievable as written. A denied uncached read is silent in the operator's log: revoking one RoleBinding produced twenty 403s, a condition carrying its own remedy, and not one line containing is forbidden: — settling a question milestone 6's handover had explicitly left as an unestablished hypothesis, and finding the place such a denial is visible, which is a metric rather than a log. And Cilium will not share a load balancer address between two Local Services that select different pods, which cost the design its plan for reaching the network and is a property of Local rather than a misconfiguration.

What it leaves open is one thing with three faces: TCP 25565 does not reach this cluster from outside, and neither does a NodePort on an address that serves HTTPS all day, so the filtering sits in front of the cluster and not in anything spawnery configures. Without it there is no real join, no player for the PodDisruptionBudget to protect, and nothing to drive the adding half of the occupied label. All three unblock together.

Since the rollout

Since the rollout, one thing has been closed that predates every milestone above it: milestone 2a promised that a compromised game server pod cannot harm any other, and the availability half of that promise had no bound at all. 6b gave the channel MaxConcurrentStreams, an idle reaper and a TokenReview cache, and none of the three reaches the actual attack — a pod opening connections without limit, each carrying a valid token and a live stream, so that the idle reaper never fires and the rate limit never misses its cache. internal/agentserver now bounds connections per peer address on the listener itself, which is before the TLS handshake and therefore before the expensive half of a connection is paid for.

The number is measured rather than chosen. A legitimate agent's peak is 2, because renewal is make-before-break and each attempt builds its own channel; cmd/spawnery-stubop counts connections for exactly this, and the high-water mark was 2 in every run across the plain renewal, the operator-supersede path, the give-up path and the Velocity agent — roughly seventy renewals, never a third connection. The bound is 8, four times that, and hack/agent-test.sh now asserts the peak against it, so an agent change that needs more fails there rather than by being refused in a cluster. spawnery_agent_open_connections and spawnery_agent_connections_refused_total publish both halves, and the chart's PrometheusRule alerts on the second.

What that does not close is in docs/known-issues.md unchanged: the bound is per peer, so a set of compromised pods is bounded only by their number.

The second thing closed since the rollout is half of the drain's oldest gap, and the measurement is the interesting part. A player whose connection to a draining server is still in flight is counted by nobody, so DrainPlayers moved everyone except them and the operator then read an empty server. The obvious fix — have the proxy report its own playersConnected — would not have worked: disassembling velocity 3.5.1 build 615, VelocityRegisteredServer.addPlayer is called from exactly one place, the backend's play phase, so the proxy does not count such a player either. Neither would simply widening the drain's filter, because ConnectionRequestBuilderImpl refuses a move while a connection is in flight and would have made the change a silent no-op.

So the agent catches them on arrival instead. Drain now remembers which servers are draining and moves whoever lands on one, driven from ServerPostConnectEvent — the first point at which Velocity has cleared the in-flight connection and a move actually takes. The set is rebuilt from what the operator says rather than aged out on a timer: a resync is a FullSync followed by one DrainPlayers per draining server, so the messages after a FullSync are a complete statement, and a drain the operator stops naming is gone within one further resync. It also stops a drain sending players onto another draining server, which the single-name exclusion could not see.

That is the proxy image 0.2.2, and docs/upgrading.md says why it rolls the Paper fleet too and why nothing has to be upgraded in any particular order. The operator's half stays open in docs/known-issues.md: Occupied() still reads only the backend's count.

Handovers

Anyone starting work on this project begins at docs/handover-milestone-6e.md: it says where 6e stopped, what was actually driven versus what only exists, what the next milestone finds in place, and what the RKE2 rollout still owes — carried forward from 6d's own list unchanged. It is written to be read by someone with no memory of how any of this was built. docs/handover-milestone-6d.md, written for 6e and kept because its §3 is the record of what 6e started from and had to decide, docs/handover-milestone-6c.md, written for 6d and kept because its §3 is the record of what 6d started from and had to decide, docs/handover-milestone-6b.md, written for 6c and kept because its §2 and §3 are the record of what 6c started from and had to decide, docs/handover-milestone-6.md, written for 6b and kept because its §2 and §3 are the record of what 6b started from and had to decide, docs/handover-milestone-5.md, docs/handover-milestone-4b.md, docs/handover-milestone-4.md, docs/handover-milestone-3.md, docs/handover-milestone-2c.md and docs/handover-milestone-2b.md are its predecessors, kept as the record of what those milestones started from.