Development¶

Everything builds through Nix. nix develop puts Go, controller-gen, the envtest assets, kubectl, kind, k3d, protoc with its Go and Java plugins, a JDK 21 and Gradle on the path at the versions this repository pins, so nothing here depends on what is installed on the machine.

nix develop
make test              # unit and envtest tests
make build             # bin/spawnery-operator
make agent             # both agent plugins and their JUnit suites
make e2e               # the driven run: the operator in a real kind cluster

The targets¶

The first six are the commit loop and run anywhere. Everything from agent-test down needs a container runtime and only works on x86_64-linux; pass CONTAINER=podman if docker is not your runtime. Three reach the network and are therefore part of no other target, not even make all: agent-deps, publish and publish-chart.

Generated code¶

make proto regenerates the Go code under internal/agentpb from proto/spawnery/agent/v1alpha1/agent.proto. It is checked in like zz_generated.deepcopy.go, and make test does not regenerate it: after a change to the .proto, run make proto and commit the diff with it.

The agent plugins¶

make agent (nix build .#agents) builds Paper's plugin and Velocity's (they share the session loop, token source and channel construction in agent/common) and runs both JUnit suites as the derivations' check phases.

agent/deps.json is the checked-in lockfile pinning every Maven artifact by hash across the Gradle subprojects. make agent-deps regenerates it, and is needed only when a build.gradle.kts under agent/ changes a dependency: it reaches Maven Central, and a Nix build must never depend on the network.

make agent-test runs both real images against the Go stub operator in cmd/spawnery-stubop and checks the handshake, the authorization header, the player reports, the overlapping renewal and the bound on a session the operator never answers. For the Velocity image it also checks that its readiness port stays closed until a server list has arrived and opens once one does.

AGENT_TEST_ONLY=snapshot runs the world-sync snapshot phase alone, beside AGENT_TEST_ONLY=voice for the voice phase.

The images¶

make image-test runs the game images (Purpur 26.3 and Velocity) offline under the same constraints the podspec imposes, loading each first so the target needs no separate build step of its own. hack/image-test.sh asserts Paper's behaviour (config rewritten, plugin loaded, nothing downloaded at start), which Purpur, a Paper fork, keeps.

The Paper image was deprecated in 0.2.15 and is no longer built from 0.23.0; see the release notes. nix/paper.nix stays: Purpur patches with its Mojang jar, and internal/render's tests read its repo.

Reproducibility¶

make image-repro is the standing check that the images are reproducible, worth running again after any change to nix/paper.nix, nix/purpur.nix or nix/purpur-image.nix.

The plain build in front of each --rebuild is not redundant. --rebuild compares a fresh build against the output already in the store, and with nothing there it does not fail the check, it declines to run it: "some outputs … are not valid, so checking is not possible". All three image derivations take the working tree as their source: appending one line to a file in docs/ was measured to change the derivation hash of purpur-image, velocity-image and operator-image alike (agents was unaffected). So an edit almost anywhere empties the store of them.

The operator's image¶

make operator-image-test runs the operator image under the constraints charts/spawnery/templates/deployment.yaml imposes (non-root and a read-only root filesystem) rather than more comfortable ones, plus --network none, which is the script's own choice and not the Deployment's, and cheap here because the run only asks the binary to print its usage. make image-repro covers this image beside the other two and the agent jars.

Publishing¶

Five artefacts, three scripts, none of them part of another target.

make publish (hack/publish.sh) copies the images from their Nix archives straight to ghcr.io/spawnery/ with skopeo, so the registry gets what the flake describes, not what a previous podman load left in a local store. It needs a GitHub token with write:packages. DRY_RUN=1 still builds every image it was asked for (on a machine without them cached, that is the expensive part), then prints what it would copy where, needing no credential. FORCE=1 overwrites a tag that already exists, which it otherwise refuses to do with exit 3. WRITE_DIGEST=1 writes the digest skopeo copy reported into charts/spawnery/values.yaml's image.digest key. The chart is the only installation form, so the only place a digest means anything.

Each Purpur image also gets its startup cache. Before copying the image, the script loads it into a local container store (CONTAINER, default docker), trains it with hack/aot-train.sh (two boots of the server) and pushes the result as ghcr.io/spawnery/purpur-aot:<tag> first: an operator running with --aot-cache mounts that image, and its servers do not start while it cannot be pulled. DRY_RUN=1 trains too. Canvas gets no startup cache.

Every image carries org.opencontainers.image.source naming this repository, which links a new ghcr.io package to it. ghcr.io/spawnery/canvas was created by the v0.30.0 release at 06:45 UTC on 2026-10-27 and answered an anonymous manifest request for 26.3-0.30.0 with 200 at 07:29 UTC. purpur-aot, created on 2026-10-06, answered the same request for 26.3-0.30.0 with 200 on 2026-10-27; its visibility right after creation was not measured. If an anonymous pull of a new package answers 401, set it to public in the organisation's package settings.

make publish IMAGES=operator-image publishes one image, the ordinary case: flake.nix keeps operatorVersion apart from imageVersion, so a reconciler fix makes exactly one tag new. Asking for all three stops at the first tag already published and never reaches it; FORCE=1 gets past that only by re-pushing about 1.4 GB over tags that were right. .github/workflows/release.yml therefore invokes the script once per image on a v* tag.

make publish-chart (hack/publish-chart.sh) pushes the chart to oci://ghcr.io/spawnery/charts/spawnery. It packages from git archive HEAD, not the working tree, because on the release runner those two stop being the same file: WRITE_DIGEST=1 rewrites charts/spawnery/values.yaml minutes before the chart step runs, and archiving HEAD makes the ordering irrelevant instead of a comment somebody must keep obeying. It also refuses, with no FORCE=1 escape, to publish a chart whose committed image.digest is non-empty. That is the one state nothing else catches, because internal/rbacaudit's TestTheOperatorImageIsNotAMutableTag returns early when a digest is set instead of failing. Its "already there" refusal is exit 3 too, the ordinary outcome here since most tags change nothing under charts/. make publish-chart-test drives nine cases past it, none against a registry.

hack/publish-api.sh publishes cloud.spawnery:spawnery-api to Maven Central, the only artefact that is not a container, so a plugin can compile against the API without a checkout. It is a script rather than a Gradle plugin because any plugin speaking the Central Portal's HTTP API is third-party and would enter agent/deps.json, to save one curl. Two of its inputs are secrets nobody can grant from inside a workflow: a Central Portal token pair and an ASCII-armoured signing key, both belonging to a person. release.yml therefore skips this step rather than failing it when they are absent, and says which artefact it left out; a hard failure would hold every image and the chart hostage to a secret that has nothing to do with them. DRY_RUN=1 builds the bundle, prints what would go where, and needs neither.

A tag whose whole change is under charts/ publishes a chart and no image, and is a correct release. Both image numbers therefore have gaps, and none is a miscount: imageVersion reads 0.2.5, 0.2.7, 0.2.9, 0.2.10, 0.2.12, 0.2.13, operatorVersion reads …, 0.2.9, 0.2.11, 0.2.12, and the chart's own version tracks neither: it moves whenever anything under charts/ does, while its appVersion stays with the operator it deploys. A missing number is the record of a release that built nothing on that side. A local make publish is for the case a tag cannot cover.

The end-to-end run¶

make e2e (hack/e2e.sh) installs charts/spawnery with helm install --create-namespace into platform-system, a namespace sharing nothing with the chart's documented default, spawnery-system. The CRDs come from that install too, so there is no separate apply. Its Go test package drives the operator under its own ServiceAccount, then reads the whole operator log and fails on is forbidden:. The operator runs in the cluster here, from its own image, so nothing hand-builds a Service; that is the difference between this and the local flow below. Under rootless Podman the invocation is:

systemd-run --scope --user --property=Delegate=yes -- \
  nix develop -c env KIND_EXPERIMENTAL_PROVIDER=podman make e2e

E2E_KEEP=1 leaves the cluster standing afterwards and prints its KUBECONFIG; a failed run dumps the operator log, the objects and the events before tearing down.

Running this image accepts Mojang's EULA on your behalf: the entrypoint writes eula=true, because Paper does not start otherwise.

Trying it locally against kind¶

This is the hand-driven flow, running the operator outside the cluster through go run. It gives what make e2e does not (a real Paper image, a server that reaches Ready, an agent that reports players) at the price of the workarounds below, which make e2e avoids because its Service has a selector.

Under rootless Podman (measured with 5.8.4) k3d cannot bring up a cluster at all: its tools node bind-mounts the runtime socket to /var/run/docker.sock inside itself, and rootless Podman refuses to create that mount point (mkdir /var/run/docker.sock: permission denied); no DOCKER_HOST fixes it, the failure being in the tools node's own container creation rather than in the client reaching the socket. The flow below uses kind under KIND_EXPERIMENTAL_PROVIDER=podman, which works against the same socket.

kind needs cgroup delegation under systemd as a regular user, hence the systemd-run --scope --user --property=Delegate=yes wrapper around every kind command below: without it kind refuses with a Delegate=yes error even when that property is already set on the user's systemd service; the scope is what its check looks for.

Running through go run there is no POD_NAMESPACE from the downward API, so --operator-namespace has to be set explicitly; without it the process refuses to start (see validateAgentFlags), because the serving certificate would otherwise carry the wrong SANs.

One gap decides whether a Server reaches Ready: the pod dials spawnery-operator.<ns>.svc:9443 and nothing creates that Service, the operator being outside the cluster where no selector could find it. A selector-less Service with a hand-written Endpoints pointing at the host closes it. The serving certificate already carries that DNS name, so TLS verifies against the CA the pod was given.

Which address goes into those Endpoints depends on the runtime. With a real Docker daemon it is the bridge gateway, 172.17.0.1. Under rootless Podman it is none of the obvious candidates: the gateway of the kind network (10.89.0.1 here) lives inside the rootless network namespace, where the operator is not listening and a connection is refused, and the one address that does reach the host (the pasta link-local 169.254.1.2, which Podman also publishes as host.containers.internal) is rejected by the API server in both Endpoints and EndpointSlice with may not be in the link-local range. What works, measured, is one more container on the same Podman network relaying to the host: it gets a routable address on that network, and pods reach it.

systemd-run --scope --user --property=Delegate=yes \
  env KIND_EXPERIMENTAL_PROVIDER=podman \
  nix develop -c kind create cluster --name spawnery-dev
systemd-run --scope --user --property=Delegate=yes \
  env KIND_EXPERIMENTAL_PROVIDER=podman \
  nix develop -c kind load docker-image ghcr.io/spawnery/purpur:26.3-0.22.0 --name spawnery-dev
nix develop -c kubectl apply -f config/crd/bases
nix develop -c kubectl apply -f config/samples/network.yaml
# --leader-elect=false is not needed to *start*, but it is what you want here:
# with leader election on, a local run contends for the same lease as an
# operator already installed in that namespace and waits it out first.
nix develop -c go run ./cmd/spawnery-operator --leader-elect=false --operator-namespace minecraft &

# Rootless Podman only. With a real Docker daemon, skip this and use
# 172.17.0.1 as the endpoint address below.
podman run -d --name spawnery-relay --network kind \
  -v /nix/store:/nix/store:ro \
  --entrypoint "$(nix build --no-link --print-out-paths nixpkgs#socat)/bin/socat" \
  ghcr.io/spawnery/purpur:26.3-0.22.0 \
  TCP-LISTEN:9443,fork,reuseaddr TCP:host.containers.internal:9443
RELAY_IP=$(podman inspect spawnery-relay \
  --format '{{.NetworkSettings.Networks.kind.IPAddress}}')

nix develop -c kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: spawnery-operator
  namespace: minecraft
spec:
  ports:
    - name: agent
      port: 9443
      targetPort: 9443
      protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: spawnery-operator
  namespace: minecraft
subsets:
  - addresses:
      - ip: $RELAY_IP
    ports:
      - name: agent
        port: 9443
        protocol: TCP
EOF

sleep 90
nix develop -c kubectl get networks,servergroups,servers,pods -n minecraft

The relay needs only a rootfs, which is why the Paper image stands in for one; socat comes out of the mounted Nix store.

The first server can take a good half minute: if the ServerGroup meets its network before the Network controller has accepted it, it retries only after networkRetryInterval (30 seconds). The 90 seconds also cover Paper's own start (about seven seconds to a first answered ping) and the agent's handshake. Loading the image beforehand is its own wait: at 26.2-0.2.1 the Paper image is 372 MB as a tarball and the Velocity one 170 MB. They were 735 MB and 533 MB until 2026-08-25, when both stopped shipping a whole headless JDK and started shipping a runtime jlink'd to the modules each actually resolves (see nix/paper-jre.nix and nix/velocity-jre.nix).

Expected, as measured on 2026-08-10 against kind v1.36.1 under rootless Podman:

• network production with Accepted=True and SERVER GROUPS 1,
• servergroup lobby in phase Ready with READY 1 and FREE SLOTS 100; READY is status.readyReplicas,
• a pod lobby-xxxx in Running with READY 1/1: the readiness probe spoke a real server list ping to a real Paper process,
• a server lobby-xxxx in phase Ready with SLOTS 100, PLAYERS 0 and REGISTERED true. SLOTS is what the agent reported from SPAWNERY_MAX_PLAYERS, PLAYERS what it counted: zero, because nobody can join yet.

If the Server stops in Starting, the agent cannot reach the operator; kubectl logs on the pod shows the reason, so far always the Service/Endpoints pair above rather than the agent.

The session renews after eight minutes (--agent-session-renew-after), and if the replacement stream did not overlap the outgoing one, the server would drop out of Ready on that rhythm. Measured over thirteen minutes:

nix develop -c kubectl get server lobby-xxxx -n minecraft \
  -o jsonpath='{.status.readinessLosses} {.status.readySince} {.status.playersUpdatedAt}'
# 0 2026-08-09T22:37:36Z 2026-08-09T22:49:57Z

readinessLosses still zero and readySince still the original timestamp, while playersUpdatedAt keeps moving. The renewal happened and cost the server nothing.

Afterwards, clean up:

kill %1
podman rm -f spawnery-relay
systemd-run --scope --user --property=Delegate=yes \
  env KIND_EXPERIMENTAL_PROVIDER=podman \
  nix develop -c kind delete cluster --name spawnery-dev