Skip to content

Worlds in an object store

An on-demand group whose worlds live in an S3 bucket instead of on a claim per member. It joins the Network of config/samples/network.yaml, and the installation needs the chart's worldSync values first (below):

apiVersion: spawnery.cloud/v1alpha1
kind: ServerGroup
metadata:
  name: private-servers
  namespace: minecraft
spec:
  networkRef:
    name: production
  type: OnDemand
  image: ghcr.io/spawnery/purpur:26.3-0.34.0
  maxPlayers: 10
  maxInstances: 200
  storage:
    backend: ObjectStore
    # Required by the schema, not enforced on the node; see Limits.
    size: 2Gi
    # Exactly what is synced.
    keep:
      - world
      - plugins/Example/data

Members are started, stopped and deleted as in Private servers. What changes is where the world is between runs, and what a start waits for.

What the backend does

A member gets /data from a directory on the node it runs on, through the CSI driver worldsync.spawnery.cloud. A node agent, spawnery-worldsync, runs on every node as a DaemonSet and fills that directory from the bucket, uploads it back, and holds a lease in the bucket so that a world is written by one node at a time.

At a start, the node agent takes the lease and reads the world's manifest. When its own copy is current, the world is ready at once; otherwise it downloads the files keep matches, 32 requests at a time. The container starts meanwhile, and the Paper agent holds the server back from reading the world until the download is done. It waits 10 minutes at most, and a failed download ends the run with exit code 1.

While the member runs, the agent turns autosave off every snapshot interval, runs save-all flush and asks for a snapshot. The node agent copies what changed to a directory beside the world and answers, autosave goes back on, and the upload runs from the copy in the background.

After the container ends, the node agent takes a last snapshot and answers the kubelet. The upload follows in the background and retries until it succeeds, and then the lease is released. The node keeps its copy as a cache for the next start.

A start on the same node can begin while the previous run still uploads; it waits only for an upload attempt already in flight. A start on another node waits in ContainerCreating until the lease is released, or, if the old node is gone, until the lease has been silent for 10 minutes.

Installing the node agent

The bucket is cluster configuration, set once in the chart; game namespaces never see its credentials. The operator gets the same settings, because it marks deleted worlds in the bucket and sweeps them.

# worldsync-values.yaml
worldSync:
  enabled: true
  namespace: spawnery-worldsync
  objectStore:
    endpoint: https://s3.example.net
    region: eu-central
    bucket: example-worlds
    prefix: ""
    credentialsSecret: worldsync-s3
  snapshotInterval: 5m

The node agent runs privileged: it bind-mounts worlds into the kubelet's pod directories. Its namespace must admit privileged pods, so it cannot be an operator namespace held to the restricted Pod Security level. The chart creates neither the namespace nor the Secret, and the Secret has to exist in both namespaces:

kubectl create namespace spawnery-worldsync
kubectl label namespace spawnery-worldsync pod-security.kubernetes.io/enforce=privileged
for ns in spawnery-system spawnery-worldsync; do
  kubectl create secret generic worldsync-s3 -n "$ns" \
    --from-literal=AWS_ACCESS_KEY_ID="$KEY_ID" \
    --from-literal=AWS_SECRET_ACCESS_KEY="$SECRET"
done
helm upgrade spawnery oci://ghcr.io/spawnery/charts/spawnery --version 0.34.0 \
  --namespace spawnery-system --reuse-values -f worldsync-values.yaml

With enabled: true the operator runs with --world-sync. Without it, a member of an ObjectStore group gets no pod (Accepted is False, reason WorldSyncOff) and deleteServer refuses it.

The DaemonSet tolerates every taint and runs at system-node-critical by default: a member scheduled to a node without the node agent never gets its volume. Narrow it with worldSync.nodeSelector and worldSync.tolerations only together with the game pods' own scheduling. Every value is listed in Chart values.

The store has to support conditional writes, If-None-Match: * and If-Match on PutObject, and accept the ETag in If-Match without quotes, which is the form the node agent sends. Requests use path-style addressing.

What the group needs

The API refuses ObjectStore without type: OnDemand and keep. keep is exactly what is synced, so list every level directory and every plugin directory that holds state. Paper 26.3, which the example's image runs, keeps all dimensions of a world inside its level directory (world/dimensions/minecraft/the_nether and .../the_end) and moves the world_nether and world_the_end directories of older versions in there at its first start, so world covers the nether and the end. On older versions, list world_nether and world_the_end as well.

What keep does not match stays on the node and never reaches the bucket. When a member stops, the node agent deletes those paths, except one that holds a world by the prune's rule (a level.dat, a region directory or an .mca file) and that replace does not list. Such a path stays on the node, every snapshot of a running member fails with spec.storage.keep does not keep <path>, which holds a world, and the next start on that node is refused by the prune as it would be on a claim. The snapshot at a stop still saves what keep matches. Add the path to keep, or to replace if the sources ship it.

The group's image needs to be one of spawnery's game images, 0.24.3 or later, as in the purpur:26.3-0.34.0 of the example. Its Paper bootstrapper is what holds the server back until the world is on disk. An older image, or a custom one without the agent, starts the server on a half-downloaded world, which the server can then corrupt. Before 0.24.3 the entrypoint also ran chmod over the downloaded world, which belongs to the node agent, so a resumed world did not start. The operator cannot tell which agent an image carries.

size, storageClassName, accessModes and annotations stay valid and are ignored, so a group can switch back to Claim without touching the immutable fields. Switching either way moves no data: claims stay where they are, and so does the bucket. A running member keeps the backend it started with, and the switch reaches it at its next start.

The node agent gives what it writes to the pod's fsGroup, which spawnery's server pods always set.

Plugins that touch the world early

Paper reads level.dat and the datapacks before any plugin's onLoad, so the agent waits in a Paper bootstrapper of SpawneryAgent. A plugin's onLoad and onEnable run after that wait and need nothing. A plugin whose own bootstrapper reads or stages world files must have Paper run the agent's bootstrapper first, in its paper-plugin.yml:

dependencies:
  bootstrap:
    SpawneryAgent:
      load: BEFORE

Without it, the plugin may see a world that is still downloading, or write files that the download then replaces.

Snapshots and what a node loss costs

worldSync.snapshotInterval (operator flag --world-sync-snapshot-interval, default 5 minutes) is how often a running member asks for a snapshot. Changing it restarts nothing; members pick it up at their next start.

A snapshot freezes saving for the flush and for a copy on the node's own disk, not for the upload. Files under 64 KiB are copied every time; larger ones only when their size or modification time changed. A file that keeps changing while it is copied fails that snapshot after three attempts, and the next one tries again.

The agent waits at most 60 seconds for the saves, the flush and the copy together, and writes the end of that wait next to the request (snapshot.deadline, a wall-clock time). The node agent refuses a request it picks up after that time. A copy that started in time but finishes after it is dropped and never queued for upload, though the node still reads the world until that copy ends. Agent and node agent run on one node and read its clock; a clock stepped back by an administrator or NTP while a snapshot is in flight lets a late copy through by up to the size of the step. A request without a deadline, from an older agent, is copied as before.

If a node dies, its running worlds lose the play since their last snapshot that finished uploading: at most the interval, plus an upload that was still running. After 10 minutes another node may take the lease and start the world from the bucket. When the dead node comes back, it finds its leases taken and moves its copies to orphans/ under the node directory (worldSync.hostPath), where they stay for manual recovery and never upload. A bucket that stays unreachable for more than 10 minutes after a stop has the same effect: another node may take the lease and start from the last upload, and the stopped run's copy goes to orphans/ once the bucket answers again.

A stop always uploads, whether or not the agent ever asked for a snapshot.

Deleting a world

deleteServer(group, key) writes a deletion marker into the bucket and then deletes the member's Server. The operator sweeps marked worlds once a minute, as soon as no node holds their lease. Until the sweep is done, startServer for the same key answers UNAVAILABLE; after it, the key starts an empty world. A new world whose first upload is still running when its Server is already gone still counts as a world, since the node agent holds its lease from the mount on. The delete marks it, and the sweep removes the upload once it lands. Without --world-sync, startServer refuses every key of an ObjectStore group.

Older generations

Each upload of a member's world makes a new generation: about every five minutes while the member runs, and once more when it stops. By default the bucket keeps only the newest. storage.retention keeps older ones, using the prune options of Proxmox Backup Server:

spec:
  storage:
    backend: ObjectStore
    keep: [world]
    retention:
      last: 12    # the newest 12, the current one included
      hourly: 24  # then the newest of each of 24 more hours
      daily: 7
      weekly: 4
      monthly: 3
      maxAgeDays: 30  # and none replaced more than 30 days ago

The options apply in the order last, hourly, daily, weekly, monthly, yearly. Each counts only periods that hold a generation, skips a period an earlier option already kept a generation from, and keeps the newest generation of each further period. Weeks are ISO weeks and every period is UTC. A member nobody plays makes no generations, so the counts leave its history as it is.

maxAgeDays is a limit on top of the counts. An older generation goes once the generation that replaced it is more than that many days old, whatever the counts would keep; the current generation always stays. It measures from the replacement, not from the generation's own time: a state that a reset in game or a restore replaced stays a restore point for maxAgeDays after that. Unset, there is no limit; when set, it is at least 1.

The counts only prune where a generation is written, so they never reach a world nobody starts again. A world that was large, was reset in game and then sat idle keeps its large generations for good under last alone. maxAgeDays covers that case: once an hour the operator lists each group that sets it, and for a world with a generation past the limit it takes the world's lease, prunes it by the group's whole policy and releases the lease again. It skips a world whose lease a node holds, which is a member that runs or is still uploading, and a world marked for deletion. A world whose current state comes from a restore keeps the state the restore replaced until its next upload, as without maxAgeDays. The objects this pass deletes count in spawnery_world_aged_objects_total. The pass starts an hour after the operator, and only the leader runs it.

A kept generation costs what its upload cost: the region files that changed and one pack of the small files. On a pregenerated world of about 2.75 GB, two players changed 8 region files (41.7 MB) within an hour; a member nobody played changed one file of 0.7 MB.

The operator writes the policy to .retention/<namespace>/<group>.json in the bucket, and the node agent reads it before each upload. A change reaches each world at its next upload and restarts no member. Removing retention drops a world's history at its next upload. Deleting the group leaves the file in place, so its worlds keep their history. The API refuses retention without backend: ObjectStore, so a group goes back to Claim only after dropping it.

A plugin lists a member's generations with restorePoints(group, key), whether the member runs or not, and also while its world is being deleted. The listing's settling() is true while the world may still change: while the member runs or is being stopped, and after a stop until the node's upload of the final snapshot releases the world's lease, which can take tens of seconds. Until then the list can miss that snapshot and a restore is turned away. restoreWorld(group, key, generation) makes one of them current while the member is stopped. See what a plugin can do. The restore writes the old content as a new generation, and the generation that was current stays a restore point whatever the retention says, so the restore can be undone. The member's next upload prunes by the retention alone: it keeps that generation under last: 3 or more (the upload, the restored generation and the replaced one), or when another option keeps it from an earlier period than the upload's. The next start downloads the whole world, even on a node that holds most of it. A restore takes the world's lease under a name of its own, so a second restore of the same world at the same time answers UNAVAILABLE. If the operator dies mid-restore, the world stays held until that lease goes stale after 10 minutes, and starts and further restores answer UNAVAILABLE meanwhile. Each restore records the event WorldRestored on the group and counts in spawnery_world_restores_total.

Moving existing worlds in

spawnery-worldsync import uploads the keep paths of a directory as the first generation of a world, and refuses a world that already has a manifest. Run it as a Job that mounts a stopped member's claim read-only. The claim is in the game namespace, so the Job runs there and needs the credentials Secret there for the import; delete it afterwards.

apiVersion: batch/v1
kind: Job
metadata:
  name: import-0b5c1c82
  namespace: minecraft
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: import
          image: ghcr.io/spawnery/spawnery-worldsync:0.34.0
          args:
            - import
            - --world=minecraft/private-servers/0b5c1c82-4c7f-4a6e-9d1b-2c1f2b9d5e10
            - --dir=/world
            # spec.storage.keep, one entry per line
            - |-
              --keep=world
              plugins/Example/data
          env:
            - name: WORLDSYNC_ENDPOINT
              value: https://s3.example.net
            - name: WORLDSYNC_REGION
              value: eu-central
            - name: WORLDSYNC_BUCKET
              value: example-worlds
            - name: WORLDSYNC_PREFIX
              value: ""
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef: {name: worldsync-s3, key: AWS_ACCESS_KEY_ID}
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef: {name: worldsync-s3, key: AWS_SECRET_ACCESS_KEY}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          volumeMounts:
            - name: world
              mountPath: /world
              readOnly: true
            # The image has no /tmp; the import stages its snapshot there.
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: world
          persistentVolumeClaim:
            claimName: private-servers-0b5c1c82-4c7f-4a6e-9d1b-2c1f2b9d5e10-data
            readOnly: true
        - name: tmp
          emptyDir: {}

The world name is <namespace>/<group>/<key>, the claim <group>-<key>-data. The import copies the world into /tmp before uploading it, so the emptyDir needs room for one world.

Import every world before the group switches to ObjectStore, with its member stopped. A member that starts under ObjectStore before its import begins an empty world, and once that world has uploaded, the import refuses the key. There is no export in this version.

Watching it

The node agents serve metrics on port 8090. With world sync on, the chart's metrics switches cover them too: serviceMonitor adds a PodMonitor for the agents, prometheusRule the spawnery-worldsync alerts, and dashboard a second dashboard, Spawnery world sync, beside the network one. The metrics and alerts are listed in metrics and alerts.

The dashboard shows per node the worlds mounted and cached, snapshots not yet in the bucket, download and upload times and counts, objects pruned, retries, lease conflicts, orphans and failed downloads, and the agents' network traffic, CPU and memory from the kubelet's cAdvisor series.

Limits

  • storage.size is not enforced on the node directory. A world can use as much of the node's disk as it writes.
  • Only OnDemand groups.
  • The cache is per node. A start on another node downloads the world even if a third node holds a current copy. A cache nobody uses is evicted after 24 hours, or earlier, oldest first, when the node directory's filesystem has less than 15 % free.
  • What an upload attempt that failed wrote stays in the bucket until the world's next sweep: every twelfth upload, and when a node releases the world.
  • A change to the counts does not prune worlds nobody plays, until one of their generations passes maxAgeDays.
  • The history holds what the snapshots held. Play after the last snapshot before a crash is in no generation, and a deleted world cannot be restored.

The design and its measurements are in docs/superpowers/specs/2026-10-06-object-store-worlds-design.md, docs/superpowers/specs/2026-10-08-world-history-design.md and docs/superpowers/specs/2026-10-20-retention-max-age-design.md.