Metrics and alerts¶
Generated from two sources that only agree with each other because this page checks it: the Prometheus metrics the operator and the world sync node agent actually register, read from their registries rather than from source, so this table can't drift from what is really exported; and the alerting rules in charts/spawnery/templates/prometheusrule.yaml, rendered through helm template because that file is a Helm chart template, not YAML, until Helm has expanded it.
Metrics¶
The spawnery_worldsync_ metrics come from the world sync node agents on port 8090, the rest from the operator on port 8080.
| Name | Type | Labels | Help |
|---|---|---|---|
spawnery_agent_answers_dropped_total | counter | none | Answers to agent requests dropped because the pod's mailbox was full. |
spawnery_agent_auth_failures_total | counter | role | Refused agent streams, by role. |
spawnery_agent_connections_refused_total | counter | bound | Connections refused for exceeding a connection bound, by bound. |
spawnery_agent_open_connections | gauge | none | Connections open on the agent endpoint. |
spawnery_agent_open_streams | gauge | role | Open agent streams, by role. |
spawnery_agent_rate_limited_total | counter | none | Token checks refused by the per-peer rate limit. |
spawnery_agent_rejected_reports_total | counter | role | Agent reports discarded as implausible, by role. |
spawnery_agent_requests_refused_total | counter | reason | Agent requests the operator declined, by reason. |
spawnery_agent_token_review_cache_hits_total | counter | none | Token checks answered without asking the API server. |
spawnery_agent_token_review_cache_misses_total | counter | none | Token checks that required a TokenReview. |
spawnery_agents_expected | gauge | none | Managed pods that ought to hold an agent connection. |
spawnery_ca_expiry_timestamp_seconds | gauge | none | NotAfter of the CA currently signing the serving certificate, in Unix seconds. |
spawnery_ca_key_unmatched | gauge | none | 1 while the CA key in the operator's TLS secret matches no certificate in ca.crt and the operator refuses to issue a new CA, 0 otherwise. |
spawnery_ca_rotation_blocked_namespaces | gauge | none | Namespaces holding a Network whose CA ConfigMap does not yet carry the incoming CA. |
spawnery_ca_rotation_phase | gauge | phase | 1 for the CA rotation phase currently in effect, 0 for the others. |
spawnery_group_free_slots | gauge | namespace, network, group, type | Free playable seats of the group. |
spawnery_group_players | gauge | namespace, network, group, type | Players on the group's servers. |
spawnery_group_servers | gauge | namespace, network, group, type | Servers of the group. |
spawnery_group_servers_ready | gauge | namespace, network, group, type | Ready servers of the group. |
spawnery_network_changeovers_in_flight | gauge | namespace, network | Groups of the network holding a changeover budget place. |
spawnery_network_changeovers_waiting | gauge | namespace, network | Groups of the network waiting for a changeover budget place or an earlier stage. |
spawnery_network_players | gauge | namespace, network | Players on the network's proxies. |
spawnery_permissions_missing | gauge | scope | Permissions the operator needs and the API server says it lacks, by scope. |
spawnery_proxy_heap_max_bytes | gauge | namespace, network, group, proxy, node | The proxy's JVM max heap. |
spawnery_proxy_heap_used_bytes | gauge | namespace, network, group, proxy, node | JVM heap in use on the proxy. |
spawnery_proxy_players | gauge | namespace, network, group, proxy, node | Players on the proxy. |
spawnery_proxy_sessions_cut_total | counter | none | Proxy sessions ended because the session fell too far behind. |
spawnery_server_heap_max_bytes | gauge | namespace, network, group, server, node | The server's JVM max heap. |
spawnery_server_heap_used_bytes | gauge | namespace, network, group, server, node | JVM heap in use on the server. |
spawnery_server_mspt | gauge | namespace, network, group, server, node | The server's mean tick time in milliseconds. |
spawnery_server_phase | gauge | namespace, network, group, server, node, phase | 1 for the server's current phase. |
spawnery_server_playable_slots | gauge | namespace, network, group, server, node | The server's effective playable slots. |
spawnery_server_players | gauge | namespace, network, group, server, node | Players on the server, as its agent last reported. |
spawnery_server_sessions_cut_total | counter | none | Backend sessions ended because the session fell too far behind. |
spawnery_server_slots | gauge | namespace, network, group, server, node | The server's slots, as its agent last reported. |
spawnery_server_tps | gauge | namespace, network, group, server, node | The server's one-minute ticks per second. |
spawnery_serving_cert_expiry_timestamp_seconds | gauge | none | NotAfter of the operator's serving certificate, in Unix seconds. |
spawnery_voice_packets_total | counter | namespace, group, reason | Voice chat packets the proxies' relays handled, by namespace and proxy group and by what became of them. |
spawnery_world_aged_objects_total | counter | none | Objects and packs the operator deleted from world history past storage.retention.maxAgeDays. |
spawnery_world_restores_total | counter | result | World restores plugins asked for, by result: restored, refused, unavailable or failed. |
spawnery_worldsync_download_failures_total | counter | none | World downloads that failed; the pod waiting for one fails its start. |
spawnery_worldsync_download_seconds | histogram | none | Time to download a world at publish. |
spawnery_worldsync_lease_conflicts_total | counter | none | Publishes refused because another node held the world. |
spawnery_worldsync_orphans_total | counter | none | Local copies moved aside after the lease was lost. |
spawnery_worldsync_pending_snapshots | gauge | none | Snapshots on this node not yet in the bucket. |
spawnery_worldsync_pruned_objects_total | counter | none | Objects and packs that prunes deleted. |
spawnery_worldsync_retries_total | counter | none | Uploads, final snapshots and lease releases that failed and wait for a retry. |
spawnery_worldsync_upload_seconds | histogram | none | Time to upload one snapshot. |
spawnery_worldsync_worlds | gauge | state | Worlds on this node: mounted by a pod, or cached with no pod. |
Alerts¶
Off by default (metrics.prometheusRule.enabled: false, see chart values); rendered here with that value and worldSync.enabled turned on and every other value left at the chart's own default. The spawnery-worldsync group renders only with world sync on. SpawneryCAExpiringSoon's summary interpolates metrics.prometheusRule.caExpiryWarningDays -- the number in it below is that value's chart default, not a fixed part of the alert; an install that overrides it gets a different number in that one message.
SpawneryAgentConnectionsRefused¶
Group: spawnery-agents Severity: warning For: 15m
The spawnery operator is refusing agent connections over the per-peer bound.
increase(spawnery_agent_connections_refused_total{bound="peer"}[15m]) > 0
One peer address has held MaxConnectionsPerPeer connections open and asked for more. Find the peer in the operator's log -- it is logged at the first refusal and then at every power of ten, so the line is there without the log being the flood. Compare spawnery_agent_open_connections against spawnery_agents_expected: a fleet at its pod count with one peer refusing is a single misbehaving pod, and a fleet well above it is something else.
SpawneryAgentFleetOverItsBound¶
Group: spawnery-agents Severity: critical For: 5m
The spawnery agent endpoint holds more connections than its fleet can account for.
increase(spawnery_agent_connections_refused_total{bound="fleet"}[15m]) > 0
This is the one bound in the agent channel that is a statement about the fleet rather than about one peer. Reaching it means either that several pods are opening connections they have no use for, or that addresses which are not managed pods at all are reaching the endpoint -- the NetworkPolicy admits any pod labelled spawnery.cloud/managed-by and does not count them. Start with the peers in the operator's log and check each against kubectl get pod -A -o wide. A spawnery_agents_expected of 0 beside a fleet that plainly is not empty is the third reading: the count itself is broken, and the endpoint is serving one peer's worth while it is.
SpawneryCAExpiringSoon¶
Group: spawnery-ca Severity: warning For: 1h
The spawnery agent CA expires in under 90 days.
(spawnery_ca_expiry_timestamp_seconds - time())
< 7776000
Rotating it is a deliberate multi-step procedure with a hold in the middle that waits for a person, so start it with time in hand. Annotate the operator's TLS secret with spawnery.cloud/rotate-ca=start and follow docs/guides/rotating-the-ca.md. Note that a start annotation can sit unread for up to an hour on an idle cluster, because the operator only re-reads it on its slow tick.
SpawneryCAKeyMatchesNoCertificate¶
Group: spawnery-ca Severity: critical For: 5m
The spawnery operator refuses to issue a new CA, because ca.key matches no certificate in ca.crt or is not a readable key.
spawnery_ca_key_unmatched == 1
A new CA would lock out every agent that pinned the old one, so the operator leaves the TLS secret as it is. A running operator keeps serving its last certificate and agents stay connected until it expires, at most 90 days; one that restarts serves none, refuses every agent connection and creates no new pods or proxies until the secret is repaired. Do not restart the operator to fix this. Put the certificate that matches ca.key back into ca.crt (from a backup, or from the spawnery-ca ConfigMap in any namespace holding a Network), or restore ca.key from a backup if that is the broken half; the operator picks the repair up within 30 seconds. Starting over with a new CA means deleting the secret and restarting every agent pod; see docs/guides/rotating-the-ca.md.
SpawneryCARotationBlocked¶
Group: spawnery-ca Severity: warning For: 6h
A spawnery CA rotation has been waiting on namespaces for six hours.
spawnery_ca_rotation_blocked_namespaces > 0
Read spawnery.cloud/ca-rotation-blocked-on on the operator's TLS secret for the names. A namespace with leftover agent pods and no Network blocks the gate by design; the remedy is to delete or drain those groups.
SpawneryCARotationStalled¶
Group: spawnery-ca Severity: warning For: 24h
A spawnery CA rotation has been distributing for a day.
spawnery_ca_rotation_phase{phase="distributing"} == 1
The gate waits for every namespace to show the incoming CA and then for a projection margin plus the agent session deadline -- a quarter of an hour in total, not a day. Check spawnery.cloud/ca-rotation-blocked-on, and spawnery.cloud/ca-rotation-discarded for a slot the operator repaired or threw away.
SpawneryOperatorMissingPermissions¶
Group: spawnery-permissions Severity: critical For: 5m
The spawnery operator lacks {{ $value }} permission(s) it needs ({{ $labels.scope }}).
spawnery_permissions_missing > 0
The operator's log names each missing verb and the call site that needs it -- search it for "is missing permissions it needs". It keeps running on purpose, because a missing permission may be one this cluster's paths never take, so nothing else will look wrong: the paths that need the verb simply reconcile nothing. Compare the ClusterRole and Role in the release against config/rbac/role.yaml, and check that the bindings still name the operator's ServiceAccount.
SpawneryServingCertificateNotRenewing¶
Group: spawnery-ca Severity: critical For: 1h
The spawnery operator's serving certificate expires within a day.
(spawnery_serving_cert_expiry_timestamp_seconds - time()) < 86400
This certificate renews on its own once a third of its life is left, so reaching one day means the renewal path has stopped. Every agent handshake fails when it expires.
SpawneryWorldDownloadFailed¶
Group: spawnery-worldsync Severity: warning For:
A world download on node {{ $labels.node }} failed.
increase(spawnery_worldsync_download_failures_total[15m]) > 0
The pod that waited for it failed its start. The agent's log has the error ("download failed"); an object the manifest names but the bucket lacks means the world in the bucket is damaged.
SpawneryWorldLeaseConflicts¶
Group: spawnery-worldsync Severity: warning For: 15m
Node {{ $labels.node }} has refused a world for 15 minutes because another node holds its lease.
increase(spawnery_worldsync_lease_conflicts_total[5m]) > 0
A pod waits in ContainerCreating with an Unavailable event naming the node that holds the world. That node's agent still renews the lease: either a pod there still mounts the world, or its final upload cannot finish.
SpawneryWorldOrphaned¶
Group: spawnery-worldsync Severity: warning For:
Node {{ $labels.node }} moved a world's local copy to orphans/.
increase(spawnery_worldsync_orphans_total[30m]) > 0
The node lost the world's lease while it still held changes the bucket does not have. The copy lies under orphans/ in the node directory (worldSync.hostPath) and never uploads; the agent's log names the world. Compare it with what the bucket holds before restoring anything by hand.
SpawneryWorldSnapshotsPending¶
Group: spawnery-worldsync Severity: warning For: 10m
Snapshots on node {{ $labels.node }} have not reached the bucket for 10 minutes.
max by (node, pod) (spawnery_worldsync_pending_snapshots) > 0
The node agent keeps retrying, and its log says why ("retrying"). Usually the object store is unreachable or refuses the credentials. Play since the last upload exists only on this node until then; if a stopped world's lease goes stale meanwhile, another node may start it from the last upload and this copy moves to orphans/.