Install

Docs / Disaster recovery

Disaster recovery

This guide covers losing more than one branch: a node, a zone, the controller, or the whole cluster. For restoring a single branch to an earlier time, see backup-and-restore.md.

Recovery objectives

Storage mode Data loss (RPO) Time to recover (RTO)
Object storage None while two of a project’s three WAL nodes survive; with the default archive acknowledgement, none if WAL disks are lost, because commits wait for the archive A new compute on the same data: seconds to minutes
Dedicated disk, with backups Up to 60 seconds of commits plus the current WAL segment if the disk is lost Restore of the last base backup and WAL, proportional to database size

These are design objectives verified on local clusters. Measure your own on your cloud provider with a recovery drill before you rely on them, and repeat the drill after upgrades.

What keeps running

  • Controller down (upgrade, crash, node loss): running databases keep serving, and the proxy keeps routing to them from their last known address for 5 minutes. Waking suspended branches, creating, deleting and resizing wait until the controller is back. A crashed controller’s successor takes over within about 15 seconds.
  • A WAL node down: commits continue on the other two. With two of three down, the project stops accepting writes rather than risk losing them.
  • A node with a database compute down: Kubernetes reschedules the compute. In object storage mode it starts on any node; with dedicated disks it needs its disk to attach elsewhere (same zone for most block storage).
  • Proxy replica down: clients reconnect to another replica.

Never force-delete a pod on an unreachable node to speed this up: confirm the node is stopped (fenced) at the infrastructure level first, or the old process may still be writing.

Lost cluster, bucket intact

Your bucket holds every backup, WAL archive and page layer. What a new cluster also needs is the platform’s identities, which you exported as described in backup-and-restore.md.

Dedicated disk projects can be rebuilt branch by branch from the bucket: install the same release into a new cluster with the same bucket, recreate the project identities, then create each branch with "restore": {"prefix": "<backup prefix>", "at": "<time>"}. The backup prefix is recorded on every branch (backupPrefix in the API) and in your bucket under skela/<project>/.

Object storage projects must be recovered with their pool identity, because their timelines are bound to it. Use the full regional recovery below, or keep the pool ConfigMaps, timeline ConfigMaps and signing Secrets from your exports and restore them before the controller starts. Never let a new pool identity be created over retained data.

Full regional recovery bundle

The release’s recovery image (SKELA_RECOVERY_TOOL_IMAGE in release.env) captures a complete region into one encrypted, authenticated bundle: the namespace’s resources, every disk, the management database dump and your platform secrets. It complements continuous archiving; writes after the bundle was taken come from the archives.

Run it with access to the cluster, for example:

TOOL=$SKELA_RECOVERY_TOOL_IMAGE
recover() { docker run --rm --user "$(id -u):$(id -g)" -e KUBECONFIG=/kube/config \
  -v "$HOME/.kube/config:/kube/config:ro" -v "$PWD/dr:/dr" "$@"; }

The image contains only the recovery tool, so the kubeconfig must authenticate without an exec credential plugin such as aws eks get-token, gke-gcloud-auth-plugin or kubelogin: use a short-lived token or client certificate for a recovery administrator.

Capture

  1. Stop console mutations and the cloud worker, and record the release digests, namespace and cluster identity. Export runtime configuration, encryption keys, slot tokens and mounted secret files into dr/artifacts/platform-secrets.json.
  2. Drain the controller (scale it to zero and confirm its Lease is released), stop the proxy, the database computes and then the storage pools. No pods may remain in skela.
  3. Dump the management database with pg_dump -Fc into dr/artifacts/management.dump.
  4. Capture the resources: recover -e SKELA_RECOVERY_NAMESPACE=skela -e SKELA_RECOVERY_METADATA=/dr/metadata.json "$TOOL" capture. It refuses if pods or jobs are running or anything changes between its two passes.
  5. Archive every PersistentVolumeClaim into dr/artifacts/pvc-<claim>.tar: recover -e SKELA_RECOVERY_NAMESPACE=skela -e SKELA_RECOVERY_PVC=<claim> -e SKELA_RECOVERY_IMAGE="$SKELA_PG_IMAGE" -e SKELA_RECOVERY_VOLUME_UID=<uid> -e SKELA_RECOVERY_VOLUME_ARCHIVE=/dr/artifacts/pvc-<claim>.tar "$TOOL" archive-volume. The helper image must contain a shell and GNU tar; the PostgreSQL image does. Database, seed and redo disks are owned by UID 1000, WAL disks by 65532.
  6. Generate a 32-byte key into an owner-only file ((umask 077; openssl rand -base64 32 > dr/bundle.key)), seal and check the bundle: recover -e SKELA_RECOVERY_KEY_FILE=/dr/bundle.key -e SKELA_RECOVERY_METADATA=/dr/metadata.json -e SKELA_RECOVERY_ARTIFACTS=/dr/artifacts -e SKELA_RECOVERY_BUNDLE=/dr/region.bundle "$TOOL" seal, then the same with inspect.
  7. Store the bundle and its key in separate failure domains, delete the plaintext staging directory, and restart storage, controller, proxy and console.

The bundle is encrypted with AES-256-GCM in authenticated 1 MiB frames and checks every artifact’s SHA-256. It refuses links, device files and paths that escape a volume. Artifacts are capped at 100 GiB by default (SKELA_RECOVERY_MAX_BYTES); budget local disk for input, staging and output.

Restore

  1. Fence the old cluster’s nodes at the infrastructure level and revoke their access to the bucket. Record the evidence.
  2. Prepare an isolated cluster with the same release’s resource definition, cert-manager and issuer, StorageClasses and cluster-wide RBAC. The skela namespace must not exist. Do not route clients to it yet.
  3. restore-metadata with the bundle, its key and an absent SKELA_RECOVERY_ARTIFACTS directory recreates the namespace with every workload at zero replicas and a skela-recovery-hold ConfigMap that keeps the controller from starting. Identities, generations and backup prefixes are kept: recover -e SKELA_RECOVERY_KEY_FILE=/dr/bundle.key -e SKELA_RECOVERY_BUNDLE=/dr/region.bundle -e SKELA_RECOVERY_ARTIFACTS=/dr/restored "$TOOL" restore-metadata.
  4. restore-volume for every claim, with the same UID and helper image. The destination disk must be empty; never retry over a partly restored disk.
  5. Restore management.dump into a fresh PostgreSQL with pg_restore --exit-on-error --single-transaction --no-owner, restore the console Secrets, and run cloud/render.sh check-keys.
  6. Check TLS identities, bucket access and that the old nodes are fenced. Then delete skela-recovery-hold, start the controller, storage and proxy, and verify application logins and known recent rows in both storage modes.
  7. Move DNS or load balancers to the new cluster, resume the console, and take a new bundle.

The bundle does not contain cluster-wide resources, cloud IAM and KMS, DNS, load balancers, bucket contents or your WorkOS configuration; recreate those with your infrastructure tooling.

Drill

Before production and after every upgrade, restore into an isolated cluster and record: the time of the last acknowledged row you can read, how long recovery took, and every manual step. Losing both the bundle and its key, and the original bucket, is not recoverable.