Install

Docs / Rotate keys and certificates

Rotate keys and certificates

Every credential SkelaDB uses lives in your cluster or your secret store. This guide lists each one, where it lives and how to replace it. Generate new values with openssl rand -hex 32 unless stated otherwise, and never paste them into shell history (read -r -s VALUE).

Platform tokens

Credential Where Used by
Admin API token skela-api Secret, key token Operators and automation with full access
Proxy token skela-api, key proxy-token The proxy, which can only resolve connections
Metrics token skela-api, key metrics-token Prometheus scrapes of /metrics
Project tokens skela-project-tokens Secret, one key per project Console regions and per-project automation
Proxy peer secret skela-proxy-peer, key secret Proxy replicas authenticating each other

The controller and proxy read the first three at startup. To rotate one:

kubectl -n skela patch secret skela-api --type merge -p "{\"stringData\":{\"token\":\"$NEW\"}}"
kubectl -n skela rollout restart deployment/skela-control deployment/skela-proxy

Update every client of the admin token at the same time. Rotating the proxy token or the peer secret makes logins wait for the restart to finish, so do it in a quiet period.

The controller reads project tokens on every request: replace or remove a key in skela-project-tokens and the old token stops working at once. If the console uses that slot, update the token in the cloud-regions Secret and restart cloud-api and cloud-worker in skela-cloud.

License key

Install a renewed key before the old one expires (the controller exports skela_license_expiry_timestamp):

kubectl -n skela create secret generic skela-license --from-literal=key="$SKELA_LICENSE_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n skela rollout restart deployment/skela-control

Rerunning ./install.sh apply with SKELA_LICENSE_KEY set also updates the Secret; restart the controller afterwards. See licensing.md.

Database passwords

  • Console databases: Databases → Rotate password issues a new password for that database’s role; connected sessions keep running and new logins need the new password.
  • Roles you created yourself: ALTER ROLE ... PASSWORD ... as usual.
  • The branch’s internal passwords live in the branch’s Secret; the platform reinstalls them whenever the Secret changes.

Console keys and secrets

Encryption keys. SKELA_CLOUD_ENCRYPTION_KEYS lists id:base64key, newest first, with numeric IDs from 1 to 255. To rotate:

  1. Escrow the new key, then prepend it: 2:<new>,1:<old>. Restart cloud-api and cloud-worker. New values are sealed with key 2, and the worker re-seals existing values in the background, 100 per table each minute.
  2. Wait until the API metric skela_cloud_sealed_values_not_current is 0.
  3. Remove key 1 from the list and restart. Keep it in escrow as long as any backup sealed with it is retained.

Run cloud/render.sh check-keys after any change; it fails if any stored value cannot be opened with the configured keys.

WorkOS. Rotate the API key and webhook secret in WorkOS, update cloud-config and console-config, and restart the three workloads. Changing SKELA_WORKOS_COOKIE_PASSWORD signs everyone out.

Metrics token (SKELA_CLOUD_METRICS_TOKEN): update the Secret and your Prometheus configuration together.

Certificates

cert-manager renews leaf certificates before they expire. The controller, proxy, database computes and storage services pick up renewed certificates while running (the platform services check every five seconds), and existing connections continue. Nothing needs to be done as long as the issuing CA stays the same. Watch the certificate expiry alert in observability/.

Replacing the certificate authority

A new CA must be trusted everywhere before anything presents a certificate signed by it. Plan a maintenance window and keep the old CA and issuer until the end.

  1. Inventory every verifier: application clients, the proxy, the console’s region CA files (controller_ca, compute_ca), database computes and storage services.
  2. Distribute a bundle with both the old and the new CA to every verifier (for issuers that do not write ca.crt, the SKELA_TLS_CA_SECRET Secret) and restart the proxy and controller, which read trust at startup. Keep the old certificates in place.
  3. Check application logins, the console and backups.
  4. Point the issuer at the new CA and reissue certificates (delete the Certificate Secrets one by one, or cmctl renew). Repeat the checks.
  5. When everything uses new certificates, remove the old CA from the bundles, restart the verifiers and confirm that a certificate from the old CA is refused.

Before step 5 you can roll back by restoring the old certificates. Clients that verify the proxy (sslmode=verify-full) need the new CA before step 4.

Release signing key

SkelaDB signs releases with one key published at https://skeladb.com/releases/cosign.pub. If we ever replace it, the new key and its fingerprint are announced in verify.md and the release notes; update any admission policy that pins it.