Skip to content
tcpsystems
Version: Current

Rotation

2026-11-05. repository reference/snapshot reviewed 2026-11-05; verify current behavior before using as live guidance.

Rotation is a small procedure with one rule: mint first, verify second, revoke last. The old credential stays valid until the new one is proven, and the revoke is the last step, never the first.

No value appears on this page. A rotation record cites key ids and file paths, not tokens.

Before you start

Section titled “Before you start”

Two failure modes have already happened here, so check for both.

The running config is not the file on disk

A reverse proxy loads its configuration at reload time. If someone edited the file after the last reload, the running config is authoritative and the file is stale. Reloading without looking would replace a working credential with an expired one and take the surface down.

Read the running configuration first (Caddy: GET 127.0.0.1:2019/config/) and diff it against the file before touching either.

One admin key serves several surfaces

Find every consumer of the key before revoking it. Search the host for the file that injects it, and check whether a second admin key exists for another surface. Rotating one key must not silently break the other.

Rotate an injected admin key

Section titled “Rotate an injected admin key”

Use this when a proxy injects an admin key so that an operator console can be read without a per-request credential. The plaintext lives in one file, mode 640, owned by the proxy’s group.

Mint the replacement

Section titled “Mint the replacement”

Over loopback, with an existing admin key, create a new key with the same scopes and limits and a timestamped name, e.g. console-TS``. Record the returned key id; store the plaintext only where it is about to be used.

Rewrite one line

Section titled “Rewrite one line”

Change only the header line that carries the key in the proxy’s site file. Do not reformat the file, do not change the bind, do not add routes. Keep the mode and ownership unchanged.

Validate, then reload

Section titled “Validate, then reload”
Terminal window
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

A validation failure means stop: the running config is still serving and the old key still works.

Verify the intended surfaces

Section titled “Verify the intended surfaces”
  • the tailnet console answers 200 for /console, /console/keys, /v1/router/*;
  • the public address of the console port refuses connections;
  • the public host answers 404 for /console, /metrics and /v1/router/*;
  • /healthz is 200 and /readyz is 401 anonymous, 200 with an operator key.

Revoke the old key

Section titled “Revoke the old key”

DELETE /v1/router/keys?id=<old key id> over loopback. Confirm the old value now answers 401 on loopback, and that audit rows exist for the create and the delete.

Record the rotation

Section titled “Record the rotation”

Write the timestamp, both key ids, the config file path and the verification results to the host’s backup directory. The pre-rotation copy still contains a revoked token: keep it, but note that it is inert.

Rotate a consumer key

Section titled “Rotate a consumer key”

A person rotates their own key on /app: create a new api grant, update the client, then revoke the old grant. Because KeyAuth looks the key up on every request, the revoke is effective on the next call with no restart and no cache to flush.

Operator-side rotation follows the same shape:

  1. POST /v1/router/keys with the same scopes, models and caps.
  2. Update the client’s secret store.
  3. DELETE /v1/router/keys?id=<old id>.
  4. Confirm 401 invalid virtual key for the old value.

Revoke a session

Section titled “Revoke a session”

Deleting the sessions row stops the cookie resolving immediately. A person’s POST /app/grants/{id}/revoke does this for a session grant, and marks the grant revoked.

A frozen account revokes every grant at once and is recorded in grant_events as account frozen. Unfreezing does not resurrect a revoked credential; issue a new one.

Rotate the process bearer

Section titled “Rotate the process bearer”

The process token (ACUMEN_BEARER_TOKEN) is only required on a non-loopback bind. Rotate it by writing the new value to the service’s environment file, restarting the unit, and then proving the old value is rejected:

Terminal window
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4020/healthz   # 200, exempt

Every other path on that listener must fail with the old bearer and pass with the new one. On a loopback bind the variable is not read, so removing it is a no-op until someone changes -addr.

Invite and recovery-code policy

Section titled “Invite and recovery-code policy”
  • An invite token is single-use and expires (-ttl, default 72h, max 720h).
  • An unknown, expired or consumed token answers 410; the page does not say which, so an attacker learns nothing from the difference.
  • There is no email password reset on this surface. A lost recovery code means an operator revokes the account’s grants and issues a fresh invite.