acumen gateway (snapshot runbook)

import StatusPill from ‘../../../components/StatusPill.astro’;

The gateway runs as a container on the host that serves ai.tcp.systems, with its state in PostgreSQL. This runbook covers operator checks; verify release paths and deployment state before using any command.

Deploy and roll back

Each release is an image swap with a database dump taken first, and the swap script records its own rollback:

ls ~/acumen-<sha>/scripts/          # swap-to-<sha>.sh and rollback-to-<previous>.sh
ls ~/acumen-<sha>/backups/          # pre-<sha>-<UTC>.dump

A rollback restores the dump before the older binary starts: a newer schema is not readable by an older binary, and the release notes say which migration number is the floor.

Incident

1. curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4020/healthz on the host: the process, not the edge.
2. curl -sS -o /dev/null -w '%{http_code}\n' https://ai.tcp.systems/v1/models : the edge, given a process that answers. A 502 here with a 200 above is the proxy, not the gateway.
3. Seat state, when the process answers but requests fail:

SELECT provider, account, model, cooling_until, consecutive_failures, last_error_code
  FROM seat_health
 WHERE cooling_until > now()
 ORDER BY cooling_until DESC;

Every row is a seat that is deliberately not being tried. An empty result with model_no_eligible_seat responses means the offer table has no matching row, not that Redis or the network is down.

Restore

The database is the state. Restoring the dump from the release directory brings back seats, keys, the grant ledger, the attempt ledger and console sessions together. There is no partial restore worth doing.

Rotate a key

Virtual keys are rows: rotate by inserting a new key and revoking the old one, never by editing a value. Vendor credentials live in files outside the database, referenced by path from the seat row; the file is what gets replaced, and the 30-second seat reload picks it up.

Add a provider

Add a provider is the procedure: acumen offer add, then wait one reload window, then prove one answered request.