Skip to content
tcpsystems
Version: Current

status.tcp.systems (snapshot runbook)

import StatusPill from ‘../../../components/StatusPill.astro’;

Before you act

Section titled “Before you act”
  • Confirm status-page state first; owner status (reviewed 2026-11-05) marks status dead or unverified until restored.
  • Run host commands only on the verified status host (snapshot references scaleblade).
  • Do not edit served HTML/JSON to fix an incident; update the feed source and publish it.

The status page is a generated snapshot plus a live event stream. Neither is derived from this repository.

Shape

Section titled “Shape”
scaleblade:/opt/tcpsystems-status          the generator and its schedule
scaleblade:/srv/status/index.html          the served snapshot
scaleblade:/srv/status/api/status.json     the machine snapshot
scaleblade:127.0.0.1:3046/api/events       the SSE stream
scaleblade:status-gen.timer                regenerates every 5 minutes
scaleblade:status-events.service           watches the artifact, serves SSE

Caddy serves the page and proxies /api/events with flush_interval -1, so the stream is never compressed.

Deploy

Section titled “Deploy”

The generator and service files live in the tcpsystems/status repository; the host has no GitHub credential, so a change is installed from a bundle:

Terminal window
ssh scaleblade 'ls -la /opt/tcpsystems-status'
# install the bundle, then:
ssh scaleblade 'systemctl daemon-reload && systemctl restart status-events.service && systemctl start status-gen.service'

Rollback

Section titled “Rollback”

Backups of the previous install are at /var/backups/status-live-<UTC>/ on scaleblade; restoring is cp -a of that directory back over /opt and /srv/status, then systemctl restart of the two units.

Restore

Section titled “Restore”

/srv/status/index.html is regenerated, not precious: systemctl start status-gen.service rebuilds it from the probe list. The precious file is the probe list itself, under /opt/tcpsystems-status.

Incident

Section titled “Incident”
  1. curl -sS https://status.tcp.systems/ | grep -o 'Generated [0-9T:.Z-]*'; a timestamp older than ten minutes means the timer stopped.
  2. ssh scaleblade 'systemctl is-active status-gen.timer status-events.service' A failing timer and a failing stream are independent.
  3. ssh scaleblade 'ss -ltnp | grep 3046': the stream binds loopback only; a missing listener is why the page stops updating live while the snapshot still serves.

A stale page is not a lie: the badge turns red when the timer stops. Do not “fix” it by editing the HTML.

Update the status during an incident or maintenance

Section titled “Update the status during an incident or maintenance”

The feed is the only writable surface. state/incidents.json in the tcpsystems/status repository is the source; the generator copies it verbatim to /srv/status/api/incidents.json on every run, so the page and the feed cannot drift apart.

  1. Append one entry to the entries array of state/incidents.json: id (date plus name, like 2026-11-02-search-backend-down), opened, status, severity, the affected surface, and the four prose fields impact, evidence, detection, owner_action.
  2. Evidence is commands and outputs, each with a timestamp. An entry without an observed source is not written.
  3. Publish like any change: bundle the repository change to the host and install it (see Deploy). The next timer run (within five minutes) publishes the feed; the rows move on their own as the probes observe the work.
  4. Close by updating the entry, not deleting it: set closed and status, and append the resolution command and its output to evidence. The feed is append-only history.
  5. Do not hand-edit /srv/status/index.html, /srv/status/api/*.json or the Caddy config to reflect an incident; the generator owns those files and the next run overwrites them.

Planned maintenance is an incident entry opened before the window, with the expected impact and surfaces named; there is no separate maintenance state. What each state means, and what a reader sees while the entry is open, is in the status semantics reference.

Rotate a key

Section titled “Rotate a key”

The probe list contains endpoints, not credentials. Nothing in this service holds a vendor key; a probe that needs one would be a design change, not a rotation.

Add a provider

Section titled “Add a provider”

The equivalent here is a new probe target: add it to the probe list in /opt/tcpsystems-status, and the next timer run includes it. --check refuses a push-mode entry with no endpoint, so a broken target is caught before it serves.

Next steps

Section titled “Next steps”