Edge security

Unverified2026-11-05. repository reference/snapshot reviewed 2026-11-05; verify current behavior before using as live guidance.

Last verified: 2026-10-20 18:33 UTC. Scope: what the public internet can reach on the edge host, the two operator surfaces that were exposed, their fixes, and the admin-key rotation. No credential values are recorded here; key ids are not secrets.

1. Console on a wildcard bind

The console vhost used an IP-literal upstream address. An IP-literal site address pins the port but not the interface, so the proxy listened on *:8088 and proxied to acumen while injecting an admin bearer. The host firewall was inactive and the project had no cloud firewall, so /console, /v1/router/stats, /v1/router/keys and /metrics answered 200 anonymously on the public IP.

Fixed 2026-10-20 18:21 UTC, in the console site file:

• bind <tailnet address>: the listener is pinned to the tailnet address.
• A remote_ip guard that answers 404 for anything outside the tailnet and loopback ranges, as a second line of defence if the bind is ever removed.
• The public site gained explicit respond 404 handlers for /console*, /v1/router/*, /metrics and /debug/pprof/*.
• A systemd drop-in orders the proxy after tailscaled.service and sets Restart=on-failure, so the pinned bind cannot take every vhost down at boot if the tailnet address is late.
• The console vhost answers 404 for /metrics* and /debug/*. It injects the admin key, so proxying those would make them anonymous to every tailnet peer, and a heap profile can carry request bodies or credentials. The direct loopback listener still serves both to an operator who presents a key.

Verified: the listener shows the tailnet address only; public :8088 refuses on IPv4 and IPv6; the public host returns 404 for /console, /metrics and /v1/router/stats; /healthz is 200 and /readyz is 401 anonymous, 200 with an operator key. Model doors were exercised with a short-lived chat key through the public hostname and answered 200; the key was revoked.

2. Grafana on a wildcard bind

grafana-server had no http_addr, so it bound every interface and answered on the public IP with a 302 to /login, despite a tailnet vhost proxying loopback.

Fixed 2026-10-20 18:27 UTC: http_addr = 127.0.0.1 in the [server] section, with the previous file kept in the host backup directory.

Verified: the listener is loopback only; public :3000 refuses on IPv4 and IPv6; the tailnet vhost redirects to /login and the login page is 200; /api/health reports database ok.

3. Admin key rotation

The console vhost injected a virtual key with admin scope, no model limit. A disk scan confirmed the plaintext existed only in the console site file and its backups.

Procedure, executed 2026-10-20 18:27 UTC:

1. Minted a timestamped replacement with the same scopes and limits over loopback POST /v1/router/keys.
2. Rewrote only the header line that carries the key; the file kept mode 640 and its proxy-group ownership.
3. caddy validate → valid; systemctl reload caddy → active.
4. Verified the tailnet console, keys, router stats and metrics all 200; public :8088 refused; the public host returned 404 for /console.
5. Revoked the old key (DELETE .../v1/router/keys?id=<old id> → 204). The old token now gets 401 on loopback. Audit rows for the create and the delete exist for both ids.

A second admin key used by an operator was left untouched. The pre-rotation copy still contains the old plaintext, which is now revoked and therefore inert.

See Rotation for the general procedure.

4. Listener inventory

Public (*:), by design:

Port	Process	Verdict
22	sshd	intentional; key-only root, password auth off
25, 465, 587, 993, 143	maddy	intentional mail submission and IMAPS; no web admin
80, 443	caddy	intentional public web edge; operator paths 404
21115-21119	RustDesk ID/relay	intentional; no key configured

Loopback only: the proxy admin API on 2019, Grafana on 3000, app ports on 3030-3032, open-webui on 3080, acumen on 4020, the control plane on 8080 and its metrics on 9090, containerd, and a local SSH forward on 8888.

Tailnet only: the console proxy on :8088, plus tailscaled itself.

UDP: 443 (HTTP/3), 3478 (STUN), 41641 (tailscaled, public by design), RustDesk ports, and DHCP. No admin surface.

5. Public by design

• The control-plane hostname is needed for node registration; its node API is 401 and the root is an empty placeholder page.
• The chat hostname: open-webui with auth on, signup off, default role user.
• The apex and www answer 301/308 redirects to chat.

6. Regression check

tools/edge-check.sh asserts:

• public liveness 200, operator namespace 401/403/404;
• model doors answer 401/403, so a missing door cannot pass as a closed one;
• the console port on its public address refuses connections;
• Grafana on its public address refuses connections;
• optional tailnet probes: console 200 and Grafana login 200.

Last run: ok.

ACUMEN_TAILNET_URL=http://<tailnet-ip>:8088 \
GRAFANA_TAILNET_URL=https://<tailnet-host> \
  tools/edge-check.sh        # edge-check: ok

7. Rollback

Grafana: restore the previous grafana.ini from the host backup directory and restart the service.

Console vhost: restore the pre-bind copy from the backup directory, then caddy validate and systemctl reload caddy. Do not restore the pre-rotation copy on its own: it contains a revoked token and must be paired with a fresh key.

Key rotation is one-way: the old key is deleted from the store. Rolling back means pointing the vhost at another live admin key.

8. Follow-ups

1. The RustDesk relay runs without a key and accepts anonymous clients. Decide whether to require one; changing it means reconfiguring every client.
2. Grafana root_url is still the default http://localhost:3000/. Login works through the vhost because the redirect is relative, but emailed links and absolute redirects will point at localhost.
3. The host has a global IPv6 address; both fixed ports refuse on IPv6 as well as IPv4. Keep probing v6 in future audits.
4. No cloud firewall exists for this project. Host-side binds are the only control layer, so configuration drift is a direct exposure.