Phase 8 — Polish & Hardening

Goal

The bot is operable by someone other than the original author and survives common failure modes without intervention. This phase is mostly ops work: rate limits, backups, alerting, token rotation, crash-loop guards, and runbook completeness.

Scope / Deliverables

• Per-user MCP rate limit (token bucket) — only if a real abuse case appears
• Daily Postgres logical backup to off-VPS storage (S3/B2) via cron
• Restore-from-backup drill, documented in ops/deploy.md
• journalctl alert hook (or systemd OnFailure=) on service crash
• Token rotation script (npm run rotate-token) that updates env + restarts service + prints the new token once
• Crash-loop guard: Restart=on-failure + StartLimitBurst=5 + StartLimitIntervalSec=60
• Document the entire setWebhook → DNS → systemd flow end-to-end in ops/deploy.md
• Decide whether to mirror Inngest state in Postgres permanently or to query Inngest API on demand (resolve the standing open question with real load data — see operations-and-strategy.md)

Automated Unit Testing

This phase emphasizes operational drills and ops/runbook completeness over application-level logic, so the unit-test additions are narrow and only required for the items that introduce new code paths:

• rate-limit.test.ts — only if the per-user MCP rate limit is implemented:
  • allows the first N calls inside the window
  • the (N+1)th call within the window → RATE_LIMITED JSON-RPC error
  • after the window elapses, the bucket refills and calls succeed again
  • independent buckets per userId (one user being throttled does not affect another)
• rotate-token.test.ts (script unit) —
  • generates a new 32-byte token, writes it to the env file with the existing mode preserved
  • prints the new token to stdout exactly once
  • does not leak the new token to logs or other files
  • dry-run mode (if implemented) reports what would change without writing
• inngest-mirror.test.ts — only relevant if the open question is resolved in favor of either path:
  • mirror mode: status-reconciliation logic correctly transitions running → succeeded/failed rows when corresponding events arrive
  • API-on-demand mode: get_job_status falls back to the Inngest API on cache miss and degrades gracefully on Inngest 5xx

Manual scope (Phase 8 owns this — and matters more here than code tests):

• Restore-from-backup drill end-to-end (kill the DB, restore from the latest off-VPS dump, confirm chat_messages is intact).
• Crash-loop scenario: kill -9 the service five times within 60s; assert systemd stops trying to restart.
• Token rotation drill: run npm run rotate-token, verify Inspector authenticates with the new token and fails with the old one.

Definition of Done

• Every deliverable above is shipped or has a written rationale for being deferred.
• ops/deploy.md is complete enough that a new operator can stand the system up end-to-end without asking questions.
• Restore-from-backup drill has been executed successfully at least once.
• Crash-loop guard has been verified by simulating repeated failures.
• Token rotation script has been used at least once on the production env without downtime past one systemctl restart.
• The Inngest-state-mirror open question is resolved one way or the other, with the chosen approach reflected in code and docs.