Phase 2 — Operator runbook

What you need to do to ship Phase 2 PRs to production. Tick items as you complete them. Pair with phase-2-google-ads.md for the implementation spec.

Status (2026-05-12): Phase 2 is code-complete locally (PR-1 through PR-7 on dev, 77/77 integration tests passing). Only PR-1 is in prod so far. The remaining operator work is one deploy that covers PR-2 through PR-7 (see §3a). Live end-to-end smoke is blocked on Google Ads developer-token approval (§4a) but every code path passes against mocked Google.

1. Verify PR-1 locally (15 min) — ✅ done 2026-05-11

Verified on a fresh local DB during the PR-1 → PR-4 session. Items kept here so the procedure is reproducible on another machine.

• Start Postgres.

  docker compose up -d postgres

• Apply migrations. PR-4 added 0002_new_synch.sql (unique index on platform_credentials(tenant_id, platform)). Both 0001 and 0002 apply via:

  npm run db:migrate

• Create the mcp_app role (only on a fresh DB volume). Phase 1’s roles.sql is what creates the runtime role:

  docker exec -i deneva-mcp-postgres-1 psql -U mcp_admin -d deneva_mcp `
    -v app_password="dev_only_password" -f - < src/db/roles.sql

  Skip if you’re re-using an existing DB volume; the role persists across docker compose down.

• Apply the RLS policy update. Drizzle does NOT generate RLS DDL; it lives in src/db/rls.sql and must be applied as mcp_admin after every migration that adds a tenant-scoped table. PR-1 added tenant_isolation_deks.

  docker exec -i deneva-mcp-postgres-1 psql -U mcp_admin -d deneva_mcp -f - < src/db/rls.sql

  If you see ERROR: policy "..." already exists, that’s expected — rls.sql is not idempotent today. The new policy is what matters.

• Run the cache integration tests.

  npm test -- tests/integration/cache.test.ts

  Expected: 5 passing. The thundering-herd test exercises pg_advisory_xact_lock against the real engine — if it fails, RLS context is probably missing on a code path; check src/cache/cache.service.ts setTenantContext.

• Run the full test suite.

  npm test

  After PR-7 the baseline is 77 tests across 14 files: Phase 1 — api-key, audit, ip-block, log-scrubber, oauth-state, rls; Phase 2 — cache, credentials, oauth, google-adapter, tools-google, revocation, dek-rotation, cache-metrics.

• Sanity-check lint + typecheck:

  npm run lint
  npm run typecheck

  Both should be silent.

2. Review and commit — ✅ done

All seven Phase 2 commits are on dev:

• CLAUDE.md updated with the Phase 2 patterns (envelope encryption, OAuth routes, adapters, TIMESTAMP parser, tx-aware crypto, stateless MCP transport).
• phase-2-google-ads.md “Deviations from spec” updated end-to-end; every §11 checklist item is ticked.

3. Apply PR-1 to production app.deneva.io — ✅ done

PR-1 is in prod. The migration is additive (cache + tenant_deks schema, no destructive DDL) and the cache layer is dormant until PR-6 wires it through the seven tools. Items below are kept as a checklist for reproducibility.

• Snapshot the prod database before applying the migration.
• Deploy the new code (git pull, npm ci, npm run build, systemctl restart).
• npm run db:migrate (applied 0001).
• Apply src/db/rls.sql (added tenant_isolation_deks).
• Verify the new objects exist (\d tenant_deks, metric_cache_key_idx, tenant_isolation_deks).
• Smoke-test /health + tools/list returned the expected results.

3a. Apply PR-2 through PR-7 to production (30 min) — pending

This is the operator work that’s still outstanding. One deploy lands everything from PR-2 through PR-7:

• New secrets (PR-3): GOOGLE_CLIENT_SECRET, GOOGLE_DEVELOPER_TOKEN. The startup verifyAllSecretsLoadable() will refuse to bind a port without them.
• New env vars (PR-3, PR-4): GOOGLE_CLIENT_ID, GOOGLE_OAUTH_REDIRECT_URI. loadConfig() (Zod) will throw at boot without them.
• New migration (PR-4): 0002_new_synch.sql adds a unique index on platform_credentials(tenant_id, platform).
• New npm dep (PR-5): google-ads-api@^23 — npm ci --omit=dev picks it up.
• New code surface to verify post-deploy:
  • POST /auth/google/start redirects to Google (PR-4)
  • POST /mcp tools/call get_account_health returns real (or mocked-shape) data (PR-6) instead of the Phase 1 stub
  • POST /admin/tenants/:tenantId/disconnect/:platform (PR-7)
  • GET /admin/metrics/cache (PR-7)
  • tsx scripts/rotate-dek.ts <tenantId> available as an ops command (PR-7)

3a.1. Confirm the OAuth client config

• In Google Cloud Console → APIs & Services → Credentials, confirm the OAuth 2.0 client has https://app.deneva.io/auth/google/callback listed as an authorized redirect URI.
• Grab the client ID (public) and client secret (secret) — you’ll need both in the next two steps.
• If the developer-token application has come back, grab the developer token too. If not, use a placeholder string (pending_dev_token) — PR-4 doesn’t actually call the Google Ads API yet; PR-5 will.

3a.2. Snapshot the prod DB

sudo -u postgres pg_dump -d deneva_mcp -F c -f /var/backups/deneva_mcp_pre_phase2_pr4.dump

3a.3. Encrypt the two new secrets with systemd-creds

sudo bash /home/<you>/deneva-mcp-src/scripts/encrypt-prod-secrets.sh

The script’s SECRETS list now includes GOOGLE_CLIENT_SECRET and GOOGLE_DEVELOPER_TOKEN. Existing .cred files are left alone; only the two new ones are prompted.

3a.4. Edit the systemd unit

Add four new lines to /etc/systemd/system/deneva-mcp.service:

Environment=GOOGLE_CLIENT_ID=<paste-public-client-id>
Environment=GOOGLE_OAUTH_REDIRECT_URI=https://app.deneva.io/auth/google/callback
LoadCredentialEncrypted=GOOGLE_CLIENT_SECRET:/etc/deneva-mcp/creds/GOOGLE_CLIENT_SECRET.cred
LoadCredentialEncrypted=GOOGLE_DEVELOPER_TOKEN:/etc/deneva-mcp/creds/GOOGLE_DEVELOPER_TOKEN.cred

Optional but recommended — pin the timezone to UTC so the new pg TIMESTAMP parser override never has to compensate:

Environment=TZ=UTC

3a.5. Deploy the new code

ssh user@app.deneva.io
cd /opt/deneva-mcp     # or wherever your deploy lives
git pull origin dev    # or merge dev→master and pull master, per your branching
npm ci --omit=dev
npm run build

3a.6. Apply migration 0002

npm run db:migrate
# → applies 0002_new_synch.sql (unique index on platform_credentials)

3a.7. Reload + restart

sudo systemctl daemon-reload
sudo systemctl restart deneva-mcp
sudo systemctl status deneva-mcp --no-pager

Tail the logs and watch for deneva-mcp listening:

sudo journalctl -u deneva-mcp -f

3a.8. Smoke-test the new surface

Health is still unauthenticated:

curl -fsS https://app.deneva.io/health

/auth/google/start without a key should be 401:

curl -i https://app.deneva.io/auth/google/start
# → HTTP/1.1 401

With a real prod API key, expect a 302 to Google:

curl -is -H "X-Api-Key: $PROD_KEY" https://app.deneva.io/auth/google/start
# → HTTP/1.1 302
# → Location: https://accounts.google.com/o/oauth2/v2/auth?...code_challenge=...

tools/list should now return ping + the seven Phase 2 tools:

curl -s -X POST https://app.deneva.io/mcp \
  -H "X-Api-Key: $PROD_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
# → 8 tool names: ping, get_account_health, get_search_term_waste,
#   get_quality_score, get_auction_insights, get_pmax_breakdown,
#   get_budget_optimizer, get_weekly_anomaly

tools/call get_account_health on a not-yet-connected tenant returns the no_google_credentials error (the makeTool wrapper still audits mcp.tool_failed) — proves the dispatch through cache + adapter works:

curl -s -X POST https://app.deneva.io/mcp \
  -H "X-Api-Key: $PROD_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_account_health","arguments":{"platform":"google","dateRange":"last_7_days"}}}'
# → JSON-RPC error envelope mentioning no_google_credentials

Admin routes:

# Cache metrics — empty {} until a tool call lands cache traffic
curl -s -H "x-admin-token: $ADMIN_TOKEN" https://app.deneva.io/admin/metrics/cache
 
# Disconnect a tenant from Google (idempotent — returns {status:'disconnected'} even with no row to delete)
curl -s -X POST -H "x-admin-token: $ADMIN_TOKEN" \
  https://app.deneva.io/admin/tenants/$TENANT_ID/disconnect/google

The seven tools won’t return real Google data until (a) you complete the OAuth flow for a tenant in a browser AND (b) the Google Ads developer token approval has landed. Until then, expect no_google_credentials for unconnected tenants and a Google API auth error for connected tenants on a placeholder dev token.

3a.9. Roll-back plan

If startup fails: sudo journalctl -u deneva-mcp -n 200. The likeliest causes are a missing env var (Zod will name the field) or a missing .cred file (the secrets loader will name the secret). Fix and restart. The migration is forward-compatible — the unique index can stay even if you revert the code.

4. External prereqs — start in parallel (long lead times)

These don’t block PR-1 or PR-2 code work, but they do block the live end-to-end smoke test of Phase 2 (PR-5 onwards). The Google Ads developer-token approval is the long pole — start the application today.

4a. Google Ads API developer token

• Apply for a developer token at https://ads.google.com/aw/apicenter  (sign in with a Google Ads manager account / MCC). You need at least basic access for Phase 2.
• Decide on the access tier: basic = 15k operations/day, standard = 1M+. Phase 2 only reads aggregated metrics; basic is fine to start, request standard if/when you fan out across many tenants.
• Tracking: approval can take days to weeks — note the application ID and ETA. Update the prereqs table in phase-2-google-ads.md “Implementation status” when it lands.

4b. OAuth client configuration sanity check

• Confirm in Google Cloud Console → APIs & Services → Credentials → your OAuth 2.0 client:

  • Authorized redirect URI includes https://app.deneva.io/auth/google/callback exactly (PR-4 will hit this URL).
  • Authorized JavaScript origin includes https://app.deneva.io (cosmetic, but cleaner).
  • OAuth consent screen is published (not in test mode) or all your test tenants are added as test users — test mode caps at 100 users and refresh tokens expire weekly.
  • Scope https://www.googleapis.com/auth/adwords is requested.

• Capture the client ID and secret somewhere safe (password manager). PR-3 will load them via the secrets loader, not env vars:

  • GOOGLE_CLIENT_ID (public, OK to commit to config) → set as env var or in src/config.ts
  • GOOGLE_CLIENT_SECRET → file at secrets/GOOGLE_CLIENT_SECRET in dev, systemd-cred in prod

4c. Test Google Ads account

• Pick a test customer account under your MCC that you’re OK hitting with read-only GAQL during E2E testing. A throwaway dev MCC is ideal; if not, any low-spend account.
• Note the customer ID (10-digit, no dashes). PR-5’s resolveCustomerId will pick the first accessible customer on first OAuth — if you have many under the MCC, decide which one comes first.

4d. Production secrets staging — covered by §3a

PR-3 + PR-4 require GOOGLE_CLIENT_SECRET and GOOGLE_DEVELOPER_TOKEN in /run/credentials/<unit>/ and the GOOGLE_CLIENT_ID / GOOGLE_OAUTH_REDIRECT_URI env vars on the systemd unit. The concrete steps live in §3a — encrypt the two secrets via scripts/encrypt-prod-secrets.sh (the SECRETS array already includes them) and edit the unit file.

CREDENTIAL_KEK was set up in Phase 1; PR-2 consumes it but does not require a new value.

5. Current state and next step

Phase 2 is code-complete on dev. The seven PRs (PR-1 through PR-7) cover every workstream in the spec (A through H + I tests). The full §11 checklist in phase-2-google-ads.md is ticked.

PR-1 is in prod. The remaining operator work is §3a — one deploy that lands PR-2 through PR-7 together.

After §3a is done, Phase 2 is “DONE — pending Google’s approval”. The only thing blocking a live end-to-end smoke against real Google Ads data is the developer-token application (§4a). The code path is exercised against vi.mock('google-ads-api', …) for all seven tools and against real Postgres + RLS + crypto for everything below the adapter.

The next phase to write is docs/plan/phase-3-meta-tiktok.md.

Quick reference