Skip to main content

prod-deploy

Why It Exists​

The skill was extracted directly from a real deploy: the introducing commit (32d31d9, "add prod-deploy skill") states it was "extracted from workloom," and its first 8 invariants encode what that deploy actually hit. Historically that included a managed-host egress restriction that couldn't reach the transaction pooler on :6543, only the session pooler on :5432 with sslmode=require — the follow-up fleet rollout (docs/sessions/2026-08-05-fleet-migration-progress/SUMMARY.md) records the skill being re-numbered specifically to add that :6543→:5432 note plus WEBHOOK_SIGNING_KEY after hitting the same landmine again on a second repo. That particular constraint no longer applies now the api runs on a self-hosted CapRover box with unrestricted egress, but the underlying discipline it taught — verify the connection string against the real host rather than assuming, and never "fix" one that's already working — survived into invariant 10 and is what still catches this class of bug.

Invariants 9-14 came later, from a 13-repo rollout of this same pattern (the enrichment commit e3f255f is explicit about the source). The most serious finding from that rollout is invariant 12's routine rowsecurity=false sweep, which the commit message says directly "caught a real live P0 — unauthenticated cross-tenant CRUD+TRUNCATE" on a table created after the repo's own "enable RLS on everything" migration had already run — proof that a one-time RLS migration doesn't retroactively cover tables added afterward. That sweep runs for free alongside the migration-drift check the skill was already telling you to do (invariant 11), which is why it survived into the skill as a standing invariant instead of a one-off incident note. It was serious enough that its canary-row proof methodology — insert a row via the privileged connection, confirm it's invisible to the restricted role, then delete it — was later generalized and pointed to from an unrelated planning document (.omc/plans/quality-loop-skills.md) as the reusable test-coverage pattern for "every new RLS policy," rather than being re-derived from scratch.

The fleet rollout also surfaced a structural gap the skill doesn't fully close yet: docs/sessions/2026-08-05-fleet-migration-progress/FOLLOWUPS.md flags that repos on NextAuth/OIDC (rather than Supabase Auth) get org/role from an OIDC claim instead of the access-token hook this skill assumes, making invariant 5's "register the hook" step moot for that auth path — a documented, still-open branch point rather than something already handled by a flag.

What It Does​

prod-deploy is the production-deploy checklist and reference set for an ENV-selected stack where the api runs on a self-hosted CapRover box (image-backed from ECR), the web app on Vercel, and the database on Supabase Cloud. It exists because none of the manual, one-time, per-environment setup — pushing secrets and non-secret vars into AWS SSM and Vercel's env UI, running the first cloud DB migration by hand, registering the Supabase access-token hook and captcha secret in the dashboard, and getting the pooler connection string right — happens automatically on git push. It ships a genericized PROD-DEPLOY.md checklist, an annotated caprover-service.yml, and a dry-run-by-default deploy-env.mjs that can script the env-var setup and db:deploy (never the deploy trigger itself).

It complements runtime-env-config (the app-side env loader/EnvProvider chain) and local-dev-stack (the same stack's local, not-prod, operational half) — this skill is specifically the "make it work in prod, the first time" half of that arc.

How To Use It​

Triggers on: "first production deploy", "deploy to CapRover", "caprover deploy", "deploy to Vercel", "Supabase Cloud prod", "ENV=prod not set / deployed but misconfigured", "CapRover app env reverted", "access-token hook not registered", "empty org/role in prod JWT", "set prod env vars in SSM", "migrate prod DB", "db:deploy", or standing up a first prod deploy.

skills add git@github.com:catesworks/next-starters.git --skill skills/prod-deploy -g
npm install @next-starters/skill-prod-deploy
/plugin marketplace add catesworks/next-starters
/plugin install prod-deploy@next-starters

Gotchas & Invariants​

  • .env.<stage> never reaches either platform — the api image .dockerignores .env.* and Vercel never runs server-preload.cjs — so both the secrets and the "non-secret" type:'public' set (ENV, URLs, selectors) must be entered in the platform env, not just the secrets.
  • The api's env lives in AWS SSM at /catesworks/<service>/<VAR>, and the ansible apps role prunes the CapRover app's envVars to exactly that set — so an env var typed into the CapRover dashboard is deleted on the next run, and a key you left out of SSM is a key the app simply won't have.
  • ENV=<stage> is the stage selector, distinct from NODE_ENV. Miss it and the loader falls back to base .env, so the deploy looks green but behaves like dev — the single most common "deployed but misconfigured" bug.
  • There is no migrate-on-deploy: db:deploy (db:migrate && db:post) runs by hand against the cloud direct/session URL, schema first, then RLS/the auth-hook function — running that order backwards fails against tables that don't exist yet.
  • config.toml doesn't carry to Supabase Cloud. The access-token hook, captcha secret, and "email confirmations ON" are dashboard-only settings — the hook function can exist (created by db:post) while prod JWTs still carry no org_id, because the cloud was never told to call it.
  • caprover deploy exiting 0 means the deploy was accepted, not healthy — CapRover reports success before the container is serving and does not auto-roll-back, so gate on a body-checked /health/deep poll. Use the official CLI's --appToken; a raw REST call with that token as x-captain-auth doesn't work, and omitting --caproverApp falls back to ambiguous app discovery.
  • The pinned image: tag in services.yml is a provisioning baseline, not what's deployed — and the apps role's desired-state enforcement will silently revert a CI-deployed image back to it on its next run against that service, for any reason. Read the live tag from CapRover's own /api/v2/user/apps/appDefinitions, and update the pinned tag before running the role for anything else.
  • A bare pg.Pool with no ssl option and sslmode=require throws SELF_SIGNED_CERT_IN_CHAIN on first query; postgres.js doesn't have this trap. Never "fix" a live, already-working connection string to match this rule — a working value beats a theoretically-correct one.
  • Check migration drift before running db:deploy against any pre-existing DB: Drizzle's journal compares timestamps, not hashes, so schema applied out-of-band looks like drift. If the objects already exist, baseline the ledger rows instead of re-running the migration.
  • Sweep for rowsecurity=false every time you check drift — it's nearly free and once caught a real live P0 (unauthenticated cross-tenant CRUD+TRUNCATE) on a table added after the repo's blanket RLS migration.
  • Vercel's bulk env?decrypt=true list endpoint can return ~1000-char garbage for any var regardless of type — always re-verify via the single-var endpoint before concluding a stored secret is corrupted. Also watch for ignoreCommand using a bare HEAD^, which silently skips a batched multi-commit push.
  • If org/role resolution is a per-request DB lookup rather than a JWT claim, check for the JIT-provisioning onboarding gap: a fresh zero-membership signup can get floored to a phantom non-empty role by an auth-fallback helper duplicated in more than one place in the same codebase.
  • runtime-env-config — the app-side loader/EnvProvider chain this skill's platform env entries are read by at runtime.
  • local-dev-stack — the local counterpart: the same ENV-staged stack, process-composed and migrated locally instead of by hand against Supabase Cloud.

Sourced from: skills/prod-deploy/metadata.json, skills/prod-deploy/SKILL.md, git log --follow -- skills/prod-deploy (commits 32d31d9, e3f255f), docs/sessions/2026-08-05-fleet-migration-progress/SUMMARY.md, docs/sessions/2026-08-05-fleet-migration-progress/FOLLOWUPS.md, .omc/plans/quality-loop-skills.md (local research artifact, not tracked in this repo)