Skip to main content

docker-ecr-caprover-migration

Why It Exists​

The skill was written after a real two-phase rollout in one personal portfolio: first 20+ repos moved off a platform's git-backed deploy onto Docker images in a private AWS ECR registry, then 41 of those services were cut over onto a single self-hosted CapRover box (app-02, a Hostinger VPS) over the course of a week. The original migration was motivated by the hosted platform's GitHub App integration being observed to silently drop its installation credentials, breaking auto-deploy with no visible error; the CapRover move was a cost and control decision on top of that.

The rollout's own LESSONS.md records the finding that shaped this skill's single most emphasized rule: every .dockerignore written before it was caught used bare patterns (.env*, node_modules), which only match at the Docker build context root, not recursively — unlike .gitignore's default behavior. Every real secret file in every real repo lived one directory down (apps/api/.env.prod.local), so the exclusion had been silently doing nothing. Real DATABASE_URLs, API keys, and access tokens were baked into pushed ECR images across at least four repos — including images pushed independently by a second, concurrent session working the same repos, who hit the identical bug with no coordination between the two sessions. The account owner was presented the full exposure assessment and explicitly decided against credential rotation given the contained (private-ECR-only) blast radius — a decision this skill's history preserves rather than re-litigates.

Two more real bugs from that rollout became load-bearing rules. pnpm deploy --prod --legacy silently dropped a freshly-built dist/ directory in two repos because each package had its own local .gitignore listing dist — pnpm respects that file when copying, regardless of build freshness (tracked upstream as pnpm/pnpm#7286). And in one repo, a literal port of the old deploy command's DATABASE_URL="$DIRECT_URL" shell trick into a Node env override was caught by boot-testing before it ever reached production — the repo's own prisma.config.ts already implemented the same fallback, and the naive port would have silently overwritten a real DATABASE_URL with undefined whenever DIRECT_URL happened to be unset.

The CapRover phase added its own set. The one that cost the most production trust: ansible/vars/services.yml's pinned image: tag is a provisioning baseline, not a record of what's deployed — and the Ansible apps role's desired-state enforcement silently reverted a CI-deployed image back to that stale tag when an unrelated --tags apps run touched the same service. And the one that shaped every deploy workflow since: CapRover reporting status: 100 "Deploy is done" is not the same thing as the app being able to serve. Measured live on 2026-09-17, a real 502 (nginx, "Powered by CapRover") persisted for several seconds after that response, recovering ~10–15s later.

What It Does​

Migrates one repo's service onto a Docker image in a private AWS ECR registry and deploys it to a CapRover app on the self-hosted box, which pulls the image itself. The AWS account, IAM policies, and GitHub OIDC role already exist for this portfolio — this skill is explicitly the per-repo instance of an already-decided pattern, not a from-scratch infrastructure decision each time.

A structural note worth stating plainly: there is no registry-credential object and no credential-refresh Lambda in the CapRover world. The previous pattern needed a per-workspace ECR credential plus a scheduled Lambda refreshing it ahead of AWS's 12-hour token expiry, an EventBridge rule, and a RENDER_TARGETS config listing every workspace. caprover deploy sends the image reference using the operator's own AWS credentials, and the box's Docker daemon does the pull. That entire layer is gone and must not be recreated.

It walks through: reading the app's live definition from CapRover's appDefinitions API (not a static config file, which drifts — and whose drift has caused a real production revert); checking real repo size before writing .dockerignore, since an oversized Terraform provider cache or a native mobile app tree produces the exact same symptom as the recursion bug but needs a completely different fix; choosing among three Dockerfile shapes depending on whether the target package can safely use pnpm deploy pruning; moving any migration step into a container-entry wrapper that runs before the server starts — mandatory on CapRover, where the image already is the built artifact and there is no deploy-time hook to bolt a migration onto; building for linux/amd64 with attestations disabled, since an arm64-plus-attestation-manifest image produces a pull error that reads exactly like a credential failure; and deploying with a single caprover deploy --appToken call followed by a real health poll.

How To Use It​

Triggers on: "move X to CapRover", "dockerize the API for X", "cut X over to CapRover", "deploy X to the Hostinger box", any request naming a repo plus a CapRover app name (hullkit-api, dealnight-api) in the same breath, or a report that a git-backed deploy is flaky.

skills add git@github.com:catesworks/next-starters.git --skill skills/docker-ecr-caprover-migration -g
npm install @next-starters/skill-docker-ecr-caprover-migration
/plugin marketplace add catesworks/next-starters
/plugin install docker-ecr-caprover-migration@next-starters

Gotchas & Invariants​

  • Every .dockerignore pattern needs a **/ prefix. A bare pattern only matches the build context root — the rule the whole rollout learned the hard way, repeatedly, because a bare pattern "looking like it works" in one repo's specific directory layout doesn't mean it's actually depth-agnostic.
  • Verify a .dockerignore change by extracting the real image (docker create + docker cp + find -iname ".env*"), not by reading the pattern and reasoning about whether it looks right — it will look right regardless.
  • Before writing .dockerignore for a new repo, run du -sh */ (and du -sh apps/*/ if relevant). Twice in this rollout a slow build turned out to be a domain-specific oversized directory (a Terraform provider cache, a native mobile/TV app tree) that the standard exclude list had never needed to cover before — not the recursion bug.
  • If the target package (or a dependency it needs at runtime) has its own local .gitignore listing dist, pnpm deploy --prod --legacy will silently exclude the freshly-built output. Switch to an unpruned full copy instead of fighting it.
  • Read the app's live definition from GET /api/v2/user/apps/appDefinitions before believing anything about what's deployed. Match deployedVersion against the version value inside versions[] — never index versions[deployedVersion], which is a version number, not a list position.
  • ansible/vars/services.yml's image: tag is the provisioning baseline only, and it is enforced: the apps role will silently revert a CI-deployed image back to that tag on the next --tags apps run for any reason. Keep it in sync after every CI deploy. This is a real production incident, not a style preference.
  • CapRover's appDefinitions/update endpoint replaces the entire app definition. A partial payload silently resets containerHttpPort, instanceCount, envVars, volumes, notExposeAsWebApp, forceSsl and serviceUpdateOverride to defaults. Always do a full read-modify-write.
  • CapRover defaults containerHttpPort to 80 at app creation and never infers it from the image. A healthy container listening on 4010 got a 502 from CapRover's own nginx until the port was pinned — and the app read its port from an env var, so both sides needed pinning to stop them drifting apart on the next build.
  • An App Token is deploy-only. It cannot read app status (GET appData/<app> returns "Auth token corrupted"), and its header is x-captain-app-token, not the admin token's x-captain-auth. Sending one as the other produces the same misleading error.
  • caprover deploy is genuinely one call — no config-then-deploy two-step — but a zero exit is not proof the app is serving. CapRover treats both 100 (OKAY) and 101 (OKAY_BUILD_STARTED) as success and the CLI exits 0 on either; and even a real 100 precedes readiness by several seconds. Poll the app's health endpoint: 30 attempts at 10s intervals.
  • Assert on the deep health body ("status":"ok" and "db":"ok"), not just a 200. A Swarm rolling update can leave the old container serving while the new one crash-loops, and a liveness 200 in that window is the old container answering.
  • The health gate deliberately does not auto-rollback — it prints the exact manual recovery command and fails. "Stop, alert, human decides" matches the Ansible role's own contract; an automatic rollback would hide the signal.
  • A pure background worker with no HTTP endpoint gets no health gate at all — that's the correct configuration, confirmed live, not a gap. Verify it from docker service logs instead of inventing a synthetic endpoint.
  • CapRover has no native cron/scheduled-job type. Cron-shaped work is either a type: worker app running its own scheduler loop, or an OS-level root crontab entry running docker run --rm --env-file on the box. Don't invent a CapRover cron primitive; there isn't one.
  • CapRover has no per-service command override at the platform level. The image's CMD is authoritative; serviceUpdateOverride's Command/ Args is the real escape hatch, and CapRover stores that field as a JSON string, not a nested object.
  • Always build with docker buildx build --platform linux/amd64 --provenance=false --sbom=false and verify the manifest (docker buildx imagetools inspect) before attempting a cutover — the generic pull error this prevents looks identical to a credential failure and cost real debugging time the first time it appeared.
  • ECR repos here are IMMUTABLE, so docker/metadata-action@v5 needs flavor: | latest=false. Without it, the second real release fails outright trying to overwrite :latest. Hit for real once, caught proactively on the next repo.
  • Don't "clean up" the catesworks@* pattern in the OIDC role's trust policy. Transferred repos carry a sub_claim_prefix override with org and repo ids embedded (repo:catesworks@322158710/hullkit@1341816730), which the plain catesworks/* pattern does not match.
  • Any DB migration moves into a container-entry wrapper. On CapRover there's no build-vs-deploy command distinction at all — the image is the artifact, so the migration must be in the image and invoked by the entrypoint, and a distroless/slim runtime has no shell for && anyway.
  • When porting an old deploy command's env-var handling into that wrapper, check whether the app's own config already implements the same fallback first — replicating it a second time can silently overwrite a real value with undefined. And a VAR=x command override belongs in a child-process env: option, never a process.env mutation, which persists into whatever gets imported afterward.
  • If another session might be touching the same repo, check git branch --show-current before any git operation — this rollout had a working directory silently switched to an unrelated branch mid-task by a concurrent process.
  • Secrets don't only hide in dotfiles — an env-mirror.mjs-style script mirroring real secrets into a differently-named env-mirrors/ directory leaked into a build the .env* pattern never caught. And a repo already having a working Dockerfile doesn't mean it's safe: one repo was already building its own Dockerfile with zero .dockerignore at all, so its explicit COPY steps would bake in secrets a real git clone never has.
  • A stale local *.tsbuildinfo file (gitignored, so a real clone never has one, but a dev machine's disk does) can make tsc --build believe a workspace dependency is already compiled and silently skip it — the build step reports no error; the missing module only surfaces at boot-test time. Exclude **/*.tsbuildinfo alongside **/dist.
  • When one repo has multiple services built from the same install/build base, give the shared build stage named final targets (--target runtime-api vs --target runtime-outbox) instead of rebuilding per service — push the same image to each service's own ECR repo, and give each CI matrix entry its own type=gha cache scope.
  • CapRover serializes mutating API calls per namespace and returns 429 when a second arrives while the first holds the lock. The deploy call holds it for the whole Swarm service update (4.3s warm, up to 180s on a cold pull). A single caprover deploy doesn't need to care; a concurrent Ansible run against the same box does.
  • Not every migration ends fully green, and that's the correct outcome to report. One background worker started correctly, then got an external SIGTERM about a minute in with no crash logged and no restart — confirmed it wasn't the app's own code, wasn't fleet-wide, and reproduced identically across two deploys, then stopped investigating and told the user plainly rather than guessing further or quietly calling it done.
  • Where a fact couldn't be verified, the skill says so rather than guessing. The genuine cold-pull timing on the caprover deploy path has never been measured — an attempt was invalidated because the two tags differed by 941 bytes out of ~176MB, so the box's layer cache made it meaningless. Ansible's ~34s figure is the only real data point, and the 300s health budget is sized generously rather than empirically.
  • prod-deploy — the equivalent "first production deploy" checklist for a repo that isn't image-backed at all yet; this skill is what to reach for once a repo already has a working deploy and needs to move it onto ECR + CapRover.
  • runtime-env-config — the app-side env loader this skill's containers still depend on at runtime; Docker packaging doesn't change how a repo resolves its own env vars, only how the image ships.

Sourced from: skills/docker-ecr-caprover-migration/metadata.json, skills/docker-ecr-caprover-migration/SKILL.md, the original cates-works/docs/sessions/2026-08-22-ecr-docker-migration-rollout/ dossier (SUMMARY.md, LESSONS.md, ARCHITECTURE.md, adr/0001, adr/0002), and the CapRover-phase material in hostinger/ — CLAUDE.md, ansible/roles/apps/tasks/caprover_service.yml, .../cron_service.yml, ansible/vars/services.yml, .omc/plans/fix-docker-caprover-synchronous-deploy.md, and the live hullkit/.github/workflows/docker-release.yml + docker-caprover.yml workflows.