apps/api/**
¶ API
Branch
main
Workflowdeploy-do.yml
TargetDOCR → DO App Platform component api + PRE_DEPLOY migrate
Source: docs/deployment-workflow.md
¶ Deployment Workflow
How code gets from a branch to production. This documents the CI/CD pipeline — the GitHub Actions workflows, what gates what, and how to recover when a deploy stalls.
main is production
development deploys nowhere
Nothing runs on pull requests
Alembic replay is the only gate
For one-time infrastructure provisioning (creating the DO app, databases, Spaces, env vars), see the DigitalOcean deployment guide (deployment-do.md). This doc assumes that all already exists.
1. The map
¶ What deploys where
apps/baileys-connector/**
¶ Connector
Branch
main
Workflowdeploy-do.yml
TargetDOCR → DO App Platform component connector
apps/web/**
¶ Web
developmentThe integration branch. Deploys nowhere.
ReleasingOpen a development → main PR and merge it. main is the production line.
No staging APIapi.replybox.utopiagroup.com.my is live traffic.
Nothing runs on pull requests. Neither workflow has a pull_request trigger. A development → main PR shows Checks 0 and the merge button is never blocked by CI. Every gate in this pipeline fires after the merge lands on main. That is a deliberate consequence of the current trigger config, not an oversight you're seeing for the first time — but it means a PR being green tells you nothing.
2. deploy-do.yml
¶ API and connector
Triggers
on:
push:
branches: [main]
paths: ['apps/api/**', 'apps/baileys-connector/**', '.github/workflows/deploy-do.yml']
workflow_dispatch:
inputs: { api: bool (default true), connector: bool (default false) }
Concurrency
concurrency: { group: deploy-do, cancel-in-progress: false } — overlapping runs queue, they don't cancel each other. Two concurrent rollouts of the same component is the failure this prevents.
Both build jobs depend on migration-check. A failed gate means no image is built, no tag moves, and no deployment is ever created — production keeps serving the current image untouched.
1changes
Detect which components changed.
2migration-checkThe alembic gate.
3build-connectorIf connector changed.
4build-apiIf api changed.
Job graph (text)
Job graph
changes ──▶ migration-check ──▶ build-connector ──▶ build-api
(detect) (alembic gate) (if connector) (if api)
1
Job 1 · changes
¶ Detect changed components
Sets two booleans, api and connector, that gate everything downstream.
On
workflow_dispatchTakes the inputs verbatim and does no diffing.
On pushRuns a plain git diff --name-only $BEFORE $SHA and prefix-matches ^apps/api/ and ^apps/baileys-connector/.
Why hand-rolled, not a paths-filter action: run #56 had a filter action report api=false for a commit that changed apps/api/Dockerfile and app/models/organization.py, so both builds skipped and the run went green while deploying nothing. Silent no-deploy is the worst failure mode this workflow can have, so the decision is made in shell that prints its own file list to the log.
$BEFOREgithub.event.before — whatever main pointed at immediately before this push. Not the merge base, not the PR's base.
FallbackIf it's absent or unresolvable (first push of a branch, shallow clone) it falls back to $SHA^, never to "nothing changed".
Workflow-only commitA commit touching only deploy-do.yml matches neither prefix — the run is a deliberate no-op. Use workflow_dispatch to force a rebuild.
The diff-window trap. Because the window is one push wide, changes can fall through the cracks across two runs. See §6 Failure modes → Gate failed, then I pushed a fix.
2
Job 2 · migration-check
¶ Verify alembic upgrade head
Runs when api == true || connector == true. Spins up a postgres:16 service container and replays every revision (~129) from an empty database. Four steps, two of which are non-obvious:
Assert a single head —
This is not hypothetical or rare:
alembic heads | grep -c '(head)' must equal 1. Needs no database. This is the single most common way the deploy breaks: two branches each add a migration off the same parent, neither is a merge revision, and alembic upgrade head aborts with "Multiple head revisions are present for given argument 'head'" before applying anything.This is not hypothetical or rare:
ls apps/api/alembic/versions/ | grep merge returns 14 merge revisions, one per time it has already happened. Assume it will happen to you and check before promoting (§5 step 2).Pre-create
alembic_version at production width — creates the table with VARCHAR(255). Production and local dev have it at 255, but a database bootstrapped by current Alembic gets the VARCHAR(32) default, and 22 revision ids are longer than that (longest: 0024_scheduled_messages_lifecycle_timestamps, 44 chars). Without this the replay dies at 0023_scheduled_messages_queued_status with StringDataRightTruncationError — an artifact of bootstrapping empty, not a defect in the migration being deployed. This step is what makes the check mirror prod instead of failing every run.alembic upgrade head — the actual rehearsal. Migrations create their own extensions (citext, pg_trgm, pgcrypto, all IF NOT EXISTS) and all three ship with the stock postgres image.Assert the database landed on head —
alembic current must report (head).
CatchesInvalid SQL, a bad
op.* call, a revision referencing a missing table, and multiple heads.
Cannot catchA failure that depends on production data — adding a NOT NULL column to a table that already has rows, or a unique index over values already duplicated in prod. Those still surface only at deploy time, in the PRE_DEPLOY job, after images are published and the rollout has started.
3
Job 3 · build-connector
¶ Build & push connector image
Guarded by if: needs.changes.outputs.connector == 'true'.
This job restarts every live WhatsApp session — the connector is instance_count: 1 and holds every socket, so a new image tears down and re-pairs every channel (sessions are restored from the encrypted blobs in Spaces, but it is a real blip). That is why component-level path filtering exists at all: before it, an api-only commit still pushed replybox-connector:latest, killing every session for a change that never touched it.
Record current deployment — snapshots the newest DO deployment id before the push, so the wait step can tell "the rollout I triggered" from the one already ACTIVE.
Build & push
:<sha> — one build, linux/amd64, GHA layer cache.Point
:latest at the same manifest — docker buildx imagetools create, a registry-side copy by digest.Verify
:latest moved — reads doctl registry repository list-manifests -o json and asserts the latest and :<sha> digests are identical. Retries 5× / 10s because the registry listing lags a push.Wait for the rollout to finish — polls up to 3 min for a new deployment to appear, then up to 10 min for it to reach
ACTIVE. ERROR/CANCELED/SUPERSEDED fails the job. If no deployment appears within 3 min it emits a warning and proceeds rather than blocking the api push.Do not "simplify" steps 2–3 into a multi-tag tags: | block. Tried 2026-08-03: DOCR applied only the :sha tag, latest stayed on the previous digest, and App Platform redeployed the old image while reporting success. Do not build twice either. That was the 2026-08-03 fix and it got the whole workflow reverted on 2026-08-04 — build-push-action@v6 defaults to provenance: true, so each invocation wraps the image in an OCI index plus an attestation manifest carrying per-build metadata, and two builds of byte-identical layers mint two different index digests. imagetools create copies by digest, so both tags resolve to one digest by construction, and it preserves the ordering that matters: :sha first, :latest last as the deploy trigger.
4
Job 4 · build-api
¶ Build & push api image
Job condition
needs: [changes, migration-check, build-connector]
if: always()
&& needs.changes.outputs.api == 'true'
&& needs.migration-check.result == 'success'
&& needs.build-connector.result != 'failure'
&& needs.build-connector.result != 'cancelled'always() is needed so this still runs when build-connector was skipped — the common case, an api-only commit. Note that migration-check is asserted positively (== 'success'), not by excluding failure: under always() a != 'failure' test would let cancelled and skipped through, and a skipped migration check must never read as a passing one.
Same build/tag/verify sequence as the connector, minus the rollout wait.
Connector rolls before api, always. Two :latest pushes in parallel create two independent App Platform deployments and the second CANCELs the first mid-rollout — that is how one commit killed the connector twice. It is also the ordering CLAUDE.md requires when a new queue is introduced (connector must be able to consume a job kind before the API can produce it).
How the deployment is actually triggered
¶ Moving :latest is the trigger
There is no doctl apps create-deployment step anywhere in this workflow. App Platform watches each component's :latest tag with deploy_on_push. Everything the workflow does is:
build
tag
:sha
copy to :latest
verify
(connector only) wait for ACTIVE
Corollary: if :latest doesn't move, nothing deploys — regardless of what landed on main.
PRE_DEPLOY
migrateThe app's migrate component (kind: PRE_DEPLOY, reuses the api image, runs sh scripts/release.sh → alembic upgrade head && python -m app.cli seed) runs on every deployment, including a connector-only one.
Blast radiusA migration that fails on production data doesn't just fail the api rollout — it fails the whole app's deployment, after the images are already published.
App id7894a462-65dd-4830-940c-ce274f6d2a03 (overridable via repo variable DO_APP_ID).
2b. After the rollout
¶ The post-deploy check reads an API aggregate, not the database
CI runners often cannot open network connections to managed Postgres instances, so nothing in GitHub Actions relies on direct DB access for this check.
¶ Where the numbers come from
The baseline step calls GET /api/v1/healthz/channels (aggregate numbers only, no auth). It returns connected, total, and by_status counts — never channel ids, phones, labels, or organization identifiers.
¶ First rollout tolerates a 404
The rollout that introduces the endpoint may return 404. The workflow then records an empty baseline and emits a warning rather than failing the release; subsequent rollouts compare before and after counts.
¶ Host override
Set REPLYBOX_API_BASE (or rely on the repo default) if the check needs a non-standard API host. It no longer requires a read-only DB secret for this count.
vercel build --prod
vercel deploy --prebuilt --prod --skip-domain
verify staged
report staged vs. live
The deploy is staged
¶ It does not go live on its own
--skip-domain creates the production deployment without assigning replybox-dev.vercel.app or replybox.utopiagroup.com.my to it. Vercel marks it Staged; both domains keep serving the previous build until a human promotes it. A green Deploy Web run means "the frontend built and is ready to promote", not "the frontend is live".
Why: observed on 62ba226
When this workflow assigned the domains itself, the new frontend went in front of the old api for the rest of the DO run.
Independent workflowsThis workflow and
deploy-do.yml both fire on the same push to main, neither needs: the other, and they sit in separate concurrency groups, so GitHub runs them in parallel. This one is much the shorter: --prebuilt makes the Vercel side a ~10 second upload, while deploy-do builds two Docker images.
Exposure outlasts the green tickbuild-api only pushes :latest and exits: the App Platform rollout, PRE_DEPLOY migrate included, is still in progress when deploy-do reports success.
Not a dashboard toggle--skip-domain overrides the project's Auto-assign Custom Production Domains setting, so this does not depend on a dashboard toggle staying off. Both production domains are registered on the Vercel project, so one Promote assigns both.
Promoting
¶ Wait for the API, then promote
Wait for deploy-do to finish and for the App Platform deployment to reach ACTIVE (DO console → Activity, or /healthz returning the new build). Then either:
Vercel dashboard → Deployments → the staged deployment →
… → Promoteor
vercel promote <deployment-url>Promotion assigns both domains at once and needs no rebuild. The deployment
The source doc's sentence ends here, cut off mid-thought.
The one non-obvious step
¶ Debug pulled env
Greps NEXT_PUBLIC_API_URL out of .vercel/.env.production.local and prints whether it was retrieved. That value is a public URL inlined into the client bundle at build time; if a project/scope misconfiguration means vercel pull doesn't get it, the build silently ships a bundle that POSTs to the web app's own origin and 404s on /api/v1/.... The grep makes that fail visibly.
No migration gate, no lint, no typecheck, no tests run here. pnpm build failing is the only thing that stops a bad frontend deploy.
3b. Promote Web
¶ Promoting a staged deployment from CI
A manual GitHub Action, Promote Web (.github/workflows/promote-web.yml), promotes a Vercel deployment that was intentionally staged with --skip-domain to the production domains. It replaces the Vercel dashboard click while verifying project, commit, and aliases, so nothing is promoted accidentally.
deployment_idrequiredThe staged deployment to promote
expected_shaoptionalDefaults to the tip of main at run time
VERCEL_TOKEN / VERCEL_ORG_ID / VERCEL_PROJECT_IDsecretRequired in the repository
¶ Guards before promoting
The job validates that the deployment belongs to this project, is READY, was built as a production-target deployment, and (optionally) matches the expected commit SHA.
¶ Proof, not a 2xx
A 2xx from the promote endpoint alone is not treated as proof. The job polls until both the fixed aliases and the project's production target serve the promoted deployment, within roughly four minutes, and fails loudly if any guard fails.
¶ Manual, and self-contained
Trigger is workflow_dispatch only, and the job does not trigger other deploy workflows. The run has a 10-minute timeout to allow polling and confirmation steps.
4. Required GitHub configuration
¶ Secrets and variables
DIGITALOCEAN_ACCESS_TOKENsecretdeploy-do.yml — DO API token, registry read/write + app scope
VERCEL_ORG_ID / VERCEL_PROJECT_ID / VERCEL_TOKENsecretdeploy-web.yml
DOCR_REGISTRYvariabledeploy-do.yml — DOCR registry name
DO_APP_IDvariabledeploy-do.yml — optional, falls back to the literal in env:
5. Releasing
¶ Five steps from feature branch to ACTIVE
Land feature branches on
development via PR.Before opening the promote PR, check for split alembic heads
Check
cd apps/api && alembic heads # must print exactly one lineIf two branches each added a migration, fix it on development first:
Merge heads
alembic merge heads -m "merge <branch-a> and <branch-b>"Doing this before the merge means one push, one diff window, and both components build correctly in a single clean run. Doing it after means a red main and a manual recovery (§6).
Open
development → main.Remember: Checks 0 is expected, and proves nothing.
Merge. Watch the run in Actions.
Confirm the DO deployment reached
ACTIVE and /healthz is 200.6. Failure modes and recovery
¶ Gate failed, then I pushed a fix — and the connector never deployed
The one to internalize. Sequence:
RunDiff window (
$BEFORE..$SHA)apiconnectorOutcomeA — merge the promote PR
main@old … mergetruetruemigration-check ❌, both builds skippedB — push the alembic fix
merge … fix (1 file)truefalse✅ green, api deploys, connector does notRun A detected the connector correctly but never built it. Run B's diff window contains only the merge revision under apps/api/, so connector=false, build-connector is skipped by its own if:, replybox-connector:latest never moves, and App Platform never redeploys that component.
Net result: main carries new connector code, production runs the old image, and the run is green. Exactly the run-#56 silent-no-deploy failure mode, re-entered through a different door — the diff-window logic has no memory of what a previous failed run was supposed to build.
Recovery — force both components explicitly
Dispatch both
gh workflow run deploy-do.yml --ref main -f api=true -f connector=true(or Actions → deploy-do.yml → Run workflow → branch main → tick both)
workflow_dispatch bypasses the git diff entirely and uses the inputs. Expect the connector restart to blip every WhatsApp session.
Prevention: merge alembic heads on development before promoting (§5 step 2).
Other modes
¶ Symptom, cause, fix
Run green, nothing deployedCause:
changes marked both false — check the printed file list in the job logFix: workflow_dispatch with the right boxes ticked
2 alembic heads presentCause: Two migrations share a down_revisionFix: alembic merge heads, commit, then dispatch both components
:latest did not move to this build's digestCause: Registry lag past 50s, or a tag write that silently failedFix: Re-run the job; if it persists, check DOCR manifests by hand with doctl registry repository list-manifests <repo> -o json
deployment X ended in CANCELED/SUPERSEDEDCause: A second deployment stacked on top mid-rolloutFix: Check for a concurrent dispatch or a manual DO-console deploy; re-run once settled
deployment X did not go ACTIVE within 10mCause: Rollout genuinely stuckFix: DO console → Activity → component logs. Do not stack another deploy on top
Deploy dies in PRE_DEPLOY migrateCause: Migration is data-dependent (NOT NULL on populated table, unique index over dupes) — migration-check cannot catch thisFix: Fix forward; the old version keeps serving until the new one goes live
Web bundle 404s on /api/v1/...Cause: NEXT_PUBLIC_API_URL wasn't in the pulled Vercel envFix: Check the Debug pulled env step output; fix the Vercel project env
Rollback
¶ Repoint to a known-good :<git-sha>
Repoint the component's image tag to a known-good :<git-sha> in the App Platform dashboard and redeploy. Every build pushes a :<sha> tag alongside :latest, so any previously-deployed commit is directly addressable.
Migrations don't roll backThe PRE_DEPLOY job only ever moves forward. An old image against a newer schema is usually fine (columns are additive) but is not guaranteed.
7. Known gaps
¶ Documented so nobody rediscovers them the hard way
¶ No CI on pull requests
Both workflows are push: branches: [main] only. Adding pull_request: branches: [main] to deploy-do.yml would let migration-check run on the PR — it needs no secrets, only the postgres service container — and would catch split heads before they reach main. The build jobs would need guarding off the PR event.
¶ No tests, lint, or typecheck gate anything
pytest, ruff, mypy, pnpm lint, pnpm typecheck and the connector's vitest suite run nowhere in CI. The only automated correctness gate in the entire pipeline is the alembic replay.
¶ migration-check replays against an empty database
So data-dependent migration failures reach production before anyone sees them.