Source:
docs/deployment-workflow.md
How code gets from a branch to production. This documents the CI/CD pipeline
— the GitHub Actions workflows, what gates what, and how to recover when a
deploy stalls.
For one-time infrastructure provisioning (creating the DO app, databases,
Spaces, env vars), see deployment-do.md. This doc assumes
that all already exists.
| What | Branch | Workflow | Target |
|---|---|---|---|
apps/api/** |
main |
deploy-do.yml |
DOCR → DO App Platform component api + PRE_DEPLOY migrate |
apps/baileys-connector/** |
main |
deploy-do.yml |
DOCR → DO App Platform component connector |
apps/web/** |
main |
deploy-web.yml |
Vercel (production) |
development is the integration branch and deploys nowhere. Releasing means
opening a development → main PR and merging it. main is the production line.
There is no staging API. api.replybox.utopiagroup.com.my is live traffic.
Neither workflow has a pull_request trigger. A development → main PR shows
Checks 0 and the merge button is never blocked by CI. Every gate in this
pipeline fires after the merge lands on main. That is a deliberate
consequence of the current trigger config, not an oversight you're seeing for the
first time — but it means a PR being green tells you nothing.
deploy-do.yml — API and connectoron:
push:
branches: [main]
paths: ['apps/api/**', 'apps/baileys-connector/**', '.github/workflows/deploy-do.yml']
workflow_dispatch:
inputs: { api: bool (default true), connector: bool (default false) }
concurrency: { group: deploy-do, cancel-in-progress: false } — overlapping runs
queue, they don't cancel each other. Two concurrent rollouts of the same
component is the failure this prevents.
changes ──▶ migration-check ──▶ build-connector ──▶ build-api
(detect) (alembic gate) (if connector) (if api)
Both build jobs depend on migration-check. A failed gate means no image is
built, no tag moves, and no deployment is ever created — production keeps
serving the current image untouched.
changes (Detect changed components)Sets two booleans, api and connector, that gate everything downstream.
On workflow_dispatch it takes the inputs verbatim and does no diffing.
On push it runs a plain git diff --name-only $BEFORE $SHA and
prefix-matches ^apps/api/ and ^apps/baileys-connector/. This is deliberately
hand-rolled rather than a paths-filter action: run #56 had a filter action report
api=false for a commit that changed apps/api/Dockerfile and
app/models/organization.py, so both builds skipped and the run went green
while deploying nothing. Silent no-deploy is the worst failure mode this
workflow can have, so the decision is made in shell that prints its own file
list to the log.
$BEFORE is github.event.before — whatever main pointed at immediately
before this push. Not the merge base, not the PR's base. If it's absent or
unresolvable (first push of a branch, shallow clone) it falls back to $SHA^,
never to "nothing changed".
⚠️ The diff-window trap. Because the window is one push wide, changes can
fall through the cracks across two runs. See
§6 Failure modes → Gate failed, then I pushed a fix.
A commit touching only deploy-do.yml matches neither prefix — the run is a
deliberate no-op. Use workflow_dispatch to force a rebuild.
migration-check (Verify alembic upgrade head)Runs when api == true || connector == true. Spins up a postgres:16 service
container and replays every revision (~129) from an empty database.
Four steps, two of which are non-obvious:
Assert a single head — alembic heads | grep -c '(head)' must equal 1.
Needs no database. This is the single most common way the deploy breaks: two
branches each add a migration off the same parent, neither is a merge
revision, and alembic upgrade head aborts with "Multiple head revisions are
present for given argument 'head'" before applying anything.
This is not hypothetical or rare: ls apps/api/alembic/versions/ | grep merge
returns 14 merge revisions, one per time it has already happened. Assume
it will happen to you and check before promoting (§5 step 2).
Pre-create alembic_version at production width — creates the table with
VARCHAR(255). Production and local dev have it at 255, but a database
bootstrapped by current Alembic gets the VARCHAR(32) default, and 22 revision
ids are longer than that (longest: 0024_scheduled_messages_lifecycle_timestamps,
44 chars). Without this the replay dies at
0023_scheduled_messages_queued_status with StringDataRightTruncationError
— an artifact of bootstrapping empty, not a defect in the migration being
deployed. This step is what makes the check mirror prod instead of failing
every run.
alembic upgrade head — the actual rehearsal. Migrations create their own
extensions (citext, pg_trgm, pgcrypto, all IF NOT EXISTS) and all three
ship with the stock postgres image.
Assert the database landed on head — alembic current must report
(head).
Scope — what this proves and what it does not. It catches invalid SQL, a bad
op.* call, a revision referencing a missing table, and multiple heads. It
cannot catch a failure that depends on production data — adding a NOT NULL
column to a table that already has rows, or a unique index over values already
duplicated in prod. Those still surface only at deploy time, in the PRE_DEPLOY
job, after images are published and the rollout has started.
build-connector (Build & push connector image)Guarded by if: needs.changes.outputs.connector == 'true'. This job restarts
every live WhatsApp session — the connector is instance_count: 1 and holds
every socket, so a new image tears down and re-pairs every channel (sessions are
restored from the encrypted blobs in Spaces, but it is a real blip). That is why
component-level path filtering exists at all: before it, an api-only commit still
pushed replybox-connector:latest, killing every session for a change that never
touched it.
Sequence:
:<sha> — one build, linux/amd64, GHA layer cache.:latest at the same manifest — docker buildx imagetools create,:latest moved — reads doctl registry repository list-manifests -o jsonlatest and :<sha> digests are identical. Retries 5× / 10sACTIVE. ERROR/CANCELED/SUPERSEDEDDo not "simplify" steps 2–3 into a multi-tag
tags: |block. Tried
2026-08-03: DOCR applied only the:shatag,lateststayed on the previous
digest, and App Platform redeployed the old image while reporting success.
Do not build twice either. That was the 2026-08-03 fix and it got the whole
workflow reverted on 2026-08-04 —build-push-action@v6defaults to
provenance: true, so each invocation wraps the image in an OCI index plus an
attestation manifest carrying per-build metadata, and two builds of
byte-identical layers mint two different index digests.imagetools create
copies by digest, so both tags resolve to one digest by construction, and it
preserves the ordering that matters::shafirst,:latestlast as the deploy
trigger.
build-api (Build & push api image)needs: [changes, migration-check, build-connector]
if: always()
&& needs.changes.outputs.api == 'true'
&& needs.migration-check.result == 'success'
&& needs.build-connector.result != 'failure'
&& needs.build-connector.result != 'cancelled'
always() is needed so this still runs when build-connector was skipped —
the common case, an api-only commit. Note that migration-check is asserted
positively (== 'success'), not by excluding failure: under always() a
!= 'failure' test would let cancelled and skipped through, and a skipped
migration check must never read as a passing one.
Same build/tag/verify sequence as the connector, minus the rollout wait.
Connector rolls before api, always. Two :latest pushes in parallel create
two independent App Platform deployments and the second CANCELs the first
mid-rollout — that is how one commit killed the connector twice. It is also the
ordering CLAUDE.md requires when a new queue is introduced (connector must be
able to consume a job kind before the API can produce it).
There is no doctl apps create-deployment step anywhere in this workflow.
App Platform watches each component's :latest tag with deploy_on_push.
Moving the tag is the trigger. Everything the workflow does is: build → tag
:sha → copy to :latest → verify → (connector only) wait for ACTIVE.
Corollary: if :latest doesn't move, nothing deploys — regardless of what
landed on main.
Once a deployment starts, the app's migrate component (kind: PRE_DEPLOY,
reuses the api image, runs sh scripts/release.sh → alembic upgrade head && python -m app.cli seed) runs on every deployment, including a connector-only
one. So a migration that fails on production data doesn't just fail the api
rollout — it fails the whole app's deployment, after the images are already
published.
App id 7894a462-65dd-4830-940c-ce274f6d2a03 (overridable via repo variable
DO_APP_ID).
deploy-web.yml — frontendSimple and ungated. On push to main touching apps/web/**:
checkout → pnpm 9.5.0 → node 20 → vercel pull --environment=production → vercel build --prod → vercel deploy --prebuilt --prod
The one non-obvious step is Debug pulled env, which greps
NEXT_PUBLIC_API_URL out of .vercel/.env.production.local and prints whether
it was retrieved. That value is a public URL inlined into the client bundle at
build time; if a project/scope misconfiguration means vercel pull doesn't get
it, the build silently ships a bundle that POSTs to the web app's own origin and
404s on /api/v1/.... The grep makes that fail visibly.
No migration gate, no lint, no typecheck, no tests run here. pnpm build failing
is the only thing that stops a bad frontend deploy.
| Kind | Name | Used by |
|---|---|---|
| secret | DIGITALOCEAN_ACCESS_TOKEN |
deploy-do.yml — DO API token, registry read/write + app scope |
| secret | VERCEL_ORG_ID / VERCEL_PROJECT_ID / VERCEL_TOKEN |
deploy-web.yml |
| variable | DOCR_REGISTRY |
deploy-do.yml — DOCR registry name |
| variable | DO_APP_ID |
deploy-do.yml — optional, falls back to the literal in env: |
development via PR.cd apps/api && alembic heads # must print exactly one line
development first:alembic merge heads -m "merge <branch-a> and <branch-b>"
main and a manual recovery (§6).development → main. Remember: Checks 0 is expected, and proves nothing.ACTIVE and /healthz is 200.The one to internalize. Sequence:
| Run | Diff window ($BEFORE..$SHA) |
api | connector | Outcome |
|---|---|---|---|---|
| A — merge the promote PR | main@old … merge |
true | true | migration-check ❌, both builds skipped |
| B — push the alembic fix | merge … fix (1 file) |
true | false | ✅ green, api deploys, connector does not |
Run A detected the connector correctly but never built it. Run B's diff window
contains only the merge revision under apps/api/, so connector=false,
build-connector is skipped by its own if:, replybox-connector:latest never
moves, and App Platform never redeploys that component.
Net result: main carries new connector code, production runs the old image,
and the run is green. Exactly the run-#56 silent-no-deploy failure mode,
re-entered through a different door — the diff-window logic has no memory of what
a previous failed run was supposed to build.
Recovery — force both components explicitly:
gh workflow run deploy-do.yml --ref main -f api=true -f connector=true
(or Actions → deploy-do.yml → Run workflow → branch main → tick both)
workflow_dispatch bypasses the git diff entirely and uses the inputs. Expect
the connector restart to blip every WhatsApp session.
Prevention: merge alembic heads on development before promoting (§5 step 2).
| Symptom | Cause | Fix |
|---|---|---|
| Run green, nothing deployed | changes marked both false — check the printed file list in the job log |
workflow_dispatch with the right boxes ticked |
2 alembic heads present |
Two migrations share a down_revision |
alembic merge heads, commit, then dispatch both components |
:latest did not move to this build's digest |
Registry lag past 50s, or a tag write that silently failed | Re-run the job; if it persists, check DOCR manifests by hand with doctl registry repository list-manifests <repo> -o json |
deployment X ended in CANCELED/SUPERSEDED |
A second deployment stacked on top mid-rollout | Check for a concurrent dispatch or a manual DO-console deploy; re-run once settled |
deployment X did not go ACTIVE within 10m |
Rollout genuinely stuck | DO console → Activity → component logs. Do not stack another deploy on top |
Deploy dies in PRE_DEPLOY migrate |
Migration is data-dependent (NOT NULL on populated table, unique index over dupes) — migration-check cannot catch this |
Fix forward; the old version keeps serving until the new one goes live |
Web bundle 404s on /api/v1/... |
NEXT_PUBLIC_API_URL wasn't in the pulled Vercel env |
Check the Debug pulled env step output; fix the Vercel project env |
Repoint the component's image tag to a known-good :<git-sha> in the App
Platform dashboard and redeploy. Every build pushes a :<sha> tag alongside
:latest, so any previously-deployed commit is directly addressable.
Note that a rollback does not roll back migrations — the PRE_DEPLOY job only
ever moves forward. An old image against a newer schema is usually fine (columns
are additive) but is not guaranteed.
Documented so nobody rediscovers them the hard way:
push: branches: [main] only.pull_request: branches: [main] to deploy-do.yml would letmigration-check run on the PR — it needs no secrets, only the postgresmain. Thepytest, ruff, mypy,pnpm lint, pnpm typecheck and the connector's vitest suite run nowhere inmigration-check replays against an empty database, so data-dependent