Source:
ReplyBox - Handover.pdf, prepared 1 September 2026.
ReplyBox — Handover
Prepared 1 September 2026. Repository: github.com/utopiabuilder/replybox-dev. This document
is a handover for an engineer taking over or joining ReplyBox. Read it end to end once before
touching anything. The repository's own README.md, CLAUDE.md and docs/ remain canonical
— where this document and the repo disagree, the repo wins.
Read this first — the five things that will hurt you
1. The Baileys connector runs exactly one instance and holds every live WhatsApp
socket. Deploying it tears down and re-pairs every customer channel. Never scale it past
instance_count: 1; never rebuild it for a change that didn't touch it.
2. There is no staging. api.replybox.utopiagroup.com.my is production, with live paying
customers on it. Every test you run against it is a test in production.
3. Nothing runs on pull requests. A green PR proves nothing — there are no CI checks on
PRs at all. Every gate fires after the merge lands on main.
4. Split Alembic migration heads is the single most common way the deploy breaks. It
has already happened 14 times (14 merge revisions in the repo). Check alembic heads
before every release.
5. Nobody pushes, merges or deploys without the owner's explicit approval for that
specific change. Prepare the change, show it, stop. Approval is per change, never
standing.
What ReplyBox is
ReplyBox is a multi-tenant shared inbox SaaS for WhatsApp — a team inbox where multiple
agents work one WhatsApp number together, with assignment, tagging, automation, broadcasts
and a public API. It is self-hosted on DigitalOcean rather than built on a vendor platform, which
is why the operational surface below is as large as it is.
It is both a product and an internal shared service. Roughly 105 organisations use it. Utopia
BD Lead pushes its WhatsApp outreach through it, and the payroll disciplinary bridge sends
Show Cause and Warning Letters through it. Breaking ReplyBox breaks more than ReplyBox.
Feature surface as it stands today
Area
Inbox
Messaging
What exists
Conversations, threads, assignment, status,
pinning, starring, notes, internal activity feed,
search, contact right-rail, command palette
(⌘K), keyboard shortcuts
Text, media
(image/video/audio/voice/document/sticker),
location cards, reactions, replies, edits,
revokes, contact cards, group chats with
@-mentions, read receipts
Channels
QR pairing, lifecycle state machine, health
tracking, migration/reparenting, multi-channel
per org
Transports
Baileys (WhatsApp Web) and Meta WhatsApp
Cloud API, including Coexistence (one number
live on both at once)
Automation
Message Flows (visual flow builder with
conditions, HTTP steps, AI nodes), keyword
auto-replies, scheduled messages, broadcasts
with pacing and send windows
CRM
Contacts directory, CRM board, tags at three
grains, call-log ingest, contact identity/merge
Platform
Organisations, RBAC (16 permissions, 4
system roles + custom roles), invitations,
UtopiaSpace SSO, public REST API +
outbound webhooks, per-org API tokens, bug
reports auto-filed to GitHub, Sentry, notifications
AI
Media captioning / voice transcription and CRM
title generation via Gemini, gated behind env +
per-org flags
Scale markers: ~163 Alembic revisions, ~37 public REST routers, ~20 HMAC-protected internal
endpoints, ~48 ORM models, ~80 service modules, 416 git branches.
Repository layout
apps/api/
FastAPI backend — REST + Socket.IO + queue
producer. THE SOURCE OF TRUTH.
apps/web/
Next.js 14 App Router frontend
apps/baileys-connector/ Node.js Baileys WhatsApp Web adapter —
disposable, restart-safe
infra/docker/
Local Postgres / Redis / MinIO compose stack
(+ pgweb DB GUI behind a profile)
docs/
21 design + runbook documents; all canonical
CLAUDE.md
Contributor conventions and architectural
rules — read before non-trivial work
docs/plan.md
Phased roadmap and what is shipped
SHIPPED.md
Release log with re-runnable verification
evidence
The API is the source of truth. The connector is disposable. Anything the connector knows
that matters must end up in Postgres. Its only durable state is the encrypted session blob in
object storage — you can delete and rebuild the connector container at any time without data
loss (at the cost of a re-pair blip).
Architecture — and the one rule about it
Web ── REST (cookie JWT) ──▶ API ── BullMQ "whatsapp" queue ──▶
connector ──▶ Baileys ──▶ WhatsApp
▲
│
│ Socket.IO room org:{id}
│
inbound events
│
▼
└───────────────────── API ◀── HMAC-signed HTTP ──── POST
/internal/v1/...
Excalidraw (By Nazwa):
There are exactly four transports between the parts, and each has exactly one implementation:
Direction
Mechanism
Where it lives
Web → API
REST over httpOnly cookie
apps/web/lib/api.ts
JWT (access + refresh), axios
refresh interceptor
API → Connector
BullMQ jobs on Redis. The
apps/api/app/queue/producer.p
Python bullmq producer is
y
wire-compatible with the Node
worker.
Connector → API
HMAC-SHA256 signed HTTP Verifier app/security/hmac.py;
POST to /internal/v1/....
signer src/internal/api-client.ts
Signature over
{ts}\n{METHOD}\n{path}\n{body
}, sent as
X-Replybox-Timestamp +
X-Replybox-Signature.
API → Web
Socket.IO, one room per
app/realtime/sio.py,
organisation (org:{id}). Events: app/realtime/emit.py
channel.qr, channel.status,
channel.sync,
message.created,
conversation.updated.
The rule: do not add a fifth path, and do not add a second implementation of any of the
four. No Redis pub/sub between connector and API, no websocket side-channel, no direct DB
writes from the connector. If the existing transport can't carry what you need, extend it. This rule
exists because every parallel path that has been added in similar systems became the thing
nobody remembered to update.
Queues — and which one your job belongs on
Queue
whatsapp
Consumed by
Connector
Purpose
User-initiated WhatsApp
mutations: connect, disconnect,
send, send_media,
send_location, send_reaction,
mark_read, resync_history,
group ops
whatsapp_media
Connector
Media download — slow,
background
whatsapp_avatars
Connector
Avatar fetch — slow,
background
scheduled, flow_runs,
Python workers inside the
They decide when something is
broadcasts, and other
API process
sent, then call the same
orchestration queues
producer functions the
synchronous endpoints call
Two rules. (1) Keep slow background work off the whatsapp queue — the connector has only a
handful of worker slots, and a background job occupying one is a user-initiated send that
doesn't go out. This has caused two production worker-slot deadlocks. (2) An orchestration
worker never talks to WhatsApp directly; it always goes through the producer.
Storage and encryption
● Sessions: Baileys multi-file auth state → serialised → AES-256-GCM under
CHANNEL_SESSION_KEY → one blob per channel at channels/{id}/auth.bin in Spaces.
Not in Postgres.
● Media: encrypted at rest under a separate MEDIA_OBJECT_KEY, same [iv(12) | tag(16)
| ciphertext] layout, at channels/{id}/media/{message_id}. Two distinct keys means leaking
one does not compromise the other domain — keep it that way.
Tenancy and RBAC
● Every row carries organization_id. Every user-facing public endpoint enforces
membership. The pinned /api/v1/disciplinary/* provider integration is the sole
server-to-server exception: fixed bearer token, fixed org and channel, no inbox access.
Do not add another exception without documenting its fixed tenant boundary.
● 16 seeded permissions and 4 system role templates (owner, admin, agent, viewer) cloned
per org on signup. System roles are read-only in the UI; custom roles are editable. Gate
every REST endpoint with require_permission(). Seed source: app/seeds/permissions.py.
sh scripts/release.shDetail
7894a462-65dd-4830-940c-ce274f6d2a03
(overridable via repo variable DO_APP_ID)
SGP1 (Singapore) for everything
DOCR, Basic plan — two repositories:
replybox-api, replybox-connector. Each build
pushes : and :latest.
Redis logical DBs
/0 BullMQ, /1 Socket.IO adapter, /2 rate limit.
Locally, use /15 for tests.
Frontend
Vercel, production alias replybox-dev.vercel.app
Production API
api.replybox.utopiagroup.com.my — live users,
no staging equivalent
Health endpoints
API /healthz, web /healthz, connector
:8787/health (private LAN only)
Cost
Roughly USD 45–80/month depending on
component sizing. The connector has been
raised to 4 vCPU / 8 GB.
The connector is an internal Service, not a Worker, and not public. It needs ingress from the
API on the app-private LAN for the HMAC-protected live probes, but has no public route. Do not
give it http_port or an ingress rule. The BullMQ Bull Board dashboard is only mounted when
ENV=development, so it is neither mounted nor reachable in production.
Provisioning from scratch is fully documented step-by-step in docs/deployment-do.md (661
lines, including every env var, the asyncpg URL rewrite, custom domain and TLS, and a
first-deploy troubleshooting table). You should not need it unless you are rebuilding the
environment.
apps/baileys-connector/ main
**
apps/web/**
main
deploy-do.yml
deploy-web.yml
Target
DOCR → App Platform
api + PRE_DEPLOY
migrate
DOCR → App Platform
connector
Vercel production
How a deploy is actually triggered
There is no doctl apps create-deployment step anywhere. App Platform watches each
component's :latest tag with deploy_on_push. Moving the tag is the trigger. The workflow only
builds, tags :sha, copies that manifest to :latest by digest, verifies the tag moved, and (for the
connector) waits for the rollout to go ACTIVE.
Corollary: if :latest doesn't move, nothing deploys — no matter what landed on main.
Job graph
changes ──▶ migration-check ──▶ build-connector ──▶ build-api
(detect)
(the only gate)
(if connector)
(if api)
● changes — a hand-rolled git diff --name-only $BEFORE $SHA that prefix-matches
^apps/api/ and ^apps/baileys-connector/ and prints the file list to the log. It is deliberately
not a paths-filter action: run #56 had a filter action report api=false for a commit that
changed apps/api/Dockerfile, so both builds skipped and the run went green while
deploying nothing. A silent no-deploy is the worst failure mode this pipeline can have.
● migration-check — spins up postgres:16 and replays all ~163 revisions from an empty
database. It asserts exactly one Alembic head, pre-creates alembic_version at
production's VARCHAR(255) width (22 revision ids exceed the modern 32-char default),
runs alembic upgrade head, and asserts the DB landed on head. This is a gate, not a
report: both build jobs needs: it, so a failure means no image is pushed, no tag moves,
and no deployment is created at all.
● build-connector — runs only when connector files changed. Records the current
deployment id, builds and pushes :sha, copies to :latest with docker buildx imagetools
create, verifies both tags resolve to the same digest (5 retries, registry listing lags), then
polls up to 3 min for a new deployment and up to 10 min for ACTIVE.
● build-api — same build/tag/verify sequence, no rollout wait. Guarded with always() so it
still runs when the connector job was skipped, with migration-check.result == 'success'
asserted positively so a skipped gate can never read as a passing one.
Two build rules you must not "simplify"
1. Do not consolidate the two tags into a multi-tag tags: | block. Tried 3 Aug 2026:
DOCR applied only the :sha tag, latest stayed on the previous digest, and App Platform
redeployed the old image while reporting success.
2. Do not build twice either. build-push-action@v6 defaults to provenance: true, so each
invocation wraps the image in an OCI index plus an attestation manifest carrying per-build
metadata — two builds of byte-identical layers mint two different index digests, and the
verify step becomes unsatisfiable. imagetools create copies by digest, so both tags
resolve to one digest by construction.
Connector rolls before API, always
Two :latest pushes in parallel create two independent App Platform deployments, and the
second CANCELs the first mid-rollout — which is how one commit ended up killing the
connector twice. It is also the ordering required when you add a new queue or job kind: the
connector must be able to consume a job before the API can produce it.
The migrate pre-deploy job
kind: PRE_DEPLOY, reuses the api image, runs sh scripts/release.sh = alembic upgrade head
&& python -m app.cli seed. It runs on every deployment, including a connector-only one. So a
migration that fails on production data does not just fail the api rollout — it fails the whole app's
deployment, after the images are already published and the rollout has started.
deploy-web.yml — frontend
Simple and ungated: checkout → pnpm 9.5.0 → node 20 → vercel pull
--environment=production → vercel build --prod → vercel deploy --prebuilt --prod → vercel alias
set. The one non-obvious step is Debug pulled env, which greps NEXT_PUBLIC_API_URL out
of the pulled Vercel env and prints whether it was retrieved — that value is inlined into the client
bundle at build time, and if vercel pull misses it the build silently ships a bundle that POSTs to
the web app's own origin and 404s on /api/v1/.... No lint, no typecheck, no tests. pnpm build
failing is the only thing that stops a bad frontend deploy.
Required GitHub configuration
Kind
secret
secret
variable
variable
Name
Used by
DIGITALOCEAN_ACCESS_TO deploy-do.yml — needs registry
KEN
read/write + app scope
VERCEL_ORG_ID,
deploy-web.yml
VERCEL_PROJECT_ID,
VERCEL_TOKEN
DOCR_REGISTRY
deploy-do.yml
DO_APP_ID
deploy-do.yml — optional, falls
back to the literal in env:
Release checklist
1. Land feature branches on development via PR.
2. Before opening the promote PR, check for split Alembic heads: cd apps/api &&
alembic heads — must print exactly one line. If two branches each added a migration, fix
it on development first with alembic merge heads -m "merge and ". Doing it
before the merge means one push, one diff window, and both components build correctly
in one clean run. Doing it after means a red main and manual recovery.
3. Open development → main. "Checks 0" is expected and proves nothing.
4. Merge. Watch the run in Actions and read the printed changed-file list.
5. Confirm the DO deployment reached ACTIVE and /healthz returns 200.
6. Append a line to SHIPPED.md with re-runnable evidence: the Actions run id, the
deployment id, image digests, and the probes you ran. This log is how past releases are
audited — keep the format.
The failure mode to internalise: the one-push diff window
The changes job's diff window is exactly one push wide, and it has no memory of what a
previous failed run was supposed to build. So:
Run
Diff window
api
connector
Outcome
A — merge the
main@old …
true
true
migration-check
, both builds
promote PR
merge
skipped
green, api
B — push the
merge … fix (1 file) true
false
Alembic fix
deploys, connector
❌
✅
does not
Net result: main carries new connector code, production runs the old image, and the run
is green.
Recovery — force both components explicitly:
gh workflow run deploy-do.yml --ref main -f api=true -f connector=true
workflow_dispatch bypasses the git diff entirely and uses the inputs. Expect the connector
restart to blip every WhatsApp session, so do it deliberately and at a quiet hour.
Rollback
Repoint the component's image tag to a known-good : in the App Platform dashboard
and redeploy — every build pushes a :sha alongside :latest, so any previously-deployed commit
is directly addressable. A rollback does not roll back migrations. The PRE_DEPLOY job only
ever moves forward. An old image against a newer schema is usually fine because columns are
additive, but it is not guaranteed — check the migrations in between.
apps/web
Dev
uvicorn
app.main:app
--reload --port
8000
pnpm dev
Build
—
Lint / typecheck Tests
ruff check . / mypy pytest
app
pnpm build
pnpm lint / pnpm
typecheck
pnpm typecheck
none
apps/baileys-conn pnpm dev
pnpm build
vitest
ector
API setup: python -m venv .venv && pip install -e ".[dev]" && alembic upgrade head. Requires
Node 20+, Python 3.11+, pnpm 9+. Ruff line length 100, Python 3.11 target.
Local gotchas that will waste a day if you don't know them
● On Windows, use 127.0.0.1, not localhost, for container ports. Docker's IPv6 relay
makes localhost hang; the SSL traceback you get is a red herring.
● Cookie Domain quirk: when testing the API by hand locally, use localhost as the host
and carry the Cookie header yourself.
● apps/web/node_modules silently loses files on Windows. A plain pnpm install no-ops
on the broken tree; pnpm install --force repairs it.
● Never send or retry a message against a locally-connected channel — it is a real
WhatsApp number and a real send.
● Docker Desktop is broken on the maintainer's arm64 machine (exec-format error). Don't
churn trying to fix it — the plan there is native Homebrew Postgres/Redis/MinIO.
Database GUI
pgweb is wired into the compose stack behind the tools profile so it never starts with normal
infra: docker compose -f infra/docker/compose.yaml --profile tools up -d pgweb, then open the
published port and pick a bookmark. It runs with --bookmarks-only and binds to 127.0.0.1, so no
arbitrary connection string can be typed into a UI that can reach production. The prod bookmark
is gitignored, should authenticate as a read-only replybox_ro role (create it with
infra/sql/pgweb-readonly-role.sql), and needs this machine's public IP in the DO cluster's
Trusted Sources.
Testing — what exists and what doesn't
Nothing gates a deploy except the Alembic replay. pytest, ruff, mypy, pnpm lint, pnpm
typecheck and the connector's vitest suite run nowhere in CI. Run them locally, deliberately,
before you promote.
Two opt-in integration harnesses unlock tests that otherwise skip, so a plain pytest runs
anywhere. Point both at disposable instances — the DB harness TRUNCATEs between
tests:
TEST_DATABASE_URL=postgresql+asyncpg://user:pass@localhost:5432/replyb
ox_test pytest
TEST_REDIS_URL=redis://localhost:6379/15 pytest
Skipping is silent, so a green pytest does not mean these ran — check the skip count.
TEST_REDIS_URL covers queue behaviour that can only be verified against real Redis running
the real Lua; the fake-backed unit tests assert what we send and share the code's own
assumptions, which is how a broken job promotion went unnoticed in production for its entire
life. The test database needs the citext and pg_trgm extensions, and must be recreated after
any column-altering migration.
Operations runbook
Diagnose before you read code
Symptom
"Message stuck on the sending clock"
Tags visible in one place but not another
First move
python -m app.cli diagnose-stuck-sends. It
classifies every queued outbound row by
BullMQ job state × the connector's send-guard
cache, separating "the send failed" from "the
browser's cache is wrong" in one command.
Production measures ~0% of the former. Full
history in docs/rca-stuck-sending-bubble.md.
This report has recurred six times; the answer
is settled.
python -m app.cli reconcile-tags — reports and
repairs historical drift between the three tag
tables.
Deploy stalled / component unhealthy
Connector CPU high
doctl apps list-deployments $DO_APP_ID, doctl
apps get-deployment, and component logs in
the DO console. Do not stack another deploy
on top of a stuck rollout.
CPU tracks media volume in both directions.
Relief: drain the media/avatar backlog via
BullMQ. Peak observed 75–84%.
Other CLI commands
seed, create-superuser, backfill-contact-phones, merge-username-contacts, reconcile-tags,
diagnose-stuck-sends — all under python -m app.cli from apps/api.
Known production behaviours you should recognise, not
re-investigate
● Baileys sockets flap. Production sockets close roughly 14×/hour with 408/428 codes;
one number has closed 3× in 47 seconds. The UI symptom is fixed; the root cause is
open. Flapping alone is not an incident.
● Cloud API cannot address @lid or @g.us, and it fails silently. A LID is a routing id for
a customer who messages from a username, not a phone number, and it cannot be
resolved back to one (proven by a live USync probe: PN→LID works, LID→PN does not).
Stripping the suffix yields digits that look like a phone number, so Graph accepts the send,
returns a wamid, and we record sent — one grey tick, forever. No error, no failed receipt,
nothing in the logs. This is exactly what happened when a Coexistence channel flipped its
default to Cloud API: 28 messages one day, 27 the next, 0 delivered, phone-number
customers on the same channel unaffected. Always route through
services/transport.py; never read conversations.transport directly.
● History sweeps and media re-upload are served by the paired phone. If the phone is
asleep or in a drawer, those requests time out silently. "Recovery didn't work" is often "the
phone wasn't awake".
● Messages arriving hours late are usually WhatsApp's offline queue flushing on
reconnect, mis-ingested as history.
● A chat present in another WhatsApp integration but absent from ReplyBox is a
WhatsApp multi-device fan-out miss caused by two companion integrations on one
number. Not a bug, not fixable in code.
● "Delete for me" on a linked device never reaches us, and a message sent from a
linked phone is indistinguishable from one sent by another device — there are four
columns that prove whether a message came from ReplyBox, but not which device sent it
otherwise.
Meta / Cloud API rules
● The Meta App Review is approved for all four permissions.
● Never test against a live client's WABA. Use the dev app and a fresh WABA.
● Coexistence (one number on the Business app and Cloud API simultaneously) is proven
in production, but rollouts break on two things: Meta's 5-minute embedded-signup
timeout, and the JS-SDK domain allowlist. Check both before blaming the code.
Do not
● Do not scale the connector past one instance. Each WhatsApp session must be held
by exactly one process. A second instance produces multi-device 401s and session loss.
● Do not push, merge, deploy or release without explicit approval for that specific
change. Approval is per change, never standing, even when a task looks finished and
pushing reads as the obvious last step.
● Do not rebuild the connector for a change that didn't touch it — every rebuild re-pairs
every customer channel.
● Do not add a second connector→API mechanism. No Redis pub/sub, no websockets.
Extend the HMAC HTTP path.
● Do not put slow background work on the whatsapp queue. Give it its own queue and
worker.
● Do not trigger React Query refetches on realtime events — the web client folds
Socket.IO events into the caches directly, and refetching turns a message burst into a
request storm.
● Do not read conversations.transport directly — resolve it through services/transport.py
or you will eventually route a send onto a queue nothing consumes.
● Do not compute broadcast delivery from sent_count — it only means "handed to the
connector" and diverges exactly when it matters. Delivery comes from messages.status,
i.e. WhatsApp's own receipts. Reply rate is per recipient; delivery is per message. Never
share a denominator between them.
● Do not add another server-to-server tenancy exception without documenting its fixed
tenant boundary.
● Do not merge the two encryption keys. CHANNEL_SESSION_KEY and
MEDIA_OBJECT_KEY are separate on purpose.
● Do not "simplify" the two-step tag push in CI (see §5) — both shortcuts have already
broken production deploys.
● Do not trust a green PR. No checks run on PRs.
● Do not point a test harness at a real database. TEST_DATABASE_URL TRUNCATEs
between tests.
Open security and config items worth attention
● The production Valkey/Redis cluster has no Trusted Sources configured — it is
protected by authentication alone. Send jobs carry the message body and recipient JID,
so anyone able to write to that queue could send messages with no corresponding
database row. An audit of send:done orphans found none, but the exposure should be
closed.
● Several shipped features are silently disabled in production by unset or misplaced
DO env vars — call-log ingest returns 503, connector safety knobs are off, and at least
one worker-concurrency var is set on the wrong component. Audit the App Platform env
against app/config.py when you take over.
● Baileys socket flapping root cause is still open.
● ~2.5%/day of scheduled messages never fire — the row stays queued, the BullMQ job
is gone, no error is raised.
● 416 branches, most dead. Needs a prune.
● Backup story beyond DO's default Postgres snapshots is undesigned; Spaces has no
replication.
● Multi-region Socket.IO fan-out isn't designed in — AsyncRedisManager covers a single
Redis horizontally only.
Document
CLAUDE.md
docs/plan.md
docs/deployment-workflow.md
docs/deployment-do.md
docs/public-api.md
docs/rca-stuck-sending-bubble.md,
docs/bug-rca-pending-and-media.md
docs/connector-reconnection-design.md,
docs/connector-sharding.md
docs/{tagging,notes,crm-board,flow-builder-7c,i
ncremental-history-sync,channel-migration}.md
docs/utopia-sso-{design,deploy,profile-enrichme
nt}.md
docs/disciplinary-bridge.md,
docs/toppie-webhook-{spec,tutorial}.md
SHIPPED.md