¶ Classify the symptom
Decide whether the report points at API, web, connector, queue, database, Redis, Spaces, Cloud API, or WhatsApp-device behavior.
Operations
¶ Production Troubleshooting
Triage the symptom, then pick the system. First identify the affected surface, confirm deploy and queue state, classify the channel, and write down the evidence before making changes.
Scope the blast radius
Runbook commands first
Avoid stacked deploys
Golden rule
¶ Collect signals before changing state
Known production reports usually separate quickly once you know whether the failure is API, web, connector, queue, database, Redis, Spaces, Cloud API, or WhatsApp-device behavior.
Which workspace, channel, conversation, and message?
When did it start, and did a deploy land nearby?
One user, one tenant, one channel, or global?
First ten minutes
¶ Stabilize the investigation
Use this flow before deep code reading. It preserves the original troubleshooting order while making each handoff explicit.
¶ Use the runbook first
Prefer known commands and RCA pages before opening code. The operations runbook has first moves for stuck sends, tags, deploys, connector CPU, Cloud API routing, and history or media timeouts.
¶ Check deployment state
Read DigitalOcean deployment status and component logs. If a rollout is stuck, do not stack another deploy on top of it.
¶ Check queue state
Inspect stuck sends, scheduled messages, broadcasts, media, and avatar work before assuming the application path is broken.
¶ Classify the channel
Confirm whether the affected channel is Baileys, Cloud API, or coexistence. Routing behavior and failure modes differ sharply between them.
¶ Leave an evidence trail
Write probes, timestamps, IDs, logs, and conclusions in the relevant issue, runbook, or SHIPPED.md entry.
Route by system
¶ Pick the most likely owner
Use the observed symptom to choose the next inspection point. A fast, accurate owner beats a wide search.
¶ API
Requests fail, permissions look wrong, message rows do not match UI state, or server logs show validation and status errors.
¶ Web
One browser shows stale state, optimistic bubbles disagree with the server, or realtime recovery depends on refetch and cache state.
¶ Connector
Baileys sockets flap, sends depend on paired-device reachability, media work piles up, or WhatsApp ids appear before API status lands.
¶ Queue
Jobs are waiting, delayed, failed, or missing for sends, scheduled work, broadcasts, media download, re-upload, or avatar probes.
¶ Database
Rows disagree with expected tenant, contact, channel, message, migration, or identity evidence state.
¶ Redis
Socket fan-out, queue metadata, locks, rate limits, or send-guard evidence are absent, stale, or expiring earlier than expected.
¶ Spaces
Media fails to load, encrypted object metadata is missing, copies lose envelope metadata, or retry windows disagree with UI behavior.
Cloud API cannot address @lid or @g.us, devices sleep, companion fan-out misses, or offline queue flushes arrive late.
Deployment and queue
¶ Two checks catch most false leads
Before changing code or retrying a release, compare component health with background work. Split-version serving, a stalled rollout, or a queue backlog can mimic product defects.
Deploy stateRead DigitalOcean rollout status and component logs. If deployment is stuck, pause and diagnose the active rollout.
Queue stateCheck stuck sends, scheduled messages, broadcasts, media, and avatars before escalating to code changes.
Channel typeBaileys, Cloud API, and coexistence have different addressing, delivery, and device-dependency rules.
Known behaviorCompare against documented production behaviors so ordinary WhatsApp conditions are not reopened as new incidents.
Evidence packet
¶ What to record
SymptomExact user-visible behavior and whether it is reproducible.
ScopeTenant, user, channel, conversation, message, job, and deployment identifiers.
SignalsCommands run, logs checked, queue state, channel type, and relevant timestamps.
ConclusionKnown behavior, active incident, rollout problem, data repair, or follow-up RCA.
After a connector deploy
¶ Post-deploy checks and recovery for restart races
¶ Check for demoted channels
Run a DB query for disconnected channels with status_detail of no live socket (connector heartbeat) and disconnected_at at or after the deploy time.
¶ Read the matching log lines
If the query returns rows, inspect connector and API logs for boot reconciliation complete, channel lock acquired after waiting for another instance, and would have demoted / held back by the post-boot grace window.
¶ Reconnect what is still down
Re-request connect through the request_connect path, or press Connect in the UI. Run recovery code from apps/api with DATABASE_URL and REDIS_URL from the prod env, and avoid running with ENV=production.
¶ Deployment order
Roll the API first, since it understands started_at, then deploy the connector. Either order is wire-safe, but verify the logs and the DB check after the connector deploy.