Messaging
¶ Message stuck on the sending clock
First move
Stuck sending messages RCA
python -m app.cli diagnose-stuck-sendsOperations
¶ Operations Runbook
Diagnose before reading code. Find the symptom, take the first move, then follow the deep dive if it isn't resolved.
Diagnose before reading code
6 common symptoms
CLI under
python -m app.cli
Symptom → first move
¶ What are you seeing?
Tags
¶ Tags visible in one place but not another
First move
Tag reconciliation
python -m app.cli reconcile-tagsDeploys
¶ Deploy stalled or component unhealthy
First move
doctl apps list-deployments
doctl apps get-deploymentThen read the component logs.
Do not stack another deploy on top.
Deployment failure modes
Connector
¶ Connector CPU high
First move
Drain media/avatar backlog in BullMQ.
CPU tracks media volume.
Connector CPU / queue backlogCloud API
¶ Cloud API send one-grey-tick forever
First move
Check whether the recipient is
@lid or @g.us.
Always route through
Cloud API routing issues
services/transport.py.Fleet-wide
¶ Every channel shows “Connector offline”
First move
Follow the dedicated runbook at
docs/runbook-connector-freeze.md.
Confirm the connector (API health plus the DB heartbeat), snapshot evidence, restart only the connector, verify the heartbeat, then notify users and open an issue. Collect connector runtime logs, a DigitalOcean Insights screenshot, the connector pod or hostname with its last-log time, and the DB heartbeat query.
Do not restart other services, rotate credentials, or clear session storage before taking the snapshot.
History & media
¶ History or media recovery times out
First move
Make sure the paired phone is awake and reachable.
Pending & media RCA
Ops watchdog
¶ A read-only Telegram watchdog
It alerts on WhatsApp sending delays, channels stuck on connecting, connector heartbeat silence, queued outbound messages, the public /healthz, and DigitalOcean restarts.
Run it from the repository root
scripts\watchdog.ps1
python -m app.cli watchdog
python -m app.cli watchdog --once
python -m app.cli watchdog --dry-run¶ Required environment
DATABASE_URL and REDIS_URL pointed at production, plus TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID. For DigitalOcean restart alerts, supply DIGITALOCEAN_TOKEN or be logged in with doctl.
¶ Safety
scripts\watchdog.ps1 copies only the required production variables into the process and removes them on exit, and adds this computer's current IP to the managed DB firewall for the session so the run does not time out immediately.
¶ Alerting behaviour
Edge-triggered: one start message, one recovery, and periodic reminders while an issue is ongoing. Alerts are deduplicated and confirmed to avoid flapping.
¶ Where to look
Operational code lives in apps/api/app/services/watchdog.py, and the CLI entry is python -m app.cli watchdog. --once does a single dry check; --dry-run keeps running but prints instead of sending Telegram messages.
seed
create-superuser
backfill-contact-phones
merge-username-contacts
reconcile-tags
diagnose-stuck-sends