live_jobSlow but moving. Prod count: 0.Operations RCA
¶ Stuck Sending Messages
RCA for messages that appear stuck even after WhatsApp delivered them. The measured production fault was not a failed send path; it was a browser cache reconciliation failure after the server-side send had already succeeded.
Symptom: sending clock stays visible
First command:
diagnose-stuck-sends
Investigated 2026-08-18
Verdict
¶ The clock was a wrong cache, not a failed send
An outbound bubble could sit on "Still sending — check your connection" while WhatsApp had already delivered the message. Often a second, correct copy appeared beside it.
This was the fifth recurrence. The earlier fixes each closed one missed handoff, but the handoff still had one carrier and no backstop.
Measurement first
¶ What diagnose-stuck-sends actually checks
The read-only classifier asks two questions the database cannot answer alone: whether BullMQ still has send work, and whether the connector send-guard cache has proof that Baileys returned a WhatsApp id.
Run against the affected environment
python -m app.cli diagnose-stuck-sends
BullMQ job state
producer.send_job_state, the read-only sibling of claim_send_job.
Send-guard proofsend:done:{message_id}, written as soon as Baileys returns an id and before the status webhook is attempted.
sent_webhook_lostShipped, status never posted. Prod count: 0.job_failedRetries and terminal post lost. Prod count: 0.job_evaporatedVanished before shipping. Prod count: 0.evidence_expiredToo old to diagnose. Prod count: 9.evidence_expired prevents aged residue from being mistaken for a live fault: after the send-guard 24h TTL, both signals are gone by design.
Root cause
¶ Three writers raced with different identity
Before the fix, the browser, send response, and realtime creation event did not all carry a durable shared key.
¶ Optimistic insert
onMutate in hooks/useInbox.ts minted client_message_id.
¶ Send POST response
applySendResponse could reconcile the optimistic row when the request survived navigation and unmounts.
¶ Realtime creation emit
message.created carried the id only on creation. Later status emits came from services/message_status.py.
null.
¶ What made the bubble strand
client_message_id had no database column. Each send endpoint hand-stitched it onto its own response and creation emit, but status emits had no access to the original request. If the creation emit was lost to a socket gap and the POST response was aborted by navigation or unmount, every later ack matched nothing, appended a second row, and left the first row spinning forever.
Recovery was entirely event-driven: socket reconnect and visibilitychange. A healthy foreground tab had no recovery point; MessageBubble showed the stuck tiers and never re-asked the server.
Fix
¶ Make the handoff durable and give the client a backstop
Fix 1
¶ Persist client_message_id
Migration 0119_messages_client_message_id added a nullable UUID column on messages. MessagePublic serializes it off the row, so the POST response, creation emit, every status emit, and GET thread response carry it.
ScopeThreaded through outbound text, media, and location inserts; about twenty hand-stitch sites were removed.
Null remains correctInbound rows, scheduled sends, broadcasts, flows, public API sends, and old rows cannot be backfilled and do not need to be.
Fix 2
¶ A stuck bubble re-asks the server
reconcileThrottle.ts moved the trailing-edge throttle into a shared per-key module. Reaching the 30s stuck tier, or mounting onto an already-stuck bubble, now invalidates the thread query.
Load behavior
MessageBubble uses stuck:{conversationId}, so a thread full of stuck bubbles costs one refetch.Important propertyThe throttle is trailing-edge and never dropped; a leading-only throttle discards the recovery point.
Fix 3
¶ Realtime dedupe merges instead of keep-first
reconcileRealtimeMessage now merges duplicates on both row id and real WhatsApp id, preserving the survivor's position in the thread.
Why it matteredThe discarded duplicate was often the more advanced row, which put the bubble back on the clock with nothing left to advance it.
Deploy ordering
¶ Migration before API code
Migration 0119 must land before the API code. MessagePublic reads messages.client_message_id as a real column; deploying code before the column breaks message queries, not just sends.
Apply
0119_messages_client_message_id.
Deploy API/web code that serializes and reconciles the id.
No connector change is required; the connector can deploy in any order.
Still not fixed
¶ Known limits after the fix
These caveats stay explicit because earlier rounds said "fixed" too soon.
Fix 1 only helps if an emit reaches the tab. If the socket is down for the whole ack burst, recovery falls to the single refetch from fix 2.
The media send path still re-derives client_message_id by filename and size. A cache miss creates a NULL row; two identical files sent quickly can collide.
No real send was verified during the RCA. The local connected channel is a live number, so evidence was test-driven.
There is no client-side detector. diagnose-stuck-sends covers the server side; sendTrace is opt-in and manual.
Next occurrence
¶ Diagnose in this order
¶ Classify server versus client
Run python -m app.cli diagnose-stuck-sends against the affected environment. Production is where the distribution lives; local zero rows prove little.
¶ If server is clean, inspect browser cache
Set NEXT_PUBLIC_SEND_TRACE=1 and read window.__sendTrace() on the stuck bubble in apps/web/lib/inbox/sendTrace.ts.
¶ Read the trace labels
realtime:appended means an emit created a second row. success:no-match means the POST response taught the cache nothing. No sweep:reconcile means nothing tried to heal it.
¶ Use the right latency signal
messages.sent_at is stamped at row creation, not send confirmation. Use created_at and live job state instead.
Not this bug: send latency where a bubble is slow but moving. The classifier's live_job bucket separates slow-but-moving sends from this cache reconciliation failure.