¶ CPU high
Expected under heavy media volume. Peak observed production CPU has been high under media load.
Operations
¶ Connector CPU / Queue Backlog
CPU follows media pressure. Connector CPU tracks media volume in both directions, and production peaks have been high under media load.
CPU tracks media volume
Inspect media/avatar queues
Keep sends isolated
Immediate relief
¶ Drain background work before blaming sends.
When connector CPU is high, first inspect BullMQ backlog for whatsapp_media and whatsapp_avatars. Drain slow media/avatar work before assuming user sends are broken.
Signal map
¶ Separate load from product failure
Queue depth, connector memory, and foreground send health tell different stories. Use them together before escalating to code changes.
Media envelope
¶ v2 removes the buffering pressure
Past connector CPU / backlog incidents were caused by buffering large media blobs due to the legacy v1 envelope. The streamable v2 envelope uses a tag-last layout, eliminating the need to buffer and reducing memory spikes and queue stalls.
Storm controls
¶ Bound recovery work at the source
Media age gate
¶ Recovery windows stay intentional
Background media sweeps such as _recover_failed_media and _prefetch_recent_pending_media only attempt recovery for recently hosted files. Sweeps use a 30-day recoverable window.
Interactive pathServer-side
MEDIA_EXPIRED_AFTER_DAYS = 90 prevents automatic interactive downloads for items older than 90 days, while callers may still use force=true.Quiet hours
¶ Reconnect sweeps can skip busy hours
MEDIA_SWEEP_QUIET_HOURS_UTC lists comma-separated UTC hours when reconnect-triggered background media sweeps must be skipped.
Default shapeExample:
0,1,2,3. Malformed or empty values disable the quiet window and always sweep.Avatar deferral
¶ No-answer probes back off
The connector can report avatar probe outcome deferred when there is no live socket or the phone does not answer. This stamps the TTL gate without clearing the stored avatar.
Backoff
AVATAR_DEFERRED_TTL_SECONDS defaults to 6 hours, using back-dated avatar_fetched_at without adding columns or prolonging a longer TTL.Background shedding
¶ Roll it out in dry-run first
Deploy the connector with SHED_MODE unset (off). Then set SHED_MODE=dry_run for at least one weekday peak to observe the logged shed (dry-run): status lines and the lag spread. Tune the thresholds from that data, especially the lag spread under 2s, before setting SHED_MODE=on. Deploy in the evening: a connector-only restart is required.
Defaults that matter
¶ Safe out of the box
SHED_MODE=off, SHED_LAG_MS=1000, SHED_TRIGGER_WINDOWS=3, SHED_RESUME_LAG_MS=500, SHED_RESUME_WINDOWS=6, SHED_TRICKLE_CONCURRENCY=1, SHED_MAX_MS=20m, and PRESENCE_STALE_MS=15000 (0 disables).
Never
¶ Do not use Queue.pause()
The shedding controller uses per-worker admission. Queue.pause() persists in Redis and can survive crashes, masking live ingest and causing follow-up operational surprises.
Fallback
¶ If the window stream stops
If the event-loop window stream stops arriving, the controller falls back to the existing per-job throttle, so it cannot stick in a shed state.
History-import storms
¶ Bound concurrent first-time imports
Many simultaneous first-time WhatsApp imports decode and hold large amounts of history in RAM while waiting for upload slots, which creates event-loop and heap pressure and can freeze the process. The pairing import queue limits concurrent imports: the API holds first pairings before their QR code when imports are saturated, admitting them into slots according to the import_capacity reported in heartbeats.
Sizing
¶ Keep imports under the post limit
Set MAX_CONCURRENT_IMPORTS at or below MAX_CONCURRENT_SYNC_POSTS. The connector logs a boot warning if MAX_CONCURRENT_IMPORTS is higher.
Heap admission
IMPORT_HEAP_PAUSE_PCT and IMPORT_HEAP_RESUME_PCT pause admitting new imports when the V8 heap limit is near capacity; both compare against --max-old-space-size.Deploy order
¶ API before the connector
The API changes (pairing queue plus heartbeat evict) must be deployed before the connector roll, so the connector's heartbeat and evict behaviour are understood by the API.
Monitoring
¶ What to watch
Per-import heap logs (start, at most once a minute, end), messaging-history.status events, and heartbeat import_capacity fields when tuning quiet periods and heap alarms. Collect connector heartbeats, look for evict requests in API logs, and verify import_seen_at updates.
Deleted channelsThey get immediate
410s from the ingest endpoint so imports stop.Avatar reprobe
¶ A bounded catch-up lane
A previous bug classified transport failures as “no photo”, wiping stored avatars and creating a large backlog of blacklisted rows. The API now exposes a rate-capped catch-up lane that probes contacts with no stored picture (never probed, or an expired “probed-no-photo”), bounded so it cannot storm the connector or WhatsApp.
Rate limits
¶ AVATAR_REPROBE_PER_MINUTE
Default 20 per minute per channel; 0 disables the lane. A per-process in-memory sliding-window limiter guards enqueueing, so N API workers means N × the cap, with no extra network hops or central coordination.
Burst gateHistory-replay probes stand down while a channel has a sync burst (more than 20 history messages per minute), to avoid replay storms.
Admin tool
¶ resync-avatars
Re-probes contacts recorded as having no avatar and enqueues real WhatsApp probes. Report-only unless --apply, and bounded by default with --limit 500 and --active-days 30. Use scoped invocations (organization or channel) and small limits when testing.
Operational notes
¶ Reading the logs
Watch for fetch_avatar: probe failed transiently; deferring…. If it dominates, reduce AVATAR_REPROBE_PER_MINUTE; if WhatsApp starts rate-limiting the line, set it to 0 to disable the lane quickly. If no picture (definitive ...) dominates, most contacts genuinely lack photos.
Connector sideOnly
404/401 clear a photo; 429, 5xx, socket errors, and timeouts defer, so transient failures no longer delete stored avatars.Operating rules
¶ Keep the connector breathable
Drain slow media/avatar work first.High CPU plus background backlog is a queue-pressure problem until proven otherwise.
Keep client and server media expiry aligned.
MEDIA_EXPIRED_AFTER_DAYS must match the web Retry visibility decision so the UI does not offer a silent no-op.
Preserve user-initiated recovery.The 30-90 day band still yields recoveries worth a single user-triggered round trip, so UI Retry remains available.
Avoid bulk retries for old media.Scrolling-triggered attempts for 70-144 day rows caused huge background/job storms.
Do not place slow background work on whatsapp.The main queue is for user-initiated WhatsApp mutations.