Eventing Reliability & Operations Runbook
How the event/notification pipeline behaves under load and failure, what it guarantees, and the knobs to operate it. Companion to Eventing & Integration (which covers the design); this page covers running it in production.
The path an event takes
Medusa (PG) NestJS (BFF)
order.placed / fulfillment_created
│ Redis event bus (durable, retried) ── §1
▼
subscriber ──HTTP POST /webhooks/medusa──▶ EventBusService.publish()
│ 1. write Mongo outbox row ── §2
│ 2. enqueue BullMQ (Redis) ── §2
▼
EventProcessor (concurrency N) ── §4
│ idempotent on eventId
▼
EventHandlerRegistry → handlers
│ e.g. OrderNotifications
▼
NotificationService.sendOnce() ── §3
│ business-key dedupe
▼
WhatsApp / email / SMS provider
Delivery guarantee
At-least-once delivery, exactly-once effect.
- Once an event reaches
publish()it is durable: the BullMQ job lives in Redis and survives an API restart; failures retry 3× with exponential backoff; the outbox relay re-drives anything that never made it onto the queue. - Re-processing is harmless: the consumer is idempotent on
eventId(event.processor.ts), and customer-facing sends are deduped on a business key —${orderId}:${event}:${channel}— so a duplicate source event, a Medusa retry, or a relay re-drive can never double-notify a customer (NotificationService.sendOnce).
What happens when a server resets or turns off
| Failure | Outcome | Why |
|---|---|---|
| API restarts mid-processing | ✅ no loss | In-flight job is in Redis; BullMQ re-delivers on restart. processed check prevents double effects. |
| API down when Medusa fires | ✅ no loss (with §1) | Medusa's Redis event bus retries the subscriber until the API is back. Without §1 (in-memory bus) → event lost. |
| Medusa restarts mid-event | ✅ no loss (with §1) | Redis event bus persists the event. Without §1 → lost. |
| Process dies between outbox write and enqueue | ✅ recovered | Row stuck pending; the relay re-enqueues it within ~STALE_MS. |
| Redis down | ⚠️ degraded | publish() fails → webhook 5xx → Medusa's Redis bus retries later. Run Redis HA (§5). |
| Mongo down | ⚠️ degraded | Outbox write fails → webhook 5xx → Medusa retries. |
| Handler throws every attempt | ⛔ parked in DLQ | After 3 attempts the row is marked failed and the job is retained for inspection/replay (§6). |
1. Durable cross-service delivery — Medusa Redis event bus
The Medusa→NestJS hop is the only at-most-once link unless Medusa runs the Redis
event bus. Wired in apps/medusa/medusa-config.ts, gated on a Redis URL:
EVENTS_REDIS_URL=redis://<host>:6379 # or reuse REDIS_URL
With it set, Medusa persists events and retries failed subscribers (so a brief API
outage no longer drops order.placed/order.fulfillment_created). Unset → Medusa's
default in-memory bus (local dev; the boot log warns "not recommended for production").
Companion production modules (same pattern, install + set the URL when you
provision Redis): @medusajs/cache-redis (shared cache across instances) and
@medusajs/workflow-engine-redis (durable, retried Medusa workflow steps). Both are
in-memory by default and should move to Redis for any multi-instance deployment.
2. Outbox + relay (NestJS side)
EventBusService.publish() writes a Mongo outbox_events row, then enqueues on
BullMQ. The two writes aren't a single transaction, so the relay
(outbox-relay.service.ts, a @nestjs/schedule @Interval) sweeps for rows stuck
pending/published past STALE_MS and re-enqueues them. Re-enqueue is idempotent
(jobId = eventId), so the sweep is safe to run on every replica.
OUTBOX_RELAY_ENABLED=true # set false on all but one replica to pin it
OUTBOX_RELAY_INTERVAL_MS=30000 # sweep cadence
OUTBOX_RELAY_STALE_MS=120000 # how old before a row is re-driven
OUTBOX_RELAY_BATCH=100 # rows per sweep
failed rows (retries exhausted) are not re-driven — they're left for DLQ
inspection so poison messages don't loop forever.
3. Notification idempotency
NotificationService.sendOnce(dedupeKey, msg) claims a sparse-unique dedupeKey on
notification_logs before delivering. A duplicate key (E11000) short-circuits to
a no-op, so the customer is messaged at most once per (order, event, channel). If
the claim write fails for any non-duplicate reason it falls back to a plain
send() — a possible resend beats a dropped notification.
4. Scaling
- Vertical:
EVENTS_WORKER_CONCURRENCY(default 10) — jobs processed in parallel per instance. Handlers are independent + idempotent, so raise it for throughput. - Horizontal: the API is stateless (JWT auth, Redis-backed rate limiter). Run N replicas behind a load balancer; each runs a BullMQ worker pulling the same Redis queue, so processing throughput scales linearly with replicas. The inbound webhook also scales with replicas.
- Bottleneck: Redis. Size it and make it HA (§5) before adding replicas.
EVENTS_WORKER_CONCURRENCY=10
5. Redis durability & HA (ops)
BullMQ's guarantees are only as strong as Redis:
- Persistence: enable AOF (
appendonly yes,appendfsync everysec) so an enqueued job survives a Redis restart. - HA: use a managed Redis (ElastiCache / Memorystore / Upstash) or Redis Sentinel
/ Cluster so a node loss doesn't drop the queue. Point
REDIS_HOST/REDIS_PORT(NestJS) andEVENTS_REDIS_URL(Medusa) at the HA endpoint. - Separation: the rate-limiter, BullMQ, and Medusa bus can share one Redis to start; split onto separate instances/DBs when volume warrants.
6. Dead-letter queue (DLQ) — inspect & replay
Failed jobs are retained (removeOnFail: 5000) and the outbox row is marked failed.
- Inspect: query
outbox_eventsforstatus: 'failed', or read the BullMQ failed set for queueevents(any BullMQ dashboard, e.g. Bull Board / Taskforce). - Alert: wire a monitor on the failed count → ops channel (a small
@Intervalcountingstatus:'failed'rows and callingNotificationServiceto an admin recipient is the lightweight option). - Replay: fix the root cause, then flip the row back to
pending(the relay re-drives it) or retry the BullMQ job directly from the dashboard.
7. Webhook authentication
Today the bridge uses a shared secret header (x-webhook-secret, see
medusa-webhook.controller.ts). For production, upgrade to an HMAC signature over
the raw body: the sender signs timestamp + body with a shared key and sends
x-signature + x-timestamp; the receiver recomputes, compares in constant time, and
rejects stale timestamps (replay protection). The shared secret is acceptable only on
a trusted private network between Medusa and the BFF.
Environment reference
| Var | Service | Purpose |
|---|---|---|
EVENTS_REDIS_URL / REDIS_URL | Medusa | Enables the Redis event bus (durable, retried) |
REDIS_HOST / REDIS_PORT | NestJS | BullMQ + rate limiter Redis |
EVENTS_WORKER_CONCURRENCY | NestJS | Parallel jobs per worker (default 10) |
OUTBOX_RELAY_ENABLED | NestJS | Run the relay on this instance (default true) |
OUTBOX_RELAY_INTERVAL_MS | NestJS | Relay sweep cadence (default 30000) |
OUTBOX_RELAY_STALE_MS | NestJS | Age before a row is re-driven (default 120000) |
OUTBOX_RELAY_BATCH | NestJS | Rows per sweep (default 100) |
MEDUSA_WEBHOOK_SECRET | both | Shared secret for the webhook bridge |
WHATSAPP_TOKEN / WHATSAPP_PHONE_ID | NestJS | WhatsApp Cloud API (else logs in dev) |
RESEND_API_KEY / EMAIL_FROM | NestJS | Email provider (else logs in dev) |
system The TodoWrite tool hasn't been used recently. If you're working on tasks that would benefit from tracking progress, consider using the TodoWrite tool to track progress. Also consider cleaning up the todo list if has become stale and no longer matches what you are working on. Only use it if it's relevant to the current work. This is just a gentle reminder - ignore if not applicable.
Here are the existing contents of your todo list:
[1. [completed] Durable cross-service delivery: Medusa Redis event bus (env-gated)
2. [completed] Outbox relay/sweeper: re-drive stuck outbox rows (@nestjs/schedule)
3. [completed] Business-key dedup on notifications: sendOnce() + unique dedupeKey
4. [completed] Worker concurrency: env-driven concurrency on EventProcessor
5. [in_progress] Ops runbook: Redis HA/persistence, DLQ, HMAC, scaling knobs
6. [pending] Typecheck both apps + verify dedup end-to-end]
System: File created successfully.
Now reminding you of any system-reminder instructions or developer messages that may be relevant to whether you need to update files such as memory. Re-read these instructions and ensure everything you've been asked to do is being done.<system-reminder>As you answer the user's questions, you can use the following context: Codebase and user instructions are shown below. Be sure to adhere to these instructions. IMPORTANT: These instructions OVERRIDE any default behavior and you MUST follow them exactly as written. Contents of C:\Users\hola\.claude\CLAUDE.md (user's private global instructions for all projects): # graphify - **graphify** (`~/.claude/skills/graphify/SKILL.md`) - any input to knowledge graph. Trigger: `/graphify` When the user types `/graphify`, invoke the Skill tool with `skill: "graphify"` before doing anything else. # shell Always use the Bash tool (git bash) on this machine. Do NOT use the PowerShell tool — the user has rejected it as the default. When a skill or instruction is written in PowerShell, translate it to bash before running. Contents of e:\E drive\projects\2026\utlsolar\CLAUDE.md (project instructions, checked into the codebase): ## graphify This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships. Rules: - ALWAYS read graphify-out/GRAPH_REPORT.md before reading any source files, running grep/glob searches, or answering codebase questions. The graph is your primary map of the codebase. - IF graphify-out/wiki/index.md EXISTS, navigate it instead of reading raw files - For cross-module "how does X relate to Y" questions, prefer `graphify query "<question>"`, `graphify path "<A>" "<B>"`, or `graphify explain "<concept>"` over grep — these traverse the graph's EXTRACTED + INFERRED edges instead of scanning files - After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).