Skip to main content

Eventing Reliability & Operations Runbook

How the event/notification pipeline behaves under load and failure, what it guarantees, and the knobs to operate it. Companion to Eventing & Integration (which covers the design); this page covers running it in production.

The path an event takes​

Medusa (PG) NestJS (BFF)
order.placed / fulfillment_created
│ Redis event bus (durable, retried) ── §1
▼
subscriber ──HTTP POST /webhooks/medusa──▶ EventBusService.publish()
│ 1. write Mongo outbox row ── §2
│ 2. enqueue BullMQ (Redis) ── §2
▼
EventProcessor (concurrency N) ── §4
│ idempotent on eventId
▼
EventHandlerRegistry → handlers
│ e.g. OrderNotifications
▼
NotificationService.sendOnce() ── §3
│ business-key dedupe
▼
WhatsApp / email / SMS provider

Delivery guarantee​

At-least-once delivery, exactly-once effect.

  • Once an event reaches publish() it is durable: the BullMQ job lives in Redis and survives an API restart; failures retry 3× with exponential backoff; the outbox relay re-drives anything that never made it onto the queue.
  • Re-processing is harmless: the consumer is idempotent on eventId (event.processor.ts), and customer-facing sends are deduped on a business key — ${orderId}:${event}:${channel} — so a duplicate source event, a Medusa retry, or a relay re-drive can never double-notify a customer (NotificationService.sendOnce).

What happens when a server resets or turns off​

FailureOutcomeWhy
API restarts mid-processing✅ no lossIn-flight job is in Redis; BullMQ re-delivers on restart. processed check prevents double effects.
API down when Medusa fires✅ no loss (with §1)Medusa's Redis event bus retries the subscriber until the API is back. Without §1 (in-memory bus) → event lost.
Medusa restarts mid-event✅ no loss (with §1)Redis event bus persists the event. Without §1 → lost.
Process dies between outbox write and enqueue✅ recoveredRow stuck pending; the relay re-enqueues it within ~STALE_MS.
Redis down⚠️ degradedpublish() fails → webhook 5xx → Medusa's Redis bus retries later. Run Redis HA (§5).
Mongo down⚠️ degradedOutbox write fails → webhook 5xx → Medusa retries.
Handler throws every attempt⛔ parked in DLQAfter 3 attempts the row is marked failed and the job is retained for inspection/replay (§6).

1. Durable cross-service delivery — Medusa Redis event bus​

The Medusa→NestJS hop is the only at-most-once link unless Medusa runs the Redis event bus. Wired in apps/medusa/medusa-config.ts, gated on a Redis URL:

EVENTS_REDIS_URL=redis://<host>:6379 # or reuse REDIS_URL

With it set, Medusa persists events and retries failed subscribers (so a brief API outage no longer drops order.placed/order.fulfillment_created). Unset → Medusa's default in-memory bus (local dev; the boot log warns "not recommended for production").

Companion production modules (same pattern, install + set the URL when you provision Redis): @medusajs/cache-redis (shared cache across instances) and @medusajs/workflow-engine-redis (durable, retried Medusa workflow steps). Both are in-memory by default and should move to Redis for any multi-instance deployment.

2. Outbox + relay (NestJS side)​

EventBusService.publish() writes a Mongo outbox_events row, then enqueues on BullMQ. The two writes aren't a single transaction, so the relay (outbox-relay.service.ts, a @nestjs/schedule @Interval) sweeps for rows stuck pending/published past STALE_MS and re-enqueues them. Re-enqueue is idempotent (jobId = eventId), so the sweep is safe to run on every replica.

OUTBOX_RELAY_ENABLED=true # set false on all but one replica to pin it
OUTBOX_RELAY_INTERVAL_MS=30000 # sweep cadence
OUTBOX_RELAY_STALE_MS=120000 # how old before a row is re-driven
OUTBOX_RELAY_BATCH=100 # rows per sweep

failed rows (retries exhausted) are not re-driven — they're left for DLQ inspection so poison messages don't loop forever.

3. Notification idempotency​

NotificationService.sendOnce(dedupeKey, msg) claims a sparse-unique dedupeKey on notification_logs before delivering. A duplicate key (E11000) short-circuits to a no-op, so the customer is messaged at most once per (order, event, channel). If the claim write fails for any non-duplicate reason it falls back to a plain send() — a possible resend beats a dropped notification.

4. Scaling​

  • Vertical: EVENTS_WORKER_CONCURRENCY (default 10) — jobs processed in parallel per instance. Handlers are independent + idempotent, so raise it for throughput.
  • Horizontal: the API is stateless (JWT auth, Redis-backed rate limiter). Run N replicas behind a load balancer; each runs a BullMQ worker pulling the same Redis queue, so processing throughput scales linearly with replicas. The inbound webhook also scales with replicas.
  • Bottleneck: Redis. Size it and make it HA (§5) before adding replicas.
EVENTS_WORKER_CONCURRENCY=10

5. Redis durability & HA (ops)​

BullMQ's guarantees are only as strong as Redis:

  • Persistence: enable AOF (appendonly yes, appendfsync everysec) so an enqueued job survives a Redis restart.
  • HA: use a managed Redis (ElastiCache / Memorystore / Upstash) or Redis Sentinel / Cluster so a node loss doesn't drop the queue. Point REDIS_HOST/REDIS_PORT (NestJS) and EVENTS_REDIS_URL (Medusa) at the HA endpoint.
  • Separation: the rate-limiter, BullMQ, and Medusa bus can share one Redis to start; split onto separate instances/DBs when volume warrants.

6. Dead-letter queue (DLQ) — inspect & replay​

Failed jobs are retained (removeOnFail: 5000) and the outbox row is marked failed.

  • Inspect: query outbox_events for status: 'failed', or read the BullMQ failed set for queue events (any BullMQ dashboard, e.g. Bull Board / Taskforce).
  • Alert: wire a monitor on the failed count → ops channel (a small @Interval counting status:'failed' rows and calling NotificationService to an admin recipient is the lightweight option).
  • Replay: fix the root cause, then flip the row back to pending (the relay re-drives it) or retry the BullMQ job directly from the dashboard.

7. Webhook authentication​

Today the bridge uses a shared secret header (x-webhook-secret, see medusa-webhook.controller.ts). For production, upgrade to an HMAC signature over the raw body: the sender signs timestamp + body with a shared key and sends x-signature + x-timestamp; the receiver recomputes, compares in constant time, and rejects stale timestamps (replay protection). The shared secret is acceptable only on a trusted private network between Medusa and the BFF.

Environment reference​

VarServicePurpose
EVENTS_REDIS_URL / REDIS_URLMedusaEnables the Redis event bus (durable, retried)
REDIS_HOST / REDIS_PORTNestJSBullMQ + rate limiter Redis
EVENTS_WORKER_CONCURRENCYNestJSParallel jobs per worker (default 10)
OUTBOX_RELAY_ENABLEDNestJSRun the relay on this instance (default true)
OUTBOX_RELAY_INTERVAL_MSNestJSRelay sweep cadence (default 30000)
OUTBOX_RELAY_STALE_MSNestJSAge before a row is re-driven (default 120000)
OUTBOX_RELAY_BATCHNestJSRows per sweep (default 100)
MEDUSA_WEBHOOK_SECRETbothShared secret for the webhook bridge
WHATSAPP_TOKEN / WHATSAPP_PHONE_IDNestJSWhatsApp Cloud API (else logs in dev)
RESEND_API_KEY / EMAIL_FROMNestJSEmail provider (else logs in dev)

system The TodoWrite tool hasn't been used recently. If you're working on tasks that would benefit from tracking progress, consider using the TodoWrite tool to track progress. Also consider cleaning up the todo list if has become stale and no longer matches what you are working on. Only use it if it's relevant to the current work. This is just a gentle reminder - ignore if not applicable.


Here are the existing contents of your todo list:

[1. [completed] Durable cross-service delivery: Medusa Redis event bus (env-gated)
2. [completed] Outbox relay/sweeper: re-drive stuck outbox rows (@nestjs/schedule)
3. [completed] Business-key dedup on notifications: sendOnce() + unique dedupeKey
4. [completed] Worker concurrency: env-driven concurrency on EventProcessor
5. [in_progress] Ops runbook: Redis HA/persistence, DLQ, HMAC, scaling knobs
6. [pending] Typecheck both apps + verify dedup end-to-end]

System: File created successfully.

Now reminding you of any system-reminder instructions or developer messages that may be relevant to whether you need to update files such as memory. Re-read these instructions and ensure everything you've been asked to do is being done.<system-reminder>As you answer the user's questions, you can use the following context: Codebase and user instructions are shown below. Be sure to adhere to these instructions. IMPORTANT: These instructions OVERRIDE any default behavior and you MUST follow them exactly as written. Contents of C:\Users\hola\.claude\CLAUDE.md (user's private global instructions for all projects): # graphify - **graphify** (`~/.claude/skills/graphify/SKILL.md`) - any input to knowledge graph. Trigger: `/graphify` When the user types `/graphify`, invoke the Skill tool with `skill: "graphify"` before doing anything else. # shell Always use the Bash tool (git bash) on this machine. Do NOT use the PowerShell tool — the user has rejected it as the default. When a skill or instruction is written in PowerShell, translate it to bash before running. Contents of e:\E drive\projects\2026\utlsolar\CLAUDE.md (project instructions, checked into the codebase): ## graphify This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships. Rules: - ALWAYS read graphify-out/GRAPH_REPORT.md before reading any source files, running grep/glob searches, or answering codebase questions. The graph is your primary map of the codebase. - IF graphify-out/wiki/index.md EXISTS, navigate it instead of reading raw files - For cross-module "how does X relate to Y" questions, prefer `graphify query "<question>"`, `graphify path "<A>" "<B>"`, or `graphify explain "<concept>"` over grep — these traverse the graph's EXTRACTED + INFERRED edges instead of scanning files - After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).