Background jobs and outbound merchant webhooks run on Oban, migrating off Verk

Context

Inkress notifies merchants of events (an order paid, a status change) by delivering outbound webhooks to the URLs a merchant registers, and sends order-paid emails, and runs recurring billing — all as background jobs. The original job system was Verk (Redis-backed), with per-merchant queues and stat counters in Redis. We moved to Oban (Postgres-backed) for durable, inspectable jobs colocated with the data, retiring the separate Redis job store.

This record captures the current state of that migration for the outbound merchant-webhook path, which is a known in-progress area — not a proposed fix.

Decision

Run background jobs on Oban (Postgres, the JOBS database, with Oban Web mounted at /oban), and deliver outbound merchant webhooks through a two-stage Oban pipeline, retiring the Verk/Redis system.

• Delivery model. An order event enqueues Workers.Orders.Oban.NotificationWorker, which fans out to Workers.Webhooks.DeliveryWorker to POST to the merchant's registered webhook_urls (orders.paid and related events); the same notification worker also sends the order-paid receipt/merchant emails. Delivery is at-least-once (Oban retries), and completed/discarded jobs are retained for a bounded window for inspection.
• Enqueue paths are mixed (a hybrid, not settled). commerce-worker owns the Oban workers; commerce-api enqueues in two different ways depending on the call site: a direct enqueue-only Oban client on the shared JOBS database (config :api, Oban, repo: Api.Jobs.Repo, configured to never become the Oban leader — commerce-worker runs the jobs), and an HTTP Interface.Oban.Jobs.enqueue call into commerce-worker. This inconsistency is itself part of the in-progress state below.

Consequences

• Positive: jobs are durable and queryable in Postgres (Oban Web) instead of opaque Redis counters; retries and dead-lettering are built in; no separate Redis job store to operate.
• Positive: subscription billing genuinely runs on this path — the paid-order transition (ADR 0013) enqueues Workers.Subscriptions.Oban.BillingWorker over the HTTP enqueuer — proving the enqueue path is live in production.
• Current state — outbound webhooks are broadly NOT delivered from the live payment paths. The shared helper Service.Payment.Provider.Utils.enqueue_webhook/2 still enqueues to decommissioned Verk (Telemetry.Verk.EventSubscriber.enqueue), and every provider callback path (FAC, Fygaro, Lynk, PayPal, Stripe, WiPay) routes through it — so those orders.paid webhooks and emails are dead. Session-based checkout attempts an Oban NotificationWorker enqueue, but that worker module is not defined in commerce-api, so the call raises and is rescued to a no-op (it only logs), and that same path also still calls the Verk helper. So the durable decision (Oban is the runner; delivery is NotificationWorker → DeliveryWorker in commerce-worker) holds, but the commerce-api enqueue side is only partially wired — subscription billing works, outbound webhooks/emails largely do not.
• Follow-ups: make commerce-api reliably enqueue the notification/delivery job for every paid order (fix the rescued no-op — the worker module/enqueue must actually resolve — and repoint Utils.enqueue_webhook off Verk), then remove the Verk enqueuer and its Redis dependencies. Enabling the notification worker on all paths also restores order-paid emails platform-wide — confirm the email side is desired before flipping.

Alternatives considered

• Stay on Verk (Redis): rejected — job state lived in Redis as opaque counters with weaker durability and inspectability; Postgres-backed Oban colocates jobs with the data and gives retries, Oban Web, and dead-lettering.
• A shared Oban client embedded in commerce-api instead of enqueuing over HTTP to commerce-worker: rejected — it would couple both services to the same job tables and workers and cut across the repo split (ADR 0019); the worker owns the job runtime and exposes an enqueue endpoint.
• Big-bang swap of every enqueue site from Verk to Oban: not taken — the migration proceeds per path (subscriptions and session checkout first), which is why the provider-webhook helper is the remaining Verk caller; the cost is that outbound webhooks are split across a working and a broken path until the helper is cut over.