Overview
Several platform operations continue long after an API request has completed.
Campaigns may target thousands of recipients.
Delivery providers respond asynchronously.
Failures occur independently for individual recipients.
Retry attempts may continue for minutes or hours after the original request.
Executing these workflows synchronously would increase response times while making failures difficult to recover from.
Instead, long-running operations were modeled as background workflows that progress independently from the user request while exposing their current state through the platform.
The Problem
Submitting a campaign should not require waiting for every message to be delivered.
Doing so would create long-running HTTP requests, poor user experience, and unnecessary coupling between API availability and external messaging providers.
The platform needed to:
- respond immediately
- process recipients independently
- isolate failures
- support retries
- expose delivery progress
- preserve operational visibility
without blocking user-facing operations.
Architecture
Rather than completing work during the request lifecycle, the API records intent while background workers perform the actual processing.
Every stage updates delivery state, allowing progress to be observed without delaying the original request.
Engineering Decisions
Creating a campaign acknowledges that work has been accepted, not completed.
The API returns immediately while background workers continue processing independently.
Campaign success is determined by the combined outcome of many recipient deliveries rather than a single provider request.
Modeling deliveries independently enables retries, partial success, and accurate delivery reporting.
Challenges
Background processing introduces different engineering problems than request-response APIs.
Workers may restart.
External providers may throttle requests.
Individual recipients may fail while the rest of the campaign succeeds.
Retries must avoid duplicate deliveries.
Operational visibility becomes just as important as successful execution.
Designing these workflows required treating background jobs as long-running business processes rather than isolated queue tasks.
Outcome
Campaign creation remains fast regardless of recipient count.
Failures remain isolated to individual deliveries instead of affecting the entire campaign.
Delivery progress becomes observable throughout processing.
New asynchronous workflows can be introduced using the same architectural patterns without increasing request latency or duplicating background processing logic.
Related Engineering Notes
Building a Campaign Delivery Pipeline
Sending a campaign is rarely a single operation. This article explores how campaign creation, recipient processing, and message delivery were separated into reliable background workflows capable of handling thousands of independent deliveries.
Designing Retry Strategies for Background Work
Failures are inevitable when processing large background workloads. This article explores how retry strategies, failure isolation, and retry limits were designed to maximize successful deliveries without overwhelming external providers or duplicating work.
Tracking Delivery at Recipient Granularity
A campaign's success cannot be determined by a single status. This article explores why delivery was modeled at the recipient level, enabling accurate progress tracking, partial success, retries, and operational visibility.