Failures are an expected part of any distributed system.
Networks experience temporary outages.
Third-party providers become unavailable.
Rate limits are exceeded.
Individual recipients contain invalid contact information.
None of these situations should cause an entire campaign to fail.
Instead of attempting to eliminate failures, the platform was designed to recover from the failures that can reasonably be expected.
The challenge was deciding which failures deserved another attempt and which should be considered final.
The Problem
Retrying every failure sounds attractive until systems begin operating at scale.
A provider experiencing an outage may suddenly receive thousands of identical retry requests.
An invalid phone number will never become valid because another attempt was made.
Some failures disappear after a few seconds.
Others never will.
The platform needed a strategy that distinguished temporary failures from permanent ones while preventing background workers from entering endless retry loops.
The platform treats retries as a recovery mechanism rather than a guarantee of delivery. Only failures likely to succeed on a future attempt are eligible for another execution.
Separating Retryable and Permanent Failures
Not every failure deserves identical treatment.
Some failures indicate temporary conditions.
A provider may be unavailable.
A network request may timeout.
Rate limits may delay processing.
Other failures are permanent.
The recipient may have supplied invalid contact information.
A destination may no longer exist.
Authentication may be invalid because of configuration rather than timing.
The platform evaluates failures based on their characteristics before deciding whether another attempt is appropriate.
Conceptually, the decision looks like this.
This prevents unnecessary work while improving the likelihood that retries actually resolve delivery problems.
Limiting Retry Attempts
Even temporary failures should not be retried forever.
Each additional attempt consumes infrastructure resources, increases provider traffic, and delays campaign completion.
For every delivery, the platform enforces a maximum retry limit.
Once that limit is reached, the delivery is considered permanently failed.
This guarantees that every background job eventually reaches a terminal state.
No delivery remains in processing indefinitely.
Retries improve reliability only when they eventually terminate. Every delivery progresses toward a final outcome, whether successful or failed.
Isolating Failures
Retries operate at the recipient level rather than the campaign level.
If one recipient fails, only that delivery returns to the queue.
Previously delivered recipients remain untouched.
Campaign progress continues while failed deliveries are retried independently.
This isolation significantly reduces duplicated work and prevents successful deliveries from being repeated unnecessarily.
Preserving Operational Visibility
Retrying a delivery should never hide the fact that a failure occurred.
Each attempt becomes part of the delivery history.
Operators can see:
- the original failure
- subsequent retry attempts
- the final outcome
Rather than replacing one status with another, the platform records the complete lifecycle of each delivery.
This visibility simplifies production debugging while providing accurate delivery reporting.
Designing for Idempotency
Retries introduce another important concern.
A repeated job must not produce duplicate effects.
Background workers therefore treat every retry as though the previous attempt may already have completed.
Before performing work, each execution validates the current delivery state.
If another worker has already completed the operation, the retry safely exits without sending duplicate messages or modifying delivery records.
This approach allows workers to recover confidently from crashes, restarts, and temporary infrastructure failures.
Protecting External Providers
Retry behavior affects more than the application itself.
Aggressive retry loops can overwhelm external messaging providers during service disruptions.
Instead of immediately repeating failed requests, retries are scheduled over time, allowing providers an opportunity to recover before additional traffic arrives.
This approach reduces unnecessary load while improving the probability of successful delivery once services become healthy again.
Lessons Learned
Retries should not be viewed as a mechanism for guaranteeing delivery.
They exist to recover from temporary conditions that prevented successful execution the first time.
Treating every failure identically leads to wasted work, unnecessary infrastructure load, and confusing operational behavior.
By distinguishing temporary failures from permanent ones, limiting retry attempts, and designing workers to execute safely multiple times, the platform achieved a more reliable and predictable delivery pipeline.
Future Improvements
The current retry strategy focuses on predictable recovery from transient failures.
Future enhancements could include:
- Exponential backoff between retry attempts
- Provider-specific retry policies
- Adaptive retry limits based on failure type
- Circuit breakers for unhealthy providers
- Automatic provider failover
These improvements can be introduced without changing the overall delivery architecture because retries are already isolated from campaign creation and recipient tracking.
Reliable distributed systems are not defined by the absence of failures.
They are defined by how well they recover when failures inevitably occur.
A thoughtful retry strategy transforms temporary problems into successful deliveries while ensuring permanent failures reach a clear and observable conclusion.