Designing Multi-Step Telephony Provisioning with Rollback Orchestration
Provisioning communication resources looks deceptively simple until you try to automate it.
From the outside, the requirement sounded straightforward: once a company's subscription became active, automatically provision everything needed for SMS communication.
In reality, provisioning wasn't a single operation. It was a sequence of dependent API calls, each producing resources required by the next. A failure halfway through the process could leave orphaned resources, inconsistent database state, or a tenant that was only partially configured.
This article explains how I approached that problem.
The Problem
Each company needed its own isolated communication environment.
Provisioning involved creating multiple resources in a specific order:
The challenge wasn't creating these resources.
It was handling what happens when step 4 or step 5 fails.
Without orchestration, a failure midway through provisioning leaves orphan resources behind: a Twilio subaccount with a purchased phone number that is never attached to a service, or a database record pointing to a non-existent provider resource.
Designing for Failure
To build a reliable workflow, every step in the sequence must define two things:
- How to execute the step.
- How to undo the step.
If any step fails, the system executes the undo operation for every step that already succeeded, in reverse order.
Conceptually, rollback works like a stack.
As steps succeed, their corresponding undo operations are pushed onto the rollback stack.
If an error occurs, the workflow pops operations off the stack and executes them one by one.
Thinking in Transactions
Traditional database transactions don't help when your workflow spans external systems.
Once an external API creates a resource, that operation cannot simply be rolled back with a database transaction.
Instead, the provisioning process behaves more like a distributed transaction composed of independent steps.
Each successful operation produces a new resource while simultaneously creating a rollback obligation.
That shifts the implementation from a linear sequence into a reversible workflow.
Provisioning wasn't treated as "fire a series of API calls." Instead, every successful operation immediately became something the system knew how to undo if a later step failed.
Ordering Matters
The order of operations wasn't arbitrary.
Every step depended on resources created by previous ones.
For example:
- API credentials belong to a specific subaccount.
- Phone numbers are purchased under that subaccount.
- Messaging services reference purchased numbers.
- Webhooks depend on messaging configuration.
- Database records are only meaningful after external resources exist.
Persisting configuration before successful provisioning would create local state pointing to resources that never existed.
Provisioning in the opposite order avoids that inconsistency.
The database effectively becomes the final confirmation that provisioning succeeded.
Explicit Rollback
Rather than relying on retries or manual cleanup, the implementation tracks each completed provisioning step.
If provisioning aborts midway, cleanup happens in reverse order.
Conceptually the flow looks like this:
Reverse-order cleanup is important because dependencies also exist during deletion.
Deleting a parent resource before its children often fails.
Undoing the workflow backwards guarantees every dependency still exists while cleanup is happening.
Isolating Failure
One interesting observation during implementation was that failures weren't treated equally.
Some failures happen before any resources are created.
Others happen after almost everything has succeeded.
Those two situations require completely different responses.
For example:
| Failure | Response |
|---|---|
| Invalid subscription | Abort immediately |
| Provider authentication failure | Abort immediately |
| Phone number unavailable | Roll back previously created resources |
| Messaging configuration failure | Roll back provisioning |
| Database persistence failure | Roll back external resources |
The important distinction is whether external state already exists.
Once it does, cleanup becomes mandatory.
Keeping the Database Honest
The local database acts as the source of truth for application behavior.
That means it should never reference resources that failed to provision.
One design decision was delaying persistence until provisioning reached a stable point.
Rather than writing partial configuration after every API call, the implementation waits until the external infrastructure has been successfully assembled.
Only then is the tenant marked as provisioned.
This avoids situations where the application believes a company has communication resources when those resources only exist partially—or not at all.
The database represents completed infrastructure rather than work in progress. Intermediate provisioning state remains in memory during execution, reducing the chances of inconsistent application state.
Recoverability Over Convenience
Another design goal was making failures recoverable.
External APIs occasionally fail for reasons outside the application's control.
Network interruptions.
Temporary service outages.
Provider-side errors.
Those situations shouldn't permanently corrupt tenant configuration.
By ensuring provisioning either completes entirely or cleans up after itself, retries become much safer.
Instead of wondering which resources already exist, the system always retries from a predictable starting point.
That dramatically reduces operational complexity.
Why Explicit Rollback Instead of Automatic Retries?
Automatic retries sound attractive.
In practice they introduce ambiguity.
Consider a timeout after purchasing a phone number.
Did the purchase actually fail?
Or did the provider create the resource while the response was lost?
Retrying blindly risks duplicate resources.
Instead, the implementation favors explicit cleanup followed by a fresh provisioning attempt.
While slightly more work, it produces deterministic behavior and avoids resource leakage.
Lessons Learned
The most valuable lesson wasn't about Twilio.
It was about external systems in general.
Any workflow spanning multiple third-party operations should be treated as a sequence of independent state transitions—not a single request.
Thinking this way naturally leads to questions like:
- What happens if step four fails?
- Can this operation be safely repeated?
- Which resources now require cleanup?
- Which state should be considered authoritative?
Answering those questions early makes provisioning systems significantly easier to reason about.
Future Improvements
The current implementation executes provisioning synchronously as part of the activation workflow.
As tenant volume grows, I'd move this orchestration into a background processing pipeline.
That would provide:
- Automatic retry policies
- Better resilience against temporary provider failures
- Progress tracking
- Operational dashboards
- Dead-letter handling for failed provisioning attempts
The rollback strategy itself would remain unchanged.
Only the execution model would evolve.
Reliable provisioning isn't about creating resources.
It's about ensuring that every possible execution path—success or failure—leaves the system in a state that engineers can understand, operators can trust, and users never have to think about.