<div id="readability-page-1" class="page"><div><img src="https://devstarsj.github.io/assets/https://images.unsplash.com/photo-1545987796-200677ee1011?w=1600&amp;auto=format&amp;fit=crop">Here’s a scenario every backend engineer has lived through: you write a workflow that calls three external services, processes some data, and updates a database. It works great in staging. In production, it fails on the second service call at 2 AM, leaves the system in a half-finished state, your on-call engineer spends two hours figuring out what happened, and you spend the next week writing idempotency logic, retry handlers, and compensating transactions you should never have needed.Durable execution exists to make this problem disappear.<img src="https://images.unsplash.com/photo-1545987796-200677ee1011?w=1200&amp;auto=format&amp;fit=crop" alt="Distributed Systems Architecture"> Photo by Umberto on Unsplash<h2 id="what-is-durable-execution">What Is Durable Execution?</h2>Durable execution is a programming model where your code’s execution state is automatically persisted to durable storage. If your process crashes, is killed by a scheduler, or the network drops, the execution resumes from exactly where it left off — not from the beginning.The key insight: treat workflow execution history as the source of truth, not application state.Think of it as if every <code>await</code> in your async function automatically checkpointed its state to a database. Your code looks normal, but the runtime makes it crash-proof.<h2 id="the-core-problem-it-solves">The Core Problem It Solves</h2>Traditional approaches to distributed reliability are painful:<div><pre><code># The naive approach — don't do this
def process_order(order_id: str):
 payment = charge_customer(order_id) # What if this crashes?
 inventory = reserve_inventory(order_id) # Or this?
 notification = send_confirmation(order_id) # Or this?
 return {"payment": payment, "inventory": inventory}
</code></pre></div>When this fails mid-way, you have partial state. To fix it you need:<ul><li>Idempotency keys on every external call</li><li>A state machine in a database tracking progress</li><li>Retry logic with exponential backoff</li><li>Compensating transactions for rollback</li><li>Dead letter queues for failures</li><li>Monitoring and alerting for stuck workflows</li></ul>That’s an enormous amount of plumbing for what should be simple business logic.<h2 id="the-durable-execution-model">The Durable Execution Model</h2>With a durable execution runtime (let’s use Temporal as an example):<div><pre><code># Temporal workflow — looks almost identical, behaves very differently
@workflow.defn
class ProcessOrderWorkflow:
 @workflow.run
 async def run(self, order_id: str):
 payment = await workflow.execute_activity(
 charge_customer,
 order_id,
 start_to_close_timeout=timedelta(seconds=30),
 retry_policy=RetryPolicy(maximum_attempts=3)
 )
 
 inventory = await workflow.execute_activity(
 reserve_inventory,
 order_id,
 start_to_close_timeout=timedelta(seconds=30)
 )
 
 await workflow.execute_activity(
 send_confirmation,
 order_id
 )
 
 return {"payment": payment, "inventory": inventory}
</code></pre></div>The difference: if the process crashes after <code>charge_customer</code> succeeds but before <code>reserve_inventory</code> starts, the workflow resumes from that exact point when the worker restarts. <code>charge_customer</code> is not called again. No duplicate charges. No corruption.<h2 id="how-it-works-under-the-hood">How It Works Under the Hood</h2>The magic is event sourcing + deterministic replay:<div><pre><code>Workflow execution history (stored in Temporal server):
[
 { type: "WORKFLOW_STARTED", input: { order_id: "123" } },
 { type: "ACTIVITY_SCHEDULED", activity: "charge_customer" },
 { type: "ACTIVITY_COMPLETED", result: { charge_id: "ch_abc" } },
 // &lt;- Crash happened here
]
</code></pre></div>When a worker restarts:<ol><li>It fetches the history from the Temporal server</li><li>It replays your code, but short-circuits completed steps using the history</li><li>Execution continues from the first unrecorded point</li></ol>Your code executes “as if” it never crashed. The replay is deterministic because the runtime intercepts all non-deterministic operations (time, random, external calls) and replays them from history.This means you must write deterministic workflow code:<div><pre><code># ❌ Bad — uses real time, non-deterministic
import datetime
current_time = datetime.datetime.now()

# ✅ Good — uses workflow time (replayed from history)
current_time = workflow.now()
</code></pre></div><h2 id="the-landscape-in-2026">The Landscape in 2026</h2><h3 id="temporal">Temporal</h3>The pioneer and still the leader in enterprise adoption. Originally created by ex-Uber engineers who built Cadence (Uber’s internal workflow engine). Strong Go and Java SDKs, growing Python and TypeScript support.Strengths: Battle-tested at massive scale, rich ecosystem, strong community Tradeoffs: Operational complexity of running the Temporal server, learning curve<h3 id="restate">Restate</h3>The newcomer making waves with a radically simpler developer experience. Restate uses HTTP and gRPC rather than a custom protocol, and its SDKs feel more like writing normal code:<div><pre><code>// Restate — feels very natural
const orderWorkflow = restate.workflow({
 name: "OrderWorkflow",
 handlers: {
 run: async (ctx: restate.WorkflowContext, orderId: string) =&gt; {
 const payment = await ctx.run(() =&gt; chargeCustomer(orderId));
 const inventory = await ctx.run(() =&gt; reserveInventory(orderId));
 await ctx.run(() =&gt; sendConfirmation(orderId));
 return { payment, inventory };
 }
 }
});
</code></pre></div>Strengths: Simpler mental model, lighter operational footprint, cloud-native from day one Tradeoffs: Younger ecosystem, smaller community<h3 id="aws-step-functions">AWS Step Functions</h3>The managed cloud option. No infrastructure to run, deep AWS integration, but locked into AWS and the visual/YAML workflow definition style feels limiting compared to code-first alternatives.<h3 id="inngest">Inngest</h3>Targeting the serverless/edge use case specifically. Deploys as background functions on Vercel, Cloudflare Workers, etc. No infrastructure needed — events flow through Inngest’s cloud.<img src="https://images.unsplash.com/photo-1518186285589-2f7649de83e0?w=1200&amp;auto=format&amp;fit=crop" alt="Workflow Engine Comparison"> Photo by Christopher Burns on Unsplash<h2 id="when-to-use-durable-execution">When to Use Durable Execution</h2>Durable execution shines when you have long-running, multi-step operations that cross service boundaries:Perfect use cases:<ul><li>Order fulfillment pipelines</li><li>Payment processing workflows</li><li>Document processing pipelines (ingest → parse → analyze → notify)</li><li>User onboarding flows</li><li>CI/CD pipeline orchestration</li><li>ML training job management</li><li>Scheduled/recurring business processes</li></ul>Less appropriate for:<ul><li>Simple CRUD with no multi-step logic</li><li>Real-time low-latency operations (&lt; 100ms)</li><li>Pure data streaming (use Kafka/Flink instead)</li></ul><h2 id="real-world-impact">Real-World Impact</h2>Teams that adopt durable execution consistently report:<ul><li>Massive reduction in reliability incidents — the “stuck workflow” class of bugs essentially disappears</li><li>Simpler code — business logic is no longer tangled with retry/error handling boilerplate</li><li>Better observability — execution history is inspectable, searchable, and replayable for debugging</li><li>Faster incident response — you can see exactly what happened and where a workflow stalled</li></ul>One team I know of replaced a 3,000-line saga implementation (with its own state machine, retry logic, and compensating transactions) with 300 lines of Temporal workflow code. Bugs that had been in production for months were eliminated.<h2 id="getting-started-temporal-in-5-minutes">Getting Started: Temporal in 5 Minutes</h2><div><pre><code># Start Temporal development server
brew install temporal
temporal server start-dev

# In another terminal, run your first workflow
git clone https://github.com/temporalio/samples-python
cd samples-python
pip install temporalio
python hello_activity/run_workflow.py
</code></pre></div>Visit http://localhost:8233 to see your workflow execution in the Temporal UI — complete with timeline, input/output, and replay capability.<h2 id="the-bigger-picture">The Bigger Picture</h2>Durable execution is the distributed systems equivalent of garbage collection: it takes a hard, manual problem and makes it automatic. Just as GC didn’t make memory unlimited but made memory management invisible, durable execution doesn’t make distributed systems simple but makes distributed reliability automatic.As more applications run across multiple services, clouds, and environments, the complexity of coordination failures grows superlinearly. Durable execution is one of the clearest architectural responses to this problem that’s emerged in recent years.If you’re building workflows today without it, you’re probably writing infrastructure code that belongs in a library. Worth rethinking.<hr>Building with Temporal or Restate? I’d love to hear about your experience — especially the pain points. Hit me up in the comments. <div>이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :) </div></div></div>

Dev Note

Here’s a scenario every backend engineer has lived through: you write a workflow that calls three external services, processes some data, and updates a database. It works great in staging. In production, it fails on the second service call at 2 AM, leaves the system in a half-finished state, your on-call engineer spends two hours figuring out what happened, and you spend the next week writing idempotency logic, retry handlers, and compensating transactions you should never have needed.
Durable execution exists to make this problem disappear.
Photo by Umberto on Unsplash
What Is Durable Execution?
Durable execution is a programming model where your code’s execution state is automatically persisted to durable storage. If your process crashes, is killed by a scheduler, or the network drops, the execution resumes from exactly where it left off — not from the beginning.
The key insight: treat workflow execution history as the source of truth, not application state.
Think of it as if every await in your async function automatically checkpointed its state to a database. Your code looks normal, but the runtime makes it crash-proof.
The Core Problem It Solves
Traditional approaches to distributed reliability are painful:
# The naive approach — don't do this
def process_order(order_id: str):
 payment = charge_customer(order_id) # What if this crashes?
 inventory = reserve_inventory(order_id) # Or this?
 notification = send_confirmation(order_id) # Or this?
 return {"payment": payment, "inventory": inventory}
When this fails mid-way, you have partial state. To fix it you need:
Idempotency keys on every external call
A state machine in a database tracking progress
Retry logic with exponential backoff
Compensating transactions for rollback
Dead letter queues for failures
Monitoring and alerting for stuck workflows
That’s an enormous amount of plumbing for what should be simple business logic.
The Durable Execution Model
With a durable execution runtime (let’s use Temporal as an example):
# Temporal workflow — looks almost identical, behaves very differently
@workflow.defn
class ProcessOrderWorkflow:
 @workflow.run
 async def run(self, order_id: str):
 payment = await workflow.execute_activity(
 charge_customer,
 order_id,
 start_to_close_timeout=timedelta(seconds=30),
 retry_policy=RetryPolicy(maximum_attempts=3)
 )
 
 inventory = await workflow.execute_activity(
 reserve_inventory,
 order_id,
 start_to_close_timeout=timedelta(seconds=30)
 )
 
 await workflow.execute_activity(
 send_confirmation,
 order_id
 )
 
 return {"payment": payment, "inventory": inventory}
The difference: if the process crashes after charge_customer succeeds but before reserve_inventory starts, the workflow resumes from that exact point when the worker restarts. charge_customer is not called again. No duplicate charges. No corruption.
How It Works Under the Hood
The magic is event sourcing + deterministic replay:
Workflow execution history (stored in Temporal server):
[
 { type: "WORKFLOW_STARTED", input: { order_id: "123" } },
 { type: "ACTIVITY_SCHEDULED", activity: "charge_customer" },
 { type: "ACTIVITY_COMPLETED", result: { charge_id: "ch_abc" } },
 // <- Crash happened here
]
When a worker restarts:
It fetches the history from the Temporal server
It replays your code, but short-circuits completed steps using the history
Execution continues from the first unrecorded point
Your code executes “as if” it never crashed. The replay is deterministic because the runtime intercepts all non-deterministic operations (time, random, external calls) and replays them from history.
This means you must write deterministic workflow code:
# ❌ Bad — uses real time, non-deterministic
import datetime
current_time = datetime.datetime.now()
# ✅ Good — uses workflow time (replayed from history)
current_time = workflow.now()
The Landscape in 2026
Temporal
The pioneer and still the leader in enterprise adoption. Originally created by ex-Uber engineers who built Cadence (Uber’s internal workflow engine). Strong Go and Java SDKs, growing Python and TypeScript support.
Strengths: Battle-tested at massive scale, rich ecosystem, strong community
Tradeoffs: Operational complexity of running the Temporal server, learning curve
Restate
The newcomer making waves with a radically simpler developer experience. Restate uses HTTP and gRPC rather than a custom protocol, and its SDKs feel more like writing normal code:
// Restate — feels very natural
const orderWorkflow = restate.workflow({
 name: "OrderWorkflow",
 handlers: {
 run: async (ctx: restate.WorkflowContext, orderId: string) => {
 const payment = await ctx.run(() => chargeCustomer(orderId));
 const inventory = await ctx.run(() => reserveInventory(orderId));
 await ctx.run(() => sendConfirmation(orderId));
 return { payment, inventory };
 }
 }
});
Strengths: Simpler mental model, lighter operational footprint, cloud-native from day one
Tradeoffs: Younger ecosystem, smaller community
AWS Step Functions
The managed cloud option. No infrastructure to run, deep AWS integration, but locked into AWS and the visual/YAML workflow definition style feels limiting compared to code-first alternatives.
Inngest
Targeting the serverless/edge use case specifically. Deploys as background functions on Vercel, Cloudflare Workers, etc. No infrastructure needed — events flow through Inngest’s cloud.
Photo by Christopher Burns on Unsplash
When to Use Durable Execution
Durable execution shines when you have long-running, multi-step operations that cross service boundaries:
Perfect use cases:
Order fulfillment pipelines
Payment processing workflows
Document processing pipelines (ingest → parse → analyze → notify)
User onboarding flows
CI/CD pipeline orchestration
ML training job management
Scheduled/recurring business processes
Less appropriate for:
Simple CRUD with no multi-step logic
Real-time low-latency operations (< 100ms)
Pure data streaming (use Kafka/Flink instead)
Real-World Impact
Teams that adopt durable execution consistently report:
Massive reduction in reliability incidents — the “stuck workflow” class of bugs essentially disappears
Simpler code — business logic is no longer tangled with retry/error handling boilerplate
Better observability — execution history is inspectable, searchable, and replayable for debugging
Faster incident response — you can see exactly what happened and where a workflow stalled
One team I know of replaced a 3,000-line saga implementation (with its own state machine, retry logic, and compensating transactions) with 300 lines of Temporal workflow code. Bugs that had been in production for months were eliminated.
Getting Started: Temporal in 5 Minutes
# Start Temporal development server
brew install temporal
temporal server start-dev
# In another terminal, run your first workflow
git clone https://github.com/temporalio/samples-python
cd samples-python
pip install temporalio
python hello_activity/run_workflow.py
Visit http://localhost:8233 to see your workflow execution in the Temporal UI — complete with timeline, input/output, and replay capability.
The Bigger Picture
Durable execution is the distributed systems equivalent of garbage collection: it takes a hard, manual problem and makes it automatic. Just as GC didn’t make memory unlimited but made memory management invisible, durable execution doesn’t make distributed systems simple but makes distributed reliability automatic.
As more applications run across multiple services, clouds, and environments, the complexity of coordination failures grows superlinearly. Durable execution is one of the clearest architectural responses to this problem that’s emerged in recent years.
If you’re building workflows today without it, you’re probably writing infrastructure code that belongs in a library. Worth rethinking.
Building with Temporal or Restate? I’d love to hear about your experience — especially the pain points. Hit me up in the comments.

Durable Execution: The Pattern That Makes Distributed Systems Reliable by Default

댓글