<div id="readability-page-1" class="page"><div><img src="https://devstarsj.github.io/assets/https://images.unsplash.com/photo-1518770660439-4636190af475?w=1600&amp;q=80">Distributed systems have a dirty secret: most of the code isn’t business logic. It’s glue. Retry loops, idempotency keys, saga compensations, distributed locks, timeout handling — the machinery that makes workflows reliable dwarfs the code that makes them useful. Durable execution is a paradigm that promises to make this machinery invisible.<img src="https://images.unsplash.com/photo-1518770660439-4636190af475?w=1200&amp;q=80" alt="Distributed Systems"> Photo by <a href="https://unsplash.com/@alexkixa">Alexandre Debiève</a> on Unsplash<h2 id="the-problem-everything-fails">The Problem: Everything Fails</h2>Consider a simple e-commerce checkout:<ol><li>Reserve inventory</li><li>Charge the customer</li><li>Send confirmation email</li><li>Update fulfillment system</li></ol>What happens if step 3 fails after step 2 succeeds? What if the pod dies mid-execution? What if step 2 is called twice due to a network timeout and the customer gets charged twice?The naive approach is a transaction, but these steps span multiple services, external APIs, and I/O boundaries — traditional ACID transactions don’t apply. The standard solution is the Saga pattern: define compensating transactions for each step and roll back on failure.<div><pre><code># Traditional saga implementation — just the skeleton
class CheckoutSaga:
 def execute(self, order_id: str):
 inventory_reserved = False
 payment_charged = False
 
 try:
 # Step 1
 self.inventory_service.reserve(order_id)
 inventory_reserved = True
 
 # Step 2
 payment_id = self.payment_service.charge(order_id)
 payment_charged = True
 
 # Step 3
 self.email_service.send_confirmation(order_id)
 
 # Step 4
 self.fulfillment_service.update(order_id)
 
 except Exception as e:
 # Compensations
 if payment_charged:
 self.payment_service.refund(payment_id)
 if inventory_reserved:
 self.inventory_service.release(order_id)
 raise
</code></pre></div>This is the happy path skeleton. The real implementation needs:<ul><li>Retry logic for transient failures</li><li>Idempotency to avoid double-charges on retries</li><li>Durable state so crashes don’t lose progress</li><li>Timeout handling for slow services</li><li>Visibility into what’s running</li></ul>You end up with hundreds of lines of infrastructure code per workflow. And it still has bugs.<h2 id="durable-execution-write-normal-code-get-reliability-for-free">Durable Execution: Write Normal Code, Get Reliability for Free</h2>Durable execution engines like Temporal and Restate take a different approach: they intercept your function calls and make them durable automatically. Your code looks like sequential Python (or Go, TypeScript, Java), but under the hood, every step is checkpointed.Here’s the same checkout flow in Temporal:<div><pre><code>from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.worker import Worker
from datetime import timedelta

@activity.defn
async def reserve_inventory(order_id: str) -&gt; str:
 # Your normal service call here
 return await inventory_service.reserve(order_id)

@activity.defn
async def charge_payment(order_id: str) -&gt; str:
 return await payment_service.charge(order_id)

@activity.defn
async def send_confirmation(order_id: str) -&gt; None:
 await email_service.send(order_id)

@activity.defn
async def update_fulfillment(order_id: str) -&gt; None:
 await fulfillment_service.update(order_id)

@workflow.defn
class CheckoutWorkflow:
 @workflow.run
 async def run(self, order_id: str) -&gt; str:
 # Each activity is automatically:
 # - Retried on failure (with backoff)
 # - Idempotent (won't run twice)
 # - Checkpointed (crash-safe)
 reservation_id = await workflow.execute_activity(
 reserve_inventory,
 order_id,
 start_to_close_timeout=timedelta(seconds=30),
 )
 
 payment_id = await workflow.execute_activity(
 charge_payment,
 order_id,
 start_to_close_timeout=timedelta(seconds=30),
 )
 
 await workflow.execute_activity(
 send_confirmation,
 order_id,
 start_to_close_timeout=timedelta(minutes=5),
 retry_policy=RetryPolicy(maximum_attempts=5),
 )
 
 await workflow.execute_activity(
 update_fulfillment,
 order_id,
 start_to_close_timeout=timedelta(minutes=2),
 )
 
 return payment_id
</code></pre></div>If the worker pod crashes after <code>charge_payment</code> succeeds, Temporal will restart the workflow from <code>send_confirmation</code> — not from the beginning. The payment won’t be charged twice. This is event sourcing under the hood: Temporal records every event, and replays the history to restore workflow state on recovery.<h2 id="restate-the-new-challenger">Restate: The New Challenger</h2>Restate (released 2024, matured through 2025) takes a similar approach but with some key differences:<ol><li>HTTP-native: Workflows are just HTTP handlers with a special SDK. No separate worker process.</li><li>Deterministic execution: Uses a virtual thread model, not coroutines.</li><li>Built-in key-value storage: Each workflow/service gets durable, strongly-consistent K/V storage.</li><li>Lower operational overhead: Deploy as a sidecar or standalone server.</li></ol><div><pre><code>// Restate example: same checkout flow
import * as restate from "@restatedev/restate-sdk";

const checkoutService = restate.workflow({
 name: "CheckoutWorkflow",
 handlers: {
 run: async (ctx: restate.WorkflowContext, orderId: string) =&gt; {
 // ctx.run() wraps any action as a durable step
 const reservationId = await ctx.run("reserve_inventory", async () =&gt; {
 return await inventoryClient.reserve(orderId);
 });

 const paymentId = await ctx.run("charge_payment", async () =&gt; {
 return await paymentClient.charge(orderId);
 });

 // Save intermediate state — durable, survives crashes
 await ctx.set("payment_id", paymentId);

 await ctx.run("send_confirmation", async () =&gt; {
 await emailClient.send(orderId, paymentId);
 });

 await ctx.run("update_fulfillment", async () =&gt; {
 await fulfillmentClient.update(orderId, reservationId);
 });

 return { paymentId, reservationId };
 },
 
 // Workflows can receive signals/queries
 getStatus: async (ctx: restate.WorkflowSharedContext, _: void) =&gt; {
 return await ctx.get&lt;string&gt;("payment_id") ?? "pending";
 }
 }
});
</code></pre></div>The key insight: <code>ctx.run()</code> journals the result of the closure. If the process crashes and restarts, Restate replays the journal — it won’t call <code>paymentClient.charge()</code> again because the result is already recorded.<h2 id="how-the-journal-works">How the Journal Works</h2>Both Temporal and Restate use an event-sourcing journal under the hood:<div><pre><code>Event 1: WorkflowStarted { input: "order-123" }
Event 2: ActivityScheduled { name: "reserve_inventory" }
Event 3: ActivityCompleted { result: "res-456" }
Event 4: ActivityScheduled { name: "charge_payment" }
Event 5: ActivityCompleted { result: "pay-789" }
-- CRASH --
Event 6: ActivityScheduled { name: "send_confirmation" }
-- REPLAY: Replay events 1-5, skip re-executing them, continue from event 6 --
</code></pre></div>On replay, the SDK intercepts your code’s calls and returns the recorded results instead of executing them. Your code runs top-to-bottom as if nothing happened.This is why workflow code must be deterministic — no random numbers, no <code>Date.now()</code>, no external reads outside of activities/<code>ctx.run()</code>. The SDK provides safe alternatives:<div><pre><code># ❌ Wrong — non-deterministic
import random
delay = random.randint(1, 10)

# ✅ Correct — use workflow-safe API
delay = workflow.random().randint(1, 10)

# ❌ Wrong
now = datetime.now()

# ✅ Correct
now = workflow.now()
</code></pre></div><h2 id="temporal-vs-restate-which-to-choose">Temporal vs Restate: Which to Choose?</h2><table><thead><tr><th>Dimension</th><th>Temporal</th><th>Restate</th></tr></thead><tbody><tr><td>Maturity</td><td>Production-proven (since 2021)</td><td>Newer (2024+)</td></tr><tr><td>Language support</td><td>Go, Java, Python, TypeScript, .NET</td><td>TypeScript, Java, Go, Kotlin</td></tr><tr><td>Operational complexity</td><td>High (cluster to manage)</td><td>Low (single binary)</td></tr><tr><td>Temporal Workflows</td><td>Months to years supported</td><td>Shorter workflows more natural</td></tr><tr><td>Built-in storage</td><td>No (external DB)</td><td>Yes (embedded RocksDB)</td></tr><tr><td>Cloud offering</td><td>Temporal Cloud</td><td>Restate Cloud</td></tr><tr><td>Best for</td><td>Complex, long-running workflows</td><td>HTTP services needing durability</td></tr></tbody></table><h2 id="when-durable-execution-shines">When Durable Execution Shines</h2>Perfect use cases:<ul><li>Payment processing — never double-charge, never lose a transaction</li><li>Data pipelines — resume from failure without reprocessing</li><li>AI agent workflows — long-running LLM chains with tool calls</li><li>User onboarding flows — multi-step processes that can span days</li><li>Reconciliation jobs — idempotent, retryable batch operations</li><li>Distributed transactions — saga pattern without the saga boilerplate</li></ul><h2 id="getting-started">Getting Started</h2><div><pre><code># Temporal — start dev server
brew install temporal
temporal server start-dev

# Install SDK
pip install temporalio

# Restate — start server 
brew install restatedev/tap/restate-server
restate-server

# Install SDK
npm install @restatedev/restate-sdk
</code></pre></div>The developer experience for both is excellent — you get a local dev server with a full UI showing workflow history, state, and failure reasons.<img src="https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=1200&amp;q=80" alt="Temporal UI"> Photo by <a href="https://unsplash.com/@lukechesser">Luke Chesser</a> on Unsplash<h2 id="conclusion">Conclusion</h2>Durable execution is one of those paradigm shifts that, once understood, makes you wonder how you ever built distributed systems without it. The complexity of sagas, compensations, retry loops, and idempotency keys doesn’t disappear — it moves into the runtime, where it can be thoroughly tested and maintained by specialists.For new services handling complex multi-step workflows, the question is no longer whether to use durable execution, but which engine fits your operational maturity and language preferences. Temporal for complex, long-lived processes in mature organizations; Restate for teams wanting simpler operations with modern HTTP-native ergonomics.<hr>Related: <a href="/2026/05/09/multi-agent-orchestration-frameworks-2026/">Multi-Agent Orchestration Frameworks in 2026</a> <div>이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :) </div></div></div>

Dev Note

Distributed systems have a dirty secret: most of the code isn’t business logic. It’s glue. Retry loops, idempotency keys, saga compensations, distributed locks, timeout handling — the machinery that makes workflows reliable dwarfs the code that makes them useful. Durable execution is a paradigm that promises to make this machinery invisible.
Photo by Alexandre Debiève on Unsplash
The Problem: Everything Fails
Consider a simple e-commerce checkout:
Reserve inventory
Charge the customer
Send confirmation email
Update fulfillment system
What happens if step 3 fails after step 2 succeeds? What if the pod dies mid-execution? What if step 2 is called twice due to a network timeout and the customer gets charged twice?
The naive approach is a transaction, but these steps span multiple services, external APIs, and I/O boundaries — traditional ACID transactions don’t apply. The standard solution is the Saga pattern: define compensating transactions for each step and roll back on failure.
# Traditional saga implementation — just the skeleton
class CheckoutSaga:
 def execute(self, order_id: str):
 inventory_reserved = False
 payment_charged = False
 
 try:
 # Step 1
 self.inventory_service.reserve(order_id)
 inventory_reserved = True
 
 # Step 2
 payment_id = self.payment_service.charge(order_id)
 payment_charged = True
 
 # Step 3
 self.email_service.send_confirmation(order_id)
 
 # Step 4
 self.fulfillment_service.update(order_id)
 
 except Exception as e:
 # Compensations
 if payment_charged:
 self.payment_service.refund(payment_id)
 if inventory_reserved:
 self.inventory_service.release(order_id)
 raise
This is the happy path skeleton. The real implementation needs:
Retry logic for transient failures
Idempotency to avoid double-charges on retries
Durable state so crashes don’t lose progress
Timeout handling for slow services
Visibility into what’s running
You end up with hundreds of lines of infrastructure code per workflow. And it still has bugs.
Durable Execution: Write Normal Code, Get Reliability for Free
Durable execution engines like Temporal and Restate take a different approach: they intercept your function calls and make them durable automatically. Your code looks like sequential Python (or Go, TypeScript, Java), but under the hood, every step is checkpointed.
Here’s the same checkout flow in Temporal:
from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.worker import Worker
from datetime import timedelta
@activity.defn
async def reserve_inventory(order_id: str) -> str:
 # Your normal service call here
 return await inventory_service.reserve(order_id)
@activity.defn
async def charge_payment(order_id: str) -> str:
 return await payment_service.charge(order_id)
@activity.defn
async def send_confirmation(order_id: str) -> None:
 await email_service.send(order_id)
@activity.defn
async def update_fulfillment(order_id: str) -> None:
 await fulfillment_service.update(order_id)
@workflow.defn
class CheckoutWorkflow:
 @workflow.run
 async def run(self, order_id: str) -> str:
 # Each activity is automatically:
 # - Retried on failure (with backoff)
 # - Idempotent (won't run twice)
 # - Checkpointed (crash-safe)
 reservation_id = await workflow.execute_activity(
 reserve_inventory,
 order_id,
 start_to_close_timeout=timedelta(seconds=30),
 )
 
 payment_id = await workflow.execute_activity(
 charge_payment,
 order_id,
 start_to_close_timeout=timedelta(seconds=30),
 )
 
 await workflow.execute_activity(
 send_confirmation,
 order_id,
 start_to_close_timeout=timedelta(minutes=5),
 retry_policy=RetryPolicy(maximum_attempts=5),
 )
 
 await workflow.execute_activity(
 update_fulfillment,
 order_id,
 start_to_close_timeout=timedelta(minutes=2),
 )
 
 return payment_id
If the worker pod crashes after charge_payment succeeds, Temporal will restart the workflow from send_confirmation — not from the beginning. The payment won’t be charged twice. This is event sourcing under the hood: Temporal records every event, and replays the history to restore workflow state on recovery.
Restate: The New Challenger
Restate (released 2024, matured through 2025) takes a similar approach but with some key differences:
HTTP-native: Workflows are just HTTP handlers with a special SDK. No separate worker process.
Deterministic execution: Uses a virtual thread model, not coroutines.
Built-in key-value storage: Each workflow/service gets durable, strongly-consistent K/V storage.
Lower operational overhead: Deploy as a sidecar or standalone server.
// Restate example: same checkout flow
import * as restate from "@restatedev/restate-sdk";
const checkoutService = restate.workflow({
 name: "CheckoutWorkflow",
 handlers: {
 run: async (ctx: restate.WorkflowContext, orderId: string) => {
 // ctx.run() wraps any action as a durable step
 const reservationId = await ctx.run("reserve_inventory", async () => {
 return await inventoryClient.reserve(orderId);
 });
 const paymentId = await ctx.run("charge_payment", async () => {
 return await paymentClient.charge(orderId);
 });
 // Save intermediate state — durable, survives crashes
 await ctx.set("payment_id", paymentId);
 await ctx.run("send_confirmation", async () => {
 await emailClient.send(orderId, paymentId);
 });
 await ctx.run("update_fulfillment", async () => {
 await fulfillmentClient.update(orderId, reservationId);
 });
 return { paymentId, reservationId };
 },
 
 // Workflows can receive signals/queries
 getStatus: async (ctx: restate.WorkflowSharedContext, _: void) => {
 return await ctx.get<string>("payment_id") ?? "pending";
 }
 }
});
The key insight: ctx.run() journals the result of the closure. If the process crashes and restarts, Restate replays the journal — it won’t call paymentClient.charge() again because the result is already recorded.
How the Journal Works
Both Temporal and Restate use an event-sourcing journal under the hood:
Event 1: WorkflowStarted { input: "order-123" }
Event 2: ActivityScheduled { name: "reserve_inventory" }
Event 3: ActivityCompleted { result: "res-456" }
Event 4: ActivityScheduled { name: "charge_payment" }
Event 5: ActivityCompleted { result: "pay-789" }
-- CRASH --
Event 6: ActivityScheduled { name: "send_confirmation" }
-- REPLAY: Replay events 1-5, skip re-executing them, continue from event 6 --
On replay, the SDK intercepts your code’s calls and returns the recorded results instead of executing them. Your code runs top-to-bottom as if nothing happened.
This is why workflow code must be deterministic — no random numbers, no Date.now(), no external reads outside of activities/ctx.run(). The SDK provides safe alternatives:
# ❌ Wrong — non-deterministic
import random
delay = random.randint(1, 10)
# ✅ Correct — use workflow-safe API
delay = workflow.random().randint(1, 10)
# ❌ Wrong
now = datetime.now()
# ✅ Correct
now = workflow.now()
Temporal vs Restate: Which to Choose?
Dimension
 Temporal
 Restate
 
Maturity
 Production-proven (since 2021)
 Newer (2024+)
 
Language support
 Go, Java, Python, TypeScript, .NET
 TypeScript, Java, Go, Kotlin
 
Operational complexity
 High (cluster to manage)
 Low (single binary)
 
Temporal Workflows
 Months to years supported
 Shorter workflows more natural
 
Built-in storage
 No (external DB)
 Yes (embedded RocksDB)
 
Cloud offering
 Temporal Cloud
 Restate Cloud
 
Best for
 Complex, long-running workflows
 HTTP services needing durability
 
When Durable Execution Shines
Perfect use cases:
Payment processing — never double-charge, never lose a transaction
Data pipelines — resume from failure without reprocessing
AI agent workflows — long-running LLM chains with tool calls
User onboarding flows — multi-step processes that can span days
Reconciliation jobs — idempotent, retryable batch operations
Distributed transactions — saga pattern without the saga boilerplate
Getting Started
# Temporal — start dev server
brew install temporal
temporal server start-dev
# Install SDK
pip install temporalio
# Restate — start server 
brew install restatedev/tap/restate-server
restate-server
# Install SDK
npm install @restatedev/restate-sdk
The developer experience for both is excellent — you get a local dev server with a full UI showing workflow history, state, and failure reasons.
Photo by Luke Chesser on Unsplash
Conclusion
Durable execution is one of those paradigm shifts that, once understood, makes you wonder how you ever built distributed systems without it. The complexity of sagas, compensations, retry loops, and idempotency keys doesn’t disappear — it moves into the runtime, where it can be thoroughly tested and maintained by specialists.
For new services handling complex multi-step workflows, the question is no longer whether to use durable execution, but which engine fits your operational maturity and language preferences. Temporal for complex, long-lived processes in mature organizations; Restate for teams wanting simpler operations with modern HTTP-native ergonomics.
Related: Multi-Agent Orchestration Frameworks in 2026

Durable Execution: How Temporal and Restate Are Changing Distributed Systems

댓글