<div id="readability-page-1" class="page"><div><p>On January 13, E2B had a brief infrastructure outage following a continuous production rollout to upgrade our systems. A control-plane failure in one of our regions blocked the creation of new sandboxes and began causing malfunctions in already-running sandboxes.</p>
<p>Service is fully restored. We sincerely apologize to all customers affected. We know sandboxes are on the critical path for agentic workflows, and this outage may have impacted important end-user workflows. We have applied immediate safeguards and are rolling out longer-term improvements to reduce the likelihood and impact of a recurrence. Template data and paused sandboxes were not affected by this incident.</p>
<h2 id="customer-impact">Customer impact</h2>
<ul>
<li><strong>Single node failure</strong>: At 4:31 PM PST sandboxes on one node experienced issues with requests to their management API. We stopped scheduling sandboxes on the node and started investigating this as a workload node issue. At this point sandboxes were being scheduled on the other nodes and there was no impact on their functionality.</li>
<li><strong>Degraded service</strong>: 5:02 PM to 5:12 PM PST. Elevated failures across several nodes occurred while we were triaging the problem and we began stabilizing the control plane.</li>
<li><strong>Outage</strong>: 5:12 PM to 5:35 PM PST. New sandbox creations began to fail, and existing ones started malfunctioning.</li>
<li><strong>Recovery</strong>: 5:35 PM to 5:46 PM PST—Sandbox creation and functionality restored, but capacity was temporarily constrained while additional workload nodes were coming online. Most requests were succeeding by ~5:42 PM, with a small number of errors tapering off by 5:46 PM PST.</li>
<li><strong>Full recovery</strong>: 5:46 PM PST.</li>
</ul>
<h2 id="detection">Detection</h2>
<p>We pinpointed the root cause of the incident via high control-plane CPU usage reports, elevated sandbox-creation failures, and degradation signals in production monitoring. The first signs of the possible incident were reported by users experiencing problems with the sandbox management API during the single node failure.</p>
<h2 id="root-cause">Root cause</h2>
<p>A production deployment of the orchestrator increased the likelihood of triggering a Nomad scheduling failure mode involving reserved host ports and port preemption. Nomad began repeatedly rejecting placement plans (port planning failures and port collisions). During this churn, the control plane experienced increased resource pressure (CPU, disk activity, and memory), and the affected control plane nodes began failing health checks and restarting.</p>
<p>With multiple control plane nodes unstable and without enough time for individual nodes to recover, the cluster could not reliably maintain quorum or conduct leader elections. Without a stable leader and quorum, Nomad couldn't schedule new allocations, and Consul couldn’t provide internal cluster routing, which affected several services and prevented starting or interacting with sandboxes.</p>
<h2 id="resolution">Resolution</h2>
<p>We restored service by re-establishing a healthy Nomad server quorum and re-applying job state:</p>
<ul>
<li>Brought additional server capacity online to restore a stable control plane quorum.</li>
<li>Rebooted the affected control plane nodes so they could rejoin cleanly.</li>
<li>Re-deployed Nomad jobs to the recovered cluster, restoring scheduling and sandbox functionality.</li>
</ul>
<h2 id="corrective-actions">Corrective actions</h2>
<ul>
<li><strong>Monitoring</strong>: Add tighter alerts for memory pressure and early CPU anomalies, as well as plan-failure and port-collision signals; expand Nomad/Consul dashboards.</li>
<li><strong>Stability</strong>: Speed up replacement of unhealthy control-plane servers; maintain sufficient headroom to keep quorum during node instability.</li>
<li><strong>Safer deploys</strong>: Adjust orchestrator port/deploy strategy to avoid possibly risky scheduling patterns; expand pre-deploy checks. In the long term, migrate to a different deploy strategy.</li>
<li><strong>Resilience</strong>: Improve persistence and recovery so the control plane can recover cleanly after reboots. Focus on a multi-region failover so a regional control-plane failure does not affect platform availability.</li>
</ul></div></div>

E2B Blog & Customer Stories

Vaclav Mlejnsky

Postmortem of the January 13, 2026 E2B service disruption: how a control-plane failure blocked new sandbox creation, the impact, and the changes we're making.

Postmortem: Service disruption on Jan 13, 2026

댓글