<div id="readability-page-1" class="page"><div><p><a href="https://arena.ai/code" target="_blank" rel="noopener noreferrer" data-external="true">Arena</a>, a UC Berkeley research team, created WebDev Arena to evaluate the capabilities of large language models (LLMs) in web development. It’s a free open-source arena where two LLMs compete to build a web app.</p>
<figure></figure>
<p>You can vote on which LLM performs better and view a leaderboard of the best models.</p>
<figure></figure>
<figure></figure>
<h2 id="the-challenges-of-llm-coding-evals">The challenges of LLM coding evals</h2>
<p>Arena faced key technical challenges in testing multiple LLMs building real-time web applications.</p>
<p>When benchmarking LLMs on web development tasks, there's a fundamental need to run the code output. While getting LLMs to generate code is straightforward, running it securely at scale presents several challenges:</p>
<p><strong>Speed:</strong> For meaningful comparisons, the evaluator needs to see the two LLMs' outputs nearly simultaneously. When you're comparing output from two or multiple models at once, even small delays compound quickly and distort the results of the voting.</p>
<p><strong>Code execution:</strong> Running coding evaluations means handling large amounts of code simultaneously. Each LLM can generate extensive code snippets - from simple DOM manipulations to complex React components. Running these side-by-side requires significant computational resources.</p>
<p><strong>Isolation and security:</strong> Each piece of LLM-generated code needs to run in an isolated environment. When you're comparing models like Claude and GPT-4 simultaneously, their code can't interfere with each other or share any execution context. Running LLM-generated code requires complete isolation to prevent any potential system access or interference between different models' executions.</p>
<h2 id="llm-code-execution">LLM code execution</h2>
<p>These challenges led Arena to implement E2B as their execution environment. Under the hood, each sandbox is a small VM, allowing WebDev Arena to run code from multiple LLMs simultaneously while maintaining strict isolation and performance standards.</p>
<p>Key factors in this decision included:</p>
<ul>
<li>Security first - E2B's isolated environments ensure each LLM's code runs separately and securely</li>
<li>Quick startup - E2B sandboxes start in ~150ms, essential for real-time model comparison</li>
<li>Reliability at scale - Running multiple sandboxes simultaneously for different LLM battles</li>
</ul>
<p>The E2B sandboxes provide isolated cloud environments specifically designed for running AI-generated code. E2B is agnostic towards the choice of techstack, frameworks, and models, which lets WebDev Arena run code from different LLMs (e.g., Claude, GPT-4, Gemini, DeepSeek, and Qwen).</p>
<blockquote><p>“Very quickly, we had over 50,000 users who started over 230,000 E2B sandboxes.”</p></blockquote>
<p>- Aryan Vichare, Member of Technical Staff at LMArena</p>
<h2 id="implementing-e2b">Implementing E2B</h2>
<p>Arena implemented E2B for WebDev Arena in November 2024. The integration involved working closely with E2B's team to ensure reliable execution of code from multiple LLMs.</p>
<p>The implementation consisted of handling sandbox creation and code execution. Here's how it works:</p>
<ol>
<li>Sandbox Configuration and Creation</li>
</ol>
<p>The team implemented a flexible sandbox system with configurable timeouts and templates</p>
<ol>
<li>Dynamic Dependency Management</li>
</ol>
<p>The system supports dynamic package installation based on LLM requirements:</p>
<ul>
<li>Automatic detection of additional dependencies</li>
<li>Custom installation commands for different environments</li>
<li>Real-time logging of dependency installation status</li>
</ul>
<ol>
<li>Code Execution Pipeline</li>
</ol>
<p>The system handles two types of code execution:</p>
<ul>
<li>Code Interpreter Mode: For running and evaluating code with detailed output</li>
<li>Web Development Mode: For deploying and accessing web applications</li>
</ul>
<blockquote><p>“It took 2 hours to get E2B up and running.”</p></blockquote>
<p>- Aryan Vichare, Member of Technical Staff at LMArena</p>
<h2 id="whats-next">What's next</h2>
<p>Since the launch of WebDev Arena with E2B's code execution layer, the platform has run over 50,000 model comparisons, with Claude 3.5 Sonnet currently being on top of the <a href="https://web.lmarena.ai/leaderboard" target="_blank" rel="noopener noreferrer" data-external="true">leaderboard</a>, followed closely by DeepSeek-R1. The team plans to expand the benchmark with more models and plan to do online general-purpose coding evaluations in the future.</p>
<blockquote><p>“We plan to expand to more software engineering tasks."</p></blockquote>
<p>- Aryan Vichare, Member of Technical Staff at LMArena</p></div></div>

E2B Blog & Customer Stories

Vaclav Mlejnsky

Insights from implementing E2B sandboxes for evaluating LLM-generated web apps

How Arena Collaborated with E2B to Build LLM Web Development Evals

댓글