<div id="readability-page-1" class="page"><div><p>A new paper <a href="https://arxiv.org/abs/2402.05120" target="_blank" rel="noopener noreferrer" data-external="true">More Agents Is All You Need</a> finds that, simply via a sampling-and-voting method, the performance of LLMs scales with the number of agents instantiated. This can imply that the popularity of multi-agent frameworks is justified.</p>
<p>CrewAI, also <a href="https://www.toolify.ai/ai-news/developing-autonomous-ai-agents-with-crewai-autogen-20-installation-guide-541783" target="_blank" rel="noopener noreferrer" data-external="true">called AutoGen 2.0</a>, is a recently popular multi-agent framework. I tested CrewAI and compared it to AutoGen, mainly regarding the LLM-generated code execution capabilities.</p>
<figure><figcaption>GitHub stars rating evolution of AutoGen and CrewAI (https://star-history.com/#joaomdmoura/crewAI&amp;microsoft/autogen&amp;Date)</figcaption></figure>
<p>CrewAI is built on top of LangChain and allows one to orchestrate multiple agents working on a user-defined task. Same as <a href="https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat/" target="_blank" rel="noopener noreferrer" data-external="true">AutoGen</a>, CrewAI is open-source and uses the concept of agents with different roles, but on top of that, CrewAI allows agents to delegate work to each other.</p>
<figure><figcaption>Working model of CrewAI agents. Source (https://github.com/joaomdmoura/crewai)</figcaption></figure>
<h2 id="why-the-hype">Why the hype?</h2>
<p>There are several explanations for CrewAI's popularity. It is quick to set up, and works well for a variety of interesting use cases with clear guides and demos, e.g.:</p>

<h2 id="code-execution-comparison">Code execution comparison</h2>
<h3 id="autogen">AutoGen</h3>
<p>What I like about AutoGen is that it is <a href="https://microsoft.github.io/autogen/docs/FAQ/#code-execution" target="_blank" rel="noopener noreferrer" data-external="true">execution-capable</a> of the code output it produces. That is, when I wanted to analyze and visualize a dataset, AutoGen agents generated a code for it, executed the code via Docker, and saved the resulting chart as a PDF file on my computer.</p>
<figure><figcaption>AutoGen code execution feature used for generating a chart for stock prices (/resources/microsoft-s-autogen)</figcaption></figure>
<p>By default, AutoGen <a href="https://microsoft.github.io/autogen/blog/2024/01/23/Code-execution-in-docker/" target="_blank" rel="noopener noreferrer" data-external="true">currently uses</a> Docker containers to execute Python code. They even <a href="https://microsoft.github.io/autogen/blog/2023/11/13/OAI-assistants/" target="_blank" rel="noopener noreferrer" data-external="true">added a Code Interpreter example</a> made with a new (experimental) agent called the GPTAssistantAgent that lets you add the new OpenAI assistants into AutoGen-based workflows.</p>
<p>Executing LLM-generated code locally via Docker may be limiting o for some use cases and possesses some risks, but there exists a cloud alternative. In this <a href="https://github.com/e2b-dev/e2b-cookbook" target="_blank" rel="noopener noreferrer" data-external="true">open-source code interpreter example</a>, the code produced by AutoGen agents is running in an isolated cloud environment.</p>
<h3 id="crewai">CrewAI</h3>
<p>When asked for similar data analysis tasks, CrewAI by default generates a text report. It works well with search tools like <a href="https://python.langchain.com/docs/integrations/tools/ddg" target="_blank" rel="noopener noreferrer" data-external="true">LangChain DuckDuckGo Search</a>, but to perform more complex data analysis tasks, it would need tools that allow code execution of the LLM-generated code.</p>
<figure></figure>
<figure><figcaption>Another example of CrewAI performing a stock analysis task. Source (https://www.youtube.com/watch?v=e0Uj4yWdaAg&amp;ab_channel=BuildNewThings)</figcaption></figure>
<p>I haven’t found a quick way to add such tools, but it still should be possible to integrate them. In some examples, like <a href="https://github.com/crewAIInc/crewAI-examples/tree/main/crews/landing_page_generator" target="_blank" rel="noopener noreferrer" data-external="true">generating a landing page</a>, CrewAI uses other (custom) tools, like writing a new file with content.</p>
<figure><figcaption>Source (https://github.com/crewAIInc/crewAI-examples/blob/main/crews/landing_page_generator/src/landing_page_generator/tools/file_tools.py)</figcaption></figure>
<h3 id="langchain-tools-for-code-execution">LangChain tools for code execution</h3>
<p>Lang Chain offers several <a href="https://python.langchain.com/docs/modules/agents/tools/" target="_blank" rel="noopener noreferrer" data-external="true">Tools</a> where LLM-generated code gets automatically executed.</p>
<p>One example is the <a href="https://python.langchain.com/docs/integrations/toolkits/pandas" target="_blank" rel="noopener noreferrer" data-external="true">Pandas Dataframe</a> where a Python agent is used to execute the LLM-generated Python code.</p>
<p>Another example is <a href="https://python.langchain.com/docs/integrations/tools/python" target="_blank" rel="noopener noreferrer" data-external="true">Python REPL</a> which can execute Python commands.</p>
<p>There is even one Langchain tool for remote code execution. <a href="https://python.langchain.com/docs/integrations/tools/bearly" target="_blank" rel="noopener noreferrer" data-external="true">Bearly Code Interpreter</a> allows safe LLM code execution by evaluating Python code in a sandbox environment. This environment resets on every execution.</p>
<p>Apart from these, users can even build their <a href="https://python.langchain.com/docs/modules/agents/tools/custom_tools" target="_blank" rel="noopener noreferrer" data-external="true">custom Langchain tools</a> for code execution and add them to CrewAI.</p>
<p>In conclusion, LangChain tools are able to execute code snippets for example via the Python runtime environment.</p>
<h3 id="limitations">Limitations</h3>
<p>Running LLM-generated code can pose a security risk in general. Either because a user asks the LLM to generate malicious code or the LLM generates malicious code accidentally.</p>
<p>Even the official LangChain tool Pandas Dataframe explicitly mentions “This can be bad if the LLM generated Python code is harmful. Use cautiously.”</p>
<figure><figcaption>Source (https://python.langchain.com/docs/integrations/toolkits/pandas)</figcaption></figure>
<figure><figcaption>Source (https://python.langchain.com/docs/integrations/tools/python)</figcaption></figure>
<p>LangChain recently received feedback to add a more secure way of running the LLM-generated code, e.g., the same way AutoGen does.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I can understand the popularity of both AutoGen and CrewAI as they have proven the ability to deliver some interesting and useful examples quickly. While CrewAI is younger than AutoGen, it would be cool to see benchmarks and evals from both frameworks to make it easier for developers to make the right decision when deciding.</p>
<p>I heard from some developers that they chose CrewAI because they were already familiar with LangChain, and others argued that AutoGen is more customizable. However, when discussing with developers, most said that they don’t see a big difference between CrewAI and AutoGen as they accomplish similar tasks.</p></div></div>

E2B Blog & Customer Stories

Vaclav Mlejnsky

I tested CrewAI and compared it with AutoGen, mainly regarding the LLM-generated code execution capabilities.

CrewAI vs AutoGen for Code Execution AI Agents

댓글