<div id="readability-page-1" class="page"><div>After the initial hype around AI agents, there has been a cooling-off period as people realize that AI agents are not that autonomous. An agent won’t create the whole complex program dreamed up by a no-code user. Usually, until reaching a desired quality, agents' output needs multiple iterations.
These iterations may not be just human-agent, but rather among a higher number of agents specialized in narrow areas. For example, one agent writes a code specified by the end user, another agent then takes over and debugs the code, then hands it to another agent who can visualize the data, and so on.
<a href="https://www.microsoft.com/en-us/research/blog/autogen-enabling-next-generation-large-language-model-applications/" target="_blank" rel="noopener noreferrer" data-external="true">Recently launched</a> AutoGen by Microsoft has gained especially big popularity among multi-agent <a href="https://github.com/e2b-dev/awesome-sdks-for-ai-agents" target="_blank" rel="noopener noreferrer" data-external="true">frameworks</a>.
<figure><figcaption>Fig. 1. Google trends results for "AutoGen". Source (https://trends.google.com/trends/explore?date=now%207-d&amp;q=AutoGen&amp;hl=cs)</figcaption></figure>
<h2 id="simple-guide-to-autogen">Simple Guide to AutoGen</h2>
What is special about AutoGen is that it is execution-capable of the code output it produces.
<figure><figcaption>Fig. 2. The AutoGen paper compares with Multi-agent Debate, CAMEL, BabyAGI, and MetaGPT.</figcaption></figure>
We will hence focus on that feature and create a simple data visualization Python script, where we explore different types of AutoGen pre-defined agent classes, and demonstrate how AutoGen generates and runs code. I hope it helps understand the principles of AutoGen.
<h3 id="1-install-autogen">1. Install AutoGen</h3>
Start with installing AutoGen from their <a href="https://microsoft.github.io/autogen/docs/Getting-Started" target="_blank" rel="noopener noreferrer" data-external="true">documentation</a> or from <a href="https://github.com/microsoft/autogen" target="_blank" rel="noopener noreferrer" data-external="true">GitHub</a>.
<figure><div><pre tabindex="0"><code>pip install pyautogen
pip install "pyautogen[blendsearch]" for optional dependencies</code></pre></div></figure>
AutoGen offers good support on their <a href="https://discord.gg/E5YAjB5gRY" target="_blank" rel="noopener noreferrer" data-external="true">Discord</a>, even though I don’t know whether it’s official. It also provides a page with concisely written <a href="https://microsoft.github.io/autogen/docs/Examples" target="_blank" rel="noopener noreferrer" data-external="true">Examples</a> from which I choose the data analyst one to try.
<h3 id="2-import-packages">2. Import packages</h3>
AutoGen has a default abstract class called <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/agent" target="_blank" rel="noopener noreferrer" data-external="true">Agent</a> that can communicate with other agents and perform actions. Agents can differ in what actions they perform in the receive method. We import <code>AssistantAgent</code> and <code>UserProxyAgent</code> classes, which are both subclasses of a more generic class - <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent" target="_blank" rel="noopener noreferrer" data-external="true">ConversableAgent.</a> (We will get to this later.)
<figure><div><pre tabindex="0"><code>from autogen import AssistantAgent, UserProxyAgent</code></pre></div></figure>
<h3 id="3-get-api-keys">3. Get API Keys</h3>
Now, we get our API keys. I store mine as the <code>.env</code> variable.
<figure><div><pre tabindex="0"><code>import os
import openai
from dotenv import load_dotenv
load_dotenv()</code></pre></div></figure>
<h3 id="4-create-the-agents">4. Create the agents</h3>
In this step, we can define a set of agents with specialized capabilities and roles.
We create an instance of the <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/assistant_agent" target="_blank" rel="noopener noreferrer" data-external="true"><code>AssistantAgent class</code></a> representing the chatbot that will respond to the user input and an instance of the <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/user_proxy_agent/" target="_blank" rel="noopener noreferrer" data-external="true"><code>UserProxyAgent class</code></a> representing the user that will initiate the conversation.
The LLM inference configuration in AssistantAgent can be configured via <code>llm_cofig</code>.
<figure><div><pre tabindex="0"><code>assistant = AssistantAgent(name="assistant")
user_proxy = UserProxyAgent(name="user_proxy")</code></pre></div></figure>
<h3 id="5-define-the-interaction">5. Define the interaction</h3>
After creating the agents, the script initiates a chat between the user and the chatbot by calling the <code>initiate_chat</code> method on the <code>user_proxy</code> instance. The <code>initiate_chat</code> method takes two arguments: the assistant instance, which represents the chatbot, and a message string that contains the task description.The script then creates a text completion request using the <code>openai.Completion.create</code> method.The <code>config_list</code> parameter is set to a list that contains a dictionary with the model name, API base URL, API type, and API key.The prompt parameter is set to a string that contains the text to be completed. The <code>Completion.create</code> method sends a request to the OpenAI API and returns a response that contains the completed text.
<figure><div><pre tabindex="0"><code>user_proxy.initiate_chat(
&nbsp;&nbsp;&nbsp;&nbsp;assistant,
&nbsp;&nbsp;&nbsp;&nbsp;message="""Hello, today you are my data analyst assistant and you should help me visualize data, make predictions, and explain your thinking.""",
)
response = oai.Completion.create(
&nbsp;&nbsp;&nbsp;&nbsp;config_list=[
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"model": "chatglm2-6b",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"api_base": "http://localhost:8000/v1",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"api_type": "open_ai",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"api_key": "NULL", # just a placeholder
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
&nbsp;&nbsp;&nbsp;&nbsp;],
&nbsp;&nbsp;&nbsp;&nbsp;prompt="Hi",
)
print(response)</code></pre></div></figure>
<h3 id="6-create-a-chat-completion-request">6. Create a chat completion request</h3>
Finally, we create a chat completion request using the <code>openai.ChatCompletion.create</code> method. The &nbsp;<code>config_list</code> parameter is set to the same list as before, and the messages parameter is set to a list that contains a dictionary with the role and content of the user's message.The <code>ChatCompletion.create</code> method sends a request to the OpenAI API and returns a response that contains the chatbot's response to the user's message.
<figure><div><pre tabindex="0"><code>response = oai.ChatCompletion.create(
&nbsp;&nbsp;&nbsp;&nbsp;config_list=[
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"model": "chatglm2-6b",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"api_base": "http://localhost:8000/v1",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"api_type": "open_ai",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"api_key": "NULL",
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
&nbsp;&nbsp;&nbsp;&nbsp;],
&nbsp;&nbsp;&nbsp;&nbsp;messages=[{"role": "user", "content": "Hi"}]
)
print(response)</code></pre></div></figure>
<h2 id="output">Output</h2>
As I mentioned earlier, AssistantAgent and UserProxyAgent classes are both subclasses of a more generic class - <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent" target="_blank" rel="noopener noreferrer" data-external="true">ConversableAgent.</a>
The <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/assistant_agent" target="_blank" rel="noopener noreferrer" data-external="true">AssistantAgent</a> (a subclass of ConversableAgent) is designed to solve a task with LLM. This agent doesn't execute code by default and expects the user to execute the code. After the AssistantAgent produces code output, the user can execute the code by pressing Enter.
I chose my agent to visualize data, which is a task requiring multiple steps like planning the process, writing the code, and executing it in visual form. That should best show its capabilities.
First, the program answers to my default intro message:
<figure><div><pre tabindex="0"><code>message="""Hello, today you are my data analyst assistant and you should help me visualize data, make predictions, and explain your thinking."""</code></pre></div></figure>
<figure></figure>
I now instruct the agent to plot a chart of NVDA and TESLA stock price change YTD. It then prints user input and devises an action plan - which may include even installing new libraries.
<figure></figure>
The agent returned a code that contains an error that is indicated under user_proxy. Here, the user_proxy is used as another agent that provides feedback to the assistant, as opposed to a human instructing the agent with a prompt to fix the code.
The assistant makes another iteration that seems to be functioning code. This was a nice example of <a href="https://stackoverflow.blog/2023/06/07/self-healing-code-is-the-future-of-software-development/" target="_blank" rel="noopener noreferrer" data-external="true">self-healing code</a>.
<figure></figure>
The following diagram summarizes the workflow of iterating between multiple agents.
<figure><figcaption>Fig. 3. Schema of the communication between UserProxyAgent and AssistantAgent. Source (https://microsoft.github.io/autogen/docs/Getting-Started)</figcaption></figure>
<h2 id="code-execution">Code execution</h2>
The output explains what happens if the user (you) decides to run the code. You can always execute the proposed code by pressing “enter”.
When a human user chooses to execute the code, the output opens in a new window like this:
<figure></figure>
It is possible to configure various arguments of the AutoGen agents. It is <a href="https://github.com/microsoft/autogen" target="_blank" rel="noopener noreferrer" data-external="true">open-source</a>, and I like how their <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent/" target="_blank" rel="noopener noreferrer" data-external="true">docs</a> are structured.
When modifying the ConversableAgent class, you can change the code_execution_config argument in the <a href="https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent#__init__" target="_blank" rel="noopener noreferrer" data-external="true">__init__ method</a> to even disable the execution of the code.
<figure><figcaption>Fig. 4. Configuring code execution in AutoGen docs. Source (https://microsoft.github.io/autogen/docs/Getting-Started)</figcaption></figure>
You can also modify the way to execute code blocks, single code blocks, or function calls, by overriding <code>execute_code_blocks</code>, <code>run_code</code>, <code>and execute_function</code> methods respectively.
The code from AutoGen agents is executed locally via <code>use_docker</code> - Bool value of whether to use docker to execute the code, or str value of the docker image name to use or None when code execution is disabled.
<figure><figcaption>Fig. 5. Setting up use_docker in AutoGen docs. Source (https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent)</figcaption></figure>
<h2 id="potential-limitations">Potential limitations</h2>
Why would you want to keep a close eye on the execution of the code run locally via Docker?
As the <a href="https://docs.docker.com/engine/security/" target="_blank" rel="noopener noreferrer" data-external="true">Docker security article</a> mentions,
One primary risk with running Docker containers is that the default set of capabilities and mounts given to a container may provide incomplete isolation, either independently, or when used in combination with kernel vulnerabilities.
Granting autonomous AI tools access to executing code locally may be a challenge, especially for enterprise users.
<h4 id="alternative-solutions-may-be">Alternative solutions may be:</h4>
<ol>
<li>A better isolation of containers achieved by adding some barriers between them. However, containers like <a href="https://www.docker.com/" target="_blank" rel="noopener noreferrer" data-external="true">Docker</a> would still use shared resources as the kernel.</li>
<li>Another option is using <a href="https://docs.e2b.dev/" target="_blank" rel="noopener noreferrer" data-external="true">sandboxed cloud environments</a>. This provides security for running any code, starting processes, using the filesystem, and so on.</li>
</ol>
Another challenge with agent frameworks is scalability when the product acquires hundreds or thousands of users each developing their own AI applications, which would require thousands of containers.
This problem is solved for example by using cloud with <a href="https://docs.e2b.dev/?ref=framer-microsoft-s-autogen" target="_blank" rel="noopener noreferrer" data-external="true">E2B SDK</a>.
<h2 id="autogen-use-cases">AutoGen Use Cases</h2>
I found a few examples of how people try AutoGen. It seems like it is still experimenting with the framework mostly for fun purposes, but maybe a time shows whether AutoGen becomes regularly used for work purposes too.
<ol>
<li><a href="https://github.com/Poly186-AI-DAO/AutoGen-Snake-Game" target="_blank" rel="noopener noreferrer" data-external="true">Snake Game Development with AutoGen</a> - A project structured around a group chat setup where different agents collaborate to bring the snake game to life. <a href="https://www.youtube.com/watch?v=gnn1H4H81IY" target="_blank" rel="noopener noreferrer" data-external="true">YouTube video</a></li>
<li><a href="https://github.com/Andyinater/AutoGen_EnhancedAgents" target="_blank" rel="noopener noreferrer" data-external="true">Enhanced Agents</a> - Debuting with a MemoryEnabledAgent with improvements in context/token control, portability, and PnP functionality</li>
<li><a href="https://github.com/abhilashi/ai-explorations/blob/main/ai_scene_writer.py" target="_blank" rel="noopener noreferrer" data-external="true">Scene Writer</a> - A simulation of a fictional scene with AI screenwriters, a couple of assistant agents, and a critique</li>
<li><a href="https://twitter.com/oscarmoxon/status/1708603929011863871" target="_blank" rel="noopener noreferrer" data-external="true">AgentXP</a> - A self-improving agent that is eventually able to write itself</li>
<li><a href="https://colab.research.google.com/github/githubpradeep/notebooks/blob/main/autogen_meme_creator.ipynb" target="_blank" rel="noopener noreferrer" data-external="true">Meme creator</a></li>
<li><a href="https://github.com/amadad/agentcy" target="_blank" rel="noopener noreferrer" data-external="true">Agentcy</a> - An example with agents’ roles such as Account Manager, Strategist, Marketer, Researcher, or Designer</li>
</ol>
<h2 id="resources">Resources</h2>
</div></div>

E2B Blog & Customer Stories

Vaclav Mlejnsky

A guide to Microsoft's AutoGen framework for code-executing AI agents: how multi-agent conversations work, and how to run generated code securely with E2B.

Microsoft's AutoGen - A guide to code-executing agents

댓글