<div id="readability-page-1" class="page"><div>We talked with <a href="https://twitter.com/pelaseyed" target="_blank" rel="noopener noreferrer" data-external="true">Ismail Pelaseyed</a> about AI agents, their current challenges, and potential solutions.
Ismail is the founder of <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> - a framework for web or no-code developers. Being active for around two months, Superagent already gained around 2000 users and over 1500 agents running in the production every day. It is a completely <a href="https://github.com/homanp/superagent" target="_blank" rel="noopener noreferrer" data-external="true">open-source</a> platform, with a cloud version free to use.
<h2 id="users-and-use-cases">Users and Use Cases</h2>
Superagent tries to approach no-code developers to integrate agents into their applications.
They have two main types of agents built with Superagent:
1. Chatbots - Retrieval types of agents, used to Q&amp;A over structured or unstructured documents.
2. Action-based assistant - Run either through a chat interface or just through an API.
‍
<figure></figure>
‍
<h2 id="architecture-and-techstack">Architecture and Techstack</h2>
<a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> has a database layer on top of the agents' infrastructure. When a user creates an agent, they create an entry into that database with a configuration of how an agent should run. They have an API service hosted on <a href="http://render.com/" target="_blank" rel="noopener noreferrer" data-external="true">render.com</a> which picks the configuration depending on the API call, creates the agent on the fly, and runs it.
They are currently using <a href="https://github.com/e2b-dev/awesome-sdks-for-ai-agents/tree/main#langchain" target="_blank" rel="noopener noreferrer" data-external="true">Langchain</a>, which has a very prompt-based agent infrastructure, to call various LLMs and <a href="https://github.com/e2b-dev/awesome-sdks-for-ai-agents/tree/main#llamaindex" target="_blank" rel="noopener noreferrer" data-external="true">LlamaIndex</a> to ingest various data.
They are deep into replacing currently used frameworks, moving away from the core Langchain Agent-LLM loop (e.g. <a href="https://arxiv.org/abs/2210.03629" target="_blank" rel="noopener noreferrer" data-external="true">ReAct</a>, <a href="https://www.promptingguide.ai/techniques/cot" target="_blank" rel="noopener noreferrer" data-external="true">Chain of Thought</a>).
The <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> team has a wrapper around some individual types of agents which they have tested and are ready to be run in production. These agents can then be easily configured without the need to consider various components that are involved in running agents, mainly:
<ul>
<li>Memory</li>
<li>Vectorizations</li>
<li>The agent's acting and whole logic.</li>
</ul>
<h2 id="current-challenges">Current Challenges</h2>
The agent architecture is still in its early stages, making configuration for specific use cases challenging.
“Many people attempt to create agents but often give up due to its inability to meet their desired functionality, " says Ismail. “Our goal is to develop agents that are not only demo-worthy but also practical and useful for business purposes.”
Ismail has been noticing the many and many specific ways in which people want to use their agents for the same use case. “People really do have specific ways of working on the same thing, which therefore requires deep developers’ understanding.”
Creating efficient agents that work effectively in production for specific use cases demands significant expertise at this stage. Especially with high pressure being placed on ensuring user-friendliness for no-code users.
With that come the challenges of debugging, tracing, monitoring, and observability of the agents.
<h2 id="debugging-the-agents">Debugging the Agents</h2>
The current process of solving a bug is very simple. If an agent instance fails, end users go to the <a href="https://discord.com/invite/mhmJUTjW4b" target="_blank" rel="noopener noreferrer" data-external="true">Superagent Discord</a>. There they describe the bug, for example, that a retrieval agent did not use a document to make a query, but hallucinated instead.
“It is not the actual software that is failing, it is mostly the setup that the user has. It can be prompt, it can be the document being ingested, or anything else. That is what we are trying to debug.”
Ismail emphasizes looking for a powerful debugging solution that would be user-centric. It is not the developer team, but the end users who are not currently able to debug the agent themselves.
<h2 id="tracing-and-observability">Tracing and Observability</h2>
The tracing currently lacks WHY and HOW. That is, the user sees that an agent instance failed, but doesn't understand WHY exactly it happened and HOW to fix the problem. This problem can be solved on an individual level but will grow big when Superagent gains even more regular users and bigger customers.
<figure><figcaption>Monitoring agents' logs.&nbsp;Source: Superagent (https://www.superagent.sh/) dashboard</figcaption></figure>

To overcome the challenges of tracing and debugging, the Superagent team can envision themselves using an equivalent of <a href="https://sentry.io/welcome/" target="_blank" rel="noopener noreferrer" data-external="true">Sentry</a> (a software error tracking and monitoring platform), which could provide insights into the origin of a bug. They have tried <a href="https://github.com/e2b-dev/awesome-sdks-for-ai-agents/tree/main#langsmith" target="_blank" rel="noopener noreferrer" data-external="true">Langsmith</a> (a web-based GUI to test and monitor calls from LLM apps), which, however, won't give the root cause of the bug and possible solutions.
“We need a tool that would also tell a user what they should do with the bug,” says Ismail.
The <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> team has a UI where end users can see agents’ runs. It is very similar to Langsmith or any other type of logging framework.
When a user faces a problem, the <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> team can log into the system, see what the agent tries to do in each step, and then visualize the log. This is in principle similar to regular server logs visualizations.
<h2 id="conclusion">Conclusion</h2>
Right now the <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> team focuses on re-doing their whole infrastructure, aiming to run the agents in a separate queue, and creating a queue system, instead of running them on the fly.
“We would go for big changes in our infrastructure if it solves the discussed problems,” says Ismail.
Without a doubt, in the very young agents space (<a href="https://news.agpt.co/" target="_blank" rel="noopener noreferrer" data-external="true">AutoGPT</a>, the first well-known agent, launched only in March 2023), <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> and other agents are steadily gaining traction among end users. Despite the current drawbacks, this shows the real demand for agents.
If you want to try <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Superagent</a> yourself, check the <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">web</a>, <a href="https://github.com/homanp/superagent" target="_blank" rel="noopener noreferrer" data-external="true">GitHub</a>, or join the <a href="https://www.superagent.sh/" target="_blank" rel="noopener noreferrer" data-external="true">Discord community</a>.</div></div>

E2B Blog & Customer Stories

Vaclav Mlejnsky

We talked with Ismail Pelaseyed, the founder of Superagent, about AI agents, their current challenges, and potential solutions.

Discussing agents' tracing, observability, and debugging with Ismail Pelaseyed, the founder of Superagent

댓글