The development of large language models is entering a new era, in which prompting is no longer the main way to get an AI to do something useful. The first successes of AI-assisted engineering were achieved through prompting, but the current trends in AI engineering are moving towards developing multi-agent systems, in which several language model agents work together to accomplish some task. Collaboration between multiple language model agents requires a careful design of memory, decomposition, and coordination of information between language model agents to perform more involved engineering, scientific, and enterprise tasks. The researchers have identified three key ideas that are critical to building autonomous agents, namely, memory, decomposition, and coordination. First, memory management is of particular importance in training large language models as a single agent due to limited context window sizes. At the same time, an agent that performs some involved task over a long period of time cannot rely only on the context provided through the chat interface. Modern multi-agent systems address this challenge through short- and long-term memory mechanisms. Short-term or working memory allows agents to perform logical reasoning over the chat history and tool execution results during each turn. Long-term memory, in turn, is stored in external databases using vector embeddings or other encodings. To manage long-term memory, an agent must be able to store and organize the information for later use and retrieval. If the memory is not managed properly, the agent will either fail to perform the task because it does not have the required information in its context, or, on the contrary, it will have too much information, including irrelevant data and hallucinations from previous interactions. Further, to solve complex tasks, multi-agent systems need to have the ability to decompose them into simpler subtasks. This capability is especially important for multi-agent systems since providing the agent with all the necessary information and instructions in one prompt, as is done when training large language models, is often impossible when dealing with complex tasks. The process of breaking down a task into subtasks requires understanding what needs to be done, as well as what the agent can actually do. Second, the task decomposition must be dynamic, allowing the agent to analyze the result of the executed subtask and change the plan if a mistake was made or the selected method did not work. Third, the agent must be able to reflect on the execution of the previous task when decomposing the task. Finally, the most critical aspect of building autonomous agents is coordination, that is, communication between agents. Coordinating multiple agents is a complex process that involves establishing the necessary rules for communication between agents and developing strategies for making joint decisions. Agents can communicate using peer negotiation strategies, voting, leader-follower strategies, and other concepts from game theory. The use of game-theoretic approaches allows decentralized teams of agents to coordinate actions and make decisions, for example, on what sources to use, what facts are true, and which arguments are the best. This, in turn, allows such multi-agent systems to avoid hallucination problems and achieve higher solution quality.
Source: https://nordicapis.com/what-is-a-multi-agent-system/
Researchers have proposed a variety of ways to evaluate the capabilities of autonomous agents. However, traditional natural language processing language model benchmarks are not suitable for measuring the performance of such systems. It is necessary to develop new evaluation methods that would measure the performance of various approaches to building autonomous agents, for example, by comparing the number of tasks performed, their complexity, and the amount of tokens used.
Sources
https://arxiv.org/html/2402.03578v1 https://arxiv.org/abs/2502.14743