How to Build a Multi-Agent AI System from Scratch
Zarif
You build one AI agent and it works. So you add a second to check its work, a third to search the web, a fourth to format the output. A week later, nobody can tell which agent broke the result, or why.
A multi-agent system is a set of AI agents that each do one job and pass work to each other. It can do things one agent can't. It can also fail in ways one agent never would. This guide walks through building one in seven steps, from deciding whether you need it to getting it ready for real use.
When you actually need more than one agent
Start with one. A single agent with the right tools can do more than you'd expect. OpenAI's guide to building agents says to push a single agent as far as it goes before adding more. Every extra agent is another thing to coordinate, another place to fail, and another thing to debug.
You need more than one when the jobs pull in opposite directions inside one prompt. A research agent should roam and explore. A code-writing agent should be precise and do the same thing every time. A review agent should be suspicious. Ask one agent to be all three at once and it does each one worse.
This is becoming common. Gartner projects that 40% of enterprise applications will include task-specific AI agents by the end of 2026, up from under 5% in 2025.
Step 1: Give each agent one job
Before you write any code, write down what each agent does. This is the step people skip, and it's the one that decides whether the system holds together.
For each agent, write three things: its job, its tools, and what it's not allowed to do. That last part matters as much as the first two. An agent that can touch everything will eventually touch something you didn't mean it to.
Say you're building a system that researches and writes articles. You might have three agents:
- A research agent that searches the web and pulls data. Tools: web search and scraping.
- A writer agent that drafts from the research notes. Tools: text generation and formatting.
- A quality agent that checks drafts against a checklist. Tools: grammar checking and fact checking.
Each one stays in its lane. The research agent never writes. The writer never searches. The quality agent only critiques. It never creates.
Write your agent definitions in a plain YAML or JSON config file before you start coding. It forces you to think through responsibilities, keeps scope from creeping, and makes it easy to swap an agent later without touching the rest of the system.
Step 2: Choose how the agents work together
How the agents are wired together matters more here than which model you pick. Six patterns come up again and again.
Assembly line (sequential pipeline). Agent A finishes and hands off to B, which hands off to C, always in the same order. It's the simplest pattern and the easiest to debug, because you always know where the data came from. Use it when the work has clear stages, like research, then write, then review.
Manager and specialists (supervisor). One agent plans the work, hands pieces to specialists, and decides when it's done. This is the most common place to start, and it suits tightly scoped jobs like financial analysis or compliance checks. The catch: every decision goes through the manager, and it gets slow as the work grows.
Split and combine (parallel fan-out). Several agents work on different parts of the same task at once. A code review system might send the code to a style agent, a security agent and a performance agent at the same time, then pass all three reports to one agent that writes the final verdict. When the parts don't depend on each other, this finishes faster.
Maker and checker (generator and critic). One agent creates. Another grades it. They loop until the work clears a bar. Use it when quality really matters, like generating code that has to pass tests, or writing that has to survive a fact check.
Shared whiteboard (blackboard). Every agent can read and write to one shared workspace, adding pieces of the answer as they go. Nobody routes the work through a manager. It suits open-ended, creative tasks where you can't predict the right order in advance.
Human sign-off (human in the loop). The system pauses and waits for a person to approve before anything risky, like deploying code, moving money or emailing someone outside the company.
Pick the pattern that fits the shape of the work, not the framework you like. Fixed stages get an assembly line. Coordinated but scoped work gets a manager. Independent pieces get split and combine. Work where quality matters most gets a maker and checker.
| Pattern | Best For | Complexity | Failure Mode |
|---|---|---|---|
| Sequential Pipeline | Linear stage-based workflows | Low | Cascading errors between stages |
| Supervisor/Subagents | Coordinated, scoped tasks | Medium | Supervisor bottleneck |
| Parallel Fan-Out | Independent subtasks | Medium | Output aggregation conflicts |
| Generator/Critic | Quality-critical outputs | Medium | Infinite refinement loops |
| Blackboard | Creative, exploratory work | High | Coordination chaos without constraints |
| Human-in-the-Loop | High-stakes decisions | Low-Medium | Approval bottleneck at scale |
Step 3: Pick your framework
Three frameworks are worth comparing in 2026. All three are free and open source under the MIT license: CrewAI, LangGraph, and Microsoft Agent Framework. The real costs are the model usage and the time it takes you to build and test.
CrewAI is built like an org chart. You give each agent a role, a goal and a backstory, then assign it tasks. It's the fastest way from an idea to something that runs. Start here if your workflow mostly goes in a straight line, and if you want people who don't code to be able to read and edit the agent definitions.
LangGraph treats the system as a flowchart. Each agent or step is a box, and you draw the arrows between them, including branches, loops and "if this, go there" rules. Its companion tool, LangSmith, shows a step-by-step record of each run with token counts per step, and lets you rerun a failed run with changed inputs from its interface. Reach for LangGraph when the flow gets complicated, with many decision points or work running side by side.
Microsoft Agent Framework replaces AutoGen, which earlier versions of this guide recommended. AutoGen's README now says it's in maintenance mode, gets no new features, and tells new users to start with Agent Framework. Agent Framework reached 1.0 for Python and .NET on April 3, 2026.
Microsoft's overview describes it as AutoGen's agent building blocks plus Semantic Kernel's enterprise features, with flowchart-style workflows for wiring agents together explicitly. Pick it when you want chatty agents and a fixed workflow in one framework, or when your team already runs on Azure and .NET. If you have AutoGen code, Microsoft publishes a migration guide.
| Framework | Architecture Style | Best For | Learning Curve |
|---|---|---|---|
| CrewAI | Role-based teams | Business workflows, fast prototyping | Low |
| LangGraph | Graph-based workflows | Complex pipelines, conditional logic | Medium-High |
| Microsoft Agent Framework | Agents plus graph-based workflows | Conversational agents with explicit orchestration, .NET or Azure stacks | Medium |
For your first multi-agent system, start with CrewAI unless you already know you need flowchart-style control. You can move to LangGraph later, once you know how your agents actually need to coordinate.
Step 4: Make every handoff checkable
Picture the research agent finishing its work and passing notes to the writer. The notes are missing a field, or a list came back as one long sentence. The writer doesn't complain. It writes a confident article from bad notes, and the quality agent grades the writing, not the research.
That's why the handoffs between agents matter most. One bad message early on spreads through every step after it.
Define the exact shape of every message. Models don't guess what you meant. They follow what you spell out. So write down exactly what each agent sends and receives, using Pydantic models, JSON Schema, or whatever checking your framework has built in. This is called a typed schema. Skip it, and sooner or later one agent passes broken data that trips up the next.
Wrap each handoff in a package. When agent A passes work to agent B, send more than the raw model output. Include the result, a note on what was done, a confidence score where it makes sense, and whatever context B needs.
Keep one place to check progress. Even in a simple assembly line, you want one spot where any agent can see where the overall task stands. Redis works for simple cases. If the state needs to survive between sessions, a database that saves numbered snapshots lets you replay a failed run and see what happened.
Here's a small typed handoff:
from pydantic import BaseModel
from typing import List, Optional
class ResearchOutput(BaseModel):
query: str
sources: List[str]
key_findings: List[str]
confidence: float
gaps_identified: Optional[List[str]] = None
class WriterInput(BaseModel):
research: ResearchOutput
target_word_count: int
tone: str
outline: List[str]
When the research agent finishes, it produces a ResearchOutput. The writer gets a WriterInput, which wraps that research with a few more instructions. If the data doesn't match the shape, you find out right at the handoff, not three agents later.
Step 5: Add limits and safety checks
The failure to plan for looks like this. One agent produces bad output. The next agent builds on it with full confidence. By the time anyone notices, the whole chain has produced something completely wrong. This is called a cascade failure, and a single agent can't have one.
Give each agent only the tools it needs. The research agent needs web search, not write access to your database. The writer needs text generation, not access to outside APIs. It's the old security rule of least privilege, applied to agents.
Check what's inside every handoff, not just its shape. A research agent that returns zero findings with high confidence passes the shape check and is still wrong. Add checks that the content makes sense.
Stop retrying after a few failures. If an agent fails three times in a row, stop and hand the work to a backup agent or a person. Retrying forever burns money and never produces a better answer. Engineers call this a circuit breaker.
Set token and cost limits for each agent. An agent stuck in a loop can run up a large bill before anyone looks. Put a hard cap on tokens per turn and on total cost per task.
Cap the number of revision rounds. The maker-and-checker pattern is the one most likely to loop forever. Give the checker a passing score and the loop a maximum number of rounds. When it hits the cap, hand the draft to a person.
Never give a multi-agent system unchecked access to production APIs or databases during development. Run it in a sandbox with read-only access until you've checked its behavior across a wide, varied set of test cases.
Step 6: Test with realistic inputs before you launch
Testing one agent means checking its output. Testing several means also checking that they work together: that the handoffs hold up on odd inputs, and that the system recovers when one part fails.
Test in layers. First test each agent alone, to make sure it does its own job. Then test pairs, to make sure the handoffs work. Then run the whole thing end to end on realistic, messy inputs.
Roll it out in pieces. Don't launch the whole system at once. Start with one agent doing the core job. Add the second once the first is steady. Add coordination a piece at a time.
Log everything once it's live. Record every message between agents, every tool call and every change of state. Something will go wrong, and you'll need the full record to find out what. LangSmith, Langfuse and Arize Phoenix are built for this. The production deployment guide covers what to log, what to alert on, and how to replay a failed run.
Step 7: Get from demo to real use
A demo that works once is not a system people can rely on. Three things close that gap.
Memory between runs. Agents need to remember earlier runs. A research agent that searches a topic it already covered wastes time and money. Use a vector database (Pinecone, Weaviate, Qdrant) for memory the agent searches by meaning, and a simple key-value store for task status.
Use smaller models where you can. Not every agent needs your most capable model. The manager agent deciding where work goes can often run on something smaller and faster. An agent that only checks a message's shape needs very little. Match the model to the job and you spend less without hurting the output.
Plan for parts breaking. When one agent fails, or an outside API goes down, the whole system shouldn't crash. Give it a backup path. If the research agent can't reach the web, it uses cached results and marks the output as possibly out of date. If the quality agent is overloaded, queue the work instead of dropping it.
One mix that suits real use: fast specialists work side by side, while a slower, more careful agent checks their results every so often and decides whether to keep going or stop.
Protocols that let agents and tools connect
Two open protocols matter for connecting agents to tools and to each other. Both now sit under the Linux Foundation rather than a single company.
Model Context Protocol (MCP), created by Anthropic and donated to the Agentic AI Foundation in December 2025, is a standard way for agents to reach tools and data. Instead of building a custom connection for every API, you wrap the tool once as an MCP server, and any agent that speaks MCP can use it. What is MCP explains the pieces.
Agent2Agent (A2A), created by Google and moved to the Linux Foundation in June 2025, is for agents talking to other agents. Agents built on different frameworks, running in different places, can find each other and pass tasks back and forth without one manager owning every message. IBM's Agent Communication Protocol (ACP), which earlier versions of this guide listed separately, merged into A2A in August 2025.
You don't need both on day one. Start with MCP, since every agent needs tools. Add A2A when your agents run as separate services, especially ones owned by different teams or companies.
Once the system works on your machine, the production deployment guide covers what changes when real users depend on it. For memory that lasts between runs, see building persistent agent memory.
