Multi-Agent Orchestration: Managers, Workers, and Pipelines

In Part 1 you saw how a single ReAct loop turns a model into an agent. One agent with good tools can do a lot: classify an email, draft a reply and book a meeting in the same run. So start with one agent. Split the work across several only when the sub-tasks are truly independent, need different tools or context, or will not fit in one context window. And when you do split, you need a plan to keep the agents from getting stuck.

Multi-agent orchestration is how you assign tasks to specialized agents, coordinate their work, and stop infinite retries. This article covers when not to split, sequential pipelines, manager-worker coordination, state passing, and the feedback loops that keep a team of agents from grinding to a halt. Everything is grounded in Mailmind, an AI assistant that lives in your inbox: it reads email, drafts replies, schedules meetings, tracks orders and refunds, and files receipts.

One failure deserves a warning up front. A manager that simply tells a worker "try again" can trap the whole pipeline in a loop of identical rejections. Better prompts do not fix it. A stagnation detector that notices the worker keeps giving the same wrong answer does.

When should you not split the work?

More agents cost more. In Anthropic's write-up How we built our multi-agent research system (June 2025), agents typically used about 4 times more tokens than chat interactions, and multi-agent systems about 15 times more. The same post says that domains where all agents need to share the same context, or where agents depend on each other a lot, are not a good fit for multi-agent systems today, and that most coding tasks have fewer truly parallel pieces than research does.

Research on failures points the same way. A study of traces from seven popular multi-agent frameworks found that their performance gains on popular benchmarks are often minimal, and sorted 14 failure modes into three groups: system design issues, misalignment between agents, and weak task verification (Cemri et al. 2025, Why Do Multi-Agent LLM Systems Fail?).

So the default is one agent with good tools, which matches the advice in Anthropic's guide Building effective agents (December 2024): find the simplest solution possible and add complexity only when it is needed. Reach for several agents when at least one of these is true:

• The sub-tasks are independent, so running them in parallel saves real time.
• They need different tools, permissions or context, and mixing them in one prompt makes each job worse.
• The work is too big for one context window, so each agent must read its own slice.

Everything below assumes you have passed that test.

How does a sequential pipeline work, and when should you use it?

A sequential pipeline runs agents in a fixed order. One agent's output becomes the next agent's input. This simple pattern shines when each step depends directly on the previous one. Anthropic's guide calls it prompt chaining.

Think of a restaurant kitchen. Before a dish reaches the customer, it passes through prep, cooking, assembly, and an expeditor who checks the plate. Each station builds on the work done before it. You cannot plate a steak before the grill finishes cooking it.

Mailmind uses a three-step pipeline for drafting important replies. First a triage agent reads the thread and assigns one of Mailmind's five categories: urgent, newsletter, receipt, meeting request, or other. If the thread is urgent, it flows into a drafting agent that composes a reply from the thread and the triage result. Finally a review agent checks the draft for tone, completeness and factual errors. The reviewer can reject the draft and send it back to the drafting agent with specific feedback (the same guide calls a generate and review loop like this evaluator-optimizer).

State passes directly from one agent to the next. The triage agent's output is a structured object that the drafting agent consumes. The reviewer receives both the draft and the triage result. This straight-line flow makes debugging easy: you can trace a bad reply back through exactly three agents. The tradeoff is rigidity. A pipeline's path is fixed in advance, apart from explicit loops like the review step.

The next pattern handles tasks that do not depend on each other at all.

How does a manager-worker pattern parallelize independent work?

A manager-worker setup hands independent sub-tasks to several workers at once. The manager collects the results, evaluates them, and decides whether to accept, retry, or escalate. This pattern turns a batch of unrelated jobs into a single coordinated run.

Picture a restaurant during the morning prep. The head chef tells one cook to chop vegetables, another to make soup stock, and a third to prepare sauces. All three work at once. The head chef inspects each result and deals with problems as they come.

Mailmind's morning brief is a good example. A manager agent receives a batch of new emails. It starts three workers:

• Worker one classifies each email by urgency and category.
• Worker two writes a two-sentence summary of every thread from the last 24 hours.
• Worker three extracts explicit action items: who needs a reply, who mentioned a deadline, who asked for a document.

All three run in parallel because their tasks are independent. The manager collects the raw outputs, removes duplicate action items, and combines everything into one morning brief for the user. Anthropic's guide calls a fixed split like this parallelization (sectioning), and the version where a central model decides the sub-tasks on the fly orchestrator-workers.

Parallel execution cuts waiting time. Instead of three calls in a row, the user waits for only the slowest worker plus the manager's combining step. The risk is that the manager must be good enough to combine or reject conflicting outputs. A vague manager that simply pastes raw text together creates an unreadable brief.

Next we will look at how agents share information across these patterns.

What are the three patterns for passing state between agents?

Agents must share context. Without a clear way to pass state, information gets lost and the pipeline breaks. There are three practical approaches, each with its own tradeoffs.

Direct passing is the simplest: agent A's output becomes agent B's input. This works well in short pipelines like Mailmind's triage, draft and review flow. The chain is easy to audit. As the team grows, direct passing becomes unwieldy. Every new agent needs to know the full format of the previous one, and you end up with brittle, tightly coupled steps.

A shared memory store lets all agents read and write a central store keyed by a thread_id. Mailmind uses this when several agents contribute to the same email thread over time. A scheduler writes a proposed meeting time, a drafting agent reads the last message, and a follow-up agent checks whether the user replied. The store acts as a blackboard. The downside is that any agent can corrupt the shared state, and recovering from bad writes is hard. Agent Memory: The Four Kinds and How They Fit Together covers memory stores in depth.

A context object is a structured record that gets filled in at each stage. Mailmind defines a ThreadContext that starts with the original email. The triage agent adds category and intent. The drafting agent adds draft_body and confidence. The review agent stamps approved or rejection_reason. Each stage reads the fields it needs and writes new ones without touching anything else. This keeps the format explicit and auditable. It scales better than raw direct passing and avoids the chaos of one shared blob.

Choosing among these comes down to how many agents you have, how often they run, and how much you need to debug a single run.

How state moves sets the stage for the biggest danger in orchestration: the loop that never ends.

How do you stop an agent team from retrying forever?

An infinite retry loop happens when a manager rejects a worker's output, the worker retries, produces the same thing, and gets rejected again. Work keeps happening, but nothing moves forward. It wastes time, tokens and the user's patience.

People sometimes call this a deadlock, but a deadlock is a different bug. In a deadlock two agents each wait for the other, so nothing happens at all: a scheduling agent waits for the drafter's reply text while the drafter waits for the scheduler's time slot. A timeout on every wait fixes that one. The retry loop is the opposite: plenty of activity, no progress (engineers call that a livelock). Three mechanisms stop it.

A max retries cap is the blunt instrument. After N attempts, the manager must decide: accept the best output, fall back to a simpler agent, or flag it for a human. Mailmind caps retries at three for any worker. A cap alone is not enough; if every retry produces the same failure, you burn all attempts with no progress.

Stagnation detection catches the same-answer problem early. The system stores a fingerprint of each output, a hash of the text after normalizing it. If a fingerprint comes back, the loop stops at once. Normalizing matters because a hash only catches exact repeats: change one character and the hash is completely different. So before hashing, lowercase the text, collapse the spaces, and sort the keys if the output is JSON. To catch answers that are worded differently but mean the same thing, compare their embeddings instead. Mailmind uses this in its scheduling flow. When the worker keeps returning "no available time" without checking the calendar, the detector stops the loop after the second identical answer instead of waiting for the cap.

Progressive fallback adds a ladder of degraded options. First, the manager retries with specific feedback. Second, it calls a simpler fallback, for example a rule-based scheduler instead of an LLM (large language model). Finally, it escalates to the user with the best attempt and a clear explanation. Mailmind's feedback is precise: "You missed the flight confirmation in the thread. Re-scan messages from airlines." Generic "try again" instructions make retries produce the same result.

Here is the whole ladder in a few lines of Python-style pseudo-code:

def run_with_retries(worker, task, max_retries=3):
    seen, feedback, output = set(), None, None
    for attempt in range(max_retries):
        output = worker(task, feedback)
        verdict = manager_review(task, output)
        if verdict.accepted:
            return output
        fingerprint = sha256(normalize(output))  # lowercase, collapse spaces, sort JSON keys
        if fingerprint in seen:                 # the same answer came back: stop early
            break
        seen.add(fingerprint)
        feedback = verdict.specific_feedback     # say what is wrong, not "try again"
    fallback = rule_based_fallback(task)
    return fallback if fallback else escalate_to_user(task, best_attempt=output)

These three safeguards turn a fragile pipeline into one that degrades gracefully. They also set up the next question: which tasks can run at the same time?

When do you run agents in parallel versus sequentially?

Run agents in parallel when their work does not depend on each other. Run them in sequence when one task needs the answer from the previous one. This sounds obvious, but misjudging dependencies is a common orchestration bug.

In the morning brief, classification, summarization and action-item extraction are independent. They can run in parallel. If you made classification wait for the summaries, you would waste time, because classification does not need them. Mailmind's reply pipeline is the opposite: the drafting agent must know the triage result before it writes a response.

The rule is simple: draw a dependency graph. Each edge forces one step to wait for another. Steps with no edge between them can fan out to parallel workers. Ignoring this graph leaves speed on the table or, worse, creates races where one agent reads stale data from a shared store.

With parallelism decided, the next question is who does what.

When does specialization beat a single all-purpose agent?

A single agent whose prompt mixes triage, drafting, summarizing and scheduling can do worse than focused agents, because unrelated instructions and context compete for its attention. Specialization helps when the jobs need different context or tools: each agent's prompt, tools and context can be tuned to one job. The price is more model calls and more hand-offs that can fail, so specialize where the jobs really differ, not by reflex.

Mailmind's triage agent only needs the email subject, sender and first few sentences. Its prompt is short and direct: "Classify this email as urgent, newsletter, receipt, meeting request, or other." The drafting agent receives the full thread and a different instruction: "Draft a polite reply that addresses the sender's questions." One agent doing both would need a long prompt covering every task and would carry the whole thread even when it only needed the subject line.

Specialization also makes debugging easier. When the drafting agent writes a bad reply, you know which prompt to fix. When the triage agent misclassifies, you adjust its examples (the technique in Few-Shot: Examples Beat Instructions) without risking the drafts. Single responsibility applies to agents as much as it applies to classes in software design.

Beyond these core patterns, real systems layer on more coordination strategies.

What lies beyond pipelines and managers?

The patterns so far solve most practical problems. As the workload grows, a few more become useful.

Hierarchical orchestration introduces managers of managers. Mailmind could have a weekly-digest manager that delegates to a Monday-morning manager and a Friday-summary manager. Each of those runs its own workers. The top-level manager combines the day-level reports into one weekly email.

Event-driven orchestration triggers agents on events rather than on a fixed schedule. A new email arriving in Mailmind fires a triage agent. If triage marks it urgent, that event triggers a drafting agent and a scheduling agent. No central conductor calls each step; agents react to changes. Work starts the moment the event arrives.

Blackboard architecture goes further: agents share a workspace and pick their own tasks. A new email appears on the board. A triage agent notices it has no category and claims it. Later, an unhandled action item sits on the board, and any free scheduling agent can pick it up. There is no single coordinator to fail, though the shared board itself must stay available and consistent. It is also harder to debug, because the order of events is not planned in advance.

Choreography versus orchestration frames the control decision. Orchestration uses a central conductor that knows the full workflow, so you can log every step and reason about failures. Choreography has no central brain; agents talk to each other through events. It copes better with partial failures but is far harder to trace. A common split, and the one Mailmind uses, is orchestration for critical paths and choreography for background tasks like receipt filing.

Quick Reference

Frequently Asked Questions

Q: When does a manager-worker pattern add more overhead than it saves? A manager adds at least one extra model call plus coordination code. Parallelism pays only when the slowest worker plus the manager's calls take less time than running the tasks one after another, and the token bill is worth it. For two small tasks, a plain sequence is often faster and cheaper. Measure it on your own workload rather than trusting a rule of thumb.

Q: How do you decide between a context object and a shared memory store? Use a context object when you have a fixed set of fields that get filled at known stages. You can type the record and validate it. Use a shared memory store when agents run at unpredictable times and need to read arbitrary keys. The store is more flexible and much harder to keep consistent.

Q: Can a worker be both a specialized agent and a manager itself? Yes. A worker that receives a complex task can start its own sub-workers as a local manager. That is hierarchical orchestration. The top-level manager only sees the combined result. You pay for the nested model calls, so only do it when the sub-task truly needs its own breakdown.

Q: What is the simplest code pattern for stagnation detection? Normalize the worker's output (lowercase, collapse whitespace, sort JSON keys), hash it with a fast hash such as SHA-256, and keep the hashes from earlier attempts. If a new hash matches one you have seen, stop the loop. A hash catches only exact repeats: one changed character gives a completely different hash. That is why you normalize first, and why you compare embeddings when you need to catch answers that are reworded but mean the same thing, as the Test yourself answer below does.

Q: Why not always use event-driven orchestration for everything? Event-driven systems start work quickly but are much harder to debug. When a bad reply reaches the user, you have to trace a chain of events with no single execution record. For user-facing flows like drafting an email, Mailmind uses orchestration so every step appears in one trace. For background jobs like receipt filing, events are fine.

Test yourself

Mailmind's manager keeps rejecting the scheduling worker's output. After three retries, the pipeline is stuck, and the user sees no meeting invitation. Your logs show that each retry produces a slightly different but still unacceptable suggestion. The manager's feedback is "Schedule a time that works for both parties."

What is the most likely cause, and how would you unblock and prevent this?

Answer: The feedback lacks specifics, and the worker has no way to check availability. The manager says "a time that works for both parties", but the scheduling worker cannot see the calendar, so it guesses, and each attempt is a different wrong time. A plain hash check misses this, because the outputs differ. First, unblock: take the most plausible suggestion, show it to the user flagged as unconfirmed, and let the other workers finish. Then fix the root cause. Give the scheduling worker a calendar tool that returns free slots, and let the manager use the same tool to verify. Rewrite the feedback with concrete constraints: "3 PM clashes with the user's board meeting. Propose the next free 30-minute slot tomorrow morning." To prevent recurrence, extend the stagnation detector to compare the meaning of outputs, not just exact hashes: if consecutive answers are nearly identical by embedding similarity (with a threshold you tune on past runs), stop and escalate.

Next: Agent Memory: The Four Kinds and How They Fit Together. It covers how an agent remembers across calls and sessions: working memory, episodic memory, semantic memory and procedural memory.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• ReAct: Synergizing Reasoning and Acting in Language Models
• How we built our multi-agent research system
• Building effective agents
• Why Do Multi-Agent LLM Systems Fail?