Chapter 9 of 13
Multi-agent systems
Five agents look like more capability. Usually they are more failure modes. The pattern that holds up in production is orchestrator-worker: one agent owns the goal, splits it into independent subtasks, hands each to a worker that sees only its own slice, and merges the results itself. The first reason to split is not speed, it is context isolation. In this lab each worker carries a prompt of at most 85 characters where a single agent carrying all three reviews carries 164, and at real scale that is the difference between a focused prompt and a transcript the model loses the thread in. Parallelism is the second prize, not the first. The cost is coordination, and teams reach for it far too early. If task B needs task A's output, that is a handoff, it is sequential by definition, and no fan-out helps it. If one agent with two better tools can do the job, it beats five agents every time, because there is no merge step to get wrong. This lab builds the orchestrator, then breaks it the way real systems break, with shared mutable state.
The lab: read it, then run it
#!/usr/bin/env python3
"""
LAB AE9: Multi-agent systems. Orchestrator, workers, and the shared-state race.
The dominant real multi-agent shape is orchestrator-worker: one agent owns the
goal and splits it into independent subtasks, each worker handles exactly one,
and the orchestrator merges the results. The first reason to split is CONTEXT
ISOLATION, not speed: each worker only ever sees its own slice, so its prompt
stays small while a single agent doing all three would carry all three slices at
once. This lab measures that, then reproduces the failure that bites every team
that shares mutable state between workers, and fixes it.
Run: python3 modules/academy-content/labs/agentic-engineering/ae9-orchestrator-workers.py
"""
import sys, os
_cands = [os.path.join(os.path.dirname(__file__), "..") if "__file__" in globals() else None,
os.path.join(os.getcwd(), "..", "labs"), os.path.join(os.getcwd(), "labs")]
for _c in _cands:
if _c and os.path.exists(os.path.join(_c, "academy_llm.py")):
sys.path.insert(0, os.path.abspath(_c)); break
from academy_llm import complete
# ── The work: three independent reviews. Independent is the precondition for ──
# fanning out at all. Anything where task B needs task A's output is a HANDOFF,
# which is sequential by definition, and no amount of parallelism helps it.
TASKS = [
("r1", "I love this laptop, the screen is excellent"),
("r2", "the battery is terrible and the fans are awful"),
("r3", "it arrived on tuesday in a box"),
]
TRUTH = {"r1": "positive", "r2": "negative", "r3": "neutral"}
def worker(task_id, text, state):
"""One worker agent. It sees ONE review, never the other two. `state` is the
dict it is told to report into, which is the whole point of this lab."""
prompt = "Classify the sentiment of this review: %s" % text
state["result"] = complete(prompt) # <-- the bug, kept on purpose
state["by_id"][task_id] = complete(prompt) # <-- the fix, same call
return len(prompt)
# ── STEP 1: context isolation, measured. ──────────────────────────────────────
solo_prompt = "Classify the sentiment of each review: " + " | ".join(t for _, t in TASKS)
print("STEP 1: context isolation")
worker_sizes = [len("Classify the sentiment of this review: %s" % t) for _, t in TASKS]
print(" one agent, all three reviews : %d prompt chars" % len(solo_prompt))
print(" each worker, its own review : %s chars (max %d)" % (worker_sizes, max(worker_sizes)))
isolated = max(worker_sizes) < len(solo_prompt)
print(" worker context is smaller : %s" % isolated)
# ── STEP 2: the hazard. Every worker writes the SAME key. ─────────────────────
print("")
print("STEP 2: shared mutable state, all workers writing state['result']")
shared = {"result": None, "by_id": {}}
for task_id, text in TASKS:
worker(task_id, text, shared)
print(" after %s -> state['result'] = %r" % (task_id, shared["result"]))
racy = [shared["result"]]
print(" aggregated results survived : %d of %d" % (len(racy), len(TASKS)))
# ── STEP 3: the fix. One namespace per worker, merged by the orchestrator. ────
print("")
print("STEP 3: isolated per-worker state, merged at the end")
merged = {}
for task_id, text in TASKS:
own = {"result": None, "by_id": {}} # this worker's private scratch space
worker(task_id, text, own)
merged.update(own["by_id"]) # the orchestrator owns the merge
print(" worker %s returned %r" % (task_id, own["by_id"][task_id]))
print(" aggregated results survived : %d of %d" % (len(merged), len(TASKS)))
# ── STEP 4: did the orchestrator actually get the right answers? ──────────────
print("")
print("STEP 4: correctness of the merged result")
for task_id, expected in sorted(TRUTH.items()):
print(" %s expected %-8s got %-8s %s" % (task_id, expected, merged[task_id],
"ok" if merged[task_id] == expected else "WRONG"))
lost_to_race = len(TASKS) - len(racy)
ok = (isolated
and lost_to_race == 2
and merged == TRUTH
and len(merged) == len(TASKS))
print("")
print(" results lost to the shared key : %d" % lost_to_race)
print("ISOLATED WORKER STATE FIXED THE RACE AND THE ORCHESTRATOR MERGED ALL 3: %s"
% ("YES" if ok else "NO"))
if not ok:
sys.exit(1)
print("Split for context first, parallelism second. One agent with better tools often wins.")
Runnable lab
ae9-orchestrator-workers.pyDispatch three workers from one orchestrator, measure their context isolation, then reproduce and fix the shared-state race that silently eats two of the three results.
Proves: ISOLATED WORKER STATE FIXED THE RACE AND THE ORCHESTRATOR MERGED ALL 3: YES
Runs in your browser via Pyodide. First run loads the runtime once; no install, no server.
Both runs made the same model calls with the same prompts. The only difference was where the results went. When every worker wrote state['result'], each overwrote the last, and the orchestrator aggregated one answer out of three: two results were lost with no error raised anywhere. Giving each worker a private dict and merging by task id kept all three. That is the shared-state hazard, and it never announces itself. In a threaded or async orchestrator it gets worse, because which value survives depends on which worker finishes last, so the bug is intermittent and passes in testing. The rule that avoids it: workers return values, they do not mutate shared structures, and the orchestrator owns the merge. Next you grade the thing you just built, which is harder than grading a single call.
Check your understanding
1. What is the primary reason to split work across multiple agents?
2. Two workers both write their result to shared['result']. What happens?
3. When does one agent with better tools beat a five-agent system?