Handoff Note: Multi-Agent Systems Fail at the Seams

Each agent can pass its own evals and the system still drops facts at the handoff. Pass a typed Handoff Note (facts, constraints, open questions, sources) and check it at every seam.

By Wasim Sheikh AI Architect

Every agent in the pipeline passed its own evals. The claim still came out wrong.

When you split work across agents (a planner, an extractor, a reviewer), you add something no single-agent eval covers: the handoff. Each agent can do its part well, and the system can still lose the one fact that mattered, because nobody owned the space between them.

Many of the multi-agent bugs I review aren't model bugs. They're seam bugs. The fix is boring and it tends to work: every agent passes a Handoff Note, a small typed record of what it knows, what it was told not to do, what's still open, and where each fact came from. The next agent reads the note first, and the system checks the note at every seam.

Quick use case

This is an illustrative example, a composite of a common failure pattern, not a client story.

Situation. Say a regional insurer's claims intake team runs three agents. An intake agent reads the customer's email and attachments. An extractor pulls policy number, loss date, amount, and coverage type into the claims system. A reviewer agent checks the result and either approves it for an adjuster or kicks it back. Each agent has its own eval set, and each one scores well.

What breaks. A customer writes, "the water damage started before I renewed, but the agent on the phone said the old policy covers it." The intake agent catches it and summarizes the case in a free-text paragraph. The extractor only takes structured fields from the attachments, so it files the claim under the new policy's dates. The reviewer re-reads the attachments from scratch, sees consistent fields, and approves. The phone-call exception never reaches the adjuster, and the claim is denied on a date technicality that takes an escalation to unwind.

No agent was "wrong" on its own test. The fact died in a paragraph that the next agent never had to read.

The fix (Handoff Note).

Typed note, not prose. Each agent writes four lists: facts, constraints, open questions, and sources.
Read the note before the raw input. The extractor starts from the intake note, so "possible prior-policy coverage" arrives as an open question it has to resolve or carry forward.
Open questions can't disappear. A question is either answered with a source or passed on. The reviewer can't approve while one is still open.
Check the seam. A cheap validator compares the note going in with the note coming out and flags any fact or question that vanished.
Log the notes with the trace. When a claim goes wrong, you can see which seam dropped what.

Same agents, same models. In this example, the exception now shows up on the adjuster's screen as an open question with the customer's own words attached.

Why per-agent evals aren't enough

Per-agent evals test each agent on its own input. In production, an agent's real input is the previous agent's output, and that output drifts as prompts, models, and upstream agents change. An extractor that scores well on clean attachments tells you little about whether it respects a caveat the intake agent raised.

Free-text summaries make this worse. They read well to humans and pass vibe checks, but they let the next agent pick what it pays attention to. Anything without a slot is optional, and optional facts get dropped under pressure.

Handoff Note, illustrative example: a customer email with a caveat line flows to an intake agent that emits a Handoff Note with four slots (facts, constraints, open questions, sources). A seam check diffs the note in versus out. The extractor's note still carries the open question, while a side callout shows the free-text path dropping it. The reviewer gate approves only when open questions are zero or handed to a human, and notes are stored per seam, stamped with the Release Pin

The Handoff Note (five layers)

1. Four slots, always

Facts (with values), constraints ("don't file under the new policy until coverage is confirmed"), open questions, and sources (document, line, message ID). Keep it short. If a slot is empty, it says so on purpose.

2. Note first, raw input second

The receiving agent's prompt starts with the note and treats it as the brief. It can still read the raw documents, but it has to reconcile anything that conflicts with the note instead of quietly overwriting it.

3. Questions are conserved

An open question leaves a seam in one of two states: answered, with a source, or still open. "Gone" isn't a state. This one rule catches a large share of dropped-exception bugs.

4. A boring validator at every seam

Not another LLM judge, just code. Diff the incoming and outgoing notes. Missing facts, missing questions, or new facts with no source fail the handoff and route to a human or a retry. This is the Boundary Contract idea applied between agents instead of between an agent and a tool.

5. Notes live in the trace

Store every note with the run, stamped with the Release Pin. When something goes wrong, you can replay the seams instead of guessing which agent "probably" missed it, and you can turn the bad run into a Golden Trace for the whole pipeline.

How this maps to what you already have

5-layer agent stack. The note is the contract between Orchestration and each agent's Memory, and it's recorded in Traces.
Golden Trace. Add pipeline-level golden runs that assert which facts and questions survive end to end.
Shadow Path. When you swap one agent in the chain, shadow it and compare its outgoing notes, not just its final labels.
Ship Gate. Add one check: no seam in the pipeline accepts free-text-only handoffs.

A test you can run this week

Take ten recent runs of any multi-agent flow. For each one, list every caveat, exception, or "but" in the original input. Then check whether each one is visible in the final output or explicitly resolved somewhere along the way.

If even one disappeared without a trace, you have a seam bug. Add the four-slot note at that seam before you touch a prompt or a model.

Closing

Agents don't usually fail alone. They fail at the handoff, where a fact one agent knew never became something the next agent had to deal with. Give every seam a typed note, conserve open questions, check the diff with plain code, and keep the notes in the trace. Make the handoff the part you can audit.

I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Wasim Sheikh on Substack · X: @anciwasim