Your new agent passed every offline check. Then you gave it the write path, and live traffic taught you what the eval set never saw.
Smoke Evals and Golden Traces are necessary. They are not a substitute for watching the candidate disagree with the live path on real requests before anything mutates.
Most teams flip traffic because the scorecard looked green. Production agents need a quieter step first: run the new version beside the old one, block every write, and promote only when the disagreement that matters is small enough to live with.
I call that a Shadow Path: same live inputs, no side effects, a scored compare against what actually happened, and a promote gate stamped with the Release Pin.
Quick use case
Situation. A mid-size SaaS support team's intake desk runs an agent that routes tickets: refund, billing, bug, or VIP escalate. The current setup is a rules-plus-LLM hybrid. It is imperfect, but finance gets refund tickets and VIP issues leave the queue fast. The team builds a replacement agent. Offline Smoke Evals and Golden Traces look strong.
What broke. They flipped 100% of traffic overnight. Within a week, refund tickets that used to go to finance started landing in general support. VIP escalations sat longer because the new agent asked one more clarifying question before escalating. Those patterns were not in the offline set, and "helpful" burned SLA and money lanes.
The fix (Shadow Path).
- Same inputs, no writes. The candidate sees live tickets. Mutating tools are stubbed or recorded as proposals only.
- Score disagreement, not just accuracy. Compare the proposed label or tool call to what the live path did. Track rate and severity: harmless vs money, access, or SLA-critical.
- Promote by lane. Shadow-only first, then canary writes on low-risk classes, then full cutover. Never big-bang because evals passed.
- Stamp every shadow run with the Release Pin. You always know which manifest disagreed.
- Close the shadow window with a receipt. Proposed N, agreed M, disagreed K by severity, and a decision: hold, canary, or promote.
Same traffic. Same team. The difference was that the disagreement showed up before any write path changed.
Why "evals passed" isn't enough
Offline cases are frozen. Live traffic moves. New phrasings, seasonal refund spikes, and VIP patterns that never made the golden set all show up after you ship.
Accuracy against a fixed set also hides the wrong kind of error. An agent can look "right" on a rubric and still route money to the wrong queue, or delay an escalation by one polite question. You need a disagreement score against the path already carrying production risk, and a rule that writes stay off until that score is acceptable.
The Shadow Path (five layers)
1. Same inputs, no writes
Feed the candidate the same live requests the production agent sees. Stub every mutating tool, or record the would-be call as a proposal. Traces still run. Receipts still exist, for proposed actions, not executed ones. The shadow agent can think and plan. It cannot change the world.
2. Score disagreement, not just accuracy
For each request, compare the shadow's proposed action, label, or tool call to what the live agent (or human) actually did. Bucket severity: cosmetic vs money, access, or SLA-critical. A 4% overall disagreement rate with a spike on refunds is not "almost fine." It is a hold.
3. Promote by lane
Start shadow-only. When disagreement on low-risk classes is quiet, open canary writes on those classes only. Keep money and VIP lanes in shadow longer. Full cutover is the last move, not the first.
4. Stamp every shadow run with the Release Pin
Every shadow trace carries the candidate manifest: model snapshot, prompt hash, tools, eval set. When refund disagreement jumps, you know which release caused it. Hold or promote is a decision about a version, not a vibe.
5. Close the shadow window with a receipt
End with a short receipt: proposed N, agreed M, disagreed K by severity, and the decision: hold, canary, or promote. If the counts don't support the decision, the window isn't closed. Treat the promote call like a Receipt Gate: one write into the release record, not a Slack shrug.
How this maps to what you already have
- Release Pin. Shadow runs a candidate manifest beside the live pin. Promote means pointing production at that manifest.
- Golden Trace. Offline anchors stay. Shadow adds a live disagreement layer those frozen paths can't see.
- Smoke Evals. Still the pre-deploy bar. Shadow is the post-deploy-but-pre-write bar.
- Receipt Gate. Proposed actions get proposal receipts; promote gets a release receipt.
- Ship Gate. Add one check: shadow window closed with an acceptable disagreement receipt before the write path opens.
- 5-layer agent stack. Shadow sits across Tools (writes stubbed) and Traces (proposals and compares logged).
A test you can run this week
Pick one agent write path. For 48 hours, log what a candidate agent would have done on the last N live requests without executing. Count disagreements on money, access, or SLA-critical actions.
If that count isn't near zero, or each case isn't explained and accepted, do not flip traffic. Fix the candidate, widen the golden set, or keep the lane in shadow.
Closing
Evals tell you the agent can pass the cases you already know. A Shadow Path tells you how it disagrees with production before it can hurt production. Same inputs, no writes, score the gap, promote by lane, stamp the pin, close with a receipt. Prove it before it writes.
I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.
Follow for practical AI architecture that ships.
Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Wasim Sheikh on Substack · X: @anciwasim
Related: Undo Ledger · Release Pin · Boundary Contract · Cheap Twin · Receipt Gate · Golden Trace · Smoke Evals · Ship Gate · 5-Layer Agent Stack