Insights
Writing & Architecture Notes
Named patterns and architecture notes, newest first: Ship Gate, Receipt Gate, Release Pin, Undo Ledger, and the rest of the series.
Handoff Note: Multi-Agent Systems Fail at the Seams
Each agent can pass its own evals and the system still drops facts at the handoff. Pass a typed Handoff Note (facts, constraints, open questions, sources) and check it at every seam.
Shadow Path: Prove the New Agent Before It Writes Anything
Offline evals can pass and still fail on live traffic. Run the new agent in shadow: writes off, score disagreement, promote only when the gap is safe.
My From-Scratch Phishing Classifier Matched scikit-learn. The Best Part Was a Bug.
What a logistic regression on URLs taught me about leakage, and about checking outputs instead of trusting them
The key was supposed to be random. It fit in one byte.
Eight trusted CA keys, a certified HSM, two clean audits, and nobody looked at what actually came out
Undo Ledger: Every Agent Step Ships With Its Reverse
When a multi-step agent fails or gets cancelled halfway, "stop" leaves half-finished work in real systems. An Undo Ledger records every step with its reverse, so cancel means unwinding cleanly, in order, with proof.
Release Pin: Ship the Model, Prompt, and Tools as One Version
Agents drift when the model alias updates, a prompt gets edited in a dashboard, or a tool schema changes underneath them. A Release Pin ships all of it as one versioned manifest you can test, trace, and roll back in one move.
Boundary Contract: Make Every Agent Tool Fail Boring
Agents turn tool timeouts and empty results into confident answers. A Boundary Contract gives every tool typed outcomes — ok, empty, partial, timeout, rejected — so the agent can't mistake a broken call for the truth.
Cheap Twin: Escalate Only When the Small Model Needs Help
Defaulting every agent request to the flagship model wastes money and hides which tickets needed the heavyweight path. Cheap Twin runs a small model first and escalates only when a short checklist or smoke assert says so — with a receipt that names which brain answered.
Clean Exit, Wrong Record
A support agent can close the ticket, return no tool error, and still leave the wrong record. Microsoft and Hugging Face's ThinkingBox (Oct 3, 2026) measured that gap. Gate the customer-visible step on a readback, not the model's "resolved."
Receipt Gate: No Second Write Without a Receipt
Cascading agent writes fail quietly when nobody can skim what just changed. A Receipt Gate requires a human-readable receipt after every write before the next tool fires — what changed, which tool, which inputs, success or fail in plain language.
Golden Trace: The Eval Anchor Most Teams Skip
Failure logs teach. A locked successful run proves what still works. Golden Trace freezes one end-to-end agent path — inputs, tools, checks, outcome — and regresses every change against it before users feel the drift.
Permission Envelope: Consent That Expires
Permanent tool grants are how agents outlive the moment you meant them to act. A Permission Envelope scopes what an agent can do, for how long, and under whose fresh consent — with expiry as the audit trail.
Smoke Evals: The 20 Cases That Should Block Every AI Deploy
Demos pass. Production fails. Smoke evals are a fixed set of ~20 trap cases that must pass before any AI feature ships — safe, useful, and cheap enough to run on every change.
Ship Gate: The Pre-Deploy Checklist Most AI Features Skip
Most AI features don't fail in the model — they fail at the gate. Ship Gate is a five-step pre-deploy checklist: threat model, data boundary, eval set, human override, and readable observability.
The 5-Layer Stack Behind Agents That Ship
Most AI agents fail because a chat model is glued to tools and hoped into production. Here's the five-layer stack that holds up: Orchestrator, Tools, Traces, Guardrails, Rollback.
Stay updated
Subscribe for long-form essays on AI architecture and production systems.