Skip to main content
Back to Insights

Five of Five: Run It Again Before You Trust an Agent

An agent that passes a task once can do it. An agent that passes it five runs out of five is one you can leave alone. How to test repeat-run reliability before an agent goes to production.

The demo worked. So did the second try. The third one quietly sent a claim to the wrong queue.

Most agent evaluations ask one question: can it do the task? Run it once, read the output, nod. That tells you the agent is capable. It does not tell you the agent is dependable, and production only cares about the second one.

The fix is a simple habit I call Five of Five: before an agent ships, every test case runs several times, and a case only passes if every run passes.

Five of Five: a pinned release of model, prompt, tools, and retrieval. Case 17, a claim email with a scanned PDF, is run five times and passes four, so it is flaky. The scoreboard shows 40 of 40 passed once and 31 of 40 passed five of five. Flaky cases cluster around scanned attachments and emails with two policy numbers, then get fixed with OCR first and a route to a person. The release ships only when every case is five of five. Illustrative example, not a client case

Quick use case

Situation. An insurance operations team builds an intake agent. It reads incoming claim emails and attachments, pulls the policy number, classifies the claim type, and routes it to one of six adjuster queues. The team builds 40 test cases from real past claims. The agent gets 38 right. Two fixes later it gets 40 out of 40, and it ships.

What broke. Within two weeks adjusters start forwarding misrouted claims back. Water damage lands in auto. A few policy numbers come through with two digits swapped. The team reruns the 40 test cases and they all pass again. Nobody can reproduce the problem.

Then someone runs the same 40 cases five times each. Thirty-one cases pass every time. Nine pass on some runs and fail on others. The agent was never 40 for 40. It was 40 for 40 on one lucky afternoon.

The fix (Five of Five).

  1. Repeat every case. Each test case runs five times on the exact model, prompt, and tool versions going to production.
  2. Score the worst run. A case passes only if all five runs pass. The headline number changes from "40/40" to "31/40 consistent."
  3. Look at the flaky ones first. All nine unstable cases shared two causes: attachments that were scanned images, and emails mentioning two policy numbers.
  4. Fix the cause, not the prompt wording. Scanned attachments now go through OCR before the model sees them. Emails with two policy numbers get flagged for a person instead of a guess.
  5. Gate on it. The release only ships when every case is five for five, and any new flaky case blocks the deploy.

Same model. After the fixes, 39 of 40 cases were five for five, the last one went to the human-review path on purpose, and the misroutes stopped.

Why one pass lies

Language models are not deterministic in practice. Even at low temperature, small differences in input order, tool latency, retrieved context, or provider-side changes can push a borderline case either way. A single run samples one point from a range of possible behaviors. If that range includes a failure, you will hit it in production, because production runs the task thousands of times.

The math is unforgiving. An agent that gets a case right 90% of the time sounds solid. Run that case five times and the chance all five succeed is about 59%. Over a day of real traffic, the 10% is a steady trickle of mistakes that nobody sees in the demo.

One pass measures capability. Repeat passes measure reliability. You need both, and only the second one lets you sleep.

The Five of Five framework (four layers)

1. Pin what you test

Run the repeats on the exact versions you ship: model, prompt, tools, retrieval settings. If any of those can change under you, the result means nothing. This is the Release Pin idea applied to evals.

2. Repeat and score the worst run

Run each case at least five times. Report two numbers side by side: "passed at least once" and "passed every time." The gap between them is your flakiness, and it is the most useful number on the dashboard.

3. Group the flaky cases by cause

Flaky cases cluster. Look for what they share: input format, length, ambiguity, a specific tool. Fix the cause with structure (cleaner inputs, a deterministic step, a rule in code) before you touch prompt wording. When a case is truly ambiguous, route it to a person on purpose instead of letting the model guess.

4. Make consistency a release gate

Add the consistent-pass rate to your Ship Gate. New flaky cases block the deploy, the same way a failing unit test would. Keep the flaky cases in your Golden Trace set so they get rechecked on every release.

How this maps to what you already have

  • 5-layer agent stack. Five of Five lives in Evals and feeds the release decision in Orchestration.
  • Smoke Evals. Smoke evals catch obvious breaks fast. Five of Five catches the quiet ones.
  • Done Check. A Done Check confirms one run's result is correct. Five of Five confirms it will be correct again tomorrow.
  • Cheap Twin. If you route easy work to a smaller model, run Five of Five on both models. A cheap model that is right 4 of 5 times is not cheap once you count the cleanup.

A test you can run this week

Take the eval set for one agent you already run in production, even if it is only 20 cases. Run every case five times without changing anything.

Write down two numbers: how many cases passed at least once, and how many passed all five times. If they match, you have earned some trust. If they don't, the difference is the list of production incidents you haven't had yet. Start with the case that failed most often.

Closing

An agent that gets a task right once can do it. An agent that gets it right five runs out of five is one you can leave alone overnight. Demos show the first. Production needs the second. Pin your versions, repeat every case, score the worst run, and make consistency a gate. Run it again before you trust it.

I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Wasim Sheikh on Substack · X: @anciwasim

Related: Memory Expiry · Done Check · Golden Trace · Smoke Evals · Cheap Twin · Release Pin · Ship Gate · 5-Layer Agent Stack