The agent didn't crash. That was the problem.
It ran all night, stayed "healthy" on every dashboard, and spent more in eight hours than the whole project had spent in a month. Nothing threw an error. It just kept trying.
Most teams put limits on their APIs and none on their agents. A model that can call tools, read the result, and decide to try again is a loop with a credit card attached. The fix isn't a smarter prompt. It's a Run Budget: every run starts with a fixed allowance of tokens, tool calls, wall-clock time, and dollars, and when any one runs out, the run stops cleanly and tells you why.
Quick use case
Situation. A logistics company runs an agent that matches carrier invoices to purchase orders. For each invoice it looks up the PO, compares line items, and either approves the match or flags it for a person. On a normal night it handles about 2,000 invoices for a few dollars.
What broke. A carrier changed its invoice PDF layout. The extractor started reading the PO number with one digit missing. The lookup tool returned "not found," and the agent did what its prompt told it to: "If the PO isn't found, try alternate formats and search again." It tried with dashes, without dashes, with leading zeros, with a fuzzy search, then started over with a bigger context window "to be thorough." Each invoice turned into 40 or more tool calls and a growing prompt. By morning, a few hundred invoices had used most of the monthly API budget, the PO system had rate-limited the whole company, and not one of those invoices had been flagged for a person.
Every single step looked reasonable. The total was the bug.
The fix (Run Budget).
- Budget per run, set before it starts. Each invoice gets at most 8 tool calls, 30k tokens, 90 seconds, and $0.10.
- Count in the orchestrator, not the prompt. The model doesn't get to decide whether it's over budget. Plain code counts and enforces.
- Stop with a report, not a crash. When a limit hits, the run ends in a named state, "budget_exhausted," with what it tried and where it got stuck.
- Exhausted runs go to a person. The invoice lands in the review queue with the attempts attached, which is the outcome it should have had after the second failed lookup.
- A ceiling for the whole fleet. A nightly cap pauses the job and pages someone if total spend crosses 3x the normal night.
Same model, same prompt. The next layout change cost $11 and produced 140 flagged invoices with clear notes instead of a surprise bill.
Why "it'll stop when it's done" isn't a plan
Agents are built to keep going. Retry instructions, self-reflection, and "try another approach" are exactly what make them useful on hard tasks, and exactly what turns a bad input into an endless one. Bad inputs are guaranteed in production: a changed format, a down service, a tool that returns an empty result instead of an error.
Rate limits on the provider side don't save you. They protect the provider, not your budget, and they kick in long after the money is spent. Per-month spending alerts arrive after the fact. You need a limit that's checked on every step of every run.
The Run Budget (five layers)
1. Four meters, every run
Tool calls, tokens, wall-clock time, and dollars. Each one catches a different failure. Tool calls catch retry loops, tokens catch prompts that keep growing, time catches hung tools, and dollars catch everything the other three miss. Set them from real traces: take a normal run's usage and allow about 3x.
2. Enforced outside the model
The orchestrator checks the meters before each step. The model can see how much budget is left, which helps it plan, but it can't raise its own limit. This is the same rule as the Boundary Contract: the boring code owns the hard rules.
3. A clean stop state
"budget_exhausted" is a real outcome, like "approved" or "flagged." It carries the attempts, the last error, and which meter ran out. A run that stops this way is a useful signal, not an incident.
4. A human path for every stop
An exhausted run never just disappears. It goes to the same review queue as a low-confidence answer. If your system has no place for "I tried and couldn't," the agent will keep trying.
5. Fleet ceiling and alerts
Per-run budgets stop one bad run. A fleet ceiling stops a thousand of them at once. Track spend per job per hour against a normal baseline, pause at a multiple of it, and alert a person. Stamp every run with the Release Pin so you can tell whether a spike came from a new prompt, a new model, or new data.
How this maps to what you already have
- 5-layer agent stack. Budgets live in Orchestration, and their meters are recorded in Traces.
- Cheap Twin. Route to the bigger model only while there's budget left for it.
- Golden Trace. Add a golden run with a bad input and assert that it ends in "budget_exhausted," not in 40 retries.
- Ship Gate. Add one check: no agent job ships without per-run limits and a fleet ceiling.
A test you can run this week
Pick one agent you run in production. Feed it an input that can't succeed, like an ID that doesn't exist or a tool stubbed to return empty results. Then watch three numbers: how many tool calls it makes, how long it runs, and what it costs.
If the answer to any of them is "until something else stops it," you need a Run Budget before you need a better prompt.
Closing
An agent that never gives up sounds like a feature until the bill arrives. Give every run four meters, enforce them in plain code, make "I ran out of budget" a normal outcome with a person on the other end, and put a ceiling over the whole fleet. The goal isn't a cheaper agent. It's an agent whose worst night you already know the price of.
I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.
Follow for practical AI architecture that ships.
Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Wasim Sheikh on Substack · X: @anciwasim
Related: Perfect Sprint Board · Handoff Note · Shadow Path · Undo Ledger · Release Pin · Boundary Contract · Cheap Twin · Receipt Gate · Golden Trace · Smoke Evals · Ship Gate · 5-Layer Agent Stack