Giving an AI agent a budget

The scariest thing about an autonomous agent isn't that it might do something wrong. It's that it might do something wrong twelve hundred times before anyone notices, on a schedule, at three in the morning.

Anyone can build an agent that works when you're watching it. The engineering problem is making one you can walk away from.

Three ceilings, not one

We settled on three independent limits, because they fail in different ways.

1. A token ceiling

Every agent carries a maximum token budget per run. This is the crudest limit and the most useful, because runaway loops are the most common agent failure mode. An agent that gets stuck re-reading the same document until it "understands" it will burn tokens indefinitely, unless it can't.

The ceiling turns an unbounded failure into a bounded one. The run stops, it's marked as hitting its limit, and you get told.

2. A spend cap

Tokens aren't cost. Different models are priced an order of magnitude apart, and an agent that's allowed to pick its model can be cheap on Monday and expensive on Thursday.

So spend is capped separately, in currency. This is the number a finance team actually cares about, and it's the one we surface on the dashboard next to hours saved, because a workload that saves four hours and costs six hours' wages in compute is a workload you should turn off.

3. A confirmation gate

The first two limits bound cost. Neither bounds consequence, and consequence is not proportional to cost. Sending one email to the wrong 4,000 people is cheap and catastrophic.

So tools are classified by whether they change something outside the system. A read is a read, so let it run. A send, a payment, a deletion, a public post: those can be configured to pause and wait for a human to approve, with the exact arguments shown before anyone clicks.

Awaiting confirmation
gmail.send_email · 3 recipients
[ Approve ]  [ Reject ]

That pause is the difference between an agent you pilot and an agent you deploy.

What we got wrong first

Our initial version had a single global limit per organisation. It was useless in both directions at once: generous enough that a broken agent could still do real damage, and tight enough that one heavy legitimate workload would starve everything else.

Limits have to be per-agent, because agents have wildly different appetites. A support triage agent runs hundreds of times a day for pennies. A contract reviewer runs twice a week and reads eighty pages. One global number cannot describe both.

Logs are part of the safety system

A limit tells you an agent stopped. A log tells you why it was heading there.

Every run records each step in order: the trigger that started it, each tool call with its arguments, what came back, and how long it took. When something goes wrong the question is never "did it break". It's "at which call did it start going wrong, and what did it see at that point?"

Without step-level logs you're left re-running the agent and hoping it misbehaves the same way. With them, most debugging is reading.

The rule of thumb

If you can't answer these three questions about an agent, it isn't ready to run unattended:

1. What is the most this can cost me in a day?
2. What is the worst thing it can do without a human agreeing to it?
3. Where do I look to find out what it did?

Good answers make autonomy boring. Boring is the goal.