Agents, and why they fail

The arithmetic that governs everything

Start here, because every architectural decision later on follows from it.

end_to_end_success = per_step_success ^ n_steps

  95% per step, 10 steps  ->  0.95^10  = 60%
  95% per step, 20 steps  ->  0.95^20  = 36%
  99% per step, 20 steps  ->  0.99^20  = 82%
  90% per step,  5 steps  ->  0.90^5   = 59%

A 95% success rate per step sounds excellent. Over ten steps it produces a system that fails four times in ten. That is the entire gap between the demo — which was five steps and got lucky — and the product.

Two consequences, and they point in the same direction.

Fewer steps beats better prompts. Going from 10 steps to 5 at the same per-step rate takes you from 60% to 77%. Getting per-step accuracy from 95% to 97% takes you to 74%. Cutting steps is usually both easier and more effective, and it is the lever people reach for last.

Verification between steps changes the exponent into something manageable. If a failed step is detected and retried rather than propagating, the effective per-step rate rises toward the rate at which failures are detectable — which is often far higher than the rate at which they do not occur. This is why the single most valuable thing you can add to an agent is usually a check, not a capability.

The loop

Strip away the frameworks and every agent is the same loop.

state = initial_context(task)

for step in range(MAX_STEPS):              # always bounded
    action = model(state, tools)         # decide
    if action.is_final:
        return action.answer

    result = execute(action)             # act
    state  = state + [action, result]    # observe

return give_up_gracefully(state)          # the path people forget

Three properties of this loop cause most production problems.

State grows monotonically. Every step appends. Twenty steps of tool calls and results is a very long context, which costs money, slows every subsequent step, and pushes earlier material into the region models attend to least. An agent that has been running for a while is a worse agent than it was at the start.

Errors persist in the context. A failed tool call with a confusing error message stays in the state and continues influencing every future decision. Models frequently get stuck re-attempting a variation of the same failed action because the failure is sitting right there in the context, shaping the next prediction.

The loop must be bounded and must have a give-up path. An unbounded loop with an intermittent failure is an unbounded bill. And "gave up gracefully with a partial result and an explanation" is a much better product outcome than "hit the step limit and returned nothing," yet the second is the default in most implementations.

Designing tools a model can use

Tool design is the highest-leverage and least-discussed part of building agents. A well-designed tool raises per-step success directly, and per-step success is in the exponent.

Name and describe tools for a reader who has no other context. The model sees the name, the description, and the parameter schema — nothing else. query_db with the description "queries the database" is unusable. find_customer_orders described as "returns a customer's orders from the last 90 days, most recent first; returns an empty list if the customer has none" is usable, because it says what it does, what it returns, and what happens in the empty case.

Make wrong calls impossible rather than detectable. Enums instead of free-text strings. Required parameters that are genuinely required. Types the schema enforces. Every constraint you push into the schema is a failure the model cannot make, which is strictly better than a failure you catch afterwards.

Fewer tools, better chosen. Selection accuracy falls as the tool count rises, and with thirty tools models start picking plausible-but-wrong ones. If you have many, group them behind a router — a first call that selects a small relevant subset, then a second with only those available.

Design tools around tasks, not around your API surface. Exposing your twelve REST endpoints as twelve tools forces the model to compose them correctly, which is twelve chances to fail. One tool that does the composition internally is one step instead of four, and — per the arithmetic — that is the single most effective change available.

Return results the model can act on. A 400 with a stack trace teaches it nothing. "Invalid date format; expected YYYY-MM-DD, received 03/04/2024" tells it exactly what to fix, and models recover from that reliably. Error messages are prompt engineering.

Keep results small. A tool returning 5,000 rows poisons the context for every subsequent step. Paginate, summarise, or return references the model can expand selectively.

Make retries safe. Agents retry. If a tool has side effects, give it an idempotency key so a retried call does not duplicate the action. This is the difference between an agent that occasionally fails and one that occasionally sends two emails.

Context and memory

Because state grows monotonically, managing it is not an optimisation — it is what keeps a long-running agent working at step 30 as well as it did at step 3.

Summarise the middle. Keep the original task and the most recent few exchanges verbatim, and compress what is between them. This preserves the two regions models attend to most reliably while cutting the bulk.

Drop or truncate large tool outputs once used. A 4,000-token document retrieved at step 3 rarely needs to remain verbatim at step 15. Replace it with a short summary and a reference.

Keep a structured scratchpad outside the conversation. Rather than relying on the model to remember its findings from the transcript, maintain explicit state — facts established, subtasks completed, open questions — and render it into the prompt fresh each step. This is markedly more reliable than transcript recall and it makes the agent's state inspectable, which matters enormously when debugging.

Distinguish the kinds of memory. Working memory is this task's state. Episodic memory is what happened in previous sessions, usually retrieved rather than carried. Semantic memory is durable facts about the user or domain, which is a retrieval problem and should be treated as one. Conflating them produces an agent that either forgets everything or drags everything along.

The failure taxonomy

What actually works

The patterns that survive production, roughly in order of how often they are the right answer.

Do not build an agent. Most tasks framed as agentic are a fixed sequence of steps with a model call at each. If you know the sequence, write the sequence. A deterministic pipeline with model calls inside it is dramatically more reliable, cheaper, easier to test and easier to debug than a model deciding the control flow. Reserve agency for genuine cases where the path cannot be known in advance.

Constrain the search space. Even where dynamic control flow is needed, it rarely needs to be unrestricted. A router that picks among five known workflows gets you most of the flexibility with a fraction of the failure surface.

Verify between steps. After each action, check the result programmatically where you can. Did the file get written, does the returned JSON match the schema, is the number in a plausible range. Cheap, deterministic, and it converts silent failure into a retry.

Plan once, then execute. Generating a plan upfront and then executing it is more debuggable than deciding each step in isolation, because the plan is inspectable before anything happens and a bad plan is visible immediately. Re-plan on failure rather than continuously.

Put a human at the irreversible points. Not on every step — that defeats the purpose — but on anything that spends money, sends a message, deletes data or cannot be undone. Users tolerate a confirmation far better than a mistake.

Specialise rather than generalise. An agent with four tools and a narrow task outperforms one with thirty and a broad remit, and it can actually be evaluated.

On multi-agent architectures: they are appealing and they multiply the arithmetic problem rather than solving it. Each agent has its own per-step failure rate, and the handoffs between them are additional failure points with no shared context. They are justified when subtasks genuinely need separate tool sets or separate context, not as a default structure. Be sceptical of a design where the main benefit claimed is conceptual tidiness.

The security model

Agents have a specific and serious risk profile, because they combine three things that are individually manageable and jointly dangerous: untrusted input, tool access, and actions with side effects.

The attack is straightforward. Content the agent processes — a web page, an email, a retrieved document, a file a user uploaded — contains instructions addressed to the model. The model, which has no reliable way to distinguish data from instructions, follows them. Now an attacker is issuing tool calls with your agent's permissions.

What actually helps:

Separate the three. If an agent processes untrusted content, it should not simultaneously hold destructive tools. Split into a stage that reads and summarises with no tools and a stage that acts on a validated structured result. This is architectural and it is the only mitigation that is robust rather than probabilistic.

Least privilege, per invocation. Scope credentials to the specific task and the specific user. An agent operating with broad service credentials turns any successful injection into a broad compromise.

Validate actions against a policy, not against the model's judgement. A deterministic check — this user may access this resource, this amount is under the limit, this recipient is on the allowlist — outside the model, on every call. Never rely on the system prompt to enforce a boundary.

Sandbox execution. Anything running generated code needs isolation, resource limits and no network by default.

Log every tool call with its arguments and result. Both for incident investigation and because an anomaly in the tool-call pattern is often the first detectable sign of an injection.

Attack your own agent. Write the injection payload, confirm it works, fix it, confirm the payload now fails. Until you have done that you are guessing about your system's behaviour rather than knowing it.

Evaluating an agent

End-to-end success alone is insufficient, because when it drops you cannot tell where. Measure the components:

Per-step accuracy — did each step choose the right tool with the right arguments. The number in the exponent, and the one most worth improving.

End-to-end success — was the task completed correctly. The number that matters to users.

Step count distribution, not the mean. A long tail of 40-step runs is where your cost and your timeouts live, and the mean hides it entirely.

Cost and latency per completed task, denominated by successful completion. An agent that fails half the time has twice the effective cost of its per-run figure.

Recovery rate — when a step fails, how often does the agent recover rather than derailing. This is the number that distinguishes a fragile agent from a robust one, and almost nobody measures it. Inject synthetic tool failures deliberately and see what happens; the results are usually sobering and immediately actionable.

Build the evaluation set from real tasks, including the ambiguous ones and the ones that should be refused or escalated. And log full traces — every step, every tool call, every result — because agent debugging without traces is not debugging, it is speculation.