Building LLM agents that don't fall apart in production

An LLM agent is a small idea wrapped in a loop: give the model a goal and some tools, let it decide what to do, run the tool it asks for, feed the result back, and repeat until it says it's done. In a notebook this feels like magic. In production it's where the on-call pages come from.

The demo and the outage run the exact same loop. The difference is entirely in the guardrails around it — the limits, the validation, the determinism, and the ability to see what happened when it goes wrong. This is what I've learned keeping agents running against real traffic and real bills.

The loop, and why it's dangerous

Every agent is some version of the same cycle: plan what to do next, call a tool, observethe result, decide whether the goal is met, and loop back if not. It's clean and general — which is exactly the problem. A plain whileloop driven by a probabilistic model has no natural stopping condition, no cost ceiling, and no guarantee the model's next request is even valid. Left alone, it will happily spin.

Bound everything

The first job is to make the loop finite and cheap by construction, not by hoping the model behaves.

• Max steps.A hard cap on iterations. If the agent hasn't finished in, say, 10 steps, it stops and returns a partial result or an explicit failure. No cap means an infinite bill waiting to happen.
• Wall-clock timeout.Independent of step count. A tool that hangs shouldn't hang the whole request. Budget the total run and abort when it's spent.
• Token / cost budget. Track spend per run and refuse to exceed it. This is the difference between a bad run costing cents and costing a mortgage payment.
• Bounded retries with backoff. Transient tool failures get a couple of retries; persistent ones surface as errors the agent can reason about, not silent retries forever.

Validate every tool call against a schema

The model does not call your function — it produces text that claimsto be a function call. Treat that claim as untrusted input, exactly like a request body from the internet. Define each tool's arguments as a strict schema and validate before you execute anything. If validation fails, you feed the error back and let the model correct itself, rather than crashing on a missing field or a string where you expected an integer.

from pydantic import BaseModel, ValidationError

class SearchArgs(BaseModel):
    query: str
    limit: int = 10          # bounded, typed, with a default

def run_tool(name: str, raw_args: dict) -> str:
    try:
        args = SearchArgs(**raw_args)      # reject anything malformed
    except ValidationError as e:
        # Hand the error back to the model as an observation.
        return f"invalid arguments: {e.errors()}"
    return search(args.query, limit=min(args.limit, 50))

Push determinism to the edges

The model is the one non-deterministic part of the system, so shrink its surface area. Anything that can be code should be code. Don't ask the model to do arithmetic, parse dates, or format JSON — give it a tool that does those things correctly every time. Constrain what it can output with structured decoding or a fixed set of allowed actions. The more of the run is deterministic tooling and the less is free-form generation, the more predictable — and testable — the agent becomes.

A useful mental model: the LLM is a router that decides which deterministic thing to do next. Keep the routing narrow and the actions boring.

You cannot ship what you cannot evaluate

"It seemed to work when I tried it" is not a release criterion. Agents need evals — a fixed set of representative tasks with checkable outcomes — that you run on every prompt change, model swap, and tool edit. The outcomes don't have to be a single golden string; they can be assertions: did it call the right tool, did it stay under budget, did the final answer contain the required field. Without this, every change to a prompt is a blind edit to production behavior, and LLM upgrades become terrifying instead of routine.

Watch cost and latency like a hawk

Each step is at least one model call, and agents chain steps, so cost and latency compound. A five-step agent is five sequential round trips before the user sees anything. A few things keep this sane: prefer fewer, richer steps over many tiny ones; cache stable context so you're not re-sending the same system prompt and tool definitions every call; use a smaller, cheaper model for routing and reserve the expensive one for the hard reasoning; and stream partial output so the experience feels responsive even when the full run is slow. Measure per-run cost and p95 latency the same way you'd measure any other endpoint.

The failure modes you will actually hit

• Loops. The agent repeats the same action, or oscillates between two, never converging. Max steps stops the bleeding; detecting a repeated state and breaking out is better.
• Hallucinated tool calls.It invokes a tool that doesn't exist, or invents plausible-looking arguments. An allowlist and strict schema validation catch this before execution.
• Silent errors. A tool fails, returns an empty or error value, and the agent charges ahead reasoning on garbage. Always feed failures back as explicit observations so the model — and your logs — know something went wrong.
• Cost blowups. One pathological input triggers a long, expensive run. A per-run budget caps the damage.

The unglamorous conclusion

A production agent is 20% clever prompting and 80% ordinary systems engineering: bounded loops, validated inputs, retries, budgets, logging, and tests. The loop is the easy part. The reason agents fall apart in production is almost never that the model wasn't smart enough — it's that nothing was watching the loop. Build the guards first, and the intelligence has somewhere safe to run.