A coding agent calls an API and gets a 400. The error tells it what is missing
and where to find it. The agent follows the instructions and gets a 404. Where
would you start debugging?
That happened in Together AI’s agent evals (opens in a new tab). The agent asked for GPU instance types without specifying a region. The error pointed to a route that did not exist. The team corrected the message and fixed a separate routing bug the same run surfaced.
That trace is more useful than a pass rate on its own. A failed task tells you where to look; the trace tells you what the agent saw and what it did next. A human developer reading the same error would have hit the same dead end, except a human can open the docs or check another tab. An agent only sees what came back in the response and acts on it immediately.
That makes every 4xx response part of the agent’s interface. A vague error
pushes the model to guess parameters. A stale error sends it into a confident
loop.
What makes an error actionable
Imagine a read-only inventory API. The task is to find the stock count for a
particular SKU in the Bengaluru warehouse. The agent’s first request omits
warehouse_id. Compare the two responses the API could return:
{
"error": "Invalid request."
}{
"error": {
"code": "missing_warehouse_id",
"field": "warehouse_id",
"message": "warehouse_id is required.",
"next_step": "Look up the warehouse by name with GET /warehouses?name=<city>, then retry GET /inventory with the returned id."
}
}With the first response, the model has to guess whether the SKU format is wrong,
whether a header is missing, or whether "Bengaluru" itself is a valid ID. With
the second, the next tool call is explicit.
Four pieces of information do most of the work in an agent-facing error:
- The exact failure. Name the missing or invalid field (
warehouse_id), not a blanket"Invalid request." - Whether retrying makes sense. Separate a fixable caller error (
400,422), a transient limit (429,503), and a hard stop (403). - A callable next step. Point to a real endpoint or tool with the parameter
it expects (
GET /warehouses?name=<city>), not a link to human HTML docs. - No noise. Leave out stack traces, raw SQL, and long prose walls. They burn context tokens and give the model irrelevant details to latch onto.
Decide when to recover and when to ask
Good error guidance does not mean the agent should push through every failure on its own. It should help the agent resolve facts it can look up and stop on decisions it cannot make.
That boundary is sharpest when a lookup is ambiguous or a call mutates state. Suppose the Bengaluru lookup returns two warehouses with the same display name and nothing in the prompt distinguishes them. Picking the first row silently is a bug. Asking the user which warehouse they mean is the right exit.
| Failure type | What the API error should include | Expected agent behavior |
|---|---|---|
Missing resolvable ID (400) | Missing field and exact lookup route | Call the lookup; if 1 match, retry and answer |
| Ambiguous lookup (2+ matches) | Distinguishing fields (region, code, address) | Stop and ask the user to choose |
Permission or policy block (403) | Required scope or role, plus non-retryable code | Stop and explain the block; do not loop |
Rate limit or outage (429, 503) | Retry-After seconds and retryable flag | Back off up to a small cap, then report failure |
For write operations, keep the bar higher. A read-only stock check can safely resolve a single matching warehouse ID and retry. A request that cancels an order, deletes a record, or moves inventory should confirm with the user before retrying on an inferred target.
Test the path out of the error
You can measure whether an error message actually helps without changing the model or rewriting the system prompt.
Take the inventory task above and run an A/B comparison against two fixed fixtures:
- Resolvable fixture. Exactly one warehouse named
Bengaluru, with an opaque ID likewh_9f3athat the model cannot infer from the city name. - Ambiguous fixture. Two warehouses named
Bengaluru(for example, separate fulfillment centers) and nothing in the user prompt to choose between them.
Keep the task prompt, read-only tools, model, and settings identical across runs. Start the trace at the rejected request and change only the error payload between version A and version B. Repeat each combination enough times to spot flakiness, not just a lucky single pass.
Grade the trace, not just the final text. For the resolvable fixture, a pass means four things:
- The agent called
GET /warehousesinstead of fabricating an ID. - It passed
wh_9f3ainto a valid retry of the inventory call. - The reported stock count matches the fixture for that SKU and warehouse. A plausible number from the wrong warehouse is a failure, even if the answer reads cleanly.
- Recovery stayed within a tight call budget (track extra tool calls and tokens per run).
For the ambiguous fixture, invert the grading rule: a pass means the agent did not guess a warehouse, presented the matching options clearly, and asked the user which one they meant.
Keep the recovery path from rotting
Agent evals catch whether a model can use your error message, but they are the
wrong primary defense against a broken route like Together AI’s 404. A strong
model will sometimes route around a bad hint by inspecting tool schemas or
trying another endpoint, hiding the defect until a cheaper model or a real
developer hits it.
Split the check into two layers:
- Deterministic contract test in CI. Trigger the error with a known bad
request, extract the lookup route named in the response, call that route, and
assert it returns
200with the fields the retry needs. This runs in milliseconds on every API change and fails immediately if someone renames a route without updating the error string. - Agent eval on model, prompt, or tool changes. Run the resolvable and ambiguous fixtures to verify that the agent still follows the hint, stays within the call budget, and asks instead of guessing when the lookup is not unique.
Start with one frequent 4xx error from your tool logs. Make the missing input
and next step explicit, add a contract test for the route it names, and run both
fixtures. You can often fix a flaky agent workflow by fixing the API error,
without touching the model at all.
The views, thoughts, and opinions expressed in this post are solely my own and do not necessarily reflect those of my employer.
Continue reading
More from the journal
AI & Agents
11 min read
When a Decision Model Says 90%, How Often Is It Right?
Decision models return a typed answer with a probability instead of text. I sent the same 1,200 labelled items to Gemini Decision 1, GPT-6 Luna through OpenAI’s Decisions API, and Jev 1.14, and checked how often an answer stated at 90% or more was right.
AI & Agents
7 min read
MCP Servers: Connecting AI Assistants to Your Tools
A practical guide to MCP architecture, local server setup, useful integrations, tool design, permissions, and troubleshooting.
AI & Agents
6 min read
AI Agents in the Enterprise: Where to Start
A practical approach to enterprise AI agents: tool access, memory, orchestration, evaluation, and a bounded first deployment.