When an agent breaks, which layer actually broke?

A coding agent invents a tool call. It drops a constraint halfway through a long task. Two agents working the same repo talk past each other. The first instinct is to blame the model; the second is to rewrite the prompt. Both are guesses, and a fix aimed at the wrong layer costs you a release cycle before it tells you it was wrong.

A paper published in July proposes a rule for making that call. It is three weeks old and has no replication — the caveats are at the bottom of this post and worth reading before you adopt it — but the two decision rules at its core are usable today, and the checks that follow from them settle most cases before you touch either the prompt or the model.

The rule

Raj, Gupta, Mahmoud and co-authors set this out in Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures (July 2026). A failure gets classified by which two components were talking when it originated: nine components in three families — the user side (whoever set the task, whoever grades it, third parties), the equipment around the agent (context, memory, tools, and other agents as peers), and the environment (the local shell and filesystem, external services).

A failure is then written as an edge plus a side. tool—model · fault: tool means the tool layer failed to relay an observation — the model reasoned correctly about information it never received. tool—model · fault: model is the case where the observation arrived intact and the model mishandled it. Same symptom, different file, different owner.

Two rules do the real work. Trace back to the earliest failure the run never recovered from, rather than labelling whatever broke last — agents recover from things constantly, and the visible breakage is usually downstream of the one that mattered. And define model-side sharply: a failure is the model’s when a more capable model could have prevented it or recovered from it. If a better model changes nothing, stop looking there.

What it looks like applied

Three failures from our own fleet this month, run through the rule. These are internal incidents — you have no way to check them, so treat them as worked examples rather than evidence. In each one the tempting fix, a better prompt or a stronger model, would have changed nothing.

• A function listing live sessions returned an empty list from a perfectly healthy server. Its caller had no code for “empty” distinct from “I couldn’t tell”, so it deferred recovery indefinitely. Tool-side, on the tool—caller edge; no model was involved at any point.
• A call that merely timed out was classified as “the thing is gone”, and the recovery path recreated sessions that were alive. Environment-side: a timeout is the plainest unknown a distributed system produces, and it was filed as a certainty.
• A monitor reported zero live sessions against a live fleet, because with no locale set tmux replaced the tab delimiter in its own format output with an underscore, and the parser split on tabs. Environment-side, and invisible to every test that ran in an interactive shell.

None of these are exotic. They are the ordinary shape of agent-system bugs: the model is the most expensive component and the least likely culprit.

Six checks before you blame the model

1. Look at the raw tool output, not the rendered version. Encoding, delimiters and locale mangle observations silently, and the model sees only what survives. Byte-dump it once.
2. Count your states. If an observation has three possible outcomes — yes, no, and couldn’t-tell — and your code has two codes for them, you have a bug waiting for load. Timeouts and empty results are the usual casualties.
3. Replay across your compression boundary. If a summarization or context-trim step sits between the fact and the decision, take one failing run and replay it twice from the same starting state — once with compression on, once off — and diff the actions, not the text. That is the paired closed-loop method in Toward Reliable Context Compression for Long-Horizon Agents (August 2026), which finds repeated compression degrades long-run reliability on its own: more blocked actions, more repeated exploration. Evaluate the compression event, not the summary's readability.
4. Attack your own semantic guard with reversals. Build a small mutation set against whatever cosine-similarity gate you trust — negate the verb, swap the actor, invert a quantity, flip an allow to a deny — and see how many it waves through. Similarity Gates Approve Reversals (August 2026) did exactly that to a production drift guard: 56 meaning-reversing mutations, 56 approvals; “withhold the study drug” and “administer the study drug” score 0.9608 cosine.
5. Treat dynamically loaded skills as untrusted code, because that is what they are. Pin versions, diff a skill against its last approved revision before it loads, and run anything unreviewed without network or filesystem access by default — do not rely on the agent to notice. Risk Assessment of Malicious Skill Files in Coding Agents (August 2026) ran 2,826 synthesized malicious skill files against Gemini CLI: 95.5–96.1% of 5,629 runs were exploited, and the agent flagged something wrong in 1.99% of them.
6. Run monitoring code in the environment it will actually run in. cron and systemd strip most of an interactive shell, locale included. A test suite that only asserts the failure branch of a binary check stays green while the check has stopped being able to tell the two branches apart.

Where the taxonomy is weaker than it looks

Worth knowing before you adopt it wholesale. The paper is three weeks old and has no replication. It is explicitly descriptive rather than quantitative — it catalogues 41 named failure modes but says nothing about how often each occurs in the wild, and it is built partly from public write-ups that may omit what you would need to pin a unique root cause.

It also has a structural tilt: 36 of its 41 modes land model-side. That follows from its own “could a stronger model have handled this” definition, not from a measurement that equipment problems are rare — apply it uncritically and you inherit the bias, which is the opposite of what you wanted the taxonomy for.

And if you are tempted to automate the labelling: four frontier models judging 40 worked examples agreed with human ground truth at κ=0.70–0.76 at the edge-and-side level, dropping to 0.62–0.72 accuracy and F1 0.53–0.64 on the finer named-mode level. A 3-of-4 ensemble reaches 83% category precision at 90% coverage; requiring unanimity buys 96% precision but the judges then abstain on nearly a third of cases. Judges are usable for localizing, weak at naming, and no single verdict is ground truth.

The cheap version

You do not need the 41 boxes. Name the edge a failure happened on, trace back to the earliest one nobody recovered from, and ask whether a better model would have changed anything. Most post-mortems that end in “the agent needs better instructions” have not passed that third question yet.

References

1. Raj, Gupta, Mahmoud et al. “Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures.” arXiv:2607.28802, July 2026. arxiv.org/abs/2607.28802
2. “Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability.” arXiv:2608.06503, August 2026. arxiv.org/abs/2608.06503
3. “Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems.” arXiv:2608.10216, August 2026. arxiv.org/abs/2608.10216
4. “Towards a Risk Assessment of Malicious Skill Files in Coding Agents.” arXiv:2608.05223, August 2026. arxiv.org/abs/2608.05223