Environment Reproducibility for Debugging Agent Failures

Agent failures do not announce themselves. A coding agent can return a clean success status while having retrieved the wrong document, called the wrong tool, or quietly invented a file path that does not exist, and nothing in that response tells you which one happened. That is the actual subject of this piece: not that agents fail, but that their failures leave no trace for conventional debugging to find, which makes reconstructing the exact environment an agent ran in the only way to find out what went wrong.

Why coding agents fail silently

Traditional software debugging starts with a stack trace or an error code, something the system hands you that points at the failure. Agents built on large language models do not produce that kind of evidence when they misread retrieved context, pick the wrong API, or hallucinate a result that looks plausible, and this absence is what Braintrust's 2026 agent debugging guide identifies as the core distinction between agent failures and conventional software failures.

Three failure modes make this worse in practice. A tool can respond with a changed schema, partial data, or a timeout, and instead of raising an exception, the model improvises around the broken response and keeps working with corrupted context. Output quality can also degrade in small increments rather than collapsing all at once, so the agent keeps looking functional while its decisions get steadily worse. And in a multi-step workflow, a small error at one step carries forward and compounds with the next, so by the end of the chain the result is wrong in ways that don't map back cleanly to any single step.

None of these modes trip an alarm. Each one leaves the system reporting success. So the debugging task is not locating an error signal, because there isn't one: it's reconstructing a decision path, call by call, that produced a wrong answer while reporting a right one. That reconstruction only works if you know, precisely, the environment the agent was operating in when it made each decision, not just the code it eventually wrote.

What "environment" means for a coding agent run

Most engineers, asked what "environment" means for a coding agent, will answer with a dependency list. That answer misses most of the surface area that actually matters. The environment of a coding agent run is the OS image, the exact repository commit it was pointed at, the dependency lockfile, the version of the agent client itself, the identifier of the model it called, the schemas of every tool available to it, the credentials it had access to, and its own configuration files, all at once, and a change in any single one of these between two runs can produce two different failures from what looks like the same task.

Recent research gives this idea a formal name. Vangala and Malik, in a September 2026 paper (arXiv:2610.00425), frame environment specification, the accurate identification of what dependencies are actually required to run generated code, as its own measurable axis of code quality, distinct from whether the code works, and one that current benchmarks simply do not test for. The gap this reveals is not small. An earlier study by the same group found that projects which declare a handful of dependencies actually require 13.5 times as many packages to run at all, a hidden layer of complexity that turns straightforward reproduction into an extended debugging exercise.

Tool schemas sit inside this same territory, and they fail in their own particular way. Amazon's engineering team has documented that poorly defined tool schemas and vague semantic descriptions cause agents to pick the wrong tool entirely, invoking APIs that have nothing to do with the task at hand. The n8n Vector Store Question Answer Tool began generating invalid JSON schemas for function calling, a concrete version of this failure. Nothing about that failure was a bug in the agent's reasoning. It was a change at the environment layer, invisible until something downstream broke.

There's a limit to what capturing all of this buys you, and it should be stated rather than papered over. A Terraform apply run twice against the same configuration produces the same infrastructure state, every time. An agent run twice against the same configuration and the same input is not guaranteed to produce the same output, by design. Capturing environment state does not remove that variance. It narrows the space you have to search when something goes wrong, which is a different and more modest claim, but still the one that makes debugging possible.

How bad the reproducibility gap is in practice

The scale of this problem is documented with data, not speculation. When another developer takes AI-generated code and tries to run it in a clean environment, failure is the ordinary outcome, and the cause is almost never a short list of missing packages. It's the environment as a whole being underspecified.

Vangala et al. found that nearly a third of AI-generated projects fail to execute without someone stepping in to fix something by hand, with the failure rate varying sharply by language: Java projects failed most often, Python least. The failures that do occur are mostly syntax and path errors rather than dependency problems. They trace back to malformed syntax, file paths that point nowhere, and variables the generated code never initialized. A runtime dependency count that runs more than an order of magnitude past what the agent declared means the environment specification the agent hands you is incomplete from the start.

The September 2026 follow-on paper adds a finding that should worry anyone betting on newer models to fix this on their own. Across different agents asked to solve identical tasks, agreement on the dependency set required was in the low single digits as a percentage, and newer agents showed no meaningful improvement over older ones. This is a structural property of how these systems generate code, not a problem model scale is closing, and reproduction rates bear this out from an entirely different angle: research reproduction benchmarks like AutoMat and PaperBench show even the strongest agents topping out just above half on AutoMat and dropping to 24.4% on PaperBench, with the failures concentrated not in the code itself but in everything that has to be true around the code for it to run. The Replit and Railway cases show the human cost of the technical gap.

Why benchmarks have not surfaced this problem until recently

This problem sat largely unmeasured for a straightforward institutional reason: the benchmarks engineering teams relied on were built to test whether generated code works, not whether the environment it depends on can be rebuilt by someone else. That design choice, made for reasons that made sense when benchmarks were first built around functional correctness, quietly hid environment reproducibility as a category of failure.

The consequence follows directly. An agent can score well on a standard coding benchmark while being reliably unable to produce something another developer can actually run, because functional correctness and environmental portability are measured on separate axes and only one of them was ever being tested. A benchmark that checks whether the code passes its tests tells you nothing about whether the code will even start on someone else's machine.

The Sim Reproducible Coding-Agent Benchmark, published September 20, 2026, is the clearest attempt at fixing this design gap. Under its rules, every reported run has to come with a downloadable task fixture, an isolated execution log, a machine-readable result, the tool version, the model version, the runtime, and a cost record attached to it. If an agent cannot be tested under those conditions, the benchmark reports its result as "Not tested" rather than filling in an estimate.

This benchmark design specifies what proper measurement requires: a pinned OS image, a pinned repository commit and dependency lockfile, the exact agent client and version, and the exact model identifier. That list is, nearly point for point, what production teams need to capture if they want to debug an agent failure after the fact rather than guess at it.

What environment reproducibility requires from an engineering team

Treating environment state as something you capture at the moment an agent runs, rather than something you try to reconstruct later from memory and git history, means recording six things every time: the OS image, the repository commit, the dependency lockfile, the agent configuration version, the model identifier, and the credentials and tool schemas that were in scope for that session.

Each one of these, left uncaptured, produces its own specific blind spot. Without a pinned OS image, the runtime an agent executed in and the runtime someone uses to debug it afterward can be different machines wearing the same name. Without a pinned commit, the repository has usually moved on by the time anyone looks, and the defect that caused the failure may no longer exist in the code being inspected. Without a lockfile, the packages that actually ran diverge from whatever the agent declared, the dependency explosion Vangala et al. documented means the packages that actually ran diverge from whatever the agent declared, and no one can say which set caused the behavior in question. Without a pinned agent configuration, a change to a file like CLAUDE.md or agents.yaml between one run and the next can alter the agent's behavior with no visible signal marking that anything changed. Without a recorded model identifier, a vendor updating a model under its existing name can change outputs for identical inputs, and there's no way to tell that from the session record. And without a record of credential scope, a session cannot be audited, and a destructive tool call cannot be traced back to the permissions that allowed it.

The fix for the configuration layer is to treat it the same way teams already treat application code. CLAUDE.md, AGENTS.md, and agents.yaml files, checked into the repository and reviewed through the same process as any other code change, each get a commit hash, and that hash can be pinned to the record of a specific session. This is the agents-as-code pattern, applied specifically to reproducibility rather than to deployment convenience.

Tool schema versioning is the layer teams skip most often, and the n8n incident shows exactly why that is a mistake: a schema changing underneath a running agent produces a failure that is debuggable only if someone recorded the schema's state at the moment of the failure, not its state today. Vangala and Malik's three-layer framework gives teams a concrete structure for auditing all of this: what an agent declared it needed, what it actually installed, and what was strictly necessary for the task to succeed are three different sets, and the question worth asking is not just the first one.

Full session logs as the mechanism for replaying failures, not just inspecting their aftermath

A captured environment tells you where the agent was running. It does not tell you what the agent did while it was there. Debugging an agent failure needs both, because the actual failure usually lives inside a sequence of decisions the agent made, not in a misconfigured environment variable sitting next to them.

Braintrust's debugging guide separates three distinct levels of visibility into a system: monitoring, which shows aggregate health across many requests; observability, which shows a trace at the level of a single request; and debugging, which shows root-cause analysis at the level of a single step, with the ability to replay it. Only the third level actually lets a team fix the failure rather than just notice it happened.

A full session log, built to support that third level, has to contain every model call along with its inputs and outputs, every tool invocation along with its parameters and the response it got back, every retrieval step, timing data, the cost of each call, and the intermediate decisions the agent made along the way. This is agent-trace-level logging, a different thing entirely from an HTTP access log.

The AgentS4D safety benchmark makes clear why completion status is not enough on its own: across thousands of runs, a large share came back both apparently complete and unsafe. A status code showing success cannot distinguish those two outcomes. Only a log of what actually happened during the run can.

Logs also need to answer to a person. "The AI agent did this" is not an auditable statement by itself, and every session needs to trace back to the specific person or trigger that started it. That attribution matters for the same reason credential scope matters: without it, there is no way to hold anyone accountable for what a given session did.

The payoff for doing all of this is visible at the point a failure needs to become permanent institutional knowledge rather than a one-off fire drill. The debugging workflow Braintrust describes ends with turning the failure into a case in an automated evaluation suite, and that step only works if the original trace exists to serve as ground truth for the regression test built around it. Without the log, there is no case to add, only a memory of something that went wrong once.

The infrastructure of isolated sandboxes

Environment capture and session logging both depend on a condition that is easy to take for granted: each agent run has to execute inside its own isolated sandbox, separate from every other run. A shared execution environment undermines both practices at once, because the state a team captures is no longer a clean record of what one agent did. It is contaminated by whatever every other process running in that same environment did at the same time.

When two agents run against a shared filesystem or a shared set of installed packages, a change one of them makes, a dependency version it installs, a file it writes, a schema it mutates, becomes part of the environment the other one inherits. A pinned OS image and a pinned lockfile mean very little if the actual machine they describe was never really isolated to begin with. The six-point capture described earlier only produces a trustworthy record if nothing outside that one session touched the environment while it ran. That is what isolated sandboxing provides: a boundary around each run tight enough that the captured state and the logged trace describe one agent, one task, one set of decisions, with nothing else bleeding into the record.