PaperlayerView exact revision on arXiv

Source-linked AI summary

Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses

Yuvraj Sehgal, Mahsa Panahandeh, Naser Ezzati-Jivan

arXiv:2610.36114v1cs.SEcs.AI

TL;DR

An agent’s context is shaped by harness decisions that can lose tool information before the model sees it. The paper models and measures this pipeline and finds that, when another route to the answer exists, delivery loss costs extra work rather than accuracy.

  • Problem

    An agent reads what its harness places in context, and each step between tool output and model input can lose information.

    Source: §I · p. 1

  • Method

    The paper models context as a five-stage pipeline and measures evidence survival at its boundaries, including delivered fields and yield.

    Source: §II · p. 2

  • Results

    A third of tool results were destroyed before reaching the model, while the context window remained nearly empty.

    Source: §I · p. 1

  • Takeaways & Limitations

    When the agent had another route to the answer, the loss cost extra work, not accuracy.

    Source: §VIII · p. 5

    The evidence is limited to one harness, domain, and model; other harnesses and conventional coding agents were not measured end to end.

    Source: §VI · p. 4

Abstract

from arXiv · show

LLM agents are judged by two numbers: the size of the context window, and how often the final answer is correct. Both numbers miss a third thing: how much of a tool's output is lost before the model ever sees it. We measured this loss in a real tool-using agent. The agent finds the cause of injected faults by reading Linux kernel traces. The context window was almost empty: across 720 runs, the single largest prompt used only 5.75% of it. Yet a third of all tool results (3,827 of 11,189) were cut before they reached the model, and every cut result became broken JSON. The harness cut each result by slicing its JSON text at a fixed length. This does not shorten a result evenly. It deletes whatever data sits near the end. One field reached the model in only 4.7% of calls. A second field, in the very same results, reached the model in 99.3% of calls. Which field held the fault's evidence depended on when the faulty process started: early or late in the trace. The agent's diagnosis accuracy matched this same split. So we ran controlled experiments to test one question: did the data loss cause the drop in accuracy? This paper makes four contributions. First, we model an agent's context as a five-stage pipeline, with a measured failure at each stage. Second, we replay 864 real tool results, including one from a production trace where the shipped cut deletes all 128 records of the fault's evidence. Third, a controlled experiment shows that the cut's real cost appears first as extra work, not as wrong answers: 43% more tool calls, twice the computation, and yield falling from 0.61 to 0.31. The agent still finds the right answer when it has another way to reach it. Fourth, a survey of seven agent frameworks finds two that cut tool output the same unsafe way. Whether the data cut truly causes the accuracy drop is still an open question.

I. INTRODUCTION

The paper measures information lost between tool output and model input, a failure that window size and final-answer accuracy alone do not capture. In a Linux-trace fault-diagnosis agent, position-dependent delivery loss aligned with a split in diagnosis scores; the paper’s controlled experiment and broader causal question are treated separately.

I. INTRODUCTION

  • Prior work has examined context size, memory, harness code, and tool use, but the paper identifies a gap in measuring what data survives between tool production and the model.A run log records what a tool delivered, not necessarily what it produced.Source: §I · p. 1
  • Across 720 runs, the largest prompt used only 5.75% of the 400k-token context window.The maximum prompt was 23k tokens.Source: §I · p. 1
  • A third of tool results were destroyed before reaching the model because fixed-point JSON cuts removed later fields rather than shortening results evenly.One field reached the model in 4.7% of 742 calls, while another field in the same payload reached it in 99.3%.Source: §I · p. 1
  • Faults visible only in the late list scored 0–24 out of 60, while faults that spawned a new process scored 40–55 out of 60.The evidence location depended on whether the faulty process started during the trace or was already running.Source: §I · pp. 1–2
  • The paper contributes a five-stage context-pipeline model, replay evidence, a controlled experiment, and a survey finding two of seven frameworks use the same unsafe cutting approach.The controlled experiment found a compensation tax when another route to the answer existed: extra work, not reduced accuracy.Source: §I · p. 2

II. CONTEXT AS A PIPELINE

The paper treats context as a five-boundary pipeline controlled by the harness, with distinct failure modes at each boundary. It proposes measuring evidence survival alongside accuracy, while distinguishing loss frequency from effects on final answers.

II. CONTEXT AS A PIPELINE

  • The five boundaries are produced-to-delivered, act, retain, transfer, and persist; S1 and S4 can destroy evidence, S2 can stop work, S3 can add cost, and S5 is disabled by design.The pipeline is unified by harness decisions about what survives to the next step, not by a shared failure mode.Source: §II · p. 2
  • With fixed key order and prefix slicing, field survival depends on position: a late field beyond the cut is gone before model attention can reach it.The passage notes that models already attend unevenly to positions inside the window.Source: §II · p. 2
  • Boundary failure counts alone do not show whether the final answer changes; the controlled experiment reintroduces one boundary failure at a time with other conditions held fixed.Table I reports how often boundaries fail, while the causal question is tested separately.Source: §II · p. 2, Table I · p. 2
  • The proposed measurements count delivered top-level tool-result fields at S1 and transferred findings or computations at S4, alongside yield as findings recorded divided by computations run.Yield is a measurable proxy for work reaching the decision, not a measure of relevance.Source: §II · p. 2

A. In the field

The deployed evaluation measures extensive tool-result loss, with every cut result breaking. A replay illustrates the positional delivery cliff at production scale, where the late field is either delivered whole or lost whole.

A. In the field

  • 3,827 of 11,189 tool results were cut across 720 runs, and every cut result broke.One field reached the model in 4.7% of 742 calls, while another field in the same results reached it in 99.3%.Source: A. In the field · p. 2
  • At the production trace’s 33,370 characters and 128 records, prefix slicing predicted all-or-nothing delivery or loss of the late field, while row-aware cutting kept every field.The replay used the real tool’s result shape and the shipped 6,000-character cap; no model was involved.Source: Figure 1 · p. 3

B. Mechanism and replays

Replays show that prefix slicing deterministically breaks oversized results, and a production-scale trace lost every late-list record containing a quiet culprit’s evidence.

B. Mechanism and replays

  • 864 recorded results were replayed; every over-budget result broke under prefix slicing, while none broke under the row-aware cut.The replays covered results from 31 runs.Source: B. Mechanism and replays · p. 3
  • Only 2.5% of calls were affected; breakage hit 50% of process-listing calls, 43% of analysis outputs, and 1% of small summaries.The defect was quiet on most calls but affected the longer listing and analysis outputs.Source: B. Mechanism and replays · p. 3
  • 0 of 128 late-list records reached the model in a real 21-container trace, deleting the quiet culprit’s entire evidence class.The payload was 33,370 characters, 5.6× the shipped budget, and was already broken.Source: B. Mechanism and replays · p. 3

C. The same cut in other frameworks

A survey of seven frameworks found positional tool-output cutters in two, while a structure-aware line cutter shows a safer pattern exists.

C. The same cut in other frameworks

IV. THE CONTROLLED EXPERIMENT

A controlled experiment varied tool-result delivery while holding the agent, task, tools, and model fixed. Truncation increased work and lowered yield, while alternate routes could preserve correct answers; production-scale replication remained confounded by bundled fixes.

IV. THE CONTROLLED EXPERIMENT

  • The experiment held the trace, fault, prompt, model, and tools fixed while reintroducing delivery defects one at a time, with predictions recorded before each wave.Source: §IV · p. 3
  • Across 145 runs, the corrected scorer required naming the culprit in the answer field; median input use was 2.5% of the 400k-token window.The runs paired five recordings per fault class with three repeats; the first mention-based scorer was replaced after an audit.Source: §IV · p. 3
  • 43% more tool calls (76 vs. 53) and roughly doubled computations (36 vs. 18) accompanied yield falling from 0.61 to 0.31 under S1 reversed versus repaired.Findings stayed at 11, while the order-only control returned to yield 0.61.Source: Table II · p. 4, §IV · p. 4
  • Despite the culprit’s list being absent from every reversed-key payload, the agent named the loudest process in 15/15 runs via sorted-table rows.That process accounted for 14.0% of kernel events.Source: §IV · p. 4
  • At production scale, repaired runs named the throttled service in 3/6 cases versus 0/5 under the defect, but delivery and step-budget fixes were bundled.Page-on-request never fired because every result fit on the first page; the authors do not treat this as proof the cut alone drives the field-wide split.Source: §IV · p. 4

V. WHY NOBODY NOTICES

The defect is easy to miss because latency and window occupancy look reassuring, while procedural guidance can improve results without fixing delivery.

V. WHY NOBODY NOTICES

  • Model waiting took a median 84% of wall-clock time, leaving the defect’s correctness and token costs barely visible in the latency metric teams watch.The measured range was 80–91%, including network and queueing.Source: §V · p. 4
  • A barely used context window gives no hint that evidence was discarded upstream.Source: §V · p. 4
  • A written procedure reached the same verdict as free exploration at 35% of the tokens, improving outcomes by requiring less evidence rather than better delivery.Source: §V · p. 4

VI. THREATS TO VALIDITY

The evidence is limited to one harness, one domain, one model, and kernel traces; the field accuracy split remains correlational. The controlled experiment also uses repeated recordings rather than independent fault instances.

VI. THREATS TO VALIDITY

  • The accuracy split in field results remains correlational, not causal.The evidence covers kernel traces and has not measured other harnesses end to end or a conventional coding agent.Source: §VI · p. 4
  • Five recordings per class were paired with repeats of those same recordings, not five independent fault instances.Source: §VI · p. 4
  • The registered experiment reintroduces a defect the authors wrote, rather than evaluating a repair story.Source: §VI · p. 4
  • The experiment uses smaller traces and a scaled budget, while production-scale replay retains the shipped budget.Source: §VI · p. 4

VII. FUTURE PLANS

Future work proposes a controlled test of whether delivery loss causes accuracy changes, a broader harness survey, routine loss reporting, and mechanical CI checks.

VII. FUTURE PLANS

  • A planned controlled experiment varies only delivery across complete, truncated, reversed-key-order, row-aware, and paging-on-request conditions while holding other factors fixed.It would track evidence visibility, payload validity, alternative recoverability, and final answers across independent instances.Source: §VII · p. 4
  • The planned deeper survey would classify cutters across eleven harnesses and thirteen scaffolds, then replay them against the delivery cliff.Source: §VII · p. 4
  • The authors propose reporting a loss profile beside accuracy to test whether equal-accuracy harnesses differ in yield and predict harder-task performance.Source: §VII · pp. 4–5
  • Mechanical failures such as invalid cut results, exhausted step budgets, and dropped computed results could be caught in CI without a model.The replication package includes drivers, scorer, results, replay corpus, and registered plan.Source: §VII · p. 5

VIII. CONCLUSION

The conclusion frames agent context as pipeline output rather than window size, emphasizing that tool-result loss can consume work even when answers remain recoverable.

VIII. CONCLUSION

  • A third of tool results were destroyed while the context window was 93% empty, with evidence deleted by position rather than value.Source: §VIII · p. 5
  • Where the agent had another route to the answer, delivery loss cost extra work rather than accuracy.Source: §VIII · p. 5
  • Harness delivery choices are measurable, but the conclusion says none reports them.Source: §VIII · p. 5

Related papers

Loading 2610.36114v1…