PaperlayerView exact revision on arXiv

Source-linked AI summary

Does an Agent's History Tell You When Compaction Will Hurt? A Modest, Bounded Effect on the TRACE Paired-Replay Corpus

Egor Pakhomov, Erik Nijkamp

arXiv:2610.08722v1cs.AIcs.LG

TL;DR

Long-horizon agents often compact context using global rules, and existing history-conditioned methods do not reveal whether a particular compaction hurt. Using paired replays of 590 AppWorld boundaries, the paper finds weak predictive signal: the best extension trigger reaches AUROC 0.66 against a same-boundary replicate’s 0.72.

  • Problem

    Global compaction rules ignore recent agent behavior, while task-success evaluations of history-conditioned methods cannot show whether an individual boundary’s compaction hurt.

    Source: §1 · p. 1

  • Method

    The study replays 590 harness-triggered AppWorld boundaries under pre-compaction and summary contexts, measuring paired differences in next-action burden and testing history-based triggers.

    Source: §2 · p. 2, §4 · p. 3

  • Results

    The best extension trigger reaches held-out AUROC 0.66, versus 0.72 for a same-boundary replicate; the best frozen logistic policy avoids 0.206 of positive-burden boundaries at 0.837 retained.

    Source: §4 · pp. 3–4

  • Takeaways & Limitations

    Released pre-boundary history has limited predictive value, and the evidence does not establish a per-boundary trigger that a harness designer would adopt on this corpus.

    Source: §5 · p. 4

    The release lacks ordered action sequences, token counts, compression ratios and summary text, limiting available pre-boundary observables.

    Source: §2 · p. 2

Abstract

from arXiv · show

Many long-horizon agents compact their context on a global rule, usually a token budget, blind to what the agent was doing. We ask whether the agent's recent behaviour predicts when a compaction will hurt. TRACE's public corpus of 590 harness-triggered AppWorld compaction boundaries replays each boundary from a re-executed prefix state under the pre-compaction context and under the summary, and records the burden of the next actions: calls that error or repeat a call already made. We find that pre-boundary history predicts post-compaction harm only weakly. An internally prespecified contrast by prefix placement is a wide null, and the naive "has-written" label behind it turns out to measure trajectory phase. The best extension-protocol trigger reaches held-out AUROC 0.66 (0.64 on the replicate's own label) against a same-boundary replicate of 0.72; the best frozen, interpretable trigger avoids 21% of harmful (positive-burden) boundaries while keeping 84% of compaction opportunities, and exceeds the random-rule expectation on count but not on burden mass (a post hoc comparison). Whether the best trigger beats a token-budget rule at matched retention cannot be evaluated on the release. We state what corpora should ship to answer it.

1 Introduction

Long-horizon agents commonly compact context at a global budget, without regard to recent behavior, while existing history-conditioned approaches are judged by task success rather than whether a particular compaction hurt. This paper uses TRACE’s paired boundary outcomes to test whether pre-boundary history predicts harm and finds only weak predictive signal.

1 Introduction

  • The paper asks whether recent behavior can predict harmful compactions, since task success cannot identify harm at a particular boundary.TRACE supplies paired outcomes for each boundary, comparing the next actions under pre-compaction context and under the summary.Source: §1 · p. 1
  • The contributions include a prespecified placement test, task-held-out risk ranking, and a frozen-protocol benchmark of fixed learners against a split-half replicate yardstick.The placement test also diagnoses what the naive write label measures.Source: §1 · p. 1
  • The placement contrast is a wide null; held-out AUROC is 0.404–0.635 across original-label cells for interpretable policies (0.55–0.62 on the primary target) and 0.66 for boosted policies, versus 0.72 for a same-boundary replicate.Source: §1 · p. 2

2 Corpus and measurement

The study analyzes TRACE’s 590 harness-triggered AppWorld compaction boundaries, replaying each boundary under both pre- and post-compaction renderings. The release exposes only a limited set of pre-boundary observables.

2 Corpus and measurement

  • TRACE contains 590 boundaries from 275 of 294 trajectories across 147 AppWorld tasks, with one model and a 4,096-token window.The harness triggers compactions; 275 are first compactions and 315 are later ones.Source: §2 · p. 2
  • Each boundary’s prefix is re-executed to rebuild state, then up to five next actions are sampled under PRE and POST renderings.The preregistered primary outcome, dU@3, is the paired POST-minus-PRE difference in union counts of blocked or repeated calls over the first three actions.Source: §2 · p. 2
  • The release provides call and function sets, prefix length, compaction index, task ID and difficulty, and the last three observations, but not ordered actions or summary text.Token counts and compression ratio are also unavailable.Source: §2 · p. 2

3 Is burden localised by pre-boundary placement?

The prespecified comparison of post-write and read-only boundaries does not establish that prior placement predicts compaction burden. A post hoc diagnostic indicates that the coarse write label largely tracks trajectory phase.

3 Is burden localised by pre-boundary placement?

  • The prespecified post-write-minus-read-only dU@3 contrast was −0.062 [−0.209, +0.079], a wide null that met neither informativeness criterion.Its interval includes zero, and its half-width was 0.144 against the prespecified 0.10 criterion.Source: §3 · p. 2
  • Of 494 write-prefixed boundaries, 368 involved only login or session calls and 126 involved a task-relevant state change.The “has-written” label therefore mostly marks a trajectory that has been running for a while; the uncontrolled contrast shrank from −0.22 to −0.004 after adjustment for measured phase variables.Source: §3 · pp. 2–3
  • Table 1 groups boundaries by pre-boundary function set, distinguishing the frozen no-write category from the post hoc substantive-write category.The caption identifies login-only as writes restricted to session endpoints.Source: Table 1 · p. 3

4 Can a history-based trigger beat random selection at matched retention?

History-based triggers rank harmful boundaries only modestly better than a same-boundary replicate, while the best frozen interpretable trigger improves on the random-rule expectation by count but not burden mass. The evaluation scores policies offline at equal-weight compaction opportunities retained.

4 Can a history-based trigger beat random selection at matched retention?

  • Policies block compaction at or above a score threshold; offline value uses observed POST outcomes when allowing a boundary and PRE outcomes when blocking it.Retention counts equal-weight compaction opportunities because token counts are unavailable.Source: §4 · p. 3
  • At 0.837 retained, the logistic policy avoids 0.206 [0.156, 0.256] of positive-burden boundaries, a +0.044 [+0.017, +0.070] count advantage over random expectation.Its burden-mass advantage is −0.001 [−0.034, +0.035].Source: §4 · pp. 3–4
  • The best boosted extension trigger reaches held-out AUROC 0.66 [0.61, 0.70], while a same-boundary replicate scores 0.72 [0.69, 0.75].Under the post hoc same-label comparison, the replicate leads the seventeen-summary trigger by +0.07 [+0.04, +0.11].Source: §4 · p. 4
  • Adding 144 function-and-app indicators yields held-out AUROC 0.66 [0.61, 0.71] and a +0.067 [+0.040, +0.092] advantage over random expectation at 80% retention.Hashed observation text and exact calls did not improve held-out AUROC point estimates beyond 0.61–0.65.Source: §4 · p. 4

5 Discussion and limitations

On TRACE, released pre-boundary features provide only weak separation of harmful from harmless compactions, and the tested evidence does not establish a deployable trigger. The conclusion is limited to one agent, compressor, window, and AppWorld harness, with offline value measured only over the next k actions.

5 Discussion and limitations

  • For the tested representations and learners, the count advantage over a random rule was two to seven points; against the same half-sample label, the boosted seventeen-summary trigger reached AUROC 0.64 versus replicate 0.72.Source: §5 · p. 4
  • A “has-written” label mostly tracks trajectory phase, so its apparent placement effect falls from −0.22 to −0.004 after controlling for measured phase variables.The stump’s shorter-history leaf has the higher positive-burden rate; a short-history, near-empty-observation tree leaf reaches 0.89–0.93 in four folds, mostly on first compactions.Source: §5 · p. 4
  • The study covers one agent model, compressor, and 4,096-token window on harness-triggered AppWorld boundaries, and prices only the next k actions.The release lacks ordered actions, token counts, compression ratio, and summary text; per-boundary deltas use three to five replays per arm.Source: §5 · p. 4

A Reproduction and preregistration record

The analysis uses TRACE’s frozen cohort and paired replay renderings, with channel definitions and released observables determining what can be tested. The record documents reproduction, frozen decision gates, amendments, and analysis provenance.

A Reproduction and preregistration record

  • TRACE documents 590 boundaries from 275 of 294 trajectories across 147 AppWorld tasks, with 4,641 stored rollouts and 20,133 executed steps.The analysis uses the release’s minimax-m3 agent and compressor with a 4,096-token window; it does not use the source’s annotations on 46 compactions.Source: A Reproduction and preregistration… · p. 7
  • At first compaction PRE is the full raw prefix; later PRE combines the previous real summary with the raw suffix, while POST is the recorded summary plus one retained raw turn.No previous summary is fabricated.Source: A Reproduction and preregistration… · p. 7
  • The released features omit ordered actions, token counts, compression ratio, and summary text, limiting trigger inputs to sets, prefix and compaction counts, task metadata, and three observations.Blocked actions are errors under AppWorld’s native contract; refetches repeat an exact action signature already observed before the boundary or within the rollout.Source: A Reproduction and preregistration… · p. 7
  • Four of six shipped scan tables match the Python 3.10.8 fresh scan byte-for-byte; all six match under Python 3.12.13, with no downstream number depending on the tiny differences.The two Python 3.10.8 bootstrap-table differences are at the 10−15 level and attributed to float summation order, not randomness.Source: A Reproduction and preregistration… · p. 7, Table 3 · p. 7
  • The preregistered gates required reproduction, taxonomy coverage and audit, informative placement or policy results, and distinctness from the source; the paper proceeded via G4 after G3 failed.G3’s original-label contrast was −0.062 [−0.209, +0.079], with half-width 0.144; G4’s frozen logistic met its ≤0.15 half-width criterion.Source: A Reproduction and preregistration… · p. 8
  • The frozen gates were internal dated files with hashes available on request, without a third-party registry.The analysis used seeded bootstraps and Python 3.10.8; the pipeline made no model API calls and collected no new rollouts.Source: A Reproduction and preregistration… · pp. 8–9
  • The frozen record preserves amendments and labels: the second label was post hoc, the taxonomy collapsed because three features were unmeasurable, and the frontier uses equal-weight opportunity retention without token counts.Other post-freeze additions included the replicate yardstick and extension protocol, an advantage-over-random comparison, sensitivity analyses, and signed plug-in value; these were not preregistered.Source: A Reproduction and preregistration… · pp. 8–9

B Endpoint method map and audit protocol

The endpoint map derives read/write labels from AppWorld decorator methods, and separate checks compare its classifications with encoded semantics and blind labelers. The blind audit’s relabelings leave the cohort boundaries unchanged.

B Endpoint method map and audit protocol

  • The endpoint map treats GET as read and POST, PUT, PATCH, and DELETE as write; it covers 119 observed functions, including 18 explicit aliases.The map records decorator methods; a boundary is post-write when its pre-boundary function set contains any non-GET function.Source: B Endpoint method map and audit protocol · p. 9, Table 4 · p. 9
  • An encoded semantic check agrees with decorator methods for all 43 observed non-GET functions; read-verb rules cover the other 76 functions.The 50-boundary comparison follows mechanically from the encoded table and pre-boundary function sets, rather than constituting a second audit.Source: B Endpoint method map and audit protocol · p. 10

B.1 Blind two-labeler audit

  • Two map-blind labelers agree with the decorator map on all 119 endpoints, and replacing the map with either labeler moves zero of 590 boundaries.The audit used READ/WRITE and SESSION/SUBSTANTIVE definitions; one hosted model ran at its default temperature because its endpoint rejected temperature 0.Source: §B.1 · p. 10
  • Table 5 covers session endpoints among the 494 boundaries labeled post-write, listing boundaries whose pre-boundary function set contains each endpoint.Source: Table 5 · p. 10

C Cell estimates, exploratory contrasts and regressions

The prespecified original-label placement contrast is imprecise and crosses zero, while post hoc substantive-write contrasts are negative but attenuate toward zero after phase controls. Exploratory cell contrasts are descriptive, and exposure differs only slightly between replay arms.

C Cell estimates, exploratory contrasts and regressions

  • For dU@3, the all-boundary post-write-minus-read-only contrast was −0.062 [−0.209, +0.079] (590 boundaries, 140 tasks); for first compactions only, it was +0.069 [−0.086, +0.225] (275 boundaries, 42 tasks).Both intervals cross zero.Source: C Cell estimates, exploratory… · p. 11
  • Post hoc substantive-write minus no-substantive-write is −0.221 [−0.359, −0.093] overall and −0.389 [−0.546, −0.234] for first compactions.Only five tasks contain both first-compaction cells for the latter contrast.Source: C Cell estimates, exploratory… · p. 11, Table 7 · p. 11
  • With task fixed effects, log prefix steps, and a first-compaction indicator, the post hoc substantive-write coefficient is −0.004 [−0.193, +0.176], versus −0.027 [−0.216, +0.174] for the original label.The uncontrolled binary-regressor OLS coefficient equals the difference in cell means; the horizon-by-channel exploratory contrasts are descriptive and unadjusted for multiplicity.Source: C Cell estimates, exploratory… · p. 11, Table 8 · p. 11
  • Through three actions, valid-action exposure is 2.768 PRE versus 2.761 POST; completion by three is 0.221 versus 0.199.Through five actions, valid actions are 4.249 versus 4.299 and completion is 0.368 versus 0.328; burden includes failure to continue, while reach-k sensitivity is future work.Source: C Cell estimates, exploratory… · p. 11

D The first-compaction confound

First-compaction status is unevenly distributed across the write labels and rises with prefix length, making the naive write label a proxy for trajectory phase. Controlled analyses show a positive first-compaction association, while prefix length itself is uncertain.

D The first-compaction confound

  • 275 of 590 boundaries are first compactions; mean compaction index rises from 1.22 for no-write to 3.14 for substantive-write boundaries.The login-only group averages 1.89; Figure 2 displays paired deltas by these three groups, with the login-only interval derived descriptively.Source: D The first-compaction confound · p. 11, Figure 2 · p. 12
  • Under the original label, 78/96 read-only boundaries (81%) but 197/494 write boundaries are first compactions; under the post hoc label, the corresponding counts are 257/464 and 18/126.The post hoc first-compaction-only contrast rests on five tasks containing both cells.Source: D The first-compaction confound · pp. 11–12
  • In controlled regressions, first-compaction status carries +0.274 [+0.084, +0.474] under the original label and +0.273 [+0.091, +0.466] under the post hoc label.The corresponding log-prefix-step estimates are −0.103 [−0.368, +0.178] and −0.113 [−0.366, +0.158].Source: D The first-compaction confound · p. 12

E Trigger policies: definitions, folds, frontiers and sensitivity

This section defines task-held-out trigger policies over pre-boundary observables and evaluates their operating points, robustness, and descriptive fitted patterns. The reported comparisons include count and burden-mass advantages over a random-rule expectation, with sensitivity analyses qualifying uncertainty.

E Trigger policies: definitions, folds, frontiers and sensitivity

  • Trees use six observables while logistic uses the write label and recent errors; models are scored out-of-fold across five task-held-out folds, and frontier thresholds retain at least 80% of boundaries.The six-feature definitions include prefix length, recent errors, last-observation length, compaction index, and historical exact calls.Source: E Trigger policies: definitions,… · p. 12
  • The signed plug-in value is +5.0 [−6.3, +16.8] burden-steps for frozen logistic, +18.5 [+5.5, +31.1] for the boosted seventeen-summary trigger, and +33.3 [+19.1, +47.8] for the 144-feature trigger.The logistic interval includes zero; the latter two are post hoc comparisons, each blocking 118 boundaries.Source: E Trigger policies: definitions,… · pp. 12–13
  • Nested replay resampling changes 11.5% of boundary sign labels; under this sensitivity, no burden-mass advantage excludes zero among the four Table 9 cells.The nested intervals re-estimate policy uncertainty, not fitting; the models remain task-held-out.Source: E Trigger policies: definitions,… · p. 12, Table 9 · p. 13
  • At the operating points, burden-mass advantage over the random rule is +0.042 for depth-2 and +0.038 for boosted, versus −0.001 for logistic and +0.001 for stump.The logistic and stump intervals include zero; the depth-2 and boosted intervals exclude zero in the task-only comparison.Source: E Trigger policies: definitions,… · p. 13
  • Shorter-history leaves have higher positive-burden rates in the descriptive stump fits, but these are in-sample purities from outcome-selected splits, not a mechanism.The stump’s shorter-history leaves have rates of 0.68–0.71 versus 0.40–0.47; short last observations refine the depth-2 split.Source: E Trigger policies: definitions,… · p. 14
  • The horizon-by-channel AUROC grid and exploratory contrasts are descriptive and were not adjusted for multiplicity.Table 11 refits policies for each horizon-channel target; only the k = 3 union cells match the frozen policies.Source: E Trigger policies: definitions,… · p. 14, Table 11 · p. 14
  • Under stem-clustered sensitivity, the placement contrast interval remains wide at [−0.238, +0.100], while most avoided-fraction intervals at ≥80% retention remain positive.Stem clustering changes no sign conclusion except the boosted burden-mass advantage, whose lower bound shifts from +0.003 to −0.002.Source: E Trigger policies: definitions,… · p. 14
  • Leave-one-task-stem-out mean within-stem AUROCs differ from Table 11’s pooled estimator, and post hoc ≥80%-retained operating points vary by policy.For the post hoc label, depth-2 avoids 0.235 [0.108, 0.253] at 0.805 retained and logistic avoids 0.116 [0.066, 0.181] at 0.886 retained.Source: E Trigger policies: definitions,… · p. 14

F Extension analyses: replicate yardstick and engineered-summary trigger

The extension analyses compare history-based risk ranking with a split-half replicate yardstick and test engineered summaries under a frozen protocol. The strongest summary trigger reaches AUROC 0.66, below the same-label replicate AUROC of 0.72.

F Extension analyses: replicate yardstick and engineered-summary trigger

  • The protocol and interpretation rules for EXT-1 and EXT-2 were frozen before outputs, while amendments D–F were specified post-freeze and are not preregistered.The post-freeze analyses were direction-blind and specified after their motivating result was known.Source: F Extension analyses: replicate… · pp. 14, 16
  • The split-half replicate’s half–half correlation is 0.40 and its nominal full-sample reliability is 0.57 [0.50, 0.62], an extrapolation from unequal halves.The estimated task-level share of delta variance is near zero, with intraclass correlation 0.01 [−0.05, 0.08].Source: F Extension analyses: replicate… · p. 16
  • At k = 3, the same-label replicate AUROC is 0.72, versus 0.64 for the boosted trigger; the paired replicate-minus-trigger difference is +0.073 [+0.037, +0.111].The replicate avoids 0.243 at 0.866 retained, but its few score values prevent a nearer-to-0.80 operating point.Source: F Extension analyses: replicate… · pp. 16–17
  • The best engineered-summary model is a boosted classifier over seventeen history features, evaluated with five-fold task-stem-grouped cross-validation and 20 repetitions.The primary target is 1[dU@3 > 0]; the best cell is selected among four eligible cells and its interval is not selection-adjusted.Source: F Extension analyses: replicate… · p. 17
  • The best cell reaches held-out AUROC 0.66 [0.61, 0.70]; its AUROC is 0.61 at k = 1 and k = 5, 0.56 on blocked burden, and 0.65 on refetch.Distinct prior exact calls have the largest reported permutation importance, with mean AUROC drop 0.045.Source: F Extension analyses: replicate… · pp. 17–18

F.1 Richer representations (144–8,336 features; EXT-3)

Adding function and app indicators to the summary features yields a 144-feature trigger with held-out AUROC 0.66 and a burden-mass advantage that remains positive under replay resampling. The paper also reports that LLMs were used for two audit instruments, while the statistical pipeline made no model calls.

F.1 Richer representations (144–8,336 features; EXT-3)

  • Adding function and app indicators gives held-out AUROC 0.66 [0.61, 0.71] and +0.067 [+0.040, +0.092] over the random-rule expectation at 80% retained.Hashed observation text or exact calls did not improve held-out AUROC point estimates beyond 0.61–0.65.Source: §F.1 · p. 19, Table 17 · p. 19
  • The 144-feature model’s burden-mass advantage remains above zero under replay resampling, with nested interval [+0.026, +0.123].The same-label replicate-minus-trigger gap is +0.061 [+0.025, +0.097].Source: §F.1 · p. 19, Table 18 · p. 19

G LLM-use disclosure

  • The outcomes use the source corpus’s blocked and refetch flags, and the statistical pipeline made no model calls to judge outcomes or generate rollouts.Two LLM instruments supported a semantic-map check and endpoint label audits; authors directed and reviewed model assistance with code and manuscript drafting.Source: G LLM-use disclosure · pp. 19–20
Loading 2610.08722v1…