PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Budgeted Context Projection with Unexecuted Companion Runs

Guangzhe Zhang

arXiv:2609.31381v2cs.AI

TL;DR

Sequential evaluation can suppress a paired continuation after a cap, leaving quality comparisons incomplete. This paper audits that observation process in a recorded ReVerPi campaign, retaining unknown outcomes and selector dependencies. The resulting bounds vary with the task frame, and the evidence is limited to one adaptively assembled campaign.

  • Problem

    Context projection may reduce individual requests yet require more searching, reading, and model turns, while sequential evaluation can leave a paired continuation unobserved after a cap.

    Source: §1 · p. 1

  • Method

    The paper reconstructs a recorded ReVerPi campaign’s observed and suppressed outcomes and applies finite-record bounds while preserving fixed-selector and reference-policy dependencies.

    Source: §1 · p. 2

  • Results

    Across 27 boundaries, projected-minus-full success ranges from −9 to +1 tasks; the frozen selector’s difference is [−3, 0] after excluding a task with an unresolved platform premise.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Evaluation reports should retain every boundary and endpoint state, preserve unknown outcomes and policy relationships, check task premises, and separate completion bounds from success-conditioned resource use.

    Source: §1 · p. 2, §7 · p. 13

    The evidence is one adaptively developed campaign with one proxy-reported model identity and one stochastic continuation per observed arm, so it does not establish prevalence across evaluation systems.

    Source: §6.1 · p. 12

Abstract

from arXiv · show

Context projection can shorten individual requests while changing whether an agent finishes within its budget. We examine how sequential evaluation obscures this trade-off when a capped first continuation prevents its companion from running. In a recorded ReVerPi source-reading campaign, 15 pairs with two final answers yield 12 historically scored successes per arm. Retaining all 27 intervention boundaries distinguishes observed failures from ten unexecuted companions and bounds projected-minus-full success between $-9$ and $+1$ tasks. Under the archived scoring contract, a frozen projection selector has a success difference from full context of $[-3,0]$; outside four fitting tasks, it is $[-4,-1]$ across 23 boundaries. Excluding one task whose platform premise is not established by retained actor-input evidence changes the nonfitting range to $[-3,0]$ across 22 boundaries, so strict inferiority is not robust to that exclusion. Among eleven historically joint-success pairs, projection uses 25% fewer aggregate logical tokens but more tokens for the median pair and 55 rather than 35 suffix requests. These retrospective results concern one adaptively assembled campaign, not population performance. The case motivates accounting that retains every boundary, preserves unknown outcomes and policy dependencies, checks task premises, and separates bounded completion from success-conditioned resource use.

1 Introduction

This paper examines how sequential evaluation can leave paired agent outcomes unobserved when a capped first continuation suppresses its companion. In a recorded ReVerPi campaign, preserving those unknown outcomes and task-premise concerns changes what can be concluded about projection versus full context.

1 Introduction

  • Context projection may reduce individual requests while requiring additional searching, reading, and model turns before an answer is produced.The paper frames the evaluation around downstream behavior and end-to-end resource use, not text removed alone.Source: §1 · p. 1
  • The evaluation uses read-only questions about Python package sources and controller-scored JSON answers, with two sequentially run continuations allocated twelve new requests each.A wrong final answer counts as completed for scheduling; a capped continuation without an answer is distinct from its suppressed, unexecuted companion.Source: §1 · p. 1
  • Across 27 intervention boundaries, projected-minus-full success ranges from −9 to +1 tasks when capped arms count as failures and unexecuted companions remain unknown.The 15 pairs with two final answers each show 12 correct answers per arm.Source: §1 · p. 1
  • Under the archived scoring contract, outside four fitting tasks, even the most favorable completion leaves the rule at least one success behind full context; excluding one task yields [−3, 0] over 22 nonfitting boundaries.Source: §1 · p. 2
  • The study reconstructs observed and suppressed outcomes, applies partial-identification bounds while preserving selector/reference-policy dependencies, and separates whole-frame quality from resources on the joint-success subset.Its reporting procedure retains boundary ledgers, endpoint states, denominators, and policy relationships; it is not a new missing-data theorem or validated selection algorithm.Source: §1 · p. 2
  • The evidence comes from one adaptively developed campaign and does not establish how prevalent the observation failure is across other evaluation systems.Recalculation is supported by the supplementary package, but independently auditing all answers or repeating the private service requires evidence not included.Source: §1 · p. 2

2 Related Work

Related work covers agent interfaces and context management, compression under interactive behavior, joint quality-and-resource evaluation, and methods for incomplete outcomes. This paper situates its question at the intersection of paired continuations and scheduler-suppressed observations.

Agent interfaces and context management

  • Prior systems study agent-computer interfaces, condenser or memory-tier approaches, and archiving, recall, and efficiency; ReVerPi’s intact history, compact request view, and addressable archive fit this family.Examples include SWE-agent [Yang et al., 2024], OpenHands [Wang et al., 2025; OpenHands contributors, 2026], MemGPT [Packer et al., 2023], and SoL-Pi [Liu et al., 2026].Source: §2 · p. 2

Compression and closed-loop behavior

  • Compression research includes prompt reduction, evidence-location effects, interactive trajectory changes, feedback-guided compression, and paired closed-loop continuations.The narrower question here is what remains identifiable when a scheduler never executes a continuation; paired continuation and post-compression work are not presented as new.Source: §2 · p. 2

Quality and resource accounting

  • Prior evaluation work argues for measuring accuracy and cost together and examining end-to-end cost; this paper treats token count as distinct from a monetary proxy.Its additional focus is that outcome-dependent execution changes which quality comparisons exist before resource costs are summarized.Source: §2 · p. 2

Incomplete outcomes

  • The analysis uses standard worst-case partial-identification bounds and distinguishes observed joint successes from a population always-success stratum under repeated stochastic sampling.The finite-record application requires no missing-at-random extrapolation, randomized assignment, or population noninferiority claim.Source: §2 · p. 2

3 Tasks and the Recorded Execution Protocol

The protocol evaluates read-only source questions by replaying shared prefixes into full and projected views, while a historical sequential scheduler can suppress a companion continuation. It defines projection, the recorded selection rule, and the analysis frames used for the campaign.

3.1 A read-only source question and its evidence

  • Success is a correct source answer under the stated scorer, not completion of an unrestricted coding task.Questions concern fixed subsets of Python package sources; the adapter permits allowlisted reading and archived observations but not shell execution, file writing, or network tools.Source: §3.1 · p. 3

3.2 Projection and exact archive access

  • Projection replaces eligible observations with a content hash, retrieval instructions, and a 1,024-byte combined head/tail excerpt while retaining original bytes in the archive.Search and exact-interval recovery share an allowance of three successful calls; search results can themselves provide usable content.Source: §3.2 · p. 3

3.3 A shared prefix and an outcome-dependent stop

  • The historical scheduler runs a companion only when the first arm returns a final answer; Figure 1 depicts the changed request view and this outcome-dependent stop.A cap suppresses the companion, so the figure’s paired comparison is not observed in that case.Source: Figure 1 · p. 4

3.4 The frozen rule and the analysis frames

  • The recorded rule selects projection when sealed-history size meets a 93,641-byte threshold, with four fitting pairs used in its offline selection and retrospective frames defined by exclusions.The fitting records did not validate a harm detector or uniquely establish history length as the preferred feature; exclusions are not untouched test sets.Source: §3.4 · p. 5

4 Outcome and Resource Accounting

The section distinguishes observed failures from unexecuted outcomes, preserves dependencies in policy comparisons, and limits resource comparisons to fully observed jointly successful continuations. It treats the resulting bounds and costs as properties of recorded runs, not population performance.

4.1 Known failure and unknown outcome

  • Observed rejected answers or capped branches count as failures, whereas an unexecuted companion remains unknown; observed capped branches used twelve suffix requests.A stop before the cap would require additional endpoint handling when caused by token, time, integrity, or permission limits.Source: §4.1 · p. 5
  • Table 2’s nested run sets describe available information, not independent sample sizes; ten single-arm integration runs are separate from 76 paired/capture plans.Source: Table 2 · p. 5
  • Worst-case bounds for the finite recorded success contrast have attainable endpoints and are not confidence intervals for a sampled population.The bounds encode known successes, known failures, and unknown outcomes as binary intervals.Source: §4.1 · p. 6

4.2 Preserve the selector’s dependence on its reference

  • When the selector is zero, its difference is zero even if the full arm is unexecuted; separately bounding marginal success counts would discard this shared-outcome dependence.This applies the same binary bounds to the fixed policy rather than introducing a new general bounding algorithm.Source: §4.2 · p. 6

4.3 Resource use on a fully observed success subset

  • All twelve incomplete pairs have an observed zero, so none can enter the identified 11-pair joint-success subset; its cost comparison is not a population always-success effect.The subset describes realized jointly successful continuations.Source: §4.3 · p. 6
  • Each arm’s logical expenditure includes its shared prefix once, while paired acquisition expenditure counts that prefix once across both arms; failed-arm tokens are not cost to a correct answer.Source: §4.3 · p. 6
  • The median describes the middle pair and the geometric mean symmetric multiplicative change, so they need not agree; no dollar-saving claim is made because the private bill and price schedule are unavailable.Reported tokens include uncached input, cached input already included in total input, and output.Source: §4.3 · p. 6

5 Results

The completed-pair tie masks uncertainty across the full intervention frame, and the frozen selector’s recorded contrast changes under task-premise sensitivity. Resource findings depend on which outcomes are observed and whether the measure is success-conditioned.

5.2 The completed-pair tie does not determine whole-frame quality

  • Each arm scored 12/15 in completed pairs, while first-arm counts were nine correct, three wrong, and three capped for full versus five correct and seven capped for projected.The deterministic first-period counts compare adaptively obtained cases, not a randomized treatment effect.Source: §5.2 · p. 7
  • Across 27 boundaries, projected-minus-full success ranges from −9 to +1 tasks; the frame includes 15 completed pairs and 12 stopped pairs, with ten caps suppressing companions.The corresponding percentage-point range is −33.3 to +3.7; unexecuted arms remain unknown rather than being scored as failures.Source: §5.2 · p. 7, Table 3 · p. 7, Figure 2 · p. 7

5.3 A full-frame comparison of the frozen selector

  • The frozen selector’s success difference from full context is [−3, 0] across 27 boundaries and [−4, −1] outside its four fitting tasks.The contrasts use the archived scoring contract and fixed recorded continuations.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Sensitivity to an unresolved task premise

  • Excluding pathspec-util removes the established strict nonfitting deficit, because its retained task premise is not established by actor-input evidence.This is a retrospective task-exclusion sensitivity, not a repaired experiment or a newly selected rule.Source: §5.4 · p. 8, Table 5 · p. 9

5.5 Observed policy expenditure and fitting participation

  • On 13 observed nonfitting comparisons, the selector spends 122,682 more logical tokens (+8.6%); across all 17 comparisons, it spends 16,240 more (0.9%).Excluding pathspec-util leaves 12 comparable nonfitting cases with two failures per policy; full uses 1,295,456 tokens and the selector 1,264,711.Source: §5.5 · p. 9, Table 6 · p. 9

5.6 Joint success reduces aggregate tokens but adds interaction

  • Among 11 joint successes, projected uses 949,774 versus full’s 1,267,036 logical tokens, a 0.750 sum ratio, while the median ratio is 1.292 and seven pairs increase.The aggregate saving is sensitive to more-itertools-recipes: removing its 382,538-token saving changes the sum ratio to 1.077.Source: §5.6 · p. 9, Figure 3 · p. 10, Table 7 · p. 10
  • Across the reported pairs, projected uses 55 suffix requests versus 35, a 57% increase; it uses more requests in eight pairs, fewer in two, and equally many in one.Source: §5.6 · p. 9

5.7 Successful archive access can precede a cap

  • On pathspec-util, full returns an accepted answer after three suffix requests, while projected reaches its cap after quota refusals and continued requests without a final answer.The record shows successful archive retrieval before quota refusals, but does not establish that quota alone caused failure or that retrieved fragments were sufficient.Source: §5.7 · pp. 10–11, Figure 4 · p. 11

6 A Reporting Procedure and Its Limits

The section proposes a reporting procedure that makes observation and policy comparisons explicit, then qualifies what this campaign can establish and what its supplementary materials support.

6 A Reporting Procedure and Its Limits

  • The procedure preserves every intervention boundary, allocates both continuations independently of the first arm’s cap, and records distinct endpoint states and expenditures.It also retains frozen-selector decisions and fitting participation, reusing a shared reference-arm outcome when the selector chooses it.Source: §6 · pp. 11–12
  • Report whole-frame quality ranges before conditional resource summaries, with denominators and requests, token categories, and cache categories stated separately.The procedure also separates development from confirmation by preserving revisions and post-outcome decisions; retuning creates a new intervention, not a repair to an old outcome.Source: §6 · p. 12

6.1 Limits of interpretation

  • The campaign is adaptively assembled, uses one stochastic continuation per observed arm and a proxy-reported model identity, and cannot establish how often the observation failure occurs elsewhere.Its read-only adapter is not the complete Pi product, and deterministic arm order plus possible provider or cache-state interactions constrain intervention and population claims.Source: §6.1 · p. 12
  • One pathspec task has a platform-dependent premise not established by retained actor-input evidence, and the reported deficit depends on retaining that pair.The trace does not isolate a unique mechanism; a confirmatory study should freeze tasks, model, scorer, configuration, and selector, execute arms independently, and distinguish sources of variation.Source: §6.1 · p. 12
  • The proposed methodological test applies stopping and filtering rules offline to complete pairs, but does not reproduce cache, timing, or order effects from actually skipping a continuation.The proposed repeated sampling and quota or budget-visibility tests have not been conducted.Source: §6.1 · p. 12

6.2 Data and code availability

  • The supplementary package supports recalculation of reported numerical inputs without provider calls, but does not include raw requests, responses, private settings, or all original task and answer evidence.Supplied digests establish consistency of supplied bytes, not chronology, backend weights, or billing.Source: §6.2 · pp. 12–13

6.3 Use of artificial intelligence tools

  • Codex assisted with manuscript drafting, editing, and analysis checking, but supplies no independent experimental evidence and does not replace author responsibility.Source: §6.3 · p. 13

7 Conclusion

The conclusion shows that sequential stopping can suppress companions and distort paired comparisons. It reports bounded success differences and a resource trade-off, while limiting its claims to one retrospective campaign.

7 Conclusion

  • With shared full outcomes preserved, the frozen selector’s projected-minus-full success range is [−3, 0] overall and [−4, −1] across 23 nonfitting boundaries.Excluding the task with an unresolved platform premise changes the ranges to [−2, +1] over 26 boundaries and [−3, 0] over 22 nonfitting boundaries.Source: §7 · p. 13
  • Across all 27 boundaries, projected-minus-full success is bounded between −9 and +1 tasks, rather than being determined by the completed-pair comparison alone.The historical completed-pair comparison has twelve correct answers per arm, while ten companions are omitted after first-arm caps.Source: §7 · p. 13
  • On the eleven historically joint-success pairs, projection uses fewer aggregate tokens but has greater median expenditure and more suffix requests.Source: §7 · p. 13
  • The case motivates retaining every boundary, distinguishing failure from nonexecution, preserving policy dependencies, checking task premises, and separating bounded completion from success-conditioned resource use.These retrospective findings concern one adaptively assembled campaign, not population performance.Source: §7 · p. 13

A Finite-Frame Bounds and Their Witnesses

The appendix derives finite-record bounds while preserving shared outcomes, then supplies rowwise witnesses and complete recorded tables for boundaries and selected summaries.

A Finite-Frame Bounds and Their Witnesses

  • Finite-frame bounds sum per-run extreme contributions, while selector terms choosing full vanish and projected selections use the same binary construction.The rowwise witness table lists all eight projected selections, three with unknown full outcomes.Source: A Finite-Frame Bounds and Their… · p. 16, Table 8 · p. 16
  • The frozen selector’s sharp projected-minus-full success range is [−3, 0]; outside fitting tasks it is [−4, −1] over 23 boundaries and remains [−4, −1] over 22 nonfitting-source boundaries.The selector comparison preserves outcomes shared with full context; subtracting marginal ranges instead gives the nonsharp outer bound [−7, +4].Source: A Finite-Frame Bounds and Their… · p. 16
  • The task-exclusion sensitivity removes pathspec-util’s fixed −1 contribution, while the endpoint-validity sensitivity changes only its full-score interval from [1, 1] to [0, 1].The analyses are separately labeled and do not modify archived historical labels.Source: A Finite-Frame Bounds and Their… · p. 16

B Complete Recorded Tables

  • The eleven-member joint-success subset is identified from realized continuations because each stopped pair has an observed zero; it is not a population stratum under future resampling.Table 12 gives complete token categories on this subset, and Table 9 lists all fifteen completed pairs with expenditure and score fields.Source: A Finite-Frame Bounds and Their… · p. 16, Table 9 · p. 17, Table 12 · p. 18

A Finite-Frame Bounds and Their Witnesses

  • After a first-arm cap, the companion has zero observation probability under the schedule, so inverse-observation weighting divides by zero and regression would extrapolate into that region.The stated bounds leave that extrapolation unspecified.Source: A Finite-Frame Bounds and Their… · p. 16

B Complete Recorded Tables

  • Table 10 records all twelve stopped paired/capture runs, and Table 11 records six single-arm boundary runs plus the earlier paired boundary for the same task.Table 10 marks unexecuted arms and distinguishes first-arm caps from later projected caps; Table 11 identifies repeat trajectories, not paired counterfactuals.Source: Table 10 · p. 17, Table 11 · p. 18

C.1 The recorded search and its objective

The recorded search reconstructs a selector from fitting records using a source-group objective and a candidate enumeration. Its winning rule is not uniquely supported, and the observed-harm filter rejects no candidates.

C.1 The recorded search and its objective

  • The history feature is computed from the UTF-8 byte length of a canonical serialization of the sealed boundary’s source.messages.The serializer uses compact JSON separators, sorted keys, ensure_ascii=False, and allow_nan=False.Source: §C.1 · p. 19
  • The candidate search enumerates constant policies and feature thresholds; its feature grids contain 4, 4, 2, 3, and 4 values, yielding 36 candidates.The 32-value grid cap is inactive for this search.Source: §C.1 · p. 19
  • The objective averages projected/full token ratios across fitting source groups, rejecting candidates with any observed full success and projected failure.With four fitting sources, the recorded threshold has L = 0.84730644845, while the ratio of fitting token totals is 0.736987371.Source: §C.1 · p. 19
  • The search selects history_bytes ≥93,641, but last_cached_fraction ≤0.8012967413783466 ties it and makes the same four fitting decisions.Enumeration order chooses index 14; the data do not establish that the history feature is uniquely better.Source: §C.1 · p. 19
  • The archived runner records the winning rule, not per-candidate scores, so the search trace is reconstructed rather than an originally logged artifact.Version identity and deterministic reconstruction establish internal consistency, not prospective registration or untouched later development cases.Source: §C.1 · p. 19

C.2 Hindsight threshold comparisons

Hindsight threshold comparisons and strict-format scoring examine selector outcomes on the observed campaign records. Strict scoring changes some labels and joint-success counts, while the resource pattern persists under both scoring contracts.

C.2 Hindsight threshold comparisons

  • Fourteen leave-one-source-out folds select thresholds of 95,519 twelve times, 96,013 once, and 93,641 once; concatenated held-out choices have four failures versus three for the frozen rule.The folds reuse the observed development frame and are not independent confirmation.Source: Table 13 · p. 19, §C.2 · p. 20

C.3 Strict-format scoring

  • Strict scoring gives four full failures, six projected failures, and four frozen-rule failures on 17 comparable runs, versus three, five, and three historically.Every full-frame selector contrast in Table 4 is unchanged.Source: §C.3 · p. 20
  • Joint success falls from 11 to 10 pairs under strict scoring, while aggregate-token ratios remain below one, median ratios above one, and projected suffix requests remain more numerous.Table 14 summarizes resource sensitivity under the historical and strict-format scores.Source: §C.3 · p. 20, Table 14 · p. 20

D Original Task and Recorded Projection

The task section presents a recorded pathspec question and projection example while distinguishing the task-specific prompt from the full actor input. The retained prompt does not establish the operating platform, so one answer premise requires separate scrutiny.

D Original Task and Recorded Projection

  • The actor receives a rendered prompt plus adapter notices, allowed source paths, system instructions, and tool schemas; the displayed task-specific question is not the complete actor input.The rendered prompt and plain-text version are separately preserved.Source: D Original Task and Recorded Projection · pp. 20–21
  • The pathspec task asks for five match/normalization outputs and the package’s major version, returned as one JSON object.The recorded program tests two docs paths, two root-path patterns, and normalize_file on a backslash path.Source: D Original Task and Recorded Projection · p. 21
  • The controller reference gives root_txt=true and sub_root_txt=false, but the actor receives the prompt and allowlisted source rather than that reference answer.The retained prompt fields do not explicitly state the operating platform, so the normalization field requires separate scrutiny.Source: D Original Task and Recorded Projection · p. 21
  • The first projected request replaces a 25,551-byte observation with metadata, instructions, and a combined 1,024-byte head/tail excerpt; the original remains retrievable by handle.The excerpt allowance excludes metadata and instructions, and successful search/read operations share a three-call quota.Source: D Original Task and Recorded Projection · p. 21

E Source Inventory and Record Availability

The source inventory and retained records support campaign accounting but leave limits on independent response rescoring. Audit-rejected plans remain in the census yet fall outside the paired-outcome frames.

E Source Inventory and Record Availability

  • The source-level auditor reports 73 passes among 76 paired/capture runs; three allowlist-intent rejections remain in the census.Of the passes, 71 match a submitted audit and two have no submitted counterpart; interface compliance and accounting completeness are separate checks.Source: E Source Inventory and Record… · p. 22
  • The three audit-rejected runs stop during capture without sealed boundaries and enter none of the 27-boundary, 17-comparable, 15-completed, or 11-joint-success frames.Excluding them therefore changes none of the paired success or cost results, though they remain in campaign accounting.Source: E Source Inventory and Record… · p. 22, Table 16 · p. 23
  • The distributed package supports arithmetic checks and provided examples, but lacks raw requests, databases, provider settings, invoices, and every task/answer pair for independent full rescoring.Table 15 describes the fixed source subsets for 76 paired/capture plans.Source: Table 15 · p. 22, E Source Inventory and Record… · p. 23
  • Six single-arm boundaries are repeated trajectories, not additional paired outcomes, and do not provide missing counterfactuals at earlier captured prefixes.Earlier single-prefix experiments and calibration illustrations are not pooled into the primary comparisons.Source: E Source Inventory and Record… · p. 23

Related papers

Loading 2609.31381v2…