PaperlayerView exact revision on arXiv

Source-linked AI summary

What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents

Shuyang Zhang, Jianshuo Chang

arXiv:2609.39989v1cs.AI

TL;DR

Component replacements can change an agent’s trajectory, so task-level benefit and the contribution of local decision quality require different evidence. This critical scoping review maps studies and comparison records, then evaluates selected cases. Its findings support different conclusions across comparisons and do not treat online execution or aligned local and task gains as proof of attribution.

  • Problem

    Evaluating a component replacement requires distinguishing task-level benefit from whether local decision quality explains the outcome, because the decision can alter later actions and observations.

    Source: §1 · pp. 1–2

  • Method

    The critical scoping review maps 348 studies and examines 90 comparison records through eight purposively selected cases focused on comparison conditions, measurements, controls, and remaining explanations.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    The selected comparisons support different inferences: Outcome Monitors reports package benefit, First-chunk selection a local improvement against an offline proxy, and ECT fewer premature terminations with completion non-inferiority, not superiority.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Eight conditional reporting items identify evidence needed for claims about net task benefit, decision-quality attribution, trigger effects, efficiency, or transfer.

    Source: §4.2 · p. 10, §6 · p. 11

    The purposively selected cases illustrate relationships across different tasks, budgets, and baselines but do not estimate their prevalence across the literature.

    Source: §5 · p. 10

Abstract

from arXiv · show

Background. A component replacement in a language-model agent changes an execution trajectory, potentially altering later observations, resource use, and recovery opportunities. Different evidence is needed to assess its task-level benefit and the contribution of local decision quality. Methods. This critical scoping review maps 348 studies and examines 90 comparison records: 88 from 40 included studies and two from supplementary studies. Eight purposively selected cases structure the synthesis around the replaced decision, executed conditions, measurement comparability, controls, and remaining explanations. Results. Of 222 studies reporting local decision metrics, 142 also report measured task endpoints and 49 report proxies. These counts identify studies that report both types of measurement, without establishing that the measurements come from matched comparisons. Outcome Monitors reports a package-level completion gain whose attribution to detector quality remains limited; First-chunk selection reports a local improvement assessed against an offline proxy endpoint; Evidence-Carrying Termination reports fewer premature unsupported terminations and completion non-inferiority, without establishing completion superiority. Cross-case analysis identifies three candidate mechanisms involving recovery and disruption, intervention timing, and downstream use. Attribution and deployment depend on the comparison controls, label definitions, and information available to the controller. Conclusions. The review distinguishes the task-level benefit of a component replacement from the contribution of local decision quality and derives eight claim-specific reporting items. Neither online execution nor simultaneous gains in local and task metrics alone establish that better local decisions explain the task-level gain.

1 Introduction

The review distinguishes whether a component replacement improves task outcomes from whether local decision quality explains that improvement. It examines specific replacement comparisons and situates its claim-focused analysis alongside broader reviews of agent evaluation, routing, and component attribution.

1 Introduction

  • Outcome Monitors reports improved completion for a monitor-and-recovery-receipt package, but its comparisons leave detector quality’s contribution unresolved [1].The separate always-warn comparison varies recovery tools within a different warning envelope, making recovery affordances relevant to explaining the main gain.Source: §1 · p. 1
  • A replacement’s task-level benefit and the contribution of local decision quality are separate questions; aligned local and task improvements alone do not establish attribution.The review connects decisions to subsequent agent actions and examines controls for information, intervention frequency, budget, and recovery policy.Source: §1 · pp. 1–2
  • The review maps 348 papers and analyzes 90 comparison records through eight selected cases, assessing supported claims, candidate explanations, and claim-specific reporting guidance.The records comprise 88 from 40 included studies and two supplementary records; the review does not estimate a population rate of local–task decoupling.Source: §1 · p. 2

1.1 Relationship to adjacent reviews

  • Adjacent reviews address evaluation methods and deployment, routing objectives and controls [4], and agent capability, decision-focused learning, and causal attribution [5] [6] [7]; this review traces claims to specific comparisons.Its worked cases connect those methodological concerns to reported measurements and controls.Source: §1.1 · p. 2

2 Methods

The review scopes separable decision components in multi-step language-model agents, maps included studies, and codes their evaluation evidence. It then extracts replacement comparisons and synthesizes selected cases without pooling effects or estimating prevalence.

2.1 Scope and eligibility

  • The review includes identifiable decision-component replacements for tool-using or environment-acting multi-step agents, while excluding several broader or non-comparable study types.Offline studies are eligible when designed for this setting, but their evidence is distinguished from closed-loop execution; pre-task agent configurations are coded separately from execution decisions.Source: §2.1 · p. 2

2.2 Information sources and selection

  • 348 studies were identified through searches of arXiv, OpenAlex, Semantic Scholar, ACL Anthology, OpenReview, and Scopus, with source-specific screening and verification summarized in Fig. 1.The searches were conducted on 24–25 September 2026; Fig. 1 cautions that source-specific checks and paired extraction differed in depth and coding rules, rather than forming uniform independent double-coding.Source: §2.2 · p. 2, Figure 1 · p. 3

2.3 Study-level coding and verification

  • Study-level coding records each study’s decision point, controller, and evaluation fields; local metrics measure decision quality, while task endpoints and evaluation levels distinguish outcome and execution evidence.Task endpoints are coded as measured success or completion (Y), proxy (P), or none (N); evaluation levels range from static/offline prediction (E0) through replay or simulation (E1) to online closed-loop comparison (E2). Co-reporting does not establish a matched comparison.Source: §2.3 · p. 3
  • Verification covered 140 studies with source-specific checks of varying depth; the remaining 208 underwent model-assisted full-text extraction, followed by adjudication and correction of disputed codes.Agreement was calculated from valid pre-adjudication output pairs, separately from final adjudicated codes.Source: §2.3 · p. 3

2.4 Comparison-level extraction and synthesis

  • The comparison-level dataset contains 90 replacement–comparator records from 42 studies; eight purposively selected cases structure the synthesis rather than estimate prevalence.The records retain comparison conditions and measurements when reported; the review does not pool effects or estimate a corpus-wide rate of local–task decoupling.Source: §2.4 · p. 4

3 Results

The review maps local decisions and evaluation levels, then examines selected comparisons to distinguish task-level benefit from attribution to local decision quality. Its central finding is that co-reported local and task measures do not establish matched comparisons, while case evidence supports conditional interpretations rather than general causal claims.

3.1 Corpus composition and evidence levels

  • 229 studies were coded E2, 61 E1, 54 E0, and 4 insufficiently specified; these evaluation levels do not establish the validity of causal interpretations.The corpus includes a range of decision types and controller types, with selection/shortlisting the largest decision category (84 studies).Source: §3.1 · p. 4

3.2 Co-reporting does not establish a matched relationship

  • 142 of 348 studies reporting local metrics also reported measured task endpoints, and 49 reported proxies; co-reporting does not show that measurements came from matched comparisons.Local metrics appeared in 222/348 studies (63.8%); the evidence gap concerns comparability and attribution, not simply metric presence.Source: §3.2 · p. 4
  • 178 studies tested multiple backbones, but those counts do not establish transfer of a fixed controller.After excluding 11 studies near the scope boundary, the sensitivity analysis still found that a majority reported a local metric.Source: §3.2 · p. 5

3.3 Coding checks

  • 82.7–91.4% agreement across valid pre-adjudication pairs indicates coding consistency under recorded rule versions; corpus statistics use adjudicated and corrected tables.The agreement range is across four evaluation fields.Source: §3.3 · p. 5

3.4 What the comparison evidence supports

  • Eight purposively selected cases assess the replaced decision, executed comparison, measurement comparability, controls, and remaining explanations; they are not independent effect estimates or a prevalence sample.The synthesis distinguishes net benefit of a replacement (A) from attribution to local decision quality (B); favorable system comparisons can support A while leaving B unresolved.Source: §3.4 · p. 5
  • Removing checkpoint rollback left containment at 99.5% but reduced simulated task success from 83.3% to 2.6%, illustrating that containment need not preserve task progress.Recovery and disruption are candidate pathways through which interventions can change outcomes; the selected cases do not establish a general harmful effect of accurate monitoring.Source: §3.4 · pp. 6–7
  • Timing and reversibility shape intervention opportunities: a late warning may arrive after recoverable state is lost, while RestartSmart’s comparison shows recovery policy matters in its coding setup.The studies suggest reversibility as a candidate moderator but do not isolate its effect.Source: §3.4 · p. 7
  • Better local ranking or availability need not translate into downstream use, and HYSET’s matched-feedback comparison still changes architecture and retrieved context.The cross-case synthesis treats these as candidate pathways, not effects isolated by all studies; efficiency gains also need not correspond to higher completion.Source: §3.4 · p. 7

4 Discussion

The discussion separates evidence that a replacement benefits task outcomes from evidence that local decision quality explains the benefit. It uses case comparisons to identify attribution limits and proposes claim-specific reporting and research guidance.

4.1 From net benefit to attribution

  • Outcome Monitors supports completion benefit for the monitor-plus-receipt package, but detector-quality attribution remains unresolved because the ablation also changed warning counts and recovery content.The learned versus schema-only detector comparison reports completion of 25/80 versus 21/80 (+5.0 points, p = .42), an inconclusive direct comparison.Source: §4.1 · p. 7
  • Budget allowances and precomputed outcomes do not match realized resource use or feedback under a changed online policy; replay after model switching can reuse invalid states.The replay limitation was observed in two quantized configurations within one low-success pilot.Source: §4.1 · p. 8
  • First-chunk selection shows a local improvement but no significant localization difference against its offline proxy, and does not measure completion after executing the changed agent.The evidence neither establishes equivalence nor shows that the full task fails to improve.Source: §4.1 · p. 8
  • Evidence-Carrying Termination reduces premature unsupported termination from 40/66 to 0/66, while its completion result supports non-inferiority, not completion superiority.The deterministic tools and substitute model limit transfer.Source: §4.1 · pp. 8–9

4.2 Eight reporting items, conditional on the claim

  • The reporting guide makes the needed comparison depend on the claim: task improvement need not include a local-quality metric, but maintained performance requires a non-inferiority or equivalence design.The eight items are proposed guidance, not a validated scoring instrument or universal prerequisites.Source: §4.2 · p. 9, Table 3 · p. 9
  • Local scores require scrutiny of label meaning and deployment inputs: successful reference trajectories may not identify necessary actions, and a proxy controller’s same-task online performance was not independently verified.The examples include TabAgent and AgentRouter ([33]; [34]).Source: §4.2 · pp. 9–10
  • The proposed guidance treats baseline reruns as a safeguard against stochastic variation without specifying a validated minimum, and varies evaluation needs by claim rather than decision category.Monitoring and gating often call for recovery and trigger analysis; routing may need information and allocation controls; selection may need measurement of downstream use.Source: §4.2 · p. 10

4.3 Research opportunities

  • The research agenda proposes controlled tests of recovery and timing, evaluation-mode rankings, selection versus consumption, and frozen-controller transfer across backbones or environments.These are proposed comparisons derived from evidence gaps; the review does not execute them.Source: §4.3 · p. 10

5 Limitations

The comparison synthesis uses selected cases with different tasks, budgets, and baselines, illustrating relationships between local improvements and task-level gains without estimating how prevalent those patterns are.

5 Limitations

  • The selected cases span different tasks, budgets, and baselines, limiting the scope of cross-case comparison.The synthesis illustrates relationships between local improvements and task-level gains but does not estimate their prevalence across the literature.Source: §5 · p. 10
  • The three candidate mechanisms organize observed relationships between local decisions and task outcomes rather than estimating their prevalence.The discussion develops associated attribution and deployment questions.Source: §5 · p. 10
  • The synthesis does not estimate how prevalent the observed local-improvement and task-gain relationships are across the literature.Its cases are selected, not a corpus-wide prevalence analysis.Source: §5 · p. 10

6 Conclusion

The review finds that replacement comparisons support different kinds of claims, depending on their evidence. It maps the landscape, synthesizes candidate mechanisms, and offers conditional reporting items to keep claims within evidential scope.

6 Conclusion

  • Outcome Monitors supports a package-level benefit, First-chunk selection a local improvement with an inconclusive proxy-endpoint effect, and ECT fewer premature unsupported terminations with completion non-inferiority, not superiority.These comparisons support different inferences rather than contradictory conclusions about local metrics.Source: §6 · p. 10
  • The comparison-specific evidence supports distinct conclusions rather than a single general claim about the value of local metrics.Outcome Monitors, First-chunk selection, and ECT differ in what their comparisons establish.Source: §6 · pp. 10–11
  • The review maps 348 studies, synthesizes three candidate mechanisms, and derives eight conditional reporting items for assessing component-evaluation claims.Claims about net task benefit, decision-quality attribution, trigger effects, efficiency, or transfer should remain within the supporting evidence.Source: §6 · p. 11

7 Use of AI

Language models assisted several review tasks but were analytical tools, not authors or independent scholarly sources.

7 Use of AI

  • Language models assisted literature retrieval, evidence extraction, adjudication, claim verification, and image generation, serving as analytical tools rather than authors or independent scholarly sources.The paper identifies their use across these review activities.Source: §7 · p. 11

A Search, coding, and sensitivity details

The appendix documents search and selection procedures and defines the study-level coding rules used to characterize decisions, endpoints, evaluation levels, and backbone coverage.

A.1 Search and selection records

  • The Scopus export yielded 2,985 records; after documented removals, 2,729 new records remained, and the complete abstract-stage ledger records 65 included, 177 background, 349 excluded, and five duplicates.A random recheck of 50 initially unselected records found no additional eligible study, but does not establish perfect recall; the month-level cutoff could not be verified.Source: §A.1 · p. 11
  • Four coding tables contain 348 unique normalized study identifiers: 134 first-round arXiv, 125 second-round arXiv, 24 supplementary, and 65 Scopus studies.Some supplementary search stages are documented only in logs, limiting record-level verification.Source: §A.1 · p. 11

A.2 Study-level coding rules

  • The coding distinguishes ten decision points, including selection, routing, termination, monitoring, assistance, budgeting, and task-level configuration.Controller types include rules or algorithms, classifiers, probes, language-model judges, smaller fine-tuned models, retrievers, and reinforcement-learning or bandit policies.Source: §A.2 · p. 11
  • Task endpoints are coded as measured success, pass, or completion; proxies; or no reported endpoint, while evaluation levels distinguish offline, replay or simulated, and online closed-loop comparisons.E2 requires the compared decision policy to be evaluated in the closed loop; executing branches to collect outcomes alone does not qualify.Source: §A.2 · pp. 11–12
  • Multiple-backbone coding records tests on different agent backbones but does not establish transfer of a fixed controller.Benchmark, controller-size, or model-role changes alone do not qualify as multiple-backbone testing.Source: §A.2 · p. 12

A.3 Verification, adjudication, and agreement

The appendix separates verification, agreement, and adjudication updates from comparison-level extraction, then limits interpretation to purposively selected cases.

A.3 Verification, adjudication, and agreement

  • Source-specific checks documented varying-depth verification for 140 studies, while two model-assisted coding streams extracted four evaluation fields for the other 208.Full-text extraction records cover the latter group, but checks did not constitute complete re-adjudication of eligibility.Source: §A.3 · p. 12
  • A retrospective version-2 rule check reviewed potential differences in 155 of 238 earlier records and made 108 field corrections across 85 studies.One field remained unresolved, and a separate ledger preserves original outputs and agreement calculations.Source: §A.3 · p. 12
  • Agreement describes received valid output pairs, not independent human double coding under uniform conditions; historical instruction versions and operator independence could not be fully confirmed.Re-extraction from corrected sources does not retroactively create independent pairs.Source: §A.3 · p. 12
  • In the 200-study update audit, adjudicated values differed from initial codes in 80 local-metric fields (40.0%), 60 task endpoints (30.0%), 36 evidence levels (18.0%), and 71 multiple-backbone fields (35.5%).These are recorded adjudication updates, not corpus-wide abstract-coding error rates or isolated effects of full-text access; at least one field changed in 148 studies.Source: §A.3 · p. 12
  • The current tables retain 16 cells marked insufficient information; agreement statistics and update counts describe separate samples from completed source checks..Source: §A.3 · p. 13

A.4 Comparison-level extraction and case selection

  • Comparison-level records preserve conditions, measurements, denominators, and evaluation details; a comparison without a local baseline cannot establish local improvement.Proxy endpoints remain distinct from task completion, and a non-significant difference is not coded as equality or non-inferiority.Source: §A.4 · p. 13
  • The comparison collection contains 90 records from 42 studies, including 88 records from 40 included studies and two supplementary records outside the inclusion denominator.The supplementary examples address gold-history versus autonomous-workflow measurement and budget-allocation controls.Source: §A.4 · p. 13
  • Eight purposively selected cases illustrate candidate mechanisms and attribution questions but do not exhaust co-reporting studies or estimate prevalence.Seven cases are included studies and one is supplementary.Source: §A.4 · p. 13

A.5 Descriptive summaries and sensitivity

The sensitivity analysis excludes identified scope-boundary studies, while the accompanying summaries caution that verification histories and publication labels limit interpretation.

A.5 Descriptive summaries and sensitivity

  • Pooled frequencies summarize current coding, but differences in rule versions and checking depth mean strata do not reflect a uniform measurement process.Venue labels and future volume years do not establish publication by the search date, so no unverified formal-publication count is reported.Source: §A.5 · p. 13
  • Excluding 11 identified scope-boundary studies leaves 337 studies, with 217 local-Y, 139 local/measured co-reporting, 47 local/proxy co-reporting, and 223 E2 codes.A majority still reported a local metric.Source: §A.5 · p. 13
  • This purposive sensitivity analysis is not a complete eligibility audit.Source: §A.5 · p. 13

B Included-study category index

The category index groups included records by primary decision point, with membership defined by study-level coding.

B Included-study category index

Loading 2609.39989v1…