PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents

Yanze Jiang, Mingxuan Li, Yuhao Wang, Shengfang Zhai, Jiaheng Zhang

arXiv:2608.21027v1cs.AI

TL;DR

Long-horizon LLM agents need runtime intervention that not only detects poor actions but also supports recovery, while existing methods rely on another solver or a task-capable critic. COTA uses a tiny comparator to compare sampled alternatives with the actor’s proposal, trains it from same-prefix counterfactual branches, and returns winning alternatives as non-binding advice. Across WebShop, ALFWorld, and tau^3-Retail with three actors, COTA improves all nine evaluation settings, showing that constructive intervention can work with an auxiliary model substantially weaker than the actor.

  • Problem

    Constructive runtime intervention requires useful recovery direction, but existing approaches place substantial task-solving capability in an expert handoff or corrective critic.

  • Method

    COTA trains a tiny comparator on same-prefix counterfactual branches to compare sampled alternatives with the actor’s proposal, then returns preferred alternatives as non-binding advice for actor replanning.

  • Results

    COTA improves all nine actor–environment combinations and achieves the best performance in all nine settings among the compared intervention baselines.

  • Takeaways & Limitations

    Effective constructive runtime intervention can narrow the auxiliary model’s role from solving to comparing while leaving planning and execution to the stronger actor.

Abstract

from arXiv · show

LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making. As these agents operate over longer horizons, runtime intervention offers a way to improve reliability without retraining the underlying actor. Failure detection alone is insufficient. Effective intervention must also provide a useful direction for recovery. Existing approaches often rely on an expert solver or a critic that generates task-specific corrections, incurring either the cost of another capable solver or the capacity demands of a task-capable critic. We introduce Comparison-Only Tiny Advisor (COTA), a comparison-only framework for constructive runtime intervention. In COTA, a tiny comparator judges whether sampled alternatives lead to better continuations than the actor's proposal, and repeated comparisons determine when intervention is warranted. We train the comparator using pairwise supervision constructed from same-prefix counterfactual branches. Preferred alternatives are returned as non-binding advice, leaving the original actor to replan. Across WebShop, ALFWorld, and tau^3-Retail with three actors, COTA improves all nine evaluation settings and outperforms the compared baselines. These results show that constructive runtime intervention can remain effective even when the auxiliary model has substantially weaker task-solving capability than the actor.

1 INTRODUCTION

COTA addresses the difficulty of constructive runtime intervention by replacing auxiliary task solving with local comparison. A 0.5B comparator improves every evaluated actor–environment combination while leaving replanning to the original actor.

  • Runtime intervention must both detect poor actions and provide a useful direction for recovery over long-horizon tasks.Binary failure detection alone is insufficient when actions can alter the environment, consume interaction budget, and make recovery harder.
  • Existing constructive approaches either hand control to another capable solver or require a critic to generate task-specific corrections.Both pathways place substantial task-solving capability in the intervention component.
  • COTA uses repeated pairwise comparisons between the actor’s proposal and executable alternatives to decide when intervention is warranted.Winning alternatives are returned as non-binding advice, after which the actor replans.
  • Same-prefix counterfactual branches provide pairwise supervision aligned with the comparison required at runtime.The training design avoids requiring the comparator to independently solve the task.
  • A 0.5B comparator improves all nine actor–environment combinations, including substantially stronger Qwen3.6-35B-A3B and DeepSeek-V4-Flash actors.The evaluation covers three interactive environments and three actor families, with modest online overhead.

2 PROBLEM FORMULATION

The paper formulates runtime intervention around a frozen actor’s decision state and pre-execution proposal. Constructive intervention withholds a proposal, supplies advice, and lets the actor replan before execution.

  • The frozen actor interacts with an environment by observing, reasoning, proposing actions, and receiving subsequent observations.The decision state includes the task and the actor-visible trajectory prefix.
  • Runtime intervention occurs after the actor proposes an action but before that action is sent to the environment.The intervention gate either executes the original proposal or withholds it; the response after withholding is left abstract here.
  • The actor-conditioned action value measures the completed-trajectory return after taking a current action and returning control to the same actor.This captures downstream consequences under the frozen policy.
  • Constructive intervention withholds the actor’s proposal and additionally provides an advice action that offers a better local direction.The actor receives non-binding advice, replans, and produces a new proposal before execution.

3 METHODOLOGY

COTA replaces difficult absolute action-value estimation with candidate-relative comparison. It samples alternatives, aggregates comparator judgments into an intervention gate, and trains those judgments from shared-prefix counterfactual branches.

  • 3 METHODOLOGY: Candidate-relative domination measures the probability that a sampled executable alternative has higher continuation value than the actor’s proposal.This rank-based quantity does not require the candidate distribution to contain an expert or globally optimal action.
  • 3.1 COMPARISON-ONLY TINY ADVISOR: The learned component performs only local action comparison, while the frozen actor remains responsible for candidate generation, planning, replanning, and execution.This separates the advisor’s narrow role from task solving and absolute value estimation.
  • 3.1 COMPARISON-ONLY TINY ADVISOR: COTA samples K executable alternatives, compares each with the actor’s proposal using a tiny comparator, and rejects the proposal when at least R alternatives defeat it.If the gate fires, the predicted winners are ranked and the highest-ranked candidate is returned as advice.
  • 3.2 COMPARATOR TRAINING: Same-prefix counterfactual branches generate pairwise labels by executing sibling actions from the same decision state and comparing their continuation returns.Shared prefixes and the same continuation actor isolate the downstream consequence of changing the branch-point action.
  • 3.3 STATISTICAL ACCURACY: The winner-count gate estimates the domination rate using candidate sampling, with error separated into comparator error and finite-candidate Monte Carlo error.When the domination rate is sufficiently far from the threshold R/K, the learned gate matches the oracle gate with high probability.

4 EXPERIMENTS

COTA is evaluated across diverse interactive environments, actors, baselines, and ablations to test performance, efficiency, intervention design, and action diversity. It improves all nine actor–environment settings while preserving actor-mediated replanning.

  • 4.1 SETUPS: COTA is evaluated on WebShop, ALFWorld, and τ 3-Retail with Qwen3-8B, Qwen3.6-35B-A3B, and DeepSeek-V4-Flash actors.The study uses task-level splits, reports environment-specific metrics, and compares several inference-time intervention baselines.
  • 4.2 IMPLEMENTATION DETAILS: 55k, 23k, and 9k pairwise supervision examples are constructed for WebShop, ALFWorld, and τ 3-Retail, respectively, from same-prefix branches.Branches vary only the first action before returning control to the same continuation actor, with labels based on downstream environment outcomes.
  • 4.3 MAIN RESULTS: COTA achieves the best performance in all nine actor–environment settings and improves the corresponding original actor.For Qwen3-8B, WebShop reward rises from 0.3960 to 0.5630, ALFWorld success from 82.84% to 90.30%, and τ 3-Retail success from 37.50% to 45.00%.
  • 4.3 MAIN RESULTS: 1.38× average episode time is COTA’s online overhead, with seven of nine settings below 1.5×.Larger overheads occur on τ 3-Retail, while useful intervention can occasionally reduce the number of interaction steps.
  • 4.4 ABLATION STUDY: 57.46% versus 2.24% is the Qwen3-8B ALFWorld result when constructive intervention replaces forced execution under absolute-Q supervision.The same trend holds on τ 3-Retail and for Qwen3.6; returning advice and allowing the actor to replan is better than directly executing the auxiliary model’s action.
  • 4.4 ABLATION STUDY: 90.30% versus 57.46% is the Qwen3-8B ALFWorld result when same-prefix pairwise comparison replaces absolute-Q estimation.The pairwise objective matches the advisor’s comparison decision without requiring absolute long-horizon value estimation, while constructive intervention delegates replanning to the actor.
  • 4.5 ACTION DIVERSITY: 3.71 distinct actions from four candidates versus 2.28 from eight actor samples shows that candidate generation exposes greater local action diversity.Candidate utilization is 92.7%, compared with 28.5% unique-action utilization for repeated actor sampling.
  • 4.5 ACTION DIVERSITY: COTA transfers candidate diversity to actor replanning without forcing convergence to the candidate distribution.DeepSeek-V4-Flash adopts recommendations in 29.2%–32.9% of replans across environments, while retaining the original proposal in 49.6%–63.6%.

5 RELATED WORK

Related work frames runtime intervention as external monitoring or correction and examines weaker models in auxiliary roles for stronger actors. COTA aligns with these directions while focusing on comparison-only intervention.

  • Runtime Intervention: Runtime-intervention research includes human intervention, supervised blockers, shielding, and agent-specific guardrails that monitor or correct proposed actions.These approaches place an external mechanism between the learner’s proposal and environment execution.
  • Weak Critics for Strong Actors: Weak-for-strong research uses weaker models as verifiers or workflow assistants rather than requiring them to solve the full task independently.Examples include small verifiers trained against stronger provers and a 7B meta-agent optimizing workflows for stronger executors.

6 CONCLUSION

The conclusion argues that constructive runtime intervention does not require an auxiliary task solver. COTA combines comparison-only advice with actor-controlled replanning and performs best across the evaluated settings.

  • 6 CONCLUSION: A 0.5B comparator trained on same-prefix branches identifies better local alternatives while the stronger actor handles generation, replanning, and execution.The conclusion attributes the design to narrowing the auxiliary model’s learned task to local comparison.
  • 6 CONCLUSION: COTA achieves the best performance across all evaluated settings, with ablations supporting pairwise comparison and actor-mediated replanning.The conclusion presents both components as central design choices for effective constructive intervention.

A ADDITIONAL ANALYSIS

This analysis clarifies how same-prefix branches estimate actor-conditioned continuation values and how finite sampling affects pairwise supervision. Branch-return differences isolate downstream consequences while value estimation can remain unbiased even when thresholded labels are noisy.

  • Same-prefix branch construction: Same-prefix branches restore state st, execute an action, then return control to the frozen continuation actor π.This construction holds the preceding state and continuation policy fixed.
  • Continuation-value estimation: Y(st, a) samples a trajectory return after taking action a at st and continuing with π.
  • Continuation-value estimation: bQM(st, a) converges almost surely to Qπ(st, a) as the number of independent continuations M increases, assuming integrable returns.
  • Pairwise comparison: Branch-return differences isolate the downstream consequence of changing the branch-point action and provide the quantity thresholded for empirical supervision.
  • Finite-sample effects: Thresholding empirical value differences need not produce an unbiased binary comparison label, especially when true value gaps are small.Finite-rollout noise makes close comparisons more susceptible to labeling error.
  • Finite-sample effects: Increasing branch continuations primarily improves supervision reliability for comparisons whose true continuation values are close.

A.2 PROOF OF PROPOSITION 3.1

The proposition analyzes errors in candidate-relative intervention estimates under finite candidate sampling. Its bounds apply to independent sampling and uniform sampling without replacement, with concentration controlled by the candidate count.

  • Error decomposition: The proof rewrites pairwise error using binary learned and oracle relations, reducing the analysis to single-candidate comparison error.
  • Candidate aggregation: Linearity of expectation combines single-candidate error with the aggregate behavior of K sampled candidates.
  • Sampling schemes: Hoeffding concentration applies under independent candidate sampling and also under uniform sampling without replacement from a finite population.Without-replacement sampling is at least as concentrated for bounded finite-population averages.
  • Conclusion: The proof establishes the proposition under both candidate-sampling schemes.

A.3 COMPARATOR TRAINING AND INFERENCE

COTA trains a comparator from same-prefix continuation estimates, uses ternary supervision with abstention, and converts bidirectional predictions into the binary runtime gate. The resulting gate is reliable away from an uncertainty region, while intervention remains limited by candidate support.

  • Comparator training: The comparator target is whether one action has higher actor-conditioned continuation value than another, with same-prefix branch estimates providing supervision.
  • Comparator training: As branch count increases, empirical action ordering increasingly reflects the ordering of Qπ values.
  • Ternary supervision: Training uses outputs A, B, and T, where T marks an empirical tie region and contributes no decisive winner at deployment.
  • Inference: Bidirectional evaluation maps ternary predictions to the binary comparator, requiring consistent semantic preference under both input orders.
  • Gate consistency: Away from an uncertainty region around the intervention threshold, the learned gate agrees with the oracle candidate-relative gate.Lower comparator error narrows the region through ϵθ, while increasing K reduces finite-candidate sampling error at the standard K−1/2 rate.
  • Candidate support: Constructive advice cannot expose a genuinely better alternative when the sampled candidate set contains none.The reference mechanism need not be expert, but weaker candidate support supplies useful alternatives less frequently.

B EXPERIMENTAL DETAILS

The experimental-details section records the evaluation and implementation contract underlying the paper’s experiments, including actor protocols, branch collection, comparator training, candidate construction, baselines, prompts, and cost accounting.

  • Experimental contract: The experimental contract specifies evaluation slices, actor protocols, branch collection, comparator training, candidate construction, baselines, prompts, and cost accounting.

B.1 EVALUATION SLICES AND ACTOR PROTOCOLS

The evaluation spans multiple environments, actor models, candidate sources, and intervention baselines, using conservative comparison gates and actor-controlled replanning.

  • Evaluation slices: Evaluation slices include WebShop, ALFWorld, and τ 3-Retail, with distinct held-out protocols and seeded repeats for environment assessment.DeepSeek uses cost-controlled fixed subsets for WebShop and ALFWorld, while Retail retains its full three-seed protocol.
  • Actor protocols: Actors receive observable task state, executed trajectory information, current observations, and admissible actions, without access to unexecuted actor thoughts.The representation is benchmark-specific but excludes hidden reasoning from the comparator interface.
  • Comparator interface: COTA accepts a candidate only when pairwise predictions remain semantically consistent after swapping action order; uncertainty and malformed outputs fail open.This bidirectional requirement makes deployment more conservative than single-order classification.
  • Intervention operating point: K controls reviewed alternatives and R controls required stable winners, with K=4, R=1 for WebShop and ALFWorld and K=4, R=2 for τ 3-Retail.Operating points are selected on held-out validation tasks and fixed across test actors within each environment.
  • Candidates and baselines: Candidates come from legal environment actions, grounded offline retrieval, or compact-LM proposals, while compared baselines include absolute-Q critics, self-reflection, and separate critics.Candidate sources and baseline interfaces vary by benchmark and actor family.

B.8 ABSOLUTE-Q DIAGNOSTICS

Absolute-Q diagnostics show that reasonable held-out ranking quality does not guarantee reliable online control, whereas comparison-based intervention remains more effective in closed-loop use.

  • Absolute-Q diagnostics: More than 80% of actor actions are replaced by forced absolute-Q selection in ALFWorld, despite the scalar critic’s reasonable held-out ranking accuracy.The critic frequently ranks legal but subgoal-irrelevant moves above the actor proposal.
  • Absolute-Q diagnostics: Constructive intervention absorbs some scalar-critic mistakes, but direct pairwise supervision remains substantially more effective in the closed loop.The comparison is about online control effectiveness rather than held-out ranking alone.
  • Comparator-target diagnostics: Explicit ties improve order consistency and preserve abstention when branch outcomes do not justify selecting a winner.The reported online difference is descriptive because it uses 100 independently sampled trajectories.
  • Efficiency accounting: Main-table normalized time uses paired cumulative episode time, combining extra model calls with trajectory shortening from better decisions.Offline training and branch-generation costs are reported separately from online Avg. T values.
Loading 2608.21027v1…