PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask Your Advisor: Learning When to Consult Stronger Models in Agentic Systems

Pascal Sowietzki, Steffen Holter, Yannick Metz, Emil Kasper, Raphael Baur, Mennatallah El-Assady

arXiv:2610.13870v1cs.MA

TL;DR

Agentic workflows need stronger reasoning at selected moments, but executor-advisor systems often leave consultation to executor self-assessment. ARIA learns cost-aware consultation decisions from workflow evidence; across three benchmarks, it recovers 44–81% of the executor-advisor success gap at less than half the Advisor-only cost.

  • Problem

    Capability allocation methods often act before execution, while executor-advisor systems leave consultation timing to self-assessment despite known limits in language-model self-diagnosis.

    Source: §1 · p. 1

  • Method

    ARIA uses a learned meta-controller to select SILENT or CONSULT at workflow contact points based on observable evidence, learning a cost-aware policy without fine-tuning either model.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    Across three benchmarks, ARIA recovers 44–81% of the executor-advisor success gap at less than half the Advisor-only cost, with the highest empirical excess recovery among evaluated conditions.

    Source: §7 · p. 10

  • Takeaways & Limitations

    ARIA makes consultation decisions inspectable and separates when to consult from whether the executor follows the advice.

    Source: §6 · p. 9

    ARIA learns where advice has downstream value but does not ensure executor uptake; ignored-input rates reach 36–40% at SWE-bench’s DELIVER stage.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

To balance task performance and computational cost, agentic workflows can use a lower-cost model for execution and selectively consult a stronger model. In such executor-advisor systems, the executor decides when to seek advice, making escalation depend on self-assessment, although current language models struggle to detect where their own reasoning fails. The need for stronger reasoning emerges during execution and varies across workflow phases. We introduce ARIA, a meta-controller that learns when to consult by balancing task success against consultation cost. Our approach formulates consultation as a sequential resource-allocation problem. At workflow contact points, ARIA maps execution evidence to a compact state abstraction and selects SILENT or CONSULT, with action values learned by Monte Carlo control from cost-penalized outcomes. Across evaluated tasks of SWE-bench Verified, Terminal-Bench 2.1, and $\tau^2$-bench, ARIA recovers 44-81% of the executor-advisor success gap at less than half the advisor model's cost. ARIA achieves the best observed success-cost trade-off among ten other evaluated conditions, including routing and cascading methods, fixed and prompted consultation, and random placement at matched advice volume. The policies are compact and inspectable, with a single consultation-price parameter governing the success-cost trade-off without retraining either underlying model.

1 INTRODUCTION

Long-running workflows need stronger reasoning selectively, because its value changes as execution unfolds. ARIA learns when to consult a stronger advisor while a lower-cost executor remains responsible for the work.

1 INTRODUCTION

  • Capability-allocation methods mostly choose models or computation before execution, leaving open when stronger reasoning is valuable within an evolving workflow.New observations, errors, and recovery opportunities can make only some decision points especially valuable for consultation.Source: §1 · p. 1
  • Executor-advisor systems keep a lower-cost executor in control while adding advisor guidance, but deciding when to consult is often left to executor self-assessment.Language models can struggle to localize their own errors and calibrate confidence; consultation should be judged by whether it improves the final outcome enough to justify its cost.Source: §1 · pp. 1–2
  • ARIA is a learned meta-controller that uses observable workflow evidence to choose consultation points through tabular Monte Carlo control, without fine-tuning either model.It is instantiated on Magentic-One and assumes recurring decision points such as replanning or final delivery.Source: §1 · p. 2, Figure 1 · p. 2
  • ARIA is evaluated on SWE-bench Verified, Terminal-Bench 2.1, and τ 2-bench against executor-only, advisor-only, and selective-allocation baselines.A random-placement control approximates ARIA’s consultation rate to distinguish the effect of timing from advisor access alone.Source: §1 · p. 2
  • The paper introduces an interpretable consultation-policy framework, evaluates it across three agentic domains, and analyzes failures in executor-advisor collaboration.The authors also release the framework and traces for research on capability allocation and agent collaboration.Source: §1 · p. 2

2 RELATED WORK

Prior capability-allocation work typically acts before execution, changes which model runs, or structures the workflow. ARIA instead learns when to acquire advisor guidance during execution without modifying the underlying models.

Specialized models and adaptive computation

  • Specialization and adaptive computation improve efficiency without conditioning allocation on evidence revealed during a running workflow.Input-adaptive methods assign resources from the query before response generation.Source: §2 · pp. 2–3

Routing, cascading, and orchestration

  • Routing and orchestration methods select models, escalate answers, route reasoning steps, or optimize workflows, whereas ARIA takes the executor and workflow as given.Some fine-grained routing methods require model-internal signals unavailable in ARIA’s setup.Source: §2 · p. 3

External consultation

  • Prior advice systems and human-agent tools offer execution-time guidance, but ARIA’s contribution is learning a cost-aware consultation policy while keeping the models fixed.Related systems include fixed-budget advice, human intervention, and trained agents that recognize when expert help is needed.Source: §2 · p. 3

3 METHODOLOGY

ARIA keeps the executor, advisor, environment, and workflow fixed, learning a binary policy that allocates advisor guidance at recurring workflow decision points. It represents workflow evidence compactly and learns consultation values from terminal outcomes penalized by consultation cost.

3.1 CONSULTATION DURING THE TRAJECTORY

  • ARIA adds consultation opportunities at PRE-STEP, STALL, and PRE-DELIVERY events, corresponding to an upcoming instruction, stalled progress, and impending final delivery.The contact-point type identifies why an opportunity occurs, separately from the temporal state’s representation of progress.Source: §3.1 · p. 3
  • At each contact point, SILENT leaves the executor in control, while CONSULT provides advisor guidance that the executor may use in subsequent reasoning and actions.ARIA learns consultation value from downstream outcomes rather than immediate local feedback.Source: §3.1 · p. 3

3.2 STATE ABSTRACTION

  • ARIA maps workflow history to a 24-state abstraction combining temporal position, execution evidence, and consultation history.Execution evidence distinguishes absent, clean, or error-signaled output; the temporal dimension groups progress into FRAME, EXPLORE, REASON, or DELIVER.Source: §3.2 · p. 4
  • The state omits remaining budget, so consultation is forced to SILENT once the budget is exhausted.The compact policy summarizes workflow signals without explicitly conditioning on remaining budget.Source: §3.2 · p. 4

3.3 COST-AWARE POLICY LEARNING

  • ARIA’s cost-aware return combines terminal task success with a fixed penalty for each consultation, with λ controlling the success-cost trade-off.The penalty normalizes one consultation against the domain’s spend for consulting at every contact point; maximizing expected return is a fixed-multiplier relaxation of a consultation-cost budget.Source: §3.3 · p. 4
  • Because task outcomes arrive only at episode end, ARIA learns action values with every-visit tabular Monte Carlo control and sample-average updates.Training uses an ε-greedy policy; evaluation freezes the greedy policy, resolving ties and unvisited states to SILENT.Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments evaluate ARIA across three agentic environments with fixed proprietary models and compare it with capability references and consultation-allocation alternatives. They measure task success, realized cost, and excess recovery, and use failure analysis to study collaboration breakdowns.

4 EXPERIMENTS

  • The evaluation uses SWE-bench Verified, Terminal-Bench 2.1, and τ 2-bench with disjoint train and test tasks and a fixed executor-advisor model pair.Consultation conditions allow at most four advisor calls per episode; budget sensitivity is examined separately.Source: §4 · p. 5
  • The baseline design compares capability references, routing and cascading, fixed schedules, LLM-based timing, random placement, and learned allocation.The conditions test whether model strength, allocation strategy, advisor access, or consultation timing explains the success-cost trade-off.Source: §4 · p. 5, Table 1 · p. 5
  • The consultation-price sweep compares λ = 0.10 with policies re-estimated offline at λ = 0.35 and λ = 0.50 from the same training trajectories.Because those trajectories were collected under the λ = 0.10 behavior policy, the sweep is offline policy re-estimation under that visitation distribution, not equivalent retraining at each price.Source: §4 · p. 5
  • The evaluation reports performance gap recovered and normalizes realized cost between Executor-only and Advisor-only anchors.Excess recovery is PGR minus CostShare; positive values indicate more performance gap recovered than normalized cost spent.Source: §4 · pp. 5–6
  • Failure analysis applies the Multi-Agent System Failure Taxonomy, assigning each failed trajectory one primary failure mode and additionally recording all modes that occur.LLM-judge annotations are validated on a stratified subset against human annotations.Source: §4 · p. 6

5 RESULTS

Across three benchmarks, ARIA achieves the best observed success–cost trade-off among evaluated conditions. Results show that consultation placement and price affect outcomes, and that residual failure patterns vary across benchmarks.

5 RESULTS

  • ARIA recovers 68.2%, 44.4%, and 80.6% of the success gap on SWE-bench, Terminal-Bench, and τ 2-bench at 48.1%, 40.9%, and 34.5% of Advisor-only cost, respectively.No evaluated condition achieves higher mean success at equal or lower cost, or lower cost at equal or higher mean success.Source: Table 2 · p. 6, §5 · p. 6
  • On SWE-bench, ARIA’s ER is 0.37 versus −0.02 for FrugalGPT, −0.12 for RouteLLM, and −0.63 for AutoMix; the evaluated task-level methods do not match its trade-off.These methods do not incorporate evolving evidence throughout the agentic trajectory on the tested benchmarks.Source: §5 · p. 6

5.1 PLACEMENT MATTERS BEYOND ADVISOR ACCESS AND VOLUME

  • At similar consultation volumes on SWE-bench, Random, Heuristic, and ARIA achieve 28.0%, 41.3%, and 48.0% success, respectively.The placement control is tightest on SWE-bench and weaker on τ 2-bench.Source: §5.1 · p. 7
  • On SWE-bench, raising Always’s budget to B = 10 increases success by 20.7 points, but costs more and uses nearly twice ARIA’s consultation volume.On Terminal-Bench, success rises by 4.0 points with unchanged ER; on τ 2-bench, success falls and ER becomes negative.Source: §5.1 · p. 7, Table 4 · p. 7

5.2 CONSULTATION PRICE CONTROLS THE OPERATING POINT

  • At λ = 0.10, ER is 0.37; increasing λ to 0.35 and 0.50 reduces consultation and cost while success falls from 48.0% to 34.0% and 29.3%.The policies use the same training traces, so this comparison reflects offline price sensitivity under the observed visitation distribution.Source: §5.2 · p. 7, Figure 2 · p. 8

5.3 LEARNED ALLOCATION DIFFERS ACROSS WORKFLOW DOMAINS

  • Policies consult in only 4–9 of 24 nominal states; calls cluster at DELIVER on SWE-bench (53.5%) but at REASON on τ 2-bench (82.2%).Consultation is selective and cannot be explained solely as a response to explicit errors.Source: §5.3 · p. 8

5.4 FAILURE ANALYSIS REVEALS MECHANISM AND SCOPE

  • On SWE-bench, verification-primary residual failures fall from 86% under Executor-only to 34% under ARIA, while ignored-input FM-2.5 becomes ARIA’s dominant residual mode at 58%.Human validation confirms these cases contain relevant advice that reached the executor but was not acted upon.Source: §5.4 · pp. 8–9
  • On Terminal-Bench, ARIA and Heuristic tie in success but solve 12 different episodes each; on τ 2-bench, ARIA leaves only eight judged failures.These results show that residual failure profiles depend on the interaction regime as well as the participating models.Source: §5.4 · p. 9

6 DISCUSSION AND LIMITATIONS

ARIA’s results suggest consultation timing matters more than advisor access or consultation volume alone. Its benefits are largest with recurring decision points, while the experiments are limited to one model pair and framework.

6 DISCUSSION AND LIMITATIONS

  • ARIA achieves the highest excess recovery on all three benchmarks, with gains largest when similar decision points repeat and smaller on heterogeneous Terminal-Bench tasks.The experiments focus on one model pair and the Magentic-One framework.Source: §6 · p. 9
  • ARIA makes consultation decisions inspectable, and the failure analysis identifies advice uptake as a next bottleneck after improving when to consult.The policy separates when to consult from what happens once guidance is provided.Source: §6 · p. 9

7 CONCLUSION

ARIA recovers 44–81% of the executor-advisor performance gap at less than half the Advisor-only cost. The appendices specify the intervention contract and a compact state representation based on workflow progress, execution feedback, and prior advice.

7 CONCLUSION

  • ARIA recovers 44–81% of the executor-advisor performance gap at less than half the Advisor-only cost and has the highest empirical excess recovery among evaluated conditions.Its inspectable policy needs neither fine-tuning nor access to model internals.Source: §7 · p. 10
  • The authors report that generative AI tools supported preparation tasks and that AI-assisted outputs and research claims were reviewed and verified by the authors.The reproducibility statement points to sections and appendices documenting the method, benchmarks, diagnostics, and evaluation variation.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 ADVISOR INTERVENTION SEMANTICS

  • The advisor provides read-only guidance while the executor retains responsibility for reasoning, tool use, environment interaction, and final submission.At PRE STEP, STALL, and PRE DELIVERY, guidance enters the trajectory at different workflow moments; SILENT leaves the workflow unchanged.Source: §A.1 · p. 14

A.2 STATE CONSTRUCTION

  • The state abstraction uses workflow-level signals rather than model-internal confidence and pools consultation experience across recurring contexts, trading away potentially decision-relevant information.Execution evidence records status rather than semantic correctness; mappings differ between benchmark environments.Source: §A.2 · pp. 14–15
  • Temporal state distinguishes FRAME, EXPLORE, REASON, and DELIVER using contact-point type and ordinal position, with STALL inheriting the ordinal stage.The labels are task-agnostic progress labels, not semantic classifications of executor reasoning; other systems can use their own recurring events and progress signals.Source: §A.2 · pp. 14–15
  • ARIA’s 24-state representation combines trajectory progress, observable execution evidence, and whether advisor guidance has already entered the trajectory.Remaining consultation budget is omitted from the state; after the episode budget is exhausted, CONSULT is forced to SILENT.Source: §A.2 · pp. 14–15

A.3 TRAINING AND POLICY ESTIMATION

ARIA learns a cost-aware consultation policy while keeping the executor and advisor fixed. Tabular Monte Carlo estimates action values, after which deployment uses a frozen greedy policy.

A.3 TRAINING AND POLICY ESTIMATION

  • ARIA learns only the consultation controller; the executor and advisor remain fixed throughout training.The resulting value table is frozen for deployment.Source: §A.3 · pp. 15, 17
  • Each consultation receives a fixed penalty κ = λ¯c/∆, where ¯c is mean consultation cost and ∆ is mean per-episode spend for consulting at every contact point.Headline policies use λ = 0.10, and realized evaluation costs are reported separately from the learning penalties.Source: §A.3 · p. 15, Table 5 · p. 16
  • Every-visit tabular Monte Carlo control uses sample-average updates with γ = 1, and only episodes reaching a terminal task outcome contribute value updates.Every occurrence contributes a sample; repeated visits within an episode are correlated, and long trajectories can receive more weight than under first-visit Monte Carlo.Source: §A.3 · p. 16
  • Training uses ϵ-greedy exploration, annealing ϵ from 0.40 to 0.10 over the first 110 episodes and keeping it at 0.10 thereafter.Three independent controller seeds are trained per benchmark on revisited development tasks disjoint from held-out evaluation tasks.Source: §A.3 · p. 16
  • For λ values of 0.35 and 0.50, SWE-bench policies are re-estimated from λ = 0.10 training trajectories without collecting additional episodes.These are offline re-estimates under the original behavior-policy visitation distribution, not independently trained policies at each price.Source: §A.3 · p. 17
  • Deployment selects the greedy action with ϵ = 0, resolves ties and unvisited states to SILENT, and forces SILENT after the consultation budget is exhausted.No exploration or controller updates occur on held-out tasks.Source: §A.3 · p. 17

B.1 SPLIT CONSTRUCTION AND TASK SEPARATION

The appendix documents task separation and comparison contracts, then audits randomized consultation calibration, billing, and run-to-run variability.

B.1 SPLIT CONSTRUCTION AND TASK SEPARATION

  • SWE-bench and Terminal-Bench development pools were outcome-filtered, while τ 2-bench used official splits; 15 of 50 held-out Terminal tasks were never solved by either model.The never-solved tasks were removed from Terminal-Bench development but retained in evaluation, making the development distribution easier.Source: §B.1 · p. 18

B.2 BASELINE CONTRACTS

  • Headline consultation comparisons use a B = 4 episode cap, contrasting learned allocation with task-level, fixed-schedule, native-orchestrator, and random-placement alternatives.Task-level methods allocate before or at task execution; consultation schedules and ARIA share within-trajectory advisor intervention semantics.Source: §B.2 · pp. 19–20

B.3 RANDOM CALIBRATION

  • Random’s consultation probability is calibrated from ARIA’s consultations per eligible contact point and then frozen; realized volume reaches 87.8% of ARIA’s on τ 2-bench.SWE-bench and Terminal-Bench meet the stated ±10% volume-matching criterion.Source: §B.3 · pp. 20–21

B.4 TOKEN USAGE AND BILLING

  • The cost reconstruction covers all 33 headline cells, with mean costs differing from Table 2 by at most $0.0049; pricing reproduces 100% of 106,662 complete non-failed calls.For 4,066 episode records with call records, attribution found one matching attempt in every case.Source: §B.4 · pp. 22–23

C.1 SEED-LEVEL DISPERSION

  • Across three seeds, ARIA’s success standard deviations were 3.5, 5.0, and 2.9 percentage points on SWE-bench, Terminal-Bench, and τ 2-bench; nearby operating points are descriptive, not statistically separated.The reported values are sample standard deviations across repeated runs, not confidence intervals over the task distribution.Source: §C.1 · p. 24

C.2 PLACEMENT ANALYSIS

Figure 4 compares task success with consultation frequency for ARIA, Heuristic, and Random across the three benchmarks.

C.2 PLACEMENT ANALYSIS

  • At similar consultation volumes on SWE-bench, ARIA achieves higher success than Heuristic and Random; on Terminal-Bench it matches Heuristic, while on τ 2-bench it exceeds both despite Random’s lower advice volume.The figure relates success to realized consultations per episode, highlighting differences in placement as well as consultation volume.Source: §C.2 · p. 24, Figure 4 · p. 24

C.3 TRAINING DYNAMICS AND POLICY INSPECTION

The appendix inspects ARIA’s training behavior, frozen action values, and agreement among independently learned policies.

C.3 TRAINING DYNAMICS AND POLICY INSPECTION

  • Across three seeds, τ 2-bench consultation rates remain comparatively similar even as greedy decisions change, while SWE-bench and Terminal-Bench show greater consultation-rate variation.The plots describe observed training dynamics and do not establish policy convergence.Source: §C.3 · p. 25, Figure 5 · p. 25
  • ARIA’s frozen tables compare CONSULT and SILENT values within each state; these are cost-aware return estimates, not calibrated success probabilities.The heatmaps visualize the learned values, and deployment selects the larger within-state value, subject to tie-breaking and the budget constraint.Source: §C.3 · p. 25, Figure 6 · p. 26, Figure 7 · p. 27

C.4 POLICY CONSENSUS

  • Eight of nine frozen policies select CONSULT at DELIVER with clean/OK evidence after prior consultation, despite limited overall agreement across seeds and benchmarks.Unvisited states count as the deployment default SILENT in the agreement summary.Source: §C.4 · p. 29, Table 13 · p. 29

C.5 WITHIN-STATE CONSULTATION EFFECTS

The appendix estimates within-state consultation effects using randomized training decisions and propensity weighting, while noting limits from training-distribution data and sparse cells.

C.5 WITHIN-STATE CONSULTATION EFFECTS

  • For each state cell, the self-normalized Hájek estimate contrasts outcomes under CONSULT and SILENT using logged consultation propensities; cells with fewer than five observations for either action receive no estimate.Uncertainty uses 10,000 episode-level bootstrap replicates, preserving dependence among decisions from the same trajectory.Source: §C.5 · p. 29
  • At SWE-bench DELIVER/CLEAN/k = 0, consultation’s estimated episode-success effect is 0.54 [0.15, 0.85]; positive delivery-gate effects also appear on Terminal-Bench.Terminal-Bench estimates are 0.33 [0.12, 0.50] for DELIVER/CLEAN/k ≥1 and 0.24 [0.06, 0.44] for DELIVER/ERROR/k ≥1.Source: §C.5 · p. 30, Figure 9 · p. 30
  • On SWE-bench, two REASON cells have negative one-step estimates of −0.35, although one has a positive episode-level estimate of 0.23 [−0.03, 0.46].The difference illustrates that immediate execution effects and longer-horizon task outcomes can diverge.Source: §C.5 · p. 30
  • The analysis uses randomized training trajectories rather than the frozen greedy evaluation policy; uneven cell occupancy, unavailable τ 2-telecom one-step outcomes, and unadjusted multiple comparisons limit interpretation.Only 11 of 35 primary intervals exclude zero, so individual cells should be read alongside the broader pattern.Source: §C.5 · pp. 30–31

C.6 MAST FAILURE ANALYSIS

The analysis characterizes failed trajectories using MAST labels and examines how failure profiles differ across benchmarks. It distinguishes dominant primary failures from co-occurring secondary modes, while cautioning against causal interpretation.

C.6 MAST FAILURE ANALYSIS

  • The analysis labels failed trajectories with usable traces using MAST’s 14 modes, grouped into system-design, inter-agent-misalignment, and verification failures.Each trajectory receives non-exclusive presence labels and one mutually exclusive primary failure label; presence labels serve as supporting diagnostics.Source: §C.6 · p. 31
  • MAST coverage excludes successful trajectories and failures without recoverable traces, although those failures remain in headline benchmark metrics.More than 1,600 failed trajectories were judged; trace exclusions particularly affect Terminal-Bench crash-path episodes.Source: §C.6 · p. 31
  • Human annotations agreed with the automatic judge on 35 out of 40 trajectories (87.5%), with ignored input and incorrect verification among the dominant errors.In validated advisor trajectories flagged as off track, relevant guidance reached the executor but was not followed by corrective action.Source: §C.6 · p. 31
  • Failure labels describe residual failure composition, not causal effects of individual consultations; inter-agent comparisons also depend on whether communication occurred.Secondary presence labels may capture co-occurring mechanisms rather than only the primary reason a trajectory failed.Source: §C.6 · p. 31
  • Across benchmarks, secondary profiles show that consultation can coincide with ignored guidance on SWE-bench, while Terminal-Bench and τ 2-bench have different residual patterns.On SWE-bench, verification failures remain common; Terminal-Bench profiles are mixed, and inter-agent failures are less prominent on τ 2-bench.Source: §C.6 · p. 32, Figure 10 · p. 32

Related papers

Loading 2610.13870v1…