PaperlayerView exact revision on arXiv

Source-linked AI summary

SynAct: Synergistic Active Feedback Learning for LLM Agents

Ning Liu, Ian Li, Kristina Klinkner

arXiv:2610.12651v1cs.LG

TL;DR

When retries cannot resolve an agent’s knowledge gap, scarce expert attention raises the question of which decisions merit feedback. SynAct ranks decisions by the relationship between actions and terminal outcomes, then turns expert corrections into selectively applied rules. It improves accuracy across all 16 evaluated settings, with an average gain of +11.7pp at the best iteration.

  • Problem

    Retries can recover slips but not knowledge gaps, while expert attention is scarce and task-level feedback may spend effort on decisions that do not determine outcomes.

    Source: §1 · p. 1

  • Method

    SynAct scores state-keyed decisions by mutual information between action and terminal outcome across rollouts, then distills expert corrections into gated reusable rules.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    SynAct improves accuracy in all 16 settings, by +11.7pp on average at the best iteration.

    Source: §1 · p. 3, §5.1 · p. 7

  • Takeaways & Limitations

    The same signal routes inference-time expert consultation, while offline rules can deploy without the expert in the execution loop.

    Source: §6 · p. 9

    The expert is a proxy for a strong human, so the cost of human authoring remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

LLM agents executing multi-step workflows fail in two fundamentally different ways: recoverable slips, which further attempts can repair, and knowledge gaps, which retrying cannot resolve. Expert feedback can address the latter, but expert attention is scarce. The key challenge is therefore not whether to ask for feedback, but where feedback can change the outcome. Existing approaches either avoid expert feedback, allowing agents to reinforce their own errors, or allocate feedback at the task level without identifying consequential decisions. We present SynAct, an active learning framework that treats expert attention as the binding constraint and allocates feedback at the level of individual decisions. SynAct ranks decisions by the mutual information between the selected action and the terminal outcome across repeated rollouts, queries an expert at the highest-value decisions, and distills the resulting corrections into transferable rules that the agent can reuse without expert access at deployment. We theoretically characterize what this signal identifies under finite repeated sampling: with k rollouts and at least m observations per action, outcome-constant decisions have zero mutual information, while outcome-separating decisions have mutual information at least H2(m/k) bits, yielding exact separation for any threshold t in (0,H2(m/k)). This also explains why success-based acquisition can be blind when all rollouts fail: success is constant even when failed runs reach different outcomes. Across four agent benchmarks and four models spanning three families, SynAct improves held-out accuracy in all 16 evaluated settings, by +11.7pp on average. The same signal transfers to inference-time routing, where SynAct outperforms budget-matched cascade, uncertainty, and expert-judge routers in 15 of 16 settings, while a single targeted correction outperforms seven additional unaimed retries.

1 INTRODUCTION

SynAct treats scarce expert attention as a decision-level allocation problem, using action–outcome mutual information to find consequential choices and turn corrections into reusable, selectively applied rules. Its finite-sample characterization and evaluations report broad gains, including when all rollouts fail.

1 INTRODUCTION

  • Expert feedback addresses knowledge gaps that retries cannot resolve, but reviewing whole trajectories wastes attention on decisions that do not determine outcomes.The paper therefore treats decision points, rather than episodes, as the unit of feedback.Source: §1 · p. 1
  • SynAct asks which action differences explain terminal outcomes, using mutual information I(A; O) from repeated rollouts without a gold answer or success label.Unlike action entropy, the score measures consequential disagreement and can remain informative when every rollout fails.Source: §1 · pp. 1–2
  • An expert turns selected decisions into a convention and antipattern; an error gate applies the rule only to failed attempts exhibiting that antipattern.The same score can route expert consultation per task at inference time.Source: §1 · p. 2, Figure 1 · p. 2
  • With k rollouts and at least m observations per action, outcome-constant and outcome-separating decisions are exactly distinguished below threshold H2(m/k), while success-based acquisition is blind when all runs fail.The signal ranks decisions by how strongly action choices explain downstream outcomes.Source: §1 · p. 2
  • SynAct improves accuracy in all 16 benchmark–model settings and beats budget-matched cascade, uncertainty, and expert-judge routers in 15 of 16 settings.The selective error gate also eliminates win-to-lose flips, while always-on context costs a weak agent up to 17.6pp.Source: §1 · p. 3

2 RELATED WORK

Related work addresses agent improvement through self-generated context, additional test-time computation, expert intervention, and credit assignment or routing. SynAct differs by using a metered expert budget to rank consequential decisions within episodes and gate reusable feedback.

Self-improving agents

  • Self-improvement methods distill context from an agent’s own traces, which is least reliable when it is confidently wrong; unconditional context can also introduce regressions.Some methods bootstrap only from successes, leaving no signal on tasks the agent always fails.Source: §2 · p. 3

Test-time correction and search

  • Self-feedback and trajectory search can recover slips but cannot supply knowledge the agent lacks, whereas SynAct uses scarce expert attention to address that gap.The paper notes that authored conventions repair syntactic misunderstandings of environment conventions.Source: §2 · p. 3

Budgeted acquisition and expert intervention

  • Prior active-learning and intervention methods economize supervision, but leave open which decision within an episode is most informative; SynAct acquires at that finer unit.Unlike LEAP, it makes one expert call per iteration and leaves the model frozen.Source: §2 · p. 3

Credit assignment and routing

  • Credit-attribution and process-supervision methods localize steps for policy updates or verification, while SynAct ranks decisions to route metered expert feedback to a frozen agent.Its within-query allocation differs from uncertainty and cost-aware methods that select whole queries.Source: §2 · p. 3

3 SYNACT: SYNERGISTIC ACTIVE FEEDBACK LEARNING

SynAct uses repeated agent rollouts to identify decisions whose action alternatives lead to different terminal outcomes, then spends a fixed expert budget authoring reusable rules. The rules are selectively applied and iteratively revised, while the same signal can route expert consultation at inference time.

3 SYNACT: SYNERGISTIC ACTIVE FEEDBACK LEARNING

  • Given a fixed acquisition budget, SynAct selects consequential decision points, asks the expert for a reusable rule in one call, and applies it only to matching failures.The objective is to maximize value per expert query, with optional repetition as learned rules change remaining failures.Source: §3 · p. 4

3.1 ACQUISITION: FINDING CONSEQUENTIAL DECISIONS

  • Repeated rollouts are grouped by normalized environment-state keys, and mutual information I(A; O) scores whether actions at a shared state distinguish terminal outcome classes.The score ranks decisions without a gold label or success check; state keys are assumed decision-sufficient, and the outcome projection depends only on the terminal state.Source: §3.1 · pp. 4–5

3.2 IDENTIFIABILITY UNDER REPEATED SAMPLING

  • At finite k, outcome-constant nodes have I(v)=0 and outcome-separating nodes have I(v)≥H2(m/k), enabling exact threshold separation when 0<t<H2(m/k).For k=8 and m=2, the admissible threshold range is (0, 0.811); rare actions are less likely to meet the support floor.Source: §3.2 · p. 5
  • For intermediate decisions, I(v) measures how much action variability the outcome resolves; success-based scores hide all-fail tasks, while the plug-in estimator has positive null bias.The score is used to rank decisions, not as a hypothesis test.Source: §3.2 · p. 5

3.3 AUTHORING: FROM EPISODIC EVIDENCE TO SEMANTIC RULE

  • The expert receives aggregated evidence blocks and writes a short convention for recurring repairable failure modes plus an antipattern describing the mistakes it prevents.One call can cover multiple acquired blocks, so additional evidence does not require additional expert invocations.Source: §3.3 · p. 6

3.4 APPLICATION: SELECTIVE INJECTION BEHIND AN ERROR GATE

  • An LLM gate inspects a failed trajectory and injects the convention only when its antipattern is present, so an already-successful base attempt never sees the rule.Each task gets at most two attempts, and the gate sees neither the success bit nor the reference solution.Source: §3.4 · p. 6

3.5 ITERATION AND INFERENCE-TIME REUSE

  • After each update, SynAct recollects rollouts, revises the convention for residual failures, and can instead use I(v⋆) to route inference-time expert consultation.Offline rules deploy without the expert; online consultation operates per difficult task without a training set.Source: §3.5 · pp. 6–7

4 EXPERIMENTAL SETUP

The evaluation spans four agent benchmarks and four models from three model families, with a strong-model expert proxy and comparisons against experience-distillation and acquisition-selection baselines. Results are reported over repeated runs with the test set and rule fixed, except for the ablated component.

Benchmarks

  • The evaluation covers InterCode-SQL, TextCraft, InterCode-Bash, and τ-bench retail, spanning text-to-SQL, compositional crafting, shell scripting, and tool-use dialogue.Each benchmark has observable states recurring across rollouts, and training uses a disjoint split of up to 50 tasks each.Source: §4 · p. 7

Agent models and expert

  • Four models from three families are tested, with Claude Opus serving as the fixed expert proxy across all 16 settings and iterations.Whether a human expert would author the same convention remains untested.Source: §4 · p. 7

Baselines

  • Four experience-distillation baselines receive the same expert as SynAct, while additional acquisition baselines vary only the selection rule.The baselines include ACE, ReasoningBank, AWM, ten-round AWM-iter, SynAct-Succ, SynAct-Ent, and SynAct-Rand.Source: §4 · p. 7

Protocol

  • Each setting is reported as the mean of three repeats on the full test set, holding the rule, gate, and test set fixed except for the ablated component.Base accuracy varies by 3.4–8.5pp across iterations; expert tokens are reported separately because they cost roughly 10–30× more.Source: §4 · p. 7

5 RESULTS

SynAct routes expert feedback to decisions associated with downstream outcomes, outperforming experience-distillation baselines across the reported settings. Its gains persist as expert-token budgets grow, and the allocation principle also transfers to inference-time routing.

5.1 ROUTED EXPERT FEEDBACK AGAINST EXPERIENCE DISTILLATION

  • SynAct improves over base by +11.7pp on average at its best iteration and leads four distillation baselines in 15 of 16 settings.After one iteration, it improves over base by +8.3pp on average; the comparison holds the expert, training split, test set, and agent fixed.Source: §5.1 · p. 7, Figure 2 · p. 8

5.2 COMPARISON OF ACQUISITION FUNCTIONS

  • Outcome-MI is the best acquisition rule in all 8 swept settings and 23 pairwise comparisons, gaining +12.5pp over base versus +8.2 for random, +6.9 for success-MI, and +6.8 for entropy.Under the final protocol, the four rules yield indistinguishable one-iteration gains; the reported value of outcome information is all-fail coverage and per-task routing.Source: §5.2 · p. 8

5.3 EXPERT-TOKEN AND SAMPLING BUDGETS

  • At comparable expert-token spend, distillers cluster near zero gain while SynAct’s gain grows through the first several iterations before flattening.Figure 3 plots TextCraft gain against cumulative expert tokens on a log x-axis.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 INFERENCE-TIME ROUTING

  • SynAct or its rule-augmented variant is the best router in 15 of 16 settings; three targeted attempts beat ten self-reflection retries in 14 of 16.The iteration-10 convention alongside routing lifts accuracy over routing alone in 14 of 16 settings, by +5.1pp on average.Source: Figure 4 · p. 9, §5.4 · p. 9

5.3 EXPERT-TOKEN AND SAMPLING BUDGETS

  • At k=2, all six swept settings beat base, and the top decile of MI-ranked decisions delivers most of the full-budget gain.The first iteration captures 70% of the median setting’s best-iteration gain; Qwen TextCraft rises from +11.4pp to +20.6pp across later passes.Source: §5.3 · p. 9

6 CONCLUSION

SynAct uses action–outcome mutual information to allocate expert attention and reports improved accuracy across all 16 settings. Its finite-sample analysis characterizes what the signal separates, when it can miss decisions, and why it can remain informative when every rollout fails.

6 CONCLUSION

  • SynAct’s gated convention improved accuracy in all 16 settings, while four distillers using the same expert budget averaged under +1pp.The conclusion also reports that the signal routes consultation at deployment.Source: §6 · p. 9

A IDENTIFIABILITY OF CONSEQUENTIAL DECISIONS

  • Under decision-sufficient projections, the analysis characterizes finite-k separation of outcome-separating from outcome-constant decisions.The conclusion notes that the signal remains informative where success-based scores are blind.Source: §6 · p. 9

6 CONCLUSION

  • The stated scope leaves stochastic-environment degradation unmeasured, and the consistency gate can miss confidently wrong but behaviorally consistent failures.The authors also identify proxy-expert authoring costs and rare modes below the support floor as untested or potentially missed.Source: §6 · p. 9

A.1 THE MUTUAL INFORMATION REPRESENTATION

  • On retained rollouts, the score is exactly empirical mutual information between the action and the outcome.The entropies use the same supported population, so the representation is exact for that empirical joint distribution.Source: §A.1 · p. 14

A.2 FINITE-SAMPLE SEPARATION

  • With at least m observations per action, outcome-constant decisions score zero and outcome-separating decisions score at least H2(m/k), allowing exact threshold separation below that bound.The result needs no asymptotics and permits multiple outcomes per action when outcome supports do not overlap.Source: §A.2 · p. 15

A.3 PARTIAL IDENTIFIABILITY

  • The resolved fraction ρ(v) ranges from zero when outcomes carry no information about the action to one when outcomes determine the action.For the examples given, disjoint outcome sets yield 1 bit of mutual information, while a shared outcome class yields 0.5.Source: §A.3 · p. 15

A.4 RECOVERY UNDER REPEATED SAMPLING

  • At k=8 and m=2, an arm taken in half of rollouts is retained with probability 0.965, versus 0.187 for an arm taken one-tenth of the time.The support floor favors reproducible behavior; increasing k widens coverage at linear rollout cost.Source: Table 1 · p. 16, §A.4 · p. 16

A.5 VISIBILITY BEYOND SUCCESS

  • When every rollout fails, success-based acquisition scores zero, but outcome mutual information stays positive if actions have differing empirical outcome distributions.Thus success-based acquisition is blind by construction on all-fail tasks, while outcome information can still distinguish end states.Source: §A.5 · p. 16

A.6 FINITE-SAMPLE RANKING

Finite-sample analysis shows when outcome-constant and outcome-separating decisions can be distinguished, while clarifying that the score is an acquisition ranking rather than a causal or repairability guarantee.

A.6 FINITE-SAMPLE RANKING

  • At finite k, the plug-in score is upward-biased under the null, averaging 0.124 bits at k=8 and reaching 0.346 bits at c=4.The null reaches 1.0 bits with probability 0.002, so no threshold strictly below the maximum eliminates false positives in this regime.Source: §A.6 · p. 17
  • At k=8 and m=2, a dependent node outranks a null node with probability 0.995 at ε=0, falling toward chance as true dependence vanishes.The probability is 0.517 at ε=0.375, where true information is 0.046 bits.Source: §A.6 · p. 17
  • SynAct uses mutual information to rank nodes, not threshold them: a zero threshold would admit the entire null, and no general ranking-recovery guarantee is claimed.Cross-node bias varies with support sizes; the practical mitigation is a coarse per-benchmark outcome projection, while roughly 93% of two-arm nodes are scorable at k=8 versus 38% at k=4.Source: §A.6 · p. 17

A.7 ASSUMPTIONS AND SCOPE

  • The score identifies finite-sample outcome separation, not causal effects or repairability; hidden variables can raise it, and a separating decision may resist correction by a written convention.State keys must avoid both aliasing distinct situations and fragmenting recurring ones; outcome projections must preserve outcome distinctions without encoding the branching action.Source: §A.7 · pp. 17–18
  • For a new benchmark, the authors say only the state key ϕ and outcome projection ω need redesign, to preserve recurring contexts and outcomes without encoding the branching action.Determinism is used to simplify exact matching and provide a control arm; under stochastic dynamics, a resample-only control would be needed.Source: §A.7 · p. 18

B.1 BENCHMARKS, SPLITS, AND STEP LIMITS

The appendix specifies the evaluation setup and implementation choices, including how benchmark contexts and outcomes are represented. A worked TextCraft example traces selected evidence into a transferable expert-authored rule.

B.1 BENCHMARKS, SPLITS, AND STEP LIMITS

  • The four benchmarks cover SQL, TextCraft crafting, Bash scripting, and τ-bench retail; the retail benchmark uses structured tool calls and excludes the two Gemma models.Success is benchmark-specific: matching the gold SQL result, producing the target item, matching reference-command output, or reaching a correct policy-compliant retail database state.Source: §B.1 · pp. 18–19

B.2 MODELS AND SERVING

  • The evaluated agents are Qwen3-32B, Gemma-3-12B, Gemma-3-4B, and Ministral-3-3B; acquisition uses temperature 0.7 rollouts and evaluation is greedy.Claude Opus 4.6 serves as the expert, and the τ-bench simulator is run at temperature 0 for reproducibility.Source: §B.2 · p. 19

B.3 ACQUISITION HYPERPARAMETERS

  • The default acquisition uses k=8 rollouts, minimum arm size m=2, and the top-scoring node per task; ablations vary k and the expert-budget fraction B.The online variant uses consistency threshold θ=0.6 before thresholding the selected node's score.Source: §B.3 · p. 19

B.4 ERROR GATE

  • The error gate costs no expert calls but adds one agent call of latency to each failed attempt, and defaults malformed responses to SKIP.It receives the task, part of the failed trajectory, antipattern, and rule, and decides whether the rule fires.Source: §B.4 · p. 19

B.5 BASELINE CONFIGURATIONS; B.6 MEASUREMENT PROTOCOL

  • Each setting averages three full-test-set repeats; base accuracy varies 3.4–8.5pp across iterations, so adjacent changes of that size are treated as indistinguishable.Baselines receive the same expert model, while their protocols and setup parameters are configured individually.Source: §B.5 · pp. 19–20, §B.6 · p. 20

C STATE-KEYED GROUPING

  • Decision nodes group run steps by state key, while the implementation uses ordered action prefixes because cached rollouts lack full observations at every step.The order-invariant alternative barely changes acquisition overall but often merges distinct TextCraft decisions, supporting ordered-prefix keying as the default.Source: C STATE-KEYED GROUPING · p. 20

D OUTCOME PROJECTIONS ω

  • Outcome projections map terminal states to benchmark-specific classes; coarsening can move rankings substantially without harming end-to-end results in the reported re-runs.On TextCraft, the coarse projection yields +17.2 vs. +13.8pp for Qwen and +12.1 vs. +6.9pp for Gemma-12B at one iteration.Source: D OUTCOME PROJECTIONS ω · p. 21

E WORKED EXAMPLE

  • In the TextCraft example, the expert abstracts distinct acquired failures into a convention that transfers beyond the concrete training items, lifting accuracy by 14.2pp over base.The example uses 49 scorable tasks, including 20 with no successful rollout; acquisition uses no labels, though success tags enter the digest shown to the expert.Source: E WORKED EXAMPLE · pp. 21–22

F ROLLOUT AND EXPERT BUDGET SENSITIVITY

The sensitivity analyses show how acquisition quality and coverage vary with rollout and expert budgets, while comparing outcome information with alternative selectors and self-supervised curation. The comparison's environment scope and the limits of self-supervised traces qualify those findings.

F ROLLOUT AND EXPERT BUDGET SENSITIVITY

  • At k=2, all six swept settings beat base, while the TextCraft multi-iteration sweep found no divergence or rule at k=2 but learned at k=4, 8, and 16.For Ministral SQL, k=2 gains +12.0pp versus +14.4pp at k=6; the multi-iteration TextCraft lifts at k=4, 8, and 16 were +20.7, +27.6, and +12.1pp, within single-repeat noise.Source: F ROLLOUT AND EXPERT BUDGET SENSITIVITY · pp. 22–23
  • The top decile gains +7.1pp over base on average and the quarter-budget +9.2pp, but B=50% averages +6.8pp and trails the quarter-budget in 5 of 6 settings.The authors attribute the downturn to more heterogeneous evidence being compressed into a single convention.Source: F ROLLOUT AND EXPERT BUDGET SENSITIVITY · pp. 22–23
  • Raising the support floor from m=2 to m=3 reduces coverage in 10 of 16 settings, collapsing Ministral's TextCraft set from 37 to 9 tasks.Using m=1 instead helps TextCraft with Ministral (+5.2 vs. +1.7pp at the same firing set).Source: F ROLLOUT AND EXPERT BUDGET SENSITIVITY · p. 23
  • With the authoring call and gate fixed, all four selectors yield statistically indistinguishable full-set gains of +6.6 to +7.6pp across 16 settings.The passage notes that outcome information's value is concentrated in all-fail tasks and per-task inference-time routing.Source: F ROLLOUT AND EXPERT BUDGET SENSITIVITY · p. 23

G COVERAGE OF ACQUISITION SIGNALS

  • +12.5pp average gain over base: outcome-MI beats random (+8.2pp), success-MI (+6.9pp), and entropy (+6.8pp) in all 8 settings.The selection-rule comparison holds rollouts, authoring, and gate fixed.Source: G COVERAGE OF ACQUISITION SIGNALS · p. 24
  • On Gemma-12B Bash, outcome-MI reaches 31.7 (+12.5pp) while success-MI remains at the 19.2 base with no mixed contrasts or authored rule.Outcome-MI acquires 1.9× more decisions overall than success-MI in the reported comparison (274 against 146 on one SQL model).Source: G COVERAGE OF ACQUISITION SIGNALS · p. 24

H SCOPE OF THE ACQUISITION-FUNCTION COMPARISON

  • The acquisition-function comparison covers 8 settings across SQL and TextCraft, so its evidence is broad across models but narrower across environments.Its absolute values are incomparable with Table 5 because it predates the equal-expert protocol.Source: H SCOPE OF THE ACQUISITION-FUNCTION… · p. 24

I WHY SELF-SUPERVISED CURATION SATURATES

  • Selective ACE and ReasoningBank variants mostly trade failure for stagnation; only two baseline-favoring settings occur, both on Gemma-3-4B and near the floor.The authors conclude that better memory selection cannot recover information the generating agent never possessed.Source: I WHY SELF-SUPERVISED CURATION SATURATES · p. 24

J ERROR GATE ABLATION

The error gate prevents learned rules from undoing solved attempts, while its contribution and the authored rule’s contribution vary across benchmarks. Gate fidelity and expert strength also affect deployment behavior.

J ERROR GATE ABLATION

  • Gating yields 32–36 SQL win→lose flips under unconditional injection versus 0 when gated, and SynAct regresses on none of the aggregate settings.The gate withholds a rule when the unaided attempt succeeds, avoiding the need for a held-out validation split.Source: J ERROR GATE ABLATION · p. 25
  • The served gate reaches recall 1.0 at precision 0.58–0.86 on TextCraft, Bash, and τ-bench, but SQL recall is 0.22 at precision 0.50.Across settings, 48–84% of gate firings are on unrepairable failures, wasting a retry but producing zero win→lose churn.Source: J ERROR GATE ABLATION · p. 25
  • Gating ACE and AWM libraries recovers +4.6 and +5.0pp versus SynAct’s +8.2pp over 12 settings, while SynAct rules repair 18% of fired-on failures versus 11% for either baseline.On InterCode-Bash, targeted acquisition leads (+11.7 vs. +3.1 and +2.5pp); the balance between gating and acquired knowledge varies by benchmark.Source: J ERROR GATE ABLATION · p. 25
  • Switching the authoring expert from Opus to Sonnet lowers mean firing-set lift from +8.8 to +4.9pp over eight settings, with Opus ahead or tied in seven.On Bash with Qwen, Sonnet’s convention prescribed the buggy aggregation and its antipattern never matched, so the rule deactivated.Source: J ERROR GATE ABLATION · p. 25

K ACCURACY ACROSS ACQUISITION ITERATIONS

Across ten iterations, gains vary: some settings peak early, while others continue improving. The reported stability is consistent with the gate withholding rules when their antipattern is absent.

K ACCURACY ACROSS ACQUISITION ITERATIONS

L FULL PER-ITERATION AND PER-ACQUISITION TABLES

The appendix reports statistical comparisons, acquisition-function results, and iteration-level accuracy patterns. Its full grids accompany the body’s acquisition comparison and the per-iteration curves.

L FULL PER-ITERATION AND PER-ACQUISITION TABLES

  • Across 16 settings, one-iteration SynAct beats base in all 16 and each individual baseline in 14–15; routing beats base and cascade in all 16 and the judge in 14.The report flags near-floor comparisons and uses paired exact sign tests across settings.Source: L FULL PER-ITERATION AND… · p. 26
  • The appendix supplies full grids for acquisition-function comparisons and per-iteration accuracy, alongside the training-method accuracy table.Table 6 holds rollouts, authoring call, and gate fixed while varying selection; Table 7 groups iteration accuracy by benchmark and model.Source: L FULL PER-ITERATION AND… · p. 26, Table 5 · p. 27
  • Outcome-MI wins every acquisition-comparison column; action entropy reaches +6.8pp over base, while random selection reaches +8.2pp.Action entropy can rank cosmetic variation alongside outcome-separating decisions when arm counts match.Source: L FULL PER-ITERATION AND… · p. 26
  • Success-MI reaches +6.9pp but has no usable evidence on Gemma-3-4B TextCraft when no task contains both a winning and losing rollout.Outcome-MI retains information when different actions lead to distinguishable outcomes, including when every rollout fails.Source: L FULL PER-ITERATION AND… · pp. 26–27
  • When one convention captures the dominant failure mode, gains can peak early; when tasks contain several modes, later iterations can add value.Only one of the sixteen rows peaks at iteration 10, and the base column pools unaided measurements across completed iterations.Source: L FULL PER-ITERATION AND… · p. 27

M COST OF A ROUTED EXPERT BUDGET

SynAct spends more agent computation to reduce expert usage, with expert turns especially relevant for human deployment. The accounting distinguishes single-iteration savings from costs that change as iterations accumulate.

M COST OF A ROUTED EXPERT BUDGET

  • At one iteration, SynAct uses roughly 15× fewer expert tokens than the compared training methods, calling the expert once per iteration rather than once or twice per training task.It runs k=8 rollouts per training task, so total token count alone obscures the opposing agent- and expert-compute costs.Source: M COST OF A ROUTED EXPERT BUDGET · pp. 27–28
  • For a human expert, one intervention per iteration is more feasible as training data grows than methods whose expert cost scales with the training set.A worked-example digest spans 49 blocks and roughly 27k characters, while the returned convention is 2–4 lines.Source: M COST OF A ROUTED EXPERT BUDGET · p. 28

N COST ACCOUNTING DETAILS

  • At one iteration, SynAct spends 0.01M expert tokens versus 0.15M for ReasoningBank and 0.20M for ACE; the reduction narrows to 5–6.7× at three iterations.The largest savings occur in the single-iteration regime, where most accuracy gain is obtained.Source: N COST ACCOUNTING DETAILS · p. 29
  • Expert tokens are 0.13% of SynAct’s budget at every iteration count, versus 17% for ACE and 14% for ReasoningBank; expert turns are one per iteration versus 50–100 per baseline run.Table 8 reports estimated token counts and exact turn counts.Source: N COST ACCOUNTING DETAILS · p. 29, Table 8 · p. 29
  • At one iteration, SynAct uses 8× more agent tokens because it runs k=8 rollouts; reducing k to 2 still beats base on all six settings.The passage contrasts this reducible agent-side cost with baseline expert costs that cannot be reduced without changing those methods.Source: N COST ACCOUNTING DETAILS · p. 29

O INFERENCE-TIME ROUTING DETAILS

At inference time, SynAct routes tasks for expert intervention when pooled rollouts disagree and none has already succeeded, then uses outcome-MI to select a decision for correction. Compared with action entropy, its advantage is strongest when both signals route the same tasks but choose different intervention points.

O INFERENCE-TIME ROUTING DETAILS

  • For each task, SynAct routes only when rollout outcomes disagree below θ=0.6 and no pooled rollout has already succeeded; it selects a decision with I(v⋆).The expert correction starts the retry sequence, followed by ordinary self-reflection attempts.Source: O INFERENCE-TIME ROUTING DETAILS · p. 29
  • The matched-budget comparison caps cascade, action-entropy, and expert-judge routing at SynAct’s expert-call count B.Action entropy selects the most uncertain tasks, while the judge makes routing decisions itself; Table 9 reports held-out accuracy at θ=0.6 and R=3.Source: O INFERENCE-TIME ROUTING DETAILS · p. 29, Table 9 · p. 30
  • Across the full retry range, routed variants dominate the no-expert pool at matched R.Figure 6 compares accuracy against retry budget for the no-expert pool with self-reflection, SynAct, and SynAct-rules at θ=0.6.Source: Figure 6 · p. 30, O INFERENCE-TIME ROUTING DETAILS · p. 31
  • Across the two SQL settings, sweeping θ from 0.5 to 0.8 changes accuracy by at most 1.3pp while expert calls roughly triple; TextCraft with Qwen gains +6.9pp from θ=0.5 to 0.7.The authors use θ=0.6 across benchmarks and models without per-setting tuning.Source: O INFERENCE-TIME ROUTING DETAILS · p. 30
  • At R=10, accuracy never falls and rises in 11 of 16 settings, with the largest gain, +11.3pp, for Ministral on τ-bench.This comparison uses the same expert budget and θ=0.6 as R=3.Source: O INFERENCE-TIME ROUTING DETAILS · p. 30, Table 10 · p. 31

P ROUTING SIGNAL AT INFERENCE: OUTCOME-MI AGAINST ACTION ENTROPY

  • On the shared routed set, outcome-MI rescues 13–21% of tasks versus 5–15% for action entropy, a 1.4×–2.7× advantage.The full-benchmark accuracy gap is 1.8–4.4pp across three models, but that comparison also reflects the rest of the routing pipeline; Gemma-3-4B is excluded from the controlled comparison.Source: Table 11 · p. 31, P ROUTING SIGNAL AT INFERENCE:… · p. 31
  • At inference time, each consultation has no aggregation slack across tasks, making the branch point itself a bottleneck.By contrast, aggregating evidence into a training digest can let the expert recover from imperfect selection.Source: P ROUTING SIGNAL AT INFERENCE:… · p. 31

Related papers

Loading 2610.12651v1…