PaperlayerView exact revision on arXiv

Source-linked AI summary

Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents

Xingjian Diao, Tianyu Yang, Wenjun Huang, Xiangchi Yuan, Chunhui Zhang, Xingchen Zhao, Weiyi Wu, Haotian Xu, Soroush Vosoughi, Xiangliang Zhang, Jiang Gui

arXiv:2610.16027v1cs.CLcs.AI

TL;DR

Tool-use agents can receive step-level feedback without knowing which decisions caused, inherited, or later repaired an error. ECHO distinguishes responsibility from credit allocation, and leads the compared methods on five of six benchmarks at both model scales.

  • Problem

    Step-level feedback localizes errors, but the step where an error surfaces may not be responsible when errors are inherited, consequences shared, or repairs made later.

    Source: §1 · p. 1

  • Method

    ECHO uses a frozen judge to identify responsibility from policy-visible prefixes, counts shared consequences once, and funds self-induced repair credit from the source error’s penalty.

    Source: §1 · p. 2

  • Results

    At both 4B and 8B, ECHO leads the compared methods on five of six tool-use benchmarks.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The results support decision responsibility as a useful basis for fine-grained credit assignment across complex tool-use trajectories.

    Source: §5 · p. 9

    ECHO addresses training-time credit assignment within agent–tool interaction trajectories, not inference-time intervention or broader system-level concerns such as tool construction and multi-agent coordination.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning improves tool-use agents, yet fine-grained credit assignment remains challenging even with step-level feedback. An observed error may be inherited from an earlier decision, while multiple responsible decisions may share the same downstream consequence or participate in later recovery. We introduce Error-Chain-Aware Optimization (ECHO), a responsibility-aware credit assignment framework that separates responsibility identification from credit allocation. A frozen judge combines retrospective trajectory analysis with decision-time assessment from the policy-visible prefix to distinguish newly introduced errors, inherited consequences, missed corrections, and successful corrections. ECHO then counts shared downstream consequences once across responsible decisions and constrains rewards for repairing self-induced errors through source-funded recovery, producing bounded action-level supervision alongside terminal supervision. Experiments on six diverse tool-use benchmarks at both 4B and 8B scales show strong and consistent improvements over the compared baselines, with ECHO achieving the highest score on five of six benchmarks at each scale. These results support decision responsibility as a useful basis for fine-grained credit assignment in tool-use agents. Our code is available at https://github.com/xid32/ECHO.

1 Introduction

Step-level feedback can locate observed errors without identifying responsible decisions, especially when errors are inherited, shared, or repaired. ECHO separates responsibility identification from credit allocation across error chains.

1 Introduction

  • Step-level feedback localizes errors, but the surfacing step may not be responsible: decisions can inherit errors, share consequences, or repair them later.The paper identifies responsibility and credit allocation as separate questions.Source: §1 · p. 1
  • The framework distinguishes newly introduced errors, inherited consequences, missed corrections, and successful corrections.An inherited consequence occurs before the discrepancy is visible; a missed correction leaves a visible, correctable deviation unaddressed.Source: §1 · p. 2
  • ECHO uses a frozen judge to assess responsibility from policy-visible prefixes, count shared consequences once, and fund repair credit from the source error’s penalty.Full-trajectory analysis proposes candidate events and verifies propagation or repair; bounded action-level corrections are optimized alongside terminal reward.Source: §1 · p. 2
  • At both 4B and 8B, ECHO leads the compared methods on five of six tool-use benchmarks.Case studies illustrate behavioral improvements, and ablations support each component.Source: §1 · p. 2

2 Related Work

Prior work advances tool-use learning and provides denser process feedback, but localizing a problematic step does not by itself establish decision responsibility. ECHO separates responsibility identification from credit allocation.

2 Related Work

  • Process supervision and LLM critics localize problematic steps, but do not alone establish responsibility or prevent judging decisions with later evidence unavailable at decision time.The cited process-supervision work includes Lightman et al. (2024), Wang et al. (2024), Zhang et al. (2025b), and Zheng et al. (2025).Source: §2 · p. 2
  • Tool-use RL methods primarily optimize trajectory-level outcomes or scale training environments, while ECHO addresses how supervision is distributed across agent decisions.The cited methods include Qian et al. (2026), Feng et al. (2026a), and Zhao et al. (2025).Source: §2 · p. 2
  • ECHO counts each shared consequence once across responsible decisions and bounds self-induced repair credit by the originating error’s budget.This separates responsibility identification from credit allocation.Source: §2 · p. 3

3 ECHO

ECHO complements terminal supervision with responsibility-aware action corrections. It identifies responsibility from decision-time context, counts shared consequences once, and funds self-induced recovery from source penalties.

3 ECHO

  • ECHO complements terminal rewards with signed action-level corrections, while separately clipped outcome-level and action-level objectives use KL regularization.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Problem Setup and Responsibility Events

  • Each responsibility event records the assessed action, affected requirement predicate, deviation origin, responsibility type, supporting evidence, and feasible alternative for missed corrections.Source: §3.1 · p. 3

3.2 Responsibility Identification from Decision-Time Prefixes

  • A frozen judge proposes events retrospectively but labels selected actions from their prefixes, distinguishing NEW, MISS, INH, and CORR; structural checks and reliability gating can reject evidence.Source: §3.2 · pp. 3–4

3.3 Shared-Consequence Accounting

  • Repeated manifestations of one predicate impairment form one consequence; linked responsibility slots share it equally, with ownership uncertainty reserved rather than reassigned.Source: §3.3 · p. 4

3.4 Source-Funded Recovery Budgets

  • Normalized positive and negative budgets produce action corrections; INH and abstentions receive no auxiliary budget, while self-induced repair credit is capped by its source penalty.Source: §3.4 · p. 5

3.5 Length-Balanced Policy Optimization

  • Local corrections can persist when group outcome advantages are zero, and action-wise averaging avoids length dilution, though clipping can make split and combined objectives diverge.Source: §3.5 · p. 5

4 Experiments

Experiments evaluate ECHO against scale-specific baselines on six tool-use benchmarks at 4B and 8B, alongside case studies, component and reward ablations, and frozen-judge comparisons. ECHO leads five benchmarks at both scales, while the ablations and judge comparisons examine its supervision design and judge choice.

4.1 Setup

  • Evaluation covers six benchmarks at 4B and 8B, with ECHO trained from Qwen3 policies on τ 2-Bench data and compared against scale-specific baselines.The benchmarks include function calling, conversational task completion, state milestones, and multi-step, multi-turn tool use.Source: §4.1 · p. 6

4.2 Main Results

  • ECHO leads five of six benchmarks at both scales; its sole exception is ACEBench-Agent, where it trails the best baseline by 0.87 points at 4B and 0.62 at 8B.Its largest gains at both scales are on ToolSandbox, which is not used during training.Source: §4.2 · p. 6
  • At 8B, ECHO leads all four BFCL-v3 conditions, but τ 2-Bench Airline is 42.10 versus Simia-RL’s 49.00; overall τ 2-Bench remains 40.56 versus 40.16.The Airline deficit is offset by ECHO’s Telecom and Retail advantages over Simia-RL.Source: §4.2 · p. 7, Table 3 · p. 7, Table 2 · p. 7

4.3 Case Studies

  • Figure 3 presents case studies comparing Qwen3 before and ECHO after training on the same task, with their behavior compared on matched task requirements.Source: §4.3 · p. 7, Figure 3 · p. 8

4.4 Component Ablation

  • Removing any of four ECHO components reduces performance on both benchmarks; removing prefix assessment causes the largest drop, while removing source-funded recovery causes the smallest.The ablations remove prefix assessment, shared-consequence accounting, source-funded recovery, or reliability gating.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

4.5 Reward Ablation

  • On τ 2-Bench at 8B, ECHO exceeds Final Reward Only by 9.51 points and the strongest heuristic, First Error, by 2.77 points overall.Although All Errors leads Airline and First Error leads Telecom, ECHO has the highest Retail and overall scores.Source: §4.5 · p. 8, Figure 5 · p. 8

4.6 Effect of the Frozen Judge

  • Frozen-judge rankings reverse: Nemotron-3-Super-120B has the lowest BFCL-v4 score yet beats both smaller Qwen judges downstream; Kimi-K2.6 Thinking exceeds Qwen3.5-122B-A10B by 1.24 percentage points.The comparisons include downstream ECHO performance and responsibility-identification accuracy on 100 manually verified samples before reliability gating.Source: §4.6 · p. 8, Figure 6 · p. 8

5 Conclusion

ECHO distinguishes four responsibility events and converts them into bounded action-level supervision. Across six tool-use benchmarks and two scales, it improves over compared baselines, supporting decision responsibility as a useful perspective for fine-grained credit assignment.

5 Conclusion

  • ECHO distinguishes newly introduced errors, inherited consequences, missed corrections, and successful corrections, converting them into bounded action-level supervision.The framework addresses conflating error manifestations with responsible decisions, double-counting shared consequences, and over-crediting recovery from self-induced errors.Source: §5 · pp. 8–9
  • Across six tool-use benchmarks at 4B and 8B, ECHO achieves strong and broadly consistent improvements over the compared baselines.Source: §5 · p. 9
  • The results support decision responsibility as a useful perspective for fine-grained credit assignment across complex tool-use trajectories.Source: §5 · p. 9

Limitations

The work focuses on responsibility-aware credit assignment for tool-use agents, with training-time attribution rather than inference-time intervention and a defined system-level scope.

Limitations

  • The framework focuses specifically on responsibility-aware credit assignment for tool-use agents.Source: Limitations · p. 9
  • ECHO is designed for training-time credit assignment, not inference-time intervention; reflection, replanning, and online verification are complementary but outside this work.Source: Limitations · p. 9
  • The analysis is limited to responsibility attribution in agent–tool trajectories, excluding tool construction, multi-agent coordination, and interface design.Source: Limitations · p. 9

Ethical Considerations

The paper reports no method-specific ethical concerns beyond those generally associated with tool-use agents and automated model evaluation. Its appendices detail candidate selection, responsibility boundaries, and calibration limits for selective supervision.

Ethical Considerations

  • The authors identify no method-specific ethical concerns beyond those generally associated with tool-use agents and automated model evaluation.They report using publicly available datasets and models licensed for research use.Source: Ethical Considerations · p. 9

A.1 Candidate Selection and Independent Requests

  • The audit proposes at most sixteen event hypotheses, then independently reviews up to eight selected actions using policy-visible prefixes.Six proposed actions and two sentinel actions are selected; retrospective labels and future observations are excluded from prefix requests.Source: §A.1 · pp. 12–13

A.2 Responsibility Boundaries

  • Responsibility depends on what discrepancy evidence was visible at decision time, and ambiguous origins or restoration attribution lead to abstention.A missed correction requires a feasible response; a legitimate repair attempt retains its prefix-grounded assessment even if an unpredictable outage prevents restoration.Source: §A.2 · p. 13

B Calibration of Selective Supervision

  • Calibration bounds stratum-level semantic-error rates under the calibration distribution, not individual-event validity, impact magnitude, or reliability under policy shift.Insufficiently sampled or unreliable strata abstain, and post hoc stratum selection uses an independent split.Source: B Calibration of Selective Supervision · p. 13

C Derivations and Guarantee Boundaries

These derivations bound shared-consequence and recovery budgets and characterize when the split local surrogate agrees with a combined objective. The guarantees have explicit assumptions and do not establish behavioral guarantees under shared policy parameters.

C.1 Shared-Consequence and Budget Bounds

  • The shared-consequence bound is independent of the number of implicated events, while additional responsibility events can reduce individual normalized shares.The total auxiliary budget is capped; direct-defect terms separately supervise certified decisions.Source: §C.1 · p. 14

C.2 First-Order Equivalence

  • At the behavior policy, the split objective has the same gradient as the additive token-advantage objective; objective values agree when relevant ratios remain inside the clipping interval.Outside that interval, opposite-sign advantages may activate different clipping branches and produce distinct objectives despite an identical KL term.Source: §C.2 · p. 14

C.3 Source Funding and the Clipped Local Surrogate

  • Under the stated source-funding condition, the fixed-batch bound applies to self-induced components with action-disjoint support or independently evaluated local surrogates.A counterexample shows that for H > 2/η, the update can increase error probability despite satisfying the source-funding constraint; shared policy parameters can also increase error creation.Source: §C.3 · pp. 14–15

C.4 Observational Attribution Limits

  • A failed trajectory can fit causal models with different intervention effects, so the trace alone may not determine intervention-based responsibility.ECHO instead uses an ownership convention to allocate observed constraint impairment.Source: §C.4 · p. 15

D Implementation Details

The implementation trains ECHO from Qwen3 policies on a small τ 2-Bench training split, with a frozen judge and fixed optimization settings. The update audits trajectories, applies action-level coefficients, and tracks how clipping changes realized supervision.

D.1 Training Data and Implementation

  • ECHO starts from Qwen3-4B and Qwen3-8B and trains for 75 updates on 92 τ 2-Bench tasks, with 12 development tasks.Each update uses 12 tasks per batch, eight rollouts per task, and a learning rate of 5 × 10−6.Source: §D.1 · p. 15

D.2 ECHO Configuration

  • A fixed Kimi-K2.6 Thinking judge proposes events and assesses prefixes, with at most eight prefix assessments per trajectory.The configuration sets κ = 0.25, α = 0.5, λ− = λ+ = 0.5, η = 0.5, and clipping half-width ϵ = 0.2.Source: §D.2 · p. 15

E.1 Update Sequence

  • Each update audits successful and failed trajectories, sets source penalties before normalizing and capping eligible corrections, then optimizes fixed aggregated action coefficients.The judge operates during training, and the selected coefficients satisfy the stated source-funding condition.Source: §E.1 · p. 15

E.2 Audit Cost

  • Audit cost includes one proposal request and at most K independent prefix requests; uncached prefix input scales up to O(Kni).Exact-prefix caching can reduce repeated computation, while batching preserves request count.Source: §E.2 · p. 15

E.3 Realized Local Coefficients

  • Active negative-advantage ratios can exceed the upper clipping threshold, making realized coefficient magnitudes larger than their on-policy masses.Realized coefficients and sign-specific clipping fractions characterize how supervision changes through aggregation and clipping.Source: §E.3 · pp. 15–16

F Reward Ablation on Multi-Turn Function Calling

On BFCL-v3 Multi-Turn, ECHO outperforms terminal-only and heuristic reward alternatives overall, while heuristic leaders vary across subsets. The section also specifies the responsibility taxonomy used to interpret decisions and benchmark diagnostics.

F Reward Ablation on Multi-Turn Function Calling

  • 8.76 percentage points over Final Reward Only and 4.42 points over All Errors overall on BFCL-v3 Multi-Turn at 8B.Figure 7 reports the reward ablation; ECHO improves on Missing Function while retaining gains over Final Reward Only in Missing Parameter and Long Context.Source: Figure 7 · p. 16, F Reward Ablation on Multi-Turn… · p. 16
  • Subset leaders vary across Base, Missing Parameter, and Long Context, and ECHO achieves the highest overall score on both BFCL-v3 Multi-Turn and τ 2-Bench.Random One Error leads Long Context but has a substantial Base deficit.Source: F Reward Ablation on Multi-Turn… · p. 16

G.1 Event-Level Decision Rule

  • NEW, MISS, INH, and CORR distinguish new defects, missed feasible corrections, inherited deviations, and execution-verified repairs.Types depend on decision-time evidence; insufficient or conflicting evidence can lead to abstention, while admitted INH events receive no auxiliary budget.Source: §G.1 · pp. 16–17

G Responsibility Event Taxonomy and Benchmark Coverage

  • The diagnostic tables map checker signals to evidence requirements using 23 literal error codes and 19 derived signals from 14,014 scored trajectories.The families can overlap within a trajectory, and 2,319 occurrences fall outside the listed families.Source: Table 4 · p. 17, Table 5 · p. 18, Table 6 · p. 18

G.2 Error Content versus Responsibility

  • Responsibility describes a decision’s relation to a failure, not the failure’s content, so the same failure can receive different types across decisions.ECHO uses this distinction for training-time credit across tools and domains rather than enumerating surface errors.Source: §G.2 · pp. 17, 19

G.3 Mapping Benchmark Diagnostics to Responsibility Evidence

The section maps benchmark diagnostics to evidence needed for responsibility assignment, then illustrates responsibility events and paired behavioral differences in qualitative evaluation trajectories.

G.3 Mapping Benchmark Diagnostics to Responsibility Evidence

  • The six benchmarks expose complementary error-chain evidence, while checker diagnostics identify failed checks but do not establish responsibility without decision-time, provenance, feasibility, and restoration evidence.When only aggregate scores are available, responsibility is assessed against task-requirement predicates extracted independently of outcomes.Source: §G.3 · p. 19

H Diagnostic Codebook

  • The codebook records 23 literal checker codes and 19 derived signals across 14,014 scored Qwen3 base-model trajectories, with diagnostic counts that are not additive.Literal codes come from benchmark checkers; derived signals use benchmark records, and responsibility assignment still requires evidence beyond semantic families.Source: H Diagnostic Codebook · p. 19

I Additional Case Studies

  • The case studies apply the responsibility decision rule to evaluation trajectories at 4B and 8B, but are qualitative examples that do not estimate pattern frequencies.Single-turn BFCL defects are established from visible constraints and submitted calls; other execution claims rely on displayed tool results or state changes.Source: I Additional Case Studies · p. 19

I.1 Responsibility Events in Base-Model Trajectories

  • NEW events contradict visible constraints or contracts; INH events reflect inaccessible facts lost earlier, MISS events repeat feasible-to-correct rejected calls, and CORR events verify restoration.Events are assessed per action and predicate, so one trajectory can include a MISS on one requirement and a CORR on another.Source: §I.1 · p. 19

I.2 Paired Comparisons Before and After ECHO Training

Paired Qwen3 and ECHO evaluations compare the same task and scale against shared requirements. The examples contrast error introduction, inherited information loss, and missed corrections with ECHO’s corresponding behaviors.

I.2 Paired Comparisons Before and After ECHO Training

  • Paired runs share evaluation settings, but compare each trajectory against the same requirement rather than aligning action indices because interaction histories can diverge.Figures 12–16 show paired Qwen3 and ECHO comparisons on the same tasks.Source: §I.2 · pp. 19–20, Figure 12 · p. 25, Figure 13 · p. 26
  • When Qwen3 introduces a NEW error, ECHO’s action on that requirement is CLEAN; when Qwen3 inherits lost information, ECHO preserves the fact upstream.Inherited-information events receive no local correction.Source: §I.2 · p. 20
  • When Qwen3 misses a correction, ECHO either avoids the discrepancy or completes the pending operation, with restoration verified by tool results or final state.Some Qwen3 examples first complete a prerequisite repair, such as logging in or freeing inbox capacity.Source: §I.2 · p. 20

Related papers

Loading 2610.16027v1…