PaperlayerView exact revision on arXiv

Source-linked AI summary

Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents

Haobin Li, Liang Jiang, Zhenyu Huang, Mouxing Yang, Xi Peng

arXiv:2609.38885v1cs.SE

TL;DR

Coding agents incur token overhead through frequent environment interactions, while inference-time efficiency controls risk discarding task-relevant information. HERO trains agents with resolution-prioritized, multi-granularity reinforcement learning; experiments report a favorable resolution-rate and token-efficiency trade-off across SWE-bench benchmarks.

  • Problem

    Frequent environment interactions create token overhead, while inference-time context manipulation and interaction limits risk discarding task-relevant information and complicate the resolution-efficiency trade-off.

    Source: §1 · p. 1

  • Method

    HERO uses hierarchical policy optimization that prioritizes task resolution and trajectory- and turn-level credit assignment to encourage efficient reasoning patterns.

    Source: §1 · pp. 2–3

  • Results

    Across SWE-bench Verified and SWE-bench Multilingual, HERO achieves a favorable resolution-rate and token-efficiency trade-off versus state-of-the-art coding agents and reinforcement learning methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Efficiency variation among successful paths and entropy-associated unproductive turns provide guidance for learning effective and efficient reasoning patterns.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Recently, coding agents have emerged as a dominant paradigm for real-world software engineering (SWE) scenarios, which solve complex tasks through multi-turn interactions with development environments. However, frequent interactions with environments would inevitably introduce substantial token overhead, leading to high usage costs and latency. Although recent studies have explored reducing token usage by context manipulation and interaction limits at inference time, these approaches focus on improving token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution rate and token efficiency. In this paper, we study a more general paradigm without suffering from the limitation, i.e., training token-efficient coding agents with promising resolution performance, which is a highly-practical yet less-explored problem. To this end, we reveal two core observations in SWE scenarios: i) Efficiency Variation: successful resolution could be achieved with fewer tokens; ii) Entropy Correlation: unproductive behaviors are associated with turn-level entropy. Motivated by observations, we propose a novel reinforcement learning framework, dubbed HERO. Specifically, HERO prioritizes task resolution over token efficiency during policy optimization and encourages efficient reasoning patterns at both trajectory and turn levels. Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that HERO achieves a favorable trade-off between resolution rate and token efficiency compared with state-of-the-art coding agents and reinforcement learning methods.

1 INTRODUCTION

The introduction frames token efficiency as a training problem: coding agents need to reduce interaction costs without sacrificing resolution, and existing inference-time controls risk discarding task-relevant information. HERO combines resolution-prioritized optimization with trajectory- and turn-level credit, drawing on observations about efficient successful paths and entropy-associated unproductive turns.

1 INTRODUCTION

  • Frequent environment interactions add token overhead, and inference-time context pruning or interaction limits risk discarding task-relevant information while balancing resolution and efficiency.Existing token-efficient RL methods also risk discouraging necessary reasoning steps when they simply penalize token-intensive trajectories in long-horizon SWE tasks.Source: §1 · pp. 1–2
  • For a given task, successful trajectories vary in token use, while low-entropy turns are more prone to repeated tool calls, excessive exploration, and erroneous commands.The paper uses these observations to motivate trajectory-level efficiency and turn-level identification of unproductive behavior.Source: §1 · p. 2, Figure 1 · p. 2
  • 42.2% average resolution rate, up from 37.4% across Qwen3.5 variants on SWE-bench series; HERO also achieves a favorable trade-off on Verified and Multilingual against state-of-the-art agents and RL methods.Compared with GRPO, HERO yields better average resolution-rate improvement with 39.8% less token usage.Source: §1 · pp. 2–3
  • HERO prioritizes task resolution, activates efficiency optimization after sufficient capability is acquired, and uses trajectory- and turn-level credit to encourage concise paths and penalize unproductive turns.The framework is presented as a training-time alternative that internalizes effective and efficient reasoning patterns in the agent policy.Source: §1 · pp. 2–3

2 RELATED WORK

The related-work section reviews coding agents and token-efficient reinforcement learning, positioning this work around long-horizon SWE interactions and the joint goal of resolution and efficiency.

2.1 CODING AGENTS

  • Coding-agent approaches divide into scaffold-based methods, which provide general-purpose tools and workflows, and training-based methods, which improve use of a scaffold through fine-tuning or reinforcement learning.Prior coding agents emphasize resolution on complex SWE tasks while giving limited attention to token use in long-horizon interactions.Source: §2.1 · p. 3

2.2 TOKEN-EFFICIENT REINFORCEMENT LEARNING

  • Token-efficient RL includes reasoning-oriented methods that constrain or filter chain-of-thought length and agent-oriented methods that penalize cumulative tokens, mainly in simple agentic tasks.The paper contrasts its approach as identifying and discouraging unproductive turns in long-horizon SWE tasks while exploring joint preservation of resolution and token efficiency.Source: §2.2 · p. 3

3 METHOD

HERO formulates token-efficient coding-agent training as a resolution–token-use trade-off, then combines resolution-prioritized optimization with trajectory- and turn-level efficiency credit.

3.1 PROBLEM FORMULATION

  • For each repository-and-issue task, the objective is to maximize TRS, which summarizes average resolution rate against cumulative token use.Source: §3.1 · p. 4

3.2 HIERARCHICAL POLICY OPTIMIZATION

  • HERO gates efficiency optimization on within-group resolution and uses resolution-first clipping so resolved trajectories retain priority over unresolved ones.Source: Figure 2 · p. 5, §3.2 · p. 5

3.3 MULTI-GRANULARITY EFFICIENCY CREDIT ASSIGNMENT

  • Efficiency credit is assigned at both rollout and turn granularity to target token overhead from unproductive behavior.Source: §3.3 · p. 5, §3.3.2 · p. 6

3.3.1 COARSE-GRAINED TRAJECTORY ADVANTAGE

  • To avoid conflating efficiency with resolution, HERO compares token use within same-outcome rollout subsets and standardizes the resulting rewards into trajectory advantages.Source: §3.3.1 · pp. 5–6

3.3.2 FINE-GRAINED TURN ADVANTAGE

  • Turn-level credit targets potentially unproductive low-entropy turns, using the group’s lowest-entropy 20% as its threshold.Source: §3.3.2 · p. 6

4 EXPERIMENTS

Experiments on SWE-bench Verified and SWE-bench Multilingual compare HERO with coding agents and reinforcement-learning methods, then examine its training dynamics, token use, unproductive turns, and design choices. The reported comparisons favor HERO's resolution-rate/token-efficiency trade-off, while the analyses describe how its efficiency and resolution develop.

4.1 EXPERIMENT CONFIGURATIONS

  • The evaluation covers 500 SWE-bench Verified issues and 300 SWE-bench Multilingual tasks across nine programming languages, using Qwen3.5 model sizes and a shared Claude Code scaffold.RL comparisons use 640 multilingual SWE training tasks; experiments are repeated with three random seeds and report mean results.Source: §4.1 · p. 6

4.2 COMPARISONS WITH STATE-OF-THE-ART METHODS

  • Across SWE-bench Verified and Multilingual, HERO prioritizes task resolution over token efficiency while reducing token overhead from unproductive interactions, achieving desirable resolution rates and token efficiency.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8

4.3 ABLATION AND ANALYTIC STUDY

  • During training, HERO outperforms GRPO on resolved rate, token usage, and TRS, with stronger gains later; token use rises until step 40 and then declines as resolution remains high.The authors interpret this pattern as early development of issue-solving capability followed by later token reduction while preserving resolution.Source: §4.3 · p. 8, Figure 3 · p. 8
  • HERO uses fewer tokens than GRPO for both resolved and unresolved trajectories, according to the reported token-usage distributions.The authors suggest this reflects learned efficient reasoning patterns that can achieve resolution with fewer tokens.Source: §4.3 · p. 8, Figure 4 · p. 8
  • HERO has lower rates of repeated tool calls, overexploration, and tool errors than both the base model and GRPO.The authors associate this result with penalizing low-entropy turns to discourage unproductive behaviors.Source: Figure 5 · p. 8, §4.3 · p. 9
  • Removing hierarchical policy optimization reduces token use but degrades resolution, while coarse- and fine-grained efficiency components both contribute to lower token use.The reported parameter analysis finds stable performance improvement with low token consumption for λ ∈[0.2, 0.4].Source: §4.3 · p. 9, Table 3 · p. 9, Figure 6 · p. 9
  • Across interaction turns, HERO shifts from inspection toward execution, following GRPO's reasoning progression while using fewer tokens.The authors describe this as internalizing efficient reasoning without undermining progression from code exploration to solution generation.Source: §4.3 · p. 9, Figure 7 · p. 9

5 CONCLUSION

The paper presents HERO as an RL framework for training token-efficient coding agents while preserving resolution performance. It reports favorable benchmark trade-offs and identifies future extensions to more complex long-horizon agentic tasks.

5 CONCLUSION

  • HERO combines multi-granularity credit assignment for efficient reasoning with hierarchical policy optimization that prioritizes task resolution over token efficiency.Experiments on the SWE-bench series are reported to significantly outperform existing methods and achieve a favorable resolution-rate/token-efficiency trade-off.Source: §5 · p. 9
  • The authors plan to extend HERO to more complex long-horizon agentic tasks, including scientific data analysis and engineering design.Source: §5 · p. 9

A MORE IMPLEMENTATION DETAILS

  • The appendix provides further implementation details on training data, settings, entropy estimation, and identifying unproductive behaviors.The listed training data comprise 640 multilingual SWE tasks, including tasks in Python, Java, C/C++, JavaScript/TypeScript, PHP, Go, and Rust.Source: A MORE IMPLEMENTATION DETAILS · p. 14, §A.1 · p. 14

A.2 MORE IMPLEMENTATION SETTINGS

The appendix specifies additional training and evaluation settings, describes turn-level entropy estimation, and outlines how unproductive behaviors are identified.

A.2 MORE IMPLEMENTATION SETTINGS

  • Training batches contain eight tasks with 16 sampled trajectories per task; evaluation uses the final training checkpoint, and Qwen3.5-35B-A3B uses λ=0.1 and τ=0.8.The default tool setup disables WebFetch, WebSearch, and Agent; the turn-level advantage is assigned to every generated token in that turn.Source: §A.2 · p. 14, Table 4 · p. 14

A.3 ENTROPY ESTIMATION

  • Turn-level entropy estimation uses top-10 token probabilities for token-level entropy and defines each turn's generated-token set and token entropy.Source: §A.3 · p. 14

A.4 IDENTIFICATION OF UNPRODUCTIVE BEHAVIORS

  • The appendix defines repeated tool calls, excessive exploration, and erroneous commands as unproductive behaviors, counting each behavior at most once per turn.The identification rules are presented in Table 5.Source: §A.4 · p. 14, Table 5 · p. 15

B ADDITIONAL EXPERIMENTAL RESULTS

The additional experimental results section states that, unless otherwise specified, its experiments use Qwen3.5-4B.

B ADDITIONAL EXPERIMENTAL RESULTS

B.1 COMPARISONS WITH TOKEN-EFFICIENT REINFORCEMENT LEARNING METHODS

HERO is compared with four token-efficient reinforcement-learning methods on SWE-bench Verified and SWE-bench Multilingual. It achieves a more favorable trade-off between resolution rate and token efficiency than those methods.

B.1 COMPARISONS WITH TOKEN-EFFICIENT REINFORCEMENT LEARNING METHODS

  • The comparison uses the same training and evaluation settings across methods, except for method-specific configurations.For LASER, the study uses a fixed threshold of 8,192 total output tokens across interaction turns.Source: §B.1 · p. 16
  • HERO achieves higher resolution rates with lower token usage than Training Efficient and SlimSearcher on both benchmarks, and higher TRS than the compared methods overall.LASER and OTC-GRPO use fewer tokens, but at the cost of lower resolution rates.Source: Table 6 · p. 16, §B.1 · p. 16

B.2 COMPARISONS WITH INFERENCE-TIME METHODS

Inference-time approaches and prefix-cache reuse are assessed for their effects on resolution, token usage, and inference cost. HERO-4B outperforms inference-time approaches applied to the base model on TRS, while HERO retains competitive cache reuse and relative inference cost.

B.2 COMPARISONS WITH INFERENCE-TIME METHODS

  • The study tests four inference-time approaches on Qwen3.5-4B and its HERO-trained counterpart, HERO-4B.The prefix-cache and cost analysis estimates inference cost post hoc from recorded token usage, with fixed cost weights for uncached input, cached input, and output tokens.Source: §B.2 · p. 16, §B.3 · p. 17
  • HERO-4B achieves higher TRS than inference-time approaches applied to the base model, and can still benefit from those approaches.Most tested inference-time approaches reduce token usage without improving base-model resolution; HERO can complement them.Source: §B.2 · p. 16, Table 7 · p. 17

B.3 ANALYTIC STUDY ON PREFIX-CACHE REUSE AND INFERENCE COST

  • Context-manipulation methods reduce token usage but undermine prefix-cache reuse, while interaction limits reduce relative inference cost at the expense of serious resolution degradation.HERO achieves competitive prefix-cache reuse and relative inference cost.Source: Table 8 · p. 17, §B.3 · p. 17

B.4 ABLATION STUDIES ON SWE-BENCH MULTILINGUAL

The analyses examine HERO ablations, token usage, and scaffold generalization. They report trade-offs in multilingual ablations, high input-token overhead, and better resolution with lower token consumption than GRPO across two scaffolds.

B.4 ABLATION STUDIES ON SWE-BENCH MULTILINGUAL

  • On SWE-bench Multilingual, removing hierarchical policy optimization reduces token usage at the cost of resolution, while coarse- or fine-grained efficiency advantages reduce tokens and improve resolution.The ablation comparison is reported in Table 9.Source: §B.4 · pp. 17–18, Table 9 · p. 18

B.5 ANALYTIC STUDY ON TOKEN USAGE BREAKDOWN

  • Input tokens account for more than 99% of total consumption, and HERO achieves higher resolution rates with fewer interaction turns and lower token consumption than GRPO on both benchmarks.Tables 10 and 11 report the token usage and interaction-turn analysis.Source: §B.5 · p. 18, Table 10 · p. 18, Table 11 · p. 18

B.6 ANALYTIC STUDY ON SCAFFOLD GENERALIZATION

  • Even when trained on the Claude Code scaffold, HERO achieves higher resolution rates with lower token consumption than GRPO under both Claude Code and mini-SWE-agent scaffolds.The additional scaffold comparison is on SWE-bench Verified.Source: Table 12 · p. 18, §B.6 · p. 18

B.7 MORE DETAILED ABLATION STUDIES

The detailed ablations test components of hierarchical optimization and turn-level penalties, while sensitivity analyses examine the capability threshold and entropy percentile. The findings describe how these choices relate to resolution and token efficiency.

B.7 MORE DETAILED ABLATION STUDIES

  • Either efficiency gating or resolution-first clipping helps preserve resolution, while resolution-calibrated credit assignment supports efficient reasoning without conflating token efficiency with resolution.The ablations also examine outcome conditioning in coarse-grained efficiency advantage.Source: Table 13 · p. 19, §B.7 · p. 19

B.8 SENSITIVITY TO THE CAPABILITY THRESHOLD

  • HERO achieves a desirable trade-off between resolution performance and token efficiency when the capability threshold τ is in [0.5, 0.6].Figure 8 presents the threshold analysis.Source: §B.8 · p. 19, Figure 8 · p. 19

B.9 SENSITIVITY TO THE ENTROPY PERCENTILE

  • A larger entropy percentile generally reduces token usage but degrades resolution, suggesting that penalizing too many turns might suppress necessary reasoning.Figure 9 presents the entropy-percentile analysis.Source: §B.9 · p. 19, Figure 9 · p. 20

B.10 ANALYTIC STUDY ON FINE-GRAINED TURN ADVANTAGE

  • On SWE-bench Verified, penalizing low-entropy turns achieves higher resolution with lower token consumption than randomly penalizing 20% of turns or penalizing the 20% with highest entropy.The comparison is reported in Table 14.Source: Table 14 · p. 20, §B.10 · p. 20

Related papers

Loading 2609.38885v1…