PaperlayerView exact revision on arXiv

Source-linked AI summary

Elastic Horizon: Discovering the Effective Interaction Frontier in Agentic Reinforcement Learning

Gangyi Zhang, Junjie Meng, Letian Zhang, Wei Wu, Yang Zheng, Dong Wang, Yang Liu, Guanjun Jiang, Chongming Gao

arXiv:2609.07247v1cs.AI

TL;DR

Existing horizon schedules monotonically expand interaction budgets without detecting when additional steps stop improving performance. Elastic Horizon is a closed-loop controller that estimates the dynamic frontier from the 90th percentile of successful trajectories, stabilizes within the saturation band from both initialization regimes, and achieves the best reported success rates while saving up to 25% of per-step trajectory tokens.

  • Problem

    Existing horizon curricula monotonically increase interaction budgets toward a manual maximum without detecting whether further expansion remains beneficial.

  • Method

    Elastic Horizon is a closed-loop controller that estimates the effective interaction frontier from the 90th percentile of successful trajectory lengths.

  • Results

    Elastic Horizon stabilizes within the saturation band from under- and over-capacity initializations, achieves the best success rates across 7B and 14B backbones on AppWorld and BFCL, and cuts per-step trajectory tokens by up to 25%.

  • Takeaways & Limitations

    The results shift horizon scaling from deciding how far to expand toward deciding when to stop.

  • Takeaways & Limitations

    Successful-trajectory statistics can be unreliable during cold starts or on extremely difficult tasks when successful trajectories are rare, potentially delaying adaptation.

Abstract

from arXiv · show

Scaling the interaction horizon-the maximum number of environment interactions per episode-improves LLM agents on long-horizon tasks, and curriculum-based methods that progressively expand the horizon outperform fixed-horizon alternatives. However, existing schedules are open-loop: they monotonically increase the horizon until a manually specified maximum, with no mechanism to detect when further expansion stops helping. We propose the effective interaction frontier hypothesis: a dynamic boundary beyond which additional interactions yield diminishing returns while cost grows linearly. We then introduce Elastic Horizon, a closed-loop controller that tracks this boundary via the 90th percentile of successful trajectory lengths. On AppWorld and BFCL, fixed-horizon sweeps reveal clear saturation plateaus; Elastic Horizon stabilizes the horizon inside the saturation band from both under- and over-capacity initializations, attains the best success rates across 7B and 14B backbones, and saves up to 25% of per-step trajectory tokens. Our work shifts the paradigm from how to scale interaction horizons to when to stop scaling.

1 Introduction

Existing horizon curricula expand interaction budgets monotonically toward a manual maximum, despite evidence that performance plateaus beyond a task-dependent frontier. Elastic Horizon replaces this open-loop design with a closed-loop controller that tracks demonstrated capability and stabilizes horizons within the saturation band.

  • Motivation: Existing schedules increase horizons monotonically to a manually specified maximum without testing whether further expansion remains beneficial.This design assumes performance improves monotonically with interaction length.
  • Empirical finding: Fixed-horizon sweeps show success improving in the under-capacity regime before plateauing at an effective interaction frontier, while trajectory cost continues growing linearly.The frontier is task-dependent and marks diminishing returns from additional interaction budget.
  • Methodology: Elastic Horizon estimates the capability boundary from the 90th percentile of successful trajectory lengths, filtering inefficient-exploration outliers.Unlike open-loop schedules, it adapts the horizon from demonstrated behavior rather than training progress.
  • Methodology: The bidirectional controller expands when capability grows and contracts near the task-complexity ceiling, enabling convergence from arbitrary initializations.Open-loop schedules cannot converge from an over-capacity start because they only increase the horizon.
  • Results: Elastic Horizon stabilizes within the saturation band from both high and low initializations, validating automatic frontier detection.This behavior is reported on AppWorld and BFCL experiments.
  • Results: Elastic Horizon achieves the best results across 7B and 14B backbones while reducing AppWorld per-step trajectory tokens by up to 25%.The paper frames this as shifting attention from how to scale interaction horizons to when to stop scaling.

2 Related Work

Prior work improves long-horizon agent training through progressive horizon curricula, complementary stabilization methods, and test-time resource allocation. Elastic Horizon differs by adapting training horizons from observed trajectory statistics and by using successful-trajectory percentiles as a competence signal.

  • Horizon curricula: ScalingInter-RL and related curricula progressively increase interaction horizons, while fixed large horizons can destabilize training or degrade performance.These approaches use predetermined functions of training steps.
  • Positioning: Elastic Horizon uses closed-loop control based on observed trajectory statistics, unlike prior open-loop schedules tied to training-step progress.Its target is the effective frontier during training rather than a manually scheduled horizon path.
  • Complementary methods: Other long-horizon methods address hierarchical learning, variance filtering, task curricula, or memory-efficient optimization rather than horizon-frontier tracking.These methods are presented as orthogonal aspects of long-horizon training.
  • Test-time allocation: Test-time methods allocate inference resources using explicit budget or prompt-difficulty signals, whereas Elastic Horizon optimizes training efficiency with implicit capability signals.The two classes of approaches are complementary and target different phases.
  • Curriculum learning: Elastic Horizon applies the upper percentile of successful trajectory lengths as a competence signal for episode-horizon curriculum control.This extends curriculum learning into an underexplored dimension: episode horizon length.

3 Preliminaries

The paper models multi-turn agentic tasks as sparse-reward POMDPs in which an interaction horizon limits episode length and affects both success probability and computational cost. It contrasts these task constraints with prior open-loop horizon schedules that depend only on training progress.

  • Task formulation: Multi-turn agentic tasks are formulated as episodic decision problems with sparse terminal rewards under a simplified POMDP.The task model includes instruction, state, action, observation, transition, and reward components.
  • Task formulation: The policy generates actions from the instruction and interaction history, and trajectory length counts the number of interaction turns.The policy is parameterized by an LLM with parameters θ.
  • Horizon constraints: A horizon constraint K forcibly terminates unfinished trajectories at K steps and assigns zero terminal reward.Successful completion receives terminal reward one.
  • Horizon constraints: The horizon constraint sets the maximum interaction turns per episode, directly affecting task completion probability and computational cost.Success rate is defined under trajectories sampled with the specified horizon constraint.
  • Horizon scheduling: Existing curriculum schedules set K_t from predetermined training-step rules, using additive or multiplicative horizon growth between lower and upper bounds.Both schedules are open-loop because they ignore observed performance and task-distribution complexity.

4 Method

The method is motivated by an empirical effective interaction frontier: beyond a task- and policy-dependent threshold, extra interaction budget adds linear cost without measurable success gains. Elastic Horizon therefore re-estimates this moving boundary from successful trajectories during training.

  • Frontier motivation: Fixed-horizon results show success-rate plateaus beyond a task-dependent threshold while training compute continues growing linearly.This contradicts the assumption that larger interaction budgets always improve performance.
  • Frontier motivation: The effective interaction frontier H∗ is the empirical boundary where larger horizons pay additional step cost without raising success rate, whereas smaller horizons cap achievable performance.The paper defines H∗ from the observed plateau rather than adding further formalism.
  • Dynamic frontier: Because H∗ depends on task distribution and evolving policy parameters, it can shift as the agent becomes more capable.This time variation makes a fixed horizon inadequate and motivates repeated re-estimation.
  • Capability estimation: Elastic Horizon uses successful-trajectory length distributions as a behavioral surrogate for the unobserved frontier.The supplied method passage introduces this surrogate and points to the capability-estimation procedure.

BUFFER

Elastic Horizon uses successful trajectory lengths to estimate capability and adapt the interaction horizon through a smoothed, bidirectional closed loop. Its design combines percentile-based robustness, exploration headroom, cold-start safeguards, and convergence toward the effective frontier.

  • Buffer and capability signal: Successful trajectory lengths populate a FIFO buffer that supplies the controller’s capability signal.The buffer retains successful trajectories for horizon estimation.
  • Bidirectional adaptation: Unlike monotonic open-loop schedules, Elastic Horizon expands with capability growth and contracts near saturation, enabling convergence from arbitrary initializations.The method is designed to track the effective frontier rather than increase toward a preset maximum.
  • Robust estimation: The controller uses P90 of successful trajectory lengths to capture near-maximal capability while filtering inefficient-exploration outliers.P90 balances sensitivity to harder solvable instances against robustness to anomalously long trajectories.
  • Exploration headroom: A fixed headroom increment encourages exploration beyond demonstrated capability and reduces premature convergence to a suboptimal frontier.The headroom supplies slack for longer trajectories required by harder tasks.
  • Smoothing: EMA smoothing trades adaptation speed against stability: small α adapts slowly, whereas large α responds faster but may oscillate.The smoothed update stabilizes noisy percentile estimates.
  • Cold start: When successful samples are scarce, the controller retains the current horizon until sufficient data accumulates.This cold-start strategy avoids unreliable P90 estimates during early exploration.

5 Experiments

Experiments on AppWorld and BFCL test horizon saturation, adaptive frontier discovery, and comparisons with fixed and open-loop baselines. Elastic Horizon stabilizes within task-dependent saturation bands and combines strong performance with lower token consumption.

  • RQ1: Horizon saturation: AppWorld 7B reward rises from approximately 17% at K=10 to approximately 60% at K=30 in the under-capacity regime.The sweep trains separate fixed-horizon models with K from 10 to 80.
  • RQ1: Horizon saturation: Runs plateau across K ∈[30, 60] on AppWorld and K ∈[20, 50] on BFCL, despite a 2× range of per-step cost.Beyond these bands, additional budget yields no systematic improvement; AppWorld has the higher task-dependent saturation threshold.
  • RQ2: Automatic frontier discovery: Elastic Horizon expands from K0 = 10 and contracts from K0 = 50, stabilizing inside the saturation band without reaching Kmax.Its behavior is based on demonstrated successful trajectory lengths rather than training step.
  • RQ3: Comparison with baselines: On AppWorld, GRPO (K=15) benefits from longer horizons, whereas on BFCL GRPO (K=50) suffers training instability, illustrating task-dependent fixed-horizon trade-offs.TTI and ScalingInter reach Kmax regardless of task complexity and therefore adapt inconsistently across benchmarks.
  • RQ3: Comparison with baselines: Elastic Horizon achieves the best results across both benchmarks at 7B and 14B, including AppWorld Best@8=90.42% at 14B.The trained 7B model also surpasses the cited zero-shot Qwen2.5-32B and Qwen3-235B-A22B results on AppWorld.
  • RQ3: Comparison with baselines: Elastic Horizon saves 25% per-step tokens at the mid-training peak and 11% versus the worst baseline at convergence.The reported cumulative-token saving is 28% at SR=50% on AppWorld 7B.

6 Conclusion

Elastic Horizon replaces manually scheduled horizon growth with closed-loop control based on demonstrated capability, while exposing empirical and scope limitations. Its reported results show quality gains and token savings, but the controller remains heuristic and uses a global horizon and proxy efficiency metric.

  • Contribution: Elastic Horizon estimates the effective interaction frontier H∗ from the 90th percentile of successful trajectory lengths and sets the interaction budget from demonstrated behavior.The controller uses a closed-loop heuristic rather than a fully theoretical frontier estimate.
  • Findings: Fixed-horizon training saturates beyond a task-dependent H∗, with similar reward across a twofold budget range while per-step cost continues growing.This saturation pattern is reported on AppWorld and BFCL.
  • Findings: Elastic Horizon expands and contracts the horizon and settles inside the saturation band from under- and over-capacity initializations.This capability-based behavior differs from monotonically increasing schedules and is supported by convergence experiments.
  • Practical impact: Elastic Horizon attains the best success rates across 7B and 14B backbones while cutting per-step trajectory tokens by up to 25%.On AppWorld, token reductions are also reported as 11% at convergence relative to fixed-horizon baselines.
  • Limitations: The method’s main limitations are sparse early successes, a single global horizon, heuristic P90 selection, and token counts that imperfectly proxy total training compute.These constraints can delay adaptation, misallocate budgets across heterogeneous tasks, leave percentile choice task-dependent, and diverge from wall-clock savings.
  • Theoretical scope: The paper does not claim a closed-form relationship between the P90 estimator and H∗, and treats convergence behavior as empirically supported rather than theoretically established.The EMA’s stability is attributed to the filter’s known behavior, while the frontier connection remains an empirical question.

D.3 Baseline Implementations

The experiments compare fixed-horizon and open-loop curriculum baselines using specified horizon schedules and fixed endpoints. They also report quantile statistics and percentile choices for evaluating adaptive horizon control.

  • Baseline schedules: The multiplicative schedule uses K ∈{15, 20, 30, 50} at training steps {0, 50, 100, 150}, capped at Kmax = 50.It starts from K=15 and raises the horizon at successive stage boundaries.
  • Baseline schedules: Fixed-horizon baselines use GRPO with K = 15 and K = 50 to represent conservative and maximum interaction budgets.
  • Experimental protocol: All training runs use one compute node with 8 AI accelerator devices, while the controller adds less than 0.1% overhead.A 7B run requires approximately 320 device-hours and a 14B run approximately 680 device-hours under the stated settings.
  • Experimental protocol: Main experiments use 3 random seeds, but 14B multi-seed variance is omitted because each run requires approximately 680 device-hours.Table 6 reports mean@8 ± standard deviation for the 7B configuration.
  • Evaluation statistics: The baseline analysis includes quantile statistics of successful trajectory lengths and compares percentile choices using Mean@8 across benchmarks.

E Trajectory Length Distribution Analysis

Trajectory lengths are positively skewed, so P90 is positioned as a compromise between the conservative mean or median and unstable extreme values. Ablations show that P90 offers the strongest supported tradeoff among the tested statistics and headroom settings.

  • Trajectory statistics: Successful trajectory-length distributions have positive skewness, indicating heavy-tailed rather than symmetric behavior.
  • Trajectory statistics: The mean is substantially lower than P90, so using the mean produces an overly conservative horizon for harder task instances.
  • Trajectory statistics: The gap between P90 and Max reflects inefficient-exploration outliers that would inflate the horizon estimate and destabilize training.
  • Percentile ablation: Performance increases from P50 to P90 on both benchmarks, then gradually decreases toward Max.The pattern is consistent with the heavy-tail interpretation.
  • Percentile ablation: P90 provides the best tested tradeoff by accommodating challenging tasks while remaining robust to occasional outliers.Lower percentiles converge around K ≈15-19, whereas P95 and Max push K toward Kmax.
  • Headroom and smoothing ablations: Headroom ∆= 0 collapses the horizon toward Kmin (K = 6.8), while ∆= 20 raises the converged horizon to K = 35.9 and erodes adaptive efficiency.The authors claim only that ∆= 10 is best among the tested values, not that success changes smoothly with ∆.
  • Headroom and smoothing ablations: The EMA coefficient α trades responsiveness against stability: α=0.1 adapts slowly, while α=1.0 tracks noisy raw P90 estimates and causes oscillations.

F.4 Sample Efficiency Comparison

Elastic Horizon improves sample efficiency by reaching comparable success with fewer trajectory tokens than fixed-horizon and open-loop alternatives. The reported savings arise from shorter early trajectories and automatic avoidance of exploration beyond the effective frontier, while several cold-start and task-allocation limitations remain.

  • Token efficiency: 28% fewer cumulative tokens are required to reach SR=50% than GRPO (K=50), using 36.28M versus 50.21M tokens.
  • Token efficiency: At peak performance, Elastic Horizon achieves the highest SR (56.87%) while consuming fewer total tokens than the strongest fixed-horizon baseline.
  • Mechanism: The token savings come from shorter early-training trajectories and automatic convergence that avoids exploration beyond H∗.
  • Token efficiency: Elastic Horizon reaches matched success rates with strictly fewer cumulative tokens than every fixed-horizon baseline on AppWorld.At SR=50%, the cumulative gap is 28% versus GRPO (K=50).
  • Per-step efficiency: On BFCL 7B, Elastic Horizon consumes fewer per-step tokens than open-loop baselines throughout training.The gap is most visible during the rapid expansion phase of TTI/ScalingInter.
  • Limitations: During cold start, scarce successful trajectories can make P90 unreliable, so maintaining the initial horizon until the buffer fills may delay adaptation.Future directions include pretrained-policy initialization and hybrid open-loop/closed-loop schedules.
  • Limitations: A single global horizon may waste budget on easy tasks and truncate hard tasks when task complexity varies substantially.Per-task or per-difficulty-level horizon allocation is proposed as future work.

G.3 Non-Stationary Task Distributions

Elastic Horizon’s convergence discussion assumes a stationary task distribution, leaving adaptation under distribution shift as an open boundary. The method controls a global interaction horizon during training and uses EMA smoothing in a closed-loop update, while formal guarantees remain limited.

  • Distribution shift: The convergence analysis assumes a stationary task distribution, so P90 may lag when task difficulty shifts during training.The FIFO buffer provides some adaptivity, but stronger shifts may require drift detection or weighted buffers.
  • Theoretical scope: The formal discussion does not establish a closed-form bound between ˆBt and H∗ or analyze EMA convergence beyond a stationary-target assumption.The authors identify percentile optimality, headroom selection, and frontier evolution as open questions.
  • Scope of adaptation: Elastic Horizon optimizes computational interaction complexity rather than conceptual task difficulty.Many API calls can make a conceptually simple task interaction-heavy, while a complex reasoning query may require few steps.
  • System scope: Training-time horizon adaptation and test-time budget control address different lifecycle stages and are described as complementary.
  • Controller operation: The controller updates one global horizon through a closed-loop phase using EMA smoothing after trajectory collection and buffer maintenance.The algorithm logs the horizon, estimated boundary, and success rate during training.
Loading 2609.07247v1…