Source-linked AI summary
Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Hongbang Yuan, Zhuoran Jin, Yixin Cao
TL;DR
Long-horizon RL for autonomous agents is hindered by sparse rewards, ineffective exploration, and limitations of SFT warming. The paper constructs feedback-enriched environments that adapt environmental feedback across exploration and training, finding consistent gains and several training-related effects. These effects include more stable dynamics, proactive exploration, policy internalization of guidance, and a consistency boundary for optimization.
Problem
Long-horizon RL agents face sparse rewards and ineffective exploration, while SFT warming is costly to scale and can constrain exploration.
Method
The paper constructs FEEs by adapting environmental feedback, shifting from agent-side warming to environment-side adaptation.
Results
FEEs consistently improve RL training across multiple benchmarks, model scales, and optimization algorithms.
Takeaways & Limitations
FEEs stabilize training, encourage proactive state-space exploration, internalize environmental guidance into policy weights, and make intra-group feedback consistency important for stable optimization.
Takeaways & Limitations
The study evaluates only SciWorld and BFCL, manually specifies stage-dependent settings, and observes limitations in more challenging environments.
Abstract
from arXiv · showhide
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional \textit{agent-side warming} up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to \textit{environment-side adaptation} by constructing \textbf{F}eedback-\textbf{E}nriched \textbf{E}nvironments (\textbf{FEEs}). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs \textbf{(1)} stabilizes training dynamics by reducing entropy volatility, \textbf{(2)} facilitates proactive state-space exploration in difficult tasks, \textbf{(3) }ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and \textbf{(4) }identifies intra-group feedback consistency as a critical boundary for stable optimization.
1 Introduction
Long-horizon RL agents struggle with sparse rewards, ineffective exploration, and costly or constraining SFT-based warming. The paper addresses this through feedback-enriched environments that improve training across benchmarks and reveal mechanisms behind their effects.
- Sparse rewards trap agents in zero-reward trajectories, causing vanishing gradients and leaving no learning signal.
- SFT warming is costly to scale and can overly constrain behavior, limiting exploration needed for effective RL.
- FEEs stabilize training, promote proactive exploration, internalize environmental guidance into policy weights, and make intra-group consistency important for optimization.
- FEEs shift adaptation from the agent to the environment by designing what feedback to provide and when to deliver it.
- FEEs consistently outperform standard environments across SciWorld and BFCL, Qwen3-4B/8B models, and GRPO, DAPO, and GSPO.The experiments evaluate multiple model scales and RL algorithms.
2 Preliminary
The paper models agent–environment interaction as goal-conditioned POMDP-based exploration with partial observations, history, trajectory rewards, and enriched observations. Enriched feedback changes the observation space and thereby shifts the agent’s action distribution.
- The environment is a goal-conditioned POMDP with states, actions, observations, transitions, and rewards.
- Given a goal, partial observation, and interaction history, the agent generates a textual action from its policy.
- Executing an action transitions the environment and produces the next observation; a trajectory receives a final trajectory-level reward, typically indicating success or failure.
- Enriched feedback intervenes in the observation space, shifting the policy’s action distribution and varying task difficulty to encourage trajectory diversity.
- Agentic RL samples candidate trajectories, computes group-statistics-based advantages, and assigns each trajectory’s advantage to its agent-generated tokens.
3 Feedback Design Strategy for FEEs
The feedback design study distinguishes action guidance from observation enrichment across intra-episode and inter-episode timing. Results support using action guidance early, observation enrichment later, and combining both stages for best performance.
- Feedback types: Action guidance narrows the policy search space, whereas observation enrichment supplies semantic information about partially observable states.Action guidance suggests immediate next steps; observation enrichment exposes hidden state information that can support remedial actions.
- Experimental design: The pilot evaluates four feedback schedules in 15-step SciWorld episodes, with guidance or enrichment assigned to early or late step ranges.AG-Early and AG-Late use steps 1–3 and 6–10 respectively; the passage introduces corresponding early and late observation-enrichment variants.
- Feedback types: Action guidance generally outperforms observation enrichment during training because it prescribes valid next steps more directly.Observation enrichment requires the agent to map additional features to optimal actions through more complex causal reasoning.
- Timing strategy: Action guidance is more effective early, while observation enrichment becomes more effective later across both exploration and training.AG-Early outperforms AG-Late initially, whereas OE-Late shows a sharper later-stage improvement than OE-Early.
- Hybrid strategy: Applying AG-Early for the first 100 training steps and OE-Late for the remaining 100 steps achieves the highest success rate.The hybrid strategy combines early search-space pruning with later policy refinement.
4 Experiments
Experiments evaluate feedback-enriched environments across SciWorld and BFCL using Qwen3 models and multiple RL algorithms. FEEs consistently improve performance over standard environments, while some missing-information tasks reveal a reliance-related limitation.
- Experimental setup: FEEs are evaluated on SciWorld and BFCL-v3 Multi-Turn with Qwen3-4B and Qwen3-8B models using GRPO, DAPO, and GSPO.The evaluation includes dynamic multi-turn interactions through predefined APIs and compares different RL optimization algorithms.
- Results presentation: Table 1 reports benchmark performance percentages and absolute percentage-point changes for FEE training relative to standard baselines, grouped by RL algorithm.Parenthetical values indicate either improvement or degradation.
- RL training: After RL training, Qwen3-4B and Qwen3-8B reach 47.81% and 48.11%, up from 23.58% and 27.18%, respectively.These results are reported as average scores and are described as approaching or exceeding the listed proprietary-model baselines.
- Main results: FEEs consistently outperform standard environments across model scales, optimization algorithms, and benchmarks.On SciWorld, Qwen3-8B with GSPO improves from 53.91% to 60.94%; on BFCL-Base, Qwen3-4B with GRPO gains 10.00 percentage points.
- Limitation: FEEs may reduce performance when tasks require questioning whether user-provided information or available tools are sufficient.Qwen3-4B with DAPO drops from 41.00% to 34.00% on Miss Param, while Qwen3-8B with GRPO drops from 52.00% to 48.00% on Miss Func.
5 Discussion
FEEs improve exploration, stabilize training, and help agents retain environmental guidance, while consistent feedback within rollout groups remains important for optimization stability.
- Training Stability: FEEs maintain steadier policy entropy and prevent premature collapse compared with standard environments.Without entropy regularization, FEE-trained agents remain steady across 300 steps, while the standard environment drops to zero near step 250.
- State-Space Exploration: FEEs improve exploration efficiency during training, with FEE-trained agents mastering a broader range of environment states.Later training stages contain more successful sampled environments under FEEs than under the standard baseline.
- State-Space Exploration: FEE-trained models transfer exploration capability to standard environments and improve performance across difficulty levels.The improvement reaches 4.3% on hard environments.
- Policy Internalization: FEE-trained models progressively assign higher probabilities to enriched feedback options even when evaluated in standard environments.Standard-trained models remain biased toward the original tool output, indicating that enriched information is not merely an inference-time hint.
- Intra-group Feedback Consistency: Consistent feedback within each rollout group supports steady optimization, whereas diverse intra-group feedback causes severe performance volatility.Excessive stochasticity can distort advantage estimation and produce noisy optimization signals.
6 Related Work
Related work addresses reward sparsity through linguistic scaffolding and task-difficulty modulation, but systematic reconstruction of dynamic multi-turn environments remains under-explored.
- 6 Related Work: Agentic language models require sequential interaction with environments through reasoning, tool manipulation, web navigation, and code execution.This framing distinguishes long-horizon agent tasks from static single-turn reasoning and generation.
- 6 Related Work: Interactive gym-style environments are increasingly used to train autonomous agents for sequential decision making in complex settings.Related work also modulates task difficulty with linguistic hints to address reward sparsity.
- 6 Related Work: Systematically reconstructing multi-turn, dynamic environments to facilitate agent evolution remains relatively under-explored.This gap is identified in the context of scaling environments rather than only scaling data.
7 Conclusion
The paper shifts long-horizon RL training from agent-side warming to environment-side adaptation through feedback-enriched environments, which consistently improve training across evaluated settings.
- 7 Conclusion: FEEs shift the focus from agent-side warming to environment-side adaptation by systematically designing richer feedback.The strategy improves RL training across multiple benchmarks, model scales, and optimization algorithms.
- 7 Conclusion: FEEs stabilize training, encourage proactive state-space exploration, internalize environmental guidance into policy weights, and require intra-group feedback consistency for stable optimization.These findings extend beyond performance gains to characterize how enriched environments affect agent learning.
Limitations
FEEs require environment-specific design choices and hyperparameters. The study’s manually specified phase settings remain underexplored in sensitivity and optimal configuration.
- FEEs require environment-specific design choices and hyperparameters.
- The experiments manually specify early and late phases for intra-episode exploration and inter-episode evolution.
- The sensitivity and optimal configurations of these stage-dependent settings remain underexplored.
Ethics and Artifact Use Statement
The work reports no significant risks and uses publicly available, open-source benchmark materials without sensitive or human-participant data. AI assistants were limited to minor writing refinement and language polishing.
- The study reports no significant potential risks associated with improving RL training in benchmarked long-horizon environments.
- The work uses official open-source codebases and benchmark environments from prior research, with artifacts appropriately cited.
- The study uses publicly available benchmark tasks and open-source research environments without personal, sensitive, or human-participant data.
- AI assistants were used solely for minor writing refinement and language polishing.
A RL Algorithms
The paper formulates RL training around grouped trajectories with trajectory-level rewards and compares token-level and sequence-level optimization strategies. GRPO and DAPO operate at the token level, whereas GSPO applies clipping and weighting once per trajectory.
- For each goal, the old policy samples trajectory groups, each trajectory receives a final reward, and expectations are taken over goals and groups sampled from the old policy.
- The group-normalized trajectory advantage is assigned to every agent-generated token in the same trajectory.
- GRPO uses token-level importance ratios and independently clips and weights each generated token while retaining trajectory-level rewards.
- DAPO preserves group-relative advantage estimation while modifying GRPO’s reduction and clipping rules, including decoupled lower and upper clipping ranges.
- DAPO’s dynamic sampling keeps only non-degenerate trajectory groups, excluding groups with identical rewards and zero normalized advantages.
- GSPO defines importance ratios at the sequence level, making clipping once per trajectory while all generated tokens share one sequence-level weight and advantage.
C Internalization Experiments
The internalization experiments probe whether agents retain enriched environmental feedback rather than merely following standard tool outputs. The materials cover SciWorld and BFCL examples, including original and enriched feedback choices and concrete tool interactions.
- Internalization Experiments: The probe tests whether the model prefers original feedback P(A) or enriched feedback P(B), with higher P(B) indicating recall and internalization of enriched information.
- Internalization Experiments: The experimental materials include tables for SciWorld and BFCL action guidance and observation enrichment, plus probing questions for Gorilla File System operations.
- Internalization Experiments: SciWorld examples frame the agent as operating in a text-based virtual environment for elementary science tasks.
- Internalization Experiments: The SciWorld task example requires finding a non-living thing, moving it to a yellow box, and tracks that six steps have already occurred.
- Internalization Experiments: The BFCL examples include user requests, tool errors, and tool outputs, such as a gallon conversion error followed by a numeric gallon result.