Source-linked AI summary
No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning
Zhicong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li, Yong Liu
TL;DR
Critique-guided RL suffers when static critics become stale as on-policy failure patterns evolve. ECHO jointly optimizes policy and critic through cascaded diagnosis–refinement rollouts, saturation-aware gain shaping, and synchronized updates, yielding more stable training and sustained refinement across open-world tasks. Its effectiveness depends on the quality and calibration of the external reward model, and reward evaluation remains separate from critique generation.
Problem
Static or decoupled critics become stale as evolving on-policy trajectory distributions and failure patterns reduce the usefulness of their feedback.
Method
ECHO synchronously co-evolves policy and critic using cascaded diagnostic-and-corrective rollouts, saturation-aware gain shaping, and dual-track GRPO.
Results
ECHO supports more stable training and sustained refinement by keeping diagnostic granularity aligned with evolving policy failure modes.
Takeaways & Limitations
Co-evolving the critic with the policy provides a framework for maintaining diagnostic alignment during on-policy long-horizon refinement.
Takeaways & Limitations
ECHO depends on the external reward model’s quality and calibration, because noisy, biased, or underspecified rewards can promote evaluator artifacts.
Abstract
from arXiv · showhide
Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments.
1 Introduction
Critique-guided RL adds diagnostic feedback to sparse outcome rewards, but static or decoupled critics become stale as on-policy failure patterns shift. ECHO addresses this through synchronized policy–critic co-evolution, cascaded refinement, saturation-aware rewards, and dual-track optimization.
- Outcome rewards lack the actionable diagnostic information needed to refine LLM-agent behavior efficiently.
- Template-based critics are inexpensive but cannot adapt feedback to an agent’s specific actions, while separate critics remain decoupled from policy learning.
- As on-policy policies evolve, their trajectory distributions and failure patterns shift from coarse mistakes toward subtle defects, making stationary critics stale.
- ECHO jointly updates policy and critic, evaluating critiques by the performance gains they induce after refinement rather than by plausibility alone.
- ECHO combines cascaded diagnostic-and-corrective rollouts, saturation-aware gain shaping, and dual-track GRPO to improve stability and performance across tasks.
2 Related Work
Prior work uses templates or separately trained critics to provide language feedback, but most training-based critics remain frozen or asynchronous. ECHO instead targets critic staleness by coupling critic updates to on-policy policy evolution.
- Language critics address the limited diagnostic value of scalar outcome rewards in long-horizon LLM-agent decision-making.
- Template-based methods range from generic hints and error-conditioned prompts to partial solutions and coarse-to-fine guidance.
- Training-based critics generate diagnostic feedback through dedicated models, including critique–refinement loops and staged training pipelines.
- Most training-based critics are trained off-policy and then frozen or updated asynchronously, so their usefulness decays as policy trajectories and failure patterns change.
- Figure 1 contrasts decoupled frozen critics with ECHO’s jointly updated policy–critic loop, where diagnoses guide trajectory refinement.
3 Methodology
ECHO co-evolves policy and critic models through cascaded diagnosis and refinement, group-relative advantages, saturation-aware critic rewards, and synchronized GRPO updates. Its reward shaping emphasizes difficult last-mile improvements near performance saturation.
- Co-evolutionary framework: ECHO treats policy–critic refinement as a shared on-policy synchronization problem rather than static supervision.
- Co-evolutionary framework: The policy converts critic diagnoses into corrective actions, while the critic is rewarded for feedback that increases the policy’s performance.
- Cascaded evolutionary rollout: These cascaded trajectories form baseline, critique, and refinement groups that supply empirical signals for co-evolutionary optimization.
- Cascaded evolutionary rollout: The cascaded rollout first samples an initial trajectory, scores it, generates multiple score-aware diagnoses, and then conditions corresponding policy refinements on those critiques.
- Saturation-aware reward design: Saturation-aware gain shaping assigns larger gains to equal improvements made at higher scores, encouraging critiques that identify subtle flaws in near-correct proposals.
- Saturation-aware reward design: The gain is path-consistent and antisymmetric, supporting multi-step refinement while rewarding improvements and penalizing regressions on one scale.
- Synchronized co-evolutionary optimization: ECHO builds interdependent diagnostic and corrective groups, computes group-relative advantages for policy refinements and critic rewards, and updates both models synchronously with GRPO.
4 Experiment Setup
ECHO is evaluated across four environments spanning web navigation, embodied planning, scientific reasoning, and deep search. Experiments use Qwen backbones and compare against diverse proprietary and open-source baselines.
- The evaluation covers WebShop, ALFWorld, SciWorld, and DeepSearch for web navigation, embodied tasks, scientific reasoning, and multi-turn information synthesis.
- Experiments use Qwen3-4B-Instruct-2507 and Qwen2.5-7B as primary backbone models, with the critic defaulting to the policy’s backbone.
5 Results
ECHO improves open-world agent learning across four benchmarks while addressing non-stationary failure patterns and critic staleness. Its synchronized critic-policy updates and saturation-aware shaping support stronger refinement, especially near reward ceilings.
- Overall effectiveness: 47.25 on DeepSearch versus 33.25 for GRPO represents roughly a 42% relative increase, while WebShop rises from 82.37 to 90.03.These are the most salient Qwen3-4B gains reported for long-horizon search and web interaction.
- Overall effectiveness: ECHO improves performance across all four benchmarks, achieving an average gain of 7.28 points over GRPO.It also matches or surpasses listed strong models on the other benchmarks except DeepSearch, where GPT-5 attains the best score.
- Failure-pattern drift: Failure-density centers shift substantially from early to late training, showing that dominant error causes change rather than merely shrinking a fixed mistake set.In ALFWorld and SciWorld, distributions are more dispersed and partly overlapping, but their density mass still migrates across phases.
- Critic synchronization: Frozen critics degrade performance across all environments, with the largest degradation on ALFWorld and SciWorld, where stale diagnoses can become redundant or off-target.The frozen variant can look strong early on WebShop but is overtaken later; ECHO separates from GRPO mainly in the mid-to-late stage on ALFWorld and SciWorld.
- Saturation-aware shaping: Saturation-aware shaping concentrates more probability mass where post-refinement reward exceeds pre-refinement reward and improves near-ceiling refinement on WebShop and SciWorld.Its benefit is larger on WebShop because that environment more often reaches a near-ceiling regime.
6 Conclusion
ECHO co-evolves the critic and policy for open-world LLM agents, keeping diagnostics aligned with changing failure modes. Its synchronized updates and complementary rollout and gain-shaping designs support more stable training and sustained refinement.
- ECHO synchronizes critic and policy updates to mitigate critic staleness under on-policy failure drift.
- Cascaded rollouts provide group-structured samples for group-relative optimization.
- Saturation-aware gain shaping boosts last-mile improvements, supporting sustained refinement and more stable training.
Limitations
ECHO depends on the quality and calibration of the external reward model, and its current implementation separates reward evaluation from critique generation. The evaluation environments are summarized by task settings, required abilities, and official scoring criteria.
- ECHO’s effectiveness depends on reward quality and calibration because noisy, biased, or underspecified rewards can induce evaluator artifacts.
- The policy may inherit misalignment when the critic optimizes toward artifacts of a noisy, biased, or underspecified evaluator.
- Reward evaluation and critique generation are handled by separate models in the current implementation.
- Table 3 summarizes evaluation environments by task setting, required core abilities, and official scoring criterion.
- Table 4 compares ECHO with established critique-guided baselines across four environments.
B More Implementation Details
Experiments use matched policy and critic learning rates, eight-member rollout groups, and the official policy reward and evaluation protocols.
- The policy and critic use the same learning rate, lrθ = lrψ = 1 × 10−6.
- Each query uses a rollout group of N = 8 independent critiques and eight corresponding refinements.
- The policy follows the official reward design and evaluation protocols for fair comparison.
- The critic uses the reward function in Eq. (6) with η = 0.1 across experiments.
C Additional Experimental Analyses
Additional analyses compare ECHO with critique-guided baselines, isolate the value of critic-guided refinement from extra sampling, and examine performance across environments and training phases. ECHO consistently outperforms the compared baselines, while critic-guided refinement produces larger reward gains than no-critic regeneration, especially late in training.
- Additional comparisons: Additional experiments compare ECHO with RCO and LUFFY beyond the standard GRPO baseline.
- Additional comparisons: ECHO consistently outperforms both baselines across all four evaluated environments.
- Additional comparisons: ECHO achieves higher performance than LUFFY without access to teacher-derived ground-truth solutions.
- Additional comparisons: RCO performs worse in this setting because it keeps the policy fixed while iteratively refining and evaluating trajectories.
- Critique-guided refinement: The controlled analysis compares critic-guided refinement with no-critic regeneration under the same second-pass sampling setup.
- Critique-guided refinement: Across both datasets and all training phases, critic-guided refinement yields substantially higher reward gains than no-critic regeneration.
- Critique-guided refinement: Relative gain becomes larger late in training, suggesting increasing critic effectiveness as policy and critic co-evolve.
C.3 Critique–Refinement Alignment and Granularity Evolution
Critique–refinement alignment rises across training as critiques become progressively more fine-grained, matching the policy’s shift from basic interaction errors to planning and reasoning failures.
- Critique Granularity Evolution: Early critiques are dominated by coarse guidance, while mid-level and fine-grained critiques become increasingly prevalent later.Fine-grained critiques grow substantially in the late phase, indicating more precise error localization.
- Critique Granularity Evolution: As the policy improves, critiques shift from procedural guidance toward higher-level decision-making and subtle reasoning flaws.The analyzed rollouts associate this shift with changing failure modes, from basic environment interaction to suboptimal planning and long-horizon reasoning.
- Critique Granularity Evolution: Increasingly precise and actionable feedback is associated with sustained performance gains in later training stages.The results attribute improved alignment to a systematic shift in critique granularity.
C.4 Training Time Analysis
ECHO adds critic rollout, evaluation, and refinement stages to training, with refinement supplying most of the extra computation. The reported average wall-clock increase is roughly 15% over GRPO, while the co-evolution mechanism itself is described as manageable.
- Training Time Analysis: ECHO decomposes training time into policy rollout, critic rollout, and refinement, while baseline GRPO reports rollout and total training time.The comparison uses Qwen3-4B across multiple environments under the same experimental setup.
- Training Time Analysis: The additional stages are described as negligible relative to baseline GRPO’s total training time, so co-evolution is not identified as the bottleneck.The added stages include initial trajectory generation, critic evaluation, and updates.
- Training Time Analysis: Roughly 15% average wall-clock time increase over GRPO represents ECHO’s reported training-cost trade-off.The paper attributes most extra computation to refinement because it processes longer contexts.
- Training Procedure: ECHO updates policy and critic tracks on the same on-policy batch after cascaded rollout, saturation-aware reward computation, and dual-track advantage estimation.Algorithm 1 summarizes the synchronized GRPO updates and the stages used to produce them.
- Critic Prompt: The critic prompt grounds feedback in official scoring information and restricts output to brief, high-level improvement suggestions.The prompt directs the critic to omit praise and focus on behavioral guidance.