Source-linked AI summary
Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Lei Huang, Xiang Cheng, Chenxiao Zhao, Guobin Shen, Junjie Yang, Xiaocheng Feng, Yuxuan Gu, Xing Yu, Bing Qin
TL;DR
Scalar-reward RL underuses natural-language feedback, making exploration inefficient. GOLF aggregates external critiques and intra-group attempts into refinements, injects them as off-policy scaffolds, and jointly optimizes generation and refinement; across verifiable and non-verifiable tasks, it reports superior performance and approximately 2.2× higher sample efficiency than scalar-reward RL.
Problem
Scalar-reward RL leaves natural-language feedback underused even though it can provide error diagnoses, comparisons, and targeted revision guidance.
Method
GOLF aggregates external critiques and intra-group attempts into refinements, adaptively injects them as off-policy scaffolds, and jointly optimizes generation and refinement.
Results
GOLF reports superior performance and exploration efficiency across verifiable and non-verifiable tasks, including approximately 2.2× higher sample efficiency than scalar-reward RL.
Takeaways & Limitations
The results indicate faster convergence and higher trajectory diversity, with complementary contributions from intra-group feedback and critique.
Takeaways & Limitations
The authors caution that stronger refinement may amplify persuasive or strategically optimized content, while judge or critique bias may be reinforced during training.
Abstract
from arXiv · showhide
Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2$\times$ improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.
1. Introduction
GOLF addresses inefficient exploration in scalar-reward RL by aggregating group-level natural-language feedback into actionable refinements and adaptively injecting them into training. Experiments across verifiable and non-verifiable tasks report stronger performance, exploration efficiency, diversity, and self-refinement.
- Motivation: Scalar-reward RL underuses natural-language feedback that can diagnose errors, compare attempts, and suggest revisions.Sparse outcomes force costly trial and error and can produce all-zero groups with vanishing gradients.
- Approach: GOLF aggregates external critiques with intra-group comparisons to produce richer, more diverse, and higher-quality refinements.External critiques identify errors and targeted revisions, while alternative group attempts provide partial ideas and failure patterns.
- Approach: GOLF adaptively injects high-quality refinements as off-policy scaffolds and jointly optimizes generation and refinement in one RL loop.The design aims to provide targeted guidance in sparse-reward regions while preserving exploration capacity and improving scaffold quality over time.
- Results: Across five non-verifiable benchmarks, GOLF achieves the best final performance and outperforms the strongest baseline by 22.7%.The experiments cover verifiable and non-verifiable tasks across various model families and sizes.
- Results: GOLF achieves approximately 2.2× higher sample efficiency than vanilla scalar-reward RL and improves Pass@k on verifiable tasks.The reported gains span math reasoning, instruction following, and code generation benchmarks.
- Analysis: Analysis finds higher training entropy, stronger Pass@k, and superior self-refinement, while ablations support complementary feedback and adaptive guidance.External feedback is also used at inference time for self-refinement.
2. Related Work
Prior work uses natural-language feedback mainly for inference-time self-improvement or imitation learning, while other RL methods guide exploration with external supervision. GOLF instead jointly trains problem solving and refinement using group-level feedback.
- Natural-language optimization: Natural-language optimization research includes self-reflective experiences and iterative refinement of previous attempts at inference time.Another line of work learns from natural-language feedback through imitation learning.
- RL exploration: Exploration in LLM reinforcement learning is limited by the current policy, especially in low-reward regimes.Recent methods therefore introduce external supervision to guide exploration beyond default sampling.
- RL exploration: LUFFY uses expert demonstrations and Critique-GRPO uses refined failed responses as off-policy guidance, whereas GOLF jointly trains problem solving and refinement.These methods differ in the form of external supervision used to guide exploration.
3. Preliminaries
GRPO trains LLMs without a trainable value function by sampling response groups, assigning scalar rewards, normalizing rewards within each group, and optimizing a clipped objective. This paper uses the Dr. GRPO variant without length or standard-deviation normalization.
- GRPO: GRPO is a widely used LLM RL algorithm that simplifies PPO by eliminating the need for a trainable value function.For each prompt, it samples a group of responses and assigns each response a scalar reward.
- GRPO: GRPO computes group-relative advantages by normalizing scalar rewards within each sampled response group.The group contains N responses sampled from the old policy.
- GRPO: GRPO updates the policy with a clipped surrogate objective containing clipping and KL-penalty hyperparameters.The importance-sampling ratio appears in the objective.
- Implementation: GOLF adopts Dr. GRPO by removing length normalization and standard-deviation normalization throughout training.This choice follows prior work cited by the paper.
4. Methodology
GOLF aggregates external critiques and failed intra-group attempts into refinements, injects successful refinements as off-policy guidance in low-reward regimes, and jointly optimizes generation and refinement. This design targets sparse-reward exploration while integrating on-policy and off-policy trajectories within RL.
- Group-level Feedback Aggregated Refinement: GOLF samples response groups, obtains scalar rewards and critiques, and distinguishes external feedback from alternative intra-group responses containing complementary partial ideas.External feedback is tied to a specific response, whereas intra-group feedback comes from other responses in the same rollout group.
- Group-level Feedback Aggregated Refinement: Failed attempts and their critiques are aggregated into one refinement context, exposing diverse failure modes for synthesis.The refinement stage can identify mistakes, fill gaps, and compose complementary partial ideas.
- Adaptive Guidance via Mixed Policy Optimization: High-quality refinements are injected into sampled groups as off-policy scaffolds when on-policy groups contain only zero- or low-reward samples.The augmented group combines on-policy rollouts with injected refinement trajectories to restore informative group-relative advantages.
- Adaptive Guidance via Mixed Policy Optimization: Injection is triggered when average group reward falls below threshold τ, set to 1/N by default, and replaces one failed response with a randomly selected successful refinement.A successful refinement is selected from the refinement set only when that set is nonempty.
- Adaptive Guidance via Mixed Policy Optimization: GOLF uses mixed policy optimization over on-policy and refinement trajectories, reshaping off-policy ratios with f(u) = u/(u + λ) and omitting clipping for off-policy rollouts.The reshaping function uses λ = 0.1 and emphasizes low-probability but effective injected actions.
- Joint Optimization for Self-Refinement: Generation and feedback-conditioned refinement are optimized jointly in one RL process, forming a loop in which improved refinements become stronger off-policy scaffolds.Separate generation and refinement rollout groups are combined into a joint batch, with advantages computed within each group.
5. GOLF on Non-verifiable Tasks
On five non-verifiable benchmarks, GOLF achieves the strongest overall performance and learns more efficiently than scalar-reward and critique-only baselines. Its gains include faster early progress and higher final performance across evaluation suites.
- Results: GOLF achieves the best average score across five non-verifiable benchmarks on both model families.It surpasses the strongest baseline by +9.27 points on Llama-3.1-8B-Instruct and +2.18 points on Qwen-3-8B.
- Results: Compared with Critique-GRPO, GOLF improves average performance by +9.27 and +2.30 points on the two backbones, respectively.
- Learning efficiency: GOLF reaches comparable performance with substantially fewer training steps and ultimately converges to higher evaluation plateaus.On AlpacaEval v2.0, it matches the baseline final LC win rate in 80 steps, yielding 2.25× sample efficiency.
- Learning efficiency: GOLF achieves 2.3× and 2.1× sample efficiency over the baseline on WildBench and ArenaHard v2.0, respectively.
- Interpretation: Group-level natural language feedback accelerates policy learning and raises the achievable performance ceiling.
6. GOLF on Verifiable Tasks
GOLF is evaluated across mathematical reasoning, instruction following, and code generation with verifiable feedback. It consistently improves benchmark performance and search coverage, including under richer coding-environment feedback.
- Main results: GOLF achieves the strongest results across all verifiable mathematical reasoning and instruction-following benchmarks for both Qwen-3 model sizes.
- Mathematical reasoning: GOLF improves AIME24 and AIME25 over GRPO by +6.46 and +2.68 points on Qwen3-4B, and by +4.44 and +3.63 points on Qwen-3-8B.
- Mathematical reasoning: GOLF consistently dominates the base model and GRPO across Pass@k from k = 1 to 128 on AIME25 and AMC23.The advantage includes higher Pass@1 and more pronounced gains at Pass@128.
- Code generation: On LCBv6, GOLF reaches an Avg@4 of 47.71, exceeding GRPO by +3.63 points with 1.5× sample efficiency.It also slightly outperforms SDPO, 47.71 versus 47.51, under the stated environment feedback setting.
7. Ablation Study and Analysis
Ablations and analyses identify complementary feedback sources, adaptive off-policy guidance, mixed-policy optimization, and joint refinement training as contributors to GOLF’s exploration and refinement gains.
- Feedback sources: Removing either intra-group attempts or external critiques consistently harms performance across non-verifiable, mathematical reasoning, and instruction-following tasks.On non-verifiable tasks, the respective drops are 12.2% and 18.9%.
- Off-policy learning: Mixed-policy RL outperforms SFT when learning from the same off-policy refinement samples, with an average improvement of 37.10%.
- Off-policy learning: Mixed-policy optimization reduces the frequency of all-zero-reward groups by 31.97%, yielding usable policy gradients.
- Adaptive guidance: Adaptive injection outperforms always injecting the highest-reward refinements for each rollout group by 27.37%.Adaptive injection is triggered when the generation group enters a low-reward regime.
- Self-refinement: Joint generation and refinement training improves refinement performance in both self-refinement and guided-refinement settings.The evaluation uses RefineBench checklist-based Pass ratios.
- Exploration: GOLF maintains higher policy entropy than Pairwise-GRPO, whose entropy rapidly collapses during training.GOLF also displays recurrent entropy surges over training.
8. Conclusion
The paper presents GOLF as a natural-language-feedback-guided RL framework that converts textual feedback into actionable training signals for more efficient and diverse exploration.
- Conclusion: GOLF turns rich textual feedback into actionable training signals and aggregates feedback at the group level.The framework uses refinement and critique to densify learning signals when scalar rewards are sparse.
- Conclusion: Across non-verifiable and verifiable tasks, GOLF achieves faster convergence and higher trajectory diversity.
- Conclusion: Ablations confirm complementary roles for intra-group feedback and external critique in driving the reported benefits.
Impact Statement
The work highlights potential benefits of natural-language feedback for RL exploration while noting risks from stronger optimization and biased critiques. A preliminary study compares binary, intra-group, external, and mixed feedback conditions.
- GOLF may reduce trial-and-error during RL training, potentially lowering computational cost and improving reliability in interactive settings.
- The authors caution that stronger refinement may amplify persuasive or strategically optimized content, while LLM-based judges or critiques may introduce reinforcing bias.
- The study evaluates four feedback conditions with increasing information richness: Simple, Intra-Feedback, External-Feedback, and Mixed Feedback.
- External feedback provides critiques that pinpoint concrete errors and suggest revisions, whereas intra-group feedback supplies alternative failed attempts and partial reasoning.
A.3. Results and Analysis
The preliminary refinement study finds that explicit critiques and alternative attempts provide distinct benefits, while combining them achieves the strongest results. The section also describes prompts and feedback mechanisms used to synthesize improved responses and solutions.
- External feedback lifts pass@4 from 0% to 27.60% and Acc from 0% to 17.00%, confirming that explicit critiques provide strong corrective signals.
- Intra-group feedback raises pass@4 to 18.80% and Acc to 6.65%, suggesting that alternative attempts broaden search more than they directly fix errors.
- 30.40% pass@4 and 17.55% Acc are achieved with mixed feedback, outperforming external feedback alone by +2.80% and +0.55%, respectively.
- The refinement prompts synthesize candidate responses or solution attempts by preserving valid content, correcting errors, and satisfying the original instruction.
- For mathematical reasoning, external critique uses the ground-truth answer, while instruction-following feedback is generated from code-based constraint checks.
E. Benchmarks
The benchmarks cover non-verifiable conversation and writing tasks alongside verifiable instruction-following evaluations, using generative judges, rubrics, or automated constraint checks. The section specifies benchmark sizes, scoring procedures, and evaluation-prompt formats.
- AlpacaEval-v2 contains 805 prompts and reports length-controlled win rate using GPT-4o as the judge.
- WildBench uses 1,024 open-domain prompts, manually checked instance-level rubrics, and mean scores over values from −100 to 100.
- ArenaHard-v2 contains 500 challenging real-world queries evaluated by a GPT-4.1 judge with style control.
- CreativeWriting-v3 evaluates 96 constrained story chapters with absolute scores from 0 to 100 using GPT-4.1.
- IFEval has over 500 prompts across 25 verifiable instruction types, while IFBench covers 58 diverse constraints with corresponding verification code.
- Generative reward-model prompts use pairwise comparison, rubric-based scoring, or Likert scoring to evaluate responses.
H.2. Ablation for Adaptive Injection
The adaptive-injection ablation compares GOLF with and without adaptive selection and controls for its additional rollout compute. The associated case study shows how group-level feedback removes errors and redundancy while broadening coverage.
- “w/ adaptive” denotes GOLF, whereas “w/o adaptive” always injects the highest-reward refinement sample at each step.
- GOLF doubles rollouts per prompt, so the efficiency comparison uses a rollout-matched baseline with N=16.
- The rollout-matched comparison indicates that GOLF’s improvement comes from feedback-guided refinement rather than additional sampling.
- Group-level aggregation removes surface errors, reduces redundancy, adds socioeconomic coverage, and keeps reasons distinct and tied to the prompt.
- The synthesized refinement includes empathy, accountability, ethics, context, creativity, adaptation, social connection, economic stability, cultural understanding, and meaning.