PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance

Kai Yan, Alexander G. Schwing, Yu-Xiong Wang

arXiv:2605.15012v1cs.LGcs.AIcs.CL

TL;DR

RLVR has limited sample efficiency on difficult tasks, and demonstration-guided methods can require costly SFT data. FEST addresses this with few-shot demonstration-guided RLVR using 128 randomly selected examples and three components: supervised, on-policy, and decaying-weight signals. The paper reports consistent gains over strong baselines and identifies scaling beyond 1.5B-parameter math models as future work.

  • Problem

    RLVR struggles with sample efficiency on complex tasks, while demonstration-guided RL methods require large and expensive SFT datasets.

  • Method

    FEST combines supervised guidance, on-policy learning, and decaying weights through a semi-online DPO-based few-shot demonstration-guided RLVR framework.

  • Results

    FEST consistently outperforms various strong baselines across multiple benchmarks while using as few as 128 randomly selected SFT examples.

  • Takeaways & Limitations

    Few-shot demonstration-guided RLVR can boost RLVR performance with minimal SFT data.

  • Takeaways & Limitations

    Evaluation was mostly limited to a 1.5B-parameter model focused on math reasoning, leaving scaling and broader task generalization for future work.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has achieved great success in developing Large Language Models (LLMs) with chain-of-thought rollouts for many tasks such as math and coding. Nevertheless, RLVR struggles with sample efficiency on difficult problems where correct rollouts are hard to generate. Prior works propose to address this issue via demonstration-guided RLVR, i.e., to conduct Supervised FineTuning (SFT) when RL fails; however, SFT often requires a lot of data, which can be expensive to acquire. In this paper, we propose FEST, a FEw-ShoT demonstration-guided RLVR algorithm. It attains compelling results with only 128 demonstrations randomly selected from an SFT dataset. We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset to prevent overfitting from multiple-epoch training. On several benchmarks, FEST outperforms baselines with magnitudes less SFT data, even matching their performance with full dataset.

1 Introduction

RLVR can struggle on complex tasks when sampled batches contain no correct answers, while demonstration-guided approaches require costly SFT data. FEST addresses this gap with few-shot guidance using 128 randomly selected examples and three integrated training signals.

  • Motivation: RLVR has low sample efficiency on complex tasks because batches without correct answers provide no learning signal.Repeated sampling can triple computational overhead on average.
  • Motivation: Demonstration-guided RL integrates SFT when RL sampling fails, but these methods demand large and expensive datasets.Curating 2,500 questions for Humanity’s Last Exam required 1,000 graduate-degree holders, even with brief rationales.
  • FEST: FEST jointly uses GRPO on answer-only data and semi-online DPO on few-shot SFT data.FEST-GRPO decomposes DPO and replaces its online component with GRPO using negative advantages to address gradient mismatch.
  • FEST: FEST is a few-shot demonstration-guided RLVR algorithm designed to use as few as 128 randomly selected SFT examples.The method targets performance gains from a limited, uncurated dataset.
  • FEST: FEST combines supervised learning, on-policy learning, and decaying weights to provide expert guidance, improve robustness, and prevent overfitting.Its semi-online DPO loss treats demonstrations as positive examples and agent rollouts as negative ones.
  • Results: Across multiple benchmarks, FEST consistently outperforms various strong baselines while using minimal SFT data.The paper presents few-shot demonstration-guided RLVR as a post-training paradigm that significantly boosts RLVR performance.

2 Preliminaries

This section defines RLVR, GRPO, DPO, and REINFORCE as foundational components for objective-reward language-model training. RLVR maximizes expected verifiable reward, while GRPO and REINFORCE provide policy-gradient mechanisms and DPO directly optimizes from preference data.

  • RLVR: RLVR trains language models on tasks with objective ground truths by generating chain-of-thought responses and assigning verifiable rewards.Rewards can come from unit tests, symbolic checkers, or exact string matching.
  • RLVR: RLVR optimizes the policy to maximize expected reward over responses sampled from the policy.A response probability is represented as the product of autoregressive token probabilities.
  • GRPO: GRPO samples a group of n rollouts with corresponding rewards and uses group-relative advantages for critic-free optimization.The advantage is calculated relative to the group mean reward; the method omits the KL regularizer and standard-deviation normalization here.
  • GRPO: GRPO applies clipping independently to tokens using the reference policy from the current sampling-training iteration.The formulation uses ϵ2 > ϵ1 > 0 and an upper response-length limit M.
  • DPO: DPO directly optimizes a policy from preference triples without requiring an explicit reward model.The preferred and non-preferred responses are denoted y+ and y−, with β as a positive hyperparameter.
  • REINFORCE: REINFORCE maximizes expected reward using a policy-gradient loss proportional to reward times the gradient of log policy probability.Its empirical loss gradient is −r(x, y) · ∇log π(y|x).

3 Methodology

FEST combines few-shot supervised demonstrations with on-policy RL signals and adaptive weighting to train RLVR when expert data are limited. FEST-GRPO further addresses the gradient mismatch between sequence-level DPO and token-level GRPO objectives.

  • 3.1 Challenges and design requirements: Few-shot demonstration-guided RLVR uses expert-curated reasoning traces in DE alongside a large-scale answer-only RL dataset DI.The objective is to exploit minimal reasoning traces to improve beyond standard RLVR on DI alone.
  • 3.1 Challenges and design requirements: Limited expert access prevents on-demand demonstrations, while few-shot data provide insufficient coverage of required reasoning paradigms.These constraints remove flexibility assumed by several prior demonstration-guided methods.
  • 3.1 Challenges and design requirements: The training objective requires supervised learning, on-policy learning, and a decaying weight when training on DE.Supervision supplies external knowledge, on-policy learning compares model rollouts with SFT traces, and decay controls repeated training on few-shot data.
  • 3.2 FEST: FEST combines a semi-online DPO loss on DE with a GRPO loss on DI through L = c · LE + LI.SFT demonstrations are preferred rollouts, while RL-generated responses are non-preferred rollouts; c > 0 controls the DE contribution.
  • 3.2 FEST: Semi-online DPO’s gradient aligns supervised learning, on-policy training, and decaying weights, and can be interpreted as adversarial training.The policy acts as a generator while a discriminator distinguishes preferred and non-preferred responses.
  • 3.3 FEST-GRPO: FEST-GRPO replaces the REINFORCE component implicit in semi-online DPO with GRPO to mitigate sequence-level versus token-level gradient mismatch.The variant retains LI and applies weighted SFT plus GRPO to DE, reducing reliance on exhaustive tuning of c.

4 Experiments

FEST is evaluated with 128 randomly sampled demonstrations against RLVR, multi-objective, and RL-SFT baselines across mathematical reasoning benchmarks. It remains effective across shot counts and demonstration datasets, with ablations, exploration metrics, and OOD evaluation supporting its robustness.

  • 4.1 Main Results: FEST variants outperform established baselines under the 128-shot configuration, yielding significant gains over vanilla RL in the sparse-data regime.SRFT and MIFO use the full dataset, while other methods share the 128-shot constraint.
  • 4.1 Main Results: FEST is the only method that consistently surpasses both pure RL and RL-G while maintaining higher Pass@8 and exploration potential.RL-G can achieve high nominal accuracy but suffers from overfitting and lower reasoning diversity.
  • 4.2 Scaling with Shot Counts: With as few as 64 shots, FEST remains effective; FEST-GRPO is more stable in extreme low-data settings, whereas FEST-DPO scales better with additional data.FEST-DPO eventually reaches performance comparable to HPT trained on the full 46K SFT dataset.
  • 4.3 Consistency of Performance Gain Across Different DE: FEST consistently improves performance across alternative few-shot datasets, including additional OpenR1 splits and the 257-example LIMOv2-8192 subset.The LIMOv2 subset contains Chain-of-Thought traces under 8,192 tokens.
  • 4.4.1 Components: Peak performance requires the synergy of supervised learning, on-policy learning, and adaptive weight scheduling.The ablation study evaluates the three components identified in the method section.
  • 4.4.2 Out-of-Distribution Dataset: FEST variants achieve the highest Pass@1 results on the zero-shot OOD MMLU-Pro benchmark, indicating robust cross-domain generalization.MMLU-Pro contains over 12,000 problems.

5 Related Work

Related work combines SFT with RL to improve exploration and mitigate exposure bias, but few-shot post-training remains constrained by data quality and coverage. FEST is positioned alongside reward-based, ratio-based, and online preference-learning approaches.

  • Demonstration-Guided RLVR: Demonstration-guided RL integrates SFT with RL when sampling fails to produce positive rollouts, extending post-training beyond decoupled training stages.These methods aim to provide expert guidance beyond the base model’s inherent capabilities.
  • Demonstration-Guided RLVR: Most demonstration-guided methods apply SFT to problems where the model receives no positive reward.Other strategies use fixed data ratios, gradient-norm balancing, or token-probability weighting.
  • Few-Shot LLM Post-Training: Few-shot SFT and DPO methods commonly select data through heuristic curation or automated data-selection procedures.The cited heuristics include chain-of-thought length, diversity, and difficulty.
  • (Semi-)Online DPO and Self-Play Preference Learning: Iterative and online DPO update preference data with the agent’s own rollouts to support self-generated learning and mitigate exposure bias.FEST adopts a semi-online variant in which only non-preferred responses are generated on the fly.

6 Conclusion

The paper presents FEST as a few-shot demonstration-guided RLVR framework built from supervised, on-policy, and decaying-weight components. Its evaluation is mainly limited to a 1.5B-parameter model and mathematical reasoning, while broader societal benefits and risks are discussed.

  • 6 Conclusion: FEST combines supervised learning from SFT data, on-policy learning with negative advantages, and decaying weights to improve RLVR with 128 random demonstrations.FEST-GRPO decomposes the DPO objective to bridge token-level GRPO and sequence-level DPO.
  • 6 Conclusion: The framework is presented as a scalable response to the scarcity of high-quality SFT data.The paper frames few-shot post-training as resource-efficient and potentially useful where extensive expert data are unavailable.
  • Limitations and Future Work: The evaluation was mostly conducted with a 1.5B-parameter model focused on mathematical reasoning.Scaling to larger architectures and broader domains such as code generation and general instruction following remains future work.
  • Potential Risks and Mitigations: The paper identifies potential risks from dual-use reasoning improvements, labor displacement, reward hacking, and unintended verifier-related biases.It calls for safety alignment, policy frameworks, and transparent reporting as mitigations.

B.2 Semi-Online DPO and Adversarial Training

This section formulates semi-online DPO as adversarial training between a policy generator and a reward-function discriminator. Nonlinear activation, KL regularization, and Jensen’s inequality yield a tractable objective corresponding to the semi-online DPO loss.

  • Adversarial Training: The adversarial formulation compares the policy π with an expert behavioral policy πE through a discriminator reward function f.The discriminator maximizes their discrepancy in an IPM-inspired objective.
  • Adversarial Training: A nonlinear log-sigmoid activation bounds the otherwise unbounded discriminator objective and produces a logistic-loss formulation.The activation is g(x) = −log(1 + e^-x).
  • Policy Optimization: KL regularization anchored at πref makes the generator’s inner maximization tractable.The resulting optimal policy has a Gibbs-distribution form.
  • Policy Optimization: The discriminator can be expressed through the policy-to-reference probability ratio plus an input-dependent term.Substituting this solution back into the objective enables the subsequent derivation.
  • Semi-Online DPO: Jensen’s inequality and log-sigmoid concavity produce a tractable upper bound that corresponds precisely to the semi-online DPO loss.The derivation establishes a numerical connection rather than only a conceptual analogy.

B.3 Unifying Post-Training Framework

The paper extends HPT’s unified policy-gradient taxonomy by incorporating DPO. The extension uses stability masks and sampling-policy choices to represent algorithms such as SFT, GRPO, and FEST-GRPO within one framework.

  • Unifying Post-Training Framework: HPT represents diverse post-training algorithms through a generalized policy-gradient estimator for policy πθ.Its notation includes a stability mask and a distribution used for importance sampling.
  • Unifying Post-Training Framework: The extended framework adds DPO, which was omitted from the original HPT formulation.The extension is motivated by the paper’s connection between DPO and REINFORCE.
  • Unifying Post-Training Framework: The framework distinguishes πθold as the pre-update policy and identifies clipping masks for GRPO, PPO, CIS, and sequence-level clipping.For FEST-GRPO, the table specifies a gradient involving an advantage term and token-level clipping.

C Implementation Details

FEST uses a unified implementation that combines few-shot SFT and answer-only RL data, with experiments conducted in a common codebase and verifier setup. Baseline implementations were adapted to the available code, checkpoints, and training constraints.

  • FEST implementation: FEST samples few-shot SFT data uniformly from OpenR1-Math-46K-8192 and trains with answer-only RL data.The algorithm takes DE and DI as separate data sources and uses rollouts from both during minibatch updates.
  • FEST implementation: Each training step computes verifier rewards and advantages for RL data, then combines GRPO loss with the few-shot learning loss.The implementation updates the policy using L = c · LE + LI.
  • Experimental setup: Experiments use a unified ReLIFT-derived codebase, verl 0.4.0 for reinforcement learning, and vllm 0.8.4 for rollout generation.The study also reports using a modified reward verifier because of occasional distributed-training deadlocks with math-verify.
  • Baselines: HPT, ReLIFT, LUFFY, CHORD, SRFT, and MIFO were implemented or evaluated with method-specific adaptations to learning rates, code availability, or checkpoints.CHORD-ϕ was implemented because the original codebase was not directly usable, while SRFT was evaluated from a public checkpoint.

D.1 Supplementary Results for Methodology

Supplementary analyses examine why FEST’s objective design is useful for few-shot RLVR. They identify diminishing SFT exposure, severe DPO–GRPO gradient-scale mismatch, and instability from naively applying RL to the few-shot gold set.

  • HPT Data Ratio Dynamics: SFT trajectories fall to approximately 2% of rollouts late in HPT replication while test performance continues improving.This supports decaying the SFT objective’s weight during later training.
  • Gradient Mismatch Analysis: DPO gradients typically range from 10^1 to 10^2, whereas GRPO gradients remain below 0.1 because their objectives operate at sequence and token levels.FEST-GRPO unifies the objectives at the token level, removing the need for exhaustive search over coefficient c.
  • Few-shot RL instability: HPT-G and ReLIFT-G show acute performance degradation during training on both the few-shot gold dataset and the RL dataset.The authors hypothesize that abrupt distribution shifts arise when SFT updates conflict with an RL-optimized policy on limited data.

D.3 Analysis on β

The β analysis shows that extended reasoning traces and few-shot training alter DPO’s learning-signal behavior. Smaller β can yield stronger, steadier signals, but performance degrades when hyperparameters move too far from a balance point.

  • Motivation: Reasoning traces of up to 8,192 tokens produce much larger cumulative log-probabilities than typical RLHF sequences, complicating β selection.The few-shot regime also creates larger log-ratio differences between preferred and rejected responses.
  • Gradient analysis: The implicit advantage z determines the DPO gradient coefficient w = β · σ(−z), whose strength can increase as β decreases when z is large.For fixed ∆≫1, reducing β to 0.1β scales the coefficient by approximately 0.1e^0.9z.
  • Signal behavior: Higher β produces switch-like signals that can rapidly collapse, whereas smaller β produces stronger and more constant learning signals.The corresponding z distribution is wider for higher β and more concentrated for lower β.
  • Sensitivity study: FEST-DPO works across β values from 0.001 to 0.1 but loses performance when all hyperparameters are too large or too small.The selected configuration targets a balance point rather than an extreme β.

D.4 Training Curves

Training curves show that FEST maintains advantages across datasets while several baselines converge quickly but overfit or become unstable. The supplementary experiments also document difficult few-shot settings and substantial compute requirements.

  • Primary training curves: FEST-DPO and FEST-GRPO maintain a substantial performance margin across DE, DI, and test-set metrics.LUFFY, CHORD-ϕ, and HPT converge rapidly but later stagnate on the test set, while ReLIFT provides too little expert guidance under few-shot constraints.
  • LIMOv2-8192: On LIMOv2-8192, DI and test-set performance grow continuously while DE accuracy remains relatively stationary.The authors attribute this pattern to LIMOv2’s higher reasoning difficulty and report gains from the few-shot dataset nonetheless.
  • SFT and SPIN: Standalone SFT and SPIN overfit rapidly on the 128-shot dataset and were terminated before 600 steps, so formal main-paper performance reports were omitted.Figure 11 compares their trajectories with FEST; SPIN peaks early before declining.
  • Compute: A standard 600-step training run requires approximately 4–5 days on a node with two NVIDIA GH200 superchips.Benchmark evaluation with eight rollouts per question takes approximately 30 minutes.
Loading 2605.15012v1…