PaperlayerView exact revision on arXiv

Source-linked AI summary

When Can LLMs Learn to Reason with Weak Supervision?

Salman Rahman, Jingyan Shen, Anna Mordvina, Hamid Palangi, Saadia Gabriel, Pavel Izmailov

arXiv:2604.18574v1cs.LGcs.AI

TL;DR

The paper addresses when RLVR can generalize despite scarce, noisy, or proxy supervision, a question made important by limited high-quality reward signals. It systematically studies model families and reasoning domains, finding that reward saturation and reasoning faithfulness predict success, while reasoning-trace SFT enables generalization and continual pre-training strengthens it.

  • Problem

    The paper asks when RLVR generalizes under weak supervision and what determines success or failure as reliable ground-truth reward signals become limited.

  • Method

    The authors conduct a systematic study across model families and reasoning domains, then disentangle continual pre-training from SFT on explicit reasoning traces through controlled pre-RL interventions.

  • Results

    Generalization follows reward saturation dynamics and reasoning faithfulness; explicit-reasoning SFT is necessary, while continual pre-training amplifies its effect across weak-supervision settings.

  • Takeaways & Limitations

    Training reward saturation can diagnose when further RL is unlikely to help, motivating pre-RL interventions that strengthen reasoning priors before weak-supervision RL.

  • Takeaways & Limitations

    The model-family comparison is treated as a proxy for pretraining-prior strength, and the intervention study focuses on MATH because Llama’s baseline failure is sharpest there.

Abstract

from arXiv · show

Large language models have achieved significant reasoning improvements through reinforcement learning with verifiable rewards (RLVR). Yet as model capabilities grow, constructing high-quality reward signals becomes increasingly difficult, making it essential to understand when RLVR can succeed under weaker forms of supervision. We conduct a systematic empirical study across diverse model families and reasoning domains under three weak supervision settings: scarce data, noisy rewards, and self-supervised proxy rewards. We find that generalization is governed by training reward saturation dynamics: models that generalize exhibit a prolonged pre-saturation phase during which training reward and downstream performance climb together, while models that saturate rapidly memorize rather than learn. We identify reasoning faithfulness, defined as the extent to which intermediate steps logically support the final answer, as the pre-RL property that predicts which regime a model falls into, while output diversity alone is uninformative. Motivated by these findings, we disentangle the contributions of continual pre-training and supervised fine-tuning, finding that SFT on explicit reasoning traces is necessary for generalization under weak supervision, while continual pre-training on domain data amplifies the effect. Applied together to Llama3.2-3B-Base, these interventions enable generalization across all three settings where the base model previously failed.

1. Introduction

The paper asks when RLVR can generalize under scarce data, noisy rewards, and self-supervised proxy rewards, and studies this across model families and reasoning domains. It finds that reward saturation dynamics, pretraining priors, reasoning faithfulness, and pre-RL interventions distinguish learning from memorization.

  • Research questions: The study evaluates RLVR generalization across Qwen and Llama models in MATH, SCIENCE, and GRAPH under three weak-supervision settings.The settings are scarce data, noisy rewards, and self-supervised proxy rewards.
  • Findings: Generalizing models sustain a prolonged pre-saturation phase, whereas rapidly saturating models enter diminishing-return regimes and memorize rather than learn.The paper links the pre-saturation phase to learning transferable reasoning patterns.
  • Findings: Domain-aligned pretraining priors help Qwen models generalize under all three weak-supervision settings, while Llama models and Qwen on GRAPH generally saturate rapidly.The paper interprets this model-family contrast as a proxy for pretraining-prior strength rather than an intrinsic family property.
  • Findings: Reasoning faithfulness distinguishes generalizing models from memorizing models, while output diversity alone does not.Faithfulness means that intermediate reasoning steps logically support the final answer.
  • Interventions: SFT on explicit reasoning traces is necessary for weak-supervision generalization, and continual pre-training amplifies its effect.Non-Thinking SFT on the same prompts fails, whereas the combination recovers generalization where Llama previously failed.

2. Experimental Setup

The experiments compare model families, domains, data-construction procedures, GRPO training, and benchmark evaluation across weak-supervision settings. Figure 1 tracks reward and benchmark performance over RL steps at small and maximum training-set sizes.

  • Models: The study evaluates Qwen base and math-specialized models alongside Llama instruction-tuned models.Qwen2.5-Math models receive an additional 1 trillion math-related pretraining tokens, while Llama models are aligned through instruction-tuning procedures.
  • Domains and datasets: The domains span MATH, SCIENCE, and GRAPH, representing high, moderate, and underrepresented levels of pretraining exposure.The datasets include Skywork-OR1, SCP, and Reasoning Gym tasks, respectively.
  • Data construction: Model-aware filtering retains problems with solve@16 ∈[1, 15], excluding instances that are trivial or intractable for the model.The retained problems form the candidate pool for all weak-supervision settings.
  • Training: GRPO samples response groups from the pre-update policy, uses binary ground-truth verification rewards, and applies KL regularization to a fixed reference policy.The advantage is computed by normalizing group-level rewards, and the experiments use the verl framework.
  • Evaluation: Figure 1 plots training reward, in-domain performance, and OOD performance over RL steps for 8-sample and Nmax training sets.Solid lines represent 8 samples, dashed lines represent Nmax, and colored vertical dashed lines mark saturation step tN_sat.
  • Evaluation: Qwen models exhibit extended pre-saturation phases and generalize from 8 samples, while Llama models saturate rapidly with limited gains.Reasoning performance is evaluated primarily with avg@16 accuracy across domain-specific benchmark suites.

3. RLVR Under Weak Supervision

Under weak supervision, RLVR generalizes when training remains in an extended pre-saturation phase; rapid saturation instead reflects memorization and limits transfer. Robustness also varies by model and domain: Qwen is generally more resilient than Llama to scarce data, noisy rewards, and proxy-reward failures.

  • 3.1. Scarce Data: With only 8 samples, all model-domain pairs show measurable pre-saturation learning, while post-saturation gains diminish.Positive Δ(8)sat appears across in-domain and out-of-domain metrics; Gsat,out is significantly positive for 7 of 8 pairs.
  • 3.1. Scarce Data: Qwen2.5-Math-1.5B trained on MATH sustains reward growth for over 300 steps and transfers gains to held-out MATH-500 and SCP-Hard.Domain-specific pretraining slows saturation and supports transfer beyond the training distribution.
  • 3.1. Scarce Data: Llama models saturate within the first 100 steps on 8 samples and generally require N ≥512 for meaningful generalization.Qwen on GRAPH shows similar data-scale dependence, indicating that domain alignment and pretraining priors influence scarce-data learning.
  • 3.2. Noisy Rewards: Qwen on MATH and SCIENCE maintains gains under up to γ = 0.7 corruption, whereas Qwen on GRAPH and Llama on MATH and SCIENCE degrade at γ ≥0.5.For Llama on MATH, nearly identical training rewards across corruption levels indicate fitting to incorrect answers rather than robust learning.
  • 3.3. Self-Supervised Proxy Rewards: Proxy rewards fail under prolonged training except for majority voting with math-specialized Qwen models.Majority voting can produce temporary gains before collapse, while self-certainty causes performance collapse across settings; policies may converge toward outputs maximizing the proxy.
  • 3.4. Why Do Models Fail Under Weak Supervision?: Higher raw diversity does not prevent rapid saturation: Llama retains higher diversity than Qwen but has lower reasoning faithfulness and poorer generalization.Faithful diversity, rather than raw diversity, tracks the model exploring reliable reasoning paths.

4. Improving RLVR Under Weak Supervision via Pre-RL Training

The study tests whether continual pre-training and reasoning-trace SFT can recover RLVR generalization under weak supervision. Thinking SFT is necessary, while CPT amplifies its effect by improving faithfulness and prolonging pre-saturation.

  • Setup: The controlled study crosses Base and CPT initializations with Thinking or Non-Thinking SFT, using Instruct as a reference across three weak supervision settings.The settings are scarce data, noisy rewards, and majority-vote proxy rewards, with MATH as the primary domain.
  • Results: Thinking SFT enables substantial downstream gains for Base and CPT, whereas Instruct remains flat or declines and Non-Thinking SFT rarely helps.Thinking SFT helps on scarce data and majority vote for both initializations; Non-Thinking SFT shows modest gains only with CPT under noisy rewards.
  • Results: CPT + Thinking SFT is the top-performing configuration across all three weak supervision settings and evaluated metrics.The combination produces substantially larger gains than Thinking SFT on Base alone.
  • Results: Matched CPT compute without reasoning traces fails to generalize on scarce data and majority vote, showing that extra pre-training compute alone is insufficient.The amplification depends on combining domain-aligned pre-training with explicit reasoning-trace supervision.
  • Results: Base initialization remains inadequate under noisy rewards, so Thinking SFT alone is necessary but not sufficient for generalization across all three settings.Domain-aligned pre-training is required to extend the intervention’s success beyond scarce data and majority vote.
  • Mechanism: Thinking SFT raises reasoning faithfulness, and CPT + Thinking SFT achieves the highest faithfulness alongside longer pre-saturation dynamics and strongest generalization.These findings support the proposed link between pre-RL faithfulness, training dynamics, and downstream generalization.

5. Related Work

The paper positions its study within work on RLVR, pre-training and fine-tuning, and diversity or faithfulness in reasoning. It extends prior work by characterizing weak-supervision generalization and proposing a pre-RL intervention.

  • RLVR for Reasoning: Prior RLVR studies report gains from sparse, noisy, or alternative rewards, but results vary across model families and can involve reward hacking or collapse.These mixed findings motivate the paper’s question about when weak supervision supports generalization.
  • Contribution: The paper contributes a cross-family, cross-domain account centered on saturation dynamics and faithfulness, plus an intervention that restores generalization after weak-supervision failure.The intervention is based on explicit reasoning SFT, with continual pre-training used to amplify its effect.
  • Pre-Training and Fine-Tuning: Existing work links pre-training and fine-tuning to RL generalization, whereas this paper focuses on continual-pretraining priors and reasoning SFT.The comparison targets how these pre-RL properties affect weak-supervision outcomes.
  • Diversity and Faithfulness: Output diversity has been proposed to support exploration, but prior work had not established which forms of diversity benefit generalization.The paper examines reasoning faithfulness as a complementary property.

6. Conclusion

The paper concludes that weak-supervision RLVR succeeds according to pre-RL properties, especially pretraining priors and reasoning faithfulness, rather than RL dynamics alone. It recommends monitoring saturation and improving priors before extending RL training.

  • Conclusion: Across model families and reasoning domains, weak-supervision success depends on pre-RL properties, pretraining priors, and reasoning faithfulness.Rapidly saturating models can produce correct answers without reasoning that supports them, indicating memorization rather than learning.
  • Practical Implications: Training reward saturation is a practical diagnostic: plateaued reward with flat downstream performance suggests further RL compute is unlikely to help.The paper interprets this pattern as the model exhausting what RL can extract from its priors.
  • Practical Implications: When weak supervision fails, the paper recommends allocating compute to pre-RL interventions that install strong priors rather than simply extending RL.This frames RL as the final stage of a pipeline whose success is largely determined before RL begins.

A. Limitations and Future Work

The study is limited to selected model families and scales, and its faithfulness and diversity analyses rely mainly on small-scale LLM-based judging. Broader validation and scalable metrics remain open directions.

  • Scope: The experiments cover specific model families and scales, so validation on larger architectures and broader task suites remains necessary.This limits how broadly the reported findings can currently be generalized.
  • Measurement: Reasoning faithfulness and diversity are evaluated with an LLM-as-a-judge framework, supplemented only by small-scale human verification.The paper identifies scalable metrics for both properties as an important future direction.

B.1. Training and Evaluation Datasets

The study evaluates weakly supervised RL across Qwen and Llama models in MATH, SCIENCE, and GRAPH, using filtered training subsets and multiple in-domain and out-of-domain benchmarks. It also specifies prompt templates, optimization settings, and continual-pretraining and SFT procedures for Llama3.2-3B.

  • Training and Evaluation Domains: Experiments span Qwen and Llama model families across MATH, SCIENCE, and GRAPH reasoning domains.MATH training uses Skywork-OR1, while SCIENCE uses SCP; evaluation covers domain-specific benchmark suites.
  • Training and Evaluation Domains: Evaluation uses multiple benchmarks, including MATH500, AMC, AIME 2024, AIME 2025, Minerva Math, OlympiadBench, GPQA-Diamond, Science Bench, MMLU-Science, and SuperGPQA.GRAPH evaluation includes held-out Quantum Lock and Largest Island tasks.
  • Dataset Construction: Training problems are filtered by solve@16, retaining intermediate-difficulty examples with solve@16 ∈[1, 15] and excluding solve@16 = 0 or 16.The procedure samples 16 base-model responses per problem before filtering.
  • Dataset Construction: Stratified round-robin sampling constructs training subsets of N ∈ {8, 32, 64, 512, 2048} while representing difficulty levels as uniformly as possible.Filtered problems are divided into 15 solve@16 bins and sampled without replacement.
  • Training Configuration: RL uses verl defaults including learning rate 10^-6, KL coefficient β = 0.001, clip ratio ϵ = 0.2, group size G = 8, and temperature 1.0.Training and mini-batches contain 64 prompts, and responses are capped at 2048 tokens unless otherwise noted.
  • Training Configuration: Continual pre-training of Llama3.2-3B uses approximately 52B tokens of math-relevant Nemotron-CC-Math data, followed by three-epoch SFT initialized from the CPT checkpoint.The subsequent RL phase evaluates training sample sizes N ∈ {8, 2048}.

C.1. Additional Experimental Results from Small to Large Data Scale

Across small-to-large data scales, generalization tracks reward saturation dynamics, with model family and domain specialization affecting how long training remains in the productive pre-saturation phase. During this phase, improvements extend across in-domain and out-of-domain metrics.

  • MATH: Qwen2.5-Math-1.5B sustains a pre-saturation phase for 330 gradient steps on 8 samples, driving continuous in-domain improvement.Llama models instead saturate rapidly in small-sample MATH regimes and depend more heavily on data scale.
  • MATH: Qwen models maintain comparable performance across sample sizes, whereas Llama models show stronger data dependence in small-sample MATH training.The contrast is associated with extended saturation periods for Qwen models and rapid saturation for Llama models.
  • SCIENCE: In SCIENCE, pre-saturation produces similar gains across sample sizes, while larger datasets provide distinct benefits after saturation.Models show substantially different saturation dynamics when trained on small samples.
  • GRAPH: In GRAPH, Qwen2.5-Math-7B saturates faster than in other domains, consistent with accelerated saturation when domain-specific pre-training is absent.The comparison uses Qwen2.5-Math-7B and Llama3.1-8B-Instruct.
  • Cross-domain Evaluation: During pre-saturation, models improve on both in-domain and out-of-domain benchmarks across pass@k metrics, with Δ(8)_sat retaining the same sign for most model-benchmark pairs and k ∈ {1, 4, 8, 16}.This pattern indicates improvement is not limited to merely narrowing the pass@k and pass@1 gap.

C.3. Additional Experimental Results on Large Models

Experiments on 7B and 8B models reproduce the smaller-model pattern: pre-saturation generalization appears across in-domain and out-of-domain benchmarks, while Llama models still saturate faster and remain data-dependent. The 8B Llama model nevertheless shows better cross-domain generalization than its 3B counterpart.

  • Experimental Scope: Large-model experiments evaluate Qwen2.5-Math-7B and Llama3.1-8B-Instruct across MATH, SCIENCE, and GRAPH.The reported results include in-domain, out-of-domain, and additional GRAPH benchmarks.
  • Generalization Dynamics: During pre-saturation, large models generalize on both in-domain and out-of-domain benchmarks under pass@k metrics.This reproduces the pattern observed for smaller models.
  • Generalization Dynamics: The 8B Llama model exhibits better cross-domain generalization than the 3B model, but Llama models still saturate faster than Qwen models and show clear data dependence.The data dependence is exemplified by SCIENCE results.

D.1. Additional Results on Reward Corruption

Reward-corruption and proxy-reward experiments test whether weak signals can support generalization across models and domains. Qwen models are more robust to corrupted rewards, while proxy rewards can induce collapse or reward hacking despite rising training reward.

  • Reward Corruption: Qwen models generalize under almost completely corrupted rewards, whereas Llama models can achieve high reward curves yet generalize poorly to new data.The Llama pattern suggests overfitting to incorrect responses.
  • Proxy Rewards: Majority voting and self-certainty are evaluated as self-supervised alternatives to ground-truth verification.Majority voting uses consensus among sampled responses, while self-certainty uses the model’s confidence as reward.
  • Proxy Rewards: Except for Qwen2.5-Math-1.5B, models exhibit failure modes under prolonged proxy-reward training across model-domain pairs.Qwen2.5-1.5B on SCIENCE collapses under both proxy rewards, while Llama-3.2-3B-Instruct on MATH degrades despite increasing training rewards.
  • Proxy Rewards: Qwen2.5-3B trained with majority-vote rewards produces identical final answers across SCIENCE rollouts, receiving reward 1.0 despite incorrect answers.The behavior constitutes reward hacking because consensus is maximized independently of problem content.
  • Reward Baselines: GRPO-POS and GRPO-NEG achieve comparable Pass@1 performance to standard GRPO in SCIENCE under both 8- and 1024-sample regimes.They exhibit similar saturation and generalization behavior in these experiments.
  • Faithfulness Analysis: Faithfulness measures whether a reasoning trace coherently and logically supports the produced answer, with aligned, partially aligned, and misaligned labels.LLM judges show substantial agreement, with κ= 0.752 and 0.649, and manual checks find fair alignment.

F.3. Additional results on diversity analysis

Diversity and faithfulness analyses distinguish models that learn reliable reasoning strategies from those that produce more misaligned responses under scarce-data RL. Qwen models show stronger diversity among correct responses and higher alignment than Llama in the reported settings.

  • Qwen-Math exhibits higher reasoning diversity on correct MATH-500 responses during later RL training than the compared models.This higher diversity coincides with better evaluation performance and is interpreted as evidence of diverse, reliable strategies.
  • Qwen2.5-1.5B and Qwen2.5-Math-1.5B produce more aligned and fewer misaligned responses than Llama3.2-3B after training on eight MATH samples.Qwen2.5-Math-1.5B shows the same pattern with eight SCIENCE samples.
  • Qwen2.5-Math-1.5B also exceeds Llama3.2-3B in alignment proportions for both correct, incorrect, and overall responses in the eight-sample SCIENCE setting.

G. Pre-RL Intervention

The section presents evaluation figures and analysis tools for comparing RL training dynamics, reward robustness, response diversity, reasoning faithfulness, and pre-RL interventions across models and domains.

  • Training dynamics and reward settings: Figures 13–17 compare RL training dynamics and performance across models, domains, data scales, and reward settings.The comparisons cover MATH, SCIENCE, and GRAPH, including scarce-data regimes, reward corruption, proxy rewards, and baseline variants.
  • Training dynamics and reward settings: Qwen models maintain performance under substantial reward-label corruption, whereas Llama models and GRAPH degrade when γ ≥0.5.The corruption fraction γ ranges from clean labels at γ = 0 to fully incorrect labels at γ = 1.
  • Training dynamics and reward settings: Proxy rewards without verifiers can fail during prolonged training through collapse or reward spikes followed by performance drops.Self-certainty exhibits training collapse, while majority vote shows reward spikes followed by performance declines.
  • Training dynamics and reward settings: Positive-only and negative-only GRPO updates produce performance comparable to standard GRPO on SCIENCE with both 8 and 1024 training samples.The comparison is reported for the baseline variants in Figures 28 and 29.
  • Behavioral analyses: The section evaluates response diversity and reasoning faithfulness using prompts, judge definitions, alignment analyses, and qualitative examples.The materials include diversity analysis on MATH-500, an LLM-as-a-judge faithfulness prompt, aligned-versus-misaligned response proportions, and qualitative examples.
  • Pre-RL interventions: Pre-RL intervention results are reported through pass@16, AIME 2024, and Science Bench evaluations, with Figure 34 covering pass@16 and Figure 35 covering additional benchmarks.The SFT format distinguishes Thinking SFT, which includes reasoning traces enclosed by <think> and </think>, from Non-Thinking SFT, which omits them.
Loading 2604.18574v1…