PaperlayerView exact revision on arXiv

Source-linked AI summary

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

Xiao Liang, Zhong-Zhi Li, Yeyun Gong, Yang Wang, Hengyuan Zhang, Yelong Shen, Ying Nian Wu, Weizhu Chen

arXiv:2506.08989v1cs.LGcs.CL

TL;DR

RLVR needs scarce, high-quality problems with precise answers and suitable difficulty, while existing synthesis methods often ignore model capabilities. SwS identifies persistent failures, synthesizes and verifies targeted problems, and uses them for augmented RL training, achieving consistent gains across eight benchmarks, including 10.0% and 7.7% average improvements for 7B and 32B models.

  • Problem

    RLVR scalability is limited by scarce human-labeled mathematical problems, limited verification in synthetic datasets, and synthesis strategies that ignore model capabilities.

  • Method

    SwS identifies persistent failure cases during preliminary RL, extracts and recombines their concepts, and generates, verifies, and difficulty-filters targeted problems for augmented RL training.

  • Results

    SwS consistently improves performance across eight reasoning benchmarks, with average absolute gains of 10.0% for 7B models and 7.7% for 32B models.

  • Takeaways & Limitations

    Focusing synthetic data on self-identified weaknesses supports reasoning generalization and helps mitigate model weaknesses without external knowledge distillation.

  • Takeaways & Limitations

    The framework requires both a strong instruction model and an answer-labeling reasoning model, which may increase computation and time costs.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and limited-verification answers in existing distillation-oriented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model's capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a Self-aware Weakness-driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses as questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure cases and synthesize new problems to strengthen the model's weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization byempowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks.

1. Introduction

RLVR depends on curated, verifiable problems at suitable difficulty, but scarce human-labeled data and capability-agnostic synthesis limit efficiency. SwS addresses this by using self-identified weaknesses to generate targeted training problems and reports gains across model sizes and benchmarks.

  • RLVR scalability requires well-curated problems with appropriate difficulty and precise, verifiable answers.Human-labeled mathematical problems are scarce and expensive to collect, while synthetic datasets may have limited verification quality.
  • Problems that are fully mastered or consistently unsolved provide weak learning signals in group-level RL because token-level advantages can collapse to zero.Difficulty must therefore align with the model’s current capabilities.
  • SwS identifies model weaknesses during preliminary RL and synthesizes targeted problems to improve those weak areas in subsequent training.It groups failure cases by category, extracts common concepts, and allocates augmentation according to relative category performance.
  • The framework was evaluated across model sizes from 3B to 32B and extended to Weak-to-Strong Generalization, Self-evolving, and Weakness-driven Selection settings.These extensions were used to examine the framework’s generalizability and adaptability.
  • 10.0% and 7.7% average absolute improvements were achieved for 7B and 32B models across eight mathematical reasoning benchmarks.Augmented models surpassed base models and models trained on the original dataset across all benchmarks, and could solve up to 20.0% more previously failed weak-domain problems.

2. Method

SwS first identifies persistent failures during preliminary RL, then generates, verifies, filters, and allocates targeted synthetic problems before augmented RL training. Its pipeline combines weakness criteria, concept recombination, quality and answer verification, and difficulty filtering.

  • 2.3. Self-aware Weakness Identification: The pipeline identifies weaknesses as problems whose accuracy never reaches 50% and whose accuracy trend has a negative slope during preliminary RL.These failure cases are collected as seed problems for subsequent synthesis.
  • 2.4. Targeted Problem Synthesis: SwS extracts concepts from weak problems, recombines related concepts within categories, and allocates more synthesis budget to categories with higher failure rates.Co-occurrence probabilities and embedding similarities help preserve semantic coherence during recombination.
  • 2.4. Targeted Problem Synthesis: A strong instruction model generates non-multiple-choice, non-multi-part, non-proof problems from category labels and recombined concepts, while general instruction models verify quality.Only problems rated perfect above a threshold and receiving no bad ratings are retained.
  • 2.4. Targeted Problem Synthesis: A strong reasoning model generates multiple reference answers, and Math-Verify retains problems with at least 50% consistent answers.This filters unreliable or excessively complex problems before RLVR training.
  • 2.4. Targeted Problem Synthesis: Synthetic problems are difficulty-filtered by retaining those whose initial-model accuracy falls within a target range such as [25%, 75%].The goal is to provide learnable problems and avoid uniformly correct or incorrect responses that produce weak RL signals.
  • 2.4. Targeted Problem Synthesis: The verified synthetic problems are added to the initial set, and the initially trained model continues training on the augmented set in a second RLVR stage.This stage targets mitigation of the weaknesses identified during preliminary training.

3. Experiments

Experiments evaluate SwS across model scales, benchmarks, baselines, and extensions, showing consistent gains from capability-targeted synthetic problems. The framework also improves performance on previously failed problems despite requiring fewer repetitions of each original problem.

  • Experimental Setup: Experiments span 3B–32B models and eight mathematical reasoning benchmarks, with evaluation using Pass@1 and additional AIME Avg@32 accuracy.
  • Experimental Setup: The evaluation compares base, instruction-tuned, initially trained, augmented, and recent RL-based model baselines across multiple benchmarks.
  • Main Results: +10.0% and +7.7% absolute improvements are reported for SwS-enhanced 7B and 32B models, respectively, across benchmarks, with +16.7% on AIME24 and +13.3% on AIME25 for Qwen2.5-7B initialized with MATH-12k.Table 1 covers multiple base models and benchmarks; AIME reports both Avg@1 and Avg@32.
  • Main Results: SwS-enhanced models consistently outperform base models and models trained on the original dataset across the evaluated benchmarks, also surpassing counterparts trained on curated human-labeled problem sets.
  • Weakness Mitigation from Augmented Training: Augmented training increases the proportion of previously failed problems solved across most domains, with largest gains in Intermediate Algebra (20%), Geometry (5%), and Precalculus (5%).The comparison is against continued training on the initial dataset alone.
  • Weakness Mitigation from Augmented Training: These weakness-mitigation gains occur even though each original problem is sampled four times less frequently than under training on the original dataset alone.

4. Extensions and Analysis

The extensions test SwS with weaker teachers, self-evolving generation, weakness-driven retrieval, and difficulty-controlled synthesis. Across these settings, targeted or appropriately difficult problems generally provide stronger or more sustained learning signals than untargeted or easy alternatives.

  • 4.1. Weak-to-Strong Generalization for SwS: Weak-teacher filtering retains problems that improve the student despite the student generally outperforming its teacher.Difficulty screening removes consistent student–teacher disagreements and overly complex samples.
  • 4.2. Self-evolving Targeted Problem Synthesis: A self-evolving SwS pipeline improves baseline performance by 1.2% across benchmarks, although performance declines on AIME.The authors attribute the AIME decline to initial training data already being tailored to that benchmark.
  • 4.3. Weakness-driven Selection: Weakness-driven augmentation outperforms random Big-Math selection on both full benchmarks and competition-level benchmarks.Random problems are quickly fitted and stop providing meaningful GRPO signals, whereas weakness-aligned problems remain challenging.
  • 4.4. Impact of Question Difficulty: Difficulty is defined by initially trained model accuracy over eight rollouts: simple [5, 7], medium [3], and hard [1] [4].These ranges determine which synthetic problems enter augmented RL training.
  • 4.4. Impact of Question Difficulty: Harder synthetic problems provide the longest-lasting evaluation gains, while simple problems converge quickly and then saturate.Medium and hard sets converge more slowly on training data but sustain performance improvements on evaluation benchmarks.

5. Conclusion

SwS synthesizes problems from weaknesses identified in model failure cases during preliminary RL and uses them for augmented training. The framework improves reasoning generalization and is extended to weak-to-strong, self-evolving, and weakness-driven selection settings.

  • 5. Conclusion: SwS identifies failure-case weaknesses during preliminary RL, synthesizes targeted problems, and incorporates them into subsequent augmented training.The paper reports extensions to Weak-to-Strong Generalization, Self-evolving, and Weakness-driven Selection.

6. Discussions, Limitations and Future Work

The discussion positions SwS against RLVR data-construction and distillation approaches while identifying computational, methodological, and synthesis-complexity limitations. Future work broadens weakness identification, task coverage, feedback sources, and model scope.

  • Limitations: SwS requires both a strong instruction model and an answer-labeling reasoning model, increasing computation and time costs.This is identified as a direct limitation of the framework.
  • Limitations and Future Work: The framework primarily targets RL to activate inherent reasoning without external knowledge distillation, leaving fine-tuning and distillation extensions open.The authors explicitly identify these extensions as future research directions.
  • Limitations: Open-source instruction models may generate synthetic problems insufficiently complex to elicit deeper reasoning, especially in the 14B self-evolving setting.The paper notes that performance improvements there are limited.
  • Related Work: Prior RLVR data work emphasizes difficulty matching, diversity, curriculum learning, process rewards, exploration, and long-CoT adaptation.SwS is situated within broader efforts to improve RL training stability, efficiency, and data composition.
  • Related Work: Existing reasoning-data synthesis commonly generates problem–response pairs, augments responses, or recombines concepts using advanced models, often for distillation.The paper contrasts these approaches with weakness-driven synthesis based on the target model’s failures.

B.1. Training

The experiments use GRPO-based RL with fixed rollout and optimization settings, evaluate with rule-based verification, and validate initial-RL weakness identification against base and SFT alternatives. The initial RL model is selected because its failures expose critical weaknesses while also serving as the augmented-training starting point.

  • B.1. Training: All RL experiments use GRPO with eight rollouts per problem, batch size 1024, and a 5 × 10^-7 learning rate.Training uses 200 initial steps and 600 augmented-training steps without a KL penalty.
  • B.1. Training: The 32B and 14B datasets lack human category labels, so LLaMA-3.3-70B-Instruct supplies categories for same-category concept combination.This preserves compatibility with the SwS workflow.
  • B.1. Training: Evaluation computes Pass@1 with greedy decoding for baselines and sampling for RL models, plus Avg@32 from 32 sampled responses on competition benchmarks.A hybrid Math-Verify and PRIME-RL verifier is used.
  • B.1. Training: Initial RL is preferred for weakness identification because its failed problems expose critical weaknesses and the trained policy can seed augmented RL.The comparison includes base, SFT, and initial RL models on the original training set.

D.1. Detailed Data Workflow

The SwS workflow filters and organizes weakness-derived synthetic problems before augmented RL training, combining strict quality and difficulty controls with concept sampling based on co-occurrence and embedding similarity.

  • End-to-end workflow: The workflow evaluates generated problems through quality, answer-consistency, and model-difficulty filters before using the retained set for augmented RL training.The process begins from weakness-derived categories and preserves samples suited to the target model.
  • Data filtering: The pipeline removes multiple-choice, multipart, proof-required, low-quality, inconsistent-answer, and difficulty-mismatched problems.Answer inconsistency is identified when the most frequent generated answer appears in fewer than 50% of generations.
  • Data filtering: 78.35% of generated problems are removed by quality-based filtering, the strictest filtering stage in the SwS pipeline.The filtering is intended to maintain quality and stability for subsequent training.
  • Difficulty selection: Synthetic problems selected for augmentation have model accuracy between 25% and 75%, comprising nearly 35% of generated problems across four model sizes.This range targets problems that are neither already mastered nor entirely beyond the model’s capability.
  • Concept sampling: Concept sampling combines co-occurrence statistics with cosine similarity to construct coherent, semantically related concept sets for problem generation.Previously selected concepts are excluded, and candidate probabilities are obtained through temperature-controlled softmax sampling.

F. Details for Weak-to-Strong Generalization in SwS

The weak-to-strong extension uses a weaker teacher for answer labeling while filtering by teacher consistency and student agreement, substantially improving labeling reliability on retained problems.

  • Weak-to-strong setting: A weaker teacher outperforms the stronger student on 16.4% of MATH-500 problems when each question is sampled eight times.This provides the empirical basis for using the weak teacher’s strengths in answer generation.
  • Answer-labeling criteria: The answer-labeling upper bound requires a teacher-consistent answer above 50% frequency and student agreement on that answer in at least 25% of responses.These revised self-consistency conditions are designed to support correctness of synthetic labels.

G. Details for Self-Evolving in SwS

Self-evolving SwS improves performance on simpler mathematical benchmarks but remains limited on AIME-level tasks, partly because the model favors its own generations and produces too few appropriately difficult problems.

  • Performance boundary: Self-evolving SwS remains suboptimal on AIME-level competition benchmarks despite improving performance on simple-to-medium mathematical reasoning tasks.This defines the extension’s principal scope boundary.
  • Self-assessment limitation: The model assigns more “perfect” and “acceptable” labels and fewer “bad” labels to self-generated problems than to standard-framework problems.Such self-preference may overlook low-quality samples or misclassify overly complex problems as unsolvable or poor quality.
  • Difficulty limitation: Nearly 50% of self-generated problems receive all-correct responses, while only 31% remain after difficulty filtering for augmentation.The resulting simplicity may reduce data efficiency and limit learning on more complex problems.
  • Selection pipeline: The weakness-driven selection pipeline embeds failed problems and target questions, performs domain-level nearest-neighbor selection, and returns a budgeted union of selected subsets.The algorithm allocates selections by domain and chooses target questions with the smallest domain-specific distances.

H. Details for Weakness-driven Selection

Weakness-driven selection transfers failure information from one model and dataset to targeted problems in another dataset, while the supplied figures and case studies illustrate difficulty, labeling, and weak-to-strong behavior.

  • Weakness-driven selection: The selection extension uses 915 Qwen2.5-7B failures on MATH-12k to select additional Big-Math data for weakness-focused augmented RL training.LLaMA-3.1-8B-base embeddings represent both failure cases and target-dataset problems.
  • Selection procedure: The selection procedure embeds failures and target questions, groups failures by domain, applies domain-level nearest-neighbor matching, and unions the selected subsets.The algorithm selects the smallest-distance target questions within each domain under the available budget.
  • Difficulty analysis: Figure 10 presents synthetic-problem difficulty distributions for models ranging from 3B to 32B.The figure is intended to compare how generated-problem difficulty is distributed across model sizes.
  • Weak-to-strong case study: Figure 11 illustrates a case where a weak teacher gives the correct solution but a strong student incorrectly applies a binomial identity.The example highlights a weak-to-strong mismatch in mathematical reasoning.
  • Self-evolving evaluation: Figure 12 depicts quality assessment and difficulty evaluation for Qwen2.5-14B-Instruct under self-evolving SwS.It supports analysis of how the model evaluates its own generated problems.

I. Evaluation Benchmark Demonstrations

The evaluation uses eight mathematical reasoning benchmarks spanning grade-school problems, high-school and Olympiad mathematics, and the most challenging AIME tasks. Demonstrations also show category-labeling prompts and representative problems across several mathematical domains.

  • Benchmark Coverage: GSM8K contains 8,500 human-written grade-school word problems, while its 1,319-question test set emphasizes sequential reasoning and basic arithmetic.
  • Benchmark Coverage: AIME24 & 25 are identified as the study’s most challenging benchmarks, with 30 problems each requiring deep insight, multi-step reasoning, and precise problem solving.
  • Evaluation Benchmark Demonstrations: The category-labeling prompt requires one domain from a fixed list, concise problem summaries, professional scientific style, and a terminating report marker.
  • Evaluation Benchmark Demonstrations: The demonstrations include geometry, counting and probability, intermediate algebra, precalculus, and number theory problems with accompanying summaries of their required concepts.
Loading 2506.08989v1…