PaperlayerView exact revision on arXiv

Source-linked AI summary

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

Yanqi Dai, Yuxiang Ji, Xiao Zhang, Yong Wang, Xiangxiang Chu, Zhiwu Lu

arXiv:2601.20614v1cs.AIcs.CL

TL;DR

Existing RLVR methods underemphasize challenging mathematical questions, while common augmentation mainly increases diversity rather than intrinsic difficulty. MathForge combines difficulty-aware optimization with answer-preserving, multi-aspect reformulation, and experiments show it significantly outperforms existing methods across models and benchmarks.

  • Problem

    Existing RLVR methods underemphasize challenging questions because GRPO updates are imbalanced and augmentation does not systematically increase intrinsic difficulty.

  • Method

    MathForge combines DGPO, which balances and prioritizes updates for harder questions, with MQR, which creates more difficult variants while preserving gold answers.

  • Results

    42.17% average performance was achieved by full MathForge, which significantly outperformed existing methods across models and mathematical reasoning benchmarks.

  • Takeaways & Limitations

    The results support targeting harder, answer-solvable questions from both optimization and data perspectives for mathematical reasoning training.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.

1 INTRODUCTION

The paper identifies difficulty-related weaknesses in RLVR for mathematical reasoning: GRPO underemphasizes easy and hard questions, while existing augmentation overlooks question difficulty. MathForge addresses both issues through DGPO and MQR, which prioritize challenging questions algorithmically and through answer-preserving reformulation.

  • Problem: GRPO’s advantage estimation suppresses update magnitudes for both easier and harder questions, with updates peaking at moderate difficulty.The paper states that this limitation is mathematically revealed and proved for GRPO and its variants.
  • Problem: Existing reasoning augmentation often creates new question-answer pairs whose answer quality is difficult to guarantee, while RLVR-specific rephrasing has not addressed question difficulty.The passage highlights this concern especially for competition-level problems.
  • Method: MathForge targets harder questions from both algorithmic and data perspectives through Difficulty-Aware Group Policy Optimization and Multi-Aspect Question Reformulation.DGPO rectifies GRPO’s implicit update-magnitude imbalance, while MQR reformulates questions across multiple aspects to increase difficulty while preserving the original gold answer.
  • DGPO: DGPO rectifies GRPO’s implicit imbalance and further upweights more challenging questions.Its contribution includes difficulty-balanced group advantage estimation and difficulty-aware prioritization of harder questions.
  • MQR: MQR increases reformulated-question difficulty across multiple aspects while preserving the original gold answer.The strategy is designed as data augmentation tailored for RLVR.

2 PRELIMINARIES

The preliminaries model autoregressive language models as policies trained with verifier-based rewards, and introduce GRPO’s group-relative advantage estimation and key notation.

  • Notation: An autoregressive language model parameterized by θ is treated as a policy, with πθ and πθold denoting current and old policies.For each query, responses are generated using πθold.
  • Notation: A rule-based verifier assigns a scalar reward to each query–response pair, defaulting to accuracy reward 1 for correct responses and 0 otherwise.The reward is defined for each generated response associated with a query.
  • Group Relative Policy Optimization (GRPO): GRPO is a PPO variant that removes the critic model and estimates response advantages relative to other responses generated for the same query.Later modifications remove KL divergence and use a token-level policy-gradient loss.
  • Group Relative Policy Optimization (GRPO): In the GRPO notation, G is the group size, ε is the clipping range, I_i,t(θ) is the token importance-sampling ratio, and Â_GR,i is the response advantage from GRAE.G denotes the number of responses generated for each query.

3 METHODOLOGY

MathForge targets harder mathematical questions through two complementary components: DGPO improves optimization for challenging examples, while MQR generates more difficult reformulations that preserve the original answers. Together, MQR expands the training frontier and DGPO learns from the resulting challenges.

  • Difficulty-Aware Group Policy Optimization: DGPO addresses GRPO’s difficulty-dependent update imbalance with difficulty-balanced group advantage estimation and difficulty-aware question-level weighting.GRPO gives smaller updates to harder questions, while DGPO normalizes update magnitudes and then prioritizes more challenging questions.
  • Difficulty-Aware Group Policy Optimization: GRPO’s single-question update magnitude peaks at accuracy rate p = 0.5 and decreases as p approaches 0 or 1.Thus, moderate-accuracy questions influence policy updates more than easier or harder questions.
  • Difficulty-Aware Group Policy Optimization: DGAE removes GRPO’s binary-reward constraint and makes the total update magnitude for a single question constant.Its reward normalization uses the mean absolute deviation across responses.
  • Difficulty-Aware Group Policy Optimization: DQW measures relative difficulty using the negative mean reward Ds and controls weight sharpness with temperature T.The scheme explicitly prioritizes harder questions within each training batch and is derived from GRPO’s update imbalance analysis and DGAE’s balanced advantages.
  • Multi-Aspect Question Reformulation: MQR automatically reformulates questions by adding story background, abstract terminology, or nested sub-problems while preserving the original gold answer.These variants target robustness to irrelevant information, abstract concepts, and multi-step or cross-domain reasoning.
  • Multi-Aspect Question Reformulation: MQR-augmented data combines original and reformulated questions as training material for DGPO, forming a synergistic loop between harder data and difficulty-aware learning.The default reformulator is OpenAI o3, though smaller open-source models can also handle the task.

4 EXPERIMENTS

Experiments evaluate MathForge across text-only and multimodal mathematical reasoning settings, using Qwen2.5-Math-7B on MATH as the main setup and additional models for model-agnostic analysis. DGPO, MQR, and their combination improve performance, with gains supported by difficulty assessments, ablations, cross-model results, and multimodal evaluation.

  • Experimental Setup: Experiments train Qwen2.5-Math-7B on MATH and additionally evaluate Qwen2.5-Math-1.5B, Qwen2.5-3B, and DeepSeek-Math-7B.DeepSeek-Math-7B uses 80k NuminaMath-CoT examples for cold start.
  • Experimental Setup: Evaluation covers AIME24, AIME25, AMC23, MATH500, Minerva, Olympiad, and the multimodal GeoQA test set.The study averages 32 runs for AIME24, AIME25, and AMC-23, and 4 runs for the other benchmarks.
  • Effectiveness of DGPO: 39.79% average score from DGPO exceeds GRPO’s 37.61% by 2.18%.The gain is attributed to difficulty-balanced advantage estimation and question-level weighting that prioritize challenging questions.
  • Effectiveness of MQR: 41.04% average score from MQR is 3.43% higher than GRPO.Background, Term, and Sub-Problem reformulations introduce narrative noise, abstract concepts, and nested logic.
  • Effectiveness of MathForge: 42.17% average score from full MathForge is the best reported performance, exceeding both DGPO and MQR individually.The results describe synergy between challenging augmented data from MQR and targeted updates from DGPO.
  • MQR Analysis: MQR-augmented questions are harder than Original, with accuracies of 79.77%, 77.31%, 76.87%, and 72.04% for Original, Background, Term, and Sub-Problem, respectively.MQR also outperforms original data across models, indicating gains arise from qualitative data enhancement rather than increased volume alone.

5 RELATED WORK

Related work advances mathematical reasoning through both reinforcement-learning policy optimization and data-centric augmentation. GRPO provides an efficient critic-less alternative built on PPO, while augmentation methods generate synthetic problems or reformulate existing questions while preserving answers.

  • Reinforcement Learning: Policy optimization has become standard for post-training large language models to enhance reasoning, with GRPO introducing an efficient critic-less paradigm based on group-relative advantage estimation.GRPO builds upon Proximal Policy Optimization (PPO) and spurred research on improving its stability.
  • Data Augmentation: Data augmentation improves mathematical reasoning by generating new problem-solution pairs or reformulating existing questions while preserving the original answer.Teacher-generated synthetic data has been reported to rival human-curated datasets, while MetaMath and related approaches reformulate existing questions.

6 CONCLUSION

MathForge enhances mathematical reasoning by targeting harder problems through both algorithmic and data perspectives. Its DGPO algorithm addresses update imbalance and prioritizes challenging questions, while MQR creates more difficult, answer-preserving question variants.

  • 6 CONCLUSION: MathForge targets harder mathematical problems through the DGPO algorithm and MQR data augmentation strategy.DGPO rectifies update-magnitude imbalance and prioritizes challenging questions, while MQR generates more difficult variants that preserve the original answers.

ETHICS STATEMENT … B.2 FULL DERIVATION FOR THE TOTAL UPDATE MAGNITUDE OF GRPO

The paper reports ethical safeguards, limited use of LLMs for writing assistance, and derivations showing that GRPO’s question-level update strength is governed by aligned response gradients and advantage magnitudes. The derivation treats the sum of absolute advantages as a tight proxy rather than an exact equality under stated assumptions.

  • ETHICS STATEMENT: The work follows the ICLR Code of Ethics and reformulates public MATH problems without personally identifiable information or sensitive content.The stated research goal is to enhance mathematical reasoning capabilities in AI models.
  • A THE USE OF LARGE LANGUAGE MODELS (LLMS): LLMs were used only to polish grammar, clarity, and style, while human authors conceived and executed the methods, experiments, and analysis.The passage explicitly limits LLM assistance to writing improvement.
  • B.1 FULL DERIVATION FOR GRADIENT OF GRPO: The proofs consider a single question q with its corresponding response group when deriving GRPO’s unclipped policy gradient.The response group is denoted {o_i}^G_{i=1}.
  • B.1 FULL DERIVATION FOR GRADIENT OF GRPO: The derivation defines sgn(·) as the sign function and detach(·) as the stop-gradient operator.These operators appear in the GRPO gradient derivation.
  • B.2 FULL DERIVATION FOR THE TOTAL UPDATE MAGNITUDE OF GRPO: For a fixed question q, the PPO/GRPO-style gradient is analyzed while ignoring token-length differences, clipping, and importance-sampling terms.This expression provides the starting point for deriving total update magnitude.
  • B.2 FULL DERIVATION FOR THE TOTAL UPDATE MAGNITUDE OF GRPO: Positive correlation among same-question response gradients limits cancellation, making the triangle inequality nearly tight.The gradients tend to jointly improve the policy on the specific query.
  • B.2 FULL DERIVATION FOR THE TOTAL UPDATE MAGNITUDE OF GRPO: Because gradient-norm variation is typically smaller than absolute-advantage variation, the sum of absolute advantages is a tight update-strength proxy, not an exact equality.Responses share a policy, temperature, and similar mathematical prompt.

B.3 PROOF OF THEOREM 1 · B.4 PROOF OF THEOREM 2 · C PROMPTS FOR MQR

The appendices prove update-magnitude results for GRAE and DGAE using absolute advantages, reward statistics, and groupwise decomposition. They also specify MQR prompts that make questions harder while preserving answers through constrained reformulation methods.

  • B.3 PROOF OF THEOREM 1: Theorem 1’s proof analyzes update magnitude for a single question using GRAE.It defines total update magnitude as the sum of absolute advantage values.
  • B.3 PROOF OF THEOREM 1: For binary rewards, the proof represents the mean reward as the accuracy rate p.The subsequent expression uses p(1 − p) in the update-magnitude calculation.
  • B.3 PROOF OF THEOREM 1: The GRAE numerator is decomposed into Gp terms with reward 1 and G(1 − p) terms with reward 0.This reward-count decomposition produces the next step of the proof.
  • B.4 PROOF OF THEOREM 2: The DGAE derivation moves a constant, non-negative factor outside the outer summation.The factor is constant with respect to the summation index i.
  • C PROMPTS FOR MQR: The general MQR prompt assigns the model the role of an expert Math Question Rephraser.It asks for a more challenging, logically sound, human-comprehensible question with exactly the original final answer.
  • C PROMPTS FOR MQR: MQR provides background, term, and sub-problem instructions for increasing question complexity while retaining a coherent mathematical task.Background reformulation adds an unrelated story context; term reformulation defines an abstract mathematical term; sub-problem reformulation converts a definite numerical condition into a self-contained problem with a unique solution yielding the required value.
  • C PROMPTS FOR MQR: The general MQR constraints preserve the final answer, limit added length to 100 words, and preserve the original interrogative verb.The prompt also requires LaTeX mathematical expressions and outputting only the rephrased question.

D AUGMENTED DATA OF MQR

The MQR augmented-data examples illustrate question reformulations that modify context, terminology, and subproblems while preserving the underlying purchase-and-exchange scenario. These examples show how MQR increases question complexity through multiple reformulation strategies.

  • Reformulation Examples: MQR presents generated-question examples with highlighted portions indicating the principal modifications made by each reformulation strategy.The examples are explicitly organized to show the changes introduced during reformulation.
  • Background Reformulation: Background reformulation enriches the bakery scenario with detailed Parisian setting, character roles, and dessert-specific context while retaining the original numerical task.The reformulated question adds Montmartre, culinary and anthropology backgrounds, and a named pâtisserie and cake.
  • Term Reformulation: Term reformulation introduces the defined quantity “euro-gap” ϵ to reframe the required contribution as a formal difference between price and available funds.The definition specifies the euro-gap as a non-negative, euro-denominated difference before restating the purchase conditions.
  • Sub-Problem Reformulation: Sub-problem reformulation inserts an independent integer system, x + y = 9 and x^2 + y^2 = 41, before defining an exchange ratio used in the final payment question.The added subproblem asks for positive integers and then uses the ratio of the larger to smaller solution as the exchange-rate basis.

E DATA AUGMENTATION COST OF MQR

MQR generates reformulated questions at a measurable token and monetary cost, requiring approximately $184 to produce 22.5k reformulated MATH questions.

  • Approximately $184 is required to generate 22.5k reformulated questions for the MATH dataset.
  • The average input usage is 255.05 tokens per question.
  • The average output usage is 820.27 reasoning tokens and 138.33 reformulated-question tokens per question.

F IMPLEMENTATION DETAILS … G COMBINATION FORMS OF DGPO AND OTHER METHODS

The paper specifies fixed generation and evaluation settings, dataset-specific training configurations, and integrations that adapt DGPO’s difficulty-aware estimation and weighting to GPG, DAPO, and GSPO. These integrations preserve each method’s core objective structure while applying difficulty-aware updates at the token, question, or sequence level.

  • F IMPLEMENTATION DETAILS: Training responses used temperature 1.0 with a 1024-token maximum, while evaluation used temperature 0.6, top-p 0.95, and 4096 maximum new tokens.
  • F.1 MATH: MATH training used step-by-step reasoning prompts, 512-token maximum prompts, 8 responses per question, batch size 32, and binary accuracy rewards.Correct final answers received reward 1; incorrect answers received 0.
  • F.1 MATH: The MATH experiments covered both the original 7.5k dataset and the 30k MQR-augmented dataset, with model-specific hyperparameters reported in Table 9.
  • F.2 GEOQA-8K: GEOQA-8k preprocessing removed non-standard units from gold answers and adjusted the prompt to require final answers without units.
  • F.2 GEOQA-8K: GEOQA-8k training used 2048-token maximum prompts, 8 responses per question, 2 epochs, learning rate 1e−6, batch size 32, gradient accumulation 1, and 480 training steps.Its reward mechanism remained binary accuracy.
  • G.1 GPG: For GPG, DGPO replaces the original advantage with difficulty-balanced DGAE, applies DQW question weighting, scales token updates by advantage, and averages over valid tokens.
  • G.2 DAPO: For DAPO, DGPO preserves the clipped PPO-style objective and composite accuracy-plus-length reward, while computing question difficulty from accuracy alone to avoid verbosity effects.
  • G.3 GSPO: For GSPO, DGPO preserves sequence-level importance sampling, drives sequence updates with DGAE, applies DQW question weighting, and averages loss over valid questions.

H QUALITY ASSESSMENT OF MQR

MQR quality was assessed with OpenAI o3 by testing whether reformulated questions yield the same final answer as their originals. Across 100 sampled questions per category, equivalence rates were 99% for Background and 97% for both Term and Sub-Problem reformulations.

  • Assessment method: OpenAI o3 evaluated mathematical equivalence by determining whether original and reformulated questions produce the same final answer.Equivalence was defined as yielding the same final numerical answer or symbolic solution.
  • Equivalence results: 99% equivalence was achieved for Background reformulations, compared with 97% for Term and 97% for Sub-Problem reformulations.The evaluation randomly sampled 100 questions from each category.
  • Failure criterion: MQR defines a failed reformulation as one that becomes unsolvable or produces an answer different from the original.Because RLVR requires exact canonical matching over a large open-ended answer space, accidental agreement from incorrect reasoning is highly improbable.
Loading 2601.20614v1…