PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesis

Zhengbo Jiao, Shaobo Wang, Zifan Zhang, Xuan Ren, Wei Wang, Bing Zhao, Hu Wei, Linfeng Zhang

arXiv:2602.03279v1cs.AIcs.LG

TL;DR

High-quality, verifiable reasoning data is costly to obtain, while existing synthesis methods trade structural validity against problem difficulty. The paper introduces Agentic Proposing, which composes modular skills through reflective tool use and MGPO; solvers trained on its trajectories outperform baselines across domains, including 91.6% AIME25 accuracy for a 30B solver trained on 11,000 trajectories.

  • Problem

    Existing synthesis methods constrain novel high-difficulty compositions or produce inconsistent and unsolvable problems when constraints are relaxed, while human annotation is costly.

  • Method

    Agentic Proposing models synthesis as a goal-driven process in which an agent dynamically composes atomic skills through internal reflection, tool use, and MGPO optimization.

  • Results

    Agent-synthesized trajectories outperform leading baselines across mathematics, coding, and science; a 30B solver trained on 11,000 trajectories achieves 91.6% on AIME25.

  • Takeaways & Limitations

    The results indicate that dense, high-quality synthetic signals can support strong reasoning performance with relatively limited training data and cross-domain generalization.

  • Takeaways & Limitations

    Existing synthesis approaches remain constrained by seed quality, and corpus-based methods may fail to calibrate difficulty or target learner weaknesses.

Abstract

from arXiv · show

Advancing complex reasoning in large language models relies on high-quality, verifiable datasets, yet human annotation remains cost-prohibitive and difficult to scale. Current synthesis paradigms often face a recurring trade-off: maintaining structural validity typically restricts problem complexity, while relaxing constraints to increase difficulty frequently leads to inconsistent or unsolvable instances. To address this, we propose Agentic Proposing, a framework that models problem synthesis as a goal-driven sequential decision process where a specialized agent dynamically selects and composes modular reasoning skills. Through an iterative workflow of internal reflection and tool-use, we develop the Agentic-Proposer-4B using Multi-Granularity Policy Optimization (MGPO) to generate high-precision, verifiable training trajectories across mathematics, coding, and science. Empirical results demonstrate that downstream solvers trained on agent-synthesized data significantly outperform leading baselines and exhibit robust cross-domain generalization. Notably, a 30B solver trained on only 11,000 synthesized trajectories achieves a state-of-the-art 91.6% accuracy on AIME25, rivaling frontier-scale proprietary models such as GPT-5 and proving that a small volume of high-quality synthetic signals can effectively substitute for massive human-curated datasets.

1 Introduction

Existing synthesis methods struggle to jointly support high difficulty and logical validity. Agentic Proposing addresses this by composing atomic skills through reflective, tool-assisted synthesis, yielding strong downstream results.

  • Human annotation remains costly, motivating scalable synthesis of high-quality, high-difficulty, verifiable problems.
  • Fixed templates and structural priors constrain novel high-difficulty compositions, while relaxed constraints can produce inconsistent or unsolvable problems.
  • Agentic Proposing models synthesis as goal-driven compositional logic engineering, dynamically selecting and composing atomic reasoning skills.
  • Internal reflection and tool-use enable the proposing agent to audit and correct synthesized problems for logical soundness and calibrated difficulty.
  • Agent-generated problems improve downstream 4B solvers across mathematics, science, and coding, with stronger training signals than established reasoning collections.
  • 91.6% accuracy on AIME 2025 was achieved by a 30B solver trained on 11,000 agent-generated trajectories, rivaling frontier proprietary models.

2 Related Work

Prior synthesis approaches expand reasoning data through seed evolution, corpus extraction, or self-play, but their generation strategies leave important adaptation and difficulty-calibration gaps.

  • Seed-Based Expansion: Seed-based methods expand instructions or evolve existing problems, but remain constrained by seed quality and fixed adaptation mechanisms.
  • Corpus-Based Extraction: Corpus-based methods provide broad factual grounding but often struggle to calibrate difficulty or target specific learner weaknesses.
  • The reviewed paradigms motivate a pipeline that combines skill acquisition, agentic supervision, and multi-granularity reinforcement learning.
  • Agent-Based Self-Play: Agent-based self-play methods use reinforcement learning and multi-agent or self-evolving setups to generate reasoning data.

3 The Agentic Proposing Framework

Agentic Proposing models problem synthesis as a partially observable, goal-driven process in which an agent composes reasoning skills, reflects, uses tools, and self-corrects. Its training pipeline combines skill acquisition, agentic supervised fine-tuning, verifier filtering, and MGPO-based post-training with layered rewards and curriculum sampling.

  • Problem Formulation and Skill Composition: The framework treats synthesis as a POMDP because problem solvability is latent and requires probing through tool use and internal reflection.The agent observes dialogue history, active skills, and a cognitive-stage indicator while navigating drafting, checking, and refinement.
  • Skill Composition: The agent composes atomic skills represented by reasoning intent, construction method, difficulty effect, and tool-use hint into natural-language problem constraints.A mapping operator transforms selected skill compositions into instructions, while the policy governs generation.
  • Training Pipeline: The training pipeline first builds an autonomous skill library, then imitates teacher-generated agentic trajectories, and finally applies MGPO to refine high-precision synthesis.The supervised dataset retains only trajectories whose final problems pass a high-precision validity verifier.
  • Dynamic Pruning: Internal reflection can trigger dynamic pruning, removing skills predicted to misalign with the objective or cause logical errors before generation proceeds.The action space includes cognitive reflection, sandboxed execution, editing, and final submission.
  • Curriculum-based Skill Distribution: Curriculum sampling updates category proficiency with an exponential moving average and samples lower-proficiency categories more often.Sampling follows p(c) ∝ 1/(m_c + ϵ), using verifier-validated pass rates to focus training where performance is weaker.
  • Layered Reward Function: The reward design gives invalid problems zero total reward and awards difficulty bonuses only when externally probed instances remain solvable.Terminal rewards combine binary verification with a Pass@k-based difficulty term, while process rewards supervise successful tool executions and coherent reflections.

2. Stage-level Advantage AS(at): Standardized process reward (rproc

The method combines stage-level and trajectory-level advantages, centers rewards within cognitive-stage groups, and applies asymmetric temperature weighting during token-normalized policy optimization.

  • Stage normalization: Advantages and implicit rewards are centered within subgroups sharing the same cognitive-stage indicator.Centering produces normalized signals for stable credit assignment across stages.
  • Advantage fusion: The fused advantage combines trajectory-level and stage-level advantages with a positive fusion weight ω.This integrates episode-level quality with process-level feedback.
  • Asymmetric optimization: The optimization uses asymmetric temperatures, with τ_neg > τ_pos for negative practical weights.The policy distinguishes positive and negative training signals through separate temperatures.
  • Policy update: The policy is updated using token-normalized weighted maximum likelihood under the stated training objective.The normalization uses the total token count N.

4 Experiments

Experiments evaluate Agentic Proposing across mathematics, coding, and scientific reasoning under fixed trajectory budgets, reporting gains across benchmarks and cross-domain transfer.

  • Experiment setup: The evaluation covers contest mathematics, algorithmic coding, and scientific reasoning benchmarks.The listed suites include AIME, HMMT, AMO-Bench, LiveCodeBench, MMLU variants, GPQA, SuperGPQA, and OlympicArena.
  • Experiment setup: All methods are compared under fixed budgets of 10,000–11,000 trajectories against synthetic, human-curated, and frontier-model baselines.Downstream solvers use GRPO, with complete protocols and baseline configurations specified in the appendices.
  • Mathematics results: +4.1 points is the overall gain for a 4B solver trained on 10,000 synthesized mathematics trajectories.The largest contest-mathematics gains are +4.5 points on AIME 2025 and +5.5 points on HMMT.
  • Scaling and generalization: 91.6% on AIME 2025 is achieved by a 30B solver trained on 11,000 mixed trajectories, with gains of +5.3 and +5.2 points on LiveCodeBench v5 and v6.The result is reported as state-of-the-art for its scale and surpasses leading open-source reasoning collections.
  • Cross-domain transfer: +5.3 points is the overall gain across multidisciplinary benchmarks for a 4B solver trained on coding and scientific trajectories.The reported graduate-level gains are +7.3 points on SuperGPQA and +6.3 points on GPQA; OlympicArena improves by +4.4 points.
  • Cross-domain transfer: Agentic Proposing synthesizes trajectories that foster cross-disciplinary robustness rather than the domain-specific overfitting associated with traditional augmentation.This claim is presented as the cross-domain transfer finding of the synthesis pipeline.

5 Analysis and Ablation Studies

Ablations examine proposer specialization, agentic correction, and MGPO credit assignment, finding that structured guidance, iterative quality gates, and stage-level rewards improve downstream performance.

  • Ablation protocol: The ablations train a Qwen3-4B downstream solver on fixed 10,000-trajectory budgets and evaluate AIME with Mean@64 accuracy.Additional sensitivity analyses cover MGPO hyperparameters, dynamic skill pruning, and curriculum design.
  • Proposer specialization: +3.6 points is gained when GPT-5.2 uses the Skill Library and Agentic Workflow instead of raw prompting.The specialized 4B proposer exceeds augmented GPT-5.2 by +1.9 points and raw GPT-5.2 by +5.5 points.
  • Agentic correction: Tool-use and internal reflection each improve downstream performance by over 2 points independently.Their full combination in the Draft–Check–Refine loop achieves a cumulative +6.8-point gain over one-shot proposing.
  • Agentic correction: +6.8 points is the cumulative gain from the iterative Draft–Check–Refine loop over one-shot proposing.The result supports the reported role of iterative correction in generating high-difficulty, logically sound problems.
  • MGPO credit assignment: +6.5 points is the improvement of MGPO over standard trajectory-level GRPO in the credit-assignment ablation.The stage-level advantage assigns credit to intermediate behaviors such as internal reflection and tool invocation.

6 Conclusion

Agentic Proposing reframes problem synthesis as autonomous compositional logic engineering, combining modular reasoning skills, self-correction, and MGPO to generate high-quality training data. The framework achieves strong performance across mathematics, coding, and science while emphasizing training-signal density over parameter scale.

  • Agentic Proposing transforms problem synthesis into a goal-driven process of compositional logic engineering.
  • The framework integrates modular reasoning skills, a self-correcting agentic pipeline, and MGPO to generate diverse, difficult, and verifiable training data.
  • Agentic Proposing enables solvers to achieve state-of-the-art performance across mathematics, coding, and science while outperforming strong baselines and frontier proprietary models.
  • The findings identify the density of high-quality training signals, rather than parameter scale, as the bottleneck for advanced reasoning.
  • Decomposing complex reasoning into composable atomic skills supports a shift from static prompting toward dynamic logic construction.

A Experimental Setup

The experimental setup evaluates Agentic Proposing across mathematics, coding, and scientific reasoning under controlled trajectory budgets, with standardized grading and verifier-based quality control. The synthesis pipeline combines specialized proposer, teacher, verifier, and prober models with iterative auditing and curriculum updates.

  • Models: The Agentic-Proposer-4B is initialized from Qwen3-4B-Instruct-2507 and optimized with agentic SFT and MGPO, while separate teacher and verifier models support skill acquisition and validity checking.The teacher is Qwen3-235B-Instruct-2507, and the verifier ensemble includes Qwen3-235B-Thinking, DeepSeek-V3.2-Special, and GPT-OSS-120B.
  • Benchmarks: The framework evaluates contest mathematics, algorithmic coding, and scientific or general reasoning across the listed benchmark suites.Contest mathematics uses AIME24/25, HMMT, and AMO-Bench; coding uses LiveCodeBench v5/v6; science and general reasoning use MMLU-Redux, MMLU-Pro, GPQA, SuperGPQA, and OlympicArena.
  • Datasets and Baselines: All methods are compared under a fixed budget of 10,000–11,000 trajectories against synthetic, curated, and frontier-model baselines.The baseline groups include synthetic data methods, human-annotated or curated methods, and trajectories synthesized by frontier models.
  • Evaluation: Contest mathematics uses Mean@64, coding uses best-of-5 execution testing, and science or general reasoning uses Mean@1 with fixed few-shot evaluation.Rule-based graders are used whenever available, with auxiliary LLMs restricted to answer extraction or normalization for complex formats.
  • Verifier Ensemble: Verifiers inspect the complete proposal trace and final problem, independently solve it, vote on validity, and apply a second audit before acceptance.Acceptance requires at least two valid votes, well-posed and solvable rationales, and consistent final answers or an explained discrepancy confirmed by the audit.
  • Difficulty Estimation: Difficulty is estimated with Pass@k using k = 16, while a curriculum of frozen probers spanning Qwen3 1.7B–30B and gpt-oss-20b tracks mastery and switches models as accuracy falls.The current prober is replaced when its stabilized accuracy drops below 30%.

A.3 Training Details and Hyperparameters

Training details standardize downstream optimization and isolate data quality from formatting and algorithmic variation. The appendix also specifies baseline construction, fixed-budget sampling, outcome-based rewards, and the computational environment.

  • Solver Training: All downstream solvers use a standardized training configuration, with GRPO applied consistently across data sources.This design attributes performance differences to the training data rather than algorithmic variations.
  • Solver Training: Solvers train only on final question–answer pairs, excluding proposer reflections, tool calls, and verifier or prober outputs.
  • Reward Function: The outcome reward assigns 1 for a ground-truth-matching answer and 0 otherwise, while format violations receive reward 0.Correctness is checked with LaTeX and Sympy parsers.
  • Computational Environment: Training runs on H200 GPUs with Flash-Attention-2 and 20,480-token sequences for numerical stability and consistent gradient scaling.
  • Data Construction: Data sources are compared under a fixed 10,000–11,000-trajectory budget, with random downsampling for established open-source corpora.Proprietary models receive targeted prompts for difficult STEM problem synthesis under logical-integrity constraints.
  • Baselines: R-Zero and Socratic-Zero provide 4B-specific self-play baselines, while POLARIS supplies a downsampled collection of high-quality open-source reasoning traces.Socratic-Zero filters seed problems to approximately 50% Pass@1 for the base 4B model.

B Analysis of Baseline Performance Degradation

Baseline data can degrade advanced reasoning performance when it is already too easy for the target model, because high pass rates reduce the reinforcement-learning signal. Agentic Proposing addresses this through frontier-targeted difficulty, curriculum sampling, weighted advantages, and dynamic skill pruning.

  • Baseline Diagnosis: High zero-shot pass rates on established baselines leave little room for reinforcement-learning improvement in the base 4B model.The diagnostic evaluates eight rollouts per problem and reports high baseline success rates.
  • Advantage Saturation: When pass rates reach 85%–89%, mostly correct rollout groups drive the relative advantage toward zero and weaken the learning signal.The analysis links this saturation to insufficient contrast between positive and negative trajectories.
  • Difficulty Distribution: Standard baselines can induce entropy collapse because mastered reasoning patterns reduce exploration of complex logic and eventually degrade performance.
  • Frontier Targeting: Agentic Proposing targets the model’s reasoning frontier to preserve an informative advantage signal during reinforcement learning.
  • MGPO Weighting: ω = 0.5 provides the optimal balance between trajectory-level and stage-level advantages in MGPO.Lower weights provide insufficient credit assignment, while excessive weights over-prioritize local tool-call rewards over overall difficulty and logical flow.
  • Curriculum Sampling: Dynamic curriculum sampling increases the rate of Combinatorial Proofs by 3.4× to force exploration of high-difficulty skill compositions.The mechanism is designed to prevent collapse into a low-difficulty safety zone.
  • Dynamic Skill Pruning: The agent prunes its active skill set in 14.5% of trajectories when internal reflection detects a potential contradiction, improving validity through mid-process correction.The comparison with and without pruning shows that self-correction is especially important when composing multiple complex skills.

C.3 MGPO Ablations and Sensitivity Analysis

The ablations evaluate proposer accuracy, MGPO’s gating design, temperature sensitivity, and the external mastery-state statistic used to stabilize model switching.

  • Evaluation metric: Problem-proposing accuracy is defined as the fraction of verifier-accepted problems among 1,000 samples generated under the same skill.Validity follows the verifier protocol in Appendix A.2.
  • MGPO ablations: 93.4% proposing accuracy is achieved by full MGPO, while symmetric or removed gates reduce accuracy by 5–6 absolute points.The comparison uses 1,000 proposed problems under the same skill.
  • Temperature sensitivity: Performance remains robust around the default gate temperatures, whereas extreme temperatures degrade proposing accuracy.Larger τ attenuates off-policy or noisy updates more strongly, and the default uses τneg > τpos.
  • Mastery-state tracking: The mastery-state vector is an external statistic rather than a gradient-learned parameter, tracking each prober model’s recent correctness with an exponential moving average.The stabilized accuracy is the EMA value mt, and the system switches prober models when mt < 0.30.
  • Mastery-state tracking: The accuracy EMA uses rule-based accuracy from the most recent batch of 500 problems and a smoothing factor α ∈(0, 1).The EMA determines stabilized accuracy for prober switching.

E Case Study: Refactoring a Complex Analysis Skill

The case study shows how Agentic-Proposer-4B packages a complex-analysis method as a reusable skill, loads it on demand, verifies generated problems with tools, and refines failures.

  • Skill packaging: A modular skill package turns complex contour integration with high-order poles into a reusable capability for the Agentic-Proposer-4B.The package supports progressive disclosure and tool-assisted verification rather than serving as a static prompt.
  • Skill execution: The Proposer loads metadata during drafting, then retrieves the procedural workflow only when the synthesis goal requires a contour integral with a cubic pole.This implements on-demand loading followed by progressive access to SKILL.md.
  • Verification and refinement: The closed-loop workflow invokes a symbolic evaluator to verify the generated problem and triggers refinement when the script detects an invalid singularity.Internal reflection adjusts the numerator function so the final problem is mathematically rigorous.
  • Compositional synthesis: The case composes differential-inequality and sequence-growth skills, then uses the refinement loop to resolve a conflict between f ≡0 and c > 0.The trajectory is presented as a real-world example of compositional skill synthesis and logical correction.
  • Skill representation: The formalized skill separates a problem’s structure, sequential operation, target effect, and external verification utility.Its construction logic specifies invariant constraints, imperative operations, verifiable outcomes, and a Python/SymPy validity check.
  • Complex-analysis procedure: For generalized Cauchy evaluation, the workflow checks analyticity and contour enclosure, maps pole order to derivative order, differentiates, and evaluates the formula.This converts path integration into deterministic algebraic evaluation of an n-th derivative, avoiding residue-sum calculations or parameterization.
Loading 2602.03279v1…