Source-linked AI summary
Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?
Dadi Guo, Yuejin Xie, Qingyu Liu, Weixian Huang, Jiayu Liu, Zhiyuan Fan, Qihan Ren, Shuai Shao, Tianyi Zhou, Jianjie Feng, Wenze Su, Yujiu Yang, Dongrui Liu, Yi R. Fung
TL;DR
The paper asks whether code agents can address the shortage of challenging mathematical problems by autonomously evolving existing ones. It introduces a multi-agent framework that explores candidate adaptations and verifies solvability and difficulty. Experiments show that the resulting problems remain solvable and become structurally distinct and more challenging, although the evaluation uses a relatively small seed set and does not test downstream training value.
Problem
The scarcity of challenging, high-quality mathematical problems makes automated synthesis important for training, evaluation, and model self-evolution.
Method
A multi-agent framework uses code-driven exploration to evolve seed problems, then separately verifies solvability and increased difficulty.
Results
94/98 agreement, approximately 96%, is achieved by DeepSeek-Reasoner with an external judge, while experiments show evolved problems remain solvable and substantially increase difficulty.
Takeaways & Limitations
Code-driven agents can synthesize mathematically valid, harder problems through test-time exploration and structured reasoning.
Takeaways & Limitations
The experiments use only 100 seed problems, and the paper does not verify whether generated problems improve model performance as training data.
Abstract
from arXiv · showhide
As large language models (LLMs) advance their mathematical capabilities toward the IMO and research level, the scarcity of challenging, high-quality problems has become a significant bottleneck for training, evaluation and self-evolution of LLMs. Simultaneously, recent code agents have demonstrated sophisticated skills in agentic coding and reasoning, suggesting that code execution can serve as a scalable environment for mathematical experimentation. In this paper, we investigate the potential of code agents to autonomously evolve existing math problems into more complex variations. We introduce a multi-agent framework designed to perform problem evolution while validating the solvability and increased difficulty of the generated problems. Our experiments demonstrate that, given sufficient test-time exploration, code agents can synthesize new, solvable problems that are structurally distinct from and more challenging than the originals. This work provides empirical evidence that code-driven agents can serve as a viable mechanism for synthesizing high-difficulty mathematical reasoning problems within scalable computational environments. Code and data is available at https://github.com/TarferSoul/Code2Math.
1 Introduction
The paper addresses the scarcity of challenging mathematical problems by using code agents to explore and autonomously evolve existing problems. Its multi-agent framework verifies solvability and difficulty, and experiments show that evolved problems remain solvable while becoming harder.
- The scarcity of challenging, diverse mathematical problems limits training, evaluation, and self-evolution of mathematical reasoning models.Manual curation requires deep expertise and substantial human effort.
- Code agents support computational exploration through simulation, symbolic manipulation, exhaustive search, and automated verification.These capabilities mirror exploratory processes such as conjecture formation and counterexample search.
- The study asks whether evolved problems are sound and solvable, genuinely harder for current models, and efficiently produced.The experiments use 100 seed problems from textbooks, competitions, the IMO, AIME, and other benchmarks.
- 94/98 agreement, approximately 96%, is achieved by DeepSeek-Reasoner with an external judge, while the generated problems remain highly solvable.The experiments test evolved problems across six solver models.
- Models can synthesize challenges beyond their own solving baselines, but successful evolution requires 1.56 to 6.55 failures per success on average.Difficult cases often exceed 10 iterations, indicating substantial computational overhead.
- The framework decomposes adaptation into evolution, solvability verification, and difficulty verification, with code execution supporting symbolic reasoning and structured exploration.Multiple Evolution Agent rollouts continue until both verification agents’ criteria are satisfied.
2 Related Works
Prior work uses environment exploration and verification to synthesize agent, general-purpose, and coding data, whereas this paper studies mathematical problem evolution.
- Recent systems progressively synthesize agent data by exploring environments, while TRACE and AutoCode evolve general-purpose and coding tasks through exploration with verification.
3 Method
The method uses a three-agent pipeline to evolve seed problems, verify solvability, and assess genuine difficulty increases. Test-time rollouts and executable mathematical tools support exploration, validation, and efficiency measurement.
- Seed Problems: The study collects 100 problems spanning algebra, combinatorics, calculus, sequences, and graph theory from books, competitions, the IMO, and AIME benchmarks.Six additional problem pairs serve as adaptation and evaluation examples.
- Multi-Agent System: The multi-agent system comprises an Evolution Agent, a Solvability Verification Agent, and a Difficulty Verification Agent.The framework takes original problems and solutions as inputs and produces validated new problems with reference solutions.
- Evolution Agent: The Evolution Agent analyzes the seed solution’s cognitive bottleneck, explores candidate modifications, and outputs a harder problem with proposed solution steps.It targets concealed insights, tighter bounds, combinatorial constructions, and numerical patterns to increase the Burden of Discovery.
- Solvability Verification: The Solvability Verification Agent first detects surface errors and then scrutinizes the proposed solution for logical inconsistencies.A flaw-less logical chain serves as a proxy for solvability, while flawed solutions are discarded.
- Difficulty Verification: The Difficulty Verification Agent compares original and adapted problems to assess whether the adapted problem requires a harder-to-discover Aha moment.Its five-point scale separates Cognitive Depth from Artificial Complexity; scores 1–2 fail the acceptance threshold.
- Difficulty Verification: Successful adaptations begin at Score 3 by breaking standard solution templates, while Scores 4–5 require stronger anti-templating effects and deeper Aha moments.Score 5 additionally rewards mathematical beauty such as deep symmetries or unexpected conceptual connections.
- Test-Time Exploration: Multiple rollouts use code for simulations, sequence inspection, tighter-bound exploration, and counterexample searches before candidate acceptance.The evaluation measures solvability, difficulty, code’s exploration role, and efficiency.
- Evaluation: Solvability is assessed with a unified third-party LLM judge, while difficulty is measured through changes in solver accuracy and reasoning length.Efficiency is the average number of Evolution Agent rollouts needed to obtain a qualified question.
4 Experiments
Experiments show that the multi-agent system produces mostly solvable evolved problems that are harder and more discriminative for current solvers, but the process requires substantial computational effort.
- Solvability Verification: External judges certified 74/90 to 98/98 internally accepted problems across evolution backbones, including 94/98 for DeepSeek-Reasoner and 98/98 for Gemini-3-Pro-Preview-Thinking.
- Difficulty Escalation: GPT-5.2-High solve rates fell from 70% on seeds to 64% on DeepSeek-Reasoner evolutions and 61% on Gemini-3-Pro-Preview-Thinking and Seed-2.0-Pro evolutions.
- Difficulty Escalation: Evolved problems add Burden of Discovery by forcing solvers away from standard solution templates rather than applying superficial paraphrases.
- Reasoning Strength of the Evolver: Reasoning-enhanced evolvers introduce structural modifications that transfer across solver models, creating capability asymmetries and exposing robustness gaps.Gemini-3-Flash-Thinking starts at a 56% seed solve rate and can lose more than 20 points, while Qwen3-235B-A22B-Thinking starts at 20% and often changes only slightly.
- Computational Cost: Evolved problems shift the ATC distribution toward more high-token and timeout cases, indicating longer reasoning chains and deeper search or self-correction.
- Efficiency and Failure Analysis: Qualified evolution requires multiple rollouts: Gemini-3-Pro-Preview-Thinking averages 1.56 failed rollouts, whereas Kimi-K2-Thinking averages 6.55.
- Efficiency and Failure Analysis: Most failures arise from solvability verification, making logical consistency the dominant bottleneck while strict dual verification improves reliability at added computational cost.
- Human Evaluation: Human and GPT judgments closely align: validity precision is 99.2%, solver-answer agreement is 1134/1226, and difficulty scores differ by at most one point in 95.9% of cases.
5 Conclusion and Discussion
The paper presents code-driven test-time exploration with dual verification as a way to autonomously generate mathematically valid problems that are harder for contemporary solvers. It also identifies computational efficiency and structural synthesis as remaining challenges.
- The framework combines executable environments, structured reasoning, test-time exploration, and dual verification to evolve mathematical problems autonomously.
- Generated problems are mathematically valid and empirically harder, reflected by solve-rate declines and increased reasoning effort across models.
- Multiple rollouts are often required, with logical consistency as a primary bottleneck that creates a reliability–computational-efficiency trade-off.
- Code execution enables local validation and structural probing, but more systematic mechanisms for structural synthesis remain to be explored.
Limitations
The study identifies limited seed-set scale, untested downstream training value, and partial human evaluation as constraints on the generality and reliability of its findings.
- The experiments use only 100 seed problems because the full evolution pipeline is computationally expensive.Larger and more diverse seed sets would provide stronger evidence for generality.
- The study does not verify whether generated problems improve model performance when used as training data.Downstream training value remains an important direction for future work.
- Human evaluation covers sampled cases rather than the entire generated set because assessing mathematical quality requires substantial expert labor.Comprehensive human evaluation could strengthen the reliability of the conclusions.
Ethics Statements
The ethics statement addresses risks from automatically generated mathematics, research-use restrictions, data privacy, and the roles of human annotators and writing assistants.
- Automatically generated problems may contain subtle errors, misleading solution steps, or artificial rather than genuine difficulty.The framework uses solvability and difficulty verification, and sampled outputs were manually inspected and filtered.
- Generated derivatives should remain within research contexts when original access conditions impose restrictions.
- The mathematical problem and solution data is not expected to contain personally identifying information.
- Human annotation was limited to expert inspection by the paper’s authors, who assessed sampled problems for solvability, correctness, and difficulty.
- GPT models were used only to polish writing, while authors retained responsibility for scientific judgment, experiments, annotation, interpretation, and final content.
A Appendix
The appendix supplies additional experimental setup details, agent prompts, and examples of the problem-evolution process.
- The appendix presents extra setup details, prompts for the three agent types, and evolution cases.
A.1 Experimental Setup Details
The experimental setup specifies model roles, rollout and timeout rules, reproducible evaluation settings, and metrics for evolution success, solvability, agreement, solving performance, and effort.
- Experimental Setup Details: Evolution instances succeed only when generated problems pass both verification agents within 20 rollouts.Otherwise, the evolution is recorded as a failure and terminated.
- Experimental Setup Details: Solver evaluation allows three attempts per problem and treats consistent timeouts after token or 30-minute limits as failures.Models use temperature 0 and their default maximum token limits for deterministic evaluation.
- Evaluation Metrics: Evolution Success Count measures problems passing both verification agents within the 20-rollout limit.
- Evaluation Metrics: Certified Solvability Count records evolved problems and canonical solutions independently verified as solvable by an external judge.
- Evaluation Metrics: Agreement Rate measures consistency between the internal solvability verifier and the external judge.Table cells list the externally certified count first and the internally accepted count second.
- Evaluation Metrics: Solve Rate compares solver accuracy on original seed problems and evolved problems through Origin-SR and Evolution-SR.A lower Evolution-SR than Origin-SR is treated as evidence of increased problem complexity.
- Evaluation Metrics: Average Token Consumption uses solver output length as a proxy for difficulty and imputes the maximum token limit for timeouts.Harder problems are expected to require more extensive reasoning chains.
A.2 Additional Result Analysis Details
The framework largely preserves solvability while producing evolved problems that reduce the solve rates of several reasoning models. However, maintaining a valid solution chain remains the dominant bottleneck.
- High Agreement Rates validate the Solvability Verification Agent, with external judges certifying most internally accepted problems as solvable.The passage presents solution-step checking as an effective filter for invalid generations before final external judging.
- DeepSeek-Reasoner evolutions reduce Gemini-3-Flash-Thinking’s solve rate from 56% to 35% and GPT-5.2-High’s from 70% to 64%.
- Gemini-3-Pro-Preview-Thinking and Seed-2.0-Pro reduce GPT-5.2-High’s solve rate to 61%, while DeepSeek-Chat has little effect on this solver.
- Most failures arise from solvability verification rather than difficulty verification, making valid solution chains the dominant bottleneck.
A.3 Additional Human Evaluation Details
The paper’s additional human-evaluation details provide full audit statistics and a complete breakdown of failures after evolution.
- Table 6 reports full human-audit metrics, including rank agreement, score calibration, and the high-level olympiad subset.
- Table 7 reports the counts underlying the compact failure-after-evolution table in the main text.
- The additional analyses cover both human-audit metrics and failure counts after problem evolution.
A.4 Additional Code Ablation Details
Code-enabled evolution often produces problems with concrete computational anchors that support auditing and intermediate verification, while no-code evolution can sometimes yield lower solver accuracy.
- Code-enabled evolution often yields problems with fixed constants, extremal targets, enumerable configurations, or verifiable closed forms.
- These computational anchors make evolved problems easier to audit through pattern testing, invalid-construction elimination, and intermediate-claim verification.
- No-code evolution sometimes produces lower solver accuracy.
A.5 Prompt Templates
The case studies show that problem evolution can replace familiar solution routes with deeper structural, geometric, graph-theoretic, and analytic challenges. Across examples, the adaptations increase the burden of discovery by requiring new abstractions rather than routine extensions.
- A.6 Case Study: Cubic-curve adaptation replaces local algebraic manipulation with geometric constraints involving the inflection point and a periodic sequence.The resulting reasoning requires recognizing consecutive-integer product minimization and modular constraints.
- A.6 Case Study: The sequence adaptation shifts the goal from proving periodicity to proving surjectivity through a hidden generative mechanism.The number 2 acts as a recurring cursor that generates successive larger integers.
- A.6 Case Study: Divisor-gap adaptation requires combining constraints from both ends of the divisor chain instead of relying on a largest-divisor tail strategy.The resulting solution families include n = p(2p −1)^k and n = 2^k · 3.
- A.6 Case Study: The increasing-sequence adaptation replaces local inequality tricks with global recurrence and convexity analysis.Monotonicity forces a minimum integer step size of 2 through a derived second-difference inequality.
- A.6 Case Study: The prime-product graph adaptation lifts a specific number-theoretic puzzle into an abstract theorem about excluding subdivisions of K4.It preserves quadratic reciprocity and descent while shifting the task toward graph reduction and graph-class characterization.
- A.6 Case Study: The card-partition adaptation raises the combinatorial target from a K3-type setting to a rarer K4 structure while adding a mixed sum-or-difference condition.The new condition invalidates the original direct algebraic approach and requires a creative construction.
- A.6 Case Study: The Hilbert-space adaptation replaces an elementary concavity trick with a dimension-independent argument based on negative definite kernels.Its proof uses an integral representation associated with Schoenberg/Levy-Khintchine theory.
- A.6 Case Study: The analytic-mean adaptation transforms a standard inequality task into a higher-level problem centered on the limit-defined quantities A and B.The case study asks for the ratio A/B after defining the analytic mean limit A.