PaperlayerView exact revision on arXiv

Source-linked AI summary

Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolution

Shaobo Wang, Zhengbo Jiao, Zifan Zhang, Yilang Peng, Xu Ze, Boyu Yang, Wei Wang, Hu Wei, Linfeng Zhang

arXiv:2509.24726v1cs.CL

TL;DR

Large-scale human-curated reasoning data is costly to scale, and existing synthesis approaches are described as limited in quality control and adaptation. Socratic-Zero co-evolves a Solver, Teacher, and Generator to build an adaptive curriculum. The paper reports gains across mathematical reasoning benchmarks and downstream effectiveness above the listed commercial-model results.

  • Problem

    Reasoning advances rely on costly, difficult-to-scale curated trajectories, while static curricula and existing synthesis methods are limited in adaptation and quality control.

    Source: §1 · p. 2

  • Method

    Socratic-Zero co-evolves a Solver that learns from preference feedback, a Teacher that refines problems based on Solver failures, and a Generator that distills the Teacher’s strategy.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    +20.2 points average improvement across seven mathematical reasoning benchmarks is reported for Socratic-Solver-8B, and Socratic-Generator-32B achieves 37.72% downstream training effectiveness, above each listed commercial model.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The conclusion presents co-evolution from minimal seed data as a viable path to reasoning systems in resource-constrained scenarios, with reported generalization across problem types and difficulty levels.

    Source: §5 · p. 9

    The complex agent dynamics currently lack a formal convergence analysis.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent breakthroughs in large language models (LLMs) on reasoning tasks rely heavily on massive, high-quality datasets-typically human-annotated and thus difficult to scale. While data synthesis or distillation offers a promising alternative, existing methods struggle with inconsistent data quality and an inability to dynamically adapt to the evolving capabilities of the model, leading to suboptimal training signals. To address these limitations, we introduce Socratic-Zero, a fully autonomous framework that generates high-quality training data from minimal seed examples through the co-evolution of three agents: the Teacher, the Solver, and the Generator. The Solver continuously refines its reasoning by learning from preference feedback on both successful and failed trajectories; the Teacher adaptively crafts increasingly challenging questions based on the Solver's weaknesses; and the Generator distills the Teacher's question-design strategy to enable scalable, high-fidelity curriculum generation. This closed-loop system produces a self-improving curriculum-requiring no pre-existing tasks or labels. Remarkably, starting from only 100 seed questions, our Socratic-Solver-8B achieves an average gain of +20.2 percentage points over prior data synthesis methods across seven mathematical reasoning benchmarks (AMC23, AIME24-25, Olympiad, MATH-500, Minerva, and GSM8K), with consistent gains on both Qwen3 and GLM4 series models. Even more surprisingly, synthetic data from Socratic-Generator-32B enables student LLMs to achieve superior performance compared to other state-of-the-art (SOTA) commercial LLMs on these benchmarks, including Qwen3-235B-A22B, DeepSeek-V3.1-671B, GPT-5, Gemini-2.5-Pro, Grok-4, and Claude-4.1-Opus.

1 INTRODUCTION

The introduction frames scalable mathematical reasoning as constrained by reliance on costly human-curated data and static curricula. Socratic-Zero responds with a three-agent co-evolutionary framework and reports substantial benchmark gains.

1 INTRODUCTION

  • Reasoning advances depend on massive, meticulously curated trajectories, while state-of-the-art systems also rely on millions of human-annotated pairs and hand-designed curricula.The introduction describes this dependence as costly and fundamentally unscalable.Source: §1 · p. 2
  • Static datasets and handcrafted curricula cannot adapt to changing model capabilities or target weaknesses, while recent synthesis and iterative-training methods still rely on external supervision and lack effective quality control.These limitations motivate an autonomous, adaptive curriculum-generation approach.Source: §1 · p. 2
  • Socratic-Zero co-evolves a Solver, a Teacher, and a Generator in a continuous loop, with the Generator distilling the Teacher’s problem-generation strategy.The Solver learns through preference feedback, while the Teacher generates challenging problems based on Solver weaknesses.Source: §1 · p. 2
  • +20.2 points average improvement across seven mathematical reasoning benchmarks is reported for Socratic-Solver-8B, while Socratic-Generator-32B achieves 37.72% downstream training effectiveness.The reported 37.72% exceeds the listed commercial-model results, including Claude-4.1-Opus at 37.63%.Source: §1 · p. 3

2 RELATED WORK

Prior work explores synthetic data, distillation, and preference-based self-evolution, but the described approaches remain limited in quality control, adaptation, or unified feedback generation and validation.

2 RELATED WORK

  • Synthetic-data methods progressed from prompt-engineered generation to deficiency-aware generation and autonomous self-play, but lack effective quality control.The paper says repeated use of low-value samples can severely impact effectiveness.Source: §2 · p. 3
  • Distillation methods transfer teacher reasoning or decision strategies, while GKD uses teacher feedback on student sequences (Agarwal et al., 2024); the described approaches rely on static data or passive feedback.The paper also characterizes recent data-centric methods as lacking effective quality control and adaptive curriculum generation (Wang et al., 2025a).Source: §2 · p. 3
  • Preference methods include RLHF, direct preference optimization and RWSFT, and self-correction or autonomous feedback approaches, but lack unified co-evolving feedback generation and validation.The paper contrasts RLHF’s complex and unstable process with the efficiency and stability attributed to direct preference methods.Source: §2 · p. 3

3 METHODOLOGY

Socratic-Zero couples Solver learning with Teacher-guided problem refinement and Generator distillation, so the curriculum can change with the Solver’s capabilities. The Solver learns from preference pairs, and the Generator is trained to produce problems with a target level of difficulty.

3.1 THE SOCRATIC-ZERO FRAMEWORK

  • The Solver, fixed Teacher, and Generator form a co-evolutionary loop in which Solver failures guide Teacher refinements and the Generator distills that strategy into a scalable, changing curriculum.The Teacher verifies solutions and refines problems; the Generator maps a problem and failed solution to a new problem.Source: §3.1 · pp. 3–4
  • Teacher-refined problem-solution pairs augment the curriculum based on Solver mistakes, allowing the problem set to evolve with its capabilities.The framework describes collecting Solver failures and using them to create new instructive pairs.Source: §3.1 · p. 4

3.2 SOLVER TRAINING VIA ONLINE PREFERENCE OPTIMIZATION

  • For online preference learning, the Solver generates multiple attempts that are divided into correct and incorrect sets, with the curriculum’s reference solution serving as the winning example if none are correct.The resulting preference pairs are used for DPO updates against a frozen reference policy.Source: §3.2 · p. 5

3.3 GENERATOR TRAINING VIA OFFLINE VALUE-WEIGHTED DISTILLATION

  • The Generator’s utility scores a new problem by the Solver’s success rate across attempts, using a Gaussian centered on target rate µ = 0.5 to favor frontier-level difficulty.The parameter σ controls tolerance for deviation from the target success rate.Source: §3.3 · p. 5
  • Weighted supervised fine-tuning trains the Generator on Solver failures and Teacher-refined problems to maximize utility-weighted likelihood and mimic high-utility problem generation.This is intended to let the Generator internalize the Teacher’s curriculum-design strategy.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate Solver performance across mathematical and general-reasoning benchmarks, and assess Generator output through problem validity and downstream student utility. They also test the effects of SFT initialization and reward-function choices.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses seven mathematical benchmarks for Solver accuracy and three general-reasoning benchmarks to assess transfer beyond mathematics.Solver accuracy is Mean@32, based on 32 generated solutions per test question.Source: §4.1 · p. 6

4.2 SOLVER RESULTS

  • 56.1% average accuracy: Socratic outperforms Static Augmentation by 15.4 points and LLM2LLM by 15.2 points across seven mathematical benchmarks.AIME-24 improves by 19.1 points and AIME-25 by 16.5 points.Source: §4.2 · p. 6, Table 1 · p. 7
  • GLM4-9B-base reaches 52.3% average accuracy (+17.1 points), and Qwen3-14B-base reaches 60.3% (+17.3 points); general-reasoning benchmarks improve by 6.02 points on average.The cross-model changes are relative to each model’s base/SFT reference, while Table 3 reports changes relative to zero-shot Qwen3-8B-base.Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7

4.3 GENERATOR RESULTS

  • The Generator evaluation compares intrinsic problem quality and downstream training effectiveness against its base model and larger commercial models.The downstream evaluation fine-tunes DeepSeek-R1-Distill-Llama-8B on generated question-answer pairs.Source: §4.3 · p. 7, §4.3.1 · p. 8

4.3.1 EVALUATION PROTOCOL

  • Generators receive seed problems to create variants; Qwen3-235B checks solvability under token and time limits, and valid question-answer pairs train a student model.The protocol describes 1,000 seeds and five variants per seed, yielding 3,000 generated problems per model.Source: §4.3.1 · p. 8

4.3.2 PROBLEM QUALITY ASSESSMENT

  • 95.6% of Socratic-Generator-32B’s generated problems are solvable by Qwen3-235B-A22B-Instruct-2507, improving on its base model and rivaling larger models.The passage names GPT5-0807-global and Gemini-2.5-Pro-06-17 among the larger-model comparisons.Source: §4.3.2 · p. 8

4.3.3 DOWNSTREAM EFFECTIVENESS

  • 37.72% student accuracy: data from Socratic-Generator-32B exceeds its Teacher’s result by 0.59 points despite the Generator being over 20 times smaller.The downstream comparison evaluates the fine-tuned student model.Source: §4.3.3 · p. 8, Table 5 · p. 8

4.4 ABLATION STUDIES

  • With SFT, Stage 3 reaches 28.02% versus 11.98% without SFT; the Gaussian reward N(µ = 0.5, σ = 0.2) scores 35.72% with 89.9% validity.The no-SFT model gains 2.34 points over its 9.64% base score; the Gaussian reward performs 0.20–0.30 points better than tested linear alternatives.Source: §4.4 · p. 9, Table 6 · p. 9

5 CONCLUSION AND FUTURE WORK

The conclusion presents Socratic-Zero as a multi-agent framework for bootstrapping mathematical reasoning from minimal seed data and identifies missing convergence analysis as a limitation. The appendix excerpts specify prompt structures for Solver reasoning, Teacher grading and generation, and static augmentation.

5 CONCLUSION AND FUTURE WORK

  • Socratic-Zero uses Solver, Teacher, and Generator agents to bootstrap mathematical reasoning from minimal seed data; its complex dynamics lack formal convergence analysis.The authors identify broader domains, including scientific discovery and real-world decision making, as future directions.Source: §5 · p. 9

A.1 SOLVER REASONING PROMPT

  • The Solver prompt structures solutions as understanding, planning, execution, verification, and an exact answer.Source: §A.1 · p. 13

A.2 TEACHER EVALUATION PROMPT

  • The Teacher grading prompt prioritizes final-answer correctness, reference use when available, and concise error analysis, with results formatted as JSON.Source: §A.2 · p. 13

A.3 TEACHER GENERATION PROMPT

  • The Teacher generation prompt uses student error analysis to produce an enhanced problem, detailed solution, and answer while preserving solvability and correctness.Source: §A.3 · p. 13

A.4 STATIC AUGMENTATION BASELINE PROMPTS

  • The static augmentation baseline separately prompts upward and downward difficulty changes, each through identifying elements, planning modifications, rewriting, and review.Source: §A.4 · p. 13

B IMPLEMENTATION DETAILS

The implementation adapts curriculum difficulty to the Solver’s performance, generating targeted variants from problems it can solve intermittently or reliably. It also describes the training infrastructure, Teacher deployment, and example problem enhancements.

B IMPLEMENTATION DETAILS

C.1 PROBLEM CATEGORIZATION BY SOLVER PERFORMANCE

  • The Solver’s success rate across k attempts, judged by the Teacher’s verification function, partitions problems into mastered, learning, and too-difficult zones.Mastered problems have a 100% success rate, learning-zone problems have a rate between 0 and 1, and too-difficult problems have a 0% rate.Source: §C.1 · pp. 14–15

C.2 ZONE-ADAPTIVE PROBLEM GENERATION

  • The Teacher generates targeted variants from learning-zone failures and more complex variants from mastered-zone successes, while excluding too-difficult problems.Failure-based variants target the demonstrated reasoning error; mastered-problem variants extend established competence.Source: §C.2 · p. 15

C.3 DYNAMIC RECATEGORIZATION OF PROBLEMS

  • Problems are periodically re-evaluated and recategorized as the Solver changes, allowing formerly too-difficult problems to re-enter the active learning and generation pool.The passage describes movement from too-difficult to learning as capability improves and from learning to mastered as skills consolidate.Source: §C.3 · p. 15

D TEACHER MODEL INFRASTRUCTURE

  • The Teacher, Qwen3-235B-A22B-Instruct-2507, is distributed across 16 AMD MI308X GPUs with 3,072 GB aggregate memory for curriculum generation and evaluation.The inference service also uses load balancing, failover, connection pooling, and retry policies.Source: D TEACHER MODEL INFRASTRUCTURE · pp. 15–16

E.1 EXAMPLE 1: RATIONAL INEQUALITY ENHANCEMENT

  • The rational-inequality example adds factored and squared denominators, requiring domain analysis, common denominators, and more complex algebra while targeting common solving errors.The enhanced problem retains the core mathematical concepts of the original.Source: E TEACHER-GENERATED PROBLEM ENHANCEMENT · p. 16, §E.1 · p. 16

E.2 EXAMPLE 2: NUMBER THEORY ENHANCEMENT

  • The number-theory example increases the numerical complexity of a GCD problem and requires recognizing the relation between consecutive factorials.The enhancement targets errors in modular arithmetic applications.Source: §E.2 · p. 16

F SEED SELECTION PROTOCOL

The seed-selection protocol aims to provide appropriate difficulty, mathematical coverage, diversity, and verified problem quality. Seeds are calibrated to the base model and screened to avoid overlap with evaluation benchmarks.

F SEED SELECTION PROTOCOL

  • Seeds come from MATH Levels 2–4, and candidates are retained only when the base model’s success rate across eight attempts is 10–70%.Level 1 problems are excluded as too easy and Level 5 problems as too difficult for the base models.Source: F SEED SELECTION PROTOCOL · pp. 16–17
  • The 100 seed problems cover all seven MATH subject areas, with balanced representation intended to let curriculum evolution target weaknesses across domains.Table 8 is identified as showing the seed distribution across mathematical domains.Source: F SEED SELECTION PROTOCOL · p. 17, Table 8 · p. 17
  • Within each subject area, problems are selected across clusters of solution-approach similarity and varied mathematical tools to promote methodological diversity.The selection process uses embeddings of problem statements to cluster by solution approach.Source: F SEED SELECTION PROTOCOL · p. 17
  • Candidate problems undergo clarity checks and Teacher verification of reference solutions through multiple independent attempts.The passage also specifies clear learning objectives and excludes trick questions or overly specialized knowledge.Source: F SEED SELECTION PROTOCOL · p. 17
  • Seed problems are excluded from evaluation benchmarks to prevent data leakage, and the selected initial curriculum is intended to support later co-evolution.The selection process is described as providing a foundation while maintaining diversity for curriculum expansion.Source: F SEED SELECTION PROTOCOL · p. 17

G EVALUATION PROTOCOL DETAILS

The evaluation protocol measures Mean@32 using independently sampled Solver attempts and combined rule-based and semantic verification. The accompanying pseudocode outlines Solver and Generator updates, while the quality-control mechanism checks problems and removes those that fail verification.

G EVALUATION PROTOCOL DETAILS

  • Mean@32 is the average accuracy across 32 independently sampled zero-shot solutions per test problem, assessed with rule-based answer extraction and semantic validation.The protocol uses temperature 0.7 for solution sampling and a dual-verification approach.Source: G EVALUATION PROTOCOL DETAILS · pp. 17–18
  • MathRule extracts and standardizes final numerical answers, while the Teacher judge checks numerical correctness and reasoning validity when extraction is ambiguous or insufficient.MathRule is described as achieving high precision and robustness to answer-format variation.Source: G EVALUATION PROTOCOL DETAILS · p. 18

H PSEUDO CODE OF SOCRATIC-ZERO

  • The pseudocode collects Teacher-verified winning and losing Solver attempts into preference pairs, then uses failures to generate problems and train the Generator with weighted examples.The curriculum is augmented with generated problem-solution pairs for the next iteration.Source: H PSEUDO CODE OF SOCRATIC-ZERO · p. 19

I PROBLEM QUALITY CONTROL MECHANISM

  • When all eight attempts fail, the Teacher re-solves the problem and checks its reference solution, clarity, consistency, and uniqueness.Problems that fail self-verification are excluded from subsequent Solver training and Generator learning.Source: I PROBLEM QUALITY CONTROL MECHANISM · pp. 19–20
  • MathRule–Teacher disagreements and anomalous Solver performance trigger additional review or exclusion to monitor curriculum quality.The feedback-driven monitoring is described as adapting to emerging issues and maintaining curriculum integrity.Source: I PROBLEM QUALITY CONTROL MECHANISM · p. 20

J CURRICULUM STABILITY AND DIVERSITY ANALYSIS

The curriculum analysis reports Solver reward changes across training stages and stable Generator rewards. It also describes seed diversity and stochastic sampling as mechanisms used to maintain variety during curriculum evolution.

J CURRICULUM STABILITY AND DIVERSITY ANALYSIS

  • The curriculum analysis examines both difficulty progression and preservation of problem diversity.The section also introduces three diversity mechanisms, without specifying them in the supplied passage.Source: J CURRICULUM STABILITY AND DIVERSITY… · pp. 20–21
  • 52.1% to 48.7% to 50.1%: Solver mean reward falls from Stage 1 to Stage 2, then recovers in Stage 3 as curriculum difficulty initially rises faster than Solver capability.Table 9 tracks mean reward, identified as correctness rate, across training stages.Source: J CURRICULUM STABILITY AND DIVERSITY… · p. 20
  • Around 50% of Generator problems remain high-reward, with only a 1.3% fluctuation range, alongside a reward function with µ = 0.5 and σ = 0.2.Table 10 is identified as the Generator reward-distribution analysis.Source: J CURRICULUM STABILITY AND DIVERSITY… · pp. 20–21
  • The 100 seeds span seven MATH subjects and difficulty levels 2–4, while temperature 0.8–0.9 sampling is used for Solver trajectories, Teacher error analysis, and problem generation.The passage associates multi-domain seeds and stochastic sampling with diverse failure patterns and varied problem formulations.Source: J CURRICULUM STABILITY AND DIVERSITY… · p. 21

K GENERALIZABILITY OF PROBLEM GENERATION CAPABILITIES

Socratic-Zero’s problem-generation capabilities may extend beyond mathematical reasoning, but transfer would require adapting evaluation and generation to each domain.

K GENERALIZABILITY OF PROBLEM GENERATION CAPABILITIES

  • The value function and curriculum evolution mechanisms are described as domain-agnostic in principle, suggesting possible applicability beyond mathematics.The passage names logical reasoning, scientific problem-solving, and creative tasks as possible domains.Source: K GENERALIZABILITY OF PROBLEM… · p. 21
  • Difficulty calibration, error-pattern recognition, and targeting are identified as Generator skills that may generalize across reasoning domains.The passage gives approximately 50% success as an appropriately challenging level and describes a Gaussian reward with µ = 0.5 and σ = 0.2 as a transferable calibration framework.Source: K GENERALIZABILITY OF PROBLEM… · p. 21
  • 37.72% downstream utility versus 37.13% from Qwen3-235B-A22B is reported for the Generator, alongside a suggestion that domain-specific training may need fewer computational resources.The passage presents this as evidence that strategic specialization can outperform raw parameter scaling.Source: K GENERALIZABILITY OF PROBLEM… · p. 21
  • Transfer beyond mathematics would require adapting the Teacher’s evaluation capabilities and problem-generation templates.Mathematics uses dual verification (MathRule plus an LLM judge), which the passage reports agrees with human experts 94.2% of the time; other domains may need more nuanced evaluation.Source: K GENERALIZABILITY OF PROBLEM… · p. 21
  • Future work could test whether a mathematics-trained Generator creates challenging physics or computer-science problems through few-shot adaptation or domain-specific fine-tuning.The passage proposes leveraging the framework’s value-learning mechanisms.Source: K GENERALIZABILITY OF PROBLEM… · p. 21

L FRAMEWORK SCALABILITY AND EXTENSIBILITY

The section describes Socratic-Zero’s modular scaling, performance across model architectures, and curriculum growth from a small seed set, while outlining further extensions.

L FRAMEWORK SCALABILITY AND EXTENSIBILITY

  • The separate Solver, Teacher, and Generator roles enable independent scaling and optimization of each component.The passage refers to deployments across different computational configurations.Source: L FRAMEWORK SCALABILITY AND… · p. 22
  • Qwen3-8B achieves 56.1% average accuracy (+20.2 points), with similar gains reported for GLM4-9B and Qwen3-14B.The passage presents this cross-model validation as evidence of consistency across architectures.Source: L FRAMEWORK SCALABILITY AND… · p. 22
  • Starting from 100 seed problems, the system generates thousands of problems; the Generator’s validity rate is 95.6% versus 89.1% for base Qwen3-32B.The passage says this scales curriculum generation without proportional increases in seed-data requirements.Source: L FRAMEWORK SCALABILITY AND… · p. 22
  • The authors propose extending the mathematical focus to multiple reasoning domains, potentially using specialized Teachers with shared co-evolutionary dynamics.The passage points to balanced seed coverage across seven mathematical domains and stable reward distributions across training rounds.Source: L FRAMEWORK SCALABILITY AND… · p. 22
  • The authors also propose hierarchical scaling with Solver-Generator pairs at different difficulty levels or specialization areas, coordinated by higher-level mechanisms.The passage links this proposal to oscillatory convergence patterns and possible synchronization points.Source: L FRAMEWORK SCALABILITY AND… · p. 22

M CONVERGENCE AND THEORETICAL FOUNDATIONS

The section reports oscillatory Solver performance and stable Generator rewards, and identifies convergence conditions and guarantees as open theoretical questions.

M CONVERGENCE AND THEORETICAL FOUNDATIONS

  • Solver performance falls from R1 (60.12%) to R4 (48.7%) before recovering to 50.1% in R5, a pattern the authors interpret as bounded oscillations around dynamic equilibria.The passage attributes the initial decline to Teacher difficulty increasing faster than Solver capability, followed by Solver adaptation.Source: M CONVERGENCE AND THEORETICAL… · p. 22
  • High-reward Generator problems stay around 50%, fluctuating within a 1.3% range across training rounds.The authors connect this stability to learning the target difficulty zone set by a Gaussian reward with µ = 0.5 and σ = 0.2.Source: M CONVERGENCE AND THEORETICAL… · p. 22
  • Similar improvement patterns across Qwen3-8B, GLM4-9B, and Qwen3-14B are presented as evidence that convergence behavior is not architecture-specific.The passage characterizes these observations as evidence about co-evolutionary dynamics.Source: M CONVERGENCE AND THEORETICAL… · p. 22
  • Future theoretical work should investigate when the system converges stably rather than chaotically and whether bounds can be established on oscillation amplitude.The authors also identify convergence guarantees as a basis for more principled hyperparameter selection and confidence bounds for deployment.Source: M CONVERGENCE AND THEORETICAL… · pp. 22–23
Loading 2509.24726v1…