PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning

Xueliang Zhao, Wei Wu, Jian Guan, Zhuocheng Gong, Lingpeng Kong

arXiv:2509.19894v1cs.LGcs.CL

TL;DR

High-quality training problems remain a bottleneck, as human-curated data is costly and existing synthetic corpora can lack difficulty and diversity. PromptCoT 2.0 uses EM to refine rationales that guide prompt synthesis, then applies the synthetic prompts in self-play and supervised fine-tuning. Experiments report state-of-the-art results at 30B scale and strong synthetic-only SFT results at 7B scale.

  • Problem

    Prompt synthesis methods often rely on engineered instructions and yield problems limited in difficulty, expression variation, or domain coverage, while human curation is costly.

    Source: §1 · p. 2

  • Method

    PromptCoT 2.0 uses EM to iteratively refine rationales that guide prompt construction, and applies synthesized prompts in self-play or teacher-distilled SFT.

    Source: §1 · p. 2

  • Results

    PromptCoT 2.0 set new state-of-the-art results at 30B scale across six benchmarks; synthetic-only SFT reached 73.1 on AIME 24, 65.6 on AIME 25, and 53.4 on LiveCodeBench v5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that prompt synthesis can provide harder, more diverse problems for advancing reasoning in open-source LLMs.

    Source: §7 · pp. 12–13

    The exploration of multimodal prompt synthesis, including prompts with images, videos, or tools, is deferred to future work.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language models (LLMs) are evolving from conversational systems into strong reasoners for tasks such as Olympiad mathematics and competitive programming. While scaling parameters and test-time computation has driven progress, a key bottleneck is the lack of high-quality training problems: human-curated datasets are costly and limited, while existing synthetic corpora are often too easy or narrow. PromptCoT 1.0 showed that injecting rationales into prompt synthesis increases problem difficulty. Building on this, we present PromptCoT 2.0, a scalable framework that replaces hand-crafted heuristics with an expectation-maximization (EM) loop, where rationales are iteratively refined to guide prompt construction. This produces problems that are both harder and more diverse than prior corpora. The synthetic prompts support two post-training regimes: (1) Self-Play, where strong models improve autonomously via verifiable feedback without stronger teachers; and (2) Supervised Fine-Tuning (SFT), where weaker models learn from teacher-distilled traces. Extensive experiments demonstrate the effectiveness of this approach. In self-play, applying PromptCoT 2.0 to Qwen3-30B-A3B-Thinking-2507 sets new state-of-the-art results at the 30B scale, with +4.4, +4.8, and +5.3 on AIME 24/25 and HMMT 25, +6.1 and +5.0 on LiveCodeBench v5/v6, and +35 Elo on Codeforces. In SFT, training Qwen2.5-7B-Instruct solely on synthetic prompts boosts accuracy to 73.1 (AIME 24), 65.6 (AIME 25), and 53.4 (LiveCodeBench v5), surpassing models trained on human or hybrid data. Analyses further confirm that PromptCoT 2.0 yields fundamentally harder and distributionally distinct problems. These results establish prompt synthesis as a new axis for scaling reasoning and position PromptCoT 2.0 as a scalable foundation for future open-source models. The implementation is available at https://github.com/inclusionAI/PromptCoT.

1 Introduction

PromptCoT 2.0 addresses limits in scalable prompt synthesis with an EM-based framework that iteratively refines rationales to guide problem construction. Its synthetic prompts support self-play and supervised fine-tuning, with reported gains across reasoning and coding benchmarks.

1 Introduction

  • Existing prompt synthesis often relies on engineered instructions and produces problems limited in difficulty, expression variation, or domain coverage.Prior rationale-driven synthesis improved difficulty but remained human-engineered and confined to mathematics.Source: §1 · p. 2
  • PromptCoT 2.0 uses EM to iteratively refine rationales that guide construction of more challenging and diverse problems across mathematics and programming.The framework is learnable and domain-agnostic, unlike approaches relying on hand-crafted instructions or domain-specific heuristics.Source: §1 · p. 2
  • PromptCoT 2.0 set new state-of-the-art results at 30B scale across six benchmarks, while synthetic-only SFT improved Qwen2.5-7B-Instruct results over human-curated and hybrid datasets.Reported SFT accuracies include 73.1 on AIME 24, 65.6 on AIME 25, and 53.4 on LiveCodeBench v5.Source: §1 · p. 2
  • The authors defer exploration of multimodal prompt synthesis, including prompts incorporating images, videos, or tools.This is identified as future work.Source: §1 · p. 2
  • The paper presents self-play and supervised fine-tuning as complementary ways to post-train models with synthesized prompts.Self-play uses automatically verifiable feedback without stronger teachers; SFT uses teacher-distilled reasoning traces.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries formulate prompt synthesis as generating prompts from underlying concepts. They describe rationales as an intermediate structure that grounds concepts and guides the construction of more robust prompts.

2.1 Problem Formulation: Prompt Synthesis

  • Prompt synthesis models p(x | c) to generate prompts that reflect a specified subset of concepts and aim to enhance LLM reasoning.Here, c is a set of concepts and x is a textual prompt.Source: §2.1 · p. 3

2.2 The PromptCoT Framework

  • PromptCoT factorizes prompt generation through a latent rationale because direct concept-to-prompt mapping can yield underspecified or brittle problem statements.The factorization is p(x | z, c) p(z | c).Source: §2.2 · p. 3
  • Rationales ground concepts, decompose them into explanatory steps, and guide more robust prompts; prior work reports increased mathematical-problem difficulty [Zhao et al., 2025b].The framework's reliance on human-crafted instructions and validation in a single domain motivates broader approaches.Source: §2.2 · p. 4

3 PromptCoT 2.0

PromptCoT 2.0 jointly trains rationale and prompt generation models through an EM procedure. Its variational formulation and implementation alternate rationale refinement with prompt-model updates, using observed concept–prompt pairs and inferred rationales.

3 PromptCoT 2.0

  • PromptCoT 2.0 jointly trains rationale and prompt generation models with EM, alternating posterior-based rationale updates and prompt-model training on inferred triples.The objective maximizes the expected joint log-likelihood over observed concept–prompt pairs.Source: §3 · p. 4

3.1 Variational Perspective

  • Direct marginal-likelihood optimization is infeasible because it sums over rationales, so the method introduces an approximate posterior and an ELBO.The ELBO provides the variational perspective for the EM updates.Source: §3.1 · p. 4
  • In the E-step, rationale updates approach the posterior over rationales, rewarding rationales according to their joint likelihood of fitting concepts and predicting valid prompts.The optimal posterior is proportional to pθ(z, x | c) = pθ(x | z, c) pθ(z | c).Source: §3.1 · p. 4
  • In the M-step, sampled rationales from the current posterior provide the prompt-generation model's training signal for reproducing rationale–prompt pairs.The update maximizes expected joint log-likelihood with the posterior fixed.Source: §3.1 · p. 4
  • The ELBO links rationale rewards to prompt generation, making rationales latent bridges between concepts and prompts in the iterative optimization loop.Rationale generation is refined using rewards from prompt generation, while prompt generation learns from posterior-proposed rationales.Source: §3.1 · p. 4

3.2.1 Cold-start Initialization

  • Implementation has a cold-start phase followed by iterative EM; the seed triples come from open-source math and programming problems with concept and rationale annotations.Four instruction-tuned models generate annotations, and maximum-likelihood estimation warm-starts both models.Source: §3.2 · p. 4, §3.2.1 · p. 5

3.2.2 EM Optimization

  • During practical EM optimization, the rationale model uses reinforcement learning with reward equal to the prompt model's joint log-likelihood, and the prompt model trains on current-posterior rationales.Eight candidate rationales are sampled per concept–prompt pair; the highest-reward rationale is selected for supervised fine-tuning, and the models alternate updates until convergence.Source: §3.2.2 · p. 5

4 LLM Post-Training with Synthesized Prompts

The paper uses synthesized prompts for two post-training regimes: self-play for already capable models and teacher-distilled supervised fine-tuning for weaker ones. Self-play relies on automatically verifiable feedback, while SFT trains students on complete reasoning traces.

Self-Play

  • Self-play lets already strong models improve using automatically verifiable feedback rather than external teacher traces.The paper frames this regime as addressing the lack of even stronger teachers for distillation.Source: §4 · p. 5
  • The self-play framework iteratively generates solutions, receives feedback, and updates the model without external teachers or human annotations.It is compatible with PPO, GRPO, and DPO.Source: §4 · p. 5

Supervised Fine-Tuning

  • Weaker models use supervised fine-tuning on synthesized problems paired with complete reasoning traces generated by a strong teacher.This provides trajectory-level supervision when self-play is not applicable to the model.Source: §4 · p. 6

5 Experiments

Experiments evaluate PromptCoT 2.0 in self-play and SFT across mathematics and programming benchmarks, then examine its components, scaling behavior, and generated-problem properties. The reported findings cover benchmark performance, data efficiency, distribution, difficulty, and EM optimization.

5.1 Benchmarks; 5.2 Evaluation Metrics; 5.3 Baselines

  • The evaluation spans six mathematics and coding benchmarks, using averaged pass@1 for mathematics, pass@1 for coding, and Elo for Codeforces.Self-play comparisons use open-source corpora including OpenCodeReasoning, OpenMathReasoning, OpenThoughts3, OpenR1, and PromptCoT 1.0.Source: §5.1 · p. 6, §5.2 · p. 6, §5.3 · p. 6

5.4.1 Prompt Synthesis; 5.4.2 Post-training with Synthesized Prompts

  • Prompt synthesis is warm-started with annotated Codeforces and AoPS problems, then rationale and prompt models are refined through EM and used in two post-training regimes.Self-play uses verifiable answers or tests and filters out problems solved in at least half of eight attempts; SFT uses GPT-OSS-120B traces on 4,766,890 synthesized prompts.Source: §5.4.1 · p. 7, §5.4.2 · p. 8

5.5 Main Results

  • PromptCoT 2.0 improves strong Qwen3 baselines across mathematical and programming tasks, with its advantage especially pronounced on HMMT.In the 4B setting, it uses about 90% of OpenMathReasoning’s math prompts and 10% of OpenCodeReasoning’s code prompts; the 30B setting uses less data while achieving stronger gains.Source: Table 1 · p. 7, Table 2 · p. 7, §5.5 · p. 8
  • With 100% synthesized prompts, PromptCoT 2.0 delivers strong performance across all SFT benchmarks and produces shorter reasoning traces.The authors note that shorter traces may largely stem from the teacher model and could reduce inference computation.Source: §5.5 · pp. 8–9, Table 3 · p. 8

5.6 Ablation Study

  • The full PromptCoT 2.0 outperforms both ablated variants across all six benchmarks, with EM removal causing the largest drops, especially on HMMT and Codeforces.Removing cold-start causes a moderate but consistent decline; the authors associate high-quality initial teacher annotations with stabilizing subsequent training.Source: §5.6 · p. 9, Table 4 · p. 9

5.7 Scaling Properties

  • As training data grows, PromptCoT 2.0 improves consistently in self-play and achieves higher peak performance and stronger data efficiency in SFT.Self-play covers 0–48k examples, while SFT covers 0–4.8M; curation-based self-play gains largely saturate beyond 24k examples.Source: Figure 2 · p. 9, §5.7 · p. 10

5.8 Distributional Analysis; 5.9 Difficulty Analysis

  • PromptCoT 2.0 problems occupy a distinct embedding-space region and are the most challenging in the evaluated datasets, without explicit difficulty filtering.The reported measures are 18.5% accuracy for Qwen2.5-72B-Instruct and 37.4k reasoning tokens for GPT-OSS-120B.Source: §5.8 · p. 10, Figure 3 · p. 10, §5.9 · p. 11

5.10 EM Optimization Analysis

  • With the E-step, likelihood improves faster and reaches a lower terminal NLL than without it; rationales also lower NLL before optimization, and iterative gains widen.The analysis compares the full EM procedure with fixed sampled rationales and rationale-conditioned and unconditioned initializations.Source: §5.10 · p. 11

6 Related Works

Prior work spans general-purpose, domain-specific, and agent-oriented prompt synthesis, while PromptCoT 2.0 frames rationale–prompt co-generation as an iterative alternative to heuristic or single-pass pipelines. Related work also situates the method among self-evolutionary approaches and reports NLL trajectories for EM variants.

Prompt Synthesis

  • Prior prompt synthesis progressed from general-purpose seed-instruction methods (Wang et al., 2023; Xu et al., 2024) to domain-specific math and coding problems and task-oriented prompts.Zhao et al. (2025b) incorporated a thinking process into math problem generation to increase difficulty.Source: §6 · p. 11
  • Figure 4 compares NLL trajectories for training with versus without the E-step and initialization with versus without rationales.Shaded confidence bands are reported for optimized variants.Source: Figure 4 · p. 12
  • PromptCoT 2.0 frames rationale–prompt co-generation as an EM procedure, contrasting its iterative reward-guided rationale refinement and conditioned prompt updates with heuristic or single-pass generation.The stated goal is more faithful and difficult problems in mathematics and programming.Source: §6 · p. 12

Self-Evolution

  • PromptCoT 2.0’s self-play resembles earlier self-evolution work but avoids self-judgment and joint optimization of synthesis and model post-training.The passage contrasts it with Yuan et al. (2024) and Zhao et al. (2025a), respectively.Source: §6 · p. 12

7 Conclusions and Future Work

PromptCoT 2.0 uses expectation–maximization to refine rationales and synthesize harder, more diverse mathematics and programming problems. The paper reports gains from self-play and SFT, and proposes extending the framework to multimodal settings, richer verification, and agentic intelligence.

7 Conclusions and Future Work

  • The framework formulates concept–rationale–prompt generation as expectation–maximization, iteratively refining rationales to create more difficult and diverse problems across mathematics and programming.The paper presents these problems as addressing the shortage of high-quality training problems for reasoning-focused LLMs.Source: §7 · p. 12
  • PromptCoT 2.0’s problems support self-play and SFT, with reported state-of-the-art results at 30B scale and competitive performance from 7B models trained purely on synthetic prompts.The 7B models are reported as competitive with models trained on human-curated or hybrid datasets.Source: §7 · p. 13
  • Future work includes multimodal extensions, richer verification signals for broader self-play, and exploration of applications to agentic intelligence.The authors position prompt synthesis as an axis for scaling reasoning beyond model size or computation.Source: §7 · p. 13

A.1 Proof of the ELBO (Eq. 2)

The proof derives the evidence lower bound from the marginal likelihood using Jensen’s inequality, identifying the result with Eq. 2.

A.1 Proof of the ELBO (Eq. 2)

  • The appendix provides a detailed derivation of the evidence lower bound.Source: §A.1 · p. 16
  • The ELBO derivation starts from the marginal likelihood in Eq. 1 and applies Jensen’s inequality.Source: §A.1 · p. 16
  • The proof identifies the resulting bound with Eq. 2.Source: §A.1 · p. 16

A.2 Proof of the Optimal Posterior (Eq. 3)

The proof establishes the optimal approximate posterior by rewriting Eq. 2, using non-negativity of the KL divergence, and matching the resulting condition to Eq. 3.

A.2 Proof of the Optimal Posterior (Eq. 3)

  • The optimal approximate posterior is established by showing the condition under which the ELBO is maximized.Source: §A.2 · p. 16
  • The proof rewrites Eq. 2 as the ELBO plus a KL divergence, then uses the divergence’s non-negativity to identify the maximizing condition.The ELBO is maximized when the KL term is zero.Source: §A.2 · p. 16
  • The resulting posterior expression is stated to match Eq. 3.Source: §A.2 · p. 16

B Additional Results on Ring-Lite

PromptCoT 2.0 improves Ring-Lite across mathematics and programming benchmarks in self-play. The accompanying instructions specify how to extract concepts and generate problem-design rationales.

B Additional Results on Ring-Lite

C Instruction for Concept Extraction

  • Concept extraction targets precise, measurable core-curriculum knowledge points that are essential building blocks, not problem-specific techniques.The prompt asks the model to identify foundational concepts underlying each seed problem.Source: C Instruction for Concept Extraction · p. 17

D Instruction for Rationale Generation

  • Rationale generation uses the problem, concepts, and difficulty level to elicit a natural, detailed design process that another teacher could reproduce.The requested process should explain how the concepts lead to the problem and include the key design decisions.Source: D Instruction for Rationale Generation · p. 17

E Case Study

The case studies cover generated problems in number theory, 3D geometry, and competitive programming.

E Case Study

  • Figure 5 presents a rationale-and-problem-generation case study for a number theory AIME-level problem.Source: Figure 5 · p. 18
  • Figure 6 presents a rationale-and-problem-generation case study for a 3D geometry AIME-level problem.Source: Figure 6 · p. 19
  • Figure 7 presents a rationale-and-problem-generation case study for a Codeforces-level programming problem.Source: Figure 7 · p. 20
Loading 2509.19894v1…