PaperlayerView exact revision on arXiv

Source-linked AI summary

CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks

Ping Yu, Jack Lanchantin, Tianlu Wang, Weizhe Yuan, Olga Golovneva, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, Jing Xu

arXiv:2507.23751v2cs.AIcs.CL

TL;DR

High-quality synthetic training data is difficult to obtain, motivating methods that improve both generation and selection. CoT-Self-Instruct reasons over seed tasks to create similarly complex examples, filters them automatically, and achieves strong results across reasoning and instruction-following benchmarks.

  • Problem

    High-quality synthetic data for reasoning and non-reasoning tasks remains insufficiently understood despite the cost, scarcity, privacy concerns, biases, and errors associated with human data collection.

    Source: §1 · p. 1

  • Method

    CoT-Self-Instruct uses Chain-of-Thought reasoning to generate synthetic instructions from seed examples, then applies Answer-Consistency or RIP filtering before training.

    Source: §1 · p. 1, §3 · p. 3, §3.1 · p. 3

  • Results

    CoT-Self-Instruct achieves the best results among tested methods, outperforming Self-Instruct and existing or human datasets across reasoning and general instruction-following benchmarks.

    Source: §1 · p. 2, §5.1 · p. 8, §5.2 · p. 9

  • Takeaways & Limitations

    Reasoning-based generation combined with automatic curation produces high-quality synthetic instructions for both verifiable reasoning and non-verifiable instruction-following training.

    Source: §1 · p. 1, §6 · p. 9

Abstract

from arXiv · show

We propose CoT-Self-Instruct, a synthetic data generation method that instructs LLMs to first reason and plan via Chain-of-Thought (CoT) based on given seed tasks, and then generate a new synthetic example of similar quality and complexity. This is followed by a filtering step to select high-quality data using automatic metrics, which are then used for LLM training. In verifiable reasoning, our synthetic data significantly outperforms existing training datasets, such as s1k and OpenMathReasoning, when evaluated on MATH500, AMC23, AIME24, and GPQA-Diamond. For non-verifiable instruction-following tasks, our method surpasses the performance of both human and standard Self-Instruct training data on the AlpacaEval 2.0 and Arena-Hard benchmarks.

1 INTRODUCTION

CoT-Self-Instruct addresses the difficulty of obtaining high-quality synthetic data by combining reasoning-based generation with automatic quality filtering. It improves training data for both verifiable reasoning and non-verifiable instruction-following tasks.

  • High-quality synthetic data remains difficult to obtain because human collection is costly, scarce, privacy-sensitive, and prone to bias and errors.Source: §1 · p. 1
  • Existing Self-Instruct variants improve complexity, diversity, scale, or self-improvement, but high-quality generation for reasoning and non-reasoning tasks remains insufficiently understood.Source: §1 · p. 1
  • CoT-Self-Instruct reasons over seed examples to generate high-quality synthetic instructions, then self-filters the data to retain the strongest examples.For verifiable tasks, Answer-Consistency compares generated target answers with majority-vote solutions; for non-verifiable tasks, RIP scores sampled responses.Source: §1 · p. 1, Figure 1 · p. 2
  • For reasoning tasks, CoT-Self-Instruct outperforms Self-Instruct and curated datasets including s1k and OpenMathReasoning on MATH500, AMC23, AIME24, and GPQA-Diamond.Source: §1 · p. 2
  • For non-verifiable tasks, it outperforms WildChat human data and Self-Instruct data on AlpacaEval 2 and ArenaHard, achieving the best results among tested methods.Source: §1 · p. 2

2 RELATED WORK

Prior work established synthetic instruction generation and increasingly sophisticated data selection, but the quality of synthetic data remains central to effective post-training. Recent curation methods favor smaller, higher-quality datasets and use model-based quality signals.

  • Self-Instruct uses seed examples to prompt LLMs to generate additional instruction-following demonstrations, while later variants target complexity, diversity, scale, and self-improvement.Source: §2 · p. 2
  • Data selection is critical for post-training because smaller, higher-quality instruction sets can improve instruction following and reasoning more effectively than larger uncurated sets.Source: §2 · pp. 2–3
  • Curation methods include deduplication, clustering, quality classifiers, heuristics, and LLM-based judgments of instruction complexity, diversity, and quality.Source: §2 · p. 3
  • RIP filters low-quality instructions using reward-model scores on generated responses, while Self-Consistency filtering rejects verifiable tasks whose solution votes show insufficient agreement.Source: §2 · p. 3

3 CHAIN-OF-THOUGHT(COT)-SELF-INSTRUCT

CoT-Self-Instruct has three stages: reason over seed instructions to create synthetic examples, curate those examples with task-appropriate filters, and train models on the retained data. The generation and curation procedures differ for verifiable and open-ended tasks.

3 CHAIN-OF-THOUGHT(COT)-SELF-INSTRUCT

  • 3 CHAIN-OF-THOUGHT(COT)-SELF-INSTRUCT: The pipeline uses a small amount of high-quality human seed data to generate, curate, and train on synthetic instructions.Source: §3 · p. 3

3.1 SYNTHETIC INSTRUCTION CREATION VIA COT

  • 3.1 SYNTHETIC INSTRUCTION CREATION VIA COT: The LLM analyzes seed domains, complexity, and purpose, then plans a new self-contained instruction of similar quality and complexity.Source: §3.1 · p. 3
  • 3.1 SYNTHETIC INSTRUCTION CREATION VIA COT: For verifiable tasks, reasoning generates both an instruction and its target answer, with strict formatting separating them.Source: §3.1 · p. 3, Figure 2 · p. 4
  • 3.1 SYNTHETIC INSTRUCTION CREATION VIA COT: For open-ended tasks, the model generates only an instruction, while domain-grouped seed sampling avoids unnatural cross-domain combinations.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 SYNTHETIC INSTRUCTION CURATION

  • 3.2 SYNTHETIC INSTRUCTION CURATION: Answer-Consistency rejects verifiable examples when majority-vote answers disagree with the generated target answer.The target answer benefits from the extensive reasoning used during example creation, adding a filtering signal beyond inference-time solutions.Source: §3.2 · p. 5
  • 3.2 SYNTHETIC INSTRUCTION CURATION: For non-verifiable tasks, a modified RIP procedure scores sampled responses and selects the highest-scoring instruction within each sampled topic group.The modification avoids topic-distribution shifts caused by applying one global score threshold across categories.Source: §3.2 · p. 5

3.3 SELF-TRAINING WITH SYNTHETIC DATA

  • 3.3 SELF-TRAINING WITH SYNTHETIC DATA: Retained reasoning data is used with GRPO, while general instruction-following data is trained with offline or online DPO.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

The experiments test CoT-based generation and curation across verifiable reasoning and general instruction-following tasks using established seeds, training methods, baselines, and benchmarks. The setup compares synthetic-data variants with human and curated datasets.

4.1 VERIFIABLE REASONING TASKS

  • 4.1 VERIFIABLE REASONING TASKS: The reasoning study seeds generation from 893 verifiable questions selected from the 1,000-question s1k dataset.Source: §4.1 · p. 5
  • 4.1 VERIFIABLE REASONING TASKS: Generation compares Qwen3-4B-Base, Think, and NoThink modes, producing up to 10,000 instructions under specified sampling settings.Source: §4.1 · pp. 5–6
  • 4.1 VERIFIABLE REASONING TASKS: Models are trained with GRPO from Qwen3-4B-Base using rule-based verifiable rewards, 40 epochs, batch size 128, and 16 rollouts per prompt.Source: §4.1 · p. 6
  • 4.1 VERIFIABLE REASONING TASKS: Baselines test standard Self-Instruct, mixed CoT variants, original s1k, OpenMathReasoning, alternative filters, and Best-of-K targets.Source: §4.1 · p. 6
  • 4.1 VERIFIABLE REASONING TASKS: Evaluation reports average accuracy on MATH500, AIME 2024, AMC 23, and GPQA Diamond from 16 sampled solutions per problem.Source: §4.1 · p. 6

4.2 GENERAL INSTRUCTION FOLLOWING

  • 4.2 GENERAL INSTRUCTION FOLLOWING: The study uses 4,000 categorized WildChat-RIP-Filtered-by-8b-Llama seed instructions spanning eight domains.Source: §4.2 · p. 6
  • 4.2 GENERAL INSTRUCTION FOLLOWING: It compares long- and short-CoT generation with Self-Instruct using 5,000 same-category prompts and RIP selection among 32 candidates per prompt.Source: §4.2 · p. 6
  • 4.2 GENERAL INSTRUCTION FOLLOWING: Offline and online DPO are evaluated on AlpacaEval 2.0 and Arena-Hard, with length-normalized preference rewards used to reduce length explosion.Source: §4.2 · p. 7

5 EXPERIMENTAL RESULTS

Across reasoning and general instruction-following tasks, CoT-Self-Instruct consistently outperforms standard Self-Instruct and other training sources, with filtering producing additional gains.

5.1 REASONING TASKS

  • 53.0% average accuracy exceeds Self-Instruct’s 42.7% without filtering, while Self-Instruct instructions paired with CoT-generated targets reach 49.5%.The comparison uses Qwen3-4B-Base models trained with GRPO on Qwen3-4B-generated data.Source: §5.1 · p. 7, Table 1 · p. 8
  • Answer-Consistency filtering improves CoT-Self-Instruct from 55.1% to 57.2%, outperforming Self-Consistency filtering.Filtering improves both methods despite reducing the total amount of training data.Source: §5.1 · p. 7
  • 57.2% average accuracy makes filtered CoT-Self-Instruct the strongest tested reasoning-data approach, exceeding s1k at 44.6% and OpenMath-Reasoning at 47.5%.Using 10k Answer-Consistency-filtered examples raises performance further to 58.7%.Source: §5.1 · pp. 7–8
  • The CoT-Self-Instruct advantage persists when targets come from alternative models or generation procedures, including base-model, majority-vote, and Best-of-K targets.Each variant has different maximum performance but preserves the same ordering against Self-Instruct and competing baselines.Source: §5.1 · p. 8

5.2 GENERAL INSTRUCTION FOLLOWING TASKS

  • 53.9 average performance exceeds Self-Instruct’s 47.4 on AlpacaEval 2 and ArenaHard without filtering, and longer CoT chains provide additional gains.These experiments use DPO-trained models for general instruction following.Source: Table 2 · p. 8, §5.2 · p. 9
  • RIP filtering raises CoT-Self-Instruct from 53.9 to 54.7 and Self-Instruct from 47.4 to 49.1, while preserving CoT-Self-Instruct’s lead.RIP also raises WildChat human-instruction performance from 46.8 to 50.7.Source: §5.2 · p. 9
  • With offline DPO, CoT-Self-Instruct with RIP filtering yields 54.7 versus 27.1 for Llama 3.1-8B-Instruct and 46.8 or 50.7 for WildChat with or without filtering; online DPO reaches 67.1 versus 63.1.Source: §5.2 · p. 9

6 CONCLUSION

CoT-Self-Instruct combines reasoning-based synthetic instruction generation with task-appropriate quality filtering. The resulting data improves trained models across verifiable reasoning and general instruction-following domains, surpassing existing human-annotated and public datasets.

  • CoT-Self-Instruct plans and reasons from seed examples before generating synthetic instructions, then applies Answer-Consistency or RIP filtering according to task verifiability.The pipeline targets both verifiable reasoning and non-verifiable general instruction following.Source: §6 · p. 9

7 APPENDIX

The appendix reports matched-size experiments and alternative prompt templates for reasoning, including no-CoT, no-solve, and target-generation variants. It also documents evaluations using multiple target-generation strategies and longer CoT prompts.

  • 893-example train-size matching compares CoT-Self-Instruct with baselines under equal training-set sizes.Source: §7 · p. 14, Table 3 · p. 14
  • Self-Instruct-Then-Solve generates a question and answer without CoT, whereas CoT-Self-Instruct NoSolve reasons to generate a question without a reference answer.Source: §7 · p. 14, Figure 9 · p. 16, Figure 8 · p. 16
  • The appendix includes standard Self-Instruct templates that generate instructions alone or instructions with target answers without reasoning.Source: Figure 6 · p. 15, Figure 7 · p. 15
  • Additional reasoning experiments compare prompt templates using direct, majority-voted, and base-model-sampled targets, with tables reporting pass@1 across four benchmarks.Source: Table 4 · p. 17, Table 5 · p. 17, Table 6 · p. 17
  • Long-CoT CoT-Self-Instruct outperforms short-CoT and standard Self-Instruct templates on general instruction-following evaluations.Both AlpacaEval 2 and ArenaHard are evaluated with GPT-4 Turbo and GPT-4o judges.Source: Table 11 · p. 19

Related papers

Loading 2507.23751v2…