PaperlayerView exact revision on arXiv

Source-linked AI summary

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs

Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

arXiv:2410.12881v2cs.AIcs.CL

TL;DR

Synthetic data often lacks the complementary knowledge and step-by-step structure needed for complex mathematical and multi-hop reasoning. MIND turns OpenWebMath into structured synthetic conversations and evaluates them in pretraining. Models trained on MIND-OWM outperform raw-data models, including against a 3.6× larger raw corpus.

  • Problem

    Synthetic-data approaches often fail to add complementary knowledge to raw corpora and may lack the step-by-step structure needed for complex mathematical and multi-hop reasoning.

    Source: §1 · p. 2

  • Method

    MIND prompts a pretrained LLM with raw web documents to generate step-by-step, multi-turn conversations, then refines the generations with heuristic filters.

    Source: §1 · p. 2

  • Results

    Models trained on MIND-OWM outperform raw-data models, with up to a 6.29% improvement across mathematical reasoning benchmarks and gains that persist against 3.6× larger raw data.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    Structured conversational data can enhance reasoning, especially where high-quality domain-specific data is limited, supporting more resource-efficient pretraining.

    Source: §7 · p. 11

    The TWO PROFESSORS setting may produce lower performance when both participants assume the other already has sufficient knowledge, leading to less detailed discussion.

    Source: §5 · p. 7

Abstract

from arXiv · show

The utility of synthetic data to enhance pretraining data quality and hence to improve downstream task accuracy has been widely explored in recent large language models (LLMs). Yet, these approaches fall inadequate in complex, multi-hop and mathematical reasoning tasks as the synthetic data typically fails to add complementary knowledge to the existing raw corpus. In this work, we propose a novel large-scale and diverse Math Informed syNthetic Dialogue (MIND) generation method that improves the mathematical reasoning ability of LLMs. Specifically, using MIND, we generate synthetic conversations based on OpenWebMath (OWM), resulting in a new math corpus, MIND-OWM. Our experiments with different conversational settings reveal that incorporating knowledge gaps between dialog participants is essential for generating high-quality math data. We further identify an effective way to format and integrate synthetic and raw data during pretraining to maximize the gain in mathematical reasoning, emphasizing the need to restructure raw data rather than use it as-is. Compared to pretraining just on raw data, a model pretrained on MIND-OWM shows significant boost in mathematical reasoning (GSM8K: +13.42%, MATH: +2.30%), including superior performance in specialized knowledge (MMLU: +4.55%, MMLU-STEM: +4.28%) and general purpose reasoning tasks (GENERAL REASONING: +2.51%).

1 INTRODUCTION

The introduction presents MIND as a way to turn raw mathematical web text into structured conversations, addressing synthetic data’s limited step-by-step structure for complex reasoning. It reports gains over raw-data training, including comparisons with a larger corpus.

1 INTRODUCTION

  • Prior synthetic-data methods often lack the step-by-step problem-solving structure needed for multi-hop and complex mathematical reasoning.The introduction also notes that curating high-quality structured mathematical data is costly and resource-intensive.Source: §1 · pp. 1–2
  • MIND prompts a pretrained LLM with web documents to create multi-turn conversations that decompose context step by step and explore each step in depth.The generated conversations are refined with heuristic filters before use in pretraining.Source: §1 · p. 2
  • 2.94% average improvement across GSM8K and MATH, and 1.56% across all benchmarks, when MIND-OWM-4B is compared with 3.6× larger OWM-14B.The comparison uses conversations generated from OWM-4B and is shown for GSM8K, MMLU, and general reasoning.Source: Figure 1 · p. 2, §1 · p. 3
  • The experiments identify participant knowledge imbalance as important to high-quality mathematical data and examine conversational styles, scaling, and raw-text source quality.The contribution summary also emphasizes reformatting raw data when integrating it with synthetic data during pretraining.Source: §1 · pp. 2–3
  • 6.29% average improvement across three mathematical reasoning benchmarks, 4.30% on MMLU, and 2.20% across 10 zero-shot tasks versus raw OWM-4B.These results compare continuous pretraining on MIND-OWM-4B with training on raw OWM-4B.Source: §1 · p. 2

2 MIND: MATH INFORMED SYNTHETIC DIALOGUE GENERATION

MIND converts raw text into multi-turn synthetic conversations using prompts designed to structure explanations and reasoning. The resulting corpus is heuristically filtered and evaluated through continued pretraining.

2 MIND: MATH INFORMED SYNTHETIC DIALOGUE GENERATION

  • MIND is motivated by the limits of surface rephrasing: multi-turn conversations can break context into steps and add explanations or complementary reasoning.The paper argues that varied syntax alone does not address complex logical and mathematical reasoning as effectively as structured complexity.Source: §2 · p. 3
  • MIND combines each raw example with one of seven conversational prompts and uses a pretrained LLM to generate a conversation.The prompt set spans styles such as two students, teacher–student, debate, problem solving, and interview.Source: §2 · p. 3, §2.1 · p. 4
  • Heuristic filtering produces a refined synthetic dialogue corpus, which the authors evaluate by continuously pretraining a base LLM to reduce computational cost.They report that model rankings between synthetic and raw data remain consistent under continuous and full pretraining in prior experiments.Source: §2 · pp. 3–4

2.1 COMPOSE CONVERSATIONAL PROMPTS

  • The seven manually composed prompts preserve the source context while guiding step-by-step discussion and in-depth explanations across diverse conversational settings.The intended settings include two students, teacher–student, two professors, debate, problem solving, layman know-all, and interview.Source: §2.1 · p. 4

2.2 GENERATE CONVERSATION

  • For each raw text, the LLM generates a multi-turn conversation using a selected prompt from the prompt set.The paper identifies OpenWebMath as its seed corpus because it contains high-quality mathematical web text suited to step-by-step analysis.Source: §2.2 · p. 4

3 EXPERIMENTAL SETUP

The experiments use a 7B decoder-only model and a controlled continued-pretraining data blend. Evaluation covers mathematical, specialized-knowledge, and general-purpose reasoning tasks.

3 EXPERIMENTAL SETUP

  • The study uses zero-shot generation with temperature 1.0 and top_p 0.9, and evaluates downstream zero-shot and few-shot capabilities with LM Eval Harness.The base model is a 7B decoder-only Transformer with a 4096-token sequence length.Source: §3 · pp. 4–5, §3.2 · p. 5

3.1 TRAINING DETAILS

  • Continued pretraining uses 50B tokens in a 2:1 blend of OpenWebMath and CommonCrawl, with the OpenWebMath portion varied between raw, synthetic, or combined data.The CommonCrawl portion remains constant; OWM-4B is used to test seven prompts, while OWM-14B contains the full 14.7B tokens.Source: §3.1 · p. 5

3.2 EVALUATION METRICS

  • Evaluation includes GSM8K, MATH, and MMLU-STEM for math and science reasoning, MMLU for specialized knowledge, and ten tasks averaged as GENERAL REASONING.The math benchmarks span elementary through college-level challenges; the listed reasoning tasks include commonsense, logical reasoning, and reading comprehension.Source: §3.2 · pp. 5–6

4 EXPERIMENTS AND RESULTS

MIND-generated conversations improve reasoning performance over raw or rephrased data, with gains persisting at the full OpenWebMath scale. The experiments also find that conversational style and data formatting affect the benefits.

4 EXPERIMENTS AND RESULTS

  • Using OWM-4B and seven conversational prompts, MIND produces 43 billion tokens; adding OWM-14B with TWO STUDENTS yields 64 billion tokens total.Source: §4 · p. 6
  • Across individual styles, MIND data improves GSM8K by 4.78%–12.82%, MATH by 0.54%–1.28%, and MMLU-STEM by 0.79%–4.28% over raw or rephrased data.The LONGEST CONVERSATION yields the highest gains across tasks.Source: §4 · p. 6, Table 1 · p. 6
  • Unlike rephrasing, which adds linguistic variation without new semantic or pragmatic variation, MIND conversations provide structured, interactive material for complex reasoning.Source: §4 · p. 6
  • For the full OWM-14B scale experiment, the authors select TWO STUDENTS from the top-performing prompts and compare synthetic conversations with raw OWM-14B.They also test 50B additional tokens of rephrased data.Source: §4 · p. 7
  • In the reported 7B comparison, continuous training with synthetic conversations outperforms models trained on the original pretraining blend and raw OpenWebMath across all tasks, leading math benchmarks and improving average performance on other reasoning tasks.Source: §4 · p. 7, Table 2 · p. 7

5 ABLATIONS

The ablations examine dialogue style, source quality, generator size, and data formatting. They find that knowledge gaps and the choice between synthetic-only and mixed data relate to different reasoning outcomes.

5 ABLATIONS

  • TWO PROFESSORS conversations perform equivalently to or worse than raw data; the authors hypothesize that the participants’ assumed expertise produces surface-level discussion.Source: §5 · p. 7, Table 3 · p. 7
  • TWO PROFESSORS has the highest similarity to raw text, whereas LAYMAN KNOWALL has the lowest BLEU and ROUGE scores and richer explanations.Source: §5 · pp. 7–8, Figure 3 · p. 7
  • Explicit knowledge gaps benefit MMLU and general reasoning, while collaborative problem-solving to bridge shared knowledge gaps is important for math performance.Source: §5 · p. 8
  • On MATHPILE, MIND improves all three math benchmarks and specialized knowledge tasks, with a 3.95% absolute GSM8K gain over raw data.General reasoning scores are comparable to the raw-data counterpart.Source: §5 · p. 8, Table 4 · p. 8
  • The tested data formats include concatenating raw text with all seven conversations, selecting the longest conversation, and using all conversations.The longest-conversation mix uses a 1:1 raw-to-synthetic token ratio.Source: §5 · pp. 8–9
  • Conversations generated from OWM-4B outperform OWM-14B, a 3.6× larger corpus, on GSM8K, MMLU, and general reasoning, with comparable performance on other tasks.The comparison is reported for all-conversation combinations.Source: §5 · p. 9, Table 5 · p. 9
  • Synthetic-only training gives the best results on GSM8K and MATH, while combining raw and conversational data benefits specialized and general reasoning tasks.The authors note that raw data retains special tags, styles, and code indentations.Source: §5 · p. 9
  • With LLAMA3-8B-INSTRUCT as the dialogue generator, MIND data still improves math and general reasoning over raw or rephrased data.The reported gains are not solely dependent on the larger generator and are attributed largely to dialogue quality and structure.Source: §5 · pp. 9–10, Table 6 · p. 9

6 RELATED WORKS

Prior math corpora collect diverse web content, but noisy data remain difficult to filter and leading models often keep corpus details undisclosed. Earlier synthetic-math work differs from MIND in its objectives, generation approach, and reliance on proprietary models.

Mathematical Data Curation

  • OpenWebMath contains 14.7B tokens of math-filtered CommonCrawl pages, while MATHPILE aggregates 8.9B tokens, mostly from arXiv.Both illustrate large-scale mathematical data curation, though noisy data remain difficult to detect and filter.Source: §6 · p. 10

Synthetic Math Data

  • Prior synthetic-math approaches often target fine-tuning, whereas MIND is designed to upsample structured data for pretraining.MIND adds complementary reasoning and explanations through diverse conversational styles; Dai et al. (2022) converts documents into dialogues without altering the original document.Source: §6 · p. 10
  • Some synthetic pretraining datasets rely on proprietary models to produce billions of tokens and may bias generation toward target tasks.MIND instead uses an off-the-shelf open-source LLM to create structured data from diverse web contexts.Source: §6 · p. 10

7 CONCLUSION

The paper introduces MIND-OWM, a conversational synthetic math corpus for improving mathematical reasoning in open-source LLMs. Models trained on it outperform raw-data-trained models across mathematical benchmarks, including models trained on larger datasets.

7 CONCLUSION

  • MIND generates a conversational synthetic math corpus, MIND-OWM, using an off-the-shelf open-source LLM.The paper focuses on improving mathematical reasoning abilities in open-source LLMs.Source: §7 · p. 10
  • Models trained on MIND-OWM achieve up to 6.29% improvement across mathematical reasoning benchmarks and outperform models trained on 3.6× larger datasets.The reported gains persist across general-purpose reasoning tasks and when scaling up the data.Source: §7 · p. 11
  • The authors present structured conversational data as a resource-efficient way to enhance reasoning when high-quality domain-specific data are limited.They describe the results as demonstrating the potential of this approach for pretraining LLMs.Source: §7 · p. 11

A PROMPTS AND DATASETS

The appendix evaluates models on general-purpose reasoning, mathematical reasoning, and specialized-knowledge tasks, using zero-shot or few-shot prompts depending on the task group.

A.2 EVALUATION METRIC DETAILS

  • Evaluation covers ten general-reasoning tasks, three mathematical tasks, and one specialized-knowledge task.General-reasoning benchmarks use zero-shot evaluation; math and specialized-knowledge tasks use few-shot evaluation.Source: §A.2 · pp. 17–18
  • ARC-E, ARC-C, RACE, PIQA, WinoGrande, and HellaSwag assess science questions, reading comprehension, physical commonsense, and contextual reasoning.ARC-C contains more difficult questions than ARC-E and requires higher-order reasoning.Source: §A.2 · p. 17
  • OpenBookQA tests applying elementary science facts to novel situations, while TruthfulQA probes factual answers to questions designed around common misconceptions.TruthfulQA contains 817 questions across 38 categories.Source: §A.2 · p. 18
  • CommonSenseQA tests commonsense reasoning using ConceptNet relations, and Social-IQA assesses reasoning about people’s actions and their social implications.Source: §A.2 · p. 18
  • GSM8K evaluates multi-step and logical reasoning on linguistically diverse grade-school math word problems using eight chain-of-thought examples.Answers are selected by majority vote from greedy decoding.Source: §A.2 · p. 18
  • MATH evaluates challenging competition problems requiring step-by-step processing, using a 4-shot prompt.The evaluation prompt follows Lewkowycz et al. (2022b).Source: §A.2 · p. 18
  • MMLU measures multitask accuracy across 57 subjects, with MMLU-STEM selected for its math and science problems requiring complex reasoning.The evaluation uses 5-shot prompts.Source: §A.2 · p. 18

B.1 RESULTS OF PRETRAINING LLM FROM SCRATCH

Scratch-pretrained 8B models were evaluated across raw and synthetic OpenWebMath data configurations. Synthetic-only data performed best on mathematical benchmarks, while concatenating raw text with conversations gave the best average general-reasoning score.

B.1 RESULTS OF PRETRAINING LLM FROM SCRATCH

  • The study pretrained an 8B model from scratch on 300B tokens, including 35B OWM tokens seen over eight epochs.Other datasets were maintained at 0.46 epochs.Source: §B.1 · p. 18
  • The four OWM configurations were student-to-student conversations, a 1:1 raw-synthetic token blend, raw text concatenated with all seven conversations, and the longest conversation.The 1:1 blend uses the two-students conversations.Source: §B.1 · p. 18
  • Concatenating raw text with conversations yields the best average score across the general-reasoning data combinations.Table 7 reports improvements from conversational synthetic data over raw data on general-purpose reasoning tasks.Source: §B.1 · p. 18, Table 7 · p. 19
  • Synthetic-only conversations give the best improvement on mathematical benchmarks, while specialized-knowledge tasks gain most from combining raw and synthetic data.The reported pattern reflects the composite, structured nature of conversational data for step-by-step problem solving.Source: Table 8 · p. 19, §B.1 · p. 19

B.2 BREAKDOWN OF INDIVIDUAL TASKS RESULTS OF CONTINUED PRETRAINED LLM

Across individual styles and larger OWM data, MIND conversations improve reasoning performance, while dialogue length alone does not predict accuracy. Context size and quality filtering also shape the generated conversations.

B.2 BREAKDOWN OF INDIVIDUAL TASKS RESULTS OF CONTINUED PRETRAINED LLM

  • All conversational styles improve general-purpose reasoning over raw or rephrased data, while all-style OWM-4B conversations yield the highest mathematical-reasoning gains among the compared models.The comparisons cover general reasoning and specialized knowledge tasks as well as math benchmarks.Source: §B.2 · p. 19, Table 9 · p. 19, Table 10 · p. 20
  • With complete OWM-14B, conversational data also improves both general-purpose reasoning and mathematical reasoning, consistent with the smaller-data results.The tables cover general reasoning and math and specialized-knowledge tasks.Source: §B.2 · p. 20, Table 11 · p. 20, Table 12 · p. 20

C.1 CONTEXT LENGTH VS CONVERSATION QUALITY

  • A 2K-token context can yield a 1K-token conversation with substantial information loss, whereas a 500-token context can produce a longer conversation that adds reasoning.The study evaluated 140K samples and sought conversations that retain relevant information, remain grounded, and add complementary explanations.Source: §C.1 · p. 21, Figure 4 · p. 21

C.2 CONVERSATION LENGTH VS MODEL PERFORMANCE

  • PROBLEM SOLVING reaches comparable accuracy to LONGEST CONVERSATION with an average dialogue about 188 tokens shorter, so length is not the only factor in reasoning gains.The authors identify conversation structure and dynamics as additional factors.Source: §C.2 · p. 21, Table 13 · p. 21

C.3 CONVERSATION QUALITY ASSESSMENT

  • For 140K conversations, the LLM’s average across four 1-to-5 metrics labels around 96% high quality; despite rating its generations higher almost always, its distribution is less skewed than FineWebEdu’s, making it more suitable for evaluating seed-corpus-derived synthetic data.Source: §C.3 · p. 22

C.4 COMPARE WITH DEEPSEEKMATH

On a stronger pretrained model, MIND-OWM training improves results over raw data; on a smaller seed corpus, it reaches math accuracy comparable to DEEPSEEKMATH, which uses 120B unique math tokens.

C.4 COMPARE WITH DEEPSEEKMATH

  • MIND is presented as an alternative for generating math data from a small seed corpus, in contrast to DEEPSEEKMATH's 120B-token corpus-building approach.The passage describes DEEPSEEKMATH's iterative extraction of math content from CommonCrawl.Source: §C.4 · pp. 22–23
  • The DEEPSEEKMATH comparison uses a 2:1 blend of MIND conversations and other data, continuously trains a converged DEEPSEEKCODER-V1.5 model for 500B tokens, and uses the published converged weights.The unconverged model weights are described as unpublished to the authors' knowledge.Source: §C.4 · p. 23
  • A model trained on MIND-generated conversations from a small seed corpus achieves math accuracies comparable to DEEPSEEKMATH, which used 120B unique math tokens in its training blend.The MIND-trained model also outperforms DEEPSEEKMATH in MMLU and general-reasoning tasks and has higher average accuracy across all tasks.Source: Table 14 · p. 23, §C.4 · p. 23
  • 17.90% gain on GSM8K, 6.90% average improvement across math tasks, and 3.43% average improvement across ten general-reasoning tasks are reported for MIND-OWM over raw data.These results use DEEPSEEKMATH-7B as the pretrained model.Source: Table 15 · p. 23, §C.4 · pp. 23–24

C.5 CONVERSATIONS ON CODE TASKS

The section contrasts code-task limitations with examples of raw text and conversations in several styles. The examples range from worked mathematical questions to discussions of scientific, statistical, and other source material.

C.5 CONVERSATIONS ON CODE TASKS

  • Conversational synthetic data does not enhance coding performance, as natural-language wrapping fragments code and obscures its intended sequence and logic.The authors conclude that conversations are not suited to preserving code integrity; rephrasing also degrades coding accuracy when tags and indentation are removed.Source: §C.5 · p. 24, Table 16 · p. 24

RAW TEXT

  • The raw example is a CAT alloy-mixture question asking which range of x keeps the mixture between 40% and 50% tin.It provides two alloy weights and tin-to-lead ratios as the problem context.Source: RAW TEXT · p. 25

SYNTHETIC DIALOGUE [TWO PROFESSORS ]

  • The TWO PROFESSORS dialogue works through the alloy problem and settles on the answer 80 kgs ≤ x ≤ 240 kgs.The speakers recast the ratios as tin and lead quantities before considering the answer choices.Source: SYNTHETIC DIALOGUE [TWO PROFESSORS ] · p. 25

SYNTHETIC DIALOGUE [TEACHER STUDENT ]

  • The TEACHER STUDENT dialogue identifies reactants and products by decreasing and increasing concentration, respectively, and says the graph lacks enough information to calculate Kc.Source: SYNTHETIC DIALOGUE [TEACHER STUDENT ] · p. 26

SYNTHETIC DIALOGUE [TWO STUDENTS ]

  • The TWO STUDENTS dialogue uses overlap of the survey intervals to discuss reliability, then says exact values are needed to identify other unreliable results.Its numerical example identifies company B as not reliable when its interval does not overlap company C's.Source: SYNTHETIC DIALOGUE [TWO STUDENTS ] · pp. 27–28

SYNTHETIC DIALOGUE [LAYMAN KNOWALL ]

SYNTHETIC DIALOGUE [DEBATE ]

  • The DEBATE dialogue connects the sum of the first n natural numbers to the sum of the first odd natural numbers through a pairing analogy.It uses the Gauss 1-to-100 sum story to explain the pairing idea.Source: SYNTHETIC DIALOGUE [DEBATE ] · pp. 30–31

SYNTHETIC DIALOGUE [INTERVIEW ] / SYNTHETIC DIALOGUE [PROBLEM SOLVING ]

Loading 2410.12881v2…