PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Mathematical Proof Supervision from Structured Textbooks

Youqing Fang, Zhouqi Hua, Yinhao Tang, Yuzhe Gu, Lingkai Kong, Bin Liu, Weiming Zhang, Kai Chen, Yanan Sun, Wenwei Zhang

arXiv:2610.40394v1cs.CLcs.AI

TL;DR

Mathematical supervision often flattens source material into isolated problem–solution pairs, leaving textbook structure underused. TOME-Math turns textbooks into structured proof-supervision records and evaluates structure-aware retrieval and training. Its fine-tuned model raises the six-benchmark mean by 16.66 percentage points over the base model.

  • Problem

    Mathematical supervision commonly uses isolated problem–solution pairs, while scaling proof supervision that preserves textbook order, proposition dependencies, source locations, and exercises remains unresolved.

    Source: §1 · p. 2

  • Method

    TOME-Math converts textbook propositions and proof gaps into linked QA records and uses retained structure for controlled retrieval and curriculum studies.

    Source: §1 · p. 2

  • Results

    16.66 percentage points: TOME-Math raises the unweighted six-benchmark mean from 29.17% to 45.83% over the base model and achieves the best result on five benchmarks.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    TOME-Math provides a large-scale resource for studying how textbook structure, curriculum ordering, and source coverage relate to mathematical reasoning.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Mathematical supervision for language models is typically flattened into isolated problem-solution pairs, discarding the curricular organization and evidential structure of the original sources. We introduce TOME-Math (Textbook-Oriented Mathematical Evidence), a large-scale corpus that converts 2,118 textbooks spanning the core areas of university-level mathematics subjects into approximately 380K structured proof-supervision records while preserving chapter order, proposition-proof links, dependency references, and source provenance. Leveraging these structural signals, we explore curriculum learning across multiple mathematical subjects and characterize scaling patterns with supervision volume and source breadth. At full scale, fine-tuning on TOME-Math raises the unweighted mean score across our held-out evaluations and established public benchmarks by 16.7 percentage points over the base model, while exceeding all three public instruction-data baselines evaluated in aggregate. Our findings highlight structured textbooks as both a scalable source of mathematical supervision and an empirical framework for studying the impact of data organization on mathematical reasoning. We will release the corpus to facilitate further research on structured mathematical supervision.

1. Introduction

Mathematical supervision commonly reduces knowledge to isolated problem–solution pairs, leaving textbook organization underused. TOME-Math preserves textbook hierarchy and proof dependencies in a large corpus and supports controlled studies of supervision structure.

1. Introduction

  • The dominant problem–solution format does not explicitly leverage source organization, while textbook order, proposition dependencies, and source locations remain difficult to preserve at scale.Textbooks combine definitions, propositions, proofs, and exercises within chapter hierarchies that reflect dependencies and pedagogical progression.Source: §1 · p. 2
  • TOME-Math contains 380,292 proof-oriented QA pairs from 2,118 textbooks across 29 subjects, preserving hierarchy, proposition–proof links, dependencies, and curriculum order.Its records connect theorem-level questions to intermediate proof obligations derived from proof gaps.Source: §1 · p. 2
  • The corpus supports controlled studies of retrieval structure, curriculum order, and source coverage rather than treating supervision scale as the only variable.The introduction reports consistent benefits from structure-guided retrieval, textbook-aligned curricula, and broader source coverage.Source: §1 · p. 2

2. Related Work

Prior mathematical resources span instruction data, textbook-informed corpora, and proof resources, but differ in how they represent mathematical knowledge. TOME-Math combines broad textbook coverage with local proof obligations, chapter order, and proposition references.

Mathematical instruction data

  • Instruction datasets such as GSM8K and MATH provide authored problem–solution pairs, while MetaMathQA and OpenMathInstruct-2 expand seeds synthetically and NuminaMath-CoT aggregates sourced chain-of-thought solutions.These resources generally retain metadata such as source, seed question, or difficulty rather than textbook hierarchy and proposition links.Source: §2 · p. 3

Textbook and knowledge-informed resources

  • Textbook-informed resources include pretraining corpora, extracted exercises, topic-graph QA, and theorem-application evaluation, motivating retention of textbook organization and mathematical statement links.Examples include OpenWebMath, MathPile, CollegeMath, MathScale, and TheoremQA.Source: §2 · p. 3

Proof resources and evaluation

  • Proof resources preserve statements, proofs, or reference graphs, while TOME-Math combines 2,118-book coverage with local proof obligations, chapter order, and proposition references.Prior directions include NaturalProofs, ProofNet, Lean-GAP, and FormaRL’s uproof.Source: §2 · p. 3

3. TOME-Math

TOME-Math converts textbook content into linked proof-supervision records while preserving chapter structure and source provenance. Its construction combines proposition-based Proof QA with Gap QA, alongside structural and semantic quality checks.

3.1. Data Construction

  • The pipeline converts 2,118 canonicalized textbook PDFs into 380,292 QA pairs: 205,861 Proof QA records and 174,431 Gap QA records.Proof QA comes from explicit proposition–proof units, while Gap QA targets missing proof obligations.Source: §3.1 · p. 3
  • Page-wise parsing reconstructs textbook chapter hierarchy and separates core concepts from exercises, retaining exercises separately but excluding them from Proof QA generation.Ambiguous page assignments are retried before the chapter state is updated.Source: §3.1 · p. 4
  • Proof QA extraction links theorem, lemma, and definition proofs to earlier propositions; Gap QA then identifies implicit or missing reasoning and produces self-contained intermediate obligations.Structural checks assess hierarchy, page boundaries, continuity, alignment, and standalone completeness, while semantic audits check completeness, accuracy, formatting, self-containment, and atomicity.Source: §3.1 · p. 4

3.2. Characterization

  • The corpus spans 29 subject categories, and each record carries book, chapter, and source-location metadata, with proposition identifiers and resolved references for Proof QA.Evaluation scopes are held-out exercises for item-level generalization and held-out books for source-level generalization, with duplicate groups respected.Source: §3.2 · pp. 4–5
  • The Gap QA stream retains 174,431 of 918,235 generated candidates, a yield of 19.00%.Figure 3 characterizes corpus scale from source textbooks through filtering to the final corpus, and compares model configurations across quality-control criteria.Source: §3.2 · p. 5, Figure 3 · p. 5

4. Evaluating TOME-Math

Across six evaluations, TOME-Math fine-tuning improves the base model’s unweighted mean and leads on five benchmarks. The evaluation uses book-held-out textbook exercises and public benchmarks to compare independently fine-tuned corpora.

4.1. Experimental Setup

  • The six-subject TOME-LONGFORM evaluation uses 300 proof-heavy textbook exercises from held-out books, with 64 books excluded from training and 364,631 examples remaining.Solutions are sampled 16 times per problem and assessed with a reference-guided step verifier; complementary evaluations include TOME-OBJECTIVE and four public benchmarks.Source: §4.1 · p. 5

4.2. Full-Corpus Comparison

  • 45.83% versus 29.17% is the unweighted six-benchmark mean for TOME-Math and the base model, respectively; TOME-Math leads on five benchmarks and ranks second on AIME 2026.Its largest margins over the strongest public baseline are on TOME-LONGFORM and ProofNet, while advantages on TOME-OBJECTIVE, MiniF2F, and MATH-500 extend beyond long-form proof generation.Source: Table 2 · p. 6, §4.2 · p. 6

4.1. Experimental Setup

  • All models start from Qwen3-30B-A3B-Base and use the same fine-tuning setup, differing only in supervision corpus; each is trained for one epoch over its corpus.Table 2 reports percentages for six evaluations and an unweighted mean.Source: §4.1 · p. 5, §4.2 · p. 6, Table 2 · p. 6

4.2. Full-Corpus Comparison

  • TOME-Math outperforms OpenMath1 on all six benchmarks despite OpenMath1 having approximately 4.8 times as many examples, and exceeds the similarly sized MetaMath corpus throughout.Because the corpora differ in several respects, this comparison alone does not isolate textbook structure as the causal factor.Source: §4.2 · p. 6

5. Leveraging Textbook Structure

The section tests whether textbook structure helps mathematical reasoning during retrieval and training, and whether benefits extend across subjects and source breadth. Controlled studies find gains from structured evidence, forward curricula, broader book coverage, and the complete training recipe.

5.1. Training-Free Structure-Guided Retrieval

  • Under a shared context-token budget, TOME (PROPS + GAP) reaches the best overall score, 17.38% pass@1, a 4.13-point gain over closed-book generation.Proposition–proof retrieval reaches 15.54%, and adding Gap QA contributes 1.84 points; the complete policy improves over CLOSED by 1.75–6.00 points across all six subjects.Source: §5.1 · p. 7

5.2. Curriculum Learning Across Subjects and Data Scales

  • At 2.4M tokens, FORWARD–GAP reaches 50.93% macro-average pass@16 versus 43.21% for SHUFFLE, and it is strongest in all 18 subject–scale cells.Across all three budgets, forward progression is the primary contributor, while Gap-first ordering adds a smaller, consistent gain over FORWARD–MIX.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3. Source Breadth and Cross-Subject Transfer

  • At the same 0.75M-token budget, spreading FORWARD–GAP training across 16 rather than 8 books improves pass@16 in every subject by 1.6–4.8 points, averaging 3.3.This matched-budget comparison identifies source breadth as a scaling axis distinct from token volume.Source: Figure 5 · p. 8, §5.3 · p. 9
  • Subject-specific checkpoints achieve a 0.92 macro-average cross-subject transfer coefficient, with eight of 30 ordered source–target pairs exceeding 1.0.Topology and differential geometry have the strongest source-side means, at 0.98 and 0.97, respectively.Source: Figure 5 · p. 8, §5.3 · p. 9

5.4. Full-Corpus Ablation of the TOME-Math Training Recipe

  • Removing components from the full recipe lowers pass@1 across all six benchmarks; removing Gap QA causes the largest drops on the four proof benchmarks (5.05–6.77 points).On the two TOME benchmarks, replacing Gap-first with within-stage mixing costs 3.74 and 2.59 points, while global shuffling costs another 0.25 and 0.95 points; the complete recipe also leads on MATH-500 and AIME-26.Source: §5.4 · p. 9

6. Conclusion

TOME-Math is a 380K-record proof corpus from 2,118 textbooks that preserves curricular and dependency structure. It improves performance across textbook-derived and public benchmarks, and controlled studies report benefits from structured retrieval, curriculum ordering, and broader source coverage.

6. Conclusion

  • TOME-Math contains 380K proof records from 2,118 textbooks and preserves curricular and dependency structure.The corpus improves performance across textbook-derived and public benchmarks; controlled studies report benefits from structure-guided retrieval, curriculum ordering, and broader source coverage.Source: §6 · p. 9
  • Controlled studies demonstrate benefits from structure-guided retrieval, curriculum ordering, and broader source coverage.The conclusion presents TOME-Math as a resource for mathematical reasoning systems that learn from better-organized knowledge as well as more data.Source: §6 · p. 9
  • The authors state that they hope TOME-Math inspires systems that learn not only from more data, but from better-organized knowledge.Source: §6 · p. 9

learning-to-reason-with-llms/. 1

The appendix documents TOME-Math’s construction, from recovering textbook structure and separating exercises to extracting propositions and proof gaps. It also outlines the intermediate artifacts and model-allocation details.

Appendix Contents

  • The appendix contents list construction details, corpus characterization, synthesized QA samples, evaluation protocols, full-corpus comparisons, and retrieval, curriculum, and transfer studies.Source: Appendix Contents · p. 13

A. TOME-Math Construction Details

A.1. Intermediate representation

  • The intermediate representation includes a recovered table of contents, main text, chapter files, exercise records and index, and any detached answer chapter.Source: §A.1 · p. 14

A.2. M1–M2: Table-of-contents and chapter recovery

  • M1 recovers and verifies table-of-contents entries; M2 uses that hierarchy to segment the textbook body into ordered chapters.M2 checks boundary pages and writes completed chapters to the content directory.Source: §A.2 · p. 14

A.3. M3–M6: Exercise extraction and answer alignment

  • M3 separates exercises from exposition, while M4–M6 serialize problems, partition and match answers, and filter incomplete questions.The answer-matching branch depends on whether problem numbering is absolute or restarts across chapters.Source: §A.3 · pp. 14–15

A.4. Four-stage proposition pipeline

  • The four-stage proposition pipeline extracts propositions, resolves references, audits proofs for gaps, and refines candidate gaps into self-contained statements.Its output distinguishes original proposition records from candidate sub-propositions marked as potential gaps.Source: §A.4 · p. 15

A.5. Proposition extraction and model allocation

  • The proposition-extraction model allocation changed during development, was versioned by run, and had per-book costs recorded in the design document.The cost-optimization note assigns reference semantics to a lower-cost non-thinking model and pairs it with a thinking model for other reasoning-intensive operations.Source: §A.5 · pp. 15–16, Table 5 · p. 16

A.6. Model-Quality Evaluation and Model Selection

The appendix compares extraction configurations across six model-quality dimensions, including cost-effectiveness. The combined pipeline is reported to balance extraction quality and cost.

A.6. Model-Quality Evaluation and Model Selection

  • Extraction quality is evaluated on completeness, accuracy, LaTeX formatting, self-containment, atomicity, and cost per QA pair.A higher C/P Ratio indicates a lower cost per pair.Source: §A.6 · p. 16
  • The Combined configuration uses Flash Thinking for primary extraction and auditing and Flash Lite for reference-semantic parsing.Source: §A.6 · p. 16
  • The Combined pipeline is reported to preserve competitive extraction quality while improving cost-effectiveness, offering a favorable balance between quality and cost per QA pair.Source: §A.6 · p. 16

B. Extended Characterization Analysis

The appendix characterizes TOME-Math’s task mix, coverage, quality, provenance, reasoning scope, structural complexity, and confidence. Five synthesized examples illustrate progression from direct fact recall to cross-chapter synthesis.

B.1. Task and Answer-Format Composition

  • 79.0% of records are proof generation; derivation and calculation use derivation-format answers 87.0% of the time, while proof answers split between full and concise formats.Full textbook proof accounts for 51.0% and concise final proof for 41.9% of proof-generation answers.Source: Figure 9 · p. 17, §B.1 · p. 17

B.2. Subject and Language Coverage

  • The ten leading subjects account for 84.4% of records, with mathematical analysis drawn from 307 books and graph theory concentrated in 25.Source-language composition also varies by educational level, from all mixed-language high-school records to predominantly English graduate records.Source: Figure 10 · p. 18, §B.2 · p. 18

B.3. Quality Behavior at Subject and Dimension Granularity

  • Groundedness and evidence alignment pass below 50% of records and fail at 28.6%, while ambiguity and question–answer consistency pass above 90%.The two leading exclusion causes are broken formulas and incomplete proofs; the top four causes cover 80% of exclusions, and “no verbatim evidence” flags 48.4% of the corpus.Source: Figure 11 · p. 19, §B.3 · pp. 19–20, Figure 12 · p. 20

B.4. Corpus Redundancy and Generation Provenance

  • 99.15% of records are singletons; generator assignments are disjoint across Proof QA and Gap QA, and the highest-yield book contributes 0.80% of the corpus.gemini-3-flash-preview-thinking produces 88.9% of Proof QA records, while gpt-5-mini produces 53.7% of Gap QA records; the top twenty books are dominated by graph theory, combinatorics, and Chinese competition texts.Source: Figure 13 · p. 21, §B.4 · p. 21, Figure 14 · p. 21

B.5. Reasoning Scope and Subject Composition Across Level

  • Graduate records have a 32.5% share of multi-step same-chapter and cross-chapter reasoning, versus 18.7% for undergraduate and 17.7% for competition records.Reasoning scope reflects curriculum breadth rather than solving skill; mathematical analysis is the only leading subject with substantial undergraduate and graduate coverage.Source: Figure 15 · p. 22, §B.5 · p. 22

B.6. Structural Complexity and Confidence Dynamics

  • Prerequisite count correlates more strongly with estimated reasoning steps than concept count, at 0.633 versus 0.522.Confidence falls with difficulty only at stages four and five, and answer length correlates with confidence at −0.231 versus −0.060 for question length.Source: Figure 16 · p. 23, §B.6 · p. 23

B.7. Structural Difficulty and Record Length

  • 51.0% of records occupy stage three at difficulty three, while question and answer lengths vary substantially within every quality tier.Stage three accounts for 65.9% of records, and length alone cannot determine tier assignment.Source: Figure 17 · p. 24, §B.7 · p. 24

B.8. Confidence Distributions by Tier and Curriculum Stage

  • Mean confidence declines from tier A at 0.944 to tier C at 0.821; across curriculum stages it follows a non-monotonic pattern.Stage three has the largest sample share at 66.0% and the tightest confidence spread.Source: Figure 18 · p. 24, §B.8 · p. 24

D.1. Evaluation set construction

The appendix defines book-held-out, proof-heavy and objective evaluation sets, then specifies response generation, verification, calibration, scoring, and matched corpus-training procedures.

D.1. Evaluation set construction

  • 300 textbook-native problems form the frozen, book-held-out TOME-LONGFORM suite, with 50 problems per subject and expert review.The six subjects are sampled across earlier and later chapter halves; the set is proof-heavy, comprising 227 proof, 50 calculation, 18 construction, and five explanation problems.Source: §D.1 · p. 27, Table 6 · p. 27
  • 145 TOME-OBJECTIVE items test final-answer correctness, alongside 30 AIME 2026 problems and the 500-problem MATH-500 subset.MiniF2F and ProofNet are additional natural-language proof benchmarks, scored by a common LLM judge using adapted official procedures.Source: §D.1 · p. 27

D.2. Response generation

  • TOME-LONGFORM uses 16 question-only rollouts per problem; objective benchmarks use fixed, benchmark-specific sampling counts and decoding configurations.The TOME-LONGFORM reference solution is withheld from the solver.Source: §D.2 · p. 28, Table 7 · p. 28

D.3. Reference-guided step verification

  • The reference-guided verifier evaluates segmented responses against the problem and reference, accepting valid alternative proofs but requiring justified mathematical steps.It treats unsupported claims and unjustified leaps conservatively; a matching conclusion alone does not excuse an invalid derivation.Source: §D.3 · p. 28

D.4. Human calibration of the pessimistic verifier

  • The eight-review pessimistic verifier rejects a response if any review flags an error, and retains the earliest flagged step.Human calibration samples 600 responses across six subjects, deliberately balancing agreement strata and retaining sampling probabilities for weighted estimates.Source: §D.3 · p. 28, §D.4 · p. 29
  • Qualified annotators independently label a stratified 30% subset, with disagreements, unjudgeable cases, and low-confidence labels adjudicated.The calibration reports weighted false-acceptance and false-rejection rates, agreement, and whether human-labeled correctness preserves model rankings.Source: §D.4 · p. 29, Table 8 · p. 30

D.5. Objective and additional proof scoring

  • Objective benchmarks score answer correctness only, while MiniF2F and ProofNet use LLM-assessed natural-language proof correctness.Equivalent expressions are accepted for objective scoring; AIME 2026 and MATH-500 use rule-based checking with an LLM fallback for unresolved cases.Source: §D.5 · p. 30

D.6. Metrics and validity checks

  • TOME-LONGFORM reports pass@1 and pass@16 as primary metrics, and complete rollouts and successful proof reviews are required for reported models.With 16 responses per problem, pass@16 is the observed fraction solved at least once; subject scores are macro-averaged.Source: §D.6 · p. 30

E.3. All benchmark results

The extended comparison reports pass@1 across six benchmarks with problem-clustered bootstrap uncertainty estimates. AIME 2026 has the widest intervals because it contains only 30 problems.

E.3. All benchmark results

  • The comparison estimates pass@1 from 16 responses for TOME-LF, MiniF2F, and ProofNet, eight for TOME-OBJECTIVE, four for MATH-500, and 32 for AIME 2026.MiniF2F and ProofNet natural-language proofs are assessed by a common LLM judge using adapted official procedures.Source: §E.3 · p. 32
  • AIME 2026 has the widest confidence intervals because its evaluation contains only 30 problems.The intervals are problem-clustered bootstrap estimates.Source: §E.3 · p. 32
  • Pass@1 scores across six benchmarks are reported with average half-widths of 95% problem-clustered bootstrap confidence intervals.The intervals use 2,000 problem-clustered bootstrap resamples.Source: Table 11 · p. 32, §E.3 · p. 32

F. Training-Free Dependency-Context Study

The dependency-context study compares retrieval conditions on held-out textbook exercises, while the curriculum protocol controls training order, scale, and source breadth. The full structure-guided retrieval condition achieves the highest aggregate score among the reported context conditions.

F. Training-Free Dependency-Context Study

  • Retrieval compares no context, random same-book context, local-chapter context, proposition-dependent retrieval, and proposition-plus-Gap retrieval under a shared context budget.Candidates come from the exercise's source book; leakage-filtered inventories and the same prompt template constrain all context-augmented conditions.Source: F. Training-Free Dependency-Context… · p. 32
  • 17.38% with proposition-plus-Gap retrieval is the highest aggregate score, versus 15.54% for proposition-only retrieval and 13.25% without context.The final condition is best in every subject, though intermediate conditions are not monotonically ordered within every subject.Source: F. Training-Free Dependency-Context… · p. 33

G. Curriculum-Scaling Protocol

  • The curriculum protocol divides each book into ten normalized chapter stages and interleaves examples across books within each stage.It contrasts shuffled exposure, forward or reverse stage order, and whether Gap examples precede other examples within a stage.Source: G. Curriculum-Scaling Protocol · p. 33
  • Curriculum scales use nested 8-, 16-, and 24-book sets, with matched schedules within each subject and held-out evaluation books excluded.The source-breadth comparison doubles coverage from eight to 16 books while matching the 0.75M-token training budget.Source: G. Curriculum-Scaling Protocol · pp. 33–34, Table 12 · p. 34

H. Cross-Subject Transfer Protocol and Results

The cross-subject transfer protocol evaluates subject-specific checkpoints on held-out exercises from other subjects. It reports pairwise pass@16 scores and off-diagonal averages.

H. Cross-Subject Transfer Protocol and Results

Related papers

Loading 2610.40394v1…