StatEval: A Comprehensive Benchmark for Large Language Models in Statistics

1 Introduction

• Statistical problems comprise less than 3% of recent reasoning benchmarks, and existing tasks often omit layered inference, asymptotic analysis, and uncertainty quantification.This motivates a dedicated benchmark reflecting real statistical practice rather than isolated probability puzzles.Source: §1 · pp. 2–3
• StatEval contains over 100,000 curated problems, including 22,262 foundational questions and 84,179 research-level proof questions from six peer-reviewed journals.The foundational set draws on more than 70 textbooks and course materials and spans undergraduate through doctoral levels.Source: §1 · pp. 3–4
• A two-level taxonomy covering more than 30 statistical subdomains lets users slice problems by both difficulty and topic for behavioral analysis or targeted retrieval and training.Topics include probability, inference, regression, Bayesian methods, multivariate statistics, asymptotic theory, experimental design, and machine learning.Source: §1 · p. 4
• TRACE is a multi-agent pipeline that converts unstructured statistical research texts into self-contained theorem-level proof tasks and can generalize to other mathematical and scientific domains.Its modular extraction and logical-parsing framework supports both benchmark construction and model training.Source: §1 · p. 4
• The Adaptive Process-Based Scoring Pipeline evaluates proofs beyond final-answer matching through step-based verification and synthesis of logical coherence, technical precision, and terminal accuracy.It checks aligned solution strategies granularly and independently validates divergent approaches.Source: §1 · p. 4
• LLMs perform reasonably on foundational concepts and standard applied exercises but remain substantially limited on research-level derivations and proofs.The authors identify this gap as evidence that rigorous statistical reasoning in current LLMs has substantial room for improvement.Source: §1 · pp. 4–5
• Retrieval-augmented inference and supervised adaptation yield consistent gains across benchmark tasks, particularly on proof-style questions.StatEval supports both pathways through retrieval from its corpus and fine-tuning on targeted foundational or research-level subsets.Source: §1 · p. 5
• Sections 4–6 demonstrate StatEval’s practical utility through adaptive-scoring evaluation of contemporary LLMs, RAG-based augmentation, and domain-specific fine-tuning.The fine-tuning study investigates aligning open-weights models with rigorous statistical derivations.Source: §1 · p. 6

2.1 Foundational Knowledge Dataset

• The foundational dataset comprises 22,262 problems, including 9,382 undergraduate-level and 12,880 graduate-level instances, curated from 76 classical statistics and related-fields textbooks, over 1,000 verified exam-style questions, and selected problems from publicly available courses.Source: §2.1 · p. 7
• Foundational content is organized across Probability, Statistics, and Machine Learning, with course-level subjects and distinct undergraduate and graduate scopes.Graduate topics include Measure Theory, Empirical Processes, and Reinforcement Learning.Source: §2.1 · pp. 7–8, Figure 2 · p. 8

2.2 Statistical Research Dataset

• The dataset comprises 84,179 proof-based research tasks, divided into 40,366 Easy, 22,013 Medium, and 21,800 Hard instances, respectively.Source: §2.2 · p. 8
• Research tasks vary in prerequisite support: Easy supplies prerequisite results as facts, Medium requires proving supplied prerequisites, and Hard provides no prerequisite hints.Hard tasks require constructing the proof chain, including intermediate lemmas.Source: §2.2 · p. 9
• Research tasks are classified by subfield and theoretical property, a stratification intended to pinpoint weaknesses in theoretical reasoning.The theoretical-property taxonomy includes eight result types, including convergence and stability, optimality, and testing validity.Source: §2.2 · pp. 9–10, Figure 4 · p. 11

3 Data Processing Pipeline: TRACE

• TRACE combines document conversion, context segmentation, problem generation, dependency parsing, difficulty generation, and validation to build theorem-proof corpora and evaluation problems at scale.The pipeline targets both reliable extraction from unstructured literature and systematic problems for evaluating and improving LLM mathematical reasoning.Source: §3 · p. 11

3.1 File Conversion and Context-Segmentation Engine

• TRACE converts PDFs into structured Markdown, isolates theoretical elements, and represents distributed proof components as linked objects for later reconstruction.This preserves hierarchical relations when a theorem depends on auxiliary results or arguments spread across sections.Source: §3.1 · pp. 12–13

3.2 Notation Harmonization and Base Problem Generation

• A document-level notation reference and theorem-proof assembly add needed definitions and assumptions, so benchmark questions test statistical reasoning rather than inference of missing definitions.TRACE combines local context, globally declared notation, assumptions, and indexed equations into a validated reference table.Source: §3.2 · p. 14

3.3 Dependency Parsing and Multi-Difficulty Question Synthesis

• TRACE builds a dependency graph and creates Easy, Medium, and Hard variants, using fidelity-focused concatenation or localized additive patches to clarify skipped reasoning.Refine Mode follows a Zero-Deletion Policy, preserving the original derivation while adding intermediate steps or correcting local artifacts.Source: §3.3 · pp. 14–15
• Multi-stage checks screen completeness, verify assumptions and prerequisites, test variant consistency, and send a sample for human expert review.Expert feedback is used to refine extraction rules and prompts and suppress hallucinated or logically inconsistent outputs.Source: §3.3 · pp. 16–17

4.2 Evaluation Protocol

• For proof responses, a selector routes aligned solutions to rubric-based step verification and divergent solutions to independent checks of assumptions, coherence, and technical precision.The protocol averages three independent judgments; multiple-choice questions use exact-match binary scoring.Source: §4.2 · p. 19

4.3 Evaluation Results

• Top frontier models cluster around 90% overall accuracy on foundational tasks, while mid-tier and smaller models mostly exceed 70%.The authors describe limited headroom for differentiation on foundational tasks.Source: §4.3 · p. 21, Table 1 · p. 22
• Research-proof scores decline monotonically with difficulty; GPT-5.4 remains above 50% on Hard, but the gap from foundational performance remains large.The authors interpret the contrast as stronger performance on familiar curriculum-level patterns than on long-horizon, out-of-distribution theoretical reasoning.Source: Table 2 · p. 23, §4.3 · p. 23

5.1 Dual-Channel Retrieval Mechanism

• Dual-channel retrieval uses keywords for mathematical constraints and a slogan for proof intent, then re-ranks candidates for strategy transferability and domain compatibility.Candidates are retained when either channel provides a strong signal; proof-aware verification filters mathematically incompatible references.Source: §5.1 · pp. 24–25

5.2 Experiment Setup and Main Results

• The 200-core-proof-problem test compares direct generation with RAG-enhanced generation for GPT-5.4, Qwen3.6-plus, and Gemini-3.1-Pro, using hybrid-retrieved references and a proof-oriented prompt designed to encourage proof transfer.Source: §5.2 · pp. 25–26
• RAG gains are especially pronounced on Medium and Hard problems, where retrieved references help bridge missing intermediate reasoning steps.The authors report that models reconstruct missing intermediate arguments and complete multi-step logical dependencies.Source: §5.2 · pp. 26, 28
• RAG consistently improves all three tested models across difficulty levels and theoretical categories: GPT-5.4 rises from 68.01 to 76.03 (+11.8% relative), Qwen3.6-plus from 54.22 to 63.63 (+17.4%), and Gemini-3.1-Pro from 54.01 to 61.65 (+14.2%).Source: §5.2 · p. 26
• In the Γ-selection case, RAG identifies three obligations—selection constraint, weak-null ATE inequality, and observed-data matching—then constructs a valid full-data law.The baseline matches the observed-data law but leaves null membership incomplete and introduces an unsupported threshold argument.Source: §5.2 · p. 28, Figure 12 · p. 29

6 Application III: Domain-Specific Fine-tuning

• The two-stage training uses Qwen3-4B to test whether general-purpose models can benefit from targeted statistical supervision for more rigorous derivations.The model first trains on general mathematical reasoning tasks and is then aligned with specialized statistical derivation data.Source: §6 · p. 29

6.1 Phase I: General Mathematical Reasoning

• Phase I curates approximately 605k reasoning samples, generates medium-length Chain-of-Thought solutions, verifies them with an LLM judge, and filters responses for validity and format.Sources include mathematical reasoning and instruction-following data.Source: §6.1 · p. 30
• Phase I uses progressive length-based buckets to train on variable-length reasoning traces without compromising the model’s ability to process the final target context length.The strategy avoids starting with a fixed 64K context window.Source: §6.1 · p. 30

6.2 Phase II: Domain-Specific Statistical Alignment

• Phase II stratifies problems by Pass@8, excluding mastered problems, synthesizing correct traces for hard problems, and using preference alignment for mid-tier problems.Training applies SFT on synthetic correct responses before DPO on internal preference pairs.Source: §6.2 · p. 31
• 27.08% relative improvement on Easy tasks, 14.02% on Medium tasks, and 31.89% on Hard tasks versus vanilla Qwen3-4B followed the complete alignment pipeline.The comparison is reported for the statistical research dataset.Source: Table 4 · p. 32, §6.2 · p. 32
• Experiments are limited to compact Qwen3-4B because of computational constraints, and systematic investigation of scaling to larger models is left for future work.The authors suggest the strategy may also benefit larger foundation models, but do not systematically test that scaling behavior.Source: §6.2 · p. 32

• StatEval introduces over 106,000 undergraduate and research-level problems alongside a standardized pipeline for constructing statistical datasets from unstructured text.Source: §7 · p. 33
• State-of-the-art models struggle with research-level tasks, while integrating retrieval-augmented generation and targeted fine-tuning significantly improves statistical reasoning.Source: §7 · p. 33
• StatEval is presented as a rigorous foundation for advancing research-oriented statistical AI tools.Source: §7 · p. 33

• The full datasets are not yet publicly available.Source: §8 · p. 33
• A sample dataset is provided on Hugging Face to verify the results.Source: §8 · p. 33