PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Language Models for Statistical Problem Formulation

Chen Wang, Junzhe Zhao, Xin Cong, Wanlu Deng, Ke Deng

arXiv:2609.01982v1cs.AI

TL;DR

The paper addresses the lack of evaluation for how LLMs formulate statistical tasks from informal requests and heterogeneous data. It defines two formulation subtasks, introduces StatFormBench, and finds that current models do not reliably lead across them, with limited and inconsistent prompting gains.

  • Problem

    Existing evaluations generally assume the statistical analysis target is already specified, leaving the upstream formulation of informal requests and heterogeneous data largely untested.

  • Method

    The paper formalizes statistical problem formulation as Statistical Problem Classification plus Variable Identification & Role Assignment and evaluates them with StatFormBench.

  • Results

    No model performs consistently best across the two formulation tasks; the highest zero-shot results are 72.0 fine-grained classification accuracy and 63.2 variable set overlap, achieved by different models.

  • Takeaways & Limitations

    Statistical problem formulation remains beyond the reliable reach of current models, while enhanced prompting yields only limited and inconsistent gains.

  • Takeaways & Limitations

    The benchmark uses solved textbook exercises and a case library rather than real-world statistical consultations, so samples may not fully capture real consulting ambiguity and underspecification.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as assistants for statistical and data science work, yet existing evaluations largely assume the analysis target is already specified. In practice, users arrive with informal goals and heterogeneous data, leaving the model to decide what statistical task is implied and which data are relevant. We first formalize this upstream step as Statistical Problem Formulation and decompose it into two subtasks: (1) Statistical Problem Classification and (2) Variable Identification & Role Assignment. We then introduce StatFormBench, a benchmark built from five cross-domain statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles. It contains 1,013 samples spanning 20 coarse-grained and 85 fine-grained statistical problem categories. Across 14 open- and closed-source LLMs, the best zero-shot models reach only 72.0 fine-grained classification accuracy and 63.2 variable set overlap. No model performs consistently best across the two subtasks, while enhanced prompting strategies yield only limited or inconsistent gains. We release the benchmark data on Hugging Face at https://huggingface.co/datasets/THU-CongLab/StatFormBench and the evaluation code on GitHub at https://github.com/THU-CongLab/StatFormBench.

1 Department of Statistics and Data Science, Tsinghua University

The section lists contact email addresses associated with the Department of Statistics and Data Science at Tsinghua University.

  • The listed contact addresses include wangchen_23 and zhaojz26 at mails.tsinghua.edu.cn.
  • The section identifies the affiliation as the Department of Statistics and Data Science at Tsinghua University.
  • The listed contact addresses include congxin1995, wanludeng, and kdeng at tsinghua.edu.cn.

1 Introduction

The paper identifies statistical problem formulation as an upstream challenge in which models infer statistical tasks and relevant variable roles from informal requests and heterogeneous data. It introduces a benchmark that decomposes this challenge into problem classification and variable identification with role assignment.

  • 12 practical requests may require distinguishing a hypothesis test from a superficially plausible prediction task and selecting only the data objects needed for valid analysis.
  • Statistical problem formulation maps informal analysis requests to a statistical problem type and relevant data objects with analytical roles.
  • Models must interpret ambiguous, domain-specific language, heterogeneous data representations, and a broad, weakly delimited space of statistical problem types.
  • STATFORMBENCH evaluates this capability using diverse statistical problem types, data representations, and scenario styles from textbooks and real-world data science cases.
  • Statistical Problem Classification identifies the expressed problem type, while Variable Identification & Role Assignment selects relevant data objects and assigns their analytical roles.

2 Related Work

Prior benchmarks mainly evaluate statistical reasoning or data science execution after analytical goals are specified, whereas statistical problem formulation evaluates how models infer those goals and relevant data.

  • Existing statistics and data science benchmarks generally assume that the analytical goal is already specified or strongly implied.
  • Statistical knowledge benchmarks test reasoning with statistical concepts, method applicability, or variable selection for pre-formulated questions.
  • Data science execution benchmarks evaluate code generation and workflows for specified preprocessing, analysis, modeling, or research tasks.
  • Optimization benchmarks explicitly map natural-language descriptions to formal representations, typically using objectives and constraints within a fixed template.
  • Unlike optimization formulation, statistical formulation infers analytical questions across heterogeneous problem types and data objects without a fixed schema.

3 Task Formulation

Statistical problem formulation maps practical context, user requests, and heterogeneous data to a statistical problem type and the variables needed, with their analytical roles. The benchmark separates this process into hierarchical problem classification and variable identification with role assignment.

  • 3 Task Formulation: Statistical problem formulation is decomposed into classification and variable identification with role assignment over input (B, Q, D).B describes practical background, Q states the request, and D refers to data that may vary in form and content.
  • 3 Task Formulation: The input data can include tables, textual descriptions, separate files, raw observations, summary statistics, and contingency tables.
  • 3.1 Task 1: Statistical Problem Classification: Problem classification predicts a two-level label distinguishing coarse analytical intent from specific statistical problem types.The coarse level includes categories such as descriptive statistics, classification and prediction, and hypothesis testing and interval estimation.
  • 3.1 Task 1: Statistical Problem Classification: The task identifies statistical problem types rather than solving methods because one problem can admit multiple methods.This keeps problem identification distinct from method selection.
  • 3.2 Task 2: Variable Identification and Role Assignment: Variable identification produces the set of data variables required to solve the user request, representing each variable with an identifier, meaning, values, and role.
  • 3.2 Task 2: Variable Identification and Role Assignment: Variable roles are predictor, response, both, or N/A, depending on whether the implied relationship is directional and how each variable participates in the analysis.N/A covers needed variables without directional roles, including identifiers, filters, and inputs to unsupervised analyses.
  • 3.2 Task 2: Variable Identification and Role Assignment: Roles are question-conditional: the same data may receive different analytical roles under different user requests.

4 Benchmark Construction

STATFORMBENCH combines textbook exercises with open-ended data-science cases, converts them into standardized labeled samples, and applies filtering, annotation, rewriting, and review. The resulting benchmark contains 1,013 samples spanning diverse categories and variable counts.

  • 4.1 Data Sources: The benchmark combines five cross-domain statistics textbooks with a case-based data science library.Textbook materials provide canonical problems with compact data, while case-library samples provide open-ended contexts, larger files, and diverse formats.
  • 4.2 Construction Pipeline: Both sources are converted into samples (B, Q, D, c, V), containing evaluation inputs, a problem-type label, and a reference variable set.
  • 4.2 Construction Pipeline: Raw materials are parsed into entries and multi-subquestion materials are split into individual samples by pairing each subquestion with its answer and shared background.
  • 4.2 Construction Pipeline: Quality filtering removes value lookup, pure concept recall, and questions dependent on answers to previous problems.
  • 4.2 Construction Pipeline: The taxonomy contains 20 coarse-grained and 85 fine-grained categories, with labels proposed by an annotator LLM and adopted or merged by domain experts.
  • 4.2 Construction Pipeline: Scenario rewriting preserves analytical intent, numerical values, and named entities while removing direct category cues and adding minor irrelevant context.
  • 4.2 Construction Pipeline: All 1,013 samples are manually reviewed for problem categories and variable sets against source reference answers.Textbook samples receive two expert review rounds, with senior review resolving disagreements.
  • 4.2 Construction Pipeline: 1,013 samples comprise 629 textbook samples and 384 case-library samples, with broad category coverage and a long tail in variables per sample.The dataset has substantial mass in data visualization, mathematical calculation, and hypothesis testing and interval estimation.

5 Evaluation Metrics

The evaluation measures statistical problem classification and variable identification, including overlap, precision, recall, and role assignment. Variable matching uses exact value fields, while incorrect or missing variables are penalized.

  • Statistical Problem Classification: Coarse-grained and fine-grained accuracy evaluate statistical problem classification at different levels of specificity.A correct fine-grained prediction also counts as correct at the coarse-grained level, so their gap reflects missed specificity.
  • Variable Identification: Predicted variables are matched to reference variables when their value fields are exactly equal.Unmatched predictions count as false positives, and unmatched reference variables count as false negatives.
  • Variable Identification: The Jaccard coefficient of variable sets measures overall overlap between predicted and reference variable sets.The metric compares the matched variables with the union of predicted and reference variables.
  • Variable Identification: Variable precision and recall distinguish over-prediction from missed variables.These metrics complement the overall overlap measure by separating extra predictions from unrecovered reference variables.
  • Variable Role Identification: Variable role identification measures the fraction of reference variables that are recovered with the correct roles.VRI jointly penalizes missed variables and incorrect role labels.

6 Experiments

Across 14 LLMs, statistical problem formulation remains difficult: models vary by task, data source, category, and prompting strategy, with no consistently dominant system.

  • Main results: 72.0 fine-grained classification accuracy was the highest reported zero-shot score, achieved by Gemini 3.1 Pro.Claude Opus 4.6 and Gemini 3 Flash followed at 69.5 and 68.4.
  • Main results: 63.2 JCV was the highest variable-set overlap, achieved by Claude Opus 4.6, whose model ranking differed from classification.Claude Opus 4.6 also led RV and VRI, while Gemini 3 Flash led PV.
  • Overall finding: No model consistently led across both formulation tasks, and prompting strategies yielded only limited or inconsistent gains.Performance also diverged within model families and across individual variable-related metrics.
  • Data source impact: Textbook samples had ACCFG of 74.1–84.1 and JCV of 33.6–51.0, whereas case-library samples had JCV of 82.4–83.6 and ACCFG of 34.6–54.7.Textbooks support classification through clearer categories but require more careful variable extraction; case libraries reverse this pattern.
  • Prompting strategies: Category definitions improved fine-grained classification by 1.9–3.8 points, while effects on JCV and VRI were smaller and mixed.Few-shot examples produced modest or model-dependent changes and did not substantially improve overall formulation.
  • Main results: Across models, coarse-grained accuracy exceeded fine-grained accuracy, indicating difficulty distinguishing closely related subtypes after identifying broad intent.Confusions such as model comparison versus model interpretation often remained within the correct first-level category.

7 Conclusion

The paper frames statistical problem formulation as an LLM evaluation target and introduces StatFormBench to test it across realistic statistical scenarios. Experiments show that current models remain unreliable, with task-specific strengths and limited prompting gains.

  • Contribution: The study decomposes statistical problem formulation into statistical problem classification and variable identification with role assignment.The two tasks map informal analytical needs to a problem type and relevant data objects with analytical roles.
  • Benchmark: StatFormBench contains 1,013 samples from five cross-domain statistics textbooks and a data science case library.It covers 20 coarse-grained and 85 fine-grained categories using scenario-style language.
  • Findings: The highest zero-shot fine-grained classification accuracy was 72.0, while the highest variable set overlap was 63.2, achieved by different models.Fine-grained performance varied substantially across categories, and prompting strategies produced limited and inconsistent gains.
  • Conclusion: Statistical problem formulation remains beyond the reliable reach of current models.This conclusion follows from the absence of a consistently best model across the two formulation tasks.

Limitations

The benchmark trades some realism for accessible, curated data and faces limitations in representativeness, category balance, and bilingual evaluation.

  • Samples come from solved textbook exercises and a case library rather than real-world statistical consultations.Constrained scenario rewriting improves realism, but confidentiality and accessibility constraints prevent direct use of many consultations.
  • Rewritten samples may still underrepresent the ambiguity and underspecification of real statistical consulting.
  • Category imbalance concentrates many samples in common types such as data visualization and hypothesis testing, reducing reliability for low-frequency categories.
  • The textbook and case-library subsets originate in English and Chinese respectively, so translation may introduce unexamined cross-lingual effects.

Ethical Statements

The supplied passages describe a curated benchmark-construction pipeline that extracts, filters, annotates, rewrites, and organizes statistical problems across multiple data sources.

  • Textbook pipeline: The pipeline extracts questions, answers, data, and metadata from textbook problems, followed by manual correction and proofreading.MinerU produces structured outputs, while annotators fix OCR, equations, and missed tables.
  • Textbook pipeline: An LLM splits shared scenarios into independent analytical questions, and deterministic flattening preserves the shared background and data.
  • Quality control: Quality filtering removes overly basic, incomplete, or unanswered exercises before annotation.
  • Annotation: Initial labels assign taxonomy categories and extract answer-grounded variables using a schema containing identifiers, values, classes, roles, and descriptions.The variable-extraction rules include raw columns, grouped statistics, contingency tables, sample sizes, and success counts while excluding model parameters and test statistics.
  • Scenario rewriting: Scenario rewriting converts statistical terminology into intuitive language while preserving numerical values and checking for intent drift.
  • Taxonomy: The taxonomy contains 20 first-level and 85 second-level categories based on canonical statistical types and collected samples, without claiming exhaustive coverage.

D Additional Dataset Statistics

Figure 7 presents the benchmark’s category distribution at coarse and fine granularity, with fine-grained composition shown within displayed coarse categories.

  • Figure 7 uses an inner ring for coarse-grained categories and an outer ring for their fine-grained composition.
  • Coarse-grained categories containing no more than 20 samples are grouped as Others.

E.1 Exact-Value Matching Implementation and Validation

The appendix specifies exact-value variable matching and validates its effect, then reports supplementary comparisons across prompting strategies, data sources, and category types.

  • Exact-value matching: Variable matching uses serialized value fields as exact keys, without normalization or semantic matching.This prevents synonymous names or descriptions from affecting the score but requires identical serialized contents.
  • Validation: The validation study samples 60 cases from 387 challenging textbook samples and uses PhD-level expert review to identify semantically correct matches rejected by exact matching.
  • Validation: About 10% of sampled outputs for each model contain matching-induced errors, which do not explain the substantially lower main-experiment JCV scores.
  • Supplementary results: Adding examples does not produce consistent gains across the two evaluated tasks and six metrics.
  • Supplementary results: Classification accuracy is higher on textbook samples, whereas variable-related scores are higher on case-library samples.
  • Supplementary results: Performance is higher when analytical objectives are explicit and lower when classification depends on subtle distinctions between related intents.
Loading 2609.01982v1…