PaperlayerView exact revision on arXiv

Source-linked AI summary

SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals

Jingyun Jia, Antoine Remond-Tiedrez, Aaron Alvarez, Joshua Shunk, Rich Caruana, Ben Lengerich

arXiv:2609.34228v1cs.LG

TL;DR

Public-table benchmarks can conflate evidence recovery with memorized dataset knowledge, while reliable ground truth for statistical questions is costly to curate. SLEUTHBENCH injects and validates controlled phenomena in public tables and computes answers automatically; models perform well on data-quality questions but less well on feature contributions, with the Empirical Layer improving feature-contribution accuracy.

  • Problem

    Evaluating statistical discovery requires verifiable answers, but public-table familiarity can influence LLM responses and expert ground-truth curation for real tables is costly.

    Source: §1 · pp. 1–2

  • Method

    SLEUTHBENCH injects controlled statistical phenomena into public tables, validates the resulting patterns, and computes reference answers; its Empirical Layer provides structured statistical evidence.

    Source: §1 · p. 2

  • Results

    Feature-contribution accuracy rises from 41.9% to 68.0% with the Empirical Layer, while Python-only agents achieve 83.8% accuracy on data-quality questions but 41.9% on feature contributions.

    Source: §4.2 · pp. 6–7, §5.3 · p. 8

  • Takeaways & Limitations

    The benchmark and Empirical Layer provide a foundation for studying how data-science agents find and use statistical evidence.

    Source: §6 · p. 9

Abstract

from arXiv · show

Evaluating statistical discovery by large language model (LLM) agents requires verifiable analytical ground truth. Establishing such ground truth for real-world datasets is costly, and prior knowledge of public datasets can influence agent responses. We introduce SLEUTHBENCH, a benchmark that addresses both problems by injecting controlled data-quality problems and feature effects into public tabular datasets: the injected pattern determines the answer, so reference answers are computed automatically and memorized knowledge of the original table is insufficient, while the table keeps its background structure. The injected patterns are modeled on phenomena reported in real data analyses. The benchmark defines 17 question templates in two families: data-quality questions and feature-contribution questions. We evaluate six state-of-the-art LLMs that analyze the data using a Python coding tool, on data-science and business phrasings of 70 validated dataset-template combinations, yielding 1680 graded responses in total. The models detect data-quality problems reliably (83.8% accuracy) but recover feature contributions poorly (41.9%). Finding how features shape the target requires searching over both candidate variables and analytical procedures. To address this issue, we propose the Empirical Layer, a set of precomputed statistical artifacts comprising summaries, fitted feature and interaction effects, and dataset descriptions, which exposes candidate patterns for direct inspection. Access to these artifacts raises feature-contribution accuracy from 41.9% to 68.0%.

1 INTRODUCTION

SLEUTHBENCH targets statistical patterns distributed across observations, addressing the difficulty of verifying analytical answers on public tables where memorized dataset knowledge can influence performance. Controlled interventions preserve source-table structure while making the injected phenomenon determine automatically computed answers.

1 INTRODUCTION

  • Public-table accuracy can reflect both prior dataset knowledge and the ability to recover evidence from the supplied table.Earlier benchmarks often focus on table-value operations, while statistical phenomena such as nonlinear effects and interactions are implicit across observations.Source: §1 · pp. 1–2
  • SLEUTHBENCH injects prescribed effects or data-quality anomalies into public tables and validates that the intended pattern is detectable and unambiguous.The source retains background structure, while the intervention specifies answer-defining evidence and supports automatic reference answers.Source: §1 · p. 2
  • Across six LLMs, the Empirical Layer raises feature-contribution accuracy from 41.9% to 68.0% and overall accuracy from 64.6% to 76.7%.The benchmark includes 17 templates covering data-quality problems and feature contributions, including single-feature effects and pairwise interactions.Source: §1 · p. 2

2 RELATED WORK

Prior work spans table question answering, data-analysis workflows, and statistical reasoning; SLEUTHBENCH complements these lines by testing recovery of controlled statistical phenomena from tables. The Empirical Layer draws on exploratory statistics and interpretable-model outputs.

2 RELATED WORK

  • SLEUTHBENCH complements agent systems by specifying a target phenomenon and reference answer while leaving agents free to choose their analysis procedure.Related agent systems support code execution, iterative refinement, or analytical methods through different workflow designs.Source: §2 · pp. 2–3
  • Existing benchmarks cover table question answering, code, interactive analysis, statistical reasoning, and data-driven discovery; SLEUTHBENCH adds controlled phenomena validated in public source tables.These benchmarks use varied construction and evaluation approaches, including functional tests, execution environments, synthetic tasks, and semi-automated ground truth.Source: §2 · p. 3
  • The Empirical Layer combines statistical summaries with interpretable-model outputs, including feature effects and pairwise interaction surfaces.Exploratory analysis examines distributions, relationships, and unusual observations; Explainable Boosting Machines decompose predictions into inspectable effects.Source: §2 · p. 3

3 SLEUTHBENCH: A BENCHMARK FOR STATISTICAL DISCOVERY

SLEUTHBENCH evaluates recovery of prescribed statistical phenomena using edited public tables, questions, and verifiable answers. Its construction combines compatible templates, validated injections, and deterministic answer computation, while its design also addresses realism, memorization, and refreshability.

3.1 BENCHMARK OVERVIEW

  • SLEUTHBENCH instances pair an edited table and natural-language question with a verifiable answer; reusable templates specify the pattern, eligible variables, question, and answer form.The benchmark uses 1,000-row tables from six public datasets, and each accepted instance has data-science and business question framings that share the table and answer.Source: §3.1 · pp. 3–4

3.2 PHENOMENON INJECTION AND QUALITY CONTROL

  • Construction applies a randomized injection, validates the resulting pattern against template-specific criteria, and computes a reference answer only for accepted candidates.For an inverted-U effect, the answer combines the feature from the edit record with a peak value computed from two fitted smooth curves on the edited table.Source: §3.1 · p. 3, §3.2 · pp. 4–5

3.3 INJECTION DESIGN

  • The two task families mirror data analysts’ inspection of data-quality problems and analysis of how features affect a target; injections recreate phenomena those analyses are meant to detect.Specific injected patterns make recovery require analysis rather than recall of a previously seen table.Source: §3.3 · p. 5

3.4 COMPARISON WITH EXISTING BENCHMARKS

  • Controlled injections into public source tables balance ground-truth effort with retained background structure, while reusable procedures reduce per-instance curation.SLEUTHBENCH changes answer-defining patterns so original-table recall is insufficient, although schema knowledge and domain priors may still help.Source: §3.4 · pp. 5–6
  • Rerunning the pipeline with a new seed produces fresh instances from the same source tables, unlike refresh strategies that depend on newly released material.The approach follows from the reusable injection, validation, and answer-computation procedures.Source: §3.4 · p. 6

4 SLEUTHBENCH REMAINS CHALLENGING FOR PRETRAINED LLMS

With Python access, six LLMs were evaluated on validated table-question instances, and they performed much better on data-quality questions than on feature contributions. The section highlights the search and prompt-specification challenges behind weaker feature-contribution results.

4.1 EXPERIMENTAL SETUP

  • Six LLMs analyzed the same 70 validated dataset–template combinations, each framed as a data-science and a business question, using a Python tool on the edited table.Accuracy counted only CORRECT responses; numerical and column–value answers used a 5% relative tolerance, while structured answers required more exact matches.Source: §4.1 · p. 6

4.2 RESULTS AND CHALLENGES

  • Pooled Python-only accuracy was 64.6%; data-quality accuracy was 83.8%, compared with 41.9% for feature-contribution questions.Per-model accuracy ranged from 50.7% to 80.0% overall and from 18.8% to 71.9% on feature-contribution questions.Source: §4.2 · pp. 6–7
  • Statistical discovery requires choosing both candidate variables and an analytical procedure, and different criteria can produce different feature rankings.Agents must match their procedure to the question’s operational definition.Source: §4.2 · p. 7
  • Python-only accuracy was 67.1% for data-science questions and 62.1% for business counterparts, which may require translating an operational concern into a statistical diagnostic.Source: §4.2 · p. 7
  • Familiar column names and public-dataset regularities can suggest answers, but interventions on feature patterns make those priors insufficient to determine the reference answer.Source: §4.2 · p. 7

5 LLMS AUGMENTED WITH STRUCTURED DATA ANALYSIS TOOLS

The Empirical Layer adds precomputed statistical artifacts to Python analysis, with other evaluation conditions held constant. It raises pooled accuracy, with the largest gains on feature-contribution questions and business framings.

5.1 EMPIRICAL LAYER

  • The Empirical Layer combines table statistics and diagnostics with EBM feature importance, fitted single-feature effects, and interaction effects; Figure 2 shows a Bike Sharing example.Source: §5.1 · p. 7, Figure 2 · p. 8

5.2 EXPERIMENTAL SETUP

  • The comparison pairs Python alone with Python plus the Empirical Layer on the same questions, holding models, tables, answers, and grading constant.At each step, the agent chooses which tool to call.Source: §5.2 · p. 7

5.3 RESULTS

  • Adding the Empirical Layer raises pooled accuracy from 64.6% to 76.7%, or from 543 to 644 correct answers.Source: §5.3 · p. 8
  • Feature-contribution accuracy rises from 41.9% to 68.0%, while data-quality accuracy moves only from 83.8% to 84.0%.The respective correct-answer counts were 161 versus 261 for feature contributions and a net gain of 1 for data quality.Source: §5.3 · p. 8
  • All six models improve overall by 2.1 to 16.4 percentage points, with feature-contribution gains of 6.3 to 35.9 points and data-quality changes no greater than 2.6 points.GPT-5.6 Sol, the strongest Python-only model, moves from 71.9% to 78.1% on feature-contribution questions.Source: §5.3 · p. 8
  • Business questions gain 14.8 percentage points overall versus 9.3 for data-science questions; on feature-contribution templates, gains are 32.3 versus 19.8 points.Feature-contribution accuracy moves from 32.8% versus 51.0% for business versus data-science prompts to 65.1% versus 70.8%.Source: §5.3 · p. 8

6 CONCLUSION

SLEUTHBENCH evaluates statistical discovery with controlled interventions in public tables, and the Empirical Layer supplies structured evidence for finding patterns. Across six LLMs, the layer raises feature-contribution and pooled accuracy while leaving data-quality accuracy unchanged.

6 CONCLUSION

  • Feature-contribution accuracy rises from 41.9% to 68.0%, and pooled accuracy from 64.6% to 76.7%, with data-quality accuracy unchanged.The conclusion attributes these results to access to the Empirical Layer across six LLMs.Source: §6 · p. 9

AI USE STATEMENT

  • The AI-use statement reports tools used for research feedback, implementation, figure creation, coding, drafting, editing, literature identification, and reference formatting.The authors state that they reviewed AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendix specifies templates, source-data preparation, reference answers, validation, paired evaluation and grading, detailed results, and an interaction case study.Appendix A catalogs templates, injections, and paired question wordings; Appendix E reports detailed results.Source: APPENDIX · p. 13

A TEMPLATES, INJECTIONS, AND QUESTIONS

  • SLEUTHBENCH defines 17 reusable question templates implemented by 14 injectors, with 70 validated dataset–template combinations entering evaluation.Each template specifies compatible variables, an injection, two question framings, an answer format, and validation criteria.Source: A TEMPLATES, INJECTIONS, AND QUESTIONS · pp. 13, 17

B SOURCE DATA AND PREPARATION

The experiments use standardized 1,000-row versions of six public datasets, spanning several application settings and mixed variable types.

B SOURCE DATA AND PREPARATION

  • Metro and Steel timestamps become month variables, and Metro’s literal holiday category None is recoded as No Holiday.The experiments use the 1,000-row standardized tables.Source: B SOURCE DATA AND PREPARATION · p. 17
  • Each experimental base table has 1,000 rows before injection; the six sources include five observational datasets and AI4I, synthetic data designed to reflect industrial predictive maintenance.The datasets cover transportation, housing, maintenance, and energy-use settings, with numerical and categorical variables.Source: B SOURCE DATA AND PREPARATION · p. 17, Table 5 · p. 18
  • Bike Sharing is associated with Fanaee-T & Gama (2014), while California Housing derives from census data studied by Pace & Barry (1997).Source: B SOURCE DATA AND PREPARATION · p. 18

C CONSTRUCTION AND VALIDATION DETAILS

Benchmark construction applies an intervention to a prepared table, validates the resulting signal, and computes a reference answer for accepted instances.

C CONSTRUCTION AND VALIDATION DETAILS

  • Each instance begins with a compatible template and prepared base table; injection edits a copy, records the intervention, and leaves incompatible or unsuccessful attempts out.Distinct injector settings use a fixed random seed, and each edited table contains one intervention.Source: C CONSTRUCTION AND VALIDATION DETAILS · p. 18
  • Task-specific validators measure the intended signal and check criteria such as group size, shape, strength, or separation from competing features.Candidate interventions may be tried until one passes; the instance is rejected if none passes within the attempt limit.Source: C CONSTRUCTION AND VALIDATION DETAILS · p. 18
  • For each validated instance, a template-specific answer function uses the edited table, slot assignments, and intervention record to write the reference answer.Only instances with passing validation and a computed answer enter model evaluation; paired phrasings share the same answer definition.Source: C CONSTRUCTION AND VALIDATION DETAILS · p. 18

D.1 PAIRED EVALUATION AND COVERAGE

The paired evaluation compares Python-only analysis with access to the Empirical Layer on the same model–question instances. Responses are graded against computed reference answers, with fixed prompts and explicit scoring rules.

D.1 PAIRED EVALUATION AND COVERAGE

  • 1,680 responses cover six models, each evaluated on 140 questions from 70 dataset–template combinations under each of two tool conditions.The 140 questions comprise data-science and business phrasings; each condition has 840 model–question cases.Source: §D.1 · p. 19
  • Agents retrieved structured statistics, EBM outputs, or language-based metadata in 611/840 P+E responses (72.7%), making the comparison one of access to the combined interface.Those responses involved 4,120 retrieval calls in total.Source: §D.1 · p. 19

D.2 GRADING

  • Accuracy counts only CORRECT grades; grading combines deterministic checks and an LLM judge, with numerical answers using a 5% relative tolerance.Reported results use pipeline grades without manual adjustment; GPT runs stopped by iteration or token limits remain in counts as INCORRECT.Source: §D.2 · p. 19

D.3 PROMPT FORMATS

  • Evaluation prompts require concise answers from the preloaded dataframe and declared tools, and prohibit filesystem, network, and environment access.The evaluation prompt is given to each model; the grading prompt is given to the LLM judge with the question, reference answer, and response.Source: §D.3 · p. 19

E DETAILED RESULTS

The detailed-results materials define the comparison conditions and identify the accuracy and grade-distribution breakdowns reported in Tables 6 and 7.

E DETAILED RESULTS

F CASE STUDIES AND AGENT TRAJECTORIES

The case study contrasts Python-only exploration with Empirical Layer use for a Metro Interstate Traffic Volume interaction question. The layer-assisted trajectory selects the reference pair after inspecting stored interaction artifacts.

F CASE STUDIES AND AGENT TRAJECTORIES

  • Python-only exploration compares several interaction proxies, including empirical mutual information, before selecting weather description, month.Source: F CASE STUDIES AND AGENT TRAJECTORIES · pp. 20, 22
  • Tables 8–10 describe accuracy breakdowns by model, question framing, template family, and pooled framing; the supplied captions state no numerical outcomes.Table 11 identifies the Luna Metro Interstate Traffic Volume business-language case and its reference pair, month, temp.Source: Table 8 · p. 21, Table 9 · p. 21, Table 10 · p. 21
  • The Empirical Layer trajectory compares six EBM interaction graphs and selects month, temp, matching the reference pair.The agent also inspects schema and metadata and retrieves target associations.Source: F CASE STUDIES AND AGENT TRAJECTORIES · pp. 20, 22, Table 11 · p. 22
  • The stored temp–month EBM term has the largest interaction importance, consistent with the layer-assisted answer.The Python-selected weather description–month pair spans 336 joint cells across 1,000 rows; sparse counts may favor it under uncorrected empirical mutual information.Source: F CASE STUDIES AND AGENT TRAJECTORIES · p. 22

Related papers

Loading 2609.34228v1…