BLADE: Benchmarking Language Model Agents for Data-Driven Science

• Existing benchmarks emphasize single-answer execution or machine-learning metrics, while scientific analysis requires assessing many intermediary decisions involving data semantics, domain knowledge, and statistics.Source: §1 · p. 1
• BLADE evaluates agents on research questions and datasets by comparing conceptual variables, transformations, and statistical models against expert-grounded analyses.Source: §1 · p. 2, Figure 1 · p. 2
• Its automatic evaluation framework uses representations and matching methods designed to accommodate different ways of expressing analysis decisions.Source: §1 · pp. 2–3
• BLADE contains 188 multiple-choice and 536 ground-truth analysis decisions across 12 real-world datasets and research questions.Source: §1 · p. 2
• LM coverage of ground truth stayed below 13% for statistical models with conceptual variables and below 27% for operationalizing variables, while a ReAct agent consistently improved coverage.Source: §1 · p. 3
• The benchmark provides a first step toward evaluating agents’ capabilities in data-driven scientific discovery and is released with its evaluation framework.Source: §1 · p. 3

• BLADE targets open-ended scientific questions rather than only precise, single-answer code-execution tasks, addressing limited evaluation of agent decision-making.Source: §2 · p. 3
• The benchmark requires agents to understand data semantics, integrate domain knowledge, plan across abstraction levels, and distinguish theoretically or statistically justifiable decisions.Source: §2 · p. 3
• Its evaluation should be automatic, decision-based around intermediary choices, and flexible to multiple equivalent decision representations.Source: §2 · p. 3
• Together, these requirements assess whether agents can plan, develop, and execute a justifiable analysis for a real-world research question.Source: §2 · p. 3

• Research questions and datasets come from scientific publications, meta-analysis papers, and statistics textbooks selected for complex, non-trivial analyses.Source: §3 · pp. 3–4
• Eleven trained analysis experts independently analyzed each dataset, recording decisions and thereby producing multiple analytical approaches.Source: §3 · p. 4
• Experts validated peer and LM-generated decisions as justified or unjustified before combining unique validated decisions into the ground truth.Source: §3 · p. 4
• Expert agreement was 75% for transformations and 80% for conceptual variables, compared with 27% and 13% for LM-generated decisions, respectively.Source: §3 · p. 4
• The expert team resolved ambiguities through discussion and consensus, so the ground truth reflects alternative approaches validated by multiple experts.Source: §3 · p. 4

4 Benchmark Tasks

• BLADE tasks target decisions needed for data-driven science, including conceptual variables, transformations, and statistical models requiring domain, statistical, and data knowledge.Source: §4 · p. 4

4.1 Task 1: Discern Justifiable Decisions

• Task 1 uses multiple-choice questions to test which conceptual variables and transformations are most or least justifiable for a research question and dataset.Source: §4.1 · p. 4
• Task 1 labels answers using expert reviews, with decisions considered justifiable only under unanimous expert agreement and unjustifiable under majority disagreement.Source: §4.1 · p. 4
• Task 1 contains 188 multiple-choice questions with correct and incorrect decision examples.Source: §4.1 · pp. 4–5

4.2 Task 2: Generate an End-to-end Analysis

• Task 2 requires an end-to-end analysis containing conceptual-variable descriptions and types, an executable transformation function, and a statistical model function.Source: §4.2 · p. 5

Matching Conceptual Variables and Statistical Models

• BLADE converts conceptual variables and statistical-model specifications into natural-language representations matched by semantic equivalence.Source: §5 · pp. 5–6

Matching Data Transformations

• Data transformations are represented as column-level data-flow graphs so equivalent execution orders and partial results can receive credit.Source: §5 · pp. 5–6
• GPT-4o decomposes submitted transformation functions into transform units and matches outputs using column values and approximate graph isomorphism.Source: §5 · p. 6

Evaluation of LM Evaluation Modules

• The evaluation modules achieved correctness rates of 92%–97% across conceptual-variable, transformation, modeling-code, and statistical-model matching.Source: §5 · p. 6

Models

• The benchmark evaluates nine models spanning general-purpose, open-source, and coding-specific language models.Source: §6 · p. 6

Experiment Settings

• ReAct agents iteratively inspect notebook execution outputs and generate next-step actions, whereas one-turn models produce analyses from a single prompt.Source: §6 · pp. 6–7
• Experiments use 40 one-turn runs and 20 agent runs per model setting, with temperature 0.8 to encourage diversity.Source: §6 · p. 7

Evaluation Metrics

• Multiple-choice performance is measured by accuracy, while generation uses average precision and coverage@10 combined through an adapted F1-score.Source: §6 · p. 7

World Knowledge and Discrimination

• LMs achieve reasonable conceptual-variable performance and strong multiple-choice performance for general models, whereas coding-specific models struggle with correct decisions.Source: §7 · p. 7, Figure 3 · p. 7

Executable Analyses

• GPT-4o generates executable analyses in 96% of runs, compared with 73% for Mixtral-8x22B and 65% for DeepSeek-Coder.Source: §7 · p. 7, Figure 5 · p. 8

Statistical Models and Operationalization

• Precision is below 35% for statistical models and below 60% for operationalized variables, with coverage@10 below 13% and 27%, respectively.Source: §7 · pp. 7–8

Main Findings

• Coverage remains below 13% for statistical models and below 27% for operationalizing variables, while ReAct consistently improves coverage but remains non-optimal.Source: §7 · p. 8, Figure 4 · p. 8

Basic Analyses

• Many submissions contain empty transforms and match basic decisions with decent precision but poor coverage across runs.Source: §7 · p. 8, Figure 5 · p. 8

Agent Diversity

• Interactive agents increase coverage, performing best overall for transformations and statistical modeling that require detailed data-semantic understanding.Source: §7 · p. 8, Figure 4 · p. 8

BLADE versus HumanEval

• Coverage correlates more weakly with HumanEval than precision, indicating that coding ability does not directly translate into diverse BLADE analyses.Source: §7 · pp. 8–9, Figure 6 · p. 9
• Gemini 1.5 Pro exceeds its HumanEval-implied precision, Mixtral-8x22B excels in precision and coverage, and CodeLlama underperforms on BLADE.Source: §7 · p. 9

Benchmarks for Data Science

• Many data-science benchmarks assess code execution or machine-learning metrics without measuring complex reasoning, external knowledge, or intermediate decisions.Source: §8 · p. 9
• Other benchmarks address causal or quantitative reasoning but often use closed-ended solutions, while DiscoveryBench evaluates hypotheses rather than analysis decisions.Source: §8 · p. 9
• BLADE evaluates open-ended, data-driven research questions across multiple valid analysis approaches.Source: §8 · p. 9

Agents for Science

• The work contributes to broader efforts applying language-model agents to automate scientific discovery and evaluate scientific analyses across domains.Source: §8 · p. 9

9 Conclusion

• BLADE combines research questions, data tables, expert ground-truth analyses, and matching algorithms for automatic, decision-based, input-flexible evaluation.The benchmark is designed to advance LM agents for data-driven scientific tasks.Source: §9 · p. 9
• Current LMs sometimes produce analyses matching the ground truth, but those analyses remain limited in complexity and lack diversity.Source: §9 · p. 9

10 Limitations

• BLADE excludes interpretation of data-analysis results, leaving this important end-to-end capability for future work.The authors note that interpretation can require subjective judgments about the problem and model results.Source: §10 · p. 10
• BLADE does not explicitly evaluate the exploratory parts of an analysis.Source: §10 · p. 10
• The benchmark assumes each dataset is contained in a single potentially extremely large table, which may not fit all research datasets.The authors argue this does not significantly reduce scope because language models commonly perform table joins.Source: §10 · p. 10
• Some evaluation components rely on language models that are known to hallucinate, although the authors report validation efforts and open-source the modules.Source: §10 · p. 10

A.1 Data Collection Recruitment

• The final annotation included 11 analysts recruited through open calls, screened by expertise and analysis reports, with diverse statistical backgrounds and occupations.Analysts reported an average of 6 years of data-analysis experience, and at least five reported high familiarity with common analysis methods.Source: §A.1 · p. 14

A.2 Data Collection Procedure

• The annotation procedure combined training, structured inputs, collaborative review, and LM-generated examples to improve consistency, reliability, and diversity, yielding 118 variable, 246 transformation, and 172 modeling decisions.Analysts received tutorials, toy examples, meetings, peer input, and approximately 40 LM-generated examples per annotator per dataset for correctness judgments.Source: §A.2 · p. 14

A.3 Analysis Decision Representations

• BLADE represents each analysis as alternative transformation graphs plus conceptual variables and statistical models, enabling agent artifacts to be matched against all ground-truth alternatives.Transformation graphs encode transform and column-pointer nodes; conceptual variables link descriptions and types to columns, while models link descriptions and model columns to variables.Source: §A.3 · pp. 14–15

A.4 Decision Matching Procedure

• The matching pipeline uses GPT-4 to convert submitted transformation and modeling code into standardized decision representations before comparing them with ground truth.The conversion includes decomposing transformations into units and translating modeling functions into model specifications with their used columns.Source: §A.4 · pp. 15–16

A.4.1 Matching Transforms

• Because transformations are open-ended, BLADE evaluates both exact value matches and more flexible fuzzy graph-isomorphism matches across alternative transformation graphs.Value matching requires identical resulting column values, whereas fuzzy graph matching compares transform verbs, parameter columns, and dataflow structure despite numerical-parameter differences.Source: §A.4.1 · pp. 16–17

A.4.2 Matching Conceptual Variables

• Conceptual variables match when their types agree and their descriptions are semantically equivalent, using GPT-4o to judge equivalence.The matcher receives JSON-formatted conceptual-variable specifications and returns matching point IDs with similarity values.Source: §A.4.2 · p. 17

A.4.3 Matching Statistical Models

• Statistical models receive coarse semantic matching and a second match based on whether they contain the appropriate conceptual variables.The conceptual-model comparison reflects that justifiable model choices depend on including the right variables.Source: §A.4.3 · p. 17

A.5 Baseline ReAct Agent Details

• The baseline ReAct agent iteratively alternates thought, notebook action, and observation stages before producing a final analysis, using up to 10 steps and models with at least an 8k context window.Experiments used Mixtral-8x22b, GPT-3.5 Turbo, GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet.Source: §A.5 · pp. 17–18

A.7 Evaluation Metrics Details

• Bootstrap estimates average F1-scores over 1,000 resampled iterations and report 95% confidence intervals to account for run-selection variability.Each iteration recomputes average precision, coverage@10, and F1-score for each dataset.Source: §A.7 · p. 19

A.8 Case Studies with Qualitative Insights

• Qualitative case studies found that GPT-4 missed interpretable composite variables, while GPT-3.5 often used irrational interaction terms without corresponding main effects.GPT-4 failed to create variables such as evaluation response rate, whereas GPT-3.5 produced terms such as eval ~ beauty * gender.Source: §A.8 · p. 19

GPT-4o

• GPT-4 generally produced more comprehensive models but sometimes included redundant variables, whereas GPT-3.5 often produced minimal models with one independent variable and no controls.The case-study figures provide examples of these contrasting variable-selection patterns in LM-generated analyses.Source: §A.8 · p. 19, Figure 20 · p. 34, Figure 21 · p. 35