Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data

• QRDATA targets statistical and causal reasoning with real data, addressing the limited evaluation of advanced reasoning applied to data.Source: §1 · pp. 1–2
• QRDATA requires advanced quantitative knowledge and multi-turn reasoning, including interpreting execution results and deciding whether estimates are reliable.The example requires checking data structure before indexing variables and comparing estimates before and after a refutation test.Source: §1 · p. 2
• QRDATA contains 411 questions and 195 data sheets; its auxiliary QRText set contains 290 questions solvable from text descriptions alone.Source: §1 · p. 2
• GPT-4 reaches 58% accuracy with the code interpreter assistant, while Deepseek-coder-instruct reaches 37% with Program-of-Thoughts.Source: §1 · p. 2
• Models struggle with data analysis, especially smaller open-source models, and all models struggle to integrate causal knowledge with provided data.Source: §1 · p. 2

Benchmark construction

• Eight open-source resources were selected for their quantitative-reasoning content, accompanying questions, data, and answers, spanning books, an online resource, and academic-paper datasets.The resources include OpenIntro Statistics, Quantitative Social Science, and datasets from academic papers.Source: §2 · p. 3
• Questions and unique gold answers came from exercises and solutions or were annotated from resource content, with data-sheet descriptions supplied alongside the questions.For existing causal-discovery datasets, the authors collected multiple-choice questions; for causal-estimation datasets, they collected numerical ATE, ATT, and ATC questions.Source: §2 · p. 3
• Most QRDATA questions use one data sheet, while 4% use multiple sheets.The authors report dataset statistics in Table 1 and describe a wide range of covered concepts.Source: §2 · p. 3, Table 1 · p. 3, Figure 3 · p. 3
• Min-K% Prob checks on GPT models and Llama-2 indicate that QRDATA is unlikely to appear in their pretraining corpora.The detection method is attributed to Shi et al. (2023).Source: §2 · p. 3

QRText

• QRText contains 290 questions—100 statistical and 190 causal—that can be answered without data and support comparison with QRDATA.Its statistical-to-causal question ratio is similar to QRDATA's.Source: §2 · pp. 3–4

Evaluation metric

• Numerical answers within ±3% of the gold answer count as correct to accommodate small differences between measurement methods.The paper gives differing causal-effect estimation methods as an example of why minor numerical variation may occur.Source: §2 · p. 4

Table Question Answering

• Table QA tests whether a model trained on TableInstruct can answer questions based on tables.The authors evaluate a competitive Table QA method that fine-tunes models on TableInstruct.Source: §3 · p. 4

Chain-of-Thought

• Chain-of-Thought (CoT) prompts models to reason in natural language from the data-sheet contents, truncating sheets when needed to fit the input limit.ChatGPT extracts final answers from model outputs; a manual check of 50 sampled CoT outputs found 98% answer-extraction accuracy.Source: §3 · p. 4

Program-of-Thoughts

• Program-of-Thoughts (PoT) has models generate Python code that an external computer executes, avoiding the data-sheet length limit of prompt-based processing.Code can also call data-analysis packages for computation.Source: §3 · p. 4

ReAct-style Prompting

• ReAct restricts agent actions to Python execution and returns execution output as an observation, enabling multi-turn reasoning and code revision before a final answer.The model generates thoughts and actions, then either continues from the observation or predicts its final answer.Source: §3 · p. 4

Code Interpreter Assistants

• GPT code-interpreter assistants similarly support multi-turn code generation and execution in a sandbox, ending with a text-form answer.The code interpreter is built into GPT-model assistants.Source: §3 · p. 4

4.2 Main Results

• GPT-4 with the code interpreter assistant achieves the best QRDATA performance, Deepseek-coder-instruct with PoT is the best open-source model, and the best model is 18% worse than human performance.Table 2 reports model accuracies; human performance was measured on a random sample of 50 questions answered by three graduate students who could use a Python executor and search for concepts.Source: Table 2 · p. 5, §4.2 · pp. 5–6, §4.1 · p. 5
• CoT results for some open-source models are close to or worse than random, and numerical-question accuracy with CoT is below 10% at best.Natural-language methods may not receive entire data sheets; even on a subset where full sheets fit, CoT accuracy on numerical questions remains below 20% across models.Source: §4.2 · p. 6
• Deepseek-coder-instruct reaches an 89% code-execution success rate versus CodeLlama-instruct's 73%, while code execution improves numerical-question accuracy for some non-code models.Overall performance for several non-code models nevertheless drops with PoT compared with CoT.Source: §4.2 · p. 6
• ReAct raises average code-execution success for open-source models other than code LLMs from 25.5% to 35.6%, and 10% of sampled outputs fixed initially non-executable code in later steps.GPT-4 and Gemini-Pro perform best with agent reasoning, while smaller open-source models struggle to follow ReAct instructions.Source: §4.2 · p. 6
• Deepseek-coder-instruct outperforms similarly sized models by more than 10%, but its 33B version scores 31.9% with PoT and 27.0% with ReAct, below the 6.7B model.AgentLM and WizardMath gain only marginally over Llama-2-chat, while CodeLlama-instruct underperforms it.Source: §4.2 · p. 6
• Across models, the performance gap over random is larger for statistical than causal questions; even GPT-4 answers only 51% of causal questions correctly.Most human errors were on causal-discovery questions (71%).Source: §4.2 · pp. 5–6

5.2 Difficulty of Causal Reasoning

• Two of the three tested models perform better on multiple-choice QRDATA causal questions without data access than with it.The comparison is between the same questions with and without data access; without data, the questions are not self-contained and models must rely on commonsense.Source: Table 4 · p. 7, §5.2 · p. 7

5.1 Difficulty of Data Analysis

• All models perform 1.8% to 11.5% better on QRText than QRDATA, and same-source OpenIntro questions yield a 6% average improvement, indicating difficulty with data analysis.On QRText, GPT-4 reaches 59.7% accuracy and Deepseek-coder-instruct leads open-source LLMs at 39.0%; GPT-4's code-interpreter advantage over PoT is 0.7% there, versus 8.8% on QRDATA.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Difficulty of Causal Reasoning

• Causal reasoning remains harder than statistical reasoning on QRText: GPT-4 reaches 89% on statistical questions but only about half that accuracy on causal questions.The reported gap across both QRDATA and QRText indicates that causal difficulty is not limited to data analysis.Source: §5.2 · p. 7
• In one GPT-4 example, a wrong plan for calculating conditional probabilities leads to a wrong causal prediction despite successful code execution.The model correctly predicts no causal relationship without data, but is misled when the data is provided.Source: §5.2 · pp. 7–8, Table 5 · p. 8

6.1 Data-related Benchmarks

• FinQA and TABMWP mainly test basic arithmetic, whereas QRDATA requires more advanced quantitative reasoning skills.Table QA benchmarks also test parsing, lookup, and integration of table information.Source: §6.1 · p. 8
• Unlike multi-hop benchmarks that can be solved in one turn, QRDATA requires multi-turn reasoning between the model and data.Source: §6.1 · p. 8
• Data-science code-generation benchmarks do not focus on advanced quantitative reasoning; DAEval includes correlation analysis, but none of these benchmarks includes causal questions.The authors identify DAEval as a concurrent dataset and note that it includes correlation analysis as one question concept.Source: §6.1 · pp. 8–9

6.2 Quantitative Reasoning Abilities

• Earlier work studies quantitative reasoning in math and science (Hendrycks et al., 2021; Cobbe et al., 2021; Lu et al., 2022a; Chen et al., 2023a) and causal reasoning, while this paper extends the inquiry to quantitative reasoning with data.Prior causal-reasoning work examines commonsense reasoning, causal knowledge, and causal-inference skills (Bhagavatula et al., 2019; Qin et al., 2019; Liu et al., 2023; Willig et al., 2022; Gao et al., 2023; Kıcıman et al., 2023; Jin et al., 2023a,b).Source: §6.2 · p. 9

7 Conclusion

• QRDATA evaluates LLM statistical and causal reasoning with real-world data, using questions from teaching and research materials.The authors compare natural-language, program-based, and agent reasoning approaches across a range of models.Source: §7 · p. 9
• Models struggle with data analysis and causal reasoning, and even powerful LLMs have difficulty integrating acquired causal knowledge with the provided data.Source: §7 · p. 9
• The authors call for greater attention to data-based quantitative reasoning and specialized methods, including agent-style training with data input and more quantitative reasoning knowledge.Source: §7 · p. 9

Limitations

• QRDATA focuses on explicit statistical and causal reasoning with data and is limited to English questions and data.The authors look toward future benchmarks covering broader domains and other languages.Source: Limitations · p. 9
• The experiments use representative models and reasoning methods, not all models or frameworks.The authors also identify training-data construction and open-source model fine-tuning as areas for further exploration.Source: Limitations · p. 9

Ethics Statement

• Collected data should not be used commercially.The authors state that data collection follows the permissions of the applicable licenses.Source: Ethics Statement · p. 9
• The authors advise readers not to trust solutions provided by LLMs, citing their imperfect quantitative reasoning with data.Source: Ethics Statement · p. 9

A.1 Question Collection of QRDATA

• QRData causal-discovery questions test pairwise direction and full-graph relations, while its causal-estimation questions cover ATE, ATT, and ATC in IHDP and ATT in Jobs.For IHDP, outcomes and counterfactuals are simulated using real covariates; Jobs treatments come from randomized experiments, enabling true ATT values to be calculated.Source: §A.1 · p. 13

A.6 Error Analysis

• 44% of GPT-4 code-interpreter errors were attributed to knowledge recall, 34% to data analysis, 16% to equation formulation, and 6% to calculation.The analysis sampled 50 incorrect predictions; the authors note that calculation is also a main source of human errors because state-of-the-art causal methods cannot always reach the correct answer.Source: Table 7 · p. 13, §A.6 · p. 14

A.2 Question Collection of QRTEXT

• QRText contains 290 questions: 100 statistical questions from OpenIntro Statistics, 94 causal questions from CLADDER, and 96 causal questions from CORR2CAUSE.The benchmark samples CLADDER's interventional and counterfactual questions, and converts selected yes-or-no causal-estimation questions into numerical questions using metadata.Source: §A.2 · p. 13

A.3 Prompt Design

• Chain-of-Thought prompts include data-sheet content up to 3,500 tokens, whereas Program-of-Thoughts prompts show shuffled data's first ten lines.For questions based on multiple sheets, prompts show each sheet's rows separately; GPT-3.5 Turbo extracts answers after Chain-of-Thought reasoning.Source: §A.3 · p. 14

A.4 Implementation Details

• Inference used temperature 0 and at most 512 new tokens, with ReAct prompting capped at 10 reasoning steps.Experiments ran on NVIDIA A800 GPUs; evaluating a 7B model on QRData took five minutes with CoT/PoT and one hour with ReAct on a single GPU.Source: §A.4 · p. 14

A.5 Case Study

• Tables 14 and 15 present GPT-4's full output on an example QRData case, respectively with and without access to the data.The appendix identifies the paired outputs as a case study but does not state a comparison outcome in these passages.Source: §A.5 · p. 14, Table 14 · p. 20, Table 15 · p. 21