PaperlayerView exact revision on arXiv

Source-linked AI summary

The Mighty ToRR: A Benchmark for Table Reasoning and Robustness

Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer

arXiv:2502.19412v3cs.CL

TL;DR

Prior work offered limited evidence about LLM table reasoning and did not systematically test robustness to table-format variation. ToRR evaluates table tasks across serialization formats and structural perturbations. The results show brittle model behavior, while testing multiple prompt configurations improves evaluation reliability.

  • Problem

    Prior evaluations were often limited to particular tasks, narrow domains, or few models and did not systematically assess robustness to table-format variation.

    Source: LLMs · pp. 1–2

  • Method

    ToRR evaluates tabular tasks across seven table serializations and four structural perturbations, and measures robustness across prompt configurations.

    Source: §2.1 · p. 3, §2.2 · p. 4

  • Results

    Models show extreme sensitivity to minor table-format variations, although no particular format consistently yields better performance.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Incorporating multiple prompt configurations improves evaluation reliability and can help address the limitations of smaller test sets.

    Source: §7 · p. 8

    ToRR covers tables embedded in prompts that can be parsed into standard formats, not scenarios requiring independent extraction or search or handling hierarchical tables.

    Source: §8 · p. 9

Abstract

from arXiv · show

Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to adopt. To address this gap, we create ToRR, a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks. The benchmark includes 10 datasets that cover different types of table reasoning capabilities across varied domains. ToRR goes beyond model performance rankings, and is designed to reflect whether models can handle tabular data consistently and robustly, across a variety of common table representation formats. We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR. Our results reveal a striking pattern of brittle model behavior, where even strong models are unable to perform robustly on tabular data tasks. Although no specific table format leads to consistently better performance, we show that testing over multiple formats is crucial for reliably estimating model capabilities. Moreover, we show that the reliability boost from testing multiple prompts can be equivalent to adding more test examples. Overall, our findings show that table understanding and reasoning tasks remain a significant challenge.

1 Introduction

ToRR addresses the limited and fragmented evaluation of LLM table reasoning, including the lack of systematic testing for robustness to table-format variation. Across diverse table tasks, existing models show brittle, inconsistent performance, while no single format consistently yields superior results.

1 Introduction

  • Prior evaluations were typically limited to specific tasks, narrow domains, or few models and did not systematically assess robustness to table-format variation.This left a gap in comprehensive understanding of LLM capabilities in structured-data reasoning.Source: LLMs · pp. 1–2

LLMs

  • ToRR evaluates table reasoning and robustness across 10 datasets from six diverse tabular tasks and multiple domains.Its design examines table serialization formats and structural perturbations.Source: LLMs · p. 2, benchmark · p. 2

benchmark,

  • Existing models struggle with table understanding and reasoning, respond brittly and inconsistently, and show no consistently superior table format.The evaluation covers 14 leading LLMs spanning seven model families.Source: benchmark · p. 2
  • Testing more prompt configurations consistently enhances evaluation reliability and can help compensate for smaller test sets.The paper’s meta-evaluation focuses on incorporating prompt configurations into evaluation.Source: benchmark · pp. 2–3
  • The benchmark also provides a leaderboard, complete model inference outputs, and code.Source: benchmark · p. 3

2 ToRR Construction

ToRR selects diverse table tasks and probes models across equivalent table formats and structural perturbations. It defines performance and robustness measures and evaluates models over multiple examples and prompt configurations.

2 ToRR Construction

  • ToRR selects challenging downstream tasks that use text-serializable tables, span diverse domains, and require different levels of textual and numerical reasoning.Knowledge extraction is required across all datasets.Source: §2 · p. 3

2.1 Prompt Configurations

  • Each input table is tested with 7 serializations and 4 structural perturbations, including row shuffling or transposition.The perturbations are combined with the serializations as orthogonal configuration dimensions.Source: §2.1 · p. 3

2.2 Metrics

  • Performance P averages example scores across prompt configurations, while robustness R is one minus the average per-example score range across configurations.The dataset-defined score function S ranges from 0.0 to 1.0.Source: §2.2 · pp. 3–4

2.3 Setup

  • The evaluation tests 14 models on up to 100 examples per dataset under 35 prompt configurations.TableBench FC contributes 79 examples; WikiTQ is truncated to one-shot prompting because of length.Source: §2.3 · p. 4

3 Results & Analysis

ToRR finds that models achieve limited table-task scores and vary substantially across prompt configurations. Performance patterns also differ across model families, tasks, and prompt choices.

3.1 Model Capabilities

  • 0.50 is the best absolute score; leading models are more robust, yet scores vary widely across 35 configurations and extremes yield different performance estimates.Most pairwise model comparisons show small, often nonsignificant practical differences.Source: §3.1 · pp. 4–5, Table 2 · p. 5, Figure 3 · p. 5

3.2 Model Performance Trends

  • Task strengths differ: claude-3-5-sonnet leads classification examples, whereas mixtral-8x22b-instruct performs better in table-to-text examples.The comparison includes TabFact for classification and QTSumm for table-to-text.Source: §3.2 · p. 5
  • Closed models outperform open models across examined tasks, while within-family score differences average 0.07 and size does not always predict cross-family performance.qwen2-72b-instruct outperforms llama-3.1-405b-instruct in both performance and robustness.Source: §3.2 · p. 5

3.3 Performance by Prompt Configuration

  • No serializer consistently outperforms others, although a given model’s overall performance can differ by up to 0.06 across serializers.The serializer effects are weak and model-specific.Source: §3.3 · p. 6
  • The tested structural perturbations have no consistent effect, neither reliably increasing nor decreasing model performance.Source: §3.3 · p. 6

4 Properties of ToRR

ToRR’s properties are assessed through statistical testing, dataset separability, and agreement among dataset-level model rankings. The results support using varied datasets while showing that agreement patterns differ across tasks.

Statistical Testing

  • Many pairwise model comparisons are statistically significant, though large sample size may explain differences between p-values and effect sizes.Source: §4 · p. 6

Benchmark and Dataset Separability

  • 79% separability across aggregated ToRR examples supports the need for diverse datasets.Individual dataset scores vary: WikiTQ exceeds 71%, while TableBench FC scores 38%.Source: §4 · p. 6

Dataset Agreements

  • Most datasets show medium-to-high ranking agreement, while table-to-text datasets diverge and SciGen and NumericNLG align closely.Source: §4 · p. 6

5 Implications for Reliable Evaluation

ToRR tests how semantically equivalent prompt configurations affect evaluation reliability, defined as consistency of model rankings across experimental choices. Its analyses find low agreement from single prompts, improved consistency from multiple prompts, and reliability gains comparable to adding examples.

5 Implications for Reliable Evaluation

  • Reliability means ranking consistency across experimental choices, measured with Kendall’s W; prompt configurations vary table structure while preserving semantics.Source: §5 · p. 6

5.1 Using a Single Prompt Config is Unreliable

5.2 Multiple Prompt Configs Increase Reliability

  • Increasing prompts from 1 to 10 raises Kendall’s W by more than 0.35 on average, with the largest gains around 2–8 prompts.Source: §5.2 · p. 7, Figure 4 · p. 7
  • With 11 prompts, Kendall’s W rises from 0.35 to 0.93 for FinQA and from 0.29 to 0.54 for NumericNLG.The authors report that datasets show differing robustness patterns; ToRR’s aggregate is roughly a lower bound across datasets.Source: §5.2 · p. 7

5.3 Prompt Configs can Substitute Examples

6 Related Work

Prior work evaluates table tasks and robustness, but the reviewed efforts often cover limited tasks, datasets, or model settings. ToRR is positioned against this narrower scope.

Related work on table benchmarks

  • Existing table benchmarks cover varied QA categories and capabilities, but are described as limited in the tasks and model capabilities they reflect compared with ToRR.Source: §6 · p. 8

Related work on robustness

  • Prior robustness studies examine table formats, noise, structural variance, or adversarial perturbations across table understanding and QA tasks.Source: §6 · p. 8
  • These robustness efforts are often restricted to simple synthetic tasks or narrow QA datasets and target models, leaving broader complex table reasoning challenges underexplored.Source: §6 · p. 8

7 Discussion

The discussion presents ToRR as a broad benchmark for table reasoning and robustness. Its results show low performance and format sensitivity, motivating evaluation practices that account for multiple prompts.

Discussion

  • ToRR is presented as a comprehensive benchmark for table reasoning and robustness in realistic tabular scenarios.Source: §7 · p. 8
  • State-of-the-art LLMs show relatively low table-reasoning performance and sensitivity to minor formatting changes, reflecting broader prompt sensitivity rather than a preference for one format.Source: §7 · p. 8
  • Incorporating multiple prompt configurations is proposed as a way to improve evaluation reliability and address limitations of smaller test sets.The authors also frame ToRR as an example for robust evaluation and encourage its adoption and refinement.Source: §7 · p. 8

8 Limitations

ToRR evaluates table reasoning with prompt-embedded tables in standard formats, while excluding some real-world settings. Its design also specifies dataset metrics, skill annotation, serializations, and structure-preserving perturbations.

8 Limitations

  • ToRR’s selected inputs are tables embedded in prompts and parseable into standard formats, excluding independent table extraction or search and non-standard formats such as hierarchical tables.These choices leave some real-world scenarios outside the benchmark’s scope.Source: §8 · p. 9

A.2 Other Table Dataset Types

  • Tool-dependent datasets were excluded to isolate core tabular reasoning from tool use and agentic skills, trading broader coverage for usability.The authors note that tool-use evaluation frameworks are typically not designed for such settings.Source: §A.2 · p. 11

A.3 Evaluation metrics

  • Evaluation uses original dataset metrics with exceptions: WikiTableQuestions uses F1, FinQA uses program accuracy, and SciGen adopts NumericNLG’s primary metric.The paper says FinQA execution accuracy can overestimate performance, while SciGen’s base-paper metrics do not align with human judgment.Source: §A.3 · p. 12

A.4 Dataset Properties Evaluation

  • Three research scientists independently assessed 15 examples per dataset, with required skills labeled by consistency and final labels determined by at least two annotators’ agreement.A skill was “required” if present in all 15 examples, “not required” if absent from all, and “partially required” if present in at least seven.Source: §A.4 · p. 12

A.5.1 Serializations

  • ToRR tests seven common table serializations: HTML, CSV, JSON, Markdown, Indexed Row Major, Data Frame, and Concatenation.The benchmark treats these as methods for converting structured data into prompt-embedded text.Source: §A.5.1 · p. 12

A.5.2 Structural Perturbations

  • Four structural perturbations alter presentation without changing table content or cell relations: row swapping, column swapping, transposition, and adding empty rows.The intended effect is a changed table structure without a significant change in model performance.Source: §A.5.2 · p. 12

A.6 Data Contamination

ToRR uses multiple prompt variations to address memorization concerns, while acknowledging that contamination cannot be ruled out. The appendix also describes dataset-quality challenges, evaluation costs, and statistical comparisons across model configurations.

A.6 Data Contamination

  • To mitigate memorization, ToRR uses 35 distinct prompt variations per example; the authors expect unseen formats to reduce success based solely on memorized phrasing.They say a model succeeding on only one format would receive a 2.8% performance score for that example, but they cannot rule out contamination entirely.Source: §A.6 · p. 12

A.7 Challenges

  • The benchmark identifies three data challenges: possible memorization from Wikipedia-derived tables, metrics that may misalign with human judgments, and errors in automatically aggregated data.Broken extracted table HTML is given as an example of data that can compromise benchmark integrity.Source: §A.7 · p. 13

A.8 Computation Cost

  • Running the full benchmark uses approximately 110–140 million tokens per model, with about $2,400 in inference API credits for all evaluations at May 2025 pricing.The cited pricing covers the inference platforms used in the paper.Source: §A.8 · p. 13

B Usage

  • ToRR can be run using the unitxt package, and the paper provides a code snippet for reproducing its results.The supplied passage includes a usage figure showing example code.Source: B Usage · p. 15, Figure 6 · p. 15

D Statistical testing of models

  • Statistical testing treats each serializer, augmenter, and example-index combination as a configuration, compares model pairs on matched configurations, and aggregates results across datasets.There are 3,500 configurations per dataset except TableBench FC, which has 2,765; aggregated p-values use Wilson’s harmonic mean and effect sizes use inverse-variance weighting.Source: D Statistical testing of models · p. 17
  • Across datasets, all pairwise model comparisons were significant by p-value (p < 0.05), although nearly all pairwise effect sizes were smaller than 0.5.Within individual datasets, some pairs were not significant after multiplicity adjustment.Source: D Statistical testing of models · p. 17

E.1 Model Performance across Datasets

Performance varies by dataset, with lower scores on several more challenging reasoning and table-to-text tasks. Closed-weight proprietary models outperform open-weight models across most benchmark datasets; the section also introduces analyses of serializers and structural perturbations.

E.1 Model Performance across Datasets

  • Performance tends to be lower on FinQA, TableBench Numerical Reasoning, TableBench Data Analysis, and Table2Text datasets, and relatively higher on question-answering and fact-checking datasets.Figure 7 compares model performance across each dataset.Source: §E.1 · p. 18, Figure 7 · p. 18

E.2 Open vs. Closed models

  • Closed-weight proprietary models considerably outperform open-weight models across most benchmark datasets.Figure 8 presents this comparison across the benchmark datasets.Source: §E.2 · p. 18, Figure 8 · p. 18

F Robustness Analysis

  • The robustness analysis examines two prompt-configuration components: table serializers and structural perturbations.The analysis includes performance scores and example-level win rates for configurations.Source: F Robustness Analysis · p. 19

F.1 Serializers Impact

  • Serializer analysis proceeds from model–dataset pairs to model- and dataset-oriented aggregates, then to an overall aggregation.The section examines trends at progressively broader levels of aggregation.Source: §F.1 · p. 19

F.1.1 Model-Dataset Score Variability

Serializer and perturbation effects vary across models and datasets, with no single configuration consistently favored overall. Further analyses indicate that both datasets and models shape robustness, while ToRR includes examples spanning difficulty levels.

F.1.1 Model-Dataset Score Variability

  • Serializer score drops average about 0.05, with llama-3-1-8b-i dropping 0.22 on TableBench FC.Table 7 presents the largest variations in model scores across serializers by dataset.Source: §F.1.1 · p. 19, Table 7 · p. 19

F.1.2 Serializer Preference across Models

  • Serializer preferences differ by model, with no single serializer ranking highest across all models; choosing one may require case-by-case tuning.Figure 10 shows model-wise serializer preferences.Source: §F.1.2 · p. 19, Figure 10 · p. 20

F.1.3 Serializer Preference across Datasets

  • Serializer rankings vary across datasets: some differences are marginal while others are clear, supporting dataset-specific evaluation rather than a one-size-fits-all choice.Figure 11 presents dataset-wise serializer comparisons.Source: §F.1.3 · p. 19, Figure 11 · p. 20

F.1.4 Overall Serializer Preferences

  • Across models and datasets, no serializer consistently outperforms the others.The overall aggregation follows observed inconsistencies in model- and dataset-specific preferences.Source: §F.1.4 · p. 19

F.2.1 Perturbations Impact

  • Perturbations produce an average overall performance difference of 0.03, with small but inconsistent score variability.The scores are compared with a baseline using the same serializer and no perturbation.Source: §F.2.1 · p. 21, Figure 12 · p. 21

F.2.2 Granular Look: Absolute Perturbations Impact

  • Perturbation impacts are higher for Table QA and fact-checking datasets than table-to-text datasets, and generally more pronounced in smaller models.The authors identify datasets and models as key factors influencing robustness.Source: §F.2.2 · pp. 21, 23, Figure 13 · p. 22, Figure 14 · p. 22

F.2.3 Overall Perturbations Effect

  • Structural perturbations have nearly identical aggregated win rates, with no one perturbation consistently producing under- or over-performance.Figure 15 reports win rates aggregated across benchmark models and datasets.Source: §F.2.3 · p. 23, Figure 15 · p. 23

G Additional Properties of ToRR

  • Score distributions differ across datasets even when they use the same metric, and example-level means show a mix of easy and challenging cases, mostly of medium difficulty.The score-distribution comparison includes TableBench, NumericNLG, and QTSumm, which all use ROUGE.Source: §G.1 · p. 24, Figure 16 · p. 24, §G.2 · p. 24

G.3 An Analysis for Interpreting Dataset Separability

Dataset separability varies and does not strongly correlate with other analyses; score distributions suggest it reflects dataset difficulty and diversity, score differences, and model selection.

G.3 An Analysis for Interpreting Dataset Separability

  • Separability varies across datasets and does not strongly correlate with other analyses; score distributions were examined using 100 bootstrap seeds.The score may reflect overall score distribution, selected models and score differences, and question difficulty and diversity.Source: §G.3 · p. 24, Figure 18 · p. 25
  • NumericNLG and SciGen have more homogeneous question types and stable model scores, while TableBench NR spans approximately 0.15–0.40 and has high score variability.WikiTQ’s high separability may reflect model selection; weaker models vary more than stronger ones.Source: §G.3 · p. 24

H Related Work

I ToRR Properties

  • ToRR’s separability score is the proportion of model pairs whose confidence intervals do not overlap, with intervals estimated by bootstrapping over 1K seeds.The accompanying figures show dataset separability scores and model-ranking agreement between datasets.Source: Figure 19 · p. 27, Figure 20 · p. 27

J Usage in AI

  • AI models were used exclusively for language-related tasks, such as rephrasing and surface-level linguistic transformations.Source: J Usage in AI · p. 27

K Prompt Examples

  • Benchmark prompts use task instructions, five demonstrations with expected answers, and an input example; displayed task examples use one demonstration.The prompt structure is shared across datasets, with task-specific instructions; figures illustrate examples for the listed tasks.Source: K Prompt Examples · p. 28, Figure 21 · p. 29, Figure 22 · p. 30
Loading 2502.19412v3…