PaperlayerView exact revision on arXiv

Source-linked AI summary

TabReX : Tabular Referenceless eXplainable Evaluation

Tejas Anvekar, Junha Park, Aparna Garimella, Vivek Gupta

arXiv:2512.15907v2cs.CL

TL;DR

Structured-generation evaluation must account for semantic fidelity as well as schema, syntax, and cell correctness. TabReX uses graph-based alignment and property-driven scoring, and the authors report strong human alignment and robustness under harder perturbations.

  • Problem

    Evaluating structured generation requires assessing semantic fidelity alongside schema alignment, syntactic consistency, and cell-level correctness.

    Source: §1 · p. 1

  • Method

    TabReX converts source text and generated tables into graphs, aligns factual correspondences and discrepancies, and computes interpretable, rubric-aware scores.

    Source: §1 · p. 2

  • Results

    TabReX correlates most strongly with human evaluations and remains robust under harder perturbations.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Takeaways & Limitations

    TabReX provides fine-grained cell- and table-level diagnostics for actionable analysis of model and prompt behavior.

    Source: §5 · p. 8

    TABREX-BENCH remains limited to English and synthetic perturbations, leaving real-world noise, multilingual data, and complex layouts for future exploration.

    Source: §6 · p. 9

Abstract

from arXiv · show

Evaluating the quality of tables generated by large language models (LLMs) remains an open challenge: existing metrics either flatten tables into text, ignoring structure, or rely on fixed references that limit generalization. We present TabReX, a reference-less, property-driven framework for evaluating tabular generation via graph-based reasoning. TabReX converts both source text and generated tables into canonical knowledge graphs, aligns them through an LLM-guided matching process, and computes interpretable, rubric-aware scores that quantify structural and factual fidelity. The resulting metric provides controllable trade-offs between sensitivity and specificity, yielding human-aligned judgments and cell-level error traces. To systematically asses metric robustness, we introduce TabReX-Bench, a large-scale benchmark spanning six domains and twelve planner-driven perturbation types across three difficulty tiers. Empirical results show that TabReX achieves the highest correlation with expert rankings, remains stable under harder perturbations, and enables fine-grained model-vs-prompt analysis establishing a new paradigm for trustworthy, explainable evaluation of structured generation systems.

1 Introduction

Reliable evaluation of structured generation must assess semantic fidelity alongside schema alignment, syntactic consistency, and cell-level correctness. TabReX combines graph-based alignment and rubric-aware scoring with a benchmark designed to test metric robustness across domains and perturbations.

1 Introduction

  • Structured generation requires evaluation of semantic fidelity, schema alignment, syntactic consistency, and cell-level correctness.Even a misplaced column, unit mismatch, or corrupted cell can propagate costly downstream errors.Source: §1 · p. 1
  • Existing metrics can miss table structure, unit semantics, or structural perturbations, while reference-less approaches may over-penalize layout changes.Prior explainable metrics are described as limited by small, single-pass benchmarks and one-shot perturbation schemes.Source: §1 · pp. 1–2
  • The paper calls for property-driven metrics that balance sensitivity and specificity, remain robust to outliers, and adapt through interpretable weights.The desired properties include permutation and format invariance, schema- and unit-consistent alignment, and monotonic improvement as errors are fixed.Source: §1 · p. 2
  • TabReX converts source text and generated tables into graphs, aligns factual correspondences and discrepancies, and computes interpretable, rubric-aware scores.Its scoring captures both structure and content quality in a reference-less evaluation framework.Source: §1 · p. 2
  • TabReX-Bench covers six domains and twelve planner-driven perturbation types across three difficulty levels.The benchmark is presented as a way to stress-test metric reliability.Source: §1 · p. 2
  • TabReX achieves strong human correlation and robustness under harder perturbations.The introduction also highlights rubric-wise diagnostics at both table and cell levels for model–prompt alignment.Source: §1 · p. 2

2 TABREX

TABREX evaluates tables against source text by representing both as knowledge graphs, aligning their facts, and scoring structural and factual discrepancies. TABREX-BENCH supplies varied, planner-generated perturbations for evaluating tabular metrics across domains and difficulty levels.

2 TABREX

  • TABREX converts source text and generated tables into knowledge graphs, then scores their alignment for explainable, reference-less evaluation.Its scoring yields interpretable table- and cell-level penalties.Source: §2 · p. 2, Figure 2 · p. 2

2.1 Pipeline Overview

  • Text2Graph extracts atomic facts, while rule-based Table2Graph turns non-empty table cells into triplets; alignment matches normalized pairs and refines remaining matches with an LLM.The refinement resolves paraphrases, abbreviations, and compound attributes.Source: §2.1 · pp. 2–3
  • Alignment-derived counts of missing, extra, and partially matched triplets drive complementary structural and factual scoring components, with weights controlling sensitivity versus specificity.Increasing βMI favors coverage, while increasing βEI penalizes hallucinated entries.Source: §2.1 · p. 3

2.2 TABREX-BENCH

  • 710 source tables expanded with 12 perturbations each yield 9,120 instances across six datasets and finance, healthcare, hierarchical-table, and narrative-to-table tasks.The benchmark includes compact clinical sheets and large multi-column tables.Source: §2.2 · p. 4
  • TABREX-BENCH separates data-preserving presentation changes from data-altering semantic edits, with each group stratified into easy, medium, and hard tiers.Examples range from reordering or unit conversion to row or column changes, swapped values, and injected noise.Source: §2.2 · p. 4
  • A planner generates executable code for all 12 perturbations in one pass; perturbations are paired with fact-aligned summaries, and validation reports 87% summary and 91% perturbation agreement.The benchmark is intended to support robustness, sensitivity, and human-alignment evaluation.Source: §2.2 · p. 4

3 Experiments

Experiments compare automatic metrics with human rankings on synthetic perturbations and real text-to-table outputs, then use TABREX’s rubric-aware scores to examine model–prompt alignment at cell and table levels.

3 Experiments

  • The experiments use GPT-5-nano on TABREX-BENCH, comparing metric correlations with human rankings of systematically perturbed variants and evaluating text-to-table outputs across domains.The benchmark includes factual-preserving and data-altering perturbations at easy, medium, and hard levels.Source: §3 · p. 4, §3.1 · p. 5, §3.3 · p. 6

3.1 Correlation Analysis of Metrics Category.

  • TABREX achieves Spearman’s ρS = 0.75, Kendall’s τK = 0.64, and weighted τw = 0.62 on synthetic perturbations, with balanced agreement and finer discrimination.Its tie ratio is 13.6%, and rank dispersion is ζF = 27.0.Source: Table 2 · p. 5, §3.1 · p. 5
  • The best ensemble, LLM (Harmonic), reaches ρS = 0.56 and τK = 0.47, below TABREX’s ρS = 0.75 and τK = 0.64.All ensemble variants fall short of TABREX in correlation and rank stability.Source: §3.1 · p. 5, Table 3 · p. 6

3.2 Can TABREX Generalize Across Perturbation Regimes?

  • TABREX remains near the ideal sensitivity–specificity zone under harder perturbations, while embedding-driven metrics show large drops and EM, H-SCORE, and P-SCORE lose sensitivity.QUESTEVAL can penalize harmless re-orderings or formatting changes; TABREX’s tabular-structure-specific fact reasoning better separates meaningful discrepancies from superficial variation.Source: §3.2 · p. 6

3.3 Evaluation on Text-to-Table Task

  • On real-world text-to-table generation, TABREX achieves the strongest correlations with expert judgments across all measures: Spearman’s ρ = 0.39, Kendall’s τb = 0.30, and RBO = 0.41.Source: Table 4 · p. 7, §3.3 · p. 7

3.4 Rubric-wise Model–Prompt Alignment

  • Larger models improve fine-grained cell fidelity more than global coherence, while reasoning-oriented variants can improve precision yet reduce semantic coverage.At table level, zero-shot improves row-column coherence but increases rubric variance; Map&Make yields steadier rubric alignment, and prompt design contributes as much as model scale to balanced alignment.Source: §3.4 · p. 7

4 Comparison with Related Work

Related work progresses from early text-to-table testbeds and surface-overlap metrics toward structural and reference-less evaluation, but existing approaches retain gaps in structural fidelity, error traceability, or robustness.

From Text-to-Table to Structural Benchmarks

  • Early text-to-table datasets offered limited schema diversity, while newer benchmarks introduced header permutations, schema reshuffling, and multi-hop reasoning.These benchmark weaknesses motivate evaluation beyond surface overlap toward structural and semantic fidelity.Source: §4 · p. 7

Metric Families: From Overlap to Explainability

  • Overlap and embedding metrics flatten tables, while structural, entailment-based, and two-phase methods trade among interpretability, layout tolerance, and diagnostic detail.TABXEVAL’s TabAlign and TabCompare provide interpretable cell-level diagnostics and strong human correlation, while TABEVAL can over-penalize harmless layout differences.Source: §4 · pp. 7–8

Reference-less Evaluation and Remaining Gaps

  • Generic question-answering signals can miss table-specific structural errors, unit inconsistencies, and localized discrepancies; many metrics also compress diverse errors into opaque scores.This limits error traceability and robustness under realistic perturbations.Source: §4 · p. 8

5 Conclusion and Future Work

The paper concludes that TABREX offers reference-less, graph-based evaluation with interpretable rubric-aware scores, and reports human-aligned judgments and explainable diagnostics. It identifies richer table formats and lighter evaluators as future directions.

5 Conclusion and Future Work

  • TABREX unifies structured alignment, factual comparison, and sensitivity–specificity control in a reference-less framework, with robustness across domain shifts and perturbation difficulty.TABREX-BENCH supports systematic stress testing across six domains and twelve controlled perturbation types.Source: §5 · p. 8
  • Experiments report the strongest correlation with human evaluations and fine-grained, explainable diagnostics at cell and table levels.The diagnostics support actionable analysis of model and prompt behaviors, and the paper reports outperforming reference-based and LLM-judge baselines.Source: §5 · pp. 8–9
  • Future work will extend TABREX to hierarchical or multimodal tables and distill its LLM components into lightweight, domain-adaptive evaluators.The paper frames trustworthy, interpretable evaluation as relevant to model selection, alignment, and reward learning in real-world applications.Source: §5 · p. 9

6 Limitations

TABREX’s current implementation and benchmark have computational, format, language, and perturbation-scope limitations.

6 Limitations

  • LLM-based fact extraction and alignment add computational cost and mild variability, and the implementation handles digital tables but not tables in images or PDFs requiring visual parsing.The benchmark is limited to English and synthetic perturbations, leaving real-world noise, multilingual data, and complex layouts for future exploration.Source: §6 · p. 9

7 Ethics Statement

The authors report that they followed research and publication ethics, controlled LLM stochasticity with fixed parameters, and identified no additional ethical risks.

7 Ethics Statement

  • The authors state that the study introduces no additional ethical risks to the best of their knowledge.They affirm adherence to research and publication standards and responsible computational linguistics practices.Source: §7 · p. 9
  • Fixed temperature, topp, and topk settings were used to mitigate variability inherent in black-box LLMs.The authors report using GPT-5-nano, Gemma, and InternVL in compliance with their respective usage policies.Source: §7 · p. 9
  • Human annotators who were also authors contributed to data labeling and verification, while Grammarly and ChatGPT assisted text refinement.Source: §7 · p. 9

Maintenance and Adoption Plan

The project makes TabReX and TabReX-Bench available under permissive terms and describes processes for maintenance, versioning, and community contributions.

Maintenance and Adoption Plan

  • The TabReX reference implementation is open-sourced under the MIT License, and TabReX-Bench is hosted on Hugging Face with versioned revisions.The authors say versioning preserves reproducibility of published numbers as the dataset grows.Source: Maintenance and Adoption Plan · p. 9
  • Junha Park coordinates maintenance, while GitHub Issues and Pull Requests handle bug reports, feature requests, and new-domain contributions.Source: Maintenance and Adoption Plan · p. 9
  • The project plans tagged releases for breaking changes and tracks upstream LLM-judge deprecations, while welcoming new domains, perturbations, and judge back-ends.The contribution options include the embedding-based matcher described in Appendix C.Source: Maintenance and Adoption Plan · p. 9

A Human Evaluation Protocol

Human evaluators ranked generated tables against gold tables when available, or against source text, using structural and contextual fidelity criteria.

A Human Evaluation Protocol

  • Each annotation batch contained one gold table and five generated candidates, which annotators ranked from best to worst.The worst rank was 5 or 12 depending on the task.Source: A Human Evaluation Protocol · p. 11
  • Structural ranking penalized missing columns most, followed by extra columns, row omissions or additions, cell omissions or additions, and partial mismatches.Source: A Human Evaluation Protocol · p. 11
  • Within equal structural quality, annotators ranked contextual errors from string mismatches through numeric, boolean, date-time, listtype, and other data-type deviations.Source: A Human Evaluation Protocol · p. 11
  • Ties were broken by the number of affected cells, and semantically incorrect column headers counted as wrong columns equivalent to missing columns.Source: A Human Evaluation Protocol · p. 11
  • The authors say the rubric produced consistent, interpretable human rankings aligned with TabReX's property-driven principles.Source: A Human Evaluation Protocol · p. 11

B Walk-Through Example of TABREX

The appendix gives default TabReX hyperparameters and a reference-less worked example, showing how structural and factual discrepancies contribute to the score.

B Walk-Through Example of TABREX

C Validation of LLM-based KG Extraction

The extractor study finds strong agreement between paraphrase-tolerant matchers while testing whether extractor choice shifts TABREX scores. The prompt templates specify graph construction, semantic alignment, and controlled perturbation generation.

C Validation of LLM-based KG Extraction

  • The validation uses 30 summaries sampled across six domains, with human annotators constructing gold knowledge graphs containing 364 triples.The full pipeline is also run four times per sample, varying the summary-KG source across gold and LLM-extracted graphs.Source: C Validation of LLM-based KG Extraction · pp. 12–13
  • Extraction quality is measured against gold triples using micro-averaged precision, recall, and F1 under exact, cross, embedding, and semantic matchers.The embedding matcher uses whole-triple cosine similarity and greedy one-to-one matching; the semantic matcher uses TABREX’s Graph Alignment prompt.Source: C Validation of LLM-based KG Extraction · p. 12, Table 6 · p. 12
  • All three extractors achieve high F1 under paraphrase-tolerant matchers, which agree within 0.06 F1; exact and cross F1 is low because extractors paraphrase entities and predicates.The agreement compares an LLM judge with dense embeddings, independent matching signals.Source: C Validation of LLM-based KG Extraction · p. 12
  • Table 8 compares rank correlations between TABREX scores from the gold summary graph and scores from each LLM-extracted summary graph across 30 samples.The supplied passage identifies the comparison but does not state its results.Source: Table 8 · p. 13

D Prompt Templates

  • The perturbation prompts distinguish meaning-preserving changes from fact-changing ones and assign easy, medium, and hard tiers by combining increasing numbers or complexity of changes.Semantically identical changes exclude adding or deleting rows or columns; semantically different changes falsify facts.Source: D Prompt Templates · p. 14
  • The graph-extraction prompt represents facts as subject–predicate–object triples, using specific entities or time slices, normalized property keys, and canonical values.The instructions also call for preserving units and using a placeholder for missing values.Source: D Prompt Templates · p. 15
  • The alignment prompt matches graph facts by content rather than position, marks unmatched facts as missing or extra, and computes numeric differences with range handling.Each source triple is used once, and overlapping ranges receive a difference of zero.Source: D Prompt Templates · p. 16
Loading 2512.15907v2…