Source-linked AI summary
When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors
Yuqing Yang, Qi Zhu, Zhen Han, Boran Han, Zhengyuan Shen, Shuai Wang, Vassilis N. Ioannidis, Huzefa Rangwala
TL;DR
LLMs frequently make data referencing errors in table reasoning, such as confusing table entries or omitting required information, undermining response quality and final accuracy. The paper systematically evaluates these errors and uses critic-based filtering and rejection sampling, improving accuracy by up to 11.96%.
Problem
LLMs still confuse table entries or overlook required information, creating prevalent data referencing errors across table reasoning tasks.
Method
The paper categorizes incorrect citations and omissions, evaluates them with an LLM-as-a-Judge framework, and applies critic-based filtering and rejection sampling.
Results
DREs are prevalent across models, table formats, and tasks, while critic-guided rejection sampling improves accuracy by up to 11.96%.
Takeaways & Limitations
Data referencing is a key evaluation dimension beyond final-answer accuracy for developing more reliable table reasoning systems.
Takeaways & Limitations
The study focuses on table-related tasks and does not establish whether its findings generalize to other domains or modalities.
Abstract
from arXiv · showhide
While large language models (LLMs) perform well on table tasks, they still make data referencing errors (DREs), i.e., incorrectly citing or omitting table values, despite understanding the table structure. Beyond final-answer accuracy, DREs directly compromise the correctness and reliability of intermediate reasoning steps. Yet prior studies have only offered limited, small-scale analyses. In this work, we present the first systematic evaluation of tabular data referencing errors across different models and tasks. Our results show that DREs occur across all tested models (1.7B to 20B parameters). Furthermore, we demonstrate that incorporating data referencing as a critic significantly improves answer accuracy up to 12.0%, through critic-based filtering and rejection sampling. Finally, we trained a lightweight 4B-parameter critic model that achieves an average F1 score of 78.2% in detecting both in-distribution and out-of-distribution DREs, and effectively assists inference for larger models.
1 Introduction
The paper systematically studies tabular data referencing errors (DREs), which arise when models incorrectly cite or omit table information even after parsing table structure. It categorizes DREs, measures their prevalence, and shows that critic-based detection and lightweight critic training can improve reliability and performance.
- Motivation and scope: DREs are ubiquitous across models from 1.7B to 20B parameters and across diverse table-related tasks, motivating systematic study beyond final-answer accuracy.Prior analyses were narrow, typically examining one model and a small set of human-annotated cases.
- DRE taxonomy: DREs comprise Incorrect Citation of individual values and Omitted Information involving entire relevant portions of a table.Dense, structurally similar tables make reliable value localization and citation difficult.
- DRE mitigation: Critic-based filtering selects responses with minimal DREs and substantially improves accuracy over using all samples, while also enhancing majority voting.The approach targets intermediate reasoning quality beyond what final-answer accuracy rewards capture.
- DRE mitigation: Rejection sampling uses DRE detection to improve overall performance by repeatedly sampling and rejecting responses with detected referencing errors.This is the second critic-based approach explored alongside filtering.
- Lightweight critic training: Critic-4B outperforms the untrained baseline in-distribution and out-of-distribution, achieving an average improvement of 8.65% F1 after two-stage training.Training combines supervised fine-tuning with RLVR to improve robust DRE detection without constraining Chain-of-Thought format.
2 Related Work
Prior work has advanced table-focused LLMs from specialized pretraining architectures to prompting, supervised fine-tuning, and reinforcement learning, while evaluation has largely emphasized final accuracy over intermediate reasoning. Although existing studies identify data referencing errors (DREs) and propose related evaluation or reward-based approaches, systematic characterization remains limited.
- Table LLMs: Table-focused language models evolved from specialized pretraining architectures to prompting, supervised fine-tuning, and reinforcement learning.Examples of specialized architectures include TaPas and TaBERT; later methods adapt general-purpose LLMs through prompting, supervised fine-tuning, and reinforcement learning.
- Evaluation Beyond Accuracy: Evaluation benchmarks mostly prioritize final accuracy, overlooking the quality of intermediate reasoning in table tasks.Prior work addresses intermediate reasoning through process-level reward models and multidimensional evaluation, but table-reasoning benchmarks still focus almost exclusively on final correctness.
- Evaluation Beyond Accuracy: Prior evaluation efforts decompose reasoning quality into dimensions such as validity, redundancy, instruction-following, truthfulness, relevance, and completeness.These dimensions were studied in mathematical reasoning, alignment, and long-form question answering, respectively.
- Existing Work on DREs: More than 80% of errors in 50 WTQ samples from Distill-Llama-8B arose from incorrect locating and citation, yet DREs remain unsystematically characterized.This finding shows the importance of accurate and complete table-value use while motivating broader systematic analysis.
3 Characterizing DREs
The section characterizes table-related data referencing errors (DREs) as incorrect citations or omitted information and evaluates their prevalence, detection, and impact across models, formats, and tasks. DREs remain common despite model reasoning and prompting, and can occur even when final answers are correct.
- DRE Definition: The study defines DREs as incorrect citations or omitted information, including wrong values, row or column confusion, fabricated content, and missing table information.The analysis focuses on table-related tasks because their dense, repetitive rows and columns make incorrect referencing particularly likely.
- DRE Evaluation: Sonnet-3.7 with ground truth detects the two DRE types with 92.67% accuracy and high consistency, enabling automatic response evaluation.The judge checks whether generated responses use table information accurately, while segmenting lengthy reasoning responses at reflection tokens addresses verbosity-related evaluation difficulty.
- Severity Analysis: DREs are prevalent across models, table formats, and table-related tasks, with larger Qwen3 models generally producing fewer errors but cross-family rates depending on overall capability.Llama4-Scout shows a relatively high DRE rate of 46.48%, while reasoning models still exhibit rates from 5.71% to 46.04%.
- Severity Analysis: DREs persist despite self-reflection and explicit data-referencing prompts, with models often repeating an initial error instead of returning to the original table.The evaluated instruction required models to use only the table and ensure cited values exactly matched it.
- Severity Analysis: Correct final answers can contain DREs, so final-answer accuracy alone cannot guarantee correct intermediate reasoning or overall response quality.The Correct-in-DRE Ratio is especially high for SciTab at 65.57%, where binary True/False answers may remain correct despite numerical citation errors.
4 Reducing DREs with Critics
Critic-based filtering and segment-level rejection sampling reduce data referencing errors and improve final accuracy, especially on data-referencing-challenging cases. These methods show that many DREs are avoidable through inference-time intervention without changing the generation model.
- Critic-based Filtering: Because the critic detects DREs rather than overall answer correctness, filtering selects a subset instead of a single best response.Multiple responses can be DRE-free yet differ because errors arise later in reasoning, motivating subset selection rather than standard Best-of-N judging.
- Critic-based Filtering: Critic-based filtering improves final accuracy by selecting responses with fewer data referencing errors, then enables majority voting over a higher-quality subset.The method samples N = 8 responses per question and filters them by DRE count; on some cases, even random selection from the filtered subset exceeds majority voting.
- Rejection Sampling: Rejection sampling with a critic improves accuracy for reasoning and non-reasoning models, with larger gains on the DRE subset than on the full set.The procedure resamples only flagged reasoning segments, or the full response when necessary, until the critic accepts it or N = 8 retries are reached.
- Rejection Sampling: Segment-level rejection sampling obtains DRE-free responses by resampling without altering the generation model, indicating that DREs are largely avoidable errors.The authors characterize critic-guided rejection sampling as a promising practical solution while noting that reliably reducing DRE frequency remains open.
5 Training a Small-Scale Critic
The authors train Critic-4B, a lightweight DRE detector using SFT followed by RLVR, and evaluate its generalization across models and table-question-answering datasets. Critic-4B reaches 78.16% overall F1 versus 69.51% for Qwen3-4B-Instruct and improves inference through rejection sampling.
- Critic Training: Critic-4B detects DREs by classifying response segments as positive when errors are present and negative otherwise, including responses that cite no table values.The critic receives a table, question, and response segment; its two-stage training uses SFT warm-up followed by RLVR.
- Results: 78.16% overall F1 lets Critic-4B outperform Qwen3-4B-Instruct’s 69.51% across all evaluation scenarios and generalize beyond WTQ to TableBench and FinQA.The evaluation spans responses from reasoning and non-reasoning models across diverse table-question-answering tasks.
- Results: Critic-4B-Synthetic gains more on in-distribution settings but declines on FinQA and Llama4-Scout, suggesting overfitting to synthetic-data biases rather than learning real-world DREs.The decline appears when the domain or model differs substantially from the synthetic-data setting.
- Inference Assistance: Rejection sampling with Critic-4B consistently improves accuracy over no rejection sampling, providing a lightweight, cost-effective alternative to the stronger Sonnet-3.7+gt critic.It is smaller than all three generation models and does not use extended thinking, although it is less effective than Sonnet-3.7+gt.
6 Conclusions
Data referencing errors are a pervasive weakness in LLM table reasoning, undermining response quality and final accuracy. Systematic analysis supports inference-time mitigation and treating data referencing as an evaluation dimension beyond final-answer accuracy.
- DREs are pervasive in LLM table reasoning and undermine both response quality and final accuracy.
- Systematic LLM-as-a-Judge analysis demonstrates DRE prevalence and motivates inference-time strategies and lightweight critics for mitigation.
- Data referencing should be evaluated alongside final-answer accuracy when developing more reliable table reasoning systems.
Limitations
The study is limited to data referencing errors in table-related tasks, although such errors also occur in other domains and modalities. It also leaves the causes of these errors underexplored because attention analyses and steering experiments were not scaled up.
- Scope: The evaluation covers only DREs in table-related tasks, despite evidence that similar errors also arise in other domains and modalities.Cobbe et al. (2021b); Mirzadeh et al. (2025) describe a non-tabular example in which Qwen3-8B misinterprets the order of ice-cube categories.
- Interpretability: The study does not examine DRE causes from an interpretability perspective because attention analyses and steering experiments were not scaled up under resource constraints.Future work could investigate this direction more extensively.
- Interpretability: Preliminary experiments suggest that increasing attention to the entire table before referencing a value can reduce subsequent DREs, linking errors to insufficient attention.This observation motivates future attention-analysis and steering experiments.
A LLM-as-a-Judge · B Critic-based Filtering
The study evaluates table-referencing accuracy with a ground-truth-aware LLM judge and reports near-human judgment reliability, while critic-based filtering compares average accuracy with accuracy on a critic-selected subset.
- A LLM-as-a-Judge: Ground-truth answers were provided to Sonnet-3.7+gt to mitigate false negatives when judging data referencing errors.The prompt design explicitly incorporates ground-truth answers as described in Section 3.2.
- A LLM-as-a-Judge: The judge was instructed to compare model responses solely with table data when assessing table-referencing accuracy.This explicit focus separates table-referencing assessment from broader answer evaluation.
- A LLM-as-a-Judge: The judge distinguishes calculation mistakes from genuine data-referencing errors rather than treating every incorrect final answer as a DRE.An example shows accurate extraction and citation of all relevant table values despite a calculation error.
- A LLM-as-a-Judge: The example model accurately extracted and cited all relevant table values even though its final answer differed from the reference answer.The discrepancy was attributed to a calculation error rather than a table-referencing error.
- A LLM-as-a-Judge: 92.67% average accuracy was achieved by Sonnet-3.7+gt’s judgments in independent assessments by three PhD-level annotators.The evaluation sampled 100 instances from the critic evaluation dataset and indicated near-human reliability.
- B Critic-based Filtering: Critic-based filtering compares all-sample average accuracy with accuracy on a critic-selected subset over the full set.The supplied passage identifies this comparison but does not provide the corresponding values.
C Synthetic Positives Construction
Synthetic positives are constructed by deliberately inserting data referencing errors (DREs) into tables or model responses, then retaining cases whose final answers change under repeated inference. Four perturbation strategies cover value swaps and row omissions, while mismatches between modified inputs and original responses identify DREs.
- DRE identification: A retained case indicates a DRE when the modified table with the original response, or the original table with the modified response, fails to fully match.The synthetic positives start from tables and responses with correct final answers before perturbation.
- Perturbation strategies: Four perturbations create DREs: swapping values across rows or columns, deleting the row containing a used value, and removing a listed row from enumerated responses.The first two strategies preserve either the column or row while changing the referenced value; the latter two alter table or response content.
- Filtering criteria: Qwen3-8B performs inference three additional times, and only cases with changed final answers are retained.This filtering step selects examples where the modification affects the model’s answer.
D Training Details · E Generation Details
Training uses specified supervised fine-tuning and reinforcement-learning configurations, while generation evaluation applies task-dependent metrics, restricted TableBench subsets, and recommended decoding settings. The trained critic uses greedy decoding at inference.
- D Training Details: The critic is trained with SFT for 2 epochs, then RLVR with GRPO for 20 epochs and 8 rollouts per prompt.SFT uses Llama-Factory (Zheng et al., 2024), while RLVR uses verl (Sheng et al., 2024) and GRPO (Shao et al., 2024).
- D Training Details: SFT uses learning rate 1e-5, batch size 8, and 2,000 examples; RLVR uses learning rate 1e-6, batch size 256, and temperature 1.0.During inference, the trained critic uses greedy decoding.
- E Generation Details: Accuracy uses string matching for table-related tasks except ToTTO, which uses (BLEU + ROUGE-L)/2 following Yang et al. (2025b).The ToTTO metric averages BLEU and ROUGE-L.
- E Generation Details: TableBench evaluation covers only Fact Checking and Numerical Reasoning, totaling 493 examples, because the other subsets exceed tested-model capabilities.Data Analysis and Visualization are excluded from evaluation.
- E Generation Details: Generation models use their recommended decoding hyperparameters, as detailed in Table 6.Table 6 specifies the decoding hyperparameters used for the generation models.
- E Generation Details: Llama4-Scout generation uses its fp4 quantized version.This is an implementation choice within the generation setup.
F Code of Ethics
The study uses only public datasets and models and reports no ethical, safety, or privacy risks. It lists the relevant licenses and states that LLMs were used only to polish author-generated writing.
- The study uses public datasets and models and reports no ethical, safety, or privacy risks.
- The licenses for the datasets and models used are listed in Table 7.
- LLMs were used to polish writing, while all original content came from the authors.