Understanding Errors in LLM-Based Question Answering over Imperfect Tables

1 INTRODUCTION

• Reliable imperfect-table QA requires both finding errors that matter to the question and deciding how to handle them before computing an answer.Error locations alone do not specify supported corrections or when a record should be excluded.Source: §1 · pp. 1–2
• Answer-preserving row-placement changes alter discovery of relevant errors, with complete discovery higher for critical rows near the back than the front.Across tested mean positions, Compact layouts yield higher complete discovery rates than Wide layouts.Source: §1 · p. 2
• 39.0–59.1 percentage points: across three systems, reviewed repairs yield higher accuracy than error-marked tables.Providing verified error locations alone leaves a substantial gap in final QA accuracy.Source: §1 · p. 2
• 3.8–18.5 percentage points: GBDI raises observed accuracy over the Code Agent baseline across five systems.Paired 95% confidence intervals exclude zero for four systems.Source: §1 · p. 2
• GBDI combines inspection across shuffled table views with explicit guidance for verifying and handling reported errors.The workflow is presented as a practical application of the study findings.Source: §1 · p. 2

2 PROBLEM FORMULATION

• Each TQA instance pairs an observed table and question with a gold answer computed from the corresponding recovered table.The observed table may contain one of five artifact types: missing values, bad values, outliers, formatting errors, or logical inconsistencies.Source: §2 · p. 3

2.1 CRITICAL CELLS

• Critical cells are both human-verifiable errors and cells whose values can affect the current question’s answer.Errors irrelevant to the answer and relevant but unerroneous cells are excluded from the critical-cell set.Source: §2.1 · p. 3
• A critical row is any row containing at least one critical cell, linking answer-relevant error locations to the unit used in row-order experiments.The critical-cell set identifies locations; the critical-row set identifies their containing rows.Source: §2.1 · p. 3

2.2 INTERVENTION

• Each critical row receives a human-reviewed EDIT or DROP intervention: correct values when recoverable, otherwise exclude the row from computation.EDIT uses supported corrections; DROP applies when at least one required correction cannot be reliably recovered.Source: §2.2 · p. 3
• Reviewers annotate 154 RADAR-T instances with critical cells and rows, supported EDIT replacements, and DROP decisions for shared evaluation references.These annotations support discovery evaluation and the error-marked and repaired-table comparisons.Source: §2.2 · p. 3

2.3 ANSWER-PRESERVING ROW PERMUTATIONS

• For questions that do not refer to row positions, data-row permutations keep the header fixed and preserve the gold answer while changing critical rows’ displayed positions.Critical cells remain unchanged; only their row positions change.Source: §2.3 · p. 3

2.4 COMPLETE DISCOVERY RATE

• Complete discovery rate counts a view as successful only when every reference critical row is found; extra nominations do not lower the score.The section uses controlled permutations to study critical-row position and dispersion, then compares answers from original, marked, and repaired tables.Source: §2.4 · p. 4

3.1 SETUP: POSITION AND DISPERSION

• Position is the mean normalized location of critical rows, while dispersion averages each critical row’s distance to its nearest other critical row.Larger position values place rows nearer the table’s end; larger dispersion values indicate greater spacing.Source: §3.1 · p. 4

3.2 EXPERIMENTAL DESIGN

• The study tested position on 60 instances and dispersion on 59, varying placements while keeping the question, cell values, and critical rows fixed in each comparison.Position used five front-to-back bands; dispersion compared Compact, Medium, and Wide layouts at fixed mean positions.Source: §3.2 · pp. 4–5, Figure 1 · p. 4

3.3 RESULTS

• Direct CDR rose from 61.1% to 72.8% for Qwen, 47.8% to 59.4% for MiniMax, and 43.9% to 60.6% for GPT when critical rows moved from q1 to q5.The Code Agent showed the same direction, with smaller increases for all three systems.Source: Figure 2 · p. 5, §3.3 · p. 6
• For Direct, Wide layouts lowered CDR versus Compact by 16.2 points for Qwen, 9.6 for MiniMax, and 7.9 for GPT, averaged over tested mean positions.The decrease held at every tested mean position for each system.Source: Figure 2 · p. 5, §3.3 · p. 6

4 HANDLING ERRORS: DISCOVERY ALONE IS INSUFFICIENT

• Error locations leave unresolved whether to correct a value or exclude an affected record when the needed value cannot be recovered.The results distinguish finding relevant errors from resolving them for QA, motivating an intervention stage after discovery.Source: §4 · p. 6, §4.2 · p. 7

4.1 EXPERIMENTAL SETUP

• The comparison used 154 human-reviewed instances across Qwen Plus, MiniMax-M2.5, and GPT-5 Mini in Direct and Code Agent modes.RAW retained the original table; DISCOVERY marked critical cells; ACTION applied reviewed replacements and removed reviewed DROP rows.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 RESULTS

• With Code Agent, repaired rather than error-marked tables raised accuracy by 39.0–59.1 percentage points across all three systems, with paired intervals excluding zero.Changes from error marking relative to Raw inputs were smaller and statistically uncertain.Source: §4.2 · p. 7, Figure 4 · p. 7

5.1 GBDI WORKFLOW

• Random-M independently shuffles row order across M views, maps nominations to original rows, and merges them into a union ledger.The default M is 5; a Code Agent then verifies candidates and applies supported actions before answering.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2 GBDI EVALUATION

• On 313 RADAR-T instances, GBDI raises QA accuracy point estimates by 3.8–18.5 percentage points over Code Agent across all five systems.Source: §5.2 · p. 8, Figure 6 · p. 8
• With Qwen Plus Random-5 reports fixed, the Intervention Skill improved QA by 6.4 percentage points over general guidance on allowed operations.Removing formatting rules reduced accuracy; effects of removing recovery rules or execution checks were less certain.Source: §5.2 · p. 8
• Even after complete discovery, incorrect answers remained for Qwen, GLM, and MiniMax; Skill guidance changed more answers from incorrect to correct than vice versa.This comparison held discovery reports fixed.Source: §5.2 · p. 8
• For Qwen Plus, CDR showed diminishing returns as M increased to five, while final QA at M = 5 improved over using the same Skill without discovery reports.Individual QA increments remained uncertain.Source: §5.2 · p. 8

LLMs in TQA

• TQA approaches develop structured representations or reasoning pipelines for answering questions from supplied tables, whereas this work studies discovering and handling errors.Examples include specialized retrieval interfaces, intermediate-table construction, cell selection, aggregation, and execution-oriented pretraining.Source: §6 · p. 9

Robustness for LLMs

• Robustness research covers prompt and context perturbations, table perturbations, row placement, and data-cleaning workflows, while this paper examines erroneous cells relevant to a question.It distinguishes marking verified errors from supplying completed repairs.Source: §6 · p. 9

Order Sensitivity in LLMs

• The paper distinguishes its focus on question-relevant error discovery and the QA gap between supplied error locations and completed repairs from prior work on answer prediction under table permutations.Prior work includes permutation robustness and adversarial row and column ordering aimed at degrading final-answer alignment.Source: §6 · p. 9

• The diagnostics rely on tables with explicit rows and columns, and extensions to text, images, or multimodal inputs require new definitions and reliable evaluation annotations.The required extensions include defining question-relevant errors, answer-preserving transformations, and supported interventions.Source: §7 · p. 9
• Black-box interfaces constrain investigation of how specific internal representations relate to the observed effects.The authors identify analysis of internal activations as a possible direction for investigating this connection.Source: §7 · p. 9

8 CONCLUSION

• GBDI achieves higher observed QA accuracy than Direct and Code Agent across all five systems, combining discovery across shuffled views with guidance for verifying and handling errors.The workflow aggregates candidate errors, checks them against the visible table, and applies supported actions before final QA.Source: §8 · p. 9

A.1 REFERENCE COVERAGE

• The human-reviewed reference covers 154 non-clean RADAR-T instances, with annotations recording affected rows, implicated columns, visible evidence, and supported EDIT or DROP actions.EDIT annotations include supported replacement values.Source: §A.1 · p. 14, A HUMAN-REVIEWED REFERENCE · p. 14
• The table-state comparison uses all 154 instances, while multi-view CDR uses 141 with nonempty critical-row sets; 13 empty-set instances are excluded from that analysis.False-positive rates for the excluded instances are reported separately.Source: §A.1 · p. 14

A.2 REVIEW PROCEDURE

• Annotators independently reviewed each instance within pairs, and an author reviewer checked all annotations and adjudicated disagreements.Reviewers used the table and question to identify errors and interventions; the gold answer checked the computation, not error locations or replacements.Source: §A.2 · p. 14

A.3 GEOMETRY COHORT

• The geometry study uses 60 instances for row-position tests and 59 for dispersion tests, selected from 29 base tasks and 21 source tables.The cohort includes 12 instances per error type; the dispersion subset omits the single-critical-row instance.Source: §A.3 · p. 14

• The position and dispersion cohorts share 29 base tasks from 21 source tables; their sole membership difference is one logical-inconsistency variant included only in the position cohort.That olympics-country variant has one critical row.Source: §A.4 · p. 14
• The cohort tables list task coverage and exact error-type membership; each listed error type identifies one RADAR-T instance.Membership cells use M, B, O, F, and L for missing values, bad values, outliers, formatting errors, and logical inconsistencies.Source: §A.4 · p. 14, Table 1 · pp. 15–16
• The tables also identify source datasets represented in the human-reviewed reference.Source identifiers and dataset names follow RADAR.Source: Table 2 · p. 16

B.1 BENCHMARK AND SHARED INFERENCE CONDITIONS

• RADAR-T contains 313 instances from 53 base tasks and 27 source tables; constructed views repeat layouts rather than create independent tasks.End-to-end QA uses the full benchmark, while human annotations support the controlled table-state study and reviewed geometry subsets.Source: §B.1 · p. 17

B.2 GEOMETRY DIAGNOSTIC

• For geometry discovery, Direct returns candidate row IDs in one response, while Code Agent may use up to five responses and read-only Python inspection.The 60-instance position study evaluates five position bands across three backgrounds; the 59-instance dispersion study crosses three mean positions, three backgrounds, and Compact, Medium, and Wide layouts.Source: §B.2 · p. 17

B.3 TABLE-STATE DIAGNOSTIC

• The table-state diagnostic compares Raw, Discovery, and Action on 154 reviewed instances under Direct and Code Agent.Discovery marks verified errors without replacements; Action applies reviewed replacements and removes rows assigned DROP.Source: §B.3 · pp. 17–18

B.4 GBDI AND MATCHED CONTROLS

• GBDI compares independently permuted Random-M inspections with call-matched repeated-order controls, then gives a Code Agent the canonical table, question, evidence ledger, and Intervention Skill.The primary configuration is Random-5; the guidance factorial holds each ledger fixed while comparing Generic and Skill guidance.Source: §B.4 · p. 18

B.5 SYSTEM CONFIGURATIONS

• The Direct baseline uses the released RADAR prompt and canonical table; GBDI and Code Agent runs use a valid done submission with the answer in kwargs.answer.Systems, providers, and request identifiers are listed in Table 3.Source: §B.5 · p. 18, Table 3 · p. 18, §B.6 · p. 19

B.6 SCORING AND REPORTING

• Discovery is scored against reference critical-row sets, with invalid submissions counted as empty predictions or incorrect answers in the primary denominator.CDR measures complete coverage without penalizing extra nominations; row-level precision, recall, and F1 complement it, with metrics computed before macro averaging.Source: §B.6 · p. 19
• Paired discovery contrasts average instance-level differences over backgrounds and, for dispersion, mean positions; QA accuracy is reported for the stated evaluation population.Uncertainty procedures are specified in Appendix E.Source: §B.6 · p. 19

C.1 POSITION AND DISPERSION RESULTS

• Table 4 reports CDR and macro precision, recall, and F1 for five position bands and three dispersion levels, using 60 and 59 instances, respectively.The section supplements the main geometry study with complete marginal results, joint position–dispersion breakdowns, and WikiSQL validation.Source: §C.1 · p. 19, C ADDITIONAL GEOMETRY RESULTS · p. 19, Table 4 · p. 20

C.2 JOINT POSITION-BY-DISPERSION RESULTS

• Under Direct, CDR decreases from Compact to Wide at each tested mean position for every system.The joint layouts use the same 59 instances and three backgrounds; Code Agent results also vary across mean positions.Source: §C.2 · pp. 21–22, Table 6 · p. 21, Table 7 · p. 22

C.3 ERROR DISCOVERY ON WIKISQL

• The WikiSQL validation uses 41 tables and missingness instances with three question-relevant rank-column values removed; Direct discovery is evaluated across position and dispersion layouts.Each Figure 7 point averages 123 views of the same 41 instances.Source: §C.3 · p. 22, Figure 7 · p. 23
• On WikiSQL, CDR falls from Compact to Wide from 80.5% to 48.0% for Qwen-Turbo and from 64.2% to 35.0% for Qwen3-8B.Position effects are less uniform: both systems have higher CDR at q5 than q1, but neither curve is monotonic.Source: §C.3 · p. 23, Figure 7 · p. 23, Table 8 · p. 23

D ADDITIONAL GBDI ANALYSES

• The additional GBDI analyses test equal-call view randomization, intervention guidance, answer recovery, and sensitivity to view count and ledger support.They follow the protocols in Appendix B.4 and use the reviewed reference in Appendix A.Source: D ADDITIONAL GBDI ANALYSES · p. 23

D.1 EQUAL-CALL VIEW RANDOMIZATION

• Random-5 yields higher CDR point estimates than Repeated-5 for all three systems, with paired intervals excluding zero for Qwen and MiniMax but not GLM.Random-5 also raises recall for all three, while precision and F1 do not improve uniformly and mean ledger size increases.Source: §D.1 · pp. 23–24, Table 9 · p. 24

D.2 VIEW AND GUIDANCE CONTROLS

• With the ledger held fixed, Skill raises QA accuracy by 5.8 points under Repeated-5 and 6.4 points under Random-5; the latter interval excludes zero.Random-5 raises CDR from 58.9% to 78.7%, while its estimated QA advantage over Repeated-5 is 4.2 points with an interval including zero.Source: §D.2 · p. 24, Table 10 · p. 24
• The aggregate guidance gain is concentrated in Logic, Formatting, and Missingness; Skill adds 10 correct answers on Formatting under Random, while reducing correct answers on Bad values by three.The factorial compares complete guidance text, including artifact-specific action rules and parsing, validation, and submission guidance.Source: §D.2 · p. 25, Table 11 · p. 25
• Under Random-5, Skill corrects 47 answers and changes 27 correct answers to incorrect, for a net gain of 6.39 points; the view-by-guidance interaction is uncertain.The interaction estimate is 0.64 points with a 95% CI of [−7.92, 9.20].Source: §D.2 · p. 25

D.3 INTERVENTION SKILL RULES

• Removing formatting rules lowers overall accuracy from 64.9% to 60.7%, while the recovery-rule contrast is less certain and the execution-check contrast is uncertain.The execution-check variant retains the required intervention turn, so it does not test a workflow without execution constraints.Source: §D.3 · pp. 25–26, Table 12 · p. 25

D.4 DISCOVERY AND ANSWER RECOVERY

• Even with complete discovery, a substantial share of instances still receive incorrect answers; with ledgers held fixed, Skill yields more corrections than regressions for every system.Some incomplete ledgers still yield the gold answer, and the final agent can inspect the complete table.Source: §D.4 · p. 26, Table 13 · p. 26

D.5 DISCOVERY BUDGET AND LEDGER SUPPORT

• Under the union rule, CDR rises from 45.5% with one view to 78.7% with five, while mean nominated rows rise from 6.92 to 10.03 and Empty FP from 64.6% to 76.9%.At five views, requiring support from two views reduces CDR by 9.9 points while raising precision from 72.5% to 73.6%; the analysis does not extrapolate beyond five views.Source: §D.5 · pp. 26–27, Table 14 · p. 27

D.6 FINAL QA BY VIEW COUNT

• Qwen Plus QA accuracy rises from 57.5% at M = 0 to 64.9% at M = 5, a paired gain of 7.3 points (95% CI [3.2, 11.9]).Intervals for the individual increments at M = 1 and M = 3 include zero.Source: §D.6 · pp. 27–28, Table 15 · p. 28

E.1 CLUSTER BOOTSTRAP

• Uncertainty estimates use clustered resampling to retain dependence among related observations, rather than treating repeated views as independent instances.The geometry analysis resamples base-task clusters; table-state and end-to-end QA comparisons resample source-table clusters.Source: §E.1 · p. 28, E UNCERTAINTY ESTIMATES · p. 28

E.4 GBDI DISCOVERY AND QA

• Paired 95% intervals for GBDI QA gains over the Code Agent baseline exclude zero for Qwen, GLM, MiniMax, and DeepSeek-V4.1-Flash, but not GPT.The comparison covers all 313 RADAR-T instances.Source: Table 20 · p. 30, §E.4 · p. 31

F.1 GEOMETRY DISCOVERY

• Direct discovery flags query-relevant rows with visible evidence without repairing the table or answering the question; the Code Agent also provides a read-only inspection interface.The Code Agent endpoint uses separate inspection and localization-submission commands, with at most five response rounds.Source: §F.1 · p. 32

F.2 TABLE-STATE DIAGNOSTIC

• The table-state diagnostic distinguishes original, error-marked, and reviewed-repaired inputs; markings identify verified errors, not corrections.The action notice directs the agent to correct a marked value only when the table supports a reliable replacement, or exclude its row if it cannot be repaired.Source: §F.2 · pp. 33–34

F.3 MULTI-VIEW INSPECTION SCHEMAS

• Multi-view discovery requests evidence for query-relevant unreliable values while keeping stable, nonordinal row identifiers to map records across table views.GPT-5 Mini’s schema also requests the displayed value and why the field can enter the requested computation.Source: §F.3 · p. 35

F.4 EVIDENCE LEDGERS

• The evidence ledger retains every row with at least one valid nomination, but treats nominations as leads to verify rather than proof of corruption or authorization to alter data.Before answering, the final agent checks each ledger row against the canonical table and directly computes the answer.Source: §F.4 · pp. 35–36

F.5 INTERVENTION GUIDANCE

• Intervention guidance permits EDIT only for an exact evidence-supported replacement, DROP only for a demonstrably unusable required record, and otherwise NO-OP.The unchanged answer program is rerun after justified operations are applied to work_df.Source: §F.5 · p. 36

Inconsistent Formatting

• The Inconsistent Formatting skill activates only when representation blocks parsing or comparison of an answer-relevant value whose semantic value remains recoverable.It requires role-specific parsing of complete tokens and says formatting alone never justifies dropping a row.Source: Inconsistent Formatting · p. 38