Source-linked AI summary
TabFact: A Large-scale Dataset for Table-based Fact Verification
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, William Yang Wang
TL;DR
Fact verification has been studied mainly with unstructured evidence, so this paper introduces TabFact for semi-structured table evidence and develops Table-BERT and LPA for its mixed reasoning demands. The models achieve reasonable but similar accuracy while remaining far below human performance, exposing unresolved challenges in combining linguistic and symbolic reasoning.
Problem
Fact verification under structured evidence such as tables, graphs, and databases remains under-explored compared with verification using unstructured text.
Method
The paper constructs TabFact and develops Table-BERT for linearized table–statement classification and LPA for executable latent program synthesis over tables.
Results
Both proposed models achieve reasonable accuracy but remain far below human performance.
Takeaways & Limitations
TabFact establishes semi-structured fact verification as a research direction requiring models that perform both linguistic and symbolic reasoning.
Takeaways & Limitations
LPA covers the correct rationale in roughly under 30% of cases and can still reach 65% overall accuracy through spurious reasoning.
Abstract
from arXiv · showhide
The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. The data and code of the dataset are provided in \url{https://github.com/wenhuchen/Table-Fact-Checking}.
1 INTRODUCTION
Fact verification asks whether a textual hypothesis is entailed or refuted by evidence, but prior work largely addresses unstructured text. This paper targets semi-structured evidence and proposes two complementary approaches for its mixed linguistic and symbolic reasoning demands.
- Existing fact-verification studies mainly use unstructured text, leaving structured evidence such as tables, graphs, and databases under-explored.
- TabFact studies verification against semi-structured tables, where statements must be judged as entailed or refuted.
- Latent Program Algorithm links entities, filters candidate APIs, searches executable programs, and uses a discriminator to select consistent traces.
- Both models achieve reasonable accuracy but remain far below human performance, motivating methods that combine linguistic and symbolic reasoning.
2 TABLE FACT VERIFICATION DATASET
The dataset is built from filtered Wikipedia tables and crowdsourced statements collected through simple and complex channels, followed by rewriting, verification, and quality control. Its design targets diverse reasoning difficulty while reducing annotation artifacts and retaining explicitly supported binary claims.
- The authors extract Wikipedia tables with captions, filter overly complicated tables, and retain relatively clean tables with fewer than 50 rows and 10 columns.
- Annotators generate statements through simple and complex channels to obtain different difficulty levels and reasoning patterns.
- Negative statements are created by rewriting entailed statements while preserving their sentence content, reducing exploitable linguistic cues.
- Quality control rejects statements with errors, vague claims, or unsupported intermediate cases requiring knowledge beyond the table.
- The dataset statistics compare simple and complex channels and report their higher-order operation distributions and train, validation, and test divisions.
3 MODELS
The paper formulates table verification as binary classification over table–statement pairs and develops both neural sequence encoding and latent program synthesis models. Table-BERT compresses relevant tables into sequences, while LPA searches executable programs and ranks their consistency.
- Each example consists of a table, a natural-language statement, and a binary label indicating whether the table entails the statement.
- Latent Program Algorithm: LPA treats verification as weakly supervised program synthesis because gold programs are unavailable.
- Latent Program Algorithm: LPA searches programs by linking statement entities, triggering plausible APIs, executing typed intermediate values, and storing candidates in caches.
- Latent Program Algorithm: A Transformer-based discriminator ranks candidate programs using statement–program representations and weak labels derived from execution consistency.
- Table-BERT: Table-BERT links statement entities, retains relevant table columns, linearizes the resulting sub-table, and classifies the table–statement pair with pretrained BERT.
- Table-BERT: The model concatenates the linearized sub-table and statement, derives a BERT representation, and predicts entailment probability through an MLP.
4 EXPERIMENTS
The experiments compare neural, symbolic, and program-synthesis approaches on simple, complex, and test partitions, alongside human evaluation. Results show complementary strengths, substantial spurious reasoning in LPA, and unresolved weaknesses in both methods.
- Evaluation setup: The evaluation uses standard validation and test sets, separate simple and complex test partitions, and a 2K-sample human-evaluation set.Each human-evaluation pair is assigned to five workers, with majority voting used to approximate human judgments.
- Evaluation setup: Table 2 reports model and human performance in percentage, including T+F and F+T table-fact orderings and a modified NSM baseline.The caption defines T+F as table followed by fact and F+T as fact followed by table.
- Results: LPA-Ranking performs best among LPA variants, while LPA remains on par with Table-BERT on the simple and test splits without external pre-training.The results attribute LPA-Ranking’s advantage to suppressing spurious programs through ranking rather than voting.
- Limitations: 58% of sentences are correctly linked, systematic search recalls 51% of correct-link cases, and LPA’s overall accuracy is 65%.The resulting coverage of the correct rationale is roughly under 30%, indicating severe spurious-program behavior.
- Limitations: Table-BERT can misclassify simple cases while correctly classifying hard cases, whereas LPA is more consistent but requires handcrafted operations and accurate entity linking.The paper identifies combining the two methods as an open question.
- Additional resources: The dataset is supplemented with roughly 1,400 human-annotated programs paired with statements for bootstrapping semantic parsers or providing rationales.The annotations cover operations including superlative, counting, comparison, and uniqueness.
5 RELATED WORK
The paper positions table-based fact verification at the intersection of natural language inference, table question answering, program synthesis, and fact checking. It extends these lines of work to semi-structured evidence and emphasizes the distinctive compositional demands of verification.
- Natural Language Inference & Reasoning: Natural language inference determines whether a hypothesis can be inferred from a premise, and TabFact generalizes this setting to semi-structured tables treated as collections of premises.The related-work discussion connects TabFact to RTE, NLI, and multimodal statement-verification tasks.
- Table Question Answering: Unlike table question answering, TabFact provides no question-type signals that specify the answer form.Its statements must be verified directly against semi-structured evidence.
- Table Question Answering: A TabFact claim is false when any part contains misinformation, requiring decomposition into multiple sub-clauses or question-answer pairs.This conjunctive structure requires models to identify the component pairs and their relationships while handling longer semantic forms and harder logical inference.
- Program Synthesis & Semantic Parsing: Program synthesis and semantic parsing provide precedents for converting language into executable representations in question answering, navigation, code generation, and SQL synthesis.These approaches motivate the paper’s symbolic reasoning direction.
- Fact Checking: Earlier fact-checking datasets mainly use textual evidence, while AggChecker uses relational databases; TabFact contributes a much larger dataset for this direction.The comparison places TabFact between text-based fact checking and database-supported verification.
6 CONCLUSION
The paper frames semi-structured fact verification as an under-explored problem and addresses it with a large-scale dataset and two complementary reasoning methods. It concludes by calling for architectures that combine linguistic and symbolic reasoning.
- Conclusion: The paper studies semi-structured fact verification, constructs a large-scale dataset, and proposes Table-BERT and LPA.The methods are based respectively on a pre-trained natural language inference model and program synthesis.
- Conclusion: Future work should pursue architectures capable of performing both linguistic and symbolic reasoning.The conclusion presents this as the direction the dataset and methods are intended to encourage.
A.1 FUNCTION DESCRIPTION
TabFact represents statements through functions covering aggregation, logical operators, comparisons, ordering, uniqueness, quantification, and basic cell-level facts. The appendix illustrates these functions and lists trigger words used to narrow search.
- Aggregation: The appendix defines functions for aggregation, such as averaging ages or totaling scores.Aggregation handles statements that summarize values across table records.
- Negation: Negation functions capture claims involving absence or failure, such as not achieving the best score or never exceeding a threshold.These examples express negative conditions over table facts.
- Comparison: Superlative and comparative functions represent highest or lowest values and pairwise higher-score relationships.They encode both extremal and relative comparisons.
- Ordering and uniqueness: Ordinal functions encode ranked positions such as first or second, while unique functions count distinct entities or test whether duplicates exist.These operations capture ordering and distinctness properties.
- Quantification and direct facts: The all and none functions represent universal conditions and statements that require no higher-order operation beyond direct facts.Examples include every train departing in the morning and a player having a specified score or country.
C ERROR ANALYSIS
The error analysis contrasts symbolic execution's coverage limitations with Table-BERT's difficulty modeling long-range dependencies, while each model performs better on different reasoning categories.
- Symbolic: Symbolic execution fails when entity linking cannot identify implicitly referenced table cells.Its failure cases arise from entity-linking or function-coverage problems.
- Symbolic: Symbolic execution also cannot handle linguistic or mathematical implications without a defined function.For example, it lacks a function that maps tennis scores to the claim that a player won two games.
- BERT: Table-BERT struggles when relevant information is separated by long spans in the linearized table.Long dependencies and memorizing history information challenge the model despite its lack of coverage problems when the whole table is available.
- Statistics: Linguistic-inference statements are handled much better by BERT, whereas LPA performs only slightly above random guessing.The comparison uses 200 validation samples grouped into linguistic inference, trivial, aggregation, superlative, count, and other logic categories.
- Statistics: LPA outperforms BERT on higher-order logic cases, especially statements involving Count and Superlative operations.BERT handles trivial cases well, which the analysis attributes to its horizontal scan order.
D REASONING DEPTH
The analysis measures claim difficulty by the number of logic-inference steps required by LPA. Most reasoning depths fall between four and seven steps.
- D REASONING DEPTH: 4 to 7 reasoning steps contain the concentrated portion of the claim-depth histogram.The steps measure how many logic-inference operations are required to verify claims.
- D REASONING DEPTH: The reasoning-depth distribution indicates that fact verification in TabFact is difficult.The analysis presents this conclusion in connection with LPA's breadth across a large semantic space.
E WHETHER TO KEEP WIKIPEDIA CONTEXT
The annotation study examines whether Wikipedia titles should provide context. Titles are supplied as constrained context and serve as placeholders that make statements sound more natural.
- E WHETHER TO KEEP WIKIPEDIA CONTEXT: The study compares worker annotations with and without Wikipedia-title context.The comparison is motivated by differing context practices in WikiTableQuestion and WikiSQL.
- E WHETHER TO KEEP WIKIPEDIA CONTEXT: Annotators receive the Wikipedia title but may not add unrelated background information.The title acts only as a placeholder in statements to make them sound more natural.
F ENTITY LINKING
The entity-linking system finds candidate table entities through longest-string matching and resolves multiple candidates using minimum edit distance. A transformer-based discriminator then scores statement–program matches.
- F ENTITY LINKING: Longest-string matching identifies candidate entities in the table, with minimum edit distance resolving multiple candidates.The procedure is illustrated in the entity-linking system visualization.
- F ENTITY LINKING: Two transformer encoders represent the program and statement before an MLP classifies the verification label.The discriminator concatenates the encoders' [CLS] outputs to compute the statement–program matching probability.
H HIT INTERFACE
The HIT interface guides annotators to create diverse, non-trivial table-supported statements and contradictory rewrites while screening out trivial, ambiguous, unverifiable, or erroneous examples.
- Eight expert reviewers perform sanity checks on each HIT to help ensure the annotated dataset is clean and meets the requirements.
- Annotators are instructed to compose statements supported by the table, with a minimum length of nine words and varied descriptive types.
- Non-trivial statements require understanding multiple table rows and may involve summarization, comparison, negation, relations, inclusion, superlatives, aggregation, rephrasing, or combinations.
- The interface presents examples contrasting rejected trivial statements with accepted non-trivial statements, including superlative, comparative, relational, summarization, rephrase, combination, negation, and inclusion forms.
- For fake statements, annotators must preserve fluency while making the statement explicitly contradictory, clear, natural, and verifiable from the table.
- Erroneous inputs receive N/A when they contain serious grammatical errors, serious misspellings, or facts contradictory to the table.