INFOTABS: Inference on Tables as Semi-structured Data

1 Introduction

• The task separates understanding a premise’s linguistic structure from reasoning over implicit information conveyed by juxtaposed text snippets.For example, judging H2 requires combining information across multiple rows and interpreting “Mixed gender” alongside the fact that Dressage is a sport.Source: §1 · pp. 1–2
• INFOTABS contains 23,738 premise-hypothesis pairs from 2,540 Wikipedia info-boxes, with short human-written hypotheses labeled as entailment, contradiction, or unrelated.The authors report that label decisions typically compose multiple inference types across table rows and use world knowledge; separate subsample verification experiments support dataset quality.Source: §1 · p. 2
• All evaluated approach families underperform across the three test sets, and the paper presents this as evidence that existing NLI strategies struggle with the task.The approaches include word-overlap models, models exploiting premise structure, and derivatives of state-of-the-art NLI systems.Source: §1 · p. 2
• The challenge test sets introduce controlled differences to probe whether models rely on superficial patterns rather than reasoning about the table information.The paper presents these tests alongside the standard train/dev/test split.Source: §1 · p. 2

2 The Case for Reasoning about Semi-structured Data

• Semi-structured data ranges from verbose web pages to terse fact sheets and tables, with structured scaffolding populated by free-form text and potentially no fixed schema or explanatory context.People routinely infer relationships among elements and fill gaps using expectations about those relationships.Source: §2 · p. 2
• Reasoning can involve cell-level lookup, lexical interpretation, recognizing text types, aggregating rows, and combining temporal information with world knowledge.The examples include interpreting 1912 as a year and combining dressage’s non-contact status with who competes in it.Source: §2 · p. 2
• The paper proposes NLI with tabular premises and textual hypotheses, and introduces INFOTABS to evaluate reasoning about semi-structured information.The task asks whether the table entails, contradicts, or is unrelated to a hypothesis.Source: §2 · p. 2

3 The Need for Multi-Faceted Evaluation

• NLI datasets can contain artifacts that enable hypothesis-only prediction, so high performance on one fixed test set does not establish that a model reasons about the premise.The authors argue for multiple similarly sized test sets with controlled differences to assess how models handle changes that may differ from the training data’s surface patterns.Source: §3 · pp. 2–3
• The α1 test set has a distribution similar to training data in hypothesis lexical makeup and premise domains.It serves as the comparatively similar-distribution test set among the three introduced here.Source: §3 · p. 3
• For α2, expert annotators change as few words as possible, reverse entailments and contradictions, and leave neutral labels unchanged.The altered sentences are intended to confuse models that learn superficial textual artifacts.Source: §3 · p. 3
• The cross-domain α3 set uses premises from domains absent from training while generally requiring similar kinds of reasoning.The authors expect models overfitting to domain-specific artifacts to underperform on α3.Source: §3 · p. 3
• The three test sets are not presented as an exhaustive set of overfitting dimensions; future work may identify further dimensions and construct corresponding tests.The paper anticipates additional ways models may overfit their training data.Source: §3 · p. 3

4 The INFOTABS Dataset

• The dataset draws on 2,540 Wikipedia info-boxes, excluding tables with fewer than three rows, non-English cells, or technical information requiring expertise.The authors removed images and split large tables at subheadings; the resulting tables use key-value pairs.Source: §4 · p. 3
• Annotators wrote three self-contained sentences per table: one true, one false, and one that may or may not be true given the table.Instructions discouraged relying on information outside the table, trivial edits, or simply restating table contents.Source: §4 · p. 3
• The α2 examples come from hypotheses similar to α1 but separate tables, with about 2.2 words changed on average; α3 uses 200 tables from domains absent from training.No more than two words change in 72% of α2 hypotheses, and the authors describe α3 hypotheses as involving similar reasoning types but different distributions.Source: §4 · p. 4
• The 23,738 collected sentences are split almost equally among entailments, contradictions, and neutrals, with similar average hypothesis lengths across splits.The dataset also partitions tables into training, development, α1, and α2 sets, with annotator distributions in development and test splits kept similar to training.Source: §4 · p. 4
• Human verification found Cohen’s Kappa scores between 0.75 and 0.80 and majority agreement ranging from 93% to 97% across splits.The majority label matched the gold label in 80% to 84% of cases across splits.Source: §4 · p. 4

5 Reasoning Analysis

• Reasoning categories included simple lookup, multi-row reasoning, and subjective or out-of-table inference.The latter includes value judgments or references to information absent from the table and neither well-known nor common sense.Source: §5 · p. 4
• Many examples required multiple rows and temporal or numerical reasoning; Dev and α3 examples needed averages of 2.32 and 1.79 distinct reasoning kinds, respectively.Only a small number could be resolved by simple lookup, simple negation, or syntactic alternation.Source: §5 · pp. 4–5
• 48.75% of Dev examples required world knowledge and common sense, compared with 25.72% in MultiNLI.Annotators used knowledge about weather, seasons, and social and cultural norms to infer information not explicit in tables.Source: §5 · p. 5
• Neutral hypotheses were more often subjective or out-of-table, yet models found neutrals almost as hard as other labels, with an approximately 3% F-score gap, except on α2.Source: §5 · p. 5
• α3 tables came from different domains and distributions than the other splits, with corresponding differences in reasoning-type distributions.For example, temporal reasoning is not expected from tables without temporal quantities.Source: §5 · p. 5

6.1 Representing Tables

• Tables are represented as templated row sentences, selected sentences closest to the hypothesis, key-value sequences, or contextualized key-value vectors.Source: §6.1 · p. 5

6.2 Modeling Table Inferences

• The study applies standard NLI models to these representations, including unigram/bigram SVMs, BERT-class encoders, and decomposable attention for TabAttn.Source: §6.2 · pp. 5–6

6.3 Results

• Pre-trained NLI systems perform poorly on INFOTABS, while full-premise input slightly improves performance over WMD-1.Source: §6.3 · p. 7, Table 5 · p. 7
• ∼9% over RoBERTaB and ∼14% over BERTB, RoBERTaL performs better; providing the full premise also beats selecting a subset of sentences.Source: §6.3 · pp. 7–8, Table 6 · p. 7
• Performance is worse on α2 and α3 than α1, indicating sensitivity to lexical patterns in α2 and domain-specific patterns in α3.Source: §6.3 · p. 8
• TabFact’s structural representation improves over RoBERTaL+Para, but TabAttn does not improve performance; both structured approaches also drop on α2 and α3.Source: §6.3 · p. 8, Table 7 · p. 8
• On Dev, RoBERTaL performs poorly on knowledge and common sense, multi-row, coreference, and temporal reasoning examples.Source: §6.3 · p. 8
• Task-trained models remain well below human performance, and the authors recommend using all three test results rather than any single split to characterize a model.Source: §6.3 · p. 8

• NLI datasets range from PASCAL RTE and image-caption SNLI to multi-domain MultiNLI, converted QA or Winograd tasks, and adversarially constructed datasets.Source: §7 · p. 8
• Recent reasoning datasets address abductive, common-sense, temporal, numerical, and multi-hop reasoning.Source: §7 · pp. 8–9
• Prior semi-structured-data work includes table, graph, and database tasks, including semantic parsing and question answering over table entries resembling database records.Source: §7 · p. 9
• Unlike TabFact’s homogeneous tables and entailment/contradiction labels, INFOTABS uses heterogeneous premises, includes an undetermined class, covers more varied reasoning, and uses α2 and α3 to examine annotation and domain artifacts.Source: §7 · p. 9
• Prior work reports that NLI models can exploit spurious patterns, motivating controlled test sets that probe artifacts along specific dimensions.Source: §7 · p. 9

• INFOTABS combines heterogeneous semi-structured premises with natural-language hypotheses and encompasses several kinds of inference.Source: §8 · p. 9
• Its multiple test sets are designed to challenge models that learn superficial correlations rather than reason about the information.Source: §8 · p. 9
• Derivatives of several popular model classes find this inference task challenging, and the authors expect INFOTABS to serve as a testbed for models and representations handling semi-structured information.Source: §8 · p. 9

A Examples of Data

• The table-reasoning inventory adapts GLUE diagnostics and includes lexical, named-entity, numerical, temporal, coreference, and quantification categories.Source: B Reasoning for INFOTABS · pp. 12–13
• The annotation scheme allows multiple reasoning types per pair but marks a repeated type only once, and assumes each table is correct and complete for relevant field information.Source: B Reasoning for INFOTABS · p. 12
• Multi-row reasoning requires multiple table rows, while entity-type reasoning identifies an entity’s type and uses it to infer something about it.A simple lookup literally restates table information; entity-type reasoning is separate from multi-row reasoning even when identifying the type uses multiple rows.Source: B Reasoning for INFOTABS · p. 12
• The inventory also distinguishes explicit negation, knowledge and common sense, syntactic alternations, ellipsis, and subjective or out-of-table inferences.World knowledge and common sense are not separated because the distinction is not always clear for table reasoning.Source: B Reasoning for INFOTABS · pp. 12–14

C INFOTABS Worker Analysis

• The top 40 frequent annotators produced about 90% of the data.Source: Figure 6 · p. 14, C INFOTABS Worker Analysis · p. 14

D INFOTABS Dataset Statistics

• Across splits, premises and annotators overlap substantially except for α3 premises; α3 tables are longer, and most splits share annotators.Source: D INFOTABS Dataset Statistics · p. 14, Table 8 · p. 15
• The dataset statistics examine hypothesis length as a possible source of classification bias and report that table-hypothesis word overlap is almost similar across labels.Source: D INFOTABS Dataset Statistics · p. 14, Table 9 · p. 15, Table 10 · p. 15
• The split-wise category statistics group categories occurring below 3% into “Other.”Source: D INFOTABS Dataset Statistics · p. 14

E F1 Score Analysis

• On the α2 test set, neutral is much easier than entailment and contradiction because those two labels were adversarially flipped.Source: E F1 Score Analysis · p. 15
• α3 is the hardest split, followed by α2 and α1, and neutral is almost as hard as the other labels in evaluations except α2, with only an ∼3% F-score gap to the next-best label.Source: E F1 Score Analysis · p. 15, Table 13 · p. 16
• The analysis reports F1 scores (%) for various baselines, with all models trained using RoBERTaL, and category distributions (%) for splits excluding α3 and for α3 separately, each with “Others” below 3%.Source: Table 11 · p. 15, Table 12 · p. 15, Table 13 · p. 16

F Statistics of INFOTABS Verification

• 84.8% of individual labels matched the majority label on average, while 75.15% matched the gold label on average across verified splits.Source: F Statistics of INFOTABS Verification · p. 16
• Across splits, at least 3, 4, and 5 verifiers agreed on >95.85%, >74.50%, and 43.91% of examples, respectively; those agreements matched gold on >81.76%, >67.09%, and 40.85%.Source: F Statistics of INFOTABS Verification · p. 16
• Five annotators verified each pair, and the analysis reports agreement among verifiers as well as matches to the intended gold label.Source: Table 14 · p. 16, F Statistics of INFOTABS Verification · p. 16