Source-linked AI summary
Understanding tables with intermediate pre-training
Julian Martin Eisenschlos, Syrine Krichene, Thomas Müller
TL;DR
Table entailment is less studied than textual entailment despite requiring language understanding and reasoning over table structure and values. The paper adapts TAPAS with intermediate pre-training on synthetic and counterfactual statements, improving results on TabFact and SQA.
Problem
Table entailment is less studied than textual entailment, although it requires language and table-structure parsing as well as numerical and discrete reasoning.
Source: §1 · p. 1Method
The paper adapts TAPAS for entailment and introduces intermediate pre-training tasks using synthetic and counterfactual statements.
Source: §1 · p. 1Results
Takeaways & Limitations
The pre-training tasks also support SQA, and comparable accuracy to previous approaches is possible using only 10% of the data.
Source: §1 · p. 2, §8 · p. 9The methods were tested on English, mainly because the end-task resources were available for that language.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TabFact (Chen et al., 2020) and SQA datasets.
1 Introduction
Table entailment remains challenging because it combines language understanding with structured and numerical reasoning, while models lag human performance. The paper introduces intermediate pre-training tasks and column pruning, reporting gains on TabFact and SQA alongside efficiency and data-use results.
1 Introduction
- TabFact requires higher-order operations such as arg max, averaging, and comparison, and human accuracy is 18 points ahead of the best models.The paper notes that tabular data has received less pre-training and representation attention than text.Source: §1 · p. 1
- The paper introduces intermediate pre-training tasks based on synthetic and counterfactual statements, using logical expressions to generate synthetic examples.The synthetic statements draw on operations that filter, combine, and compare table information.Source: §1 · p. 1
- Doubling inference speed costs less than one accuracy point, while using 10% of the data yields accuracy comparable to previous approaches.The paper also releases its pre-training checkpoints, data-generation code, and training code.Source: §1 · p. 2
- Pre-training improves TabFact over previous models by 6 points with BERT-base and 9 points with BERT-large, and improves SQA state-of-the-art by 4 points.Using only 10% of the data yields comparable accuracies to previous approaches.Source: §1 · p. 2
2 Model
The model adapts a BERT-based architecture to encode table structure and classify whether a statement is entailed by a table. It represents the statement and flattened table together, adding structural embeddings before predicting from the [CLS] representation.
2 Model
- The statement and table are tokenized and concatenated with [CLS] and [SEP] tokens, with the table flattened row by row and no added cell or row separators.Source: §2 · p. 2
- Six learnable embedding types include BERT-style token, position, and segment embeddings, plus table column, row, and numeric-rank embeddings.Column and row embeddings encode cell position; rank embeddings encode a numeric cell's rank within its column.Source: §2 · p. 2
- A transformer contextualizes the combined input, and a single-hidden-layer network uses its [CLS] output to model entailment probability.The hidden layer uses tanh and the output layer uses sigmoid.Source: §2 · p. 2
3 Methods
The methods introduce two intermediate pre-training tasks that require sentence–table reasoning, then prune tables to fit long examples within BERT input limits. The tasks generate counterfactual and synthetic statements from Wikipedia tables.
3 Methods
- The two pre-training tasks address a limitation of blank-filling approaches: they require sentence–table reasoning and complex operations grounded in table content.The counterfactual data is drawn from Wikipedia tables and related sentences; tables are recursively split until they contain at most 50 cells, producing 3.7 million tables.Source: §3 · p. 3, §3.1 · p. 3
3.1 Counterfactual Statements
- Counterfactual training pairs replace same-type entities with alternatives, aiming for minimally different statements whose truth depends on the table; 4.1 million pairs were extracted.Of these, 546 thousand have a supporting mention; the remainder do not.Source: §3.1 · p. 3
3.2 Synthetic Statements
- Synthetic statements compare two simplified SQL-like expressions sampled from a grammar, using filters, aggregations, counts, and comparisons over table columns.The expressions are constrained to comparable domains, and positive and negative statements are generated for each Wikipedia table, yielding 3.7M pairs.Source: §3.2 · pp. 3–4
3.3 Table pruning
- Column pruning ranks columns by relevance and adds them in descending order, skipping columns that exceed the input limit; heuristic exact match scores statement–column token Jaccard overlap.Heuristic entity linking is the baseline pruning method; word2vec, character-overlap, and TF-IDF variants generally perform worse than heuristic exact match.Source: §3.3 · p. 4
4 Experimental Setup
Experiments use TAPAS initialized from its public checkpoint, train on the proposed data, and fine-tune on TABFACT or SQA. The evaluation includes both datasets and compares against prior table-entailment models.
4 Experimental Setup
- Results are reported as median accuracy over nine runs: three pre-training runs crossed with three fine-tuning runs, with error margins estimated as half the interquartile range.The setup also explores sequence lengths and their speed–downstream-result trade-off.Source: §4 · pp. 4–5
4.1 Datasets
- TABFACT contains 118,000 Wikipedia table–statement pairs, while SQA contains 6,066 sequences averaging 2.9 questions per sequence.The experiments use the standard splits and official evaluation procedures for both datasets.Source: §4.1 · p. 5
4.2 Baselines
- The baselines include TABLE-BERT, which predicts statement truth without table-structure embeddings, and LPA, which predicts and executes a latent program to verify statements.LPA uses lexical heuristics to restrict program search and weak supervision from program outputs.Source: §4.2 · p. 5
5 Results
Intermediate pre-training improves TABFACT and SQA results, while subset experiments show gains with less training data and pruning reduces input cost with modest accuracy loss.
5 Results
- More than 6 points (Base) and 9 points (Large): the approach exceeds prior TABFACT state of the art; the best comparable-parameter BERT-base model also beats TABLE-BERT by more than 12 points.Joining counterfactual and synthetic data improves accuracy over either alone.Source: §5 · p. 5, Table 2 · p. 6
- With 10% of the data, counterfactual plus synthetic pre-training is comparable with LOGICALFACTCHECKER; on smaller subsets, the combined setup consistently beats either alone.At 5% of examples, C+S and synthetic-only surpass TABLE-BERT; C and SQA do so at 10%.Source: §5 · pp. 5–6, Figure 3 · p. 6
- Almost 4 points: the Large model outperforms the MASK-LM model on both SQA average question accuracy and sequence accuracy.The best Base model is comparable to the previous state of the art.Source: Table 3 · p. 6, §5 · p. 6
- Both counterfactual and synthetic data outperform MASK-LM on SQA, and the Large model improves on MASK-LM by almost 4 points on both reported metrics.The best Base model is comparable to the previous state of the art.Source: Table 3 · p. 6
- At input length 256, accuracy is 0.7 points below the full-input model, while training and application are twice as fast.HEM outperforms HEL at all input sizes; at length 128 it beats TABLE-BERT by 6.6 points.Source: §5 · p. 6, Table 4 · p. 7
- Shorter pre-training inputs of 512 and 256 give similar accuracy, while length 128 is about 1 point lower.The authors use shorter pre-training inputs to improve training time before fine-tuning.Source: §5 · p. 7
6 Analysis
The analysis groups examples by operations inferred from wording, then examines model accuracy, error distribution, and agreement across repeated runs.
6 Analysis
- Intermediate-task data reduce errors most for superlatives (39%) and negations (31%).The analysis groups examples into aggregations, superlatives, comparatives, negations, and cases requiring multiple operations; groups are assigned using word heuristics.Source: §6 · p. 7
- Many examples that remain errors after pre-training require multi-hop reasoning or complex numerical operations.Figure 4 also presents examples where the proposed model is correct in most of nine trials while the MASK-LM baseline is not.Source: §6 · p. 7
- Across nine runs, all models agree on the correct answer for 55.5% of examples with counterfactual and synthetic pre-training, versus 24.2% with MASK-LM.The authors interpret the increase as substantially reduced guessing.Source: §6 · p. 7
7 Related Work
Prior work spans structured-data entailment, table-aware BERT models, and specialized pre-training objectives. The paper builds on these strands with harder synthetic and counterfactual tasks aimed at table entailment.
7 Related Work
- Across nine runs, all models answered 55.5% of samples correctly with Synthetic + Counterfactual pre-training, versus 24.2% with MASK-LM.Figure 4 highlights persistent errors involving deeper reasoning chains or more complex numerical operations.Source: Figure 5 · p. 8, Figure 4 · p. 8
- TAPAS pre-training based on detecting whether a table and sentence came from the same article was ineffective, motivating harder tasks for TABFACT.TAPAS uses learned row, column, and numerical-order embeddings.Source: §7 · pp. 7–8
- The paper’s approach combines special table embeddings with synthetic and counterfactual pre-training, extending BERT-based systems for entailment over tables.Prior TABFACT approaches include flattened tables represented with textual templates.Source: §7 · p. 8
- Intermediate task fine-tuning has been used to teach higher-level abilities, and this work targets discrete and numerical reasoning.The related work distinguishes it from domain-adaptive pre-training for covariate shift.Source: §7 · p. 8
- Counterfactual generation uses other table rows to produce plausible negatives and replaces dates and numbers, unlike an entity-replacement approach described by Xiong et al. (2020).The paper relates paired similar examples with different labels to Kaushik et al. (2020) and Gardner et al. (2020).Source: §7 · p. 8
- Generated numerical examples motivate synthetic pre-training, while related work also applies synthetic data to semantic parsing and textual entailment.The paper describes its table-oriented generation method as better suited to table entailment than the approaches discussed.Source: §7 · p. 9
8 Conclusion
The paper concludes that synthetic and counterfactual pre-training improves TABFACT and SQA, while column selection can make training and inference faster with an accuracy trade-off. The appendix documents experimental setup, hyper-parameter search, and model configurations.
8 Conclusion
- Pre-training with synthetic and counterfactual tasks yields state-of-the-art TABFACT results and improves SQA, including with only a subset of training data.The paper adapts TAPAS to binary classification and reports column-selection speedups of 2× at about a one-point accuracy drop, or 4× with a larger drop but higher accuracy than previous approaches.Source: §8 · p. 9
Appendix
- The appendix covers experimental setup, column-pruning details, full SQA results, held-out pre-training accuracy, and trigger words used in the analysis.It also provides model and TABFACT dataset information.Source: Appendix · p. 13
A.1 Hyper-Parameter Search
- Hyper-parameter search used 200 Bayesian-optimization runs, with validation accuracy assessed after 8,000 steps and median values retained from the top 20 trials.From 10 to 200 trials, estimated accuracy increased 0.4% and standard deviation decreased from 2% to 1.3%.Source: §A.1 · p. 13
A.2 Hyper-Parameters
- Pre-training used input length 256 and fine-tuning length 512 by default, with 80,000 steps, learning rate 2e−5, and warm-up ratio 0.05.For SQA, fine-tuning started from an intermediate-pre-training checkpoint and otherwise used Herzig et al. (2020)’s model and hyper-parameters.Source: §A.2 · p. 13
A.3 Number of Parameters
- The model sizes match BERT: 110M parameters for base and 340M for large models.Source: §A.3 · p. 13
A.4 Training Time
Input length strongly affects processing speed and training time. The appendix also illustrates the model’s input representation using six embedding types.
A.4 Training Time
- At input lengths 128, 256, and 512, BERT-Base processes approximately 8,700, 5,100, and 2,600 examples per second, respectively.The corresponding training times are approximately 78, 133, and 262 minutes.Source: §A.4 · p. 13
- A BERT-Large model processes approximately 800 examples per second at length 512 and takes 14 hours to train.Source: §A.4 · p. 13
B Model
- The model input representation uses six embedding types, as illustrated in the appendix figure.The representation is depicted following Herzig et al. (2020).Source: B Model · p. 13
C Dataset
The appendix points to TABFACT dataset statistics and describes column-selection methods for pruning table inputs. It also specifies how SQA result variability is summarized.
C Dataset
- Table 6 presents statistics for the TABFACT dataset, while Figure 7 shows the validation-set input-length histogram after BERT tokenization.Source: Table 6 · p. 13, C Dataset · p. 13, Figure 7 · p. 14
D Columns selection algorithm
- Column pruning ranks columns by statement relevance and retains those that fit the token budget; scoring options include word embeddings, inverse word frequency, and character n-grams.The algorithm continues through the score order when a column does not fit, retaining the maximum number that fit under the budget.Source: D Columns selection algorithm · p. 14
E SQA
- SQA results are reported for the first development fold and test set, with error margins defined as half the interquartile range across nine runs.This range contains half of the runs and serves as a robustness measure.Source: E SQA · p. 15
F Pre-Training Data
The pre-training data includes synthetic and counterfactual examples, with held-out evaluation showing some bias but a gap between models trained with and without tables. Removing counterfactual statements without supporting mentions discards most examples and hurts results.
F Pre-Training Data
- Approximately 1% of the pre-training data is held out; both datasets show bias, yet table-trained models retain a sufficient accuracy gap over models trained without tables.The authors interpret the gap as evidence that the data remains useful despite accuracy above 50% without tables.Source: F Pre-Training Data · p. 15
- Synthetic examples are solved almost perfectly, while counterfactual held-out accuracy is 84.3%, with some statements lacking enough information to decide their truth.Table 9 reports accuracy on the synthetic and counterfactual held-out sets.Source: F Pre-Training Data · p. 15, Table 9 · p. 15
- Removing counterfactual statements without supporting entities raises the chance of incorrect negative pairs, but discards 7 out of 8 examples and hurts results.Table 10 compares counterfactual training with and without statements lacking support mentions.Source: F Pre-Training Data · p. 15, Table 10 · p. 15
G Salient Groups Definition
The salient groups are defined through trigger words linked to manually identified operations, and the grouping heuristic is checked by manually inspecting samples. The section also reports SQA evaluation metrics and trial aggregation.
G Salient Groups Definition
- Trigger words link salient groups to operations identified as commonly needed to solve the task.Table 11 lists the trigger words for the different groups.Source: G Salient Groups Definition · p. 15, Table 11 · p. 16
- Manual inspection of 50 samples from each group validated the heuristic with higher than 90% accuracy.Source: G Salient Groups Definition · p. 15
- Table 8 reports SQA development and test results using ALL, SEQ, and question-position accuracy, with medians over 9 trials.Errors are estimated with half the interquartile range.Source: Table 8 · p. 16