Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

1 Introduction

• Few studies offer controlled comparisons of diverse pretraining objectives while keeping model architecture constant.Representation-learning approaches can differ in architecture, training objective, or both.Source: §1 · p. 1
• The study isolates pretraining-objective effects by holding sentence-encoder architecture constant.It examines how different objectives affect information encoded in output representations.Source: §1 · pp. 1–2
• Function words are a focus because they are critical to language understanding but under-studied in representation learning.They contribute to compositional meaning, including identifying discourse referents and expressing relationships between entities or ideas.Source: §1 · pp. 1–2
• The paper releases nine new datasets, quality-controlled by linguists and non-linguist annotators, to support follow-up analysis.The datasets target function-word understanding.Source: §1 · p. 2

2.1 Approach

• Nine probing tasks target specific function-word types, using acceptability judgments or NLI according to the phenomenon.The suite contains four acceptability tasks and five NLI tasks, totaling 3,544 annotated examples.Source: §2.1 · pp. 2–3

2.2 Acceptability Judgment-Based Tasks

• Acceptability probes identify target constructions, then mutate half the sentences to create infelicitous versions; tasks cover wh-words, articles, conjunctions, and sentence boundaries.The end-of-sentence task forms running text by removing punctuation and capitalization from sentence pairs and testing valid versus incorrect segmentation.Source: §2.2 · p. 3

2.3 NLI-Based Tasks

• NLI probes mutate premise-hypothesis pairs to test how function-word choices affect entailment, targeting negation, prepositions, comparatives, quantification, and spatial relations.The negation task generates 16 patterns by applying or omitting lexical and explicit negation mutations to premises and hypotheses.Source: §2.3 · pp. 3–4

2.4 Annotation

• Annotators label examples, with majority judgments retained and examples without majority consensus discarded.Acceptability sets keep mutated sentences labeled unnatural and original sentences labeled natural, balancing labels and prioritizing perfect agreement.Source: §2.4 · p. 4

3.1 Pretraining Architecture

• A shared character-CNN and two-layer bidirectional LSTM encoder was used across pretraining tasks, with the encoder’s top-layer states and original word representations passed to task-specific models.Classification objectives used pairwise word attention and a matching-feature MLP; sequence-to-sequence objectives used an LSTM decoder initialized from the encoder output.Source: §3.1 · p. 5

3.2 Pretraining Tasks

• The seven pretraining objectives included language modeling, skip-thought, CCG supertagging, DisSent, NLI, machine translation, and image-caption matching.They used different training data and targets, including next-sentence generation, syntactic supertag prediction, discourse-marker prediction, and entailment classification.Source: §3.2 · pp. 5–6
• The translation and image-caption objectives trained encoders respectively on English–German translation and matching sentences to their corresponding images.The image-caption objective minimized cosine distance between sentence representations and image features.Source: §3.2 · p. 6

3.3 Classifiers for Probing Tasks

• For probing, the authors froze pretrained encoders for task-specific models: NLI probes used an MNLI-trained attention model, while acceptability probes used pooled states and an MLP.BERT instead was fine-tuned with a sequence-level classifier for acceptability tasks, while its fine-tuned MNLI classifier was used for NLI probes.Source: §3.3 · p. 6
• A general acceptability model trained on CoLA performed around chance when transferred directly to the acceptability probes.The authors suggest this may reflect CoLA’s intrinsic difficulty for their base model.Source: §3.3 · p. 6

3.4 Variation from Random Restarts

• Five random initializations of unpretrained encoders established a baseline variation of about one percentage point in average standard deviation across probing sets.The authors compared this restart variation with variation across pretrained models to assess whether pretraining measurably affected probing results.Source: §3.4 · p. 7

4.1 Overall Performance

• NLI outperforms other pretrained models on negation and conjunction, while CCG performs best on EOS, whose labels directly represent compositional structure.The NLI advantage on negation is specifically associated with understanding explicit “not,” rather than lexical antonymy.Source: §4.1 · p. 7
• Pretraining can help or hurt depending on the task: NLI helps negation, whereas image-caption matching and CCG lower performance on it.Skip-thought and image-caption pretraining also appear to hurt performance on spatial relations and prepositions relative to a randomly initialized encoder.Source: §4.1 · p. 7
• Language modeling is most useful on average, while CCG and NLI perform comparably within random variation.CCG achieves competitive results with an order of magnitude less training data than NLI and two orders less than language modeling.Source: §4.1 · p. 8
• BERT substantially improves many probing results and exceeds human performance on conjunctions and EOS, yet loses to an untrained baseline on wh-words and articles.Only a few tasks reach human-level performance, with conjunctions among the exceptions.Source: §4.1 · pp. 7–8

4.3 Data Size and Genre Effects

• More pretraining data does not imply better probing performance, and vocabulary overlap is not a significant predictor of probing results.The regression found no significant overall effect of vocabulary overlap (p = 0.39); no individual probing set was significantly affected after Bonferroni correction (all p > .05).Source: §4.3 · p. 8

4.2 Correlations between Pretaining Tasks

• Prediction overlap differs across pretraining objectives; image-caption matching and skip-thought diverge most from other tasks, and NLI and image-caption matching are the least-correlated pair.Figure 2 compares how often models trained on different tasks make identical predictions across probing sets.Source: §4.2 · p. 8, Figure 2 · p. 9

5 Related Work

• Task-based probing evaluates representations through tasks requiring specific linguistic knowledge, and prior work has also targeted phenomena such as agreement, ellipsis, and negative polarity.Related approaches have used recast NLI to probe semantic phenomena in machine-translation encoders.Source: §5 · p. 9
• Unlike adversarial modifications intended to preserve meaning, this paper’s structural mutations are designed to induce semantic changes and test comprehension of the modified expressions.Its modifications are more constrained and lexically targeted than Naik et al. (2018).Source: §5 · p. 9
• Unlike general-purpose diagnostic formats that aggregate phenomena, these tasks isolate individual function-word types with targeted modifications for narrower analysis.The NLI-style format follows work advocating NLI for diagnostic tasks and is also comparable in spirit to question answering, edge probing, and GLUE.Source: §5 · p. 9

6 Conclusion

• The nine function-word challenge tasks are architecture- and evaluation-setup-agnostic.They are used to compare pretraining objectives, and the paper hopes they will spur further work on function words and compositional meaning.Source: §6 · p. 9
• Different pretraining objectives generate different predictions, including an image-model bias toward concrete preposition senses, but no single objective performs best or worst across all tasks.The paper suggests that gaps in linguistic knowledge learned from one objective could be complemented by others.Source: §6 · p. 9
• The authors call for further research into productively combining pretraining objectives and modeling function words’ contribution to compositional meaning.Source: §6 · p. 9

A Experimental Details

• Image-caption matching trains sentence representations to minimize cosine distance from corresponding image features.The setup uses MSCOCO image-caption pairs and a ResNet-101 image encoder.Source: A Experimental Details · p. 12
• Dropout with p = 0.2 is applied after the input, each LSTM layer, and each task-specific classifier MLP layer.Source: A Experimental Details · p. 12
• Preprocessing uses Moses tokenization with a maximum length of 40 tokens, with BPE for translation and a 20,000-token output vocabulary for sequence-to-sequence tasks.The character-based word encoder requires no word-level vocabulary.Source: A Experimental Details · p. 12
• Training uses AMSGrad (Reddi et al., 2018), with validation every 1,000 iterations and stopping after 20 evaluations without a best result; learning rates are 1e-3 for text generation and 1e-4 otherwise.The rate is halved when validation fails to improve for more than 4 checks, and training also stops below 1e-6; the best checkpoint is loaded at the end.Source: A Experimental Details · p. 13
• Acceptability-task accuracy is averaged across test folds in 10-fold cross-validation, with the other nine folds split into training and development sets.Each fold has a balanced label ratio.Source: A Experimental Details · p. 13

• Table 2 reports MNLI development-set accuracy for each pretrained model.Source: Table 2 · p. 13

C Annotation Protocol

• Three annotators judged acceptability examples as natural, unnatural, or neither, and NLI pairs on a five-point inferrability scale.Source: C Annotation Protocol · p. 13
• The final probing dataset table reports pairwise agreement, unanimous-agreement rates, and individual annotator accuracy against expected labels for each task.Source: Table 3 · p. 13

D List of Prepositions Used for the NLI Probing Set

• The NLI probing set uses a list of prepositions including about, above, across, after, and without.Source: D List of Prepositions Used for the… · p. 13

E Random Initialization Variance

• Across five random initializations, reported figures were wh-words 51.37 (±0.36), def/indef articles 57.77 (±0.97), coordination 54.39 (±0.96), and EOS 52.74 (±1.30).Source: E Random Initialization Variance · p. 13, Table 4 · p. 13

F Subset Accuracy for the Negation Probing Set

• The NLI model’s negation improvement mostly comes from explicit negation with not rather than lexical negation with antonyms.Source: F Subset Accuracy for the Negation… · p. 14, Figure 3 · p. 14

G Prediction Overlap between Models

• Figure 4 shows prediction-overlap heatmaps between models pretrained with different objectives for all probing tasks.Source: G Prediction Overlap between Models · p. 14, Figure 4 · p. 15