Source-linked AI summary
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Nora Kassner, Hinrich Schütze
TL;DR
The paper asks whether pretrained language models store and use factual knowledge in a human-like way. It introduces negation and mispriming probes built on LAMA, finding that models often ignore negation and are distracted by misleading context. The results suggest that LAMA-style question answering relies more on shallow pattern matching than learned factual knowledge and inference.
Problem
The paper investigates whether pretrained language models can adequately recall factual knowledge when cloze questions contain negation or distracting context.
Method
The paper extends LAMA with positive/negative negation probes and automatically generated misprimes preceding cloze questions, evaluating pretrained language models.
Results
PLMs often fail to distinguish positive from negated facts and are misled by simple distractors, while BERT can learn unseen truth/false classifications after finetuning.
Takeaways & Limitations
The results suggest that PLMs address LAMA-style open-domain question answering through relatively shallow pattern matching rather than learned factual knowledge and inference.
Takeaways & Limitations
Automatically generated misprimes are less natural than manually generated ones, although automation enables a large dataset for this initial study.
Abstract
from arXiv · showhide
Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated ("Birds cannot [MASK]") and non-negated ("Birds can [MASK]") cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add "misprimes" to cloze questions ("Talk? Birds can [MASK]"). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.
1 Introduction
The paper probes whether pretrained language models handle factual knowledge like humans when cloze questions contain negation or distracting misprimes. It finds brittleness in both settings, while noting that automatically generated misprimes are less natural than manually written ones.
- Motivation: The study extends LAMA by testing factual recall under negation and mispriming, where human-like interpretation should resist misleading sentence context.LAMA reformulates question answering as masked-token cloze questions.
- Negation: Negated LAMA inserts negation elements into cloze templates, producing positive/negative pairs for comparing model predictions.An example contrasts “The theory of relativity was developed by [MASK]” with its negated counterpart.
- Negation: PLMs produce highly overlapping fillers for positive and negative probes, generating facts such as “Birds can fly” and their incorrect negation; BERT performs best but still fails on most negated probes.BERT can memorize positive and negative facts when they occur in training but generalizes poorly to unseen sentences.
- Mispriming: Mispriming adds an automatically generated, misleading word-question pair before a cloze prompt, such as “Talk? Birds can [MASK].”The misprime is designed to mislead PLMs while humans would ignore it and answer “fly.”
- Mispriming: PLMs are distracted by misprimes and fill “talk” rather than the factual answer “fly,” unlike a human respondent.The paper presents this as evidence of brittle factual behavior under distracting context.
- Study scope: Automatically generated misprimes are less natural than manually generated alternatives, but they enable a large dataset for this initial study.The paper illustrates the trade-off with a manually constructed “regent of Antioch” example.
2 Data and Models
The experiments build on LAMA’s cloze formulations from knowledge-base triples and question-answer pairs, then construct negated and misprimed variants for evaluating several pretrained models.
- LAMA data: LAMA generates cloze questions from subject-relation-object triples and question-answer pairs by masking the object or answer.Knowledge-base templates substitute a subject for X and MASK for Y.
- LAMA data: The data sources include Google-RE, T-REx, ConceptNet, and SQuAD, covering factual, commonsense, and question-answering relations.The listed sources provide relation triples or context-insensitive questions reworded as cloze prompts.
- Models: The evaluation uses Transformer-XL large, ELMo original, ELMo 5.5B, BERT-base, and BERT-large with source code from earlier LAMA work.The models are evaluated as pretrained language models.
- Dataset variants: Negated LAMA manually inserts a negation element into each template or question and restricts ConceptNet to an easy-to-negate subset.This creates the positive/negative probe setting used to test negation.
- Dataset variants: Misprimed LAMA inserts an incorrect word and question mark at the beginning of a statement, retaining only questions answered correctly by BERT-large.The construction removes correct primes using manual and automatic filtering rules across datasets.
3 Results
Across negation and mispriming probes, PLMs often preserve predictions despite changed sentence meaning or distracting context. BERT can learn negation with explicit supervision, but pretrained models remain brittle.
- Negation: ρ > 85% in most cases, with high overlap in rank-1 predictions between original and negated queries.BERT performs slightly better, while Google-RE date-of-birth queries are an outlier because the negated pattern is rare in corpora.
- Negation: High prediction overlap between original and negated cloze questions indicates that PLMs poorly distinguish positive from negative sentences.The evaluation compares Spearman rank correlation and overlap in rank-1 predictions.
- Negation: BERT memorizes seen positive and negative sentences but generalizes poorly to unseen examples in both forms.The balanced synthetic corpus tests whether patterns and within-group regularities support distinguishing facts from non-facts.
- Negation: 100% test accuracy follows finetuning BERT to classify sentences as true or false, showing that supervision enables negation learning.Without supervision, pretrained BERT fails to distinguish true from false sentences reliably.
- Mispriming: Mispriming BERT-large with highly ranked incorrect objects usually causes a precision drop exceeding 60%, and the effect persists across 20 inserted sentences.Random-word misprimes are less disruptive, while difficult misprimes matching selectional restrictions still affect the model.
4 Discussion
The experiments indicate that pretrained language models largely ignore negation and are highly vulnerable to misleading context, suggesting reliance on shallow similarity rather than robust factual understanding.
- PLMs largely do not learn the meaning of negation, instead predicting fillers from subject–filler co-occurrence.This pattern makes them prone to generating both facts and their incorrect negations.
- The LAMA setup prevents models from expressing uncertainty, and no prediction-probability threshold reliably separates valid positive from invalid negative answers.
- BERT can memorize negative facts present during training but generalizes poorly to unseen positive and negative sentences without supervision.After finetuning, it reaches 100% truth-value classification accuracy, showing that supervised training can teach negation.
- BERT often handles random misprimes correctly but is highly sensitive to misleading context that would not alter human question-answering behavior.A single distracting word can be sufficient to change its prediction.
- Overall, the work presents negation and mispriming as tests exposing differences between PLMs and human-level language understanding.
5 Related Work
Prior work has examined linguistic, commonsense, grammatical, and adversarial robustness in language models, while this paper focuses on factual knowledge under negation and mispriming.
- Research on commonsense knowledge did not explicitly examine negation or priming, leaving those factual-knowledge interactions less directly addressed.
- Earlier studies analyzed linguistic knowledge stored in pretrained embeddings across syntax, agreement, heuristics, and related phenomena.
- Work on natural language inference found that finetuned BERT relies heavily on syntactic heuristics, also raising concerns about commonsense acquisition.
- Studies of negative polarity items and finetuned negation address grammaticality or supervised understanding, whereas this paper examines factuality learned during pretraining.
- Adversarial and minimally modified-sentence studies differ by targeting task-specific examples or preserved semantics, while this work tests factual robustness under negation and distracting context.
6 Conclusion
The paper concludes that PLMs often answer open-domain factual questions through shallow pattern matching rather than learned factual recall and inference. It argues that future systems need better handling of discrete truth conditions and confidence assessment.
- PLMs appear to address open-domain QA through relatively shallow pattern matching rather than recalling learned factual knowledge and performing inference.
- Factual knowledge and logic are discrete phenomena requiring models to distinguish sharply different truth values despite similar representations.
- Better confidence assessment is needed because PLMs struggle to distinguish evidence-based best guesses from guesses made without training-corpus evidence.
- The authors frame human-easy negation and resistance to mispriming as desirable targets for future pretrained language models.
7 Appendix
The appendix describes dataset preparation, synthetic pretraining, and finetuning procedures for evaluating negation and generalization in BERT-base.
- Data preparation: The evaluation uses source code from Petroni et al. and reformulates Google-RE location templates to request cities for single-answer queries.T-REx continues to query for places of death without changing its template.
- Data preparation: ConceptNet is reduced to 2,996 of 11,458 samples, selecting short sentences or sentences matching predefined relation patterns.The selected sentences are automatically negated using a manually created verb-negation dictionary.
- Mispriming setup: The mispriming-distance experiment inserts up to 20 neutral sentences, reaching 89 byte pair encodings, without substantially reducing the prime’s effect.The neutral fillers include brief conversational statements such as “This is great.” and “That is okay.”
- Synthetic pretraining: BERT-base is pretrained from scratch on equally many positive and negative sentences, with duplicated training data and a 349-token vocabulary.Figure 1 reports training loss and test accuracy for this balanced corpus.
- Results and finetuning: Test accuracy stagnates around 0.5 despite training, indicating no better-than-random generalization to unseen positive and negative sentences.The appendix contrasts training behavior with test performance and then finetunes BERT to classify sentences as true or false.
- Results and finetuning: The appendix identifies the finetuning setup and its implementation as separate from pretraining, using listed hyperparameters and Wolf et al.’s source code.The cited implementation source is the Hugging Face Transformers codebase.