PaperlayerView exact revision on arXiv

Source-linked AI summary

What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models

Allyson Ettinger

arXiv:1907.13528v2cs.CLcs.AI

TL;DR

Language-model pre-training succeeds broadly, but the linguistic capacities it confers remain unclear. The paper introduces psycholinguistic diagnostics to probe contextual predictions and applies them to BERT. BERT shows strengths on hypernyms, category distinctions, and role reversals, but weaknesses on challenging inferences, role-based event prediction, and especially negation.

  • Problem

    The paper asks what linguistic capacities language-model pre-training confers, a question left unresolved despite pre-training’s success across NLP tasks.

  • Method

    The paper introduces psycholinguistic diagnostics that test contextual word predictions across linguistic capacities without task-specific fine-tuning.

  • Results

    BERT handles hypernyms, same-category distinctions, and role reversals, but struggles with challenging inferences and role-based event prediction and shows clear failures with negation.

  • Takeaways & Limitations

    The diagnostics clarify which kinds of linguistic information pre-trained BERT uses for contextual prediction and expose areas where its generalization is weak.

  • Takeaways & Limitations

    The small, non-comprehensive test sets require conservative conclusions because different formulations may produce different performance.

Abstract

from arXiv · show

Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about the information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inferences and role-based event prediction -- and in particular, it shows clear insensitivity to the contextual impacts of negation.

1 Introduction

The paper introduces targeted psycholinguistic diagnostics for probing what language models learn during pre-training and applies them to BERT. BERT shows selective strengths and weaknesses across linguistic capacities, with especially clear failure on negation.

  • The paper addresses the gap in understanding what linguistic knowledge language-model pre-training provides despite its broad task success.
  • It introduces psycholinguistically derived diagnostics that examine language-model predictions in context without task-specific fine-tuning.
  • The diagnostics target commonsense and pragmatic inference, semantic roles and event knowledge, category membership, and negation, extending beyond syntactic evaluations.
  • BERT handles hypernyms, same-category distinctions, and role reversals, but struggles with challenging inferences and role-based event prediction.
  • BERT fails completely to show generalizable understanding of negation, raising questions about language models’ ability to learn this meaning.

2 Motivation for use of psycholinguistic tests on language models

Understanding linguistic knowledge acquired through language-model pre-training matters because word prediction in context increasingly underlies state-of-the-art NLP systems. Psycholinguistic diagnostics provide a controlled way to examine the information models use for prediction in their natural operating setting.

  • The tests examine language models’ general linguistic knowledge through the information they use when assigning probabilities to words in context.
  • The diagnostics are designed for contextual predictions, allowing models to be tested without task-specific fine-tuning.
  • Clarifying pre-training’s acquired linguistic competence is important for understanding the strengths, limitations, and generalization of models built around word prediction in context.

3 Related Work

Prior language-model analyses largely emphasize syntactic competence, while this paper broadens evaluation to semantic, pragmatic, and event-related capacities. It also combines probability sensitivity with direct word-prediction accuracy for a richer assessment.

  • The work complements studies probing sentence or contextual embeddings, overall understanding, and BERT’s attention or syntactic representations.
  • Existing language-model diagnostics are dominated by targeted tests of syntactic awareness and structural dependencies.
  • The paper expands evaluation to commonsense and pragmatic reasoning, semantic roles and event knowledge, category membership, and negation.
  • Unlike evaluations that only compare word probabilities, the paper also measures word-prediction accuracy directly.

4 Leveraging human studies

The diagnostics adapt psycholinguistic experiments that measure human expectations for words in context. They use cloze behavior and N400 responses to target information that may have reduced influence on prediction, while retaining controlled, human-designed test items.

  • Psycholinguistic studies provide carefully designed items for testing specific aspects of language processing and predictive behavior.
  • Cloze probability is the proportion of people choosing a word to complete an incomplete sentence and serves as a human prediction benchmark.
  • N400 amplitude is an EEG-measured brain response whose sensitivity to contextual fit provides another measure of word expectancy.
  • The selected tests come from human studies where N400 responses diverged from cloze predictions, targeting information with reduced effects on N400.

5 Datasets

The paper builds three psycholinguistically motivated diagnostic datasets to test prediction accuracy, sensitivity to targeted distinctions, and qualitative use of context. They cover commonsense and pragmatic inference, semantic roles and event knowledge, category membership, and negation, while remaining small and vocabulary-filtered for BERT.

  • Each diagnostic supports word prediction accuracy, sensitivity testing, and qualitative analysis of top predictions.The tests are constructed to constrain the information relevant to predicting target words.
  • The datasets are small psycholinguistic test sets, and items are filtered so expected target words occur in BERT’s single-word vocabulary.The small size reflects their origin in human experiments, while vocabulary filtering ensures expected words can be predicted.
  • CPRAG-102: The CPRAG-102 set tests commonsense and pragmatic inference alongside sensitivity to distinctions within semantic categories.Its items require inference from a first sentence and interpretation of how a second sentence relates.
  • ROLE-88: ROLE-88 tests event knowledge and semantic-role interpretation by requiring models to distinguish noun orders and role reversals.Its paired contexts use the same noun pair in reversed roles, while good and bad contexts differ in human cloze probability.
  • NEG-136: NEG-136-SIMP contains 72 simple sentences testing category membership and the effect of negation on completion truth.Affirmative contexts assess hypernym information, while sensitivity testing targets whether models adjust to negation.

6 Experiments

The experiments evaluate pre-trained BERT models by converting target positions into masked-token prediction problems and comparing their resulting probabilities. They use BERTBASE and BERTLARGE, with the latter containing more parameters, while providing bidirectional context through the input format.

  • The study tests BERTBASE and BERTLARGE, two uncased models with the same basic architecture but different parameter counts.BERTBASE has 110M parameters, while BERTLARGE has 340M.
  • For each item, the target word is replaced by [MASK], and the model’s predictions at that position are measured.A [CLS] token is added at sentence start to mimic BERT’s training conditions.
  • The input includes a period and [SEP] after [MASK], giving BERT an explicit indication that the target position is followed by sentence-ending material.Although the items place relevant information in the left context, this format allows BERT to use its bidirectionality.
  • BERT’s target-position logits are softmax-transformed into probabilities comparable to human cloze probabilities.

7 Results for CPRAG-102

On CPRAG-102, BERT often identifies or prefers plausible completions, but its performance drops when contextual information is removed and its sensitivity is weaker than humans’ responses.

  • Word prediction accuracy: At k = 1, BERTLARGE identifies the expected completion in just over a third of contexts, while both models reach about half at k = 5.The models converge at k = 5 despite BERTLARGE outperforming BERTBASE at k = 1.
  • Word prediction accuracy: Shuffling the first sentence or truncating the second causes notable accuracy drops, showing that BERT uses word order and more distant context.Some items remain intact after perturbation, especially when distinctive words such as checkmate, touchdown, or stone-washed identify the target.
  • Completion sensitivity: BERT assigns the highest probability to the expected same-category completion in 73.5% of items for BERTBASE and 79.4% for BERTLARGE.Thus, inappropriate semantically related words still outrank the expected completion in a substantial minority of cases.
  • Completion sensitivity: The models’ preference for good over same-category bad completions is often small, indicating weaker sensitivity than human cloze responses.Human average cloze difference is about .74, whereas the BERT analysis uses a generous probability-difference threshold greater than .01.
  • Qualitative prediction analysis: Qualitative failures show BERT producing completions suited to the second sentence while missing relevant commonsense or pragmatic information from the first.The authors therefore identify challenging inference and pragmatic reasoning as persistent weaknesses beyond simpler lexical or n-gram cues.

8 Results for ROLE-88

On ROLE-88, BERT shows limited ability to predict role-based event completions, though it can often distinguish appropriate from role-reversed continuations; BERTLARGE uses subject and object information more fully than BERTBASE.

  • Word prediction accuracy: ROLE-88 word-prediction accuracy is very low at k = 1, while expanding to k = 5 raises accuracy and gives BERTLARGE a healthy margin over BERTBASE.Even BERTLARGE at k = 5 is correct in only half of cases in the highest-constraint analysis.
  • Word prediction accuracy: Removing either subject or object information substantially disrupts BERTLARGE but has relatively little effect on BERTBASE.The pattern suggests BERTBASE relies mainly on one participating noun, whereas BERTLARGE uses both nouns and their subject-object structure.
  • Role-reversal sensitivity: BERT prefers the appropriate completion over a role reversal in 75% of items for BERTBASE and 86.4% for BERTLARGE.These proportions are comparable to the models’ same-category sensitivity results on CPRAG-102.
  • Role-reversal sensitivity: The models’ role-reversal sensitivity remains weaker than humans’, and they fail to match human word predictions on a solid majority of cases.The authors interpret this pattern as evidence that prediction failures reflect weak event knowledge or semantic-role understanding rather than inability to distinguish word order.
  • Qualitative prediction analysis: Qualitative examples show BERTLARGE excluding implausible continuations in some role contexts, yet both models can retain the same unlikely top continuation after roles are reversed.For served, probability is higher in the appropriate order by .6 for BERTLARGE and .37 for BERTBASE, but neither selects a more appropriate top continuation in the reversed context.

9 Results for NEG-136

NEG-136 shows that BERT robustly predicts category hypernyms in affirmative contexts but fails to incorporate negation when evaluating category membership. More natural sentences partly reduce this failure, though predictions can still contradict one another.

  • 9.1 Word prediction accuracies: 100% of affirmative items had the correct category predicted at k = 5, showing strong noun–hypernym associations.At k = 1, errors exclusively involved repeating the subject noun rather than producing an informative category completion.
  • 9.2 Completion sensitivity: 100% of affirmative items received higher probability for the true completion, indicating robust preference for true statements.At the .01 threshold, all but one BERTBASE comparison passed.
  • 9.2 Completion sensitivity: 0% of negative items received higher probability for the true completion; BERT preferred the false category completion every time.The models continued predicting category matches despite their falsity under negation.
  • 9.3 Qualitative examination of predictions: BERT’s top completions generally remained unchanged after negation, producing many blatantly untrue sentences.The qualitative examples illustrate that affirmative predictions were often retained when “not” was added.
  • 9.3 Qualitative examination of predictions: BERT used the preceding determiner effectively to restrict predictions to vowel- or consonant-initial words, with only a minority of exceptions.This sensitivity appeared even in negative contexts.
  • 9.4 Increasing naturalness: Even natural items sometimes yielded contradictory top candidates, and negation occasionally failed to reverse the leading completion.One example treated a fast-food dinner as both a romantic first date and not a romantic first date.

10 Discussion

Across three diagnostics, BERT shows selective linguistic knowledge: it can use context and distinguish some semantic contrasts, but struggles with commonsense inference and role-based event prediction. Negation exposes the sharpest divergence between prediction behavior and truth-sensitive interpretation.

  • 10 Discussion: On CPRAG-102, both BERT models predicted the best completion approximately half the time at k = 5.Perturbation results indicate reliance on word order and full-sentence context, while challenging items reveal weaknesses in commonsense and pragmatic inference.
  • 10 Discussion: BERTLARGE matched top human predictions with only 37.5% accuracy on ROLE-88.The low accuracy coexists with an ability to use noun position to prefer good completions over role reversals.
  • 10 Discussion: BERT’s role-reversal distinctions suggest that ROLE-88 errors reflect weak event knowledge or semantic-role implications rather than word-order insensitivity.Both models typically used noun position, although their probability differences were smaller than on CPRAG-102.
  • 10 Discussion: NEG-136 found complete failure to prefer true over false completions for negative category-membership sentences, despite strong noun–hypernym associations.On more natural sentences, BERT reliably preferred true completions with or without negation, likely reflecting higher training-data frequency rather than naturalness itself.
  • 10 Discussion: Because language models optimize prediction using reliable cues, negation may not be learned as a semantic operation when it does not support clear predictions.The discussion contrasts predictive training with human language processing aimed at meaning and truth judgments.

11 Conclusion

The paper introduces psycholinguistically derived diagnostics for probing linguistic competencies acquired through language-model pre-training and applies them to BERT. The results reveal strengths in hypernyms and some semantic distinctions, alongside weaknesses in inference, event knowledge, and negation.

  • 11 Conclusion: The diagnostic suite tests word-prediction accuracy and probability sensitivity to linguistic distinctions across multiple capacities, then uses it to analyze BERT.The tests target commonsense/pragmatic inference, semantic roles and event knowledge, category membership, and negation.
  • 11 Conclusion: BERT distinguishes role reversals and same-category contrasts with lower sensitivity than humans, succeeds with noun hypernyms, and struggles with challenging inferences, role-based event prediction, and negation.The conclusion presents this as the paper’s overall case-study pattern.
  • 11 Conclusion: The test sets are not comprehensive, and their small size requires conservative conclusions because different formulations may produce different performance.Future work can expand the tested capacities and verify the generality of these results.
Loading 1907.13528v2…