Source-linked AI summary
Sorting through the noise: Testing robustness of information processing in pre-trained language models
Lalchand Pandia, Allyson Ettinger
TL;DR
Pre-trained language models perform impressively, but it remains unclear whether they robustly process, retain, and apply information presented in context. This paper uses cloze tasks with controlled distractors to test that ability and finds that irrelevant content, semantic similarity, and word position strongly affect predictions, consistent with reliance on superficial contextual cues.
Problem
The paper asks whether pre-trained LMs robustly process, retain, and apply new contextual information rather than relying on superficial mechanisms.
Method
The authors evaluate cloze predictions from multiple pre-trained LMs after inserting irrelevant distractors and systematically varying distractor type and position.
Results
Irrelevant distractors disrupt predictions, with especially strong effects from semantic similarity and relative word position.
Takeaways & Limitations
The findings suggest that LM predictions rely more on coarse-grained superficial contextual cues than on robust representations of relevant facts from context.
Takeaways & Limitations
The study leaves open whether comparatively strong performance of RoBERTaLARGE reflects more robust representations or superficial cues not targeted by these experiments.
Abstract
from arXiv · showhide
Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. In this paper we tackle a component of this question by examining robustness of models' ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systematically manipulate the nature of these distractors, to shed light on dynamics of models' use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to factors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by robust representations of context meaning.
1 Introduction
The paper tests whether pre-trained language models robustly process, retain, and apply new facts from context, rather than relying on superficial cues. Cloze experiments with irrelevant distractors show that predictions are vulnerable to semantic similarity and word position.
- The study asks whether pre-trained LMs robustly represent and apply new contextual facts or instead rely on superficial mechanisms.It focuses on processing, retaining, and applying information presented in the input.
- The authors use cloze tasks containing critical context information and add irrelevant distracting content to test whether models maintain correct predictions.They also vary distractor characteristics to investigate the mechanisms influencing predictions.
- Distracting content undermines model predictions, while distractor manipulations indicate sensitivity to semantic similarity and relative word position.These results support the conclusion that predictions rely substantially on superficial contextual cues rather than robust representations of relevant facts.
2 Related Work
Prior work has examined language models as knowledge bases, tested linguistic knowledge, and studied contextual attractors and reading comprehension. This paper differs by testing whether models form robust representations of new facts presented after training.
- Earlier cloze probes tested whether LMs recall world knowledge from training, whereas this work tests representations of new information presented in the input.The distinction is between retrieving stored knowledge and processing newly supplied facts.
- Research on linguistic knowledge has studied syntactic, semantic, pragmatic, and contextualized-embedding behavior in pre-trained LMs.Examples include agreement tests, broader syntactic phenomena, semantic and pragmatic phenomena, and information in contextualized embeddings.
- The study adapts attractors from syntactic testing but defines them through semantic relationships to investigate robustness in processing new contextual facts.It builds on work examining distance, contextual constraint, and semantic priming with a more systematic focus on attractor impacts.
- The approach also relates to reading-comprehension and adversarial-testing tasks that insert additional material to make prediction or comprehension more difficult.These connections include bAbI, CBT, and adversarial testing studies.
3 Methods
The method uses synthetic cloze tasks in which models infer a target from a critical background fact, then face controlled attractors that test robustness. Attractors vary by semantic relation, entity structure, quantity, and intervening content.
- 3 Methods: Each base item gives an imaginary entity a background fact and asks the model to complete a related statement, such as predicting Paris from Sebastian living in France.Models must prefer the correct target over closely related alternatives, establishing baseline competence before distraction tests.
- 3 Methods: Attractors are irrelevant inserted content used to test whether models retain the critical fact when making predictions.A robust representation should preserve preference for the correct target despite attractors.
- 3.1 Attractor manipulations: B-type attractors share a semantic class with the critical background word, whereas T-type attractors share a semantic class with the target; unrelated attractors provide a contrast.These manipulations test whether coarse-grained semantic similarity influences predictions.
- 3.1 Attractor manipulations: The study uses single-entity attractors attached to the key entity and multiple-entity attractors linked to different entities.This distinguishes sorting different facts about one entity from sorting different entity–property links.
- 3.2 Dataset construction: The synthetic dataset varies attractor counts from zero to three across four paired domains, with additional unrelated material inserted between the key entity and critical fact.The domains are countries–capitals, professions–objects, monuments–countries, and sports–scoring metrics.
- 3.2 Dataset construction: Successful prediction is defined relatively: the correct target must receive higher probability than every other target in its item set in the base context.This criterion separates baseline knowledge of the relation from robustness under distraction.
4 Experiments
The experiments evaluate multiple pre-trained LM families and size settings using controlled inputs and model-specific special tokens. The tested models include BERT, RoBERTa, and GPT-2 variants.
- The evaluation covers three classes of pre-trained LMs with multiple size settings within each class.The models are analyzed using the implementation of Wolf et al. (2020).
- The tested BERT models are BERTBASE (110M parameters) and BERTLARGE (340M parameters), both uncased.
- The tested RoBERTa models are RoBERTaBASE (125M parameters) and RoBERTaLARGE (355M parameters).
- The tested GPT-2 models range from GPT2SMALL (117M parameters) to GPT2XL (1558M parameters).The intermediate variants are GPT2MEDIUM (345M parameters) and GPT2LARGE (774M parameters).
- Inputs use model-specific start, separator, and mask conventions, while GPT-2 does not require a masked token.
5 Results
Semantically related attractors substantially disrupt models’ preference for correct completions, with the largest effect appearing after the first attractor. Unrelated attractors also reduce target probabilities, but more gradually, indicating sensitivity to semantic similarity and contextual position rather than robust use of relevant facts.
- 5 Results: A single semantically related attractor substantially reduces models’ preference for the correct completion, although RoBERTaLARGE is the most resistant.B-type and T-type attractors show largely similar patterns, and the first attractor produces clear disruption across models.
- 5 Results: Adding further semantically related attractors often causes little additional harm, and some models’ accuracy improves as more attractors are inserted.The improvement is mitigated when attractors form more complex statements in the multiple-entity setting; the authors speculate that models may attend less to list-like content.
- 5.1 Impact on probabilities: One semantically related attractor causes a dramatic drop in target probability relative to the base context, with additional attractors producing comparatively smaller effects.The first-attractor effect is especially uniform in the single-entity setting, while GPT2SMALL shows a less dramatic initial impact in the multiple-entity setting.
- 5.2 Semantically unrelated attractors: Semantically unrelated attractors preserve high accuracy overall but still produce a gradual, non-trivial decline and reduce target probabilities as more attractors are added.Their presence affects confidence in the correct target even though they are irrelevant to the prediction.
- 5.2 Semantically unrelated attractors: Related and unrelated attractors differ chiefly in their first-attractor effect: related items cause a sharp dip, whereas unrelated items exert a smaller but cumulative influence.The contrast suggests that models rely heavily on coarse-grained semantic similarity cues when identifying context words relevant to prediction.
6 Varying position of information
Varying the positions of entities and attractors shows that semantic similarity and relative word position jointly shape model susceptibility to distraction.
- 6.2 Varying key entity position: A single semantically related attractor significantly disrupts accuracy whether it intervenes between the key entity and fact or between the fact and target.These outcomes suggest models do not rely heavily on proximity between the key entity and critical fact, though they may fail to form robust entity-fact links.
- 6.2 Varying key entity position: When the queried entity occurs later, accuracy generally decreases steadily as attractors increase, rather than showing the usual first-attractor dip.This pattern appears except for GPT2SMALL.
- 6.2 Varying key entity position: Semantically related attractors have less influence when they occur before the key entity, indicating a strong role for recency and relative word position.Models may preferentially link entity mentions with following descriptors, though that heuristic does not distinguish relevant from irrelevant later words.
- 6.2 Varying key entity position: RoBERTaLARGE is notably more robust to attractor effects than other models, although it still shows a marked decrease in performance.In the separated-entity setting, RoBERTaLARGE even improves in one multiple-entity pattern while GPT2MEDIUM performs poorly.
7 Discussion
The discussion concludes that irrelevant context changes LM predictions and that semantic similarity and word position act as superficial cues. It also notes unresolved uncertainty about why RoBERTaLARGE is more robust.
- 7 Discussion: Irrelevant distracting content changes model predictions instead of leaving them consistent with the critical contextual information.This directly challenges the expectation that models robustly process and store new facts from context.
- 7 Discussion: A single semantically related attractor has a dramatic effect, whereas unrelated attractors produce more gradual, additive impacts.Relative position further modulates semantic effects: related attractors are much less influential when placed before the key entity.
- 7 Discussion: These patterns suggest heuristics based on semantic similarity to critical words and the relative position of entities and descriptors.Together, they support the conclusion that predictions rely substantially on superficial cues rather than robustly retained context meaning.
- 7 Discussion: RoBERTaLARGE is frequently least susceptible to attractors, but it is still disrupted; larger pretraining datasets and models may contribute to its relative robustness.The paper leaves open whether its advantage reflects more robust representations or superficial cues not tested here.
8 Conclusion
The paper tests whether pretrained LMs retain and use newly presented facts under distraction. The results show that semantic similarity and relative word position strongly influence predictions, which appear driven more by superficial cues than robust context representations.
- 8 Conclusion: Models can make correct predictions in simple settings, but inserting distracting material easily disrupts those predictions.The experiments manipulate inputs to test whether models represent and retain information conveyed by text.
- 8 Conclusion: Systematic manipulation identifies semantic similarity and relative word position as key influences on models’ selection of relevant contextual cues.Overall, the findings favor coarse-grained superficial cues over robust representations of prior context.
A Appendix
The appendix provides dataset examples and visualizations for attractor conditions, including B-type, T-type, and semantically unrelated attractors.
- A Appendix: Table 2 presents the base context used for the dataset.
- A Appendix: Table 3 gives examples of semantically unrelated attractors.
- A Appendix: Table 4 provides examples in which the key entity and critical fact are separated.
- A Appendix: Figures 6–8 show accuracy as the number of attractors increases for B-type, T-type, and semantically unrelated attractors.