Source-linked AI summary
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, Willem Zuidema
TL;DR
The paper asks how neural language models represent and maintain subject-verb number agreement, especially when intervening nouns create difficult dependencies. It uses diagnostic classifiers to track these representations and their corruption, then intervenes in LSTM hidden activations. Diagnostic-classifier-guided intervention raises congruent-verb preference from 78.1% to 85.4%.
Problem
Prior work showed strong long-distance agreement performance but did not clarify which LSTM components store or process syntactic features or how those features are represented.
Method
The paper trains diagnostic classifiers on LSTM internal states to analyze agreement representations across timesteps and uses them to alter hidden activations during difficult sentences.
Results
78.1% to 85.4%: after intervention, the percentage of sentences where the model prefers the congruent over the incongruent verb increases substantially.
Takeaways & Limitations
Diagnostic classifiers reveal dynamic agreement representations, locate where information is corrupted, and support targeted intervention that improves verb-number prediction.
Takeaways & Limitations
The replication matches the public evaluation script but has slightly lower overall scores than the reported results, with no explanation for the discrepancy.
Abstract
from arXiv · showhide
How do neural language models keep track of number agreement between subject and verb? We show that `diagnostic classifiers', trained to predict number from the internal states of a language model, provide a detailed understanding of how, when, and where this information is represented. Moreover, they give us insight into when and where number information is corrupted in cases where the language model ends up making agreement errors. To demonstrate the causal role played by the representations we find, we then use agreement information to influence the course of the LSTM during the processing of difficult sentences. Results from such an intervention reveal a large increase in the language model's accuracy. Together, these results show that diagnostic classifiers give us an unrivalled detailed look into the representation of linguistic information in neural models, and demonstrate that this knowledge can be used to improve their performance.
1 Introduction
LSTM language models capture long-distance relations, but prior work did not clarify which components store syntactic features or how those features are represented. This paper uses diagnostic classifiers to analyze subject-verb agreement representations in detail.
- LSTM-based language models have learned short- and long-distance relations in text, supporting high performance in language technologies.
- Prior agreement results showed strong performance but did not identify which LSTM components store or process syntactic features.
- The paper analyzes English subject-verb agreement with diagnostic classifiers to examine how number information is represented, changes across timesteps, and relates to errors.
- The authors also intervene in the LSTM’s representation to test whether agreement information can improve verb-number prediction.
2 Data
The paper treats subject-verb agreement as a variable-distance dependency complicated by intervening nouns, and uses complementary datasets to study number tracking and attractor effects.
- 2.1 Subject-verb agreement: Subject-verb agreement requires an English present-tense verb to match its subject head in grammatical number across potentially arbitrary distances.
- 2.1 Subject-verb agreement: Intervening nouns with the opposite number are agreement attractors that complicate identifying which subject and verb should agree.
- 2.2 Datasets: The Gulordava dataset contains 410 sentences from 41 originals, with nine nonce variants per original created by same-part-of-speech and same-feature substitutions.
- 2.2 Datasets: The Wikipedia-derived corpus contains approximately 1.5 million annotated sentences, enabling controlled subsets by context size, attractor position, and related conditions.
- 2.2 Datasets: Dataset notation records minimum words before the subject, context size, minimum words after the verb, and attractor position; A− denotes sentences without an attractor.
- 2.2 Datasets: Figure 1 illustrates a context size of 7 with two plural attractors, estimates and economists, between singular subject average and verb put.
3 Replication
The replication evaluates whether an LSTM assigns higher likelihood to congruent than incongruent verbs on Gulordava et al.’s English datasets. Results are broadly similar, with slightly lower accuracy scores.
- The replication processes long-distance subject-verb sentences and compares the likelihoods assigned to the original congruent verb and its incongruent counterpart.
- Table 1 reports the percentages of sentences where the correct verb receives higher likelihood than the incorrect form.
- The replication obtains overall results similar to Gulordava et al. (2018), although its accuracy scores are slightly lower.
- The authors’ implementation exactly matches the publicly shared evaluation script, but they report no explanation for the discrepancy in overall scores.
4 Diagnostic Classification to Predict Number
Diagnostic classifiers probe number information in LSTM components over time, revealing where agreement information is represented and how attractors affect it.
- Diagnostic-classifier setup: Diagnostic classifiers predict the main verb’s number from activation data extracted from 10 LSTM components.The setup uses hidden, memory-cell, and gate activations from both layers of a pretrained two-layer LSTM.
- Results: Layer 1 achieves the highest diagnostic-classification accuracies across almost all LSTM components for both correct and wrong test sets.This suggests the last LSTM layer best captures long-distance dependencies.
- Results: Diagnostic classifiers perform best at or just after the subject and verb timesteps across conditions, showing that the model recognises number information at both locations.The pattern holds across context sizes, attractor positions, and sentence-boundary conditions.
- Results: Layer 1 cell-state and hidden-activation classifiers are the most stable on correctly processed sentences, while all components exceed the 50% random baseline.These components appear most specialised for processing number information.
- Results: Agreement attractors lower diagnostic-classification accuracy, with errors rising and diverging between correct and wrong sentences at the attractor position.Without attractors, this divergence does not occur.
5 Representations Across Timesteps
Temporal generalization reveals that agreement information is encoded differently across processing stages, and that incorrect sentences lose this information after an attractor.
- Temporal generalization: Temporal generalization matrices test whether classifiers trained at one timestep generalize to representations at every other timestep.Each timestep-specific classifier is trained on one timestep and evaluated across all T timesteps.
- Encoding failures: Correctly processed sentences show high diagonal accuracies, but incorrect sentences are lower even during the first two timesteps before the attractor appears.This suggests that relevant information is already encoded differently when the subject occurs.
- Attractor effects: For incorrectly processed sentences, internal-state information falls below chance after the attractor, whereas correctly processed sentences show only a slight accuracy dip.The attractor occurs at timestep 3, followed by below-chance accuracies at timesteps 4 and 5 for incorrect sentences.
- Representations Across Timesteps: Agreement is represented at least two ways: a short-term surface representation near the subject and a longer-term deep representation during subsequent processing.The deep representation is most generic at timestep 4, whose classifier has the highest accuracy across intermediate timesteps.
6 Comparing Representations Across Components
Comparing LSTM components at timestep 4 shows that deep agreement information is concentrated in similar representations in the layer 1 hidden activation and memory cell.
- Component comparison: At timestep 4, deep agreement information is best represented in the layer 1 hidden activation and memory cell.The spatial generalization matrix compares classifiers trained separately on each LSTM component.
- Component comparison: The layer 1 hidden activation and memory-cell representations are similar in how they encode deep agreement information.This similarity is observed in the spatial generalization matrix.
7 Improving the Language Model Using Diagnostic Classifiers
The authors use diagnostic classifiers to intervene in LSTM activations at the subject timestep, correcting corrupted agreement information while minimally affecting overall processing. This intervention improves agreement prediction substantially for both original and nonce sentences.
- Intervention: Diagnostic classifiers are used to actively influence LSTM behavior rather than merely analyse its internal representations.The intervention targets the network’s processing after the subject, where agreement information was found to be corrupted.
- Intervention: The experiment uses sentences with the subject at timestep 0, an attractor at timestep 3, and the main verb at timestep 6.Four diagnostic classifiers predict number from hidden and memory-cell activations across both LSTM layers.
- Intervention: The intervention slightly adapts hidden and memory-cell activations toward the diagnostic classifier’s gold-label prediction before normal processing resumes.The adapted activations are then used to continue processing the remainder of each sentence.
- Effects on representations: Intervention effects persist across later timesteps and spread indirectly to gate values, changing representations beyond the directly modified components.Diagnostic-classifier accuracy can grow stronger as processing proceeds after the intervention.
- Language-model performance: The intervention is minor for overall language-model behavior but strongly improves agreement accuracy.Perplexity comparisons show no strong overall anomalies, while agreement results improve from 78.1% to 85.4% for original sentences and from 70.7% to 75.6% for nonce sentences.
- Implications: These results indicate that diagnostic classifiers capture features used by the LSTM and can support active modification of recurrent neural-network processing.The authors frame this as a move from analysing black-box models toward influencing them.
8 Conclusions
The paper finds that number information in an LSTM is dynamically represented, with a relatively stable deep representation between sentence beginning and end. Agreement errors can originate early, and diagnostic-classifier interventions substantially improve final verb predictions.
- Conclusions: Number information is encoded dynamically over time rather than remaining constant throughout a sentence.The representation differs substantially at the beginning and end of sentences, with a relatively stable deep representation in between.
- Conclusions: When the LSTM prefers an incongruent verb, number information appears to be stored incorrectly from the beginning of the sentence.The corruption occurs before the attractor and well before the verb appears.
- Conclusions: Using diagnostic classifiers to alter hidden activations raises congruent-over-incongruent verb preference from 78.1% to 85.4%.The intervention demonstrates that diagnostic information can be used to change subsequent LSTM processing.