Source-linked AI summary
Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
Mariya Toneva, Leila Wehbe
TL;DR
NLP networks achieve strong performance without explicit language rules, but their learned representations remain difficult to interpret. The paper aligns four models’ representations with brain recordings during naturalistic reading and finds model-, context-, and layer-dependent patterns, while brain-guided BERT alterations improve syntactic-task performance.
Problem
Neural networks’ strong NLP performance without explicit language rules raises the question of what their learned representations encode, especially for complex language.
Method
The paper aligns intermediate representations from ELMo, USE, BERT, and Transformer-XL with fMRI and MEG recordings of subjects reading naturalistic text.
Results
The models show different context-related representations, with transformer middle layers most brain-relevant; altering shallow BERT attention improves brain prediction and most syntactic-task results.
Takeaways & Limitations
Brain recordings can provide additional information for interpreting NLP representations and can guide alterations associated with better syntactic-task performance.
Takeaways & Limitations
Brain-based interpretations require caution because reverse inference can be erroneous when regions support multiple processes.
Abstract
from arXiv · showhide
Neural networks models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. This has generated a lot of research interest in interpreting the representations learned by these networks. We propose here a novel interpretation approach that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transformer-XL. We study how their representations differ across layer depth, context length, and attention type. Our results reveal differences in the context-related representations across these models. Further, in the transformer models, we find an interaction between layer depth and context length, and between layer depth and attention type. We finally hypothesize that altering BERT to better align with brain recordings would enable it to also better understand language. Probing the altered BERT using syntactic NLP tasks reveals that the model with increased brain-alignment outperforms the original model. Cognitive neuroscientists have already begun using NLP networks to study the brain, and this work closes the loop to allow the interaction between NLP and cognitive neuroscience to be a true cross-pollination.
1 Introduction
Neural networks achieve strong NLP performance without explicit language rules, motivating new ways to interpret their learned representations. This paper aligns NLP representations with brain recordings to analyze four models and investigate how context, layers, and attention shape their information.
- Motivation: Neural networks often outperform rule-informed approaches despite lacking explicit language rules, raising questions about what their representations encode.Generic pretrained representations can also improve downstream performance.
- Proposed approach: The proposed approach aligns network representations with brain activity during naturalistic reading to interpret information that standard NLP probes may miss.The method uses mappings from intermediate network representations to fMRI or MEG recordings for corresponding words.
- Interpretation framework: Brain-region predictions are interpreted using prior knowledge about regional functions, with classification and significance testing identifying predictive alignments.A layer is considered to share information with a brain region when it accurately predicts that region’s activity.
- Models and questions: The analysis examines ELMo, BERT, USE, and Transformer-XL across layer depth, context length, and attention-related questions.The study asks whether word-level information remains confined to input layers and how representation properties differ across models.
- Caveat: The authors caution against reverse inference and restrict interpretations to well-studied regional functions or experimentally demonstrated overlap with specific processes.They also cross-validate models and report performance on unseen test sets.
- Findings: The study reports stronger brain-activity prediction in transformer middle layers, context-dependent differences across models, and improved prediction from uniform attention in shallow BERT layers.Transformer-XL is reported not to degrade as context increases, unlike the other models, while altered BERT improves syntactic-task performance.
2 Related work on brains and language
Prior work has used controlled and naturalistic brain experiments, along with neural-network representations, to study language processing. However, comparatively little work has used brain recordings to evaluate or improve NLP models.
- Brain and language studies: Brain-language research has commonly contrasted controlled conditions differing in complexity, linguistic properties, or incongruities, while some studies use naturalistic stories.Naturalistic stimuli include stories and other extended linguistic materials.
- Neural representations of brain activity: Earlier studies aligned RNN or LSTM representations with MEG or fMRI recordings to model brain activity and trace sentence comprehension.These studies used neural-network feature spaces to relate word-by-word processing or representational content to brain data.
- Brain-guided NLP: Only limited prior work evaluates or improves NLP models through brain recordings, including cognition-relevant embedding evaluation and brain-constrained word embeddings.Brain-aligned embeddings were reported to improve alignment with behavioral semantic measures.
3 Approach
The approach interprets representations from four NLP models by aligning them with brain recordings collected while subjects read naturalistic text. It varies model, layer, context, and brain-imaging data to evaluate how well representations predict brain activity.
- Network-derived representations: The study examines ELMo, BERT, USE, and Transformer-XL as case studies for interpreting network-derived representations.ELMo and BERT are bidirectional models; USE encodes sentences, and Transformer-XL adds segment-level recurrence for longer context.
- Network-derived representations: Representations are computed from intermediate layers using the most recent k words presented to the participant.The context length k varies across representations, enabling comparisons of layer depth and contextual information.
- Brain recordings: Brain activity is measured with fMRI and MEG, which provide complementary spatial and temporal resolution.fMRI offers high spatial but low temporal resolution, whereas MEG offers lower spatial but high temporal resolution.
- Encoding models: Linear encoding models predict brain recordings from each network representation using the same words that generated that representation.The encoding function maps representation x_l,k to brain activity y.
- Evaluation: Prediction accuracy is evaluated through a held-out two-way classification task under four-fold cross-validation.The classifier predicts which of two word sets was being read from the corresponding predicted brain features.
- Evaluation: MEG is used to study word-embedding components because its temporal resolution exceeds the rate of word presentation.The proof-of-concept analysis examines whether word properties and ELMo embeddings predict brain recordings.
4 Interpreting long-range contextual representations
The models differ in how their representations retain recent and long-range context, with transformer behavior depending on layer depth, context length, and attention pattern. Transformer-XL benefits most consistently from longer context, while BERT’s early-layer attention changes improve brain-activity prediction.
- Contextual representations: ELMo, BERT, and Transformer-XL long-context representations predict both group 1 and group 2 regions, while USE predicts a smaller subset of group 2 regions.The authors attribute the difference partly to USE’s deep averaging and sentence-level representation, which may discard recent information.
- Layer depth and context length: Middle layers perform best beyond 15 words, while deepest layers peak below 10 words and then decline.This layer-depth pattern appears across ELMo, BERT, and Transformer-XL.
- Layer depth and context length: Transformer-XL is the only model whose prediction performance continues increasing as context length grows.The result is consistent with Transformer-XL’s design goal of capturing longer context.
- Layer depth and context length: Adjusting BERT layers relative to layer 1 makes their context-layer relationship resemble Transformer-XL’s pattern.The authors suggest BERT layer 1 combines token-level information in a way that limits longer-context retention.
- Attention: Uniform attention harms deep BERT layers but benefits shallow layers for context lengths up to 25 words.The manipulation replaces learned attention at one layer while keeping other pretrained parameters fixed.
5 Applying insight from brain interpretations to NLP tasks
The authors test whether brain-informed alterations to BERT improve language-task performance without additional fine-tuning. Altered shallow-layer models outperform the pretrained baseline on most evaluated syntactic tasks.
- BERT is evaluated on 13 tasks without fine-tuning beyond pretraining, preserving transfer from brain-based interpretations.The evaluation targets syntactic NLP tasks after pretraining.
- The probing protocol masks a focus verb in a complete sentence and compares scores for the correct and incorrectly numbered verbs.
- 8 of 13 tasks show significant improvement over base, while 4 of the remaining 5 achieve parity.The comparison uses paired t-tests at significance level 0.01 with FDR control.
- The altered models therefore improve or match pretrained BERT on 12 of the 13 evaluated tasks.
6 Discussion
The discussion presents brain recordings as a way to interpret language representations across models and context conditions. It reports improved brain prediction from uniform attention in shallow BERT layers and better performance on most syntactic tasks after alteration.
- Brain recordings of naturalistic reading are used to interpret representations from ELMo, USE, BERT, and Transformer-XL across context lengths.MEG also shows that ELMo’s non-contextualized word embeddings contain information about word length and part of speech.
- Uniform attention on the previous layer improves brain-prediction performance in BERT’s shallow layers 1-6 compared with learned attention.
- The altered BERT performs better on the majority of syntactic NLP tasks, supporting the paper’s proposed link between brain alignment and language performance.The authors state this result as a suggestion rather than a demonstrated causal mechanism.
- Future work will expand MEG analyses and derive more informative fMRI brain priors for studying high-level semantic information.
8 Brain areas included in prior
A supplementary table identifies the regions of interest shown in Figure 1. The regions are approximations based on prior results.
- Table 2 lists the names of regions of interest included in Figure 1.
- The regions of interest were approximated from results reported by Lerner et al. (2011).
- The table provides anatomical labels for the regions used in the figure rather than introducing a new brain-area measurement.
9 Data Preprocessing
The study preprocesses fMRI and MEG recordings collected while subjects read naturalistic text. The modalities differ in sampling setup and preprocessing procedures.
- fMRI: fMRI data come from 8 subjects reading chapter 9 of Harry Potter, with words presented one at a time every 0.5 seconds.Images were acquired every 2 seconds using 3 × 3 × 3 mm voxels.
- fMRI: The fMRI images were slice-time and motion corrected, detrended, and smoothed with a 3 mm full-width measure.
- MEG: MEG data were recorded for 3 subjects using 306 sensors organized in 102 locations and sampled at 1 kHz.The data were preprocessed using signal-space separation and its temporal extension.
10 Encoding Models
The paper aligns intermediate NLP representations with fMRI and MEG recordings by learning models that predict brain activity, then tests predictive regions or sensors with classification and significance procedures.
- Encoding models: Ridge regression estimates representation-to-brain mappings, with regularization selected independently through nested cross-validation for each output dimension.For fMRI, parameters are estimated per voxel; for MEG, they are estimated per sensor/time tuple.
- Encoding models: Layer representations are aligned with brain activity by fitting predictive models to fMRI or MEG signals for each brain output dimension.fMRI models incorporate delayed features to account for the hemodynamic response, whereas MEG models use word-level representations without those delays.
- Spatial analysis: A cortical-searchlight groups neighboring grey-matter voxels on the cortical sheet rather than relying only on a 3D voxel grid.Results from different subjects are transformed into MNI space before averaging.
- Predictive evaluation: Predicted signals are evaluated using classification of contiguous fMRI chunks or pooled MEG sensor locations to assess whether learned representations predict held-out brain activity.The fMRI procedure classifies 20TR chunks, while MEG pools the three sensors at each location across subjects to improve signal-to-noise ratio.
- Statistical significance: Voxel-level fMRI significance uses a chance-accuracy distribution assumed symmetric around 0.5, rejecting voxels at a threshold chosen to target FDP ≤0.05.The threshold is increased from δ = 0.001 in increments of 0.001, and voxels with accuracy ≥0.5 + δfinal are selected.
- Shared information: Shared explained variance is probed by comparing classification based on a word embedding, another feature, and their concatenation.The proof-of-concept analysis uses features such as one-hot POS representations to identify information common to both vectors.
11 MEG results as proof of concept
MEG provides a proof of concept that word embeddings predict time-resolved brain activity during single-word reading. The predicted activity shifts from visual regions toward higher-level reading areas and overlaps with word length and POS information at different latencies.
- Proof of concept: MEG is used because it can measure brain activity during presentation of individual words, enabling a proof-of-concept test of non-contextualized word embeddings.The analysis targets information expected from word embeddings, including word length and part of speech.
- Temporal progression: Around 100ms, the current word embedding predicts activity in the back of the sensor helmet, overlapping with word-length predictions in visual cortex.This timing corresponds to the expected arrival of visual signals in visual cortex.
- Interpretation: The results support the hypothesis that the word embedding contains both word-length and POS information.The conclusion is based on the distinct overlaps observed across processing times and sensor locations.
12 Complete Attention Results
Table 3 reports syntactic-task performance for BERT models with uniformly altered attention in layers 1-6, using pretrained BERT as the base reference.
- Table 3: Table 3 compares uniformly altered attention in BERT layers 1-6 across a range of syntactic tasks.‘Base’ denotes pretrained BERT.