Toward a realistic model of speech processing in the brain with self-supervised learning

Juliette Millet; Charlotte Caucheteux; Pierre Orhan; Yves Boubenec; Alexandre Gramfort; Ewan Dunbar; Christophe Pallier; Jean-Remi King

Source-linked AI summary

Toward a realistic model of speech processing in the brain with self-supervised learning

Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King

arXiv:2206.01685v2Jun 3, 2022q-bio.NCcs.AIcs.CL

TL;DR

Existing brain-like deep-learning models rely on implausible data, supervision, input, or memory requirements, motivating biologically plausible alternatives. This study tests self-supervised Wav2Vec 2.0 trained on raw speech against fMRI responses and behavior, finding brain-like representations, hierarchical alignment, and language specialization.

Problem
Existing models can resemble brain responses but often require extraordinarily large datasets, supervised labels, textual input, or implausibly large memory.
Method
The study trains Wav2Vec 2.0 self-supervised on 600 hours of speech and compares its activations with fMRI responses from 412 English, French, and Mandarin listeners.
Results
Self-supervised Wav2Vec 2.0 learns brain-like speech representations, aligns with the cortical processing hierarchy, and captures auditory-, speech-, and language-specific organization confirmed by behavioral comparisons.
Takeaways & Limitations
Self-supervised learning on limited raw speech can account for a rich organization of speech processing in the human brain.
Takeaways & Limitations
Wav2Vec 2.0 remains unlike the brain because its transformer layers access all elements within a contextual window and it shows several human-behavior mismatches.

Abstract

from arXiv · show

Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input. These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts of data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input, and / or (4) implausibly large memory (e.g. thousands of contextual words). These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses. Focusing on the issue of speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate. Specifically, we compare a recent self-supervised architecture, Wav2Vec 2.0, to the brain activity of 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to ~1h of audio books. Our results are four-fold. First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech -- a quantity comparable to what infants can be exposed to during language acquisition. Second, its functional hierarchy aligns with the cortical hierarchy of speech processing. Third, different training regimes reveal a functional specialization akin to the cortex: Wav2Vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices. Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants. These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.

1 Introduction

The paper addresses whether brain-like speech representations can emerge from learning systems without the implausible data, labels, input formats, and memory demands of prior models. It tests self-supervised wav2vec 2.0 trained on raw speech against fMRI responses and behavioral language specialization.

1 Introduction: Prior brain-like models rely on extraordinarily large datasets, supervised labels, textual inputs, and/or implausibly large contextual memory.These differences motivate identifying algorithms that can account for behavioral and brain responses under more human-relevant constraints.
1 Introduction: Wav2vec 2.0 is trained self-supervised on raw speech waveforms using 600 h of effective speech, roughly comparable to infant speech exposure during early language acquisition.The architecture combines convolutional and transformer layers to predict quantized latent speech representations.
1 Introduction: The study compares model activations with fMRI responses from 412 English-, French-, and Mandarin-speaking volunteers listening to approximately one hour of native-language audiobooks.Standard encoding analyses test how well model activations predict brain activity.
1 Introduction: The experiments compare model layers and variants trained on random weights, non-speech sounds, non-native speech, and native speech.These comparisons probe hierarchy and acoustic-, speech-, and language-specific representations.
1 Introduction: The paper reports brain-like speech representations, a cortical-aligned functional hierarchy, convergent acoustic-, speech-, and language-specific representations, and behavioral confirmation from 386 additional participants.Together, these contributions support self-supervised learning as a candidate account of speech-processing organization.

2 Methods

The study trains wav2vec 2.0 variants on raw speech or non-speech audio and evaluates their representations against multilingual fMRI recordings and speech-discrimination behavior. The analysis combines self-supervised and supervised objectives with layer-wise and region-wise brain comparisons.

2 Methods: Wav2vec 2.0 transforms a 16 kHz raw waveform into latent features, quantized sound representations, and contextualized embeddings produced by transformer blocks.The feature encoder uses temporal convolutions, followed by quantization and a 12-block context network.
2 Methods: Self-supervised training predicts masked quantized representations from contextual embeddings and regularizes the diversity of those quantized representations.The supervised alternative discards quantization and trains a phoneme-prediction layer with CTC loss.
2 Methods: Models are trained on approximately 600 h of French, English, Mandarin, or non-speech acoustic-scene data, with untrained and supervised variants included for comparison.The non-speech dataset excludes human-produced sounds, while speech models use language-specific corpora.
2 Methods: The study compares model representations with brain activity across auditory, temporal, and inferior frontal regions using encoding analyses and layer-wise brain scores.Figure 4 additionally evaluates human and model phoneme discrimination and compares acoustic-, speech-, and language-specificity patterns.

3 Results

Self-supervised wav2vec 2.0 representations predict distributed brain responses, organize along a hierarchy aligned with cortical speech processing, and become increasingly speech- and language-specific with relevant training data. Supervised learning is slightly less brain-like under the matched training condition.

3 Results: Self-supervised wav2vec 2.0 activations significantly predict brain activity across nearly all cortical areas in 412 participants.The strongest brain scores occur in primary and secondary auditory, superior temporal, and inferior frontal regions, and exceed those of a randomly initialized model across language groups.
3 Results: ∆R = 0.002: self-supervised learning produces modestly but significantly higher brain scores than supervised learning on the same 600 h of speech.The comparison uses matched wav2vec 2.0 architectures and speech data with different learning objectives.
3 Results: Early transformer layers best predict low-level auditory cortices, whereas deeper layers best predict higher-level regions including the inferior frontal gyrus and superior temporal sulcus.Convolutional layers are less predictive than transformer layers, and the hierarchy extends into supplementary motor and motor areas.
3 Results: 600 h of self-supervised speech training suffices for wav2vec 2.0 to learn language-specific representations.The study evaluates this specialization with language-specific phoneme discrimination and brain comparisons.
3 Results: ∆R = 0.006: non-speech training improves over random initialization; non-native speech adds ∆R = 0.002, and native speech adds another ∆R = 0.002 over non-native training.These ordered comparisons distinguish general auditory, speech-specific, and native-language-specific representation effects.

4 Discussion

The study argues that self-supervised wav2vec 2.0 can reproduce several organizational features of speech processing in the human brain while retaining important gaps and scope limits. Its discussion links brain-like hierarchy and specialization to a potentially biologically plausible learning principle.

A few hundred hours of speech can support infant language acquisition, but the learning principle enabling this remains unknown.
Self-supervised wav2vec 2.0 representations map onto distributed cortical regions, align with cortical hierarchy, and contain language-specific features.
The model’s functional hierarchy follows anatomical organization and extends from temporal regions through prefrontal and motor areas involved in speech processing.
Native-model representations are concentrated in superior temporal and middle temporal regions, but language-specific effects remain modest.Random and non-speech models reach 67% and 87% of native-model brain scores in STS and STG, respectively.
Interpretation remains limited because the shared representations have not yet been systematically characterized.
The study’s scope is constrained to adult speakers, passive listening in three languages, one architecture, and fMRI’s limited temporal resolution.The authors call for broader language, participant, and measurement coverage to confirm, refine, or mitigate the findings.
Important brain–model gaps remain: wav2vec 2.0 uses unconstrained contextual access, differs from humans behaviorally, and achieves brain scores below the noise ceiling.The average brain score is 19%, reaching up to 74% in Heschl’s gyrus and sulcus.
By showing that self-supervised equations can generate brain-like processes, the work challenges the view that brain complexity precludes simple explanatory theories.

A.1 Self-supervised loss formula

The self-supervised wav2vec 2.0 objective combines diversity and contrastive losses. The contrastive term trains the model to identify the correct quantized representation for a masked input from competing candidates.

The total loss combines a diversity loss Ld with a Contrastive Predictive Coding loss Lm.Ld encourages diverse quantized representations, while Lm selects the correct quantized representation from alternatives.
For a masked time step, Lm makes the context output c choose the correct quantized representation q among K + 1 candidates.
The similarity function is cosine similarity, and κ is a constant temperature used during training.
Ld encourages equal use of the V entries in each of the G quantization codebooks by maximizing averaged softmax entropy.

A.2 Supervised loss formula

The supervised version of wav2vec 2.0 optimizes a Connectionist Temporal Classification loss over model parameters. The loss compares probabilistic predictions from raw waveforms with true transcriptions while accounting for possible alignments.

The supervised wav2vec 2.0 objective is a Connectionist Temporal Classification loss parameterized by θ.The optimization uses the negative log probability of the transcription under the model predictions.
mθ(U) denotes the model’s probabilistic predictions at each time sample for input raw waveform U.The waveform and predictions are represented as multidimensional arrays over time and feature dimensions.
The objective sums over aU,V, the set of possible alignments between waveform U and transcription V.This alignment set links the time-sampled predictions to the transcription sequence.

A.3 Preprocessing of the model’s activations

Model activations are session-wise normalized, then transformed to match the temporal structure of the fMRI data. Each artificial neuron is temporally convolved and down-sampled before analysis.

The network activations X are normalized to the interval [0, 1] separately for each listening session.This session-wise normalization precedes the temporal transformation.
Each artificial neuron is temporally convolved with the Glover model response h.The convolution models the temporal response associated with the fMRI measurement.
The convolved activations are temporally down-sampled using g to obtain the analysis-time representation.The mapping g converts the higher-rate activation sequence to the fMRI sampling times.

A.4 Penalized linear model - Ridge regression

The analysis fits an ℓ2-penalized linear model for each data split to predict transformed BOLD time series from model activations independently for each dimension.

For each split s, an ℓ2-penalized linear model V predicts transformed BOLD time series from model activations.The fitted mapping has dimensions V ∈ R^dx×dz.
The model is trained independently for each activation or brain-signal dimension.This separates the prediction problem across dimensions rather than fitting one undifferentiated output.
The fitted mapping connects the model representation to the transformed fMRI response within each split.The split-specific formulation supports evaluation of how well activations predict brain signals.

A.5 Probing the linguistic features encoded in wav2vec2 activations

The probing analysis tests which auditory and linguistic features are encoded across wav2vec 2.0 layers and training regimes. Its results indicate a hierarchy from spectro-temporal information toward phonetic, word-level, and sentence-level information, with higher-level features becoming speech- and language-specific.

Features and probing setup: The study probes wav2vec 2.0 activations for acoustic, phonetic, lexical, syntactic, and sentence-level features.The feature set includes MEL spectrograms, phonemes, word embeddings, part of speech, and sentence embeddings.
Features and probing setup: Ridge regression predicts five auditory and linguistic features from activations at each model layer and training condition.The analysis compares representations across layers and models using the Timit dataset.
Features and probing setup: The probe uses 1,680 short Timit samples from 24 speakers, with activations mean-pooled to each feature’s sampling rate.Each sample is an audio recording of a sentence shorter than 10 seconds.
Layer-wise hierarchy: Early transformer layers best account for spectro-temporal information, whereas deeper layers best account for phonetic, word-level, and sentence-level information.This partially follows the hierarchy predicted by neurolinguistics.
Layer-wise hierarchy: All probed features emerge with training, but only phone-, word-, and sentence-level features appear specific to speech and the training language.The result distinguishes learned higher-level linguistic specificity from more general acoustic representations.
Training-regime comparison: Word- and sentence-level features are encoded deeper in the supervised network than in the unsupervised network.The reported best layers are 18 for the supervised network and 14 for the unsupervised network.

A.6 Noise ceiling analysis

The noise ceiling estimates the maximum explainable fMRI signal by predicting one subject’s responses from other subjects hearing the same audio. Brain scores are then normalized by voxel-specific ceilings to contextualize model–brain similarity.

Noise ceiling estimation: Noise ceilings predict one subject’s brain signals from the average responses of other subjects listening to the same stories.The analysis uses the same evaluation procedure as the brain-score calculation, replacing model activations with other subjects’ brain activity.
Noise ceiling estimation: 290 subjects from the Narrative dataset contribute noise-ceiling estimates for voxel-wise normalization.The normalized scores are reported across voxels and in selected regions of interest.
Normalized brain scores: 74% of the noise ceiling is reached in Heschl’s region, while more than 20% is reached in STS and IFG.These values describe the unsupervised wav2vec2 model’s normalized brain scores and are low on average.
Normalized brain scores: Brain scores are divided by the noise ceiling for each voxel before reporting normalized model performance.This produces a voxel-specific ratio between model-predicted brain activity and the estimated explainable signal.