Source-linked AI summary
BAT-CLIP: Trimodal Alignment of Brain, Audio and Text
Suhyun Kim, Jinmo Han, Danny Dongyeop Han, Ahhyun Lucy Lee, Jewoon Lee, Yonghyeon Gwon, Zach Paris, Chun Kee Chung, Saewoong Bahk, Nam Soo Kim, Seong Jae Hwang, Jiook Cha
TL;DR
Current brain–speech alignment methods use a single audio or text anchor, although speech comprehension draws on both acoustic and linguistic information. BAT-CLIP aligns iEEG representations to frozen audio and text anchors, and reports stronger linguistic and semantic probe performance than single-anchor alignment, with the most reliable trimodal gains when initialized from self-supervised features.
Problem
Existing CLIP-style brain–speech alignment uses a single audio or text anchor despite the brain's integration of acoustic and linguistic information and the complementary aspects each represents.
Source: §1 · p. 2Method
Results
BAT-CLIP yields stronger performance on linguistic and semantic probes than single-anchor alignment, with the most reliable trimodal gains when initialized from self-supervised neural features.
Source: §4 · p. 5Takeaways & Limitations
The findings motivate pretraining neural representations with self-supervised learning and then grounding them through trimodal contrastive alignment in a shared audio–text manifold.
Source: §4 · p. 5The probe-task evaluation does not directly test end-to-end decoding, and how BAT-CLIP scales to substantially larger iEEG collections remains unclear.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.
1. INTRODUCTION
Brain–speech alignment links neural activity with speech and language representations, supporting both neural prediction and decoding. Because audio and text models capture complementary aspects of speech processing, BAT-CLIP aligns neural representations to both anchors.
1. INTRODUCTION
- Brain–speech alignment supports neural-response prediction and decoding, and offers a way to test which computational representations match the brain and where representational levels emerge across cortex.The paper also connects stronger alignment to downstream decoding of linguistic variables and generation of coherent text.Source: §1 · p. 1
- CLIP-style methods align neural signals end-to-end to pretrained embedding spaces, leveraging pretrained audio–text models in encoding and decoding tasks.Prior work reports gains across naturalistic speech and vision in fMRI and electrophysiology.Source: §1 · p. 1
- Existing CLIP-style brain–speech methods use only audio or text anchors, despite speech comprehension integrating both linguistic and acoustic information.Audio models better explain earlier auditory regions, whereas text models better explain later language regions linked to higher-level processing.Source: §1 · pp. 1–2
- BAT-CLIP aligns a pretrained brain foundation model to frozen AudioCLIP audio and text anchors with a trimodal contrastive objective.The paper also compares random initialization with self-supervised pretraining and analyzes how trimodal alignment organizes neural representations.Source: §1 · p. 2
2. METHODS
BAT-CLIP adapts a pretrained brain encoder to paired iEEG, audio, and transcript data, aligning brain embeddings to frozen AudioCLIP anchors. Its objective combines brain–audio and brain–text contrastive losses.
2.1. Problem Formulation
- Each training example pairs an iEEG segment with its corresponding audio and transcript, with signals resampled or tokenized for the pretrained models.iEEG is downsampled for DIVER-1, audio is resampled to 44.1 kHz for AudioCLIP, and transcripts use the AudioCLIP tokenizer.Source: §2.1 · p. 2
2.2. Alignment Prior: DIVER-1 Initialization
- The trainable brain encoder maps neural windows into a 1024-dimensional shared embedding space.The encoder starts from DIVER-1 and projects flattened embeddings through a lightweight two-layer MLP followed by ℓ2 normalization.Source: §2.1 · p. 2, §2.2 · p. 2
2.3. Shared Audio-Linguistic Space: AudioCLIP Anchors
- Pretrained AudioCLIP audio and text encoders produce normalized anchor embeddings that remain frozen during training.Keeping the anchors fixed preserves a stable audio–text manifold in the low-data setting.Source: §2.3 · p. 2
2.4. Trimodal Contrastive Alignment
- The method aligns brain embeddings to both anchor modalities using symmetric contrastive losses over batch similarity logits.The logit scale α = 14.3 sharpens the distribution and stabilizes optimization.Source: §2.4 · p. 2
- The final objective sums symmetric Brain–Audio and Brain–Text contrastive losses.The method computes both pairwise losses over the global batch.Source: §2.4 · p. 2
3. EXPERIMENTS AND RESULTS
The experiments evaluate brain–speech alignment on the Podcast Benchmark, then test trimodal alignment, the role of self-supervised initialization, and the resulting embedding structure. BAT-CLIP is the strongest aligned model overall, while its advantages depend on the task and a strong pretrained initialization.
3.1. Experimental Settings.
- The Podcast Benchmark evaluates decoding from naturalistic iEEG across six probe tasks, using standardized tasks, metrics, and data splits.The setup compares aligned variants with a supervised CNN and brain foundation models, using DIVER-1 as their common initialization.Source: §3.1 · pp. 2–3
3.2. Benefits of Trimodal Alignment.
- BAT-CLIP is the best aligned model overall and leads the aligned variants on five of six probes, but it trails BA-CLIP on Content and alignment weakens Onset performance.Onset remains strongest for the CNN and unaligned DIVER-1; the authors hypothesize that pooled window embeddings can wash out transient cues.Source: Table 1 · p. 3, §3.2 · pp. 3–4
3.3. SSL Initialization for Trimodal Alignment.
- Pretrained DIVER-1 initialization substantially strengthens CLIP-aligned decoding; without it, trimodal alignment does not reliably separate from unimodal variants.The authors describe pretrained initialization as key to realizing gains from trimodal supervision.Source: §3.3 · p. 4, Table 2 · p. 4
3.4. Embedding-space analysis
- BAT-CLIP shows the strongest sentence-level grouping, and scores highest on both clustering metrics against DIVER-1 and the bimodal variants.The t-SNE visualization and quantitative clustering evaluation concern word-level embeddings for representative subject Sub03.Source: §3.4 · pp. 4–5, Figure 2 · p. 4, Table 3 · p. 5
4. DISCUSSION AND CONCLUSION
BAT-CLIP combines self-supervised brain representations with trimodal alignment, and the authors report stronger linguistic and semantic probe performance than single-anchor alignment. End-to-end decoding remains untested, and scaling to substantially larger iEEG collections is unclear.
4. DISCUSSION AND CONCLUSION
- SSL-pretrained neural features paired with trimodal grounding consistently outperform training from scratch and yield stronger linguistic and semantic probe performance than single-anchor alignment.The authors identify SSL initialization as key to reliable gains from trimodal supervision.Source: §4 · p. 5
- The proposed alignment recipe combines an SSL neural prior learned from massive unlabeled recordings with both audio and text anchors using limited paired data.The example uses DIVER-1 pretrained on about 5.3k hours and about 30 minutes of paired data.Source: §4 · p. 5
- Probe-task results do not directly test end-to-end decoding, and how BAT-CLIP scales to substantially larger iEEG collections remains unclear.The authors also note that naturalistic iEEG datasets are scarce and rarely open.Source: §4 · p. 5