PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

Zhuofan Josh Ying, Guanglong Sun, Nolan Ho, Kieran Liu, Rui Yang, Andrew Chen, Nikolaus Kriegeskorte, Pony Ma, Hossein Adeli

arXiv:2610.26802v1cs.LG

TL;DR

Frozen-backbone readouts leave pretrained representations unspecialized for brain encoding, while large-model adaptation is difficult with costly, scarce neural data. The paper uses PEFT and single-pass feature extraction to adapt large multimodal backbones. It reports improvements across nine backbones and a new published state of the art on Qwen3-Omni with about 5% as many trainable parameters.

  • Problem

    Frozen-backbone readouts leave pretrained representations unspecialized for brain encoding, while adapting large models is difficult because neural data are costly and scarce.

    Source: §1 · p. 1

  • Method

    The paper applies PEFT to pretrained backbones and exploits causal masking to extract representations for all timesteps in one pass.

    Source: §1 · p. 2

  • Results

    LoRA improves performance across all nine tested multimodal backbones, and BrainLoRA sets a new published state of the art on Qwen3-Omni with about 5% as many trainable parameters.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results suggest scaling brain encoding through stronger pretrained backbones, smaller adaptations, and more lightweight adapters specialized to a shared backbone.

    Source: §6 · p. 10

    Models were trained on one benchmark with four subjects, all watching audiovisual narrative; transfer tests also remain within narrative film.

    Source: §6 · p. 10

Abstract

from arXiv · show

Brain encoding models aim to predict neural responses to stimuli. The most widely used approach fits readouts from frozen pretrained backbones. Using neural data to train large models from scratch or fully fine-tune them after task-pretraining has proven difficult because neural data is costly and limited. We show that parameter-efficient fine-tuning (PEFT) enables us to effectively adapt very large pretrained backbones for brain encoding with a small number of trainable parameters. To make it computationally practical, we exploit the backbone's causal structure to predict multiple timepoints in a single pass instead of using sliding windows, significantly reducing feature-extraction cost. This enables efficient fine-tuning of multimodal backbones with up to 104B parameters, the largest model adapted for brain encoding to our knowledge. Trained with a low-rank adapter (LoRA) and a lightweight readout, our approach consistently improves brain encoding performance across nine diverse multimodal backbones. On the same Qwen3-Omni backbone as the prior state-of-the-art (SoTA), our method BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge. We also show the adapted multimodal backbones remain interpretable: modality attribution recovers the canonical sensory topography that prior work often obtains by construction from separate unimodal encoders, and neural adaptation increases the contribution of audio inputs and cross-modal mixing within the backbone. To scale out the number of adapted models, we develop a multi-LoRA kernel that accelerates joint training of multiple LoRAs on a shared backbone. This enables an ensemble of 200+ adapted models that ranks first on the public Algonauts post-challenge leaderboard at the time of writing. Together, our results highlight PEFT as an efficient and powerful paradigm for brain encoding.

1 Introduction

Brain encoding has largely relied on frozen pretrained backbones, leaving their representations unspecialized, while adapting large models is difficult with costly, scarce neural data. This paper applies PEFT to large multimodal backbones, reporting stronger encoding performance, interpretable adaptations, and efficient training at scale.

Motivation

  • The paper targets the gap between frozen readouts, which do not specialize backbone representations for brain encoding, and adaptation approaches challenged by costly, scarce neural data.The evaluation uses the CNeuroMod/Algonauts 2025 Challenge, which tests whole-brain fMRI prediction for naturalistic audiovisual movies and generalization to held-out out-of-distribution stimuli.Source: §1 · pp. 1–2

Approach

  • PEFT retains a pretrained model’s prior while allowing limited neural data to adapt its representations through a small set of parameters.The paper frames scaling PEFT along three axes: stronger backbones, smaller adaptations, and more specialized adapters.Source: §1 · pp. 2–3

Efficiency

  • Causal masking enables single-pass extraction of all timesteps, reducing feature-extraction cost by approximately 350× relative to the prior sliding-window protocol, at little performance cost.The prior protocol redundantly recomputes shared stimulus context across overlapping windows.Source: §1 · pp. 2–3

Effectiveness

  • With about 5% as many trainable parameters, BrainLoRA sets a new published state of the art on Qwen3-Omni, while LoRA improves performance across all nine tested multimodal backbones.The tested backbones span 4.7B–104B parameters.Source: §1 · pp. 2–3

Interpretability

  • Modality attribution recovers canonical sensory topography, while comparison with the frozen backbone shows adaptation increases audio contribution and cross-modal mixing.Early visual cortex is driven by video, and auditory and lateral temporal cortex by audio.Source: §1 · pp. 2–3

Scalability

  • A batched multi-LoRA kernel accelerates joint training on a shared backbone, enabling an ensemble of 200+ adapted models that ranks first on the public Algonauts post-challenge leaderboard.Ensemble performance is jointly governed by ensemble size, individual model quality, and model diversity.Source: §1 · p. 3

2 Related work

Most encoding methods freeze pretrained task-optimized backbones and train only readouts. Fewer studies adapt backbones with neural data; this work extends that direction to substantially larger models.

Frozen backbone, trained readout

  • Most top-ranked Algonauts 2025 methods freeze pretrained task-optimized backbones and train readouts, so learning remains confined to the readout module.Recent methods use more expressive readouts than classic voxel-wise ridge regression.Source: §2 · p. 3

Backbones fine-tuned by brain data

  • Direct neural-data adaptation has been less common: prior examples include brain-tuning speech and vision models, LoRA methods, and a 3B Llama model that ranked 10th on Algonauts 2025.MEM won Algonauts 2023; BrainWavLM matched a full fine-tune with LoRA adaptation.Source: §2 · p. 3
  • This work extends direct adaptation to backbones up to 104B parameters; the authors state that prior models above 15B had, to their knowledge, been used only as frozen feature extractors.Source: §2 · p. 3

Other ways to incorporate neural data

  • Other approaches incorporate neural data through alignment objectives, neural-data pretraining, unified stimulus–brain models, or training from scratch; the cited work reports these have not yet outperformed large task-optimized feature backbones on public benchmarks.The passage notes that shuffled-target controls sometimes reproduce much of the reported benefit of alignment objectives.Source: §2 · pp. 3–4

3 Preliminaries

An encoding model maps stimuli to predicted neural responses using a pretrained backbone and a readout. The paper frames parameter-efficient adaptation as one of three complementary scaling axes.

3 Preliminaries

  • An encoding model pairs a pretrained backbone with a readout that predicts neural responses from stimuli.The readout also temporally aligns stimulus features with neural sampling.Source: §3 · p. 4
  • The optimization set distinguishes a frozen backbone, PEFT updates around pretrained weights, and full optimization over all backbone parameters.Training from scratch uses random rather than pretrained initialization.Source: §3 · p. 4
  • PEFT retains pretrained weights while neural data modifies the representation through a low-dimensional parameterization.LoRA, adapter layers, and prompt tuning are examples of this parameterization.Source: §3 · p. 4
  • The framework identifies three complementary scaling axes: larger pretrained priors, smaller adaptations, and more adaptations trained on a shared backbone.Predictions from the adaptations can be combined.Source: §3 · p. 4

4 Method

The method adapts pretrained multimodal backbones for whole-brain fMRI prediction on naturalistic audiovisual stimuli. It reduces feature-extraction cost with causal single-pass encoding and combines LoRA with lightweight readouts and a closed-form readout refit.

4 Method

  • The evaluation uses whole-brain fMRI from four training subjects, with Season 6 for validation and Season 7 and movies for held-out ID and OOD tests.The main experiments use Qwen3-Omni, the prior SoTA backbone, alongside experiments on nine backbones.Source: §4 · p. 4
  • The single-pass procedure encodes each 149-second window once and extracts representations for all timesteps, replacing 2,800 overlapping-window passes per clip.It exploits causal masking; the modality encoders have limited local bidirectionality, which the authors assess separately.Source: §4 · pp. 4–5
  • A causal single pass cuts feature-extraction cost by approximately 350×, and the adapted configuration surpasses the sliding-window baseline.With a frozen readout, single-pass features incur only a small drop; a 60-second context lead largely recovers it.Source: §4 · p. 4, Figure 2 · p. 5
  • The Qwen3-Omni configuration inserts rank-16 LoRA in attention modules through layer 32 and jointly optimizes adapters and readout for one epoch.The readout pools modality features into temporal-lag blocks and predicts 1,000 brain parcels; an alternative cross-attention readout accesses a richer token set.Source: §4 · p. 5
  • After joint optimization, a closed-form parcel-wise ridge regression refits the linear readout using accumulated subject-specific Gram matrices.The reported improvement primarily comes from the closed-form re-estimation rather than parcel-specific regularization.Source: §4 · p. 5

5 Experiments

Across nine multimodal backbones, PEFT improves brain-encoding performance, and BrainLoRA sets a new SoTA on the Algonauts 2025 benchmark. The experiments also examine component effects, modality attribution, and scaling to ensembles of adapted models.

5.1 Adapter gains across backbones

  • LoRA improves every tested backbone by 0.032–0.066 on linear readouts, and cross-attention adds an average 0.003 gain across all nine.The tested backbones span 4.7–104B parameters; adaptation improves every parcel, with largest gains in parietal and visual areas.Source: §5.1 · pp. 5–6, Figure 3 · p. 5

5.2 A small adapter beats a large readout

  • BrainLoRA sets a new SoTA, reaching 0.327 on ID and 0.228 on OOD, while using about 5% as many trainable parameters as MIRAGE.The ensemble of 200+ models reaches 0.343 on ID and 0.248 on OOD, topping the public leaderboard at the time of writing.Source: §5.2 · p. 6, Table 1 · p. 6

5.3 Ablating the model components and the input design

  • The LoRA adapter adds +0.0356 over the frozen-backbone baseline, while closed-form refitting adds +0.0061; full fine-tuning does not outperform LoRA at this data scale.Input-context reductions can lower frozen-readout performance, but adaptation recovers the lost performance; a 60-second context lead largely recovers the window-start drop.Source: §5.2 · p. 6, §5.3 · p. 7, Table 2 · p. 7

5.4 Interpreting the multimodal backbone

  • Modality attribution recovers canonical cortical organization: early visual cortex is predominantly driven by video (72.5%) and auditory cortex by audio (72.9%); adaptation raises average audio contribution by 39%.Source: §5.4 · p. 8

5.5 Training a population of independently adapted models

  • A fused multi-LoRA step for 32 adapters costs 1.08× a single-adapter step, a 29.8× speedup over sequential training.The kernel fuses routed updates into a batched gather–matrix multiplication; the gain comes from hardware utilization rather than fewer FLOPs.Source: §5.5 · p. 9, Figure 5 · p. 9
  • A 15-model subset reaches 0.3465 validation accuracy, within 0.0018 of the full 261-model ensemble, at one seventeenth of its inference cost.For random 20-model ensembles, mean accuracy and residual correlation explain 97% of score variance jointly, versus 23% and 5% individually; architecture changes provide the most useful diversity.Source: §5.5 · pp. 9–10, Figure 5 · p. 9

6 Discussion

The discussion frames PEFT scaling along backbone size, adaptation size, and adapter-pool size, while noting that the experiments remain within narrative-film fMRI. The input-design results show how a single causal pass and context lead trade context for much lower extraction cost.

6 Discussion

  • The authors propose scaling up pretrained backbones, scaling down trainable parameters, and scaling out lightweight adapters, but say this work is not yet a brain foundation model.They suggest shared or hierarchical adapters across tasks, datasets, subjects, or species as a possible future direction.Source: §6 · p. 10
  • The models use one benchmark with four subjects watching audiovisual narrative, and neither the held-out-subject nor second-dataset test leaves narrative film.Joint training across multiple datasets and modalities is left to future work.Source: §6 · p. 10

A Extended related work

  • Earlier work includes frozen-backbone readouts and brain-data adaptation, but the paper distinguishes its 104B whole-brain encoding evaluation from prior large-backbone work.The related-work discussion also notes that adapting models above 15B for brain encoding had not been established in the literature it reviews.Source: A Extended related work · p. 15
  • For multi-adapter training, the gather-based kernel is not presented as a new primitive; its contribution is efficiency when each adapter group contains only a few rows.The paper says the fused kernel's cost is nearly independent of pool size in this regime, unlike alternatives whose costs grow with it.Source: A Extended related work · p. 16

B The full input design

  • 0.0038 is the frozen-head cost of replacing sliding windows with a whole-clip causal pass, and a 60 s context lead recovers the additional 0.0046 cost of restricting the pass to a 149 s window.The input-design comparison reports two orders of magnitude less extraction compute for the whole-clip pass.Source: B The full input design · pp. 16–17
  • 0.3247 at a 60 s lead exceeds the MIRAGE head's 0.3187 sliding-window score, while the matched linear readout scores 0.3261 against 0.3241 for the head.The head was trained in two stages and received twice the linear readout's training steps.Source: B The full input design · p. 17
  • The first 30 s of context lead adds 0.00396 and the next 30 s adds 0.00142, with accuracy flat beyond 60 s.The reported step time rises from 9.7 s with no lead to 14.4 s with a 120 s lead.Source: B The full input design · p. 17
  • At the window start, the accuracy deficit is 0.0393 without a lead, 0.0220 with a 30 s lead, and 0.0235 with the whole clip; accuracy also falls at the window end.The passage attributes at most 0.003 of the end deficit to stimulus beyond the window boundary, within the s.e.m.Source: B The full input design · p. 17

C Efficiency and the multi-adapter kernel

The section compares accuracy with parameter and compute costs, then describes a fused multi-adapter kernel whose speed advantage comes from better GPU utilization in small-group routing.

C Efficiency and the multi-adapter kernel

  • Extraction used 545× fewer forwards and saved 352× in GPU-hours (700.6 versus 1.99 GPU-h), despite longer passes (≈14,900 tokens each versus a capped prefix); measured on the authors’ hardware.Source: C Efficiency and the multi-adapter… · p. 18
  • 8.5 GPU-h versus 702.5 yields an 83× lower end-to-end cost, with equal or better accuracy and an order of magnitude fewer trainable parameters than frozen-feature baselines.Figure 7 plots held-out Friends S7 accuracy against trainable parameters and end-to-end compute, including feature extraction.Source: C Efficiency and the multi-adapter… · pp. 17–18, Figure 7 · p. 18
  • At N = 32, the fused multi-LoRA pass is 29.8× faster than running adapters one at a time, while costing 1.00× a batch through one adapter.The speedup reflects GPU utilization rather than fewer FLOPs; memory limits the rank-16 pool to between 32 and 64 adapters on an 80GB card.Source: C Efficiency and the multi-adapter… · p. 18

D Selection and evaluation protocol

The evaluation protocol selects models and ensemble weights on season-6 validation data, quantifies seed variability, and uses held-out or out-of-fold procedures for fitting penalties and ensembles.

D Selection and evaluation protocol

  • All design choices, model membership, and ensemble weights were selected on Friends season-6 validation data; leaderboard scores were reported but not used for selection or tuning.The submissions were made during the post-challenge phase of Algonauts 2025.Source: D Selection and evaluation protocol · p. 18
  • 0.32609 ± 0.00024 across 15 seeds for LoRA + linear and 0.33070 ± 0.00023 across 3 seeds for LoRA + cross-attn; differences below about 0.0008 are not separated by the table alone.The comparisons in §5.2 use paired bootstrap or seed statistics.Source: D Selection and evaluation protocol · p. 19
  • Per-parcel ridge penalties are selected on a deterministic strided 15% holdout of training windows, then re-solved using fit and holdout together.Season 6 is used only for scoring, over its 250 windows.Source: D Selection and evaluation protocol · p. 19
  • The ensemble combiner is fitted out of fold by episode to avoid leakage between windows sharing stimulus and scanner state; the 261-model pool scores 0.34007 for a uniform mean and 0.34834 for the stack.The out-of-fold predictions cover every row exactly once.Source: D Selection and evaluation protocol · p. 19
  • The 15-model subset is chosen and weighted on one episode half, scored on the other, and assembled out-of-fold; it scores 0.34651 with estimated selection bias of +0.00035.The nested procedure uses the two quarters within each half to set the greedy stopping step.Source: D Selection and evaluation protocol · p. 19

E Hyperparameter selection

The selection analyses qualify rank, layer, and training choices against seed variability and configuration constraints. Additional tests examine length-dependent representation differences and describe the ensemble pool and its plotted convergence axis.

E Hyperparameter selection

  • At the matched readout rate, rank 32 is only +0.0006 above rank 16, within seed spread; rank 64 loses 0.0025 and higher-rank runs also diverged.Rank and learning rate are coupled because α = 2r is held fixed; higher ranks were run at the rank-16-tuned learning rate.Source: E Hyperparameter selection · p. 20
  • Trained layers 31–34 lie within 0.0006 of one another, inside the 0.00077 seed spread, so layer 32 is a choice within a flat neighborhood rather than a verified optimum.The frozen-feature screen peaks at layer 30.Source: E Hyperparameter selection · p. 20, Table 6 · p. 21
  • The readout rate has an interior optimum at 3 × 10−4, while a second epoch stays below the first-epoch peak throughout its second pass.Two matched one- versus two-epoch pairs give −0.0147 and −0.0047; the learning-rate runs predate the context lead and closed-form re-fit.Source: E Hyperparameter selection · pp. 20–21

F The effect of local non-causality in the backbone

  • Shared-timestep audio representations differ most in the first chunk, with cosine 0.919 there despite identical tokens 120 s before the cut; the authors attribute this to length-dependent bf16 numerics.The proposed sources are different attention-kernel tiling across sequence lengths and top-k expert routing turning rounding differences into a different expert.Source: F The effect of local non-causality in… · pp. 21–22

G Ensemble configurations

  • The y-axis, −log10(m∞−r), expresses equal steps as equal fractions of the remaining gap, so a straight line describes convergence rate rather than unbounded growth.Subsets above the mean-subset limit m∞ fall at −∞ on this axis and are omitted.Source: G Ensemble configurations · p. 22
  • The 261-member pool lists distinct configurations and groups members that differ only by random seed; all share the shipped recipe unless a column specifies otherwise.The shipped recipe uses Qwen3-Omni, a linear readout, layer 32, rank 16, a 60 s lead, and 1,429 steps.Source: Table 10 · p. 22, G Ensemble configurations · p. 22

H Alternative PEFT methods

Alternative PEFT methods all improve over the frozen backbone, though LoRA gives the largest gain; across seeds, adapters become more similar in their computed features and predictions than their weights suggest.

H Alternative PEFT methods

  • All four non-LoRA configurations improve 998 of 1,000 parcels, with gains for every subject, showing that adaptation benefits are not specific to LoRA.LoRA nevertheless gives the largest gain.Source: H Alternative PEFT methods · p. 25
  • At 96% of LoRA’s parameters, the Houlsby adapter recovers 79% of its gain but trails LoRA by 0.0073; the FFN-only placement is within 0.0026 of Houlsby at half the parameters.The bootstrap does not resolve the placement difference.Source: H Alternative PEFT methods · p. 25
  • IA3’s 57,344 coefficients—0.64% of LoRA’s adapter parameters—recover 62% of LoRA’s gain, making IA3 the most efficient by gain per parameter.Scores do not track adapter size: the three middle configurations span 3.18M–8.52M parameters and lie within 0.0026 of one another.Source: H Alternative PEFT methods · p. 25

I The backbone comparison

  • Across the nine-backbone comparison, configurations use rank-16 LoRA and a shared recipe, with feature-extraction layers selected per backbone.Each backbone contributes one fixed run covering all four subjects, rather than its best-scoring run.Source: I The backbone comparison · pp. 25–26, Table 12 · p. 26

J Why seeds are not a source of diversity

  • Adapter changes from different seeds are nearly orthogonal in weight space (cosine 0.026), but their composed feature changes agree increasingly with depth, reaching 0.860 at the feature-extraction layer.The feature change there remains 45% of the feature norm; weight-space orthogonality follows from the LoRA parameterization and is not evidence of functionally distinct adapters.Source: J Why seeds are not a source of… · pp. 26–27
  • Under one shared readout, predictions from different seeds correlate at r = 0.934, and the seed-specific prediction component has no target signal, explaining why seed-only ensembles saturate.Pooling seeds saturates by about 12 members, while a different readout or backbone still adds accuracy.Source: J Why seeds are not a source of… · pp. 26–27

K Additional null results

Several readout, input, and subject-specific variants add little or no benefit, while the shifted-target control shows that adaptation gains depend on stimulus-response alignment. Adaptation also transfers to an unseen subject.

K Additional null results

  • The gated-MLP readout’s +0.0007 and pooled-feature cross-attention’s +0.00030 are within the seed spread; the latter’s gain comes from its larger token set, not attention itself.The cross-attention interval is [+0.00017, +0.00045].Source: K Additional null results · p. 27
  • Using the full 19,099-token sequence gives no gain, and reducing shared training stimuli from 99.3% to 44.1% changes residual correlation by only +0.00006.Randomly reducing training-data fractions lowers accuracy monotonically.Source: K Additional null results · p. 27
  • Per-subject adapters score −0.0019 relative to the shared adapter, but a compute-matched control indicates the loss reflects continued training rather than subject-specific specialization.The result was replicated at a second seed.Source: K Additional null results · p. 27

L Shift-target control

  • Training on shifted fMRI makes the adapter score 0.0733 below its matched frozen control and lose on 996 of 1,000 parcels, so the aligned-data gain does not arise from marginal fMRI statistics.Misaligned projection training alone also degrades performance (−0.0299); shifted-target training preserves response statistics but breaks stimulus pairing.Source: L Shift-target control · pp. 27–28

M Does the adapter transfer to an unseen subject?

  • A held-out subject gains +0.0301 to +0.0355 over the frozen backbone on 995–998 of 1,000 parcels, and scores within −0.0005 to +0.0021 of in-sample subjects under the three-subject adapter.The gain transfers, but the test does not cover other scanners, protocols, or datasets.Source: M Does the adapter transfer to an… · pp. 28–29, Table 14 · p. 29

N Does the adapter transfer to a new dataset?

On NNDb, the Algonauts-trained LoRA model improves encoding, with a gain pattern partly shared across datasets. Re-fitting readouts yields a larger measured gain, which the authors attribute to readout ease rather than a larger adapter effect.

Evaluation setup

  • NNDb evaluation uses 20 participants who watched the same film, without retraining the Algonauts backbone or adapter; the transferred readout is unchanged, while the re-fitted readout is solved per participant.The readout comparison also tests whether a readout fitted to other people transfers.Source: N Does the adapter transfer to a new… · p. 29

Transferred readout

Re-fitted readout

  • With a re-fitted readout, the gain is +15.4 points of the ceiling and +0.0338 in r, across 990 of 1,000 parcels and all 20 participants.The larger gain is attributed to adapted features being easier to read out, not to a larger adapter effect.Source: N Does the adapter transfer to a new… · p. 30

Spatial pattern

  • The NNDb and Algonauts parcelwise gain maps correlate 0.620 (0.677 after reliability correction), with about 46% of their spatial pattern shared and the rest dataset-specific.Both maps are reliable: split-half reliability is 0.940 on NNDb and 0.891 on Algonauts.Source: N Does the adapter transfer to a new… · p. 30
  • At the network level, the adapter improves every network; it is closest to the noise ceiling in visual and dorsal attention networks and furthest in limbic.Figure 10 compares subject-mean r per parcel with the corrected ceiling and reports mean percentage of ceiling by Yeo network.Source: Figure 10 · p. 30, N Does the adapter transfer to a new… · p. 31

O Transfer to a new subject with little data

For a new subject with little calibration data, anchoring the readout to the population estimate outperforms fitting from scratch, and the adapted model’s advantage persists across data budgets. Regional gains are widespread but vary in size.

O Transfer to a new subject with little data

  • Anchored ridge beats the population readout by +0.0143 at 20 minutes and +0.0423 at 159 minutes, while ridge from scratch needs about 80 minutes merely to match the population readout.Anchored ridge beats ridge from scratch for all 20 subject–draw pairs at every budget; its gains close 21% and 62% of the gap to the subject’s full-data readout at 20 and 159 minutes.Source: O Transfer to a new subject with… · pp. 31–32
  • The low-rank deviation adds only +0.0029 at 40 minutes and +0.0067 at 159 minutes over the population readout; anchored ridge beats it from 20 minutes onward.This tests a fixed, closed-form basis with three training subjects, not RABBiT’s end-to-end method.Source: O Transfer to a new subject with… · pp. 31–32
  • With anchored ridge, the adapted model beats the frozen backbone at every calibration budget, gaining +0.0236 with no new-subject data and +0.0326 at 159 minutes.The adapted features also gain more from the same calibration data: +0.0423 versus +0.0333 at 159 minutes.Source: Table 16 · p. 31, O Transfer to a new subject with… · p. 32
  • On NNDb, anchored ridge improves over the transferred readout by +0.0113 at 40 minutes and +0.0150 at 70 minutes; the adapter’s gain over frozen grows from +0.0215 to +0.0308.The adapter beats frozen for all 20 participants at every budget.Source: Table 16 · p. 31, O Transfer to a new subject with… · p. 32

P Where in the brain the adapter helps

  • Every region improves, with the largest gains in parietal and visual cortex; dorsal attention gains most (+0.050 on a frozen 0.330), while limbic gains least (+0.015 on 0.131).Among ROIs, MT+ leads (+0.055), while medial prefrontal cortex trails (+0.025); relative gains are largest in the somatomotor strip and intraparietal sulcus.Source: Table 17 · p. 32, P Where in the brain the adapter helps · p. 33, Figure 11 · p. 33
  • Early auditory and STS/lateral temporal regions have relatively small gains (+0.031 and +0.030), while gain is not simply larger in parcels where the frozen model scores well.Across 757 parcels with test–retest reliability above 0.1, gain correlates −0.016 with frozen score.Source: P Where in the brain the adapter helps · p. 33

Q Input-level modality attribution

Input-level attribution evaluates modality subsets by replacing absent-stream embeddings with training-mean embeddings, and exact Shapley values quantify each stream’s contribution. Adaptation shifts reliance toward audio and increases cross-stream contribution, while the streams remain substantially redundant.

Q Input-level modality attribution

  • Video alone recovers 71% of the full score and audio alone 76%, while removing either stream costs 0.0675 and 0.0665, respectively.Figure 12 contrasts the full model with each single-stream and stream-removal configuration; absent streams are replaced by training-mean embeddings.Source: Q Input-level modality attribution · pp. 33–34, Figure 12 · p. 34
  • Audio alone gains +0.0503 after adaptation, video alone gains +0.0329, and transcript alone loses 0.0065; removing the transcript costs 0.0103 after LoRA.The adapted model draws from speech audio information that the frozen model obtained from the transcript.Source: Q Input-level modality attribution · p. 34
  • Cross-stream contribution rises from 8.9% [8.6, 9.2] with the frozen backbone to 15.3% [14.9, 15.6] after LoRA, a paired difference of +6.4 points [+6.1, +6.6].Video information read at audio positions accounts for 7.5%, versus 3.9% in the reverse direction; audio positions carry 45.8% of the adapted prediction, while transcript positions fall to 14.2%.Source: Q Input-level modality attribution · p. 34
  • Audio’s contribution increases most in regions that relied least on audio before adaptation, while auditory cortex shows the smallest gain.Figure 13 reports regional audio contributions for frozen and LoRA + linear models with episode block-bootstrap intervals.Source: Figure 13 · p. 35
  • Replacing absent-stream embeddings with embeddings from another episode yields share maps correlated at 0.994 and the same dominant modality for 98.2% of parcels.The alternative baseline is in-distribution per token but wrong in content.Source: Q Input-level modality attribution · p. 35
  • The adapted and frozen contribution maps correlate at 0.963–0.990, but dominant modality agrees in only 74.4% of parcels, outside the 97.2% seed agreement.With cross-attention, the dominant modality changes in fewer parcels (8.0% versus 25.6%), in the same direction, and cross-stream contribution rises from 11.9% [11.5, 12.2] to 15.8% [15.4, 16.3].Source: Q Input-level modality attribution · p. 35

R What the pool still misses

The ensemble’s remaining residual contains stimulus-locked variance not explained by its predictions, and its character differs from low-level stimulus features or timing errors. The reproducibility details specify the hardware, training configuration, lag/readout design, and inputs needed to rerun the model.

R What the pool still misses

  • The decomposition uses the shared pool prediction and cross-subject residual covariance to distinguish explained, missed, and cross terms in inter-subject covariance.All four subjects viewed the same 250 validation windows, allowing cross-subject residual correlation to indicate stimulus-locked signal the pool does not produce.Source: R What the pool still misses · p. 35
  • 32.5% [30.0, 35.3] of cross-subject stimulus-locked variance remains unexplained by the 261-model pool, compared with 36.2% for a single model.Averaging the models recovers 3.7 points; the four subjects agree within 1.4 points.Source: Table 18 · p. 36, R What the pool still misses · p. 36
  • The 32.5% missed share is a lower bound relative to all stimulus-driven variance because inter-subject correlation excludes subject-specific stimulus-driven signal.The pool explains variance equal to 94% of ISC within a subject, while its missed share is 26% of captured plus missed variance.Source: R What the pool still misses · p. 36
  • Low-level stimulus statistics explain the shared residual at only r = 0.027, and lagged pool predictions explain 0.048; the residual is high-dimensional and slow.The top ten components hold 36% of residual variance, versus 91% of prediction variance; the authors say this fits narrative or event-level structure better than a missing low-level feature.Source: R What the pool still misses · p. 36

S Reproducibility

  • Runs use one NVIDIA A800-80GB, with bfloat16 backbone execution, FlashAttention-2, gradient checkpointing, and 1,429 single-window training steps.LoRA uses rank 16 on attention projections through layer 32, with 8.91M parameters; the vision patch embedding convolution runs in float32.Source: S Reproducibility · p. 36
  • The readout projects each modality to 512 features per frame, applies eight lag blocks, then maps to 1,000 parcels; 158 of 800 lag-design rows are zero.The eight blocks tile [0.5, 192.5) seconds, and the operator is shared by training and the closed-form solve.Source: S Reproducibility · p. 37, Table 19 · p. 37
  • The cross-attention readout uses eight learned queries per modality with a causal cutoff, while a rerun requires cached media, fMRI data, backbone weights, and readout initialization.Stage 2 re-solves the readout in closed form in one forward-only pass.Source: S Reproducibility · pp. 37–38
  • The ensemble uses per-parcel minimum-variance weights with covariance shrinkage selected on an inner episode split; the specified backbone revisions identify the loaded repositories and commits.For Phi-4-MM and LCO-Embedding-Omni, upstream main had moved past the commits used.Source: S Reproducibility · p. 37, Table 20 · p. 38

Related papers

Loading 2610.26802v1…