PaperlayerView exact revision on arXiv

Source-linked AI summary

The Platonic brain bridge hypothesis: human brain networks as an architectural prior for omni models

Pengfei Zhang, Biao Tian, Xiangang Li, Li Liu

arXiv:2609.10947v1q-bio.NCcs.LG

TL;DR

Whether brain–model correspondence can inform model architecture, rather than only brain prediction, is a question the study addresses. It builds brain-network organization into omni models and reports improved performance across three heterogeneous bases and five public benchmarks.

Abstract

from arXiv · show

We propose the Platonic brain bridge hypothesis: omni models, which process video, audio and text jointly like the brain, converge on brain-like representations, and the correspondence is bidirectional. From model to brain, brain-likeness of seven omni models is stable across participants, and our encoding models on their internal hidden states rank first on the Algonauts 2025 out-of-distribution leaderboard. From brain to model, three contributions follow. Brain-MoE gives seven cortical networks one brain-pretrained expert each and raises held-out accuracy in all 15 model-benchmark pairs by 6.42 percentage points on average. Brain-AVQA builds questions from video clips labelled by the most responsive brain network; the real network-to-expert map exceeds shuffled maps in-domain on all three models. Brain-Scope uses sparse autoencoders to localize the correspondence to a small subset whose removal weakens brain prediction in all three bases tested. Human brain networks are therefore a usable architectural prior for omni models.

Results

The results examine how omni-model representations predict human cortical responses and how brain-derived features, network structure, and expert content relate to model performance. They cover encoding across the tested bases, benchmark accuracy, and the Brain-AVQA comparison of real and shuffled brain-to-expert maps.

Omni models correspond to human cortical responses on a stable, structured spectrum

  • All seven omni models predicted held-out cortical responses, with brain-likeness from 0.337 to 0.381 and highly consistent participant rankings (mean pairwise Spearman ρ = 0.958).The encoding-model pool ranked first in all five columns of the public Algonauts 2025 out-of-distribution leaderboard. Across three bases, adding modalities raised brain-likeness, and no model peaked at its last layer.Source: Omni models correspond to human… · pp. 4–5, Figure 2 · p. 5

The correspondence localizes to sparse features shared across models

  • Matched brain-aligned features corresponded across models: over 13,383 shared time points, mean correlations were 0.5257, 0.5344 and 0.5068, 13.3–13.8 times each pair’s circular-shift maximum.The matched sets comprised 185, 184 and 190 features (1.1–2.3% of each basis); no non-identity shift or size-matched random set reached the observed values.Source: The correspondence localizes to sparse… · p. 6

Inter-model differences are confined to one shared axis

  • Models shared 97.87% of encoding-map variance; after removing that component, PC1 explained 87.8% of residual variance and the participation ratio was 1.28.The PC1 loadings were 5.37 times larger on sensory than association parcels, and neither the axis nor whole-cortex brain-likeness tracked the capability composite after Holm correction.Source: Inter-model differences are confined… · p. 6, Figure 4 · p. 9

Brain-MoE improves held-out accuracy on Brain-AVQA

network-to-expert map and transfers to unseen media

  • The real brain-network-to-expert map exceeded rotated and uniform maps on all three bases; on unseen episodes and media, routing hit rates were 0.2647–0.2997 versus 0.088–0.164 for controls.In-domain, the real map exceeded 20 random maps by 1.70 pp on average; the unseen-media hit rate did not differ from the unseen-episode rate (0.2795 versus 0.2857; P = 0.659).Source: network-to-expert map and transfers to… · pp. 9–11

Brain-pretrained experts improve unseen-question accuracy across bases and benchmarks

  • Brain-pretrained experts improved accuracy in all 15 public benchmark cells, with mean ∆1 = +6.42 pp; they exceeded random experts in 14 cells and equalled them in one.The mean ∆2 was +3.26 pp, and a question-level joint sign-flip permutation gave P = 3.2 × 10−4 for 14 positive cells.Source: Brain-pretrained experts improve… · p. 11

The size of the gain tracks base headroom

  • Across 15 cells, ∆1 and ∆2 decreased as base accuracy rose (Spearman ρ = −0.81 and −0.80), while the brain-to-random gain ratio was unrelated to base accuracy (ρ = −0.05).The geometric mean brain-to-random gain ratio was 2.55, indicating that the brain arm gained about two and a half times as much as the random arm across base accuracies.Source: The size of the gain tracks base… · p. 11

The gain comes mainly from expert content

  • Expert content accounted for the gain across five comparable cells (+1.93 to +7.31 pp), while restoring the real routing prior was significant only for MiniCPM × OmniBench (+2.28 pp).Routing effects were near zero in the three MMAU-Pro cells and Qwen3 × OmniBench; all had P ≥ 0.17.Source: The gain comes mainly from expert… · p. 11
  • Across 14 content/prior contrasts, content effects were positive (+1.40 to +7.18 pp), while prior effects stayed within 2.3 pp; the real prior helped only with brain experts on MiniCPM × OmniBench (+2.63 pp).This pattern suggests routing matters only when the selected expert holds the corresponding subspace.Source: The gain comes mainly from expert… · p. 12

Brain-aligned features are necessary for encoding on all three bases and for benchmark accuracy on Qwen3-Omni

  • Brain-aligned features caused positive excess encoding damage on all three bases: Qwen3 +0.01012, Ming +0.00128, and MiniCPM +0.00293.The Qwen3 value was largest; the positive sign also held under the reference criterion on all three bases and under the strict criterion where applicable.Source: Brain-aligned features are necessary… · p. 13
  • On Qwen3-Omni, zeroing 98 Brain-Scope features reduced OmniBench accuracy from 0.6130 to 0.5166 (−9.63 pp) and caused +9.11 pp excess damage versus matched controls.Across five benchmarks, excess damage was +1.73 pp pooled; Ming and MiniCPM showed near-zero downstream excess damage.Source: Brain-aligned features are necessary… · pp. 12–13

Discussion

The discussion argues that brain–model correspondence works in both directions: omni-model representations align with brain organization, and brain functional networks provide a usable architectural prior for omni models. It presents the gains as attributable to identified components while noting limits in scope and questions for future testing.

Discussion

  • Brain-MoE improves three heterogeneous bases across five public benchmarks, with controls attributing the gains to brain-pretrained expert content and the real network-to-expert map.Its routing prior comes from Brain-Scope features necessary for brain encoding on all three bases and benchmark accuracy on one. The injected organizational principle uses no brain data; the prior is held in a frozen feature subspace behind a read-only path, and validation uses benchmarks unrelated to the recordings. The prior is defined at seven Yeo networks; finer parcellations and other atlases remain untested.Source: Discussion · p. 13
  • The study extends organizational-prior work by testing whole-brain functional networks as the expert partition and routing prior for an omni generative model across public capability benchmarks.Earlier work includes brain-tuning and steering [14, 16–18], cognitive-specialization comparisons [20], topology controls [23], omni capability evaluation [49], and a Yeo-7 expert prior for brain-to-image reconstruction [21]. The study freezes the brain-learned feature subspace before downstream training; complementary evidence links architectural organization to brain correspondence [22, 50].Source: Discussion · p. 13
  • The discussion relates its model-to-brain findings to prior tendencies that instruction tuning need not improve overall brain alignment [34] and brain-likeness often peaks at intermediate depth [7].The full-modality increment in control and default networks is qualitatively consistent with the association-cortex benefit reported for TRIBE [9].Source: Discussion · p. 13
  • Brain-Scope links brain-response prediction, expert routing, and public-capability ablation through one feature set, while Brain-AVQA labels each question by the dominant network measured in four participants.These designs make the feature set and the network prior driving each question reusable and testable.Source: Discussion · p. 13
  • The discussion identifies uncertainty about capability–brain coupling and whether the gain decrease with base accuracy generalizes across model families and training seeds.Following one base along its training trajectory could estimate coupling without the shared axis of a seven-model panel; broader stimuli, participant, model-family, and seed coverage would test generality. Embodied agents and adding proprioception, touch, and action consequences are proposed as extensions.Source: Discussion · p. 14

Methods · Data and participants · Model panel and feature extraction

The methods link hidden states from seven omni models to fMRI responses, then use the resulting correspondence to guide brain-to-model analyses. The study uses public naturalistic-viewing data and a shared feature-extraction protocol.

Methods

  • The workflow first relates seven omni models’ hidden states to fMRI, then uses the correspondence for Brain-Scope, Brain-MoE and benchmark analyses.The methods also describe controls, ablations, headroom and routing analyses, and shared statistical procedures.Source: Methods · p. 14

Data and participants

  • Four participants watched seasons 1–6 of Friends in a 3 T scanner, with fMRI sampled every 1.49 s.The data came from the public CNeuroMod naturalistic-viewing dataset.Source: Data and participants · p. 14
  • Provider-preprocessed functional images were released in the Schaefer-1000 cortical parcellation, with each parcel labelled by its Yeo-7 network.The seven labels are visual, somatomotor, dorsal attention, salience/ventral attention, limbic, control and default.Source: Data and participants · p. 14
  • Seasons 1–5 formed the training set and season 6 the episode-level validation set; each TR paired video, audio and subtitles, with modality channels masked for modality analyses.The split prevents the sets from sharing a continuous narrative segment; responses were represented as 1,000-parcel vectors.Source: Data and participants · p. 14
  • External validity was evaluated on the official Algonauts 2025 held-out set, scored on responses to six unseen films.The passage reports a leaderboard snapshot dated 4 September 2026; the study collected no new human data and used public, de-identified data with consent and ethics approval.Source: Data and participants · p. 14

Model panel and feature extraction

  • The forward panel used seven frozen-weight omni models; per-TR hidden states were token-averaged at four relative depths, reduced by layer-wise PCA to 512 dimensions and concatenated into 2,048-dimensional features.PCA was fitted on training data and projected onto validation data; the Qwen2.5-Omni 3B checkpoint postdated its family’s cited technical report.Source: Model panel and feature extraction · pp. 14–15

Brain encoding model · Modality and depth analyses

The section defines whole-brain brain-likeness from parcel-wise encoding accuracy, then analyzes how the brain–model correspondence varies with modality and model depth. These analyses locate where correspondence is strongest; Brain-Scope next examines the individual features expressing it.

Brain encoding model

  • Parcel-wise encoding models predict responses at a fixed lag of two TRs using ridge regression, with regularization selected by leave-one-out cross-validation.The selection uses 10 log-spaced candidates between 10^2 and 10^6.Source: Brain encoding model · p. 15
  • Whole-brain brain-likeness is the mean Pearson correlation between predicted and measured responses across 1,000 parcels and participants.This score is the unit of every model-to-brain comparison.Source: Brain encoding model · p. 15

Modality and depth analyses

  • Modality analyses re-extracted hidden states from six non-empty video, audio, and subtitle subsets for Ming-flash-omni-2.0, MiniCPM-o 4.5, and Qwen3-Omni.The subset features were processed using the encoding-model equations.Source: Modality and depth analyses · p. 15
  • Channel overlap was assessed by comparing the V+A increment over V with the A increment over T, while saturation was assessed by whether V+A approached V+A+T.Uncertainty for subset values and nested contrasts came from a paired bootstrap over 1,000 parcels with 10,000 resamples; the intervals are optimistic because parcels are spatially correlated and only one validation split was used.Source: Modality and depth analyses · p. 15
  • Qwen3-Omni modality features used a separate float32 extraction pass whose whole-cortex value was 0.018 above the depth-matched pass.Within-model contrasts are exact, but audio-only and text-only values are lower bounds, and contrasts subtracting text-only are upper bounds, due to an extraction-pass artefact.Source: Modality and depth analyses · p. 15
  • Depth analyses fitted 28 encoding models across four relative depths for each of seven models, recording peak relative depth and the last-to-peak ratio.Peak position was related to model size and an external capability composite using Spearman correlations and exact P values from all 5,040 permutations at n = 7; the composite averages z-scores on World-Sense54, Daily-Omni55, and MMAU44.Source: Modality and depth analyses · p. 15
  • Modality and depth analyses identify where correspondence is strongest, while Brain-Scope examines the individual features that express it.Source: Modality and depth analyses · p. 15

Brain-Scope: sparse autoencoders and brain readout · Cross-model feature matching and exact permutation

Brain-Scope extracts sparse features from three omni models, reads them out to fMRI, and uses brain-aligned features to inform routing and cross-model matching. Cross-model correspondence is tested with one-to-one feature matching and circular-shift permutations.

Brain-Scope: sparse autoencoders and brain readout

  • TopK sparse autoencoders were trained on probe-layer hidden states from MiniCPM-o 4.5, Qwen3-Omni-30B-A3B-Instruct, and Ming-flash-omni-2.0.The probe layers were 18, 24, and 16, respectively; each autoencoder had width 4d and retained 32 active features.Source: Brain-Scope: sparse autoencoders and… · pp. 15–16
  • A parcel-wise linear readout maps sparse features to 1,000 fMRI outputs, with regularization selected per parcel by leave-one-out grid search.Feature activations are pooled over each TR using mean, maximum, or firing rate.Source: Brain-Scope: sparse autoencoders and… · p. 16
  • Brain-aligned features are selected by significant network correlations and selectivity margin, yielding sets of 185, 184, and 190 features for Qwen3-Omni, Ming-flash-omni-2.0, and MiniCPM-o 4.5.For the matched ablation, feature selection uses mean-pooled alignment; the network-level aggregation of readout predictions supplies the Brain-MoE gate's online prior.Source: Brain-Scope: sparse autoencoders and… · p. 16

Cross-model feature matching and exact permutation

  • Thirty random control sets of equal size are matched with the same algorithm, and no control draw reaches the observed value in any model pair.Controls are drawn from remaining features and preserve the pooling-scheme mix of the brain-aligned set.Source: Brain-Scope: sparse autoencoders and… · p. 16, Cross-model feature matching and exact… · p. 16
  • For each model pair, the Hungarian algorithm finds a one-to-one matching between brain-aligned features by maximizing absolute activation-series correlation on shared stimulus segments.The comparison uses 13,383 TR segments from 81 Friends half-episode segments.Source: Cross-model feature matching and exact… · p. 16
  • Significance is assessed by re-solving feature matching across 12,044 circular shifts per model pair, then applying Holm correction across the three pairs.Shifts span 5%–95% of the grid; the identity is included for 12,045 total shifts, while near-aligning shifts are excluded.Source: Cross-model feature matching and exact… · p. 16

Quantifying effective dimension

The analysis quantifies inter-model differences after removing the shared encoding map using the participation ratio of a row-centred residual matrix. It also tests alternative explanations and model-label permutations.

Quantifying effective dimension

  • The seven participant-averaged encoding maps are row-centred after subtracting their shared mean, and the residual matrix’s participation ratio quantifies inter-model differences.With seven models, the residual matrix has rank at most 6; the participation ratio is computed from the eigenvalues of EE⊤.Source: Quantifying effective dimension · p. 17
  • PR = 1 means inter-model differences lie along one direction, while PR = 6 means six directions have equal energy.Here K = 7, the number of models.Source: Quantifying effective dimension · p. 17
  • The analysis recomputes PR after testing four alternative explanations and tests the geometric corollary against 5,040 model-label permutations.The alternatives are global scalar gain per model, measurement noise, parcel resolution, and a signal-to-noise map; a low effective dimension permits injecting the seven-network partition as a whole.Source: Quantifying effective dimension · p. 17

Brain-MoE architecture · Two-stage training

Brain-MoE pairs seven Yeo-network-specific attention experts with a stimulus-derived brain prior. Its two-stage training first learns network-specific experts on Brain-AVQA, then adapts them to each benchmark.

Brain-MoE architecture

  • Seven rank-4 low-rank experts, each matched to one Yeo network, are inserted into attention projections of the frozen base models.MiniCPM-o 4.5 and Qwen3-Omni use q and v projections; Ming-flash-omni-2.0 uses fused QKV projections.Source: Brain-MoE architecture · p. 17
  • The expert injection bands sit just past each model’s half-depth layer, where the model-to-brain analysis found brain-likeness peaking.The bands span layers 20–28 for MiniCPM-o 4.5 and Ming-flash-omni-2.0, and 28–36 for Qwen3-Omni.Source: Brain-MoE architecture · p. 17
  • A read-only probe maps stimulus hidden states to predicted parcel responses and aggregates them across the Yeo-7 networks to form the brain prior.The sparse autoencoder and readout head are fitted on fMRI and frozen; the prior uses no recording at stage two or inference, and its path propagates no gradients.Source: Brain-MoE architecture · p. 17

Two-stage training

  • In stage one, each expert is trained on Brain-AVQA questions labelled with its dominant network, with a one-hot gate routing residual correction through that expert alone.Training uses a REINFORCE variant with four sampled answers per question, rewarded by correctness and confidence, plus a supervised anchor and routing-alignment term.Source: Two-stage training · pp. 17–18
  • Each network expert trains for one epoch, followed by one epoch of gate pretraining; the merged checkpoint retains frozen down-projections as the reused brain-feature subspace.Source: Two-stage training · p. 18
  • Stage two freezes the down-projections and trains only the up-projections and gate with cross-entropy plus KL divergence to the base distribution.Training runs for three epochs with one question per forward pass and gradient accumulation over four questions.Source: Two-stage training · p. 18

Control arms and factorial design

The design compares base, random-expert, and brain-expert arms, then isolates expert initialization and routing-prior effects through topology controls and a four-arm factorial design.

Control arms

  • Each base–benchmark cell compares base, randomly initialized experts with a shuffled map, and stage-one brain experts using the real network-to-expert correspondence.The random experts share the brain experts’ shape and stage-two architecture and training; stage-one expert training is not used for them.Source: Control arms and factorial design · p. 18

Topology controls

  • Topology controls compare the real prior with an adjacent-network rotation and a uniform zero prior, keeping the seven prior components’ multiset unchanged under rotation.The uniform prior removes prior information while retaining the prior term; the rotation assigns each expert the activation of an adjacently numbered network.Source: Control arms and factorial design · p. 18
  • The random-map control tests 20 sampled network permutations, excludes the identity and a cyclic shift, and estimates P as (1 + m)/21, with minimum 0.048.Expert parameters, steps, and compute are held identical across operators; gate parameters are reinitialized, while only expert LoRA parameters carry over from stage one.Source: Control arms and factorial design · p. 18

Factorial design

  • The four-arm factorial crosses brain versus random expert initialization with real versus shuffled routing priors to estimate content, prior, and interaction effects.The saturated model’s parameters are derived from the four arm accuracies; evaluation uses Brain-AVQA and five public benchmarks, with Brain-AVQA also providing stage-one supervision.Source: Control arms and factorial design · p. 18

Brain-AVQA benchmark · Public benchmarks and held-out splits · Matched ablation

This merged section defines the Brain-AVQA benchmark and held-out evaluation protocols, then specifies a matched feature ablation to test whether Brain-Scope prior features are necessary.

Brain-AVQA benchmark

  • Brain-AVQA labels each question with the Yeo-7 network most engaged during its source 8-s viewing window and verifies generated questions through independent audiovisual audits.Construction also includes translation audits and option-position permutation; the label characterizes the stimulus window, not the answering process.Source: Brain-AVQA benchmark · p. 19
  • The final Brain-AVQA benchmark contains 2,362 questions, including 2,096 requiring joint audiovisual answering, and 162 held-out consensus-hard questions.Consensus-hard questions are those answered correctly by at most one of the three frozen bases; the held-out set also includes 63 questions missed by all three.Source: Brain-AVQA benchmark · p. 19
  • Stratified question-level sampling held out 30% per dominant-network and consensus-difficulty stratum, yielding 1,653 training and 709 held-out questions.The held-out questions were excluded from training, with no detectable split differences in base accuracy (Mann-Whitney U tests P = 0.44, 0.47, 0.53) or consensus-difficulty composition (χ2 test P = 0.525).Source: Brain-AVQA benchmark · p. 19
  • Network labels showed 0.353 agreement between disjoint participant pairs (Cohen κ = 0.243 against chance 0.143), and fMRI-derived labels reproduced released values at r = 0.9988.Stage-one experts use all 2,362 questions; held-out refers to stage-two up-projections and gate, while topology analyses are in-domain controls and public benchmarks test generalization.Source: Brain-AVQA benchmark · p. 19

Public benchmarks and held-out splits

  • Generalization uses adapted multiple-choice subsets from five public benchmarks, with media-key splits and identical held-out sets shared across the three bases and two arms.The adaptations include two-to-four-option MMAU-Pro items and four-choice MUSIC-AVQA items, so results do not represent the complete native protocols; multi-audio items use only the first file.Source: Public benchmarks and held-out splits · p. 19
  • Each base-by-benchmark-by-arm cell is an independent stage-two run trained on half the data for 3 epochs with random seed 0, then evaluated on held-out media.These cell-wise gains feed the matched ablation and headroom analyses.Source: Public benchmarks and held-out splits · p. 20

Matched ablation

  • For each network, the matched ablation compares its top 24 aligned SAE features with an equal-sized, zero-overlap control set matched on firing rate and activation magnitude.Brain-set unions contain 111 features for Ming-flash-omni-2.0, 115 for MiniCPM-o 4.5, and 98 for Qwen3-Omni; zeroing subtracts each feature's decoder contribution from the hidden state.Source: Matched ablation · p. 20
  • Ablation effects are assessed as parcel-wise encoding damage and excess damage, plus held-out answer-accuracy excess on public benchmarks using paired exact tests.Encoding is re-evaluated after feature zeroing with a ridge model fitted on intact hidden states; capability tests pool five benchmarks (n = 3,868) and also report results per benchmark.Source: Matched ablation · p. 20

Headroom and routing analyses · Use of large language models

The section describes how headroom and routing analyses were conducted and documents the use of language models in constructing and checking Brain-AVQA. It states that language models did not generate the core research reasoning or analyses.

Headroom and routing analyses

  • Headroom–effect-size relationships were assessed with Spearman correlations across 15 cells, using headroom η_headroom = 1 − acc(base).The analysis considered both held-out effect size and evidence strength (−log10 P).Source: Headroom and routing analyses · p. 20
  • Headroom and held-out size were collinear at benchmark level (Spearman ρ = +0.895, n = 5), so effect size and evidence strength were jointly regressed on standardized headroom and log10 n.The joint least-squares regression used z-standardized headroom and log10 n.Source: Headroom and routing analyses · p. 20
  • Routing hit rate measured the fraction of held-out questions whose highest-weighted expert matched the annotated dominant network, against an independence-expectation baseline.The baseline was Σ_c P(route = c)P(dominant = c).Source: Headroom and routing analyses · p. 20
  • Routing was evaluated on independent episode-grouped (n = 714) and media-grouped (n = 731) splits, with significance tested by 5,000 label permutations preserving both marginals.Experts were trained on the training side of each split; the two routing splits were independent of the primary held-out set.Source: Headroom and routing analyses · p. 20

Use of large language models

  • Qwen3.5-Omni-Plus40 generated Brain-AVQA questions, Qwen3-Max58 translated occasional Chinese items, and Gemini 3.5 Flash41 audited questions on audio and video.Questions were retained only if they had a unique correct answer derivable from the window without textual shortcuts; the independent audit was repeated after translation.Source: Use of large language models · p. 20
  • The authors reviewed the audited benchmark and take responsibility for its content; language models also assisted bibliographic verification and editorial corrections, but not core research reasoning or analyses.The review included per-network examples and difficulty labels.Source: Use of large language models · pp. 20–21

Statistical analysis

The statistical analysis reports no significant correlations across the seven-model panel after Holm correction and uses paired tests and permutation procedures for comparisons. It also documents the study’s sampling, replication, exclusions, and code and data availability.

Statistical analysis

  • The study did not predetermine sample size; it reports analysis-specific n, excludes only unparseable or unavailable benchmark items and failed energy-matched controls, and uses seeded splits and single-seed matrix cells.Same-seed reruns reproduced results bit-for-bit, with additional seed repetitions for MiniCPM-o 4.5; generalization is assessed across three bases and five benchmarks. No human allocation or rating required randomization or blinding, and investigators were not blind to arm identity.Source: Statistical analysis · p. 21
  • After Holm correction across six correlations, the last-to-peak ratio’s raw P = 0.048 became P = 0.29, leaving no significant correlation on the seven-model panel.The nominal seven models are not seven independent units: two pairs share a model family or differ only in post-training, so the effective number of independent units is smaller than seven.Source: Statistical analysis · p. 21
  • Same-question arm comparisons use paired exact McNemar tests, with Holm stepdown correction for test families and exhaustive or Monte Carlo permutation tests as appropriate.When the permutation space is enumerable, all permutations are used; otherwise P values use Monte Carlo sampling. Paired-difference confidence intervals use the binomial distribution, and per-question bootstrap intervals resample questions.Source: Statistical analysis · pp. 21–22
  • Pooled inference across the three bases uses a question-level joint sign-flip permutation, and effect sizes are reported per benchmark-by-model cell.Averaging across benchmarks reverses the association between base accuracy and gain: cell-level Spearman ρ = −0.81 versus base-level ρ = +1.00.Source: Statistical analysis · p. 21

Code availability.

  • The Zenodo deposit provides the Brain-AVQA benchmark, analysis materials, data, and archived code, with CC BY 4.0 for data and annotations and MIT for code.The deposit also includes encoding maps, Brain-Scope checkpoints, readout heads, predictions, and figure files.Source: Data availability · p. 22, Code availability · p. 22

Funding.

L.L. reports support from two Chinese research funding bodies. The passage also describes the authors’ contributions and discloses Alibaba affiliations and competing-interest status.

Funding.

  • L.L. reports research support from the National Natural Science Foundation of China (grant 62471420) and the GuangDong Basic and Applied Basic Research Foundation (grant 2025A1515012296).Source: Funding · p. 22
  • P.Z. led study design, methods, experiments, analysis, and manuscript writing with L.L., who also supervised the project.B.T. and X.L. provided model infrastructure and contributed to omni-model evaluation; all authors provided manuscript input.Source: Author contributions · p. 22

Ethics declaration.

  • B.T. and X.L. are Alibaba Group employees, P.Z. completed the work during an Alibaba Token Hub internship, and L.L. declares no competing interests.Source: Competing interests · p. 22

Extended Data

Extended Data examines the robustness and scope of brain-likeness, Brain-MoE accuracy gains, and brain-aligned feature contributions to brain prediction. Analyses vary modality, depth, routing, and seed, and test network-to-expert map specificity, content-pathway effects, and feature-ablation damage.

Forward spectrum

  • Brain-likeness rankings were stable across participants (Spearman ρ = 0.958) and led all five columns of the post-challenge Algonauts 2025 leaderboard.Six of seven models peaked at half depth and none at the last layer; audio alone exceeded text alone on all three brain-to-model bases.Source: Extended Data · p. 26, Forward spectrum · p. 30

Public-benchmark matrix and headroom dependence

  • The stage-two held-out result was +3.24 percentage points, with positive gains in all 15 of 15 model-benchmark pairs across three bases and five public benchmarks.Source: Extended Data · pp. 26, 29

Extended Data

  • Real brain topology beat uniform routing on all three models, while zeroing brain-aligned features caused excess encoding damage on all three bases.In the in-domain topology control, real routing exceeded both shuffled topologies, which were indistinguishable; zeroing the 98-feature brain set caused +9.11 pp excess damage over matched control on Qwen3-Omni OmniBench.Source: Extended Data · pp. 26–27

Brain-AVQA held-out and in-domain results

  • On 709 held-out questions, MiniCPM-o 4.5 gained +3.24 pp, rising to +9.26 pp on consensus-hard questions.In-domain consensus-hard gains were +12.69 pp, +12.88 pp and +17.42 pp for Qwen3-Omni, Ming-flash-omni-2.0 and MiniCPM-o 4.5, respectively. On held-out AVQA[1], accuracy rose from 0.8778 to 0.8968; training on the same questions increased gains across the three difficulty strata.Source: Brain-AVQA held-out and in-domain… · pp. 30–31

Repeated training seeds and random network-to-expert maps

  • Brain-MoE's brain-versus-random gain was positive in all 12 repeated MiniCPM-o 4.5 seed runs, with nine differences reaching P < 0.05.A second stage-one seed gave ∆2 = +6.13 pp on OmniBench versus +6.83 pp with the first seed; the brain-arm change was 0.70 pp (P = 0.50).Source: Repeated training seeds and random… · pp. 31–32
  • In-domain MiniCPM-o 4.5 routing with the real network-to-expert map beat all 20 random maps, by 1.70 pp over their mean and 0.93 pp over the best.The empirical P was 0.048, the smallest value attainable with 20 draws; the real map's routing hit rate was 0.3400 versus at most 0.2202 for a random map.Source: Repeated training seeds and random… · p. 32

Pathways and feature ablation

  • Content pathways sum exactly to total effects, while feature ablations show significant excess encoding damage under reference and strict criteria.Content-pathway effects were +5.95 pp (MiniCPM-o 4.5 × MMAU-Pro), +7.31 pp (Ming-flash-omni-2.0 × MMAU-Pro), +3.22 pp (Qwen3-Omni × MMAU-Pro), +4.55 pp (MiniCPM-o 4.5 × OmniBench) and +1.93 pp (Qwen3-Omni × OmniBench); four-arm content effects were +1.40 to +7.18 pp under the real prior and +1.93 to +5.95 pp under the shuffled prior, with +1.40 pp exceeding every prior effect except +2.28 pp for MiniCPM-o 4.5 × OmniBench. Excess encoding damage was +0.00139, +0.00324 and +0.00891 under the reference criterion (P = 0.0022, 8.4 × 10−39 and 3.1 × 10−73), and +0.01054 under the strict Qwen3-Omni criterion (P = 2.1 × 10−40); Qwen3-Omni × OmniBench downstream excess damage had Holm-corrected P = 3.7 × 10−6, zeroing the matched control lowered accuracy to 0.6077, and pooled discordant pairs were 299/232.Source: Pathways and feature ablation · p. 32

Supplementary Methods

The supplementary methods specify the encoding, training, benchmark-construction, and evaluation procedures behind the paper’s findings, while documenting limitations in the Brain-AVQA labels and modality features. They also report the post-challenge leaderboard result and implementation details for the brain-informed experiments.

Algonauts 2025 submission system

  • The submission pool combined 24 per-layer, lag-unrolled ridge encoding models from internal omni-model states, with film-held-out selection and no external encoder features.Closed-form parcel mixing, residual readouts, hierarchical routing, and cross-participant transfer were used; deployment scales were bounded using leaderboard feedback, not training.Source: Algonauts 2025 submission system · p. 32, Algonauts 2025 evaluation · pp. 34–35

Stage-one GDPO-inspired objective

  • Stage one trained network-specific experts with a GDPO-inspired REINFORCE objective, then merged their frozen down-projections as the brain-feature subspace reused in later experiments [10, 11].A preliminary comparison motivated frozen-A training but was qualitative and is not reported as a result; routing used seven experts with an approximately uniform initial gate.Source: Stage-one GDPO-inspired objective · pp. 32–33, Brain-MoE gate settings and… · p. 35

Brain-AVQA construction

  • Brain-AVQA labels each four-choice question with the Yeo-7 network most engaged during its source 8-s stimulus window, then audits audiovisual answerability and balances answer positions.Window labels were based on participant-averaged, standardized network responses; translated questions were audited again, and correct options were balanced across positions (591, 591, 590, and 590).Source: Brain-AVQA construction · p. 33

Alternative explanations of the effective dimension of inter-model differences

  • Across five treatments addressing four alternatives, participation ratio stayed 1.17–1.33; the first component loaded 5.37 times more on sensory than association parcels, and capability-brain coupling correlated |ρ| = 0.606 versus null median 0.602.Source: Alternative explanations of the… · p. 33

Algonauts 2025 evaluation

  • The pooled encoding system ranked first in all five leaderboard columns, with participant-mean correlation 0.25034, 0.00528 ahead of the runner-up ensemble [15, 16].The 4 September 2026 snapshot covered 24 entries; evaluation used held-out out-of-distribution films with true responses withheld [17].Source: Algonauts 2025 evaluation · p. 35

Qwen3-Omni modality feature extraction

  • A hidden dimension in Qwen3-Omni audio-only and text-only features explained 90–96% of variance in some conditions, making audio-share estimates lower bounds and text-subtracted contrasts upper bounds.The whole-cortex score was 0.365 versus 0.347 for the depth-matched spectrum pass; no control without that dimension was available.Source: Qwen3-Omni modality feature extraction · p. 35

Reliability of the Brain-AVQA network labels

  • Dominant-network labels showed group-level agreement of 0.353 (κ = 0.243), not reliable window-by-window anatomy; overlap stratification found no evidence that segment memorization explained the Brain-AVQA gain.All four participants agreed on 6.39% of windows; the overlap-control contrasts were −0.59 pp for Qwen3-Omni (P = 1.00) and −1.59 pp for Ming-flash-omni-2.0 (P = 0.84).Source: Reliability of the Brain-AVQA network… · p. 36, Window-overlap control for Brain-AVQA · p. 36

Energy-matching criteria

Supplementary Tables

The supplementary tables test the low-dimensional structure of inter-model differences, the reliability of Brain-AVQA network labels, and how label reliability relates to routing performance. They also document modality-ablation contrasts and characterize the brain-likeness analyses and this work’s design.

Supplementary Tables

  • The brain-likeness summary uses mean validation Pearson correlation across four participants and 1,000 parcels, and identifies Qwen3-Omni-30B-A3B-Instruct as the Brain-Scope and Brain-MoE base.Qwen2.5-Omni 7B’s peak depth is 0.49, corresponding to its 2/4 grid layer; its 2/4 and 3/4 scores differ by 0.0002.Source: Supplementary Tables · p. 37
  • Every adjacent modality-subset contrast excluded zero in all three models except audio versus video-plus-text, which was separated only in Ming-flash-omni-2.0.For that exception, the confidence intervals were +0.0130 to +0.0215 for Ming-flash-omni-2.0, −0.0036 to +0.0055 for MiniCPM-o 4.5, and −0.0077 to +0.0002 for Qwen3-Omni.Source: Supplementary Tables · p. 37
  • The comparison table describes this work as bidirectional, using native omni models with a Yeo-7 partition and sparse-feature readout prior, and validating on five public benchmarks and Brain-AVQA.The table characterizes the prior as a frozen brain-learned subspace.Source: Supplementary Tables · p. 39
  • Gain tracks headroom (ρ = 0.80) and low-dimensional inter-model differences (PR = 1.28).The raw residual matrix has participation ratio 1.28; alternative treatments yield 1.17–1.33, while independent Gaussian noise has PR 5.96 (95% interval 5.93 to 5.98), with none of 2,000 simulations below the observed value.Source: Supplementary Tables · pp. 37, 39
  • Mean Brain-AVQA dominant-network label agreement across three participant-pair splits was 0.3533, versus 0.1464 for the permutation null.The observed agreement’s bootstrap interval did not overlap the null interval (z = 28.3 to 29.4), indicating reliability is a group-level property rather than a single-participant property.Source: Supplementary Tables · p. 39
  • Routing hit rates were 0.631, 0.450, and 0.270 in high-, middle-, and low-reliability strata, while accuracy gains were +0.0154, +0.0163, and −0.0009, respectively.The high and middle strata could not be separated from each other; the low-stratum gain’s interval included zero.Source: Supplementary Tables · p. 40

Related papers

Loading 2609.10947v1…