PaperlayerView exact revision on arXiv

Source-linked AI summary

The Platonic brain bridge hypothesis: human brain networks as an architectural prior for omni models

Pengfei Zhang, Biao Tian, Xiangang Li, Li Liu

arXiv:2609.10947v1q-bio.NCcs.LG

TL;DR

The paper asks whether brain organization can be built into omni models as an architectural prior. It introduces brain-network experts, network-based question labeling, and sparse-feature localization, finding bidirectional model–brain correspondence and improved model performance.

  • Problem

    The paper asks whether the organization of the brain can be built into omni models as an injectable architectural prior.

  • Method

    The study maps seven brain networks to brain-pretrained experts, uses dominant-network labels for video questions, and links brain encoding, routing, and ablation through sparse features.

  • Results

    Brain-MoE improves three heterogeneous model bases across five public benchmarks, while the real network-to-expert map gains more in the in-domain topology control.

  • Takeaways & Limitations

    The results support a bidirectional correspondence in which brain organization can serve as a controllable architectural prior for omni models.

  • Takeaways & Limitations

    Brain-AVQA network labels are group-level and do not support a window-by-window anatomical reading.

Abstract

from arXiv · show

We propose the Platonic brain bridge hypothesis: omni models, which process video, audio and text jointly like the brain, converge on brain-like representations, and the correspondence is bidirectional. From model to brain, brain-likeness of seven omni models is stable across participants, and our encoding models on their internal hidden states rank first on the Algonauts 2025 out-of-distribution leaderboard. From brain to model, three contributions follow. Brain-MoE gives seven cortical networks one brain-pretrained expert each and raises held-out accuracy in all 15 model-benchmark pairs by 6.42 percentage points on average. Brain-AVQA builds questions from video clips labelled by the most responsive brain network; the real network-to-expert map exceeds shuffled maps in-domain on all three models. Brain-Scope uses sparse autoencoders to localize the correspondence to a small subset whose removal weakens brain prediction in all three bases tested. Human brain networks are therefore a usable architectural prior for omni models.

Results

Across seven omni models, brain correspondence is stable across participants, structured by modality and depth, and localized to shared sparse features. Brain-MoE then uses this organization as a prior, improving held-out and in-domain performance, selecting matching experts, and transferring gains to unseen media.

  • Omni models correspond to human cortical responses on a stable, structured spectrum: 0.337–0.381 brain-likeness scores formed a narrow spectrum, with participant rankings nearly identical (mean pairwise Spearman ρ = 0.958).The measurement pool also ranked first in all five columns of the Algonauts 2025 out-of-distribution leaderboard.
  • Omni models correspond to human cortical responses on a stable, structured spectrum: 0.250, 0.244 and 0.246 text-only scores rose to 0.381, 0.361 and 0.365 with all modalities for Ming, MiniCPM and Qwen3.Every added channel increased brain-likeness, while no model peaked at its last layer.
  • Inter-model differences are confined to one shared axis: 97.87% of encoding-map variance was shared across models, while residual differences had effective dimension 1.28 and loaded 5.37 times more strongly on sensory than association parcels.The low-dimensional structure persisted under five controls, but capability correlations were not significant in this seven-model panel.
  • Brain-MoE held-out and in-domain results: Brain-MoE improved MiniCPM accuracy by +3.24 pp on 709 held-out Brain-AVQA questions and raised in-domain accuracy by +3.60, +4.87 and +7.54 pp on Qwen3, Ming and MiniCPM.The held-out gain was larger on questions the frozen bases answered poorly.
  • Brain-pretrained experts improve unseen-question accuracy across bases and benchmarks: +0.56 to +11.98 pp gains averaged +6.42 pp across all 15 public benchmark cells, with the brain arm exceeding random experts in 14 cells.The real network-to-expert map also exceeded rotated and uniform maps on all three bases and selected matching experts on unseen media.

Discussion

The paper presents brain–model correspondence as usable in both directions, while identifying its components, controls, and remaining scope boundaries. Brain-Scope links brain encoding, routing, and ablation, but the prior is tested only at the seven-network resolution and broader validation remains open.

  • Discussion: Brain-MoE improves three heterogeneous bases across five public benchmarks, with controls attributing gains to brain-pretrained expert content and the real network-to-expert map.The injected object is an organizational principle in a frozen feature subspace, not brain data, and validation uses benchmarks unrelated to the recordings.
  • Discussion: The prior is defined at the resolution of seven Yeo networks; finer parcellations and other atlases remain untested.This is an explicit scope boundary for the architectural prior.
  • Discussion: Brain-Scope reuses one feature set to predict brain responses, steer routing, and ablate features, connecting sparse-feature encoding with public capability tests.This extends prior sparse-feature work by linking brain encoding, expert routing, and capability ablation.
  • Discussion: Following one base through training and adding stimuli, participants, model families, and seeds are proposed to test remaining capability–brain coupling uncertainties.The paper also identifies embodied agents and additional sensory channels as extensions.

Methods

The methods use public naturalistic-viewing fMRI and frozen omni-model hidden states under a common train–validation protocol. Hidden representations are extracted across modalities and depths, reduced to fixed-size features, and evaluated with encoding models.

  • Methods: Four participants watched seasons 1–6 of Friends, with seasons 1–5 for training and season 6 for episode-level validation.The study used publicly released, de-identified CNeuroMod data and collected no new human data.
  • Methods: Seven frozen omni models were evaluated using video, audio, and subtitle inputs under one shared encoding protocol.The panel includes models spanning different parameter scales and architectures.
  • Methods: Hidden states were extracted at four relative depths, averaged over tokens covered by each TR, reduced independently to 512 dimensions per layer, and concatenated into 2,048-dimensional features.PCA was fitted on training data and applied to validation data.

Brain encoding model

The brain encoding model measures how well omni-model hidden states predict cortical responses across modalities and depths. Modality gains are sub-additive, while the strongest correspondence generally occurs before the final layer.

  • Brain encoding model: Brain-likeness is the mean validation Pearson correlation across 1,000 parcels and participants.Parcel predictions are obtained from participant- and parcel-specific ridge readouts.
  • Brain encoding model: Bootstrap intervals are optimistic because parcels are spatially correlated and only one validation split was used.A separate Qwen3-Omni extraction pass also makes some modality values bounds rather than directly comparable estimates.
  • Brain encoding model: Modality and depth analyses locate where correspondence is strongest before Brain-Scope identifies candidate components within each model.Depth analyses compare four relative depths across seven models and relate peak position to model size and capability.

Brain-Scope: sparse autoencoders and brain readout

Brain-Scope uses sparse autoencoders to identify hidden features aligned with the seven cortical networks and read them out to fMRI. These features provide the routing prior used by Brain-MoE and support cross-model feature comparisons.

  • Brain-Scope: sparse autoencoders and brain readout: Brain-Scope is shared by both study directions: its sparse features are read out to fMRI and supply the Brain-MoE routing prior.For each brain-to-model base, a TopK sparse autoencoder is trained on a probe layer’s hidden states.
  • Brain-Scope: sparse autoencoders and brain readout: The autoencoders use TopK activation with width W = 4d, kact = 32, and normalized decoder columns.Training samples hidden states from stimulus TR shards, with 5% of stimulus files held out for validation.
  • Brain-Scope: sparse autoencoders and brain readout: Feature activations are pooled over TR tokens using mean, maximum, or firing rate before network alignment and brain readout.The brain readout head is a parcel-wise linear model on sparse features.
  • Brain-Scope: sparse autoencoders and brain readout: Feature-to-network alignment is the clipped Pearson correlation between pooled feature activations and each network’s mean fMRI response.Significant features are ranked by selectivity margin, with the top 50 retained per network.
  • Brain-Scope: sparse autoencoders and brain readout: Brain-Scope compares brain-aligned features across model pairs by optimal one-to-one matching of activation-series correlations.Circular-shift permutations and random control sets assess whether matched similarity exceeds chance.

Quantifying effective dimension

The seven models’ encoding maps are decomposed after removing their shared map, and the effective dimensionality of residual differences is quantified with participation ratio.

  • Quantifying effective dimension: The residual encoding matrix E is formed by subtracting the seven-model mean map, limiting its rank to six.Eigendecomposition of EEᵀ yields eigenvalues used to measure inter-model differences.
  • Quantifying effective dimension: Participation ratio summarizes how evenly inter-model differences occupy the available dimensions, from one direction to six equal-energy directions.PR = 1 indicates one difference direction, whereas PR = 6 indicates six equally energetic directions.
  • Quantifying effective dimension: Four alternative explanations and 5,040 model-label permutations test whether the low effective dimension is robust to nuisance treatments and geometric chance.The tested treatments include scalar gain, measurement noise, parcel resolution and signal-to-noise structure.

Brain-MoE architecture

Brain-MoE injects seven brain-network-specific low-rank experts into frozen omni models near half depth, using a frozen brain-derived prior to route activation without brain recordings at inference.

  • Brain-MoE architecture: Seven low-rank experts are inserted into selected attention projections of frozen MiniCPM-o 4.5, Ming-flash-omni-2.0 and Qwen3-Omni, one per Yeo-7 network.Injection layers and projection types vary by base model, while each expert corresponds one-to-one with a cortical network.
  • Brain-MoE architecture: Injection begins two to four layers after each model’s half-depth layer, where model-to-brain alignment peaks in all three bases.The bases have 36, 32 and 48 layers respectively.
  • Brain-MoE architecture: The brain prior is computed from stimulus-derived hidden states through a frozen sparse autoencoder and fMRI-trained readout, with no recording used during stage two or inference.The readout predicts 1,000 parcel responses, aggregates them over Yeo-7 masks, and standardizes the network responses.
  • Brain-MoE architecture: Stage one trains each network expert on questions carrying that network’s label, then merges the experts into a checkpoint whose down-projections are reused subsequently.Training uses a one-hot expert route to prevent cross-talk during network-specific pretraining.
  • Brain-MoE architecture: Stage two freezes the brain-derived down-projections and trains only expert up-projections and routing parameters with cross-entropy plus KL divergence to the base distribution.The control arms test whether gains arise from brain pretraining rather than merely added parameters.

Control arms and factorial design

The study separates expert content from routing structure using random, shuffled, rotated and uniform controls, including a factorial model of brain initialization and real-prior effects.

  • Control arms and factorial design: Each base-by-benchmark cell compares the base, random experts with a shuffled map, and brain experts with the real network-to-expert correspondence.The two reported effects are Δ1 = acc(MoE) − acc(base) and Δ2 = acc(brain) − acc(random).
  • Control arms and factorial design: Topology controls retain the seven prior components while testing the real map against adjacent-network rotation and a uniform zero prior.The uniform operator removes prior information, whereas rotation preserves the component multiset but changes assignments.
  • Control arms and factorial design: Random-map significance uses 20 sampled permutations, with the empirical P-value computed as (1 + m)/21 and a minimum of 0.048.Identity and cyclic-shift permutations are excluded from the random draws.
  • Control arms and factorial design: The factorial design crosses brain versus random expert initialization with real versus shuffled routing priors as independent factors.Its four arms are BT, BS, RT and RS, with interaction estimated from their held-out accuracies.
  • Control arms and factorial design: The four-arm design is evaluated on Brain-AVQA and five public benchmarks, separating expert-content, routing-prior and interaction effects.The model assigns μ to the random-expert/shuffled-prior arm and derives the two main effects and interaction from the other arms.

Brain-AVQA benchmark

Brain-AVQA labels audiovisual question windows by their most responsive Yeo-7 network and evaluates brain-guided experts on held-out and downstream omni benchmarks, while Brain-Scope tests feature necessity.

  • Brain-AVQA benchmark: Brain-AVQA assigns each 8-second window its strongest participant-averaged network response, generates a four-choice audiovisual question, and independently audits its dependence on that capability.The label belongs to the stimulus window rather than the answering process.
  • Brain-AVQA benchmark: The benchmark contains 2,362 questions from 81 half-episode segments, with 2,096 requiring joint audiovisual answering and dominant-network labels spanning all seven Yeo networks.The labels include default, visual, somatomotor, limbic, control, dorsal attention and salience/ventral attention networks.
  • Brain-AVQA benchmark: The held-out split reserves 30% of questions within dominant-network and consensus-difficulty strata, yielding 1,653 training and 709 held-out questions without detectable base-accuracy differences.The split is question-level and uses seed 20260612.
  • Brain-AVQA benchmark: Network labels show group-level reliability: disjoint participant pairs agree on 0.353 of windows, with Cohen κ = 0.243 above a chance level of 0.143.Labels re-derived from fMRI reproduce released values with r = 0.9988.
  • Brain-AVQA benchmark: Downstream evaluation uses five public benchmarks covering general omni reasoning, audio and music understanding, and audiovisual question answering, with held-out evaluation after independent stage-two runs.The gains feed the matched ablation and headroom analyses.
  • Brain-AVQA benchmark: Brain-Scope selects the 24 most aligned sparse-autoencoder features per network, forms a deduplicated brain set, and compares its zeroing damage with a matched control set.Damage is measured by re-evaluating an independent ridge encoding model after reconstructing hidden states with selected features zeroed.

Headroom and routing analyses

The analyses test how Brain-MoE gains depend on base-model headroom and whether routing follows annotated dominant brain networks. Routing is evaluated on independent held-out splits against an independence baseline and shuffled labels.

  • Headroom and effect size were related using Spearman correlation across 15 base–benchmark cells, while these analyses were not used to establish the primary gain.Benchmark-level headroom and held-out size were collinear, so effect size and evidence strength were modeled jointly.
  • Routing hit rate measures how often the highest-weighted expert matches the annotated dominant network, with significance assessed by permuting routing labels while preserving both marginals.The independence expectation is P(route = c)P(dominant = c) summed over networks.
  • Routing was tested on episode-grouped and media-grouped splits, including 714 and 731 held-out questions respectively.These splits were independent of the primary 709-question held-out set.
  • Large language models generated Brain-AVQA questions and translated occasional Chinese items, while the core research reasoning and analyses were not generated by language models.Qwen3.5-Omni-Plus generated four-choice questions and Qwen3-Max translated occasional items.

Statistical analysis

The study preregisters neither sample sizes nor allocation to conditions, but reports analysis-specific n values, paired tests, permutation-based corrections, and reproducibility procedures. Its seven-model correlation analyses have limited effective independence and lose significance after Holm correction.

  • Sample sizes were not predetermined, exclusions followed parsing, media-availability, and energy-matching rules, and most main cells used a single seed with targeted replication.The study reports independent replication across three bases and five benchmarks as its generalization unit.
  • No correlation on the seven-model panel remained significant after Holm correction: the smallest raw P = 0.048 became P = 0.29.The seven models are not fully independent because some differ only by post-training or model size.
  • Two-arm same-question comparisons used paired exact McNemar tests, with familywise correction by the Holm stepdown method.The tests use discordant counts in the two directions and are two-sided unless stated otherwise.
  • Permutation inference used exhaustive enumeration when possible, including 5,040 permutations for seven models and 12,045 circular shifts, otherwise using Monte Carlo sampling.Holm-adjusted values were recomputed from raw P values whenever the tested family changed.
  • Pooled comparisons across bases used a question-level joint sign-flip permutation with 100,000 permutations rather than Stouffer or Fisher combination.Effect sizes were reported at cell level because base-level and cell-level correlations between accuracy and gain had opposite signs.
  • Public datasets, benchmarks, the Brain-AVQA benchmark, annotations, and analysis code were released through public repositories or Zenodo under stated licences.The Brain-AVQA release contains 2,362 questions with dominant-network annotations and audit records.

Funding.

The paper reports research support from Chinese national and Guangdong provincial foundations. It also discloses author roles and affiliations involving Alibaba, whose model family is evaluated.

  • L.L. discloses support from the National Natural Science Foundation of China and the GuangDong Basic and Applied Basic Research Foundation.The reported grant numbers are 62471420 and 2025A1515012296.
  • P.Z. and L.L. conceived and wrote the study, while P.Z. led the pipeline, Brain-Scope, Brain-MoE, Brain-AVQA, experiments, and analysis.B.T. and X.L. provided model infrastructure and contributed to evaluation.
  • B.T. and X.L. are Alibaba Group employees, and P.Z. completed the work during an internship at Alibaba Token Hub.Alibaba develops the Qwen model family evaluated in the study.

Extended Data

The extended data formalize the brain-bridge propositions, measurement structure, architectural interventions, controls, and headline results. Across analyses, brain alignment is stable and structured, while brain-derived experts improve held-out performance and real topology outperforms shuffled alternatives.

  • Brain-AVQA topology: The real network-to-expert map exceeded shuffled topologies in-domain on all three models, while the two shuffled maps were indistinguishable.Real-minus-uniform gains were +1.91, +1.14, and +1.27 percentage points across the three bases.
  • Measurement and propositions: Brain-likeness was positive for all seven models, rankings were consistent across participants, and the encoding pipeline ranked first in all five leaderboard columns.The leaderboard snapshot covered 24 submissions; model brain-likeness ranged from 0.337 to 0.381 with Spearman ρ = 0.958 across participants.
  • Brain-MoE: Brain-MoE improved held-out accuracy by +3.24 percentage points on MiniCPM-o 4.5, with positive gains in 15/15 model–benchmark cells.The held-out Brain-AVQA result used 709 questions; a sixth benchmark also rose from 0.8778 to 0.8968.
  • Modality and depth: Audio alone exceeded text alone on all three bases, while audiovisual and subtitle increments were sub-additive; six of seven models peaked at half depth and none at the last layer.Audio-over-text gains were +0.081, +0.085, and +0.095; the mean last-to-peak ratio was 0.947.
  • Generalization and replication: Brain-MoE gains generalized across held-out public audiovisual benchmarks and were positive in all 12 replicated seed–benchmark runs reported for MiniCPM-o 4.5.The replicated Δ2 gains ranged from +1.65 to +6.83 percentage points across four benchmarks.
  • Feature ablation: Removing brain-aligned features weakened brain prediction in all three tested bases, with excess encoding damage of +0.00139, +0.00324, and +0.00891 under the reference criterion.The strict Qwen3-Omni criterion yielded +0.01054 excess damage.

Supplementary Methods

The supplementary methods describe the brain-aligned feature extraction, Brain-AVQA construction, robustness controls, benchmark evaluation, and energy-matched comparisons used to test the brain-to-model correspondence.

  • Encoding protocol: The benchmark uses internal hidden states and per-participant, per-parcel ridge regressions, with Qwen3-Omni, MiniCPM-o 4.5, and Ming-flash-omni-2.0 among the evaluated omni families.The protocol excludes features from encoders external to the listed multimodal models.
  • Brain-MoE design: A preliminary comparison motivated freezing the brain-feature subspace because only frozen-A training allowed brain experts to outperform random experts, but this comparison was not reported as a result.The comparison varied which expert parameters entered the optimizer.
  • Brain-AVQA: Brain-AVQA labels each 8-s stimulus window with the Yeo-7 network having the largest participant-averaged standardized response, then generates four-choice audiovisual questions from that window.The label belongs to the stimulus window rather than an answering process.
  • Evaluation: The public leaderboard submission ranked first in all five Algonauts 2025 columns, reaching 0.25034 participant-mean correlation and exceeding the runner-up by 0.00528.The snapshot covered 24 leaderboard entries.
  • Feature-extraction caveat: The Qwen3-Omni audio-only and text-only features are dominated by one standard-deviation dimension, making their effective dimension about one and limiting interpretation of modality contrasts.That dimension accounts for 90–96% of feature variance; audio-share values become lower bounds and text-subtracted contrasts upper bounds.
  • Matched ablation: The primary matched-ablation criterion excludes visual and control networks because their brain-to-control energy ratios ranged from 1.57 to 7.04, leaving 709 parcels.The reference criterion includes all seven networks, while the strict Qwen3-Omni criterion uses 466 parcels.

Supplementary Tables

The supplementary tables document brain-likeness, modality and depth analyses, sparse-feature matching, ablation decompositions, reliability checks, and comparisons with related work.

  • Brain-likeness: Brain-likeness uses mean validation Pearson correlation over four participants and 1,000 parcels, with best-layer and last-to-peak metrics computed across relative depths.The encoding uses 512 principal components per layer concatenated over four layers and a two-period haemodynamic lag.
  • Modality ablation: The modality table reports whole-cortex brain-likeness for text, video, audio, and their combinations, with paired-bootstrap confidence intervals over 1,000 parcels.Most adjacent contrasts exclude zero, except audio versus video-plus-text in two models.
  • Inter-model geometry: The effective-dimension analysis finds a participation ratio of 1.28 for raw residuals, while matched independent Gaussian noise yields 5.96.The low-dimensional result remains under global-gain, standardization, parcel-resolution, and text-baseline controls.
  • Brain-MoE decomposition: The three-arm decomposition separates total, content, and routing pathways using exact paired McNemar tests and reports model-benchmark-specific effects.The pathway definitions are total effect BT − RS, content pathway BS − RS, and routing pathway BT − BS.
  • Brain-Scope: Matched-ablation excess damage is positive across the three bases, reaching +0.01012 for Qwen3-Omni under the primary criterion.Excess damage compares brain-feature zeroing with an energy-matched control and uses across-parcel Wilcoxon signed-rank tests.
  • Related-work comparison: The comparison table characterizes this work as bidirectional, native-omni, Yeo-7-partitioned, sparse-feature prior injection evaluated on five public benchmarks and Brain-AVQA.Its controls include factorial arms, topology tests, joint permutation, and matched ablation.
  • Brain-AVQA reliability: Dominant-network labels agree across disjoint participant pairs at 0.3533 versus a chance level of 0.1429, while routing accuracy and gains are highest in high- and middle-reliability strata.The all-participant identical rate is 0.0639, and the accuracy gain is absent in the low-reliability stratum.
Loading 2609.10947v1…