PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Changde Du, Kaicheng Fu, Jinpeng Li, Huiguang He

arXiv:2210.06756v2cs.CVcs.AIcs.MMcs.NE

TL;DR

Novel-category neural decoding is difficult because training has limited paired brain responses and underuses multimodal semantic knowledge. BraVL jointly models brain, visual, and linguistic features with a multimodal generative latent representation and mutual-information regularization. Experiments on three trimodal datasets report accurate novel-category decoding, stronger performance from combined visual-textual features, and possible linguistic influence on visual semantics.

  • Problem

    Existing neural decoding methods struggle to generalize to novel categories without corresponding neural training data because paired data are limited and multimodal semantic knowledge is underused.

  • Method

    BraVL jointly models brain, visual, and linguistic features using a mixture-of-product-of-experts latent representation, mutual-information regularization, and semi-supervised multimodal data.

  • Results

    BraVL achieves good-accuracy decoding of novel visual categories, while combined visual and linguistic features outperform either modality alone.

  • Takeaways & Limitations

    The results suggest that visual perception may be accompanied by linguistic influences in representing the semantic meaning of visual stimuli.

  • Takeaways & Limitations

    The extracted visual features may not accurately reflect all stages of human visual processing, and larger, more diverse datasets may be beneficial.

Abstract

from arXiv · show

Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli. Code and data: https://github.com/ChangdeDu/BraVL.

1 INTRODUCTION

BraVL addresses zero-shot neural decoding by jointly learning brain, visual, and linguistic representations, motivated by evidence that language and prior experience shape visual representations. It targets novel-category decoding despite limited paired brain data and reports improved accuracy and cognitive insights.

  • Motivation: Zero-shot neural decoding seeks to infer novel visual categories from brain activity using semantic knowledge from seen classes.The task supports investigation of brain mechanisms and provides a technical basis for brain-computer interfaces.
  • Motivation: Existing methods primarily use visual features, while human visual recognition is influenced by prior experience and linguistic knowledge.Dual-coding theory describes concrete concepts as encoded both visually and linguistically.
  • Motivation: Standard class-name word vectors are poorly aligned with visual information, leaving linguistic neural decoding accuracy below practical criteria.The paper therefore motivates richer language representations with stronger visual semantics.
  • BraVL: BraVL jointly models brain activity, image-derived visual semantics, and rich Wikipedia-based textual semantics through a multimodal generative latent representation.Its mixture-of-product-of-experts formulation supports coherent generation across all three modalities.
  • BraVL: Intra- and inter-modality mutual-information regularization improves joint-representation consistency and data efficiency with limited brain activity.The model also incorporates extra visual and textual features under semi-supervised scenarios.
  • Contributions: The study combines visual and linguistic knowledge, develops mutual-information regularizers, constructs three trimodal datasets, and reports cognitive insights about visual neural representations.The contributions include released code and datasets for further research.

2 RELATED WORK

Related work frames novel-class neural decoding as a zero-shot problem and situates BraVL within shared-latent-space, multimodal, and mutual-information learning. The paper combines semantic transfer with visual and linguistic information to address limited brain data.

  • Neural decoding of visual categories: Traditional neural decoding classifiers are restricted to predefined categories, motivating identification-based methods for novel-category decoding.These approaches characterize relationships between brain activity and semantic representations rather than relying only on fixed labels.
  • Zero-shot learning: Zero-shot learning separates seen and novel classes and transfers semantic knowledge to bridge the semantic gap between them.Methods include projections into a shared latent space, the category used by BraVL.
  • Zero-shot learning: Novel-class neural decoding has visual and textual information but no novel-class brain activity during training, making it a zero-shot classification problem.The test objective is to decode novel-class brain activity.
  • Multimodal learning: Multimodal learning combines linguistic representations with visual perceptual information, reflecting cognitive-science accounts of semantic knowledge.Prior work has also decoded linguistic nouns from brain activity using both visual and linguistic information.
  • Mutual information maximization: Mutual information quantifies dependence between variables and can be optimized through variational approximations in high-dimensional representation learning.BraVL applies mutual-information maximization at both intra- and inter-modality levels, with the latter equivalent to multimodal contrastive learning.

3 MULTIMODAL LEARNING OF BRAIN-VISUAL-

The problem setting provides trimodal data for seen classes but only visual and textual data for disjoint novel classes, whose brain activity is reserved for testing. Modalities are represented as feature matrices for brain, visual, and textual observations.

  • Seen and novel classes: Seen-class training data contain brain activity, visual features, textual features, and labels.The seen brain modality consists of fMRI features, while the other modalities represent image and text information.
  • Seen and novel classes: Novel classes provide visual features, textual features, and labels but no brain activity during training.Novel-class brain activity is used only at test time.
  • Seen and novel classes: Seen and novel class-label sets are disjoint, so the decoder must classify brain activity from categories not represented in the seen training labels.The novel categories are known candidate categories rather than arbitrary categories.
  • Feature representation: For modality m in {b, v, t}, the unimodal feature matrix X_m has N_m samples and d_m feature dimensions.The subscripts b, v, and t denote brain, visual, and textual modalities, respectively.

3.2 Brain, image and text preprocessing

BraVL preprocesses brain, image, and text inputs into modality-specific feature representations before neural decoding. Brain preprocessing includes stability selection, while visual and textual inputs use pretrained feature extractors and pooled embeddings.

  • Raw brain, image, and text inputs are converted into feature representations using modality-specific feature extractors.
  • Brain preprocessing: Brain voxel stability is assessed across repeated trials of identical stimuli using mean pairwise Pearson correlation.Voxels with the highest consistency are selected separately within each brain region.
  • Image preprocessing: Images are represented with hierarchical features extracted by the pretrained RepVGG-b3g4 visual network.The extracted features are flattened and normalized before modeling.
  • Text preprocessing: Textual descriptions are encoded with ALBERT and GPT-Neo by averaging token embeddings into sentence representations.
  • Text preprocessing: Long Wikipedia articles are split into partially overlapping 256-token sequences with 50-token overlap before pooled representation construction.

3.3 High-level overview of the proposed BraVL model

BraVL jointly models brain, image, and text modalities in a shared latent space, regularized by intra- and inter-modality mutual information. Novel-class visual and textual latent representations train an SVM whose label space is reused for brain-based decoding.

  • BraVL combines multimodality joint modeling with simultaneous intra- and inter-modality mutual information regularization.
  • Multi-modality joint modeling: The MoPoE-based model maps brain, visual, and textual features into a joint latent representation and decodes modality-specific outputs.
  • Multi-modality joint modeling: Training uses trimodal seen-class data together with bimodal or unimodal novel-class data.
  • Latent space classifier training: An SVM classifier is trained from novel-class visual and textual latent vectors while their encoders remain frozen.
  • Decoding neural representations: Brain latent representations from novel classes are passed to the frozen SVM because MoPoE aligns the three modalities in a shared space.

3.4 Multi-modality joint modeling

BraVL uses a multimodal variational autoencoder to learn a shared latent representation from brain, visual, and textual features. The MoPoE posterior supports joint modeling across complete seen-class trimodal data and visual-textual novel-class data, while MI regularization addresses latent inconsistency and posterior collapse.

  • Variational autoencoders learn scalable unsupervised representations by introducing a latent variable and optimizing a tractable ELBO.
  • Multimodality joint modeling: For neural decoding, the VAE is conditioned on brain activity, visual features, and corresponding textual features as a multimodal model.
  • Multimodality joint modeling: The model uses a MoPoE variational joint posterior over brain, visual, and textual modalities on seen classes and visual-textual modalities on novel classes.
  • Multimodality joint modeling: A high ELBO alone may still yield inconsistent or unhelpful latent representations, including posterior collapse and weak cross-modal generation coherence.

3.5 Mutual information (MI) regularization

BraVL adds intra-modality and inter-modality mutual-information regularization to improve the usefulness, consistency, and coherence of multimodal latent representations. The intra-modality term links the latent code to each modality, while the inter-modality term contrasts related and unrelated multimodal samples.

  • MI regularization maximizes both latent-to-modality and cross-modal mutual information within the multimodal ELBO objective.
  • 3.5.1 Intra-modality MI maximization: Intra-modality regularization encourages high mutual information between joint latent variable z and each brain, visual, or textual modality.It is intended to make effective use of z and alleviate posterior collapse.
  • 3.5.1 Intra-modality MI maximization: The intra-modality MI term is estimated with a variational lower bound using auxiliary distributions Qψm.The bound is tight when Qψm(z|xm) matches p(z|xm).
  • 3.5.2 Inter-modality MI maximization: Inter-modality regularization suppresses modality-specific noise by maximizing mutual information across multiple modalities.It is designed to improve data efficiency when brain activity data are limited.
  • 3.5.2 Inter-modality MI maximization: The inter-modality objective contrasts positive multimodal samples with six kinds of negative samples and thereby forms contrastive learning without extra parameters.
  • 3.5.2 Inter-modality MI maximization: Because exact joint marginal likelihoods are unavailable, ELBO and CUBO estimators approximate the positive and negative terms of the contrastive regularizer.

3.6 Overall objective and training

BraVL optimizes a multimodal objective with intra- and inter-modality mutual-information regularization, using alternating training stages. Its framework can also incorporate extra bimodal or unimodal data when brain activity data are scarce.

  • Training alternates between tightening the intra-modality MI lower bound with respect to Ψ and jointly optimizing Θ and Φ with both MI regularizers.
  • Each iteration samples a minibatch from either the seen-class or novel-class dataset before applying the two training stages.
  • The framework incorporates extra image-text pairs or unimodal image or text data to support joint representation learning despite scarce brain activity data.
  • An SVM maps latent vectors from novel-class visual and textual features to class labels, then decodes brain-derived latent vectors at test time.

4 EXPERIMENTS

Experiments evaluate BraVL on trimodal datasets spanning fMRI and EEG, with analyses of multimodal features, textual representations, mutual-information regularization, extra data, synthetic activity, and feature importance. Across these evaluations, combining visual and textual features improves decoding and supports generalization to novel classes.

  • 4.1 Brain-Visual-Linguistic datasets.: The experiments use GOD-Wiki, DIR-Wiki, and ThingsEEG-Text, with GOD-Wiki providing fMRI recordings from five subjects across seen and novel ImageNet categories.
  • 4.3.1 Does language influence vision?: 60.33% average top-5 accuracy: BraVL with visual and textual features outperforms the 10% chance level and representative methods on GOD-Wiki and DIR-Wiki.
  • 4.3.1 Does language influence vision?: Approximately 6% average Top-1 accuracy improvement occurs for most test classes when textual features are added.
  • 4.3.1 Does language influence vision?: Textual features improve voxel-wise encoding mainly in high-level visual cortex, whereas they are not beneficial or can be harmful in low-level visual cortex.
  • 4.3.2 Are Wiki articles more effective than class names?: 57.04% versus 45.47% top-5 accuracy on GOD-Wiki: ALBERT features from Wikipedia articles outperform class-name features by almost 12%.
  • 4.3.3 Ablation study.: Removing either MI regularizer degrades performance, with a clearer drop on GOD-Wiki, suggesting stronger data-efficiency benefits when only 1200 trimodal training samples are available.
  • 4.3.4 Latent-variable sampling.: MoPoE produces significantly higher decoding accuracy than PoE and MoE, indicating better suitability for the proposed multimodal learning method.
  • 4.3.5 Analyzing the impact of extra data.: Extra unimodal or bimodal data improve decoding accuracy, and larger bimodal datasets of 10k, 20k, and 50k samples produce progressively better performance.

5 DISCUSSION

BraVL is presented as a neural semantic decoding tool that can use multimodal data despite limited brain recordings, while remaining constrained by feature quality, voxel redundancy, and dataset diversity.

  • Data efficiency and applications: The method could support neuroprosthetic devices that read semantic information from human brain activity, although this application is not yet mature.
  • Data efficiency and applications: BraVL can incorporate trimodal, bimodal, and unimodal data to improve neural-decoding generalization when brain activity is available for only limited visual categories.The method aligns modality-specific distributions in a shared latent space, and application inputs are only brain signals.
  • Limitations and future work: Better visual features and larger pretrained language models could improve decoding, while non-visual Wikipedia sentences may obscure useful visual information.
  • Limitations and future work: Voxel stability selection may retain spatially redundant voxels because nearby voxels can have similar activation patterns.
  • Limitations and future work: A larger and more diverse dataset, including additional bimodal or unimodal data, may benefit the study's decoding approach.

6 CONCLUSION

The paper proposes BraVL for brain-visual-linguistic neural decoding and reports that multimodal representations with mixture-of-product-of-experts and mutual-information regularizers perform better than counterparts.

  • 6 CONCLUSION: BraVL models brain-visual-linguistic features using intra- and inter-modality mutual-information regularization principles.
  • 6 CONCLUSION: The authors constructed three trimodal matching datasets for evaluating multimodal neural decoding.
  • 6 CONCLUSION: Visual-linguistic representations using mixture-of-product-of-experts and mutual-information regularizers perform better than their counterparts.
  • 6 CONCLUSION: Experiments suggest that decoding novel visual categories from human brain activity is practically possible with good accuracy.
  • 6 CONCLUSION: The results suggest that visual perception is accompanied by language influences in representing the semantic meanings of visual stimuli.

APPENDIX REPRESENTATIONAL SIMILARITY MEASURES

The appendix measures heterogeneity in ALBERT text features by comparing cosine similarities at token, sentence, and article levels for ImageNet-class descriptions.

  • APPENDIX REPRESENTATIONAL SIMILARITY MEASURES: Cosine similarity is computed between ALBERT embedding pairs at token-level, sentence-level, and article-level representations.Mean±std values are labeled in the title of each panel.
  • APPENDIX REPRESENTATIONAL SIMILARITY MEASURES: Figure 15 presents cosine similarity matrices for Wiki pages associated with ImageNet class n02817516: ‘bearskin’, ‘busby’, and ‘shako’.
  • APPENDIX REPRESENTATIONAL SIMILARITY MEASURES: Table 9 reports cosine similarity measures as mean±std for additional examples.
Loading 2210.06756v2…