Source-linked AI summary
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, Michal Irani
TL;DR
The paper tackles fMRI-to-image reconstruction with too few labeled Image–fMRI pairs to cover natural-image and brain-response variation. It adds Encoder–Decoder self-supervision using unlabeled images and, importantly, unlabeled test-fMRI, improving reconstruction and adapting the decoder to target-data statistics.
Problem
Scarce labeled Image–fMRI pairs limit generalization to new images, because they cannot span the spaces of natural images and fMRI recordings.
Method
An image-to-fMRI Encoder and fMRI-to-image Decoder are concatenated back-to-back to train with unlabeled images and unlabeled test-fMRI.
Results
85.3% average accuracy was achieved on the 2-way identification task with the full method, compared with 80.1% for supervised training-only and 83.2% with unlabeled images.
Takeaways & Limitations
Self-supervision on unlabeled test-fMRI had the highest impact and substantially surpassed self-supervision using unlabeled natural images alone.
Takeaways & Limitations
GAN-based reconstruction methods can produce natural-looking images that are often unfaithful to the images underlying the test-fMRI.
Abstract
from arXiv · showhide
Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.
1 Introduction
The paper addresses poor generalization in fMRI-to-image reconstruction caused by scarce labeled Image–fMRI pairs and proposes self-supervision using unlabeled images and fMRI, including target test-fMRI.
- Limited labeled Image–fMRI pairs cannot span natural-image or fMRI-response spaces, while fMRI’s low resolution and SNR further reduce training reliability.
- Existing reconstruction methods are supervised and therefore prone to poor generalization on fMRI recordings of new images.
- The method trains an image-to-fMRI Encoder and an fMRI-to-image Decoder, concatenating them to enable self-supervision on unlabeled images and unlabeled fMRI.
- Training directly on unlabeled test-fMRI adapts the decoder to target-data statistics despite differences from the training data.
- On two different fMRI datasets, the same architecture achieved competitive reconstruction results, while test-fMRI self-supervision provided the greatest reported performance boost.
2 Method overview
The method trains an image-to-fMRI Encoder first, then fixes it while training an fMRI-to-image Decoder with paired data and two self-supervised objectives on unlabeled images and test-fMRI. These objectives adapt reconstruction to natural-image and test-fMRI statistics.
- Training procedure: The two-phase procedure first trains Encoder E on image-fMRI pairs, then freezes E while jointly training Decoder D with labeled and unlabeled data.The second phase uses paired examples, unlabeled natural images, and unlabeled test-fMRI.
- Encoder training: Encoder E maps images to predicted fMRI responses using a loss combining mean squared error and cosine similarity.The loss is applied against ground-truth fMRI responses, with α selected empirically.
- Decoder objective: Decoder training combines supervised image reconstruction, feature and RGB losses, total-variation regularization, and the two self-supervised losses.The image loss compares RGB values and pretrained VGG19 features while regularizing total variation.
- Unlabeled-image self-supervision: The Encoder-Decoder objective maps unlabeled natural images back to themselves, enabling training beyond the paired image-fMRI dataset.The experiments use 50K external ImageNet images without corresponding fMRI recordings.
- Unlabeled-fMRI self-supervision: The Decoder-Encoder objective maps unlabeled test-fMRI through the Decoder and Encoder, adapting the Decoder to the test cohort used at inference.This objective uses fMRI recordings without corresponding images and targets the statistics of the data being decoded.
- Implementation: The system reconstructs 112x112 RGB or grayscale images with a convolutional upsampling Decoder and trains in about 15 minutes on one Tesla V100 GPU.Inference is performed in real time.
3 Experimental results
The proposed method improves fMRI-to-image reconstruction through supervised and unsupervised training, especially by adapting to unlabeled test-fMRI. It also compares favorably with leading methods on two fMRI datasets.
- 3 Experimental results: The complete method produced more faithful and recognizable shapes, textures, colors, scenes, and objects than supervised training alone.These qualitative comparisons used reconstructions from the test-fMRI cohort.
- Ablation study of the method components: Including unlabeled test-fMRI produced a dramatic improvement, whereas excluding the target test-fMRI caused marked degradation in reconstruction quality.The comparison isolates the contribution of adapting the model to the actual inference data.
- Ablation study of the method components: 80.1%, 83.2%, and 85.3% identification accuracy were obtained with supervised-only, additional unlabeled images, and the full method, respectively.Excluding the target test-fMRI reduced accuracy to 84.1%.
- Comparison with state-of-the-art methods: Compared with and, the method better reconstructed shapes, details, and global layout, with visual and numerical support.Each leading method was evaluated on its relevant dataset.
- Comparison with state-of-the-art methods: 85.3% mean identification accuracy on fMRI on ImageNet exceeded both variants by at least 5% across n = 2, 5, 10.On vim-1, the method reached 70.5% accuracy for n = 2 and outperformed by at least 3% across difficulty levels.
Conclusion
The paper concludes that self-supervision on unlabeled target test-fMRI is especially important when labeled training data are insufficient. It suggests that adapting models to target-data statistics may also aid generalization in other ill-posed learning tasks.
- Conclusion: Self-supervised training on unlabeled test-fMRI had the strongest effect on reconstruction and substantially exceeded training on unlabeled natural images alone.The authors associate both effects with adapting the network to test-data statistics.
- Conclusion: Adapting to target test-data statistics may promote generalization in other ill-posed learning problems with scarce training data.The paper frames this as a possible use beyond fMRI inference.