Source-linked AI summary
Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks
Rufin VanRullen, Leila Reddy
TL;DR
The paper addresses the difficulty of distinguishing visually similar faces from fMRI patterns. It maps brain activity into a VAE-GAN face latent space and reconstructs faces, achieving reliable face identification while also decoding gender and imagined faces. The authors propose that deep generative-model latent spaces may approximate human face representations, although this remains a hypothesis.
Problem
Distinguishing visually similar inputs, including different human faces, remains difficult for fMRI pattern decoding.
Method
A pretrained VAE-GAN encoded faces into 1024-dimensional latent vectors, and linear regression mapped fMRI responses to those vectors for reconstruction and feature decoding.
Results
The method reliably distinguished seen faces, decoded gender, and decoded imagined faces, with 95.5% pairwise face classification against PCA’s 87.5%.
Takeaways & Limitations
Deep generative-model latent spaces provided a useful basis for probing face perception, including facial features and mental imagery.
Takeaways & Limitations
The proposed similarity between the VAE-GAN latent space and human face representations remains a hypothesis, and the method’s performance may also reflect latent-space robustness to mapping errors.
Abstract
from arXiv · showhide
While objects from different categories can be reliably decoded from fMRI brain response patterns, it has proved more difficult to distinguish visually similar inputs, such as different instances of the same category. Here, we apply a recently developed deep learning system to the reconstruction of face images from human fMRI patterns. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised training procedure over a large dataset of celebrity faces. The auto-encoder latent space provides a meaningful, topologically organized 1024-dimensional description of each image. We then presented several thousand face images to human subjects, and learned a simple linear mapping between the multi-voxel fMRI activation patterns and the 1024 latent dimensions. Finally, we applied this mapping to novel test images, turning the obtained fMRI patterns into VAE latent codes, and ultimately the codes into face reconstructions. Qualitative and quantitative evaluation of the reconstructions revealed robust pairwise decoding (>95% correct), and a strong improvement relative to a baseline model (PCA decomposition). Furthermore, this brain decoding model can readily be recycled to probe human face perception along many dimensions of interest; for example, the technique allowed for accurate gender classification, and even to decode which face was imagined, rather than seen by the subject. We hypothesize that the latent space of modern deep learning generative models could serve as a valid approximation for human brain representations.
Results
The VAE-GAN brain decoder reconstructed and identified unseen faces more accurately than PCA, with decoding supported mainly by occipital and temporal voxels. The learned mapping also supported gender classification and analyses of facial feature representations.
- Face reconstruction: 95.5% pairwise classification was achieved by the VAE-GAN decoder, compared with 87.5% for PCA.The VAE-GAN result was above 50% chance, and the model difference was significant.
- Face reconstruction: 65% full recognition was achieved for VAE-GAN reconstructions among 20 test faces, compared with 41.25% for PCA.Chance performance for identifying the exact target among 20 faces was 5%.
- Brain-region contributions: Occipital voxels provided the strongest decoding, followed by temporal voxels, whereas frontoparietal voxels alone generally failed to support reliable classification.For VAE-GAN, pairwise performance was 91.8% in occipital, 78.8% in temporal, and 60.1% in frontoparietal voxels.
- Applications: The learned mapping can visualize facial feature selectivity for individual voxels or regions of interest by sending their latent vectors through the face Generator.The resulting images provide a direct face-based representation of voxel or ROI selectivity.
- Feature decoding: The voxel-to-latent mapping supported gender classification at 70% accuracy, with informative correlations concentrated in occipital and temporal regions.Gender decoding was accurate from occipital and temporal voxels but not frontoparietal voxels; corresponding correlations involved early visual areas and fusiform cortex.
Discussion
The study argues that deep generative latent spaces provide a better image space for linear brain decoding, supporting face reconstruction, gender decoding, and imagined-face decoding. These results reinforce the hypothesis that generative-model latent spaces may approximate human brain representations and enable further studies of face processing.
- VAE-GAN decoding produced qualitatively and quantitatively superior face reconstructions compared with PCA, while reliably distinguishing individual faces and decoding gender.The authors attribute the performance partly to the latent space’s organization of nearby points as visually similar and plausible faces.
- The model also decoded faces that subjects imagined rather than saw, extending brain decoding beyond directly presented sensory inputs.
- The work is situated among concurrent generative-model approaches to reconstructing natural scenes, geometric shapes, and faces from brain activity.A closely related face-reconstruction study reported 46.7% structural similarity, whereas this study reported 50.5% for VAE-GAN and about 45.8% for PCA.
- The authors propose that deep generative-model latent spaces may be homologous to, or approximate, human brain representations of faces.
- The framework could support studies of facial-feature selectivity, socially relevant features, attention, memory, and mental imagery.The paper highlights a regional distinction: occipital voxels contributed strongly to perceived-face decoding, whereas temporal voxels contributed to both perceived and imagined faces.
Methods
The methods combine a pretrained VAE-GAN face representation with fMRI recordings from repeated face presentations and a linear mapping between latent codes and voxel activity. Novel test-face activity was inverted through this mapping, and reconstructions were evaluated against PCA and chance-based statistical baselines.
- The VAE-GAN was trained for 15 epochs on 202,599 CelebA celebrity faces, producing 1024-dimensional latent codes that a generator converted into plausible face images.The encoder reconstructed input faces, the generator synthesized faces from latent vectors, and the discriminator distinguished real from generated images during training.
- PCA provided a linear baseline by encoding faces into the first 1024 principal components and reconstructing them with the inverse transform.
- Four subjects underwent eight scan sessions with repeated face presentations while fMRI data were slice-time corrected, realigned, coregistered, and modeled with GLMs.The study used a 3T scanner and presented face, fixation, test-face, one-back, and imagery conditions.
- More than 8,000 training faces per subject supplied latent dimensions as parametric regressors for estimating a 1025-by-nvoxels weight matrix.The model included a face-versus-fixation bias term and assumed a linear mapping from latent vectors to brain activation patterns.
- Testing estimated each novel face’s latent vector by multiplying its brain activity pattern by the transposed learned weights and inverse covariance matrix.Twenty unseen faces were each presented approximately 52.8 times on average to improve signal-to-noise ratio.
- Reconstructions were compared using full and pairwise recognition, PCA contrasts, perceptual judgments, gender decoding, and imagery decoding with nonparametric or binomial tests.The analyses included Friedman tests for model and voxel-selection comparisons and chance tests for recognition, gender, imagery, and perceptual judgments.
Supplementary Materials
The supplementary materials document the PCA comparison, voxel-selection procedure, dataset and model availability, regional variance contributions, gender-selective voxel mapping, and imagery-reconstruction displays. Together, these materials detail how the supplementary analyses operationalize and extend the main decoding results.
- The full fMRI datasets, subject-specific decoding models, and pretrained VAE-GAN code are publicly available through OpenNeuro and GitHub.
- The PCA supplementary model encodes faces as linear combinations of pixel values using the first 1024 principal components and decodes them with inverse PCA.
- Voxel selection combined face responsiveness with improvement in adjusted residual variance when latent face features were included in the model.The selection boundary included voxels with t-values of 4 or greater or more than 8% adjusted residual-variance improvement.
- Decoding accuracy was plotted as a function of training-set size, comparing approximately 1K, 2K, 4K, and 8K faces per subject.
- Variance partitioning separated the unique and jointly explained latent variance contributed by occipital, temporal, and frontoparietal regions.
- Gender-selective voxels were mapped by correlating a male-minus-female latent vector with columns of the brain-decoding matrix.
- Imagery reconstructions were displayed alongside 20 candidate faces ranked by latent-vector similarity, with the imagined face highlighted and assigned a rank.