Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons

Introduction

• The neuron doctrine treats single cells as building blocks, while population accounts suggest meaningful variables such as face gender are encoded across neurons.Large-scale recordings helped motivate doubts about one-to-one mappings between external features and individual neurons.Source: Introduction · p. 1
• Deep supervised classifiers can form high-dimensional, distributed representations, reinforcing computational accounts in which single-unit tuning is difficult to interpret.These networks can perform strongly on object recognition and predict neural responses to synthesized images.Source: Introduction · p. 2
• Because natural vision relies heavily on untutored statistical learning, self-supervised generative models offer a way to learn interpretable sensory factors without external teaching signals.The β-VAE reconstructs sensory data from a low-dimensional embedding while regularization encourages individual units to encode meaningful variables.Source: Introduction · p. 2

Results

• The study tested 159 macaque AM neurons recorded while animals viewed 2,100 natural face images, a domain with well-studied image statistics and neural responses.The authors note that faces are a small subset of visual objects, although the mechanisms may generalize more broadly in the ventral stream.Source: Results · pp. 2–3
• β-VAE units varied interpretable face factors such as hairstyle, age, face shape, and smile, while individual neurons showed corresponding sensitivities.Examples include neuron 95 responding to hair thickness and neuron 136 responding to smile presence.Source: Results · p. 3
• β-VAE alignment met the theoretical ceiling (p = 0.43), while linear recombinations scored lower for both β-VAE and neural subsets (p < 0.01).The ceiling was estimated by subsampling neural data to match the dimensionality of the β-VAE representation.Source: Results · p. 3
• Networks with higher unsupervised disentanglement ranking had higher alignment with macaque neural data across networks with the same or different β values.This links stronger disentanglement of face factors with their greater expression in single IT neurons.Source: Results · p. 4
• β-VAE alignment was statistically higher than every tested baseline, including supervised, generative, linear, and hand-engineered face models (all p-values < 0.01).The comparison included VGG, AE, VAE, ICA, PCA, a β-VAE encoder-only classifier, and the active appearance model.Source: Results · p. 4
• A separate correlation-ratio measure also favored β-VAE, and different neurons tended to align with different β-VAE units.The authors relate this heterogeneity to the population's capacity to encode varied image features.Source: Results · p. 4
• No single feature of the β-VAE—architecture, training-data distribution, or an isolated objective component—accounted for its neurally aligned units; the design choices worked together.The comparisons used AE, VAE, classifier, VGG, PCA, and ICA baselines.Source: Results · pp. 4–5
• Using as few as twelve neurons, β-VAE latent decoding reconstructed held-out faces more accurately than other models (all p-values < 0.01), with more identifiable images than AE or VAE using twice as many neurons.The β-VAE was optimized for disentangling, whereas the AE was explicitly optimized for reconstruction quality.Source: Results · p. 5

• Single IT neurons’ axes of variation align with semantically meaningful β-VAE latent units, supporting a low-dimensional, interpretable code for facial identity.Source: Discussion · p. 5
• The study reports one-to-one model-unit/neuron correspondences, unlike earlier few-to-one findings [3], and extends proposed disentangling to IT after V1 results [38].Source: Discussion · p. 5
• β-VAE offers a theoretically grounded [41] account of how disentangled, low-dimensional codes may be learned from visual-input statistics [18], a question earlier work did not address.Source: Discussion · p. 6
• The authors argue the learning mechanism generalizes beyond faces [18–20], while applying β-VAE to richer visual domains depends on face-processing mechanisms generalizing across domains [27, 28].Source: Discussion · p. 6
• Their comparison measures align individual neurons with model units, retaining representational form and distinguishing computational models that transformation-insensitive methods may score similarly.Source: Discussion · p. 6
• The authors hope machine-learning insights will inform neuroscience tests of disentangled representations in abstract reasoning [47] and generalizable, efficient task learning [48].Source: Discussion · p. 6

Dataset

• Models used 2,162 natural grayscale frontal-face images, with 62 held out; training, validation and test splits were 80/10/10, and model-training faces were mirror-flipped relative to primate stimuli.The primates and models shared 2,038 images, while the held-out faces had been presented to the primates but not used to train the models.Source: Dataset · p. 11

Neurophysiological data

• The neural dataset comprised well-isolated single units recorded from two macaques viewing faces, and artificial responses were generated by linearly recombining standardized responses from 159 neurons.Neurons were recorded in face patches; stimuli were shown for 150 ms, and spikes were counted 50–350 ms after onset. Random subsets of up to 50 neurons were also sampled for comparisons.Source: Neurophysiological data · pp. 11–12

Model details

• The β-VAE used a convolutional encoder, a 50-dimensional latent representation and a reverse decoder, trained with Adam for 1 million iterations at a 1e-4 learning rate.The encoder had four convolutional layers followed by a 256-dimensional fully connected layer; the decoder parameterized a Bernoulli distribution.Source: Model details · p. 12

Baseline models

• Baselines tested alternative objectives and representations, including matched-architecture AE/VAE models, identity classifiers, VGG-Face, PCA, ICA and active appearance model (AAM) units.AE/VAE shared β-VAE’s architecture and training settings but used different objectives; classifier and VGG baselines used face-identity supervision, while PCA, ICA and AAM provided dimensionality-reduction or shape/appearance representations.Source: Baseline models · pp. 12–13

Training procedure and model selection

• They trained 400 β-VAE instances using 40 uniformly sampled β values from [0.5, 20] and 10 seeds per value, selected 51 among the top 15% UDR scores, and filtered low-VE models.Models were filtered when VE < mean VE − SD(VE), with the mean and standard deviation calculated across all models.Source: Training procedure and model selection · pp. 14–15

Analysis methods

• Lasso regression predicted each neuron’s responses from model units to measure test-set variance explained and alignment, while UDR compared β-VAE representations across random seeds without labels or neural data.Variance explained used cross-validated sparse weights; alignment used completeness, and UDR used pairwise latent-unit similarity across models trained with the same β.Source: Analysis methods · pp. 15–17
• For novel-face decoding, researchers matched model units one-to-one with neurons using correlations over 2,038 shared faces, regressed unit responses from neural activity, and decoded the 62 held-out faces.The predicted latent responses were passed through the pretrained decoder; cosine distance was averaged across held-out faces, and pairwise model comparisons used two-tailed Welch’s t-tests.Source: Analysis methods · p. 17
• For the AAM, the first N/2 principal components of keypoint locations defined shape units, while components of shape-normalised images defined appearance units.Keypoints were manually placed on 2,100 face images.Source: Analysis methods · p. 24