Generic decoding of seen and imagined objects using hierarchical visual features

Tomoyasu Horikawa; Yukiyasu Kamitani

Source-linked AI summary

Generic decoding of seen and imagined objects using hierarchical visual features

Tomoyasu Horikawa, Yukiyasu Kamitani

arXiv:1510.06479v3Oct 22, 2015q-bio.NCcs.CV

TL;DR

A method to decode generic object categories from brain activity would provide a practical capability. Generic object decoding predicts visual-feature values from fMRI and uses them to identify object categories from computed image features. Seen and imagined categories were predicted from fMRI; imagined categories were identified beyond decoder training, and higher-/lower-order features tended to be better predicted from higher-/lower-level visual areas, respectively.

Problem
A method to decode generic object categories from brain activity would provide a practical capability.
Source: Introduction · p. 4
Method
Generic object decoding predicts visual-feature values from fMRI and uses them to identify object categories from computed image features.
Source: Introduction · pp. 4–5, Discussion · p. 36
Results
Seen and imagined categories were predicted from fMRI; imagined categories were identified beyond decoder training, and higher-/lower-order features tended to be better predicted from higher-/lower-level visual areas, respectively.
Source: Discussion · p. 36
Takeaways & Limitations
The findings support fMRI-based prediction of seen and imagined object categories.
Source: Discussion · p. 36
Some CNN layers (CNN1–7) originally had more than 100,000 units, and the paper notes a challenge for future work.
Source: Discussion · pp. 37, 42

Abstract

from arXiv · show

Object recognition is a key function in both human and machine vision. While recent studies have achieved fMRI decoding of seen and imagined contents, the prediction is limited to training examples. We present a decoding approach for arbitrary objects, using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features including those from a convolutional neural network can be predicted from fMRI patterns and that greater accuracy is achieved for low/high-level features with lower/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, the decoding of imagined objects reveals progressive recruitment of higher to lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

Introduction

The study investigates decoding seen and imagined object categories beyond decoder-training classes by predicting visual features from fMRI and matching them to category representations. It describes the experiments, visual-cortex regions and computational features used, and examines feature predictability across visual hierarchy and imagery-related recruitment over time.

Introduction

Training-based classifiers are limited to classes used in training and cannot predict unseen classes.Source: Introduction · p. 3
Earlier visual image identification14,15,17 and reconstruction16 approaches decode according to image-based similarity, but do not explicitly identify the object a person is seeing or imagining.Source: Introduction · p. 4
Object categories are modeled as sets of mid- or high-level features invariant to rotation, scale, and position, unlike low-level image features.Source: Introduction · pp. 4–5
The study tests 13 feature types or layers from CNN, HMAX, GIST, and SIFT+BoF models.Source: Introduction · p. 5
The feature-based approach identifies seen and imagined objects among 15,372 ImageNet categories, including categories absent from decoder training.Source: Introduction · p. 6
Predicted seen-object features tracked feature complexity across hierarchical visual areas, while imagined-object decoding indicated progressive top-to-bottom recruitment.Source: Introduction · p. 6

Generic object decoding

The system predicts image-derived feature vectors from fMRI and identifies objects by comparing them with category-average vectors from an annotated image database.Source: Generic object decoding · p. 8
Training used 1,200 images across 150 categories; 50 held-out categories were tested 35 times each and imagined 10 times each.Source: Generic object decoding · pp. 8–9
Sparse linear regression decoders trained on fMRI from viewed images predicted feature vectors for test and imagined object categories not used in training.Decoders were trained separately for feature types or layers and brain regions of interest.Source: Generic object decoding · pp. 9–10

Image feature decoding

Higher-order features were better predicted from higher visual regions and lower-order features from lower regions, with significant feature-by-region interactions for CNN and HMAX features.Predicted feature values positively correlated with true values across all feature–ROI combinations (uncorrected P < 0.05).Source: Image feature decoding · p. 13, Figure 3 · p. 14
Preferred CNN-unit images increased in complexity across layers, from edge-like representations to shapes, textures, object parts, and intact objects.CNN6–8 are fully connected layers, so position information about preferred patterns was lost.Source: Image feature decoding · p. 15, Figure 4 · p. 17
Voxel-weight distributions matched the hierarchy: lower-region voxels were more often selected for lower visual features, while higher-region voxels were more often selected for higher features.CNN2 predictive voxels clustered mainly around V1–V3, whereas CNN8 predictive voxels were distributed around more anterior areas.Source: Image feature decoding · p. 15, Figure 4 · p. 17
Decoding accuracy was positively correlated with category discriminability for all visual features except HMAX1, and high-level CNN and HMAX features tended to be more discriminative.Category discriminability indexed the ratio of inter- to intra-category variation in feature values.Source: Image feature decoding · p. 19
Feature units that were decodable across visual cortical regions tended to be critical for defining object categories.The tendency was robustly reproduced with each ROI.Source: Image feature decoding · p. 19

Prediction of category-average features from stimulus- and imagery-induced brain activity

Imagery-related predictions of mid- and top-level CNN features suggest that mental imagery recruits intermediate-complexity visual representations through progressive top-down processing.The proposed interpretation is that these representations are neither simply pictorial nor conceptual.Source: Prediction of category-average… · p. 23
Category-average features were significantly predicted from stimulus-induced brain activity across all feature–ROI combinations, generally more accurately in higher than lower ROIs.Imagery-induced activity showed a similar but less accurate pattern, with relatively high correlations for mid-to-high CNN features decoded from V4, LOC, FFA, and PPA.Source: Prediction of category-average… · pp. 22–23, Figure 6 · p. 24
During imagery, prediction peaks tended to occur earlier for higher CNN layers and higher ROIs, whereas these timing differences were not found during stimulus viewing.CNN1 showed poor prediction and no clear peak; the reported time-by-layer and time-by-ROI interactions had P < 0.01.Source: Prediction of category-average… · p. 25, Figure 7 · p. 26

Object category identification

Identification analysis covered thousands of object categories, including categories not used for model training; for seen and imagined objects, true categories were correctly selected or highly ranked.When true categories were not assigned, categories ranked higher tended to have shorter semantic distance to the target categories.Source: Object category identification · pp. 28–30
When the correct category was not selected, higher-ranked candidates tended to be semantically closer to the target, particularly for mid-to-high CNN layers and SIFT+BoF in both conditions.This indicates that predicted feature vectors could select semantically similar, if not correct, categories.Source: Object category identification · p. 30
Mid-level features decoded from higher visual ROIs were most useful for identifying both seen and imagined categories.The discussion interprets this result as evidence for mid-level features’ contribution to discriminative object-category representations.Source: Object category identification · p. 33, Discussion · p. 36
Categories separated by greater WordNet path length tended to have higher identification accuracy, consistent with the relation between category rank and semantic distance.The study also reports that results were qualitatively similar when the 30 test categories absent from the pretrained CNN’s 1,000-category set were analyzed alone.Source: Object category identification · pp. 33–34
Seen and imagined categories were identified significantly above chance for most feature types, with highest accuracy around mid-level CNN5–6 features; imagery exceptions were CNN1, HMAX2, and HMAX3.The identification analysis considered candidate sets of two categories; accuracy was also above chance for most feature–ROI combinations.Source: Object category identification · p. 33, Figure 10 · p. 35

Discussion

Decoders trained on presented images generalized to identify seen and imagined categories absent from decoder training, demonstrating generic object decoding from fMRI signals.The discussion also reports successful predictions of category-average features for imagined objects.Source: Discussion · p. 36
Higher- and lower-order features were better predicted from higher and lower visual areas, respectively, revealing a hierarchical correspondence between cortical areas and CNN layers.The authors describe the CNN hierarchy as a useful proxy for the hierarchical feed-forward visual system.Source: Discussion · pp. 36–37
Imagery-induced activity supported category-average feature prediction and identification, and the analyses indicated progressive top-down recruitment of intermediate-complexity visual features.The discussion describes these perceptual feature representations as recruited during imagery in a graded manner.Source: Discussion · pp. 36, 38
Identification accuracies for most feature types or layers were nearly saturated with several hundred units, and results remained qualitatively similar across unit counts.This resampling analysis addressed possible bias from selecting 1,000 units from feature populations that could exceed 100,000 units.Source: Discussion · pp. 37–38
Mid-level rather than top-level CNN features produced the highest identification accuracy, and better-predicted feature units showed high category discriminability.The authors suggest that feature decoding accuracy may help select effective units for object recognition.Source: Discussion · pp. 39–41
Highly ranked candidates tended to be semantically similar to targets for mid-to-high-level CNN layers and SIFT+BoF, but not for HMAX or GIST.The authors suggest HMAX may not capture object semantics.Source: Discussion · p. 41
The approach decodes arbitrary object categories beyond decoder training and enables generic decoding of imagined categories through shared feature-level representations between perception and imagery.The authors suggest that decoding imagined categories could extend to spontaneous imagery, while distinguishing volitional from spontaneous imagery remains future work.Source: Discussion · pp. 40–42
The authors propose brain-based information retrieval and combining brain decoding with deep neural networks for richer outputs, including possible brain-machine-interface applications.Decoding spontaneous thought such as dreaming or daydreaming is presented as a possibility, not a demonstrated result.Source: Discussion · pp. 41–42

Subjects

The study included five healthy subjects: one female and four males, aged 23–38.Source: Subjects · p. 43

Visual images

Visual-image stimuli came from 200 representative object categories selected from ImageNet synsets.Images that failed stated size or aspect-ratio criteria were excluded, and remaining images were center-cropped.Source: Visual images · p. 43

Experimental design

The study included image-presentation and imagery experiments, with separate training-image and test-image sessions in the image-presentation experiment.The image-presentation sessions used a one-back repetition task to maintain attention.Source: Experimental design · pp. 43–44
Training presented 1,200 images from 150 object categories once each; a separate test session presented 50 images from 50 categories, one per category, 35 times each, and used categories excluded from training.Source: Experimental design · p. 44

Retinotopy experiment

Retinotopic mapping used a rotating wedge and expanding checkerboard ring to delineate visual cortical borders and identify V1–V4 in each subject.Source: Retinotopy experiment · p. 46

Localizer experiment

Functional localizers identified each subject’s LOC, FFA, and PPA using intact and scrambled images from face, object, house, and scene categories.Source: Localizer experiment · p. 46

MRI acquisition

Functional fMRI used a 3-T scanner, with whole-brain coverage for image, imagery, and localizer sessions and occipital-lobe coverage for retinotopy.Source: MRI acquisition · p. 47

MRI data preprocessing

Preprocessing discarded initial scans, corrected motion, and averaged block or imagery-period voxel signals after a one-volume shift for hemodynamic delay.Source: MRI data preprocessing · pp. 47–48

Region of interest (ROI) selection

The study defined LVC from V1–V3, HVC from LOC, FFA, and PPA, and VC by combining V1–V4 with HVC.Source: Region of interest (ROI) selection · pp. 48–49

Visual features

Visual features were computed with CNN, HMAX, GIST, and SIFT combined with Bag of Features; HMAX and SIFT+BoF used model training on images from the 150 training categories.Source: Visual features · p. 49

Convolutional neural network (CNN)

The ImageNet-trained CNN had five convolutional and three fully connected layers; 1,000 units were sampled from each of layers one through seven, and all units were used in layer eight.Source: Convolutional neural network (CNN) · p. 49
Images were represented by outputs from selected units across eight CNN layers, named CNN1–CNN8.Source: Convolutional neural network (CNN) · p. 50

HMAX

HMAX computed features through hierarchical layers built by alternating template matching and max operations.The layers comprised an image layer followed by S1, C1, S2, C2, S3, and C3.Source: HMAX · p. 50

GIST

GIST represented each image with a 1,024-dimensional feature vector for scene categorization.Source: GIST · p. 50

SIFT with BoF (SIFT+BoF)

SIFT with BoF features were invariant to image scaling, translation, and rotation, and partially invariant to illumination changes and affine or 3D projection.Source: SIFT with BoF (SIFT+BoF) · p. 51

Visual feature decoding

Sparse linear regression with Bayesian parameter estimation and automatic relevance determination predicted visual-feature vectors from fMRI; standard maximum-likelihood linear regression produced qualitatively similar results.Source: Visual feature decoding · pp. 51, 53, 54

Synthesizing preferred images using activation maximization

Preferred images for individual CNN units were generated by iteratively modifying a random image through backpropagation to maximize unit activation.Source: Synthesizing preferred images using… · p. 54

Identification analysis

Identification used decoded features to select the highest-correlating category from candidate sets whose categories were not used for decoder training.The candidate set was built from category-average features computed for ImageNet categories, excluding experiment and feature-training categories.Source: Identification analysis · p. 55

Statistics; Data and code availability

Statistical tests compared correlations and identification accuracies with chance, and the study’s experimental data and codes are available from the corresponding author.Normality was not rejected in any tested case (P > 0.01).Source: Statistics · p. 55, Data and code availability · p. 56

Competing financial interests

The authors declare that they have no competing financial interests.

Competing financial interests

The authors declare that they have no competing financial interests.Source: Competing financial interests · p. 64