Source-linked AI summary
Deep Learning Human Mind for Automated Visual Classification
Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Mubarak Shah, Nasim Souly
TL;DR
The paper asks whether human visual capabilities can be read from brain signals and transferred to computer vision. It learns a discriminative EEG manifold with an RNN, trains a CNN to project images into that manifold, and reports competitive automated classification alongside stronger EEG representation learning. The authors frame these results as evidence that visual-recognition processes may be decoded for incorporation into automated methods.
Problem
Human visual recognition remains difficult to transfer to machines, motivating the question of whether brain signals can provide visual-category representations for computer vision.
Method
An RNN learns a compact discriminative representation of visual-stimulus EEG, and a CNN regresses images into that learned EEG representation for classification without new EEG data.
Results
The approach achieves competitive automated image classification, with 89.7% mean accuracy on the held-out test split, while the RNN-based EEG representation method outperforms prior methods.
Takeaways & Limitations
The results support decoding human visual-recognition processes and incorporating brain-based features into automated visual classification.
Takeaways & Limitations
The paper notes that deeper investigation is needed beyond its scope, and assumes that a low-dimensional discriminative EEG manifold exists.
Abstract
from arXiv · showhide
What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories. Afterwards, we train a Convolutional Neural Network (CNN)-based regressor to project images onto the learned manifold, thus effectively allowing machines to employ human brain-based features for automated visual classification. We use a 32-channel EEG to record brain activity of seven subjects while looking at images of 40 ImageNet object classes. The proposed RNN based approach for discriminating object classes using brain signals reaches an average accuracy of about 40%, which outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain-driven approach obtains competitive performance, comparable to those achieved by powerful CNN models, both on ImageNet and CalTech 101, thus demonstrating its classification and generalization capabilities. This gives us a real hope that, indeed, human mind can be read and transferred to machines.
1. Introduction
The paper targets the gap between human visual recognition and dataset-dependent machine representations by learning visual-category information from EEG and transferring it to image classification. It presents brain-signal-driven classification, a brain-based visual manifold, and a planned public dataset and implementation release.
- Motivation: Human visual interpretation remains superior to machine performance, while CNN representations depend strongly on their training dataset.CNN first-layer features generalize more broadly, whereas later-layer features are specific to a dataset or task.
- Proposed direction: The approach first classifies EEG signals to learn a discriminative manifold of visual categories, then projects images into that manifold for automated categorization.This transfers descriptors derived from human neural processes involved in visual scene analysis to computer vision.
- Contributions: The proposed deep-learning EEG classifier tackles more object classes and achieves higher accuracy than previous state-of-the-art methods.The contribution specifically concerns EEG data evoked by visual object stimuli.
- Contributions: The paper introduces the first computer-vision approach driven by brain signals and using visual descriptors extracted from human neural processes.This is presented as an automated classification approach based on visual scene analysis processes.
- Contributions: The authors plan to publicly release the largest EEG dataset for visual object analysis, together with source code and trained models.
2. Related Work
Prior BCI work largely focused on binary EEG classification, while visual object decoding had seen limited exploitation and mostly non-deep-learning methods. This paper extends deep learning to more visual object classes and projects images into an EEG-based manifold for brain-informed scene interpretation.
- Existing EEG classification: Most BCI studies primarily classify binary EEG patterns, such as P300 presence or seizure presence.
- Existing EEG classification: Deep-learning studies have modeled more complex brain-signal events, including cognitive load and audio stimuli, but not the paper’s visual-object setting.Reported examples include about 90% accuracy over four cognitive-load levels and 28% over 12 songs.
- Visual-object EEG decoding: Although neuroscience studies decoded up to a dozen object categories from EEG event-related potentials, visual-stimulus EEG classifiers were not deeply developed.
- Visual-object EEG decoding: Earlier visual-object EEG methods were limited in number, mostly used binary classification, and did not use deep learning.
- Paper contribution: The paper explores deep learning with more object classes than prior methods and projects images into an EEG-based manifold for automated interpretation using brain-informed features.The authors describe this projection as not previously done to their knowledge.
3. Method
The method learns discriminative low-dimensional EEG features for visual categories, then maps images directly into that EEG-based manifold for classification. It combines supervised RNN encoding of EEG sequences with CNN-based regression from image descriptors to EEG features.
- Method overview: The architecture first learns a low-dimensional, class-discriminative manifold from multichannel, time-varying EEG signals evoked by object images.EEG features are intended to summarize relevant image content in a 1D representation.
- EEG manifold learning: The RNN encoder and EEG-feature classifier are trained jointly with image class labels, allowing EEG sequences to be encoded and classified.LSTMs provide temporal analysis, while the encoder output represents the full EEG sequence.
- CNN-based regression: A second stage trains CNN-based regressors to map image representations directly to the learned EEG features, avoiding EEG recording for new images.The tested alternatives are end-to-end CNN training with a regression layer or separate regression from extracted deep features.
- Experimental protocol: The experiments record EEG while six subjects view 2,000 images from 40 recognizable ImageNet object classes.The protocol uses 128 channels, 1000 Hz sampling, and 440 retained samples per image after discarding the first 40 ms.
- EEG manifold learning: Three encoder designs are tested: common LSTM, channel LSTM followed by common LSTM, and common LSTM followed by an output layer.The channel-based design separates per-channel temporal summaries before inter-channel analysis; the output-layer design adds linear combinations and ReLU.
4. Performance Analysis
Performance analysis evaluates EEG representation learning, image-to-EEG regression, and their combination for automated visual classification. The method reaches strong EEG classification accuracy and 89.7% mean image-classification accuracy on held-out images, while testing generalization on Caltech-101.
- Evaluation setup: The evaluation separates RNN-based EEG learning, CNN-based regression, and their combination into automated visual classifiers.The dataset uses 80% training, 10% validation, and 10% test splits, with 12,000 EEG sequences for RNN training.
- EEG representation learning: About 83% classification accuracy was achieved by the proposed RNN approach, compared with 29% over 12 classes and 13% on this dataset for the prior method.
- CNN-based regression: GoogleNet feature extraction combined with a k-NN regressor produced the lowest MSE and was selected to extract EEG features from arbitrary images.The average EEG feature target performed markedly better than the best-features target.
- Automated visual classification: 89.7% mean classification accuracy was obtained on held-out dataset images never used for EEG manifold learning or CNN-based regression.This result was slightly below state-of-the-art CNN performance.
- Generalization evaluation: Generalization was tested by comparing the proposed features with VGG and GoogleNet on a non-overlapping 30-class Caltech-101 subset using separate multiclass SVM classifiers.
5. Conclusions
The paper presents a first human brain-driven automated visual classification method that learns EEG representations and maps images into that learned space. Its results support using decoded human visual processes in automated classification and motivate broader interdisciplinary research.
- Method: The proposed method combines an RNN to learn EEG-based visual representations with a CNN-based mapping from images into that representation.
- Results: The authors report competitive performance for both EEG representation learning and automated visual classification.
- Implications: The work lays foundations for transferring human visual capabilities to machines across computer vision, machine learning, and cognitive neuroscience.
- Future work: Future work targets larger numbers of image classes and interpretation of EEG-learned features to identify neural activation areas and frequency bands.