PaperlayerView exact revision on arXiv

Source-linked AI summary

ICLabel: An automated electroencephalographic independent component classifier, dataset, and website

Luca Pion-Tonachini, Ken Kreutz-Delgado, Scott Makeig

arXiv:1901.07915v2eess.SPcs.LGstat.ML

TL;DR

EEG independent components are time-consuming to interpret because ICA produces unordered and unlabeled components, while existing classifiers leave room to improve descriptiveness, accuracy, and efficiency. ICLabel addresses this gap with a diverse labeled dataset, crowdsourcing website, and cross-validated automated classifier. It performs better or comparably to publicly available methods while requiring roughly one tenth of their compute time.

  • Problem

    ICA produces an unordered and unlabeled set of components, and existing classifiers leave room to improve descriptiveness, accuracy, and efficiency.

  • Method

    ICLabel combines a diverse EEG IC dataset, crowdsourced probabilistic labels, candidate-classifier cross-validation, and expert testing across varied datasets.

  • Results

    ICLabel performed better or comparably to publicly available EEG IC classifiers while requiring roughly one tenth the compute time.

  • Takeaways & Limitations

    The classifier’s speed allows the possibility of detailed near-real-time classification of online-decomposed EEG data.

  • Takeaways & Limitations

    The results may not generalize to infant EEG because infants are absent from the training dataset and infant EEG can differ greatly from adult EEG.

Abstract

from arXiv · show

The electroencephalogram (EEG) provides a non-invasive, minimally restrictive, and relatively low cost measure of mesoscale brain dynamics with high temporal resolution. Although signals recorded in parallel by multiple, near-adjacent EEG scalp electrode channels are highly-correlated and combine signals from many different sources, biological and non-biological, independent component analysis (ICA) has been shown to isolate the various source generator processes underlying those recordings. Independent components (IC) found by ICA decomposition can be manually inspected, selected, and interpreted, but doing so requires both time and practice as ICs have no particular order or intrinsic interpretations and therefore require further study of their properties. Alternatively, sufficiently-accurate automated IC classifiers can be used to classify ICs into broad source categories, speeding the analysis of EEG studies with many subjects and enabling the use of ICA decomposition in near-real-time applications. While many such classifiers have been proposed recently, this work presents the ICLabel project comprised of (1) an IC dataset containing spatiotemporal measures for over 200,000 ICs from more than 6,000 EEG recordings, (2) a website for collecting crowdsourced IC labels and educating EEG researchers and practitioners about IC interpretation, and (3) the automated ICLabel classifier. The classifier improves upon existing methods in two ways: by improving the accuracy of the computed label estimates and by enhancing its computational efficiency. The ICLabel classifier outperforms or performs comparably to the previous best publicly available method for all measured IC categories while computing those labels ten times faster than that classifier as shown in a rigorous comparison against all other publicly available EEG IC classifiers.

1. Introduction and Overview

EEG recordings mix signals from multiple biological and non-biological sources, making ICA useful but leaving its components unordered and unlabeled. ICLabel addresses this ambiguity with a crowdsourced dataset, website, and validated automated classifier.

  • EEG electrode channels record mixtures of cortical, ocular, muscular, cardiac, and other electrical sources.
  • ICA can separate source-generator processes, but its unsupervised output remains an unordered and unlabeled set of independent components.
  • Automated IC categorization can improve consistency, accelerate large-scale IC selection, support real-time applications, and assist less-trained users.
  • ICLabel combines an IC classifier with a training and validation dataset and a website for crowdsourced labeling and education.
  • Redundant website labels are combined by a crowd-labeling algorithm to produce probabilistic training labels, after which candidate classifiers are cross-validated and the best model is validated against public classifiers.

2. Background

Prior EEG IC classifiers differ in descriptiveness, accuracy, and efficiency, motivating ICLabel’s broader probabilistic outputs and evaluation across varied recording conditions. ICLabel uses diverse data and relatively simple features to provide seven-category classifications efficiently.

  • Background: ICA may imperfectly separate sources because of artifacts, insufficient clean data, poor convergence, or nonstationary source projections.
  • Prior Methods: Publicly available methods may be excluded when they lack trained classifiers, classify only selected artifacts, or require additional recordings.
  • Prior Methods: Existing IC classifiers include MARA, ADJUST, FASTER, SASICA, and IC MARC, with differences in labels, thresholds, features, and training data.
  • Prior Methods: Prior methods leave room to improve output descriptiveness, accuracy, and efficiency.
  • Evaluation Criteria: Accuracy concerns generalization across experiments, environments, amplifiers, montages, and preprocessing pipelines, while efficiency matters especially for online streaming data.
  • The ICLabel Project: ICLabel computes probabilities across seven IC classes using diverse EEG examples, crowdsourced labels, and a neural network with relatively simple input features.

3. Materials and Methods

The ICLabel project assembled a large EEG IC dataset, crowdsourced probabilistic labels through an educational website, and developed candidate neural-network classifiers for automated IC categorization. Methods included standardized IC features, expert-labeled external testing, architecture and feature-set comparisons, and safeguards against ambiguity and overfitting.

  • Dataset and features: 203,307 unique ICs from 6,352 EEG recordings formed the source pool for the ICLabel training set.The recordings were collected from SCCN storage drives and lacked standardized IC classification metadata.
  • Dataset and features: Scalp topographies, power spectral densities, and equivalent current dipole estimates supplied standardized spatial, spectral, and source-localization features.Topographies were represented as 32 × 32 images, PSDs covered 1–100 Hz, and ECD estimates used a three-layer boundary element model.
  • Crowd labeling: Over 34,000 suggested labels from more than 250 contributors covered over 8,000 ICs, averaging 3.8 labels per labeled IC.The website combined crowdsourcing with tutorials and practice feedback to support both label collection and education.
  • Crowd labeling: 5,937 usable labeled EEG ICs were produced with CL-LDA from redundant website labels, yielding compositional probability vectors rather than forced single-class assignments.Compositional labels can represent mixtures such as partly Eye and partly Muscle or mostly Brain with some Line Noise.
  • Validation: 130 ICs labeled by six experts across ten additional datasets provided an external test set for generalization and comparison with existing classifiers.The datasets varied in recording environments, paradigms, amplifiers, montages, preprocessing pipelines, and ICA algorithms; the training set also carried risks from crowdsourced label errors and repeated optimization use.
  • Candidate classifier development: Six candidate classifiers varied ANN architecture, training paradigm, and input features, comparing GAN, CNN, and weighted CNN models with or without autocorrelation functions.All candidates used the same underlying CNN structure for inference; weighted CNN training doubled the weighting of Brain IC classification errors.

4. Results

ICLabel was compared with publicly available IC classifiers across harmonized two-, five-, and seven-class schemes. It performed strongly across categories and was substantially faster than IC MARC methods, although some categories and comparison choices limited interpretation.

  • Comparison design: Comparisons required merging non-identical classifier categories into common two-, three-, five-, and seven-class schemes.Figure 3 organizes category relationships and shows how categories can be merged into broader groups such as Other ICs and Eye ICs.
  • Classifier performance: Five-class comparisons showed IC MARCSF slightly better than ICLabel for Brain ICs, while both ICLabel methods greatly outperformed IC MARC versions for Eye ICs.IC MARCEF performed best for Muscle ICs by ROC curves, whereas IC MARCSF had higher thresholded recall among the other methods.
  • Classifier performance: ICLabel and ICLabelLite outperformed both IC MARC models for Other ICs by SOC points, with ROC performance depending on the operating regime.Most ICLabel errors involved over-classifying ICs as Other.
  • Seven-class comparison: In the seven-class comparison, ICLabel performance remained nearly unchanged from five classes except for Other ICs, while Line Noise and Channel Noise were classified relatively well by ROC curves.Channel Noise accuracy was lower than expected from ROC curves, and sparse Line Noise labels produced low-resolution ROC curves.
  • Seven-class comparison: 10% higher Brain recall for IC MARCSF than ICLabel in five classes nearly disappeared in seven classes.The authors attribute the five-class difference partly to probability summation when merging categories.
  • Classification speed: 120 ms and 170 ms median runtimes for ICLabelLite and ICLabel were 15.5 and 13.0 times faster than IC MARC’s 1.8 s median.ICLabelLite was 1.36 times faster than ICLabel; the comparison required autocorrelation feature computation.

5. Discussion

The discussion describes compositional IC labels as useful for discrete classification, multi-label detection, and manual inspection, while noting boundaries in real-time feature computation, evaluation, generalization, and trust.

  • Using Compositional IC Classifications: Compositional labels can support discrete classification by selecting the category with maximal probability and interpreting that probability as classifier confidence.They can also support multi-label classification, where a chosen threshold determines detected categories.
  • Using Compositional IC Classifications: Multi-label classification provides direct ROC-based performance estimates but may produce ICs with no category or multiple categories.These outcomes may or may not be acceptable depending on the use case.
  • Using Compositional IC Classifications: Compositional labels help manual inspection by identifying the most likely IC category while showing other categories the component resembles.They are also described as more informative for learning from mixed components and classification errors.
  • Timing: ICLabel is sufficiently efficient for near-real-time use in most cases, although its reported PSD and autocorrelation features are non-causal and require recursive estimation online.ICLabelLite was typically 36% faster, but the average per-IC difference was only 50 ms.
  • Differences Between Cross-validated Training Data and Expert-labeled Test Set Results: Performance was higher on cross-validated training data than on the expert-labeled test set, potentially reflecting labeling differences or variance from only 130 test ICs and six experts.The discussion identifies additional expert-labeled examples as necessary to resolve these possibilities.
  • Cautions: The training data limits classifier accuracy because it does not represent all EEG data types, including infants, and some recordings use template rather than precise electrode locations.These limitations constrain coverage and can introduce localization error in scalp topographies.
  • Future Development: The project can update labels, retrain the classifier, and publish new weights as additional website submissions accumulate.The proposed autonomous growth process is intended to continue expanding the project over time.

6. Conclusion

The conclusion presents ICLabel as a fast EEG independent-component classifier supported by a dataset and crowdsourcing website, with performance comparable to or better than prior public methods.

  • Conclusion: ICLabel performed better or comparably to the current state of the art while requiring roughly one tenth the compute time.It estimates classifications as compositional vectors across seven IC categories.
  • Conclusion: The project combines the ICLabel classifier with a crowdsourcing website and an evolving anonymized dataset of IC features.The classifier and dataset are available through EEGLAB and online repositories.

Appendix A. Evaluation Metrics

The appendix defines classification metrics, confusion-matrix representations, ROC analysis, and soft-label evaluation methods used to assess IC classifiers.

  • Classification Metrics: Balanced accuracy averages within-class accuracies, while TPi and FNi represent class-specific correct and incorrect detections.These quantities adapt binary classification concepts to the multi-class setting.
  • Classification Metrics: Cross entropy measures negative categorical loglikelihood or the prediction-dependent portion of Kullback–Leibler divergence.It was the primary metric optimized during ICLabel candidate training.
  • ROC Curves: ROC curves vary a binary classifier’s detection threshold and plot false positive rate against true positive rate.For multi-class evaluation, one class is treated as positive and the remaining classes as negative.
  • Confusion Matrices: Confusion matrices represent correct and incorrect classifications with reference categories as rows and predicted categories as columns.Normalized matrices divide each row by the number of examples in its reference category so row elements sum to 1.
  • Soft Confusion Matrices: Soft confusion matrices compare continuous-valued labels and predictions directly using soft AND functions rather than first discretizing them.Different soft AND assumptions produce pessimistic, expected, and optimistic estimates and associated pseudo-confidence intervals.
  • Soft Confusion Matrices: Soft-TPR and soft-FPR can share ROC axes with classical curves, but their values are not directly comparable because the underlying confusion matrices use different assumptions.ROC curves and SOC points are both presented because their relevance depends on the intended application.
  • Timing Evaluation: IC classification speed was measured as feature-extraction and inference time per IC across all ten expert-labeled test datasets.Implementations were run dataset by dataset in MATLAB on an AMD Opteron 6238 processor without specified parallelization.

Appendix B. Generative Adversarial Networks

The appendix explains GANs and the semi-supervised architecture used for ICLabel, which combines adversarial real-versus-generated discrimination with multi-class IC classification.

  • Generative Adversarial Networks: GANs use a generator to create simulated data and a discriminator to distinguish simulated examples from real data.Training typically alternates between generator and discriminator stages.
  • Semi-supervised GANs: The ICLabel SSGAN subdivides the discriminator’s real category into multiple IC classes such as Brain, Eye, and Other.Its model extends SSGAN with multiple generator networks sharing one random-noise input across IC feature sets.
  • ICLabel Architecture: The SSGAN outputs an eight-element compositional vector containing seven IC-category pseudo-probabilities plus a generated-IC element.For classification, the final element is removed and the remaining seven values are renormalized.
  • Semi-supervised GANs: SSGANs have been reported to improve classification over CNNs when labeled examples are scarce but unlabeled examples are available.The additional real-versus-generated task is theorized to encourage intermediate features useful for classification.

Appendix C. ICLabel Candidate Classifier Selection

Six candidate IC classifiers were compared using ten-fold cross-validation, varying model architecture, training paradigm, and feature sets. The results favored wCNN models with autocorrelation features, leading to the official wCNNAC ICLabel classifier.

  • Candidate comparison: Six candidate classifiers were evaluated in a three-by-two factorial design using ten-fold cross-validation on the ICLabel training set.The factors were model architecture and training paradigm, and the feature collections supplied to the classifiers.
  • Model architecture: GAN-based candidates underperformed the other candidate models across all seven ICLabel classes in ROC curves and most classes in SOC points.Exceptions included higher SOC scores for Channel Noise and similar performance for Brain and Eye ICs relative to some models.
  • Model architecture: wCNN models generally outperformed CNN models, especially on pessimistic and expected SOC performance across all classes.CNN models favored optimistic SOC points for Line Noise and Other ICs, while wCNN models favored optimistic performance for Muscle and Eye ICs.
  • Feature sets: Autocorrelation features appeared to consistently improve performance across all IC classes, especially Muscle and Other ICs.The improvement was reflected by nearly uniform gains in both ROC curves and SOC points.
  • Final selection: The official ICLabel classifier used the wCNNAC paradigm, while the faster ICLabelLite model used the wCNN paradigm without autocorrelation features.Autocorrelation improved performance but required additional feature-extraction time.

Appendix D. CL-LDA Details and Hyperparameters

CL-LDA estimated probabilistic IC labels and labeler skill from crowdsourced annotations using weighted counts, confusion-matrix priors, and class priors. Expert status and a minimum labeling threshold shaped the prior assumptions and data included in estimation.

  • CL-LDA estimation: CL-LDA estimated reference labels and labeler skill by maintaining counts of label assignments across IC categories and labelers.Labeler skill was parameterized by confusion matrices, while compositional labels used weighted counts and class priors supplied pseudo-counts.
  • Prior distributions: Expert labelers received favorable strong confusion-matrix priors, whereas labelers of unknown skill received favorable weak priors.Strong priors required more submitted labels to overcome their influence than weak priors.
  • Labeler inclusion: Only labelers who submitted at least ten labels were included to maintain an acceptable quality level for labeler-skill estimates.The threshold excluded labelers with too few observations to estimate skill reliably under the stated procedure.
  • Sampling procedure: CL-LDA used a 200-epoch Gibbs-sampler burn-in followed by 800 epochs for label estimation.The same CL-LDA procedure was applied to collected expert labels for the test data.

Appendix E. Artificial Neural Network Architecture Details

ICLabel classifiers combined separate neural networks for scalp topographies and other signal features, then fused their outputs for final classifications. Training used class-balanced sampling, early stopping, data augmentation, and TensorFlow-to-MATLAB conversion for EEGLAB distribution.

  • Network architecture: Each candidate and final classifier used feature-specific neural networks whose outputs were concatenated and passed to a final classification network.Scalp topographies used a two-dimensional CNN, while PSD and autocorrelation features used one-dimensional CNNs.
  • Input processing: Scalp topography inputs were 32-by-32 images with one intensity channel, processed using two-dimensional CNNs with dilated convolutions.PSD and autocorrelation inputs were processed by one-dimensional CNNs.
  • Training: Training used Adam with a 0.0003 learning rate, β1=0.5, β2=0.999, gradient cutoff 20, and batches of 128 ICs.Random class-balanced sampling addressed class imbalance, while holdout-based early stopping used a 5,000-batch viewing window.
  • Data augmentation: Scalp topographies were augmented through left-right reflection and negation, producing a four-fold increase in the dataset size.Negation preserves the channel data when both the ICA mixing matrix and IC time-courses are negated; reflection exploits near-symmetry of human physiology.
  • Implementation: The classifiers were built and trained in Python with TensorFlow, then converted to MATLAB using MatConvNet for EEGLAB plug-in distribution.Training files were made available through the project repository.
Loading 1901.07915v2…