PaperlayerView exact revision on arXiv

Source-linked AI summary

SLEEPNET: Automated Sleep Staging System via Deep Learning

Siddharth Biswal, Joshua Kulas, Haoqi Sun, Balaji Goparaju, M Brandon Westover, Matt T Bianchi, Jimeng Sun

arXiv:1707.08262v1cs.LG

TL;DR

Manual sleep staging from PSG is time-consuming and can have inter-rater reliability limitations, motivating automation that can handle heterogeneous EEG data. SLEEPNET trains a deep neural network on a large archive of overnight PSGs and deploys it as a clinical annotation tool. On 1,000 held-out patients, it achieved 85.76% accuracy and 79.46% Kappa, exceeding previously reported expert-expert agreement.

  • Problem

    Manual PSG annotation is time-consuming and subject to inter-rater reliability limitations, while EEG heterogeneity makes automated sleep staging challenging.

  • Method

    SLEEPNET trains a deep neural network, using expert-defined EEG features and an RNN, to annotate sequential sleep stages from large-scale PSG data.

  • Results

    85.76% accuracy and 79.46% Kappa were achieved on 1,000 held-out testing patients, exceeding previously reported expert-expert agreement.

  • Takeaways & Limitations

    SLEEPNET was deployed at MGH and has potential for incorporation into clinical workflows and extending sleep medicine beyond limited human expertise.

Abstract

from arXiv · show

Sleep disorders, such as sleep apnea, parasomnias, and hypersomnia, affect 50-70 million adults in the United States (Hillman et al., 2006). Overnight polysomnography (PSG), including brain monitoring using electroencephalography (EEG), is a central component of the diagnostic evaluation for sleep disorders. While PSG is conventionally performed by trained technologists, the recent rise of powerful neural network learning algorithms combined with large physiological datasets offers the possibility of automation, potentially making expert-level sleep analysis more widely available. We propose SLEEPNET (Sleep EEG neural network), a deployed annotation tool for sleep staging. SLEEPNET uses a deep recurrent neural network trained on the largest sleep physiology database assembled to date, consisting of PSGs from over 10,000 patients from the Massachusetts General Hospital (MGH) Sleep Laboratory. SLEEPNET achieves human-level annotation performance on an independent test set of 1,000 EEGs, with an average accuracy of 85.76% and algorithm-expert inter-rater agreement (IRA) of kappa = 79.46%, comparable to expert-expert IRA.

1. Introduction

SLEEPNET addresses the time and reliability limitations of manual sleep staging by applying deep neural networks to large-scale PSG data. Evaluated on 10,000 PSGs and deployed at MGH, it achieved performance comparable to human scoring.

  • Manual PSG scoring is time-consuming and has inter-rater reliability limitations, while EEG heterogeneity complicates automation.An 8-hour PSG can require more than one hour of technologist time, much of it spent annotating.
  • SLEEPNET is a clinical decision support tool that automatically stages sleep from EEG features using a recurrent neural network.The deployed interface presents inferences with raw EEG data and calculates sleep-quality statistics.
  • The system was trained and evaluated on 10,000 overnight PSGs, using 9,000 for training and 1,000 for testing.The paper describes this as the largest evaluation on an EEG classification problem at the time.
  • SLEEPNET was deployed at Massachusetts General Hospital with initial quantitative and qualitative evaluations of its clinical and research value.The introduction identifies deployment as a major contribution beyond proof-of-concept work.

2. Related work

Prior sleep-stage annotation methods include machine learning, neural networks, and commercial software, but studies generally used small cohorts. Deep neural networks have shown promise in healthcare, although clinical deployment has received limited attention.

  • Earlier EEG classification research emphasized seizure prediction, seizure detection, and brain-computer interfaces, often using physiologically guided feature engineering.These applications used a wide variety of machine-learning methods.
  • Most prior sleep-stage studies involved fewer than 100 subjects, and the largest reported study included 590 recordings.The paper argues that these sample sizes may not adequately address variability between subjects.
  • Deep neural networks, including CNNs and RNNs, have achieved strong results in challenging healthcare applications.Examples discussed include image classification, speech recognition, EEG representation learning, and seizure recognition.
  • Although deep neural networks have performed well in real-world tasks, most published healthcare work remained proof-of-concept rather than deployed clinical systems.The paper identifies large datasets as a key factor enabling neural networks to learn complex data distributions.

3. SLEEPNET System Description

SLEEPNET separates model training from point-of-care deployment: it converts multichannel EEG into features, selects a classification model, and outputs sleep stages for each 30-second epoch. The system displays EEG-derived results and summary statistics through a web application.

  • 3.1 System overview: SLEEPNET training maps multichannel EEG to a sequence of sleep-stage labels, assigning one label to each 30-second epoch.The resulting sequence is called a hypnogram.
  • 3.1 System overview: The best explored configuration combines expert-defined EEG features with a recurrent neural network.The training module compares feature representations and classification algorithms to identify the preferred configuration.
  • Deployment module: The deployment module loads new clinical PSG files and automatically assigns sleep stages through a lightweight Docker-based web application.It displays raw EEG, spectrograms, and summary statistics derived from the computed hypnogram.
  • 3.2 Description of EEG data and Notations: Each 30-second EEG epoch receives one of five conventional labels: wake, REM, N1, N2, or N3.Certified sleep technologists annotated the 10,000 PSG studies according to AASM standards.
  • 3.2 Description of EEG data and Notations: The input EEG is represented across six channels, and training infers a predicted hypnogram close to the expert-produced ground truth.The six-channel patient recording is represented as a matrix, while the classification target is a sequence of stage labels.

3.3 Feature Extraction

SLEEPNET represents each 30-second EEG epoch using raw, spectral, and expert-defined features. Spectral features are derived from overlapping sub-epochs, while expert-defined features capture time- and frequency-domain properties.

  • Raw EEG Features: SLEEPNET constructs raw EEG features as a 3-way tensor containing 6,000 samples from six channels for each 30-second epoch.The representation is denoted FR ∈R6000×6×ni.
  • Spectral Features: Each epoch is converted into a spectrogram by estimating power spectral density across 29 overlapping 2-second sub-epochs and 257 frequency bins from 0–100 Hz.The resulting spectrogram tensor has dimensions 29 × 257 per epoch.
  • Expert-Defined Features: Expert-defined features combine time-domain measures, including line length and kurtosis, with frequency-domain features extracted from EEG spectra.Line length measures amplitude and frequency oscillations, while kurtosis measures extreme values.

3.4 Classification in SLEEPNET

SLEEPNET evaluates conventional and deep learning classifiers on EEG representations, including CNNs, RNNs, and a hybrid RCNN. The hybrid architecture extracts spectrogram features with a CNN and models their temporal dependencies with an LSTM.

  • Model Comparison: SLEEPNET compares logistic regression, tree boosting, multilayer perceptrons, and deep learning models using the constructed EEG features.The comparison includes multiple classification-model families before focusing on deep learning methods.
  • Convolutional Neural Networks: CNNs process spectrograms with convolution, nonlinear activation, and pooling layers, while raw six-channel waveforms are averaged and processed by a 1D CNN.The final CNN uses stacked 3 × 3 filters followed by max pooling and a fully connected layer.
  • Recurrent Neural Networks: RNNs classify each 30-second epoch from its current EEG representation and previous hidden state, with SLEEPNET implementing the formulation using LSTM cells.The optimized recurrent model used five LSTM layers with tanh activation, dropout keep probability 0.9, and 1,000 hidden units.
  • Recurrent-Convolutional Neural Networks: The RCNN passes each spectrogram through a CNN to extract spatial features, then uses an LSTM to model temporal dependencies and predict sleep stages.A 29 × 257 spectrogram becomes a 4,352-dimensional feature vector before recurrent processing.

3.5 Evaluation

Evaluation separates patients into training, validation, and independent test sets, with the final test set containing EEG data from 1,000 patients. Hyperparameters are tuned through random search across recurrent and convolutional design choices.

  • Data Splitting: 1,000 patients provide the final test set, while the remaining 9,000 patients support training and validation; deep learning uses 300 validation patients.Each patient contributes approximately eight hours of data and 950–1,000 epoch labels.
  • Additional Evaluation Choices: The evaluation also tests normalization choices, including batch normalization, alongside the listed model hyperparameters.

3.6 Implementation Details of Model Training Module

The model-training module is implemented with TensorFlow and supporting machine-learning libraries on a multi-GPU workstation.

  • Software and Hardware: SLEEPNET deep learning models use TensorFlow 0.12.1, while scikit-learn supports logistic regression and multilayer perceptrons and XGBoost supports tree boosting.Training uses a machine with an Intel Xeon E5-2640, 256GB RAM, and four Nvidia Titan X GPUs.

3.7 Model Deployment Module

SLEEPNET was deployed as a Docker-backed web application that scores clinical sleep EEG cases and presents interpretable visualizations and sleep statistics for clinicians.

  • SLEEPNET runs its trained model in a Docker container behind a web application that clinicians use by selecting a case.The application then generates a report page with clean visualization.
  • The interface displays EEG waveforms, spectrograms, expert labels, predicted stages, and zoomed-in views for selected 30-second epochs.
  • SLEEPNET calculates stage durations, sleep efficiency, and fragmentation indices to summarize sleep quality.Sleep efficiency is defined as sleep time divided by total recording time.

4. Experiments

Experiments evaluated feature representations and classification models on PSG recordings, with deep recurrent modeling achieving the strongest reported performance and supporting deployment-speed requirements.

  • 4.2 Accurate Model Classification: Three input representations were compared: six-channel raw waveforms, averaged spectrograms, and 96-dimensional expert-defined features.
  • 4.2 Accurate Model Classification: Five classifier families were evaluated, including logistic regression, tree boosting, MLP, CNN, RNN, and RCNN variants.
  • 4.2 Accurate Model Classification: 79.46% Kappa was achieved by the RNN with expert-defined features, while deep learning models consistently outperformed logistic regression, tree boosting, and MLP.The best model was evaluated on a large patient cohort, and N1 was the hardest sleep stage to classify.
  • 4.3 Computational Efficiency: 2–5 minutes was the reported scoring time for an entire eight-hour EEG recording, including feature extraction.This was considered sufficiently small for deployment, with annotations progressively displayed by 30-second epoch.
  • 4.4 Sensitivity Analysis of Model Training: Performance increased with more RNN look-back steps and with more training subjects.These sensitivity analyses indicate benefits from longer temporal context and additional training data.

5. Deployment Case Studies

Deployment case studies examined SLEEPNET in a sleep laboratory and a neurology ICU, where clinicians assessed workflow utility and rapid sleep characterization.

  • 5. Deployment Case Studies: The two case studies targeted different needs: speeding expert annotation in sleep laboratories and enabling sleep assessment where ICU clinicians lack time or expertise.
  • 5.1 Qualitative evaluation in MGH Sleep Laboratory: In the MGH Sleep Laboratory, clinicians found loading and interface use fast, visualization natural, and hypnogram-based inspection compatible with clinical PSG interpretation.
  • 5.2 Qualitative evaluation in MGH Neurology Intensive Care Unit (ICU): SLEEPNET rapidly characterized sleep in ICU patients without adding significant clinical work.In 10 recordings longer than 24 hours, sleep was poor, with efficiency below 85%, fragmentation, and REM sleep below 25%.
  • 5.3 Future Applications: Future applications proposed include pre-scoring for manual editing, possible replacement of manual scoring after further validation, and augmentation of limited-channel home monitors.
  • 5.3 Future Applications: Routine automated sleep monitoring in the ICU was linked to developing strategies to prevent and shorten delirium.

6. Conclusion

SLEEPNET automated five-stage sleep annotation using a deep neural network trained on a large archive of real-world overnight recordings. Its held-out performance reached expert-algorithm agreement, while deployment experience suggested potential for clinical workflow integration.

  • 85.76% inter-rater agreement and 79.46% Kappa were achieved on 1,000 held-out testing patients, exceeding previously reported expert-expert agreement levels.
  • SLEEPNET trained a deep neural network to label sequential EEG epochs with the five conventional sleep stages used in sleep medicine.
  • A deployment framework addressed inconsistent software environments and point-of-care usability.The authors report potential for incorporation into sleep laboratories and neurocritical care settings.
  • By automating a task requiring scarce human expertise, SLEEPNET could extend access to sleep medicine beyond current settings.
Loading 1707.08262v1…