A deep learning architecture for temporal sleep stage classification using multivariate and multimodal time series

Stanislas Chambon; Mathieu Galtier; Pierrick Arnal; Gilles Wainrib; Alexandre Gramfort

Source-linked AI summary

A deep learning architecture for temporal sleep stage classification using multivariate and multimodal time series

Stanislas Chambon, Mathieu Galtier, Pierrick Arnal, Gilles Wainrib, Alexandre Gramfort

arXiv:1707.03321v2Jul 5, 2017stat.MLcs.CVq-bio.NC

TL;DR

Sleep-stage scoring is clinically important but traditionally depends on expert inspection of multimodal PSG signals, while automatic methods have not consistently exploited their multivariate and multimodal structure. The paper develops an end-to-end deep network using spatial filtering, temporal convolutions, separate modality pipelines, and temporal context. On 61 public PSG records, it reports state-of-the-art performance with low computational cost and identifies effective channel and context configurations.

Problem
Sleep-stage classification supports diagnosis of sleep disorders, but traditional scoring requires experts and existing automatic systems have seldom exploited multimodal, multichannel PSG signals together.
Method
The paper uses an end-to-end deep network that spatially filters multichannel signals, learns temporal features with convolutions, processes modalities separately, and incorporates neighboring segments.
Results
The approach achieves state-of-the-art classification performance on 61 publicly available PSG records with low runtime and computational cost.
Takeaways & Limitations
Six EEG channels can perform similarly to 20 EEG channels, EMG improves performance, and temporal context helps most when spatial context is limited.
Takeaways & Limitations
The explored temporal context may not be directly suitable for online prediction and can risk overfitting abnormal sleep transitions associated with fragmented sleep.

Abstract

from arXiv · show

Sleep stage classification constitutes an important preliminary exam in the diagnosis of sleep disorders. It is traditionally performed by a sleep expert who assigns to each 30s of signal a sleep stage, based on the visual inspection of signals such as electroencephalograms (EEG), electrooculograms (EOG), electrocardiograms (ECG) and electromyograms (EMG). We introduce here the first deep learning approach for sleep stage classification that learns end-to-end without computing spectrograms or extracting hand-crafted features, that exploits all multivariate and multimodal Polysomnography (PSG) signals (EEG, EMG and EOG), and that can exploit the temporal context of each 30s window of data. For each modality the first layer learns linear spatial filters that exploit the array of sensors to increase the signal-to-noise ratio, and the last layer feeds the learnt representation to a softmax classifier. Our model is compared to alternative automatic approaches based on convolutional networks or decisions trees. Results obtained on 61 publicly available PSG records with up to 20 EEG channels demonstrate that our network architecture yields state-of-the-art performance. Our study reveals a number of insights on the spatio-temporal distribution of the signal of interest: a good trade-off for optimal classification performance measured with balanced accuracy is to use 6 EEG with 2 EOG (left and right) and 3 EMG chin channels. Also exploiting one minute of data before and after each data segment offers the strongest improvement when a limited number of channels is available. As sleep experts, our system exploits the multivariate and multimodal nature of PSG signals in order to deliver state-of-the-art classification performance with a small computational cost.

I. INTRODUCTION

The paper introduces an end-to-end deep learning approach for temporal sleep-stage classification from multivariate, multimodal PSG signals, using spatial and temporal context. It frames the task within clinical sleep scoring, class imbalance, and existing automated methods.

Clinical context: Sleep scoring assigns one of five AASM stages to each 30-second PSG segment using EEG, EOG, EMG, and related signals.The stages are Wake, REM, N1, N2, and N3.
Related work: Existing approaches either use expert-designed features or learn representations from transformed or raw signals, including convolutional neural networks.The paper positions its approach among these established categories of automatic sleep-stage classification.
Challenges: Automatic sleep scoring is an imbalanced multi-class prediction problem because stages such as N1 occur less frequently than stages such as N2.The paper notes that standard accuracy can favor frequent-stage predictions, motivating alternative evaluation or sampling strategies.
Challenges: Temporal sleep-stage classification incorporates neighboring segments so predictive models can account for transition rules between stages.The neighboring segments provide temporal context for the final classification decision.
Related work: Although PSG datasets provide multiple EEG channels and additional EOG or EMG modalities, automatic systems have seldom used all these signals together.The paper motivates exploiting multivariate EEG information for robustness and quantifying the benefit of spatial filtering.
Contribution: The proposed study presents an end-to-end temporal model for EEG, EOG, and EMG, benchmarks it against state-of-the-art methods, and examines spatial, temporal, and training-data dependencies.The formal setup represents each 30-second multivariate segment with a sleep-stage label and can use ordered neighboring segments.

II. MATERIAL AND METHODS

The paper presents an end-to-end multivariate, multimodal network for sleep-stage classification, with separate modality pipelines, learned spatial filtering, temporal convolutions, and softmax prediction. It formalizes temporal classification as predicting a segment label from neighboring 30-second signal segments.

Problem formulation: The temporal task maps an ordered sequence of 2k + 1 neighboring segments to a probability vector over sleep-stage labels.When k > 0, the model uses temporal context; when k = 0, it reduces to standard sleep-stage classification.
Problem formulation: Training minimizes the expected categorical cross entropy between the network output and the true label.The objective is expressed over the predictive-model class F and the input-label distribution.
Network architecture: The model processes EEG/EOG and EMG through separate pipelines, then concatenates their representations for softmax classification.The EEG/EOG and EMG streams use modality-specific processing before final fusion.
Network architecture: A learned linear spatial-filtering layer forms virtual channels by combining the original input channels.The operation is implemented as a time-independent linear transformation and can be viewed as task-driven spatial filtering.
Network architecture: Temporal convolutional blocks apply nonlinearities and max pooling to capture signal patterns along the time axis.For 128 Hz, each 30-second segment contains T = 3840 time steps, and the convolution kernels span approximately 0.5 seconds.

C. Time Distributed Multivariate Network

The time-distributed network extends the multivariate feature extractor across neighboring 30-second segments. It aggregates the resulting features before softmax classification and uses class-balanced minibatches during optimization.

Time-distributed architecture: Temporal context includes k preceding and k following segments around the segment being classified.The input sequence is {X_t−k, …, X_t, …, X_t+k}.
Time-distributed architecture: The network applies the feature extractor Z independently to 2k + 1 neighboring samples and aggregates their outputs before classification.The aggregated representation has dimension D(2k + 1).
Training: Training uses minibatches of 128 with approximately 20% of samples from each of the five classes.The class-balanced batches are intended to support discrimination of under-represented stages while optimizing balanced accuracy.
Time-distributed architecture: For k = 1, features from consecutive 30-second samples are aggregated and used to predict the label of the central sample.The sequence is transformed into [z_t−k, …, z_t, …, z_t+k] before classification.
Training: The time-distributed model is trained in two stages: pretrained feature-extractor weights are distributed across time, then the final softmax classifier is trained on aggregated features.The distributed feature-extractor weights are frozen during the second stage.

III. EXPERIMENTS

The experiments benchmark the proposed architecture on publicly available polysomnography records and examine spatial context, temporal context, and training-data quantity. Preprocessing includes filtering, downsampling, per-segment standardization, and record-level cross-validation.

Experimental design: The experiments compare the feature extractor with state-of-the-art approaches in univariate and bivariate settings.They also study spatial context, temporal context, and the quantity of training data.
Dataset: The dataset contains 62 night records from different subjects, with 20 EEG channels, 2 EOG channels, and 3 EMG channels per record before exclusions.One record was removed because of preprocessing issues, and the A2-referenced electrode was excluded from the study.
Preprocessing: Signals were low-pass filtered at 30 Hz and downsampled to 128 Hz before being supplied to the neural networks.The stated purpose of downsampling is to reduce computation while retaining information up to the Nyquist frequency.
Preprocessing: Each 30-second sample was standardized independently per channel to zero mean and unit variance.The paper motivates this preprocessing for handling changes in recording conditions over nearly eight-hour records.
Evaluation: Five record-level train, validation, and test splits were used to estimate performance on unseen records.Each split contained 41 training records, 10 validation records, and 10 test records.

B. Related work and compared approaches

The benchmark includes a hand-crafted-feature gradient boosting classifier and convolutional networks operating on raw univariate signals. The gradient boosting baseline uses XGBoost with class balancing and hyperparameter search.

Compared approaches: The comparison set contains gradient boosting on hand-crafted features and two convolutional networks trained on raw univariate time series.These methods represent alternative feature-engineering and end-to-end convolutional approaches.
Gradient boosting: The gradient boosting baseline uses time- and frequency-domain features extracted from each input sensor.Its spectral features include power and relative power across five frequency bands plus inter-band power ratios.
Gradient boosting: The gradient boosting implementation uses XGBoost decision trees and minimizes categorical cross entropy with undersampled training data.The model allows up to 1000 trees and uses early stopping on validation loss.
Gradient boosting: Hyperparameters were selected from 50 random configurations using average balanced accuracy across five training-evaluation splits.The search used only the training and validation subjects available for each experiment.

2) Convolutional networks on raw univariate time series

The proposed feature extractor is benchmarked against hand-crafted features and two end-to-end convolutional networks using comparable univariate and multivariate EEG inputs. Multivariate processing achieves the strongest classification results while retaining low computational cost.

Benchmark design: The benchmark compares the proposed extractor with Gradient Boosting and two end-to-end convolutional approaches using comparable EEG inputs without temporal context.The experiment evaluates feature representations on Fz-Cz and compares the proposed multivariate input using Fz-A2 and Cz-A2.
Overall comparison: The proposed extractor reaches classification performance comparable to Supratak et al. 2017 and higher than Tsinalis et al. 2016 and Gradient Boosting on Fz-Cz.It also uses very few parameters and low training and prediction run time compared with the other deep learning approaches.
Multivariate inputs: The multivariate proposed approach significantly outperforms its univariate counterpart and the other feature extractors receiving univariate time series.Using two channels instead of one causes only a limited increase in parameters, training time, and prediction time.
Per-class performance: The univariate proposed method has equal or higher confusion-matrix diagonal coefficients for W, N1, and N3 than the other feature extractors.Supratak et al. 2017 achieves the highest classification accuracy on N1 among the compared methods.

D. Experiment 2: More sensors increase performance

Increasing EEG sensor coverage improves classification until six well-distributed channels, while adding EOG and EMG provides an especially useful boost with fewer EEG channels. Additional modalities can therefore compensate for a larger EEG montage at lower computational cost.

EEG channel selection: Both Gradient Boosting and the proposed approach benefit from increasing the number of EEG sensors.The experiment evaluates seven EEG configurations ranging from 2 to 20 sensors and varying sensor positions.
EEG channel selection: Balanced Accuracy for the proposed approach stops improving after six well-distributed EEG channels.Additional channels may still improve robustness to bad sensors, but EEG channels can become redundant.
Additional modalities: Adding EOG and EMG significantly boosts classification performance, especially when EMG is included.The comparison restricts EEG inputs to six or 20 channels before adding the additional modalities.
Additional modalities: The performance boost from additional modalities is larger with six than with 20 EEG channels, and both multimodal EEG configurations reach the same performance.Additional modalities compensate for the larger spatial context in this setting.

E. Experiment 3: Temporal context boosts performance

Close temporal context improves classification, particularly when spatial context is limited, but excessive context reduces performance and creates stage-specific trade-offs. Temporal context also smooths predicted hypnograms by altering transition probabilities.

Performance effects: Considering neighboring samples boosts classification performance, especially when the spatial context is limited.The experiment examines temporal context alongside confusion matrices and transition probabilities between sleep stages.
Stage-specific effects: Close temporal context improves detection of N1, N2, and REM, whereas excessively wide context slightly reduces detection of W and N3.The results indicate a trade-off rather than a uniformly beneficial effect of increasing context.
Transition structure: Larger temporal context smooths the hypnogram by increasing diagonal transition coefficients.Transition probabilities from W, N1, N2, and REM improve, while transitions from N3, especially N3 to N3, are negatively impacted.

F. Experiment 4: More training data boost performance

More training records improve performance across spatial configurations, with the deep model improving more strongly as data increase. Rich multimodal spatial context can partly compensate for scarce training data.

Training-data quantity: Every algorithm and spatial configuration exhibits higher classification performance with more training data.The experiment varies the number of training records across five sample sizes and evaluates three spatial configurations.
Training-data quantity: Gradient Boosting is more resilient than the proposed approach in low-data settings, especially with a large spatial context.The deep learning model shows a stronger performance increase as the quantity of data grows.
Spatial context and data scarcity: With 12 training subjects, six EEG plus two EOG and three EMG channels reaches the same performance as two EEG channels with 41 training subjects.The comparison indicates that extended spatial context can compensate for scarcity of training data.

IV. DISCUSSION

The architecture uses learned spatial filtering and separate spatial and temporal convolutions to process multivariate PSG signals efficiently. Its compact design supports temporal sleep-stage classification with relatively few parameters.

Architecture: Learned linear combinations of input channels produce virtual channels intended to enhance task-relevant information through backpropagation.The number of virtual channels equals the number of input channels, but this remains an adjustable design choice.
Architecture: The model separates spatial and temporal convolutions, replacing expensive 2D convolutions with 1D operations.This low-rank spatio-temporal strategy was successful in the reported experiments.
Architecture: A two-layer feature extractor was selected because adding or removing layers did not provide extra performance gains.The architecture was designed as simple and versatile.
Architecture: The approach uses smaller convolution filters than some earlier methods, while newer approaches combine even smaller filters with more feature maps.The authors identify this combination as worth further investigation for potentially more signal-agnostic networks.
Computational cost: The network has approximately 10^4 parameters with extended spatial context and approximately 10^5 with both extended spatial and temporal context.These sizes are substantially below the parameter counts reported for the cited comparison approaches.

D. Classification metrics

The proposed model performs competitively overall and benefits from temporal context, but context width creates stage-specific trade-offs. Its clinical interpretation is also limited by the available annotation standard.

Classification performance: The proposed approach achieves equal univariate or higher multivariate classification metrics than benchmarked feature extractors with limited training and prediction time.Per-class performance is not uniformly highest: Supratak et al. 2017 performs better on N1, while Gradient Boosting has similar accuracy on N3.
Classification performance: Sensitivity for W reaches 0.85 with specificity close to 1, a profile the authors identify as potentially relevant to detecting fragmented sleep clinically.The result concerns W-stage detection rather than every sleep stage.
Subject variability: Balanced accuracy shows no particular correlation with sleep fragmentation index in the analyzed subjects.The authors interpret this as indicating possible use with patients, within the study’s supported scope.
Evaluation caveat: Performance cannot be compared with inter-scorer agreement because each dataset night was annotated by only one expert.The authors note that reproducing consensus-based evaluation is future work.
Temporal context: Small temporal contexts improve detection of N1, N2, and REM, whereas overly large contexts can reduce detection of W and N3.Accordingly, temporal-context width should be cross-validated.
Temporal context: Temporal context may improve performance while also risking overfitting abnormal sleep transitions associated with fragmented sleep disorders.This trade-off further supports cross-validating the context width.
Future work: A recurrent-network extension is proposed as future work for capturing complex stage transitions or long-term dependencies.Integrating the current feature extractor with recurrent networks remains to be done.

F. Influence of dataset

The study examines how data selection, sampling, multimodal inputs, and temporal context affect the model. It reports compact multimodal performance and stronger temporal-context gains when channel coverage is limited.

Dataset size: The study questions how much training data is needed for a correct sleep-stage-classification benchmark, especially for deep-learning approaches.Training-set size can act as a hyper-parameter for some algorithms.
Sampling and metrics: Balanced sampling and balanced metrics were used to emphasize accurate detection of sleep stages independently of their proportions.The authors report that sampling strategy affects evaluation metrics, and metric choice should motivate sampling strategy.
Sampling and metrics: Sampling strategy and evaluation metrics remain adjustable for different experimental or clinical purposes.The preferred strategy depends on the relative importance assigned to errors on minority stages.
Multimodal architecture: The model combines linear spatial filtering, temporal convolutions, and separate modality pipelines to pool multivariate and multimodal PSG information.This is the proposed deep neural-network design for temporal sleep-stage classification.
Overall outcome: The approach reports strong state-of-the-art classification performance with little runtime and computational cost.The authors identify portable and online sleep-stage classification as potential applications.
Spatial and modality context: Six EEG channels can perform similarly to 20 EEG channels, while adding EMG channels boosts model performance.The six-channel set is F3, F4, C3, C4, O1, and O2.
Temporal context: Temporal context produces a significant performance increase when spatial context is limited.The conclusion separately notes that the explored temporal context may be more directly suitable for offline than online prediction.