PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSleepNet-Lite: A Simplified Automatic Sleep Stage Scoring Model with Uncertainty Estimates

Luigi Fiorillo, Paolo Favaro, Francesca Dalia Faraci

arXiv:2108.10600v1cs.LG

TL;DR

Deep sleep-scoring models can be computationally demanding, process long EEG sequences, and often lack uncertainty estimates. DeepSleepNet-Lite scores short EEG sequences with a lightweight architecture and Monte Carlo dropout. On Sleep-EDF it performs slightly below or on par with state-of-the-art methods, while rejecting uncertain predictions raises accuracy on both database versions.

  • Problem

    Many deep sleep-scoring models use computationally demanding architectures and long input sequences, and only a few estimate prediction uncertainty.

    Source: §I · p. 1

  • Method

    DeepSleepNet-Lite scores 90-second single-channel EEG sequences and uses Monte Carlo dropout to estimate uncertainty and enhance performance.

    Source: §I · pp. 1–2, §IV · p. 4

  • Results

    Performance is slightly lower than, or on par with, state-of-the-art methods; after selection, accuracy is 86.1% on Sleep-EDF v1-2013 ±30mins and 82.3% on v2-2018 ±30mins.

    Source: §VII · p. 8, F. Comparison among our methods · p. 8

  • Takeaways & Limitations

    The lightweight approach processes 90-second EEG sequences without buffering lengthy inputs, and its uncertainty estimates can identify misclassified epochs among those selected by the query procedure.

    Source: §VII · p. 8, §VIII · p. 10

    The authors cannot conclude that using only the first representation-learning block will achieve equally good results on larger databases.

    Source: §VII · p. 8, §VIII · p. 10

Abstract

from arXiv · show

Deep learning is widely used in the most recent automatic sleep scoring algorithms. Its popularity stems from its excellent performance and from its ability to directly process raw signals and to learn feature from the data. Most of the existing scoring algorithms exploit very computationally demanding architectures, due to their high number of training parameters, and process lengthy time sequences in input (up to 12 minutes). Only few of these architectures provide an estimate of the model uncertainty. In this study we propose DeepSleepNet-Lite, a simplified and lightweight scoring architecture, processing only 90-seconds EEG input sequences. We exploit, for the first time in sleep scoring, the Monte Carlo dropout technique to enhance the performance of the architecture and to also detect the uncertain instances. The evaluation is performed on a single-channel EEG Fpz-Cz from the open source Sleep-EDF expanded database. DeepSleepNet-Lite achieves slightly lower performance, if not on par, compared to the existing state-of-the-art architectures, in overall accuracy, macro F1-score and Cohen's kappa (on Sleep-EDF v1-2013 +/-30mins: 84.0%, 78.0%, 0.78; on Sleep-EDF v2-2018 +/-30mins: 80.3%, 75.2%, 0.73). Monte Carlo dropout enables the estimate of the uncertain predictions. By rejecting the uncertain instances, the model achieves higher performance on both versions of the database (on Sleep-EDF v1-2013 +/-30mins: 86.1.0%, 79.6%, 0.81; on Sleep-EDF v2-2018 +/-30mins: 82.3%, 76.7%, 0.76). Our lighter sleep scoring approach paves the way to the application of scoring algorithms for sleep analysis in real-time.

I. INTRODUCTION

Automatic sleep scoring seeks to reduce the time-consuming, variable manual classification of sleep epochs, while existing deep models can be computationally demanding and offer limited uncertainty estimates. DeepSleepNet-Lite uses a lightweight feed-forward architecture and Monte Carlo dropout to score 90-second single-channel EEG sequences and estimate uncertainty.

I. INTRODUCTION

  • Existing deep sleep-scoring models can have many parameters and process sequences up to 12 minutes long, while only two cited architectures provide uncertainty estimates.The cited approaches use either an additional classification block or 16 models to estimate uncertainty.Source: §I · p. 1
  • DeepSleepNet-Lite is a simplified, lightweight scoring architecture that predicts sleep stages and estimates their uncertainty without extra computation over its baseline.It uses Monte Carlo dropout at test time to capture uncertainty and enhance scoring performance.Source: §I · pp. 1–2

II. DEEPSLEEPNET-LITE

  • The architecture uses only 90-second, single-channel EEG input and applies sequence-to-epoch learning to predict the central epoch.It retains the first representation-learning part of DeepSleepNet and trains it using a small temporal context.Source: §I · p. 2, §II · p. 2

A. The Architecture

The architecture extracts temporal and frequency patterns from 90-second EEG using parallel CNNs, then combines their features to classify the central epoch. It is trained to predict one of five sleep-stage classes.

A. The Architecture

  • Two parallel CNNs use small and large first-layer filters to extract high-time-resolution and high-frequency-resolution patterns from the 90-second EEG input.Each CNN section has four convolutional layers and two max-pooling layers.Source: A. The Architecture · p. 2
  • The CNN feature vectors are concatenated and passed to a softmax layer that outputs probabilities for five mutually exclusive classes.The model is trained with cross-entropy loss.Source: A. The Architecture · p. 2
  • The softmax layer computes logits from the concatenated feature vector using its weights and bias.The output probability for a class is associated with the centered 30-second signal within the input sequence.Source: A. The Architecture · p. 2

B. Training Algorithm

Training uses three-epoch sequences to predict the central epoch, with vertical flipping and oversampling to balance classes. Regularization and validation-based early stopping guide training and model selection.

B. Training Algorithm

  • The model takes three 30-second epochs as input and predicts the central epoch, while vertical flipping and random oversampling balance the less-represented classes.Oversampling makes all sleep stages equal in number to the most represented class.Source: B. Training Algorithm · pp. 2–3

C. Regularization Techniques and Training Parameters

  • Early stopping monitors validation F1-score, allows additional iterations through a patience setting, and saves the model with the highest performance.Dropout removes units randomly during training with probability 0.5.Source: C. Regularization Techniques and… · p. 3
  • L2 weight decay adds a penalty on weight values to the loss, with lambda set to 10^-3.The training uses Adam with a mini-batch size of 100 and learning rate 10^-4, and runs up to 100 iterations with patience 50.Source: C. Regularization Techniques and… · p. 3

III. MODEL CALIBRATION

Label smoothing mixes hard targets with a distribution to support calibrated confidence estimates. The paper also introduces smoothing based on conditional probabilities over neighboring sleep stages.

III. MODEL CALIBRATION

  • Label smoothing weights hard targets with the uniform distribution 1/K when minimizing cross-entropy, to calibrate the model’s predicted confidence.The smoothing parameter for this distribution was set to 0.1.Source: §III · p. 3, A. Conditional Probability… · p. 4

A. Conditional Probability Distribution in Label Smoothing

  • The proposed conditional distribution smooths labels using the probability of the central sleep stage given the stages immediately before and after it.The K×K×K matrix is computed over all three-stage sequences, and the corresponding conditional probability vector is used to smooth the central epoch’s target.Source: A. Conditional Probability… · p. 3
  • For an awake preceding stage and an N1 following stage, the example uses the corresponding conditional-probability vector to smooth the target.The conditional probabilities are computed from sequences in Sleep-EDF v1-2013.Source: A. Conditional Probability… · pp. 3–4, Table I · p. 4
  • The conditional-distribution smoothed targets are also used in the cross-entropy loss, with the smoothing parameter set to 0.2.The study explored smoothing parameters up to 0.5.Source: A. Conditional Probability… · p. 4

IV. ESTIMATING UNCERTAINTY

DeepSleepNet-Lite uses Monte Carlo dropout at inference to estimate uncertainty from repeated predictions. It derives final stage predictions from mean probabilities and identifies uncertain predictions using both mean and variance.

IV. ESTIMATING UNCERTAINTY

  • Monte Carlo dropout applies randomized sampling at inference to produce multiple predictions and estimate model uncertainty.The paper describes the sampled subnetworks as Monte Carlo samples from the space of possible models.Source: §IV · p. 4
  • The final sleep-stage prediction is the class with the highest mean probability across the N predictions.The model computes the mean and variance of the N predictions for each stage.Source: §IV · p. 4
  • Uncertain predictions are identified by analysing their computed mean and variance, and may be sent to a physician for secondary review.The selection procedure is described in subsection VI-D.Source: §IV · p. 4

V. DATA

The study evaluates single-channel Fpz-Cz EEG from the Sleep-EDF datasets, including the 2013 and expanded 2018 releases. Its experiments use cross-validation and compare models trained with different calibration and label-smoothing choices.

V. DATA

  • Sleep-EDF provides manually scored 30-second epochs; the study uses Fpz-Cz EEG at 100 Hz without preprocessing and merges N3/N4 while excluding MOVEMENT and UNKNOWN.The database includes 153 recordings from 78 subjects in the Sleep Cassette subset; the older 2013 release contains 39 recordings from 20 subjects.Source: §V · p. 4
  • The experiments include both in-bed-only data and data extending 30 minutes before and after the in-bed period.The two trimming approaches are used to facilitate comparison with existing scoring methods.Source: §V · p. 4

A. Design of Experiments

  • Validation uses 20 folds for Sleep-EDF v1-2013 and 10 folds for v2-2018, with the same subject IDs in each fold as in [26].The authors note that subject assignment across training, validation, and test sets may affect results in these small datasets.Source: A. Design of Experiments · p. 5
  • Three training variants are compared: an uncalibrated base model, uniform label smoothing, and label smoothing based on conditional sleep-stage probabilities.Each variant is evaluated with and without Monte Carlo dropout at inference.Source: A. Design of Experiments · p. 5

B. Metrics

The section defines performance measures and describes Expected Calibration Error (ECE) as a binned comparison of accuracy and predicted confidence. The cited table reports overall performance and calibration measures for models with and without Monte Carlo dropout.

B. Metrics

  • Performance is measured using per-class F1, overall accuracy, macro-averaged F1, Cohen’s kappa, and weighted-averaged F1 to account for label imbalance.The text cautions that weighted-averaged F1 is not a realistic weighted average of precision and recall.Source: B. Metrics · p. 5
  • Table IV reports overall performance and calibration for models with and without Monte Carlo dropout on Sleep-EDF v1-2013 ±30mins using 20-fold cross-validation.The table caption states that the best results are shown in bold.Source: Table IV · p. 5
  • Expected Calibration Error approximates the expected difference between accuracy and confidence, with confidence defined by softmax output probabilities.Predictions are grouped into equally spaced bins; each bin’s accuracy and average predicted probability contribute to the weighted difference, and perfect calibration gives ECE = 0.Source: B. Metrics · p. 5

C. Analysis of Experiments

The experiments assess calibration and performance with label smoothing and Monte Carlo dropout, then examine class-level results and how many Monte Carlo samples are needed. The best model uses base+LSu with Monte Carlo sampling at test time.

C. Analysis of Experiments

  • F1-score for the three models plateaus after 30 Monte Carlo samples on average, so subsequent results use base+LSu with 30 samples.Monte Carlo sampling outperforms the no-MC experiments after approximately three samples.Source: Figure 3 · p. 6, C. Analysis of Experiments · p. 6
  • With Monte Carlo dropout, label smoothing preserves or improves overall performance, while base+LSs has an ECE of 0.031.The text reports ECE values of 0.023 and 0.071 for LSu and LSs without MC, versus 0.111 for the base model.Source: C. Analysis of Experiments · pp. 5–6
  • N1 has the lowest F1-score, 44.4% and 46.0% on the two datasets; other stages range from 82.4% to 88.2% on v1 and 76.4% to 91.5% on v2.N1 epochs are often classified as wake, N2, or REM.Source: C. Analysis of Experiments · p. 6

D. Uncertainty Estimate

Monte Carlo dropout supplies uncertainty estimates used to select epochs for possible secondary review. The analysis compares variance- and mean-based selection and reports performance on the retained and selected epochs.

D. Uncertainty Estimate

  • Selecting epochs with the lowest mean predicted probabilities yields higher F1-score on the remaining epochs than variance-based selection.The comparison varies the percentage of selected epochs and uses the best model on Sleep-EDF v1-2013 ±30mins.Source: D. Uncertainty Estimate · p. 6, Figure 4 · p. 7
  • At a 5% selection threshold, about 54 epochs per recording are selected for possible physician review.Variance selects epochs with the highest predicted-probability variance; mean-based selection uses the lowest mean values.Source: D. Uncertainty Estimate · p. 6
  • With mean-based selection, misclassified epochs outnumber correctly classified ones among selected epochs up to a 10% threshold; variance-based selection reaches this pattern at 2%.Figure 5 compares the proportions of misclassified and correctly classified selected epochs for the two procedures.Source: D. Uncertainty Estimate · p. 6, Figure 5 · p. 7
  • The model has greater difficulty with N1 and REM than with W, N2, and N3, which show lower variance and higher predicted probabilities.The per-class uncertainty estimates cover both Sleep-EDF v1-2013 ±30mins and v2-2018 ±30mins.Source: D. Uncertainty Estimate · p. 6, Table VII · p. 8

E. Comparison with state-of-the-art

DeepSleepNet-Lite is compared with deep-learning methods under matched dataset and evaluation conditions, and the paper also evaluates its query procedure. The model is near state of the art with fewer parameters, while rejecting selected epochs raises performance on the retained set.

E. Comparison with state-of-the-art

  • DeepSleepNet-Lite performs slightly below or on par with state-of-the-art methods across Sleep-EDF datasets, with about 0.6M parameters versus 1.3M–24.7M for most listed alternatives.SeqSleepNet has fewer parameters at about 0.2M; the comparison restricts methods by signal, evaluation procedure, and independent training and test sets.Source: E. Comparison with state-of-the-art · p. 8

F. Comparison among our methods

  • After query selection, accuracy reaches 86.1% on v1-2013 ±30mins and 82.3% on v2-2018 ±30mins.The model improves on kept epochs and drops below 50% performance on rejected epochs when using mean-based query.Source: F. Comparison among our methods · p. 8

VII. DISCUSSION

The discussion reports near-state-of-the-art Sleep-EDF performance while qualifying that evidence to small, low-heterogeneity datasets, and notes trade-offs in uncertainty estimation and label smoothing. The conclusion leaves larger-dataset robustness unresolved while pointing to real-time and home-monitoring applications.

VII. DISCUSSION

  • On Sleep-EDF, performance is slightly lower than or on par with existing state-of-the-art methods; Table VIII compares raw single-channel Fpz-Cz systems across accuracy, macro F1, kappa, and per-class F1.The authors say these observations hold on small datasets with low heterogeneity between subjects.Source: §VII · p. 8, Table VIII · p. 9
  • Monte Carlo dropout reuses existing layers and adds no computation over the baseline, but N sampling passes increase evaluation time by N.Its outputs include interpretable means and variances of predicted probabilities; single-sequence evaluation takes only a few milliseconds.Source: §VII · p. 8
  • Conditional-probability label smoothing seems to improve calibration, but does not always yield higher performance or better uncertainty estimates with Monte Carlo dropout.The authors say these results require further investigation.Source: §VII · p. 8

VIII. CONCLUSION AND FUTURE WORKS

  • The authors cannot conclude that the first representation-learning block will perform equally well on larger databases and call for further robustness investigation.They state that the short-context observations apply to small, low-heterogeneity datasets.Source: §VII · p. 8, §VIII · p. 10
  • Monte Carlo dropout enhanced performance and identified a relevant number of misclassified queried epochs; the low-parameter model is less prone to overfitting on a small dataset.Source: §VIII · p. 10
  • The authors present the lightweight scoring approach as paving the way to real-time applications and home-monitoring scenarios.Source: §VIII · p. 10
Loading 2108.10600v1…