PaperlayerView exact revision on arXiv

Source-linked AI summary

Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and Automated sleep staging

Antoine Guillot, Fabien Sauvet, Emmanuel H During, Valentin Thorey

arXiv:1911.03221v4q-bio.QMcs.LGeess.SPstat.ML

TL;DR

Sleep staging is time-consuming and commonly evaluated against a single scorer despite limited inter-rater agreement. The paper introduces two multi-scored datasets and a consensus-based evaluation framework, then benchmarks existing methods and SimpleSleepNet. Automated methods often match or exceed human performance, while generalization across populations and devices remains limited.

  • Problem

    Automated sleep staging is usually compared with one manual scorer despite inter-rater variability, and shared publicly available multi-scorer benchmarks are lacking.

  • Method

    The study introduces DOD-H and DOD-O, evaluates methods against multi-scorer consensus, and develops SimpleSleepNet for benchmarking.

  • Results

    SimpleSleepNet and other leading automated methods outperform the average human scorer on both DOD-H and DOD-O, while many methods reach human-level performance.

  • Takeaways & Limitations

    Multi-scored datasets provide a more realistic basis for assessing automated sleep staging, which could support clinical use when data collection is consistent.

  • Takeaways & Limitations

    Automatic sleep staging generalizes poorly outside the patient population or PSG device used for training, limiting clinical use across settings.

Abstract

from arXiv · show

Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a single human scorer annotation despite an inter-rater agreement of about 85 % only. The present study introduces two publicly-available datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients suffering from obstructive sleep apnea (OSA). Both datasets have been scored by 5 sleep technologists from different sleep centers. We developed a framework to compare automated approaches to a consensus of multiple human scorers. Using this framework, we benchmarked and compared the main literature approaches. We also developed and benchmarked a new deep learning method, SimpleSleepNet, inspired by current state-of-the-art. We demonstrated that many methods can reach human-level performance on both datasets. SimpleSleepNet achieved an F1 of 89.9 % vs 86.8 % on average for human scorers on DOD-H, and an F1 of 88.3 % vs 84.8 % on DOD-O. Our study highlights that using state-of-the-art automated sleep staging outperforms human scorers performance for healthy volunteers and patients suffering from OSA. Consideration could be made to use automated approaches in the clinical setting.

I. INTRODUCTION

Sleep staging is labor-intensive and subjective, motivating automated approaches and evaluation against multiple human annotations. The study addresses limited shared multi-scorer benchmarks while comparing expert and deep-learning methods.

  • I. INTRODUCTION: 82.6% average inter-rater agreement varies substantially by sleep stage, patient, disorder, and sleep center.Agreement is 90.5% for REM but only 63.0% for N1.
  • I. INTRODUCTION: Automated sleep staging extracts features from PSG signals and classifies 30-second epochs into sleep stages.Expert methods use hand-crafted features, whereas deep learning learns features and classification jointly.
  • I. INTRODUCTION: Expert approaches combine spectral or temporal PSG features with classifiers, increasingly modeling successive epochs using recurrent neural networks.These approaches estimate the current sleep stage from engineered signal representations and temporal context.
  • I. INTRODUCTION: Deep-learning approaches use architectures such as CNNs and RNNs to learn signal features and dependencies across contiguous sleep epochs.Prior work includes single-channel and multimodal models with multiscale feature extraction.
  • I. INTRODUCTION: Using many PSG signals can improve coverage, but larger models increase computational cost and overfitting risk while reducing data efficiency.Many existing models also rely on a single PSG signal, which may limit estimated-stage accuracy.
  • I. INTRODUCTION: Most automated methods are evaluated against one manual scorer, while publicly available multi-scorer benchmarks and consensus datasets remain scarce.Performance can vary greatly across datasets, complicating fair comparison.

A. Datasets

The study provides healthy and OSA PSG datasets and evaluates scorings against a multi-expert consensus. Its consensus procedure uses majority voting with a Soft-Agreement rule for ties.

  • A. Datasets: The datasets contain 25 healthy volunteers and 55 patients with suspected sleep-related breathing disorders, excluding disorders other than OSA.DOD-H comprises healthy sleepers; DOD-O comprises patients evaluated at Stanford Sleep Medicine Center.
  • A. Datasets: The datasets’ demographic summaries report averages across subjects for DOD-H and DOD-O.Table I provides the study’s demographic overview for both datasets.
  • A. Datasets: Performance is evaluated against a consensus of multiple human scorers to reduce known inter-scorer variability.The framework uses a majority vote from sleep experts rather than a single annotation.
  • A. Datasets: Soft-Agreement measures how closely one staging agrees with the other scorers and equals 1 when it always matches the majority vote or a tied majority.The metric supports comparison with multiple annotations.

Other metrics:

The evaluation merges multiple scorings into an epoch-level consensus and measures agreement with F1, accuracy, Cohen’s kappa, and Soft-Agreement. SimpleSleepNet uses spectrogram-based signal processing with learned dimensionality reduction.

  • Other metrics:: Each consensus label is the epoch-level majority vote, with ties resolved using the scorer having the highest record-level Soft-Agreement.This tie rule differs from selecting one scorer using agreement over the entire dataset.
  • Other metrics:: F1 is computed per class and averaged according to class proportions, alongside accuracy and Cohen’s kappa.Cohen’s kappa uses observed agreement relative to chance agreement.
  • Other metrics:: SimpleSleepNet combines recent architectural components, including channel-wise dropout, linear signal recombination, positional embedding, and smaller hidden layers.These choices reduce its parameter count relative to other approaches.
  • Other metrics:: SimpleSleepNet computes STFT representations, randomly zeros signals during training, and applies signal-wise normalization.The STFT is computed on each preprocessed epoch, with channel-wise dropout probability p_kill used to reduce overfitting.
  • Other metrics:: The model linearly reduces frequency bins and input signals before applying dropout to the projected spectrogram.Frequency and signal projections independently map the original representation into smaller dimensions.

2) Signals and frequencies reduction:

The network encodes each epoch with a bidirectional GRU and attention, augments it with global temporal positional information, and processes neighboring epochs with a SkipGRU.

  • 2) Signals and frequencies reduction:: A bidirectional GRU and attention layer aggregate the projected spectrogram across time into a current-epoch representation.Attention reweights and sums GRU hidden states along the time axis.
  • 2) Signals and frequencies reduction:: Positional embeddings encode the epoch’s global location using a scaled epoch index and five cyclic indexes.The cyclic periods are 30, 60, 90, 120, and 150 epochs.
  • 2) Signals and frequencies reduction:: The architecture diagram identifies h_t−1 and h′_t−1 as previous-epoch hidden states, h_t+1 and h′_t+1 as next-epoch states, and a_t as the current-epoch embedding.These states illustrate how the sequential model incorporates neighboring epochs.
  • 2) Signals and frequencies reduction:: The positional embedding is projected and concatenated with the attention output to form the current epoch representation.The resulting vector is denoted a_t and supplies the sequential modeling stage.

5) Sequence encoder and classifier:

SimpleSleepNet encodes a temporal context around a central epoch with a bidirectional SkipGRU, then predicts sleep-stage probabilities through a softmax classifier.

  • Sequence encoder: A temporal context of 2k + 1 epochs around central epoch t is processed by a two-layer bidirectional SkipGRU with skip connections.The encoder uses m2 hidden units and is described as similar to DeepSleepNet’s sequence encoder with additional intermediary skip connections.
  • Classifier: The forward and backward SkipGRU outputs are concatenated, dropout is applied, and the resulting sequence is passed to a final softmax layer.The classifier outputs sleep-stage probabilities for the encoded epochs.
  • Loss function: SimpleSleepNet modifies its loss because it produces multiple sleep-stage estimates rather than one estimate per input sequence.The loss averages terms across the 2k + 1 epochs in the input spectrogram sequence.

D. Evaluation

At evaluation, SimpleSleepNet aggregates multiple predictions for each epoch using a geometric mean, then selects the stage with the highest aggregated probability.

  • D. Evaluation: Multiple predictions for an epoch are aggregated using the geometric mean of the available sleep-stage probabilities.Aggregation is performed for an epoch t given a temporal context k.
  • D. Evaluation: The evaluated sleep stage is the argmax of the aggregated probability vector across the six stage indices.The predicted label is obtained after aggregation rather than from a single prediction.
  • D. Evaluation: The benchmark reimplemented recent automated sleep-staging approaches and SimpleSleepNet in PyTorch, with code released publicly for reproducibility.The approaches were selected because they reported strong performance on publicly available datasets.

2) Tsinalis et al.

The benchmark includes CNN-, recurrent-, and spectrogram-based sleep-staging approaches that differ in input context, feature extraction, and temporal modeling.

  • 2) Tsinalis et al.: Tsinalis et al.’s CNN uses 630 seconds of raw signals, applies two convolution-plus-pooling layers, and classifies the central epoch.For fair comparison, the model is trained on all PSG signals instead of only a single channel.
  • 2) Tsinalis et al.: Chambon et al.’s model handles multimodal signals by first learning virtual signals through a convolution across channels before subsequent convolution and pooling blocks.The initial convolution models dependencies among the different input signals.
  • 2) Tsinalis et al.: DeepSleepNet encodes each epoch with two convolutional networks using different filters and pooling sizes, then models epoch sequences with a recurrent network.The two branches emphasize temporal and frequency information differently.
  • 2) Tsinalis et al.: SeqSleepNet reduces spectrogram frequency bins with a learned filter bank, encodes epochs with a bidirectional LSTM and attention, and models consecutive epochs afterward.Its attention layer compresses the temporal dimension into a single epoch representation.
  • 2) Tsinalis et al.: The reported implementation uses n = 30 frequency filters, retains all channels, sets dropout probabilities to 0.5, and uses 25 hidden units in both encoders.The attention context size is also set to 25.

B. Benchmark setup

The benchmark evaluates automated methods and human scorers against consensus hypnograms built from multiple scorers, with SimpleSleepNet’s full model and ablations assessed across DOD-H and DOD-O.

  • B. Benchmark setup: Consensus hypnograms are formed by majority vote, with ties resolved using the scorer with the highest record-level Soft-Agreement.Human scorers are evaluated against consensuses from the other four scorers, while automated methods use consensuses from the four overall best scorers.
  • C. Benchmark on DODO and DODH: Figure 2 compares row-normalized confusion matrices for SimpleSleepNet and human scorers across DOD-H and DOD-O, with epoch counts shown in parentheses.The figure contrasts predictions against consensus hypnograms constructed from different scorer groups.
  • C. Benchmark on DODO and DODH: SimpleSleepNet achieves the best considered performance on both datasets, exceeding human and other automated approaches; its F1 is 89.9 ± 4.1% on DOD-H.On DOD-H, it is better than the best scorer and has lower standard deviation.
  • C. Benchmark on DODO and DODH: Human scorers average 86.8 ± 7.6% F1 on DOD-H and 84.8 ± 8.6% on DOD-O, with greater stage-level variability on DOD-O.N1 is the lowest-F1 stage on both datasets, while N3 is notably lower on DOD-O.
  • D. SimpleSleepNet ablation study: Removing frequency reduction or channel dropout most affects performance on both datasets, while filtering and the filter bank have substantial effects specifically on DOD-O.The full model has the best overall ranking on both datasets.

E. Influence of the experimental setup

SimpleSleepNet performance improves with larger models, while single-channel training reduces F1 relative to full-montage training but remains comparable to average human scoring.

  • 1) Model size:: SimpleSleepNet-Small uses approximately three times fewer parameters and SimpleSleepNet-Large approximately three times more parameters than the original model.The benchmark includes baseline models, human scorer averages, and ablated SimpleSleepNet variants for comparison.
  • 1) Model size:: F1 increases by 0.5% on DOD-H and 0.7% on DOD-O with SimpleSleepNet-Large, while the Small variant reduces F1 by 0.6% and 1.1%, respectively.Larger models also reduce performance variance significantly on both datasets.
  • 2) Performances on a single EEG derivation:: Single-channel training lowers F1 by 3.9 percentage points on DOD-O and 3.3 points on DOD-H versus training on the full PSG montage.Despite the reduction, the single-channel model remains on par with the average human scorer.

3) Size of the training set:

SimpleSleepNet reaches human-level performance with relatively small training sets, benefits from temporal context, and transfers imperfectly between DOD-H and DOD-O.

  • Temporal context:: F1 rises as temporal context increases from 1 to 7 epochs and then plateaus, starting at 85.5 ± 6.5% on DOD-H and 83.0 ± 11.6% on DOD-O.A single epoch already provides decent performance on both datasets.
  • Direct transfer learning:: Cross-dataset transfer reduces F1 from 89.9% to 84.8% when training on DOD-O and testing on DOD-H, and from 88.3% to 62.6% in the reverse direction.The standard deviation nearly doubles for DOD-O-to-DOD-H transfer, and the reverse transfer shows the larger performance drop.

G. Benchmark on external dataset

External-dataset benchmarks show that SimpleSleepNet performs best on Sleep EDF, while DeepSleepNet leads MASS; broader analyses indicate automated methods can match or exceed human scoring under aligned conditions.

  • SimpleSleepNet achieves the best performance on Sleep EDF, while DeepSleepNet has the highest Macro-F1 on MASS, closely followed by SimpleSleepNet.The benchmark uses epoch-wise Macro-F1 on both external datasets.
  • DOD-O exhibits lower, more variable accuracy and more scorer ties than DOD-H, reflecting greater difficulty associated with fragmented OSA sleep.Models require more recordings to reach human performance on DOD-O than on DOD-H.
  • Transfer learning reveals a practical limitation: automated staging generalizes poorly outside the population or PSG device used for training.The authors propose mixed cohorts, devices, and scorer annotations to improve generalization, while noting that differing modalities and missing signals remain unresolved.
  • SimpleSleepNet, DeepSleepNet, and SeqSleepNet outperform the average human scorer on both DOD-O and DOD-H.
  • Performance comparisons on external datasets remain limited because those datasets were scored by a single expert, allowing possible overfitting to that scorer.
  • Data-driven feature extraction generally outperforms expert feature extraction, especially on the more variable DOD-O and SleepEDF-Extended datasets.
  • SimpleSleepNet outperforms the best human scorer and other sleep-staging models on DOD-O and DOD-H while using significantly fewer parameters.It reaches close-to-human performance with approximately 10 recordings, and its ablation study supports the contribution of its building blocks on DOD-O.

V. CONCLUSION

The study introduces open multi-scored sleep datasets and a methodology for evaluating automated staging against multiple human scorers. SimpleSleepNet and other leading automated approaches often match or exceed human-scoring performance across the reported datasets.

  • The study introduces two open multi-scored sleep staging datasets covering 25 healthy subjects and 55 patient nights with OSA.
  • SimpleSleepNet outperforms previous state-of-the-art models and human scorers on both datasets and two frequently benchmarked datasets.
  • Recent automated sleep staging methods are often on par with average human scorers, while the best automated methods exceed the best human scorer.
Loading 1911.03221v4…