Source-linked AI summary
Inter-database validation of a deep learning approach for automatic sleep scoring
Diego Alvarez-Estevez, Roselyne M. Rijsman
TL;DR
Manual sleep staging is time-consuming, and database-specific validation offers limited evidence about performance on new sources. This study evaluates a deep-learning approach using local and external validation and an ensemble of local models. The authors report good general performance against human and automatic references, with improved generalization from the architecture and ensemble.
Problem
Manual PSG scoring is time-consuming, while local testing and database-specific overfitting provide limited evidence about generalization to external databases.
Source: §1 · p. 1Method
The study evaluates a deep-learning sleep-staging approach in local and external database scenarios and tests an ensemble of individual local models.
Source: §1 · p. 2Results
The CNN+LSTM architecture improved generalization in local and external validation, and ensembles of local models generalized better than individual local models alone.
Source: §5 · pp. 15–16Takeaways & Limitations
The reported ensemble results motivate further investigation of this scalable and flexible design for inter-database generalization.
Source: §5 · p. 16Comparisons with other studies require caution because validation protocols, recording counts, and patient selections differ; only the authors’ earlier study uses the exact same database benchmark.
Source: §5 · p. 16
Abstract
from arXiv · showhide
In this work we describe a new deep learning approach for automatic sleep staging, and carry out its validation by addressing its generalization capabilities on a wide range of sleep staging databases. Prediction capabilities are evaluated in the context of independent local and external generalization scenarios. Effectively, by comparing both procedures it is possible to better extrapolate the expected performance of the method on the general reference task of sleep staging, regardless of data from a specific database. In addition, we examine the suitability of a novel approach based on the use of an ensemble of individual local models and evaluate its impact on the resulting inter-database generalization performance. Validation results show good general performance, as compared to the expected levels of human expert agreement, as well as state-of-the-art automatic sleep staging approaches
1. INTRODUCTION
Sleep staging is a clinically important but labor-intensive task, while automated systems have struggled to generalize beyond the databases used in development. This work evaluates a deep-learning sleep-staging approach across local and external database scenarios and examines an ensemble of local models.
1. INTRODUCTION
- Automated sleep staging could reduce the time and human resources required for manual scoring, but analysis remains demanding as sleep-study demand grows.PSG examinations typically involve 8–24 hours of continuous recording, and rising demand strains sleep centers.Source: §1 · p. 1
- Limited validation and database-specific overfitting have left many systems unable to generalize across sources of sleep-staging variability.Sources include differences in subjects, signal acquisition, and expert interpretation.Source: §1 · pp. 1–2
- The study evaluates a deep-learning approach using independent local and external validation, and tests whether ensembles of local models improve inter-database generalization.It contextualizes results against reported human-expert agreement and state-of-the-art automatic systems.Source: §1 · p. 2
2. MATERIALS AND METHODS
The methods combine signal preprocessing, CNN-based epoch feature extraction, and optional LSTM sequence modeling, then evaluate local, external, and ensemble predictions across sleep databases. The experiments compare architecture and preprocessing configurations using a shared training protocol and Cohen’s kappa.
2.1. Neural Network Architecture
- The proposed sleep-staging architecture comprises preprocessing, a Convolutional Neural Network (CNN), and a Long-Short Term Memory (LSTM) module.Figure 1 presents the preprocessing steps and general CNN-LSTM architecture.Source: §2.1 · p. 2, Figure 1 · p. 3
2.1.1. Pre-processing
- Preprocessing resamples two EEG, one chin EMG, and one EOG channel to 100 Hz, segments them into 30-second epochs, and applies Gaussian amplitude normalization.Optional filtering is applied to raw signals before resampling to remove noise and artifacts that may impair generalization.Source: §2.1.1 · p. 3
2.1.2. CNN block
- The CNN processes 4x3000 inputs and either produces an epoch-level sleep-stage output or supplies features to the LSTM in the CNN-LSTM configuration.For CNN-only classification, the model outputs posterior probabilities for five sleep stages.Source: §2.1.2 · pp. 3–4
2.1.3. LSTM block
- The LSTM models epoch-sequence context by processing CNN feature vectors from neighboring epochs to score the current epoch.The sequence length L determines how many epoch feature vectors are assembled around the epoch under evaluation.Source: §2.1.3 · p. 4
2.2. Ensemble of local models
- The ensemble combines independently trained local models by majority vote to predict data from external sources, allowing new local models to be added as datasets evolve.The approach is proposed as a scalable alternative to training on pooled data from all sources, which requires more resources and can reduce flexibility.Source: §2.2 · pp. 4–5
2.3. Experimental design
- The experiments compare local testing, external-database prediction, and ensembles that exclude the target database’s model, alongside architecture variants and filtering conditions.Models use an 80/20 training/testing split, a validation subset for early stopping, and Cohen’s kappa as the reference score.Source: §2.3 · pp. 5–6
2.4. Datasets
- The benchmark combines mostly public PSG datasets with the nonpublic HMC set, and retains all selected recordings while aligning reference stage labels to AASM.Included sources described here include HMC, Dublin, SHHS, and Telemetry.Source: §2.4 · pp. 6–7
3. RESULTS
The results section reports sleep-staging performance across local validation, external validation, and an ensemble comparison, with results expressed as kappa against clinical reference scorings.
3. RESULTS
- Table 1 reports individual-model kappa on local testing partitions, with dataset-specific training, validation, and testing splits and effective training iterations.Source: §3 · p. 7, Table 1 · p. 8
- Table 2 evaluates individual local models on external datasets; its main diagonal represents biased predictions on each model’s own complete dataset.Source: §3 · p. 9, Table 2 · p. 9
- Table 3 compares individual-model and ensemble performance in local and external validation scenarios, reporting kappa against clinical reference scorings.Source: Table 3 · p. 10, §3 · p. 10
- Table 4 aggregates performance across datasets for individual models on local testing sets, individual models on external datasets, and ensemble predictions on external datasets.Source: Table 4 · p. 12, §3 · p. 12
4. ANALYSIS OF EXPERIMENTAL DATA data
Across tested datasets, local validation overestimates external generalization, while the ensemble improves external performance. The CNN_LSTM_5 model performs in the range of reported human agreement and the upper range of automatic-scoring results.
4.1. Best model (CNN vs CNN-LSTM)
- CNN_LSTM_5 achieves the best generalization across tested datasets, and CNN-LSTM models outperform matched CNN-only models in both local and external scenarios.Performance improves as sequence length increases to L = 5, then decreases; increasing sequence length does not improve CNN-only prediction.Source: §4.1 · p. 12
4.2. Signal prefiltering
- Filtering effects are inconclusive or favorable across datasets except Dublin, where prediction difficulties appear under external but not local validation.The authors advise validation with heterogeneous, independent external data to reveal database-specific effects.Source: §4.2 · pp. 12–13
4.3. Database generalization performance
- Local generalization estimates exceed external performance across tested datasets and configurations, with external kappa decreasing by 0.21–0.34.The local estimate also remains an upper bound for the external ensemble, with kappa differences of 0.12–0.26.Source: §4.3 · p. 13
- The ensemble improves external inter-dataset generalization over individual models, increasing kappa by 0.08–0.10.The improvement is reported consistently across experimental simulations.Source: §4.3 · p. 13
4.4. Analysis in the context of the expected human performance
- Local performance (κ = 0.78–0.83) is in the range of expected human agreement (0.73–0.87); external performance (0.59–0.69) is within the reported human range (0.46–0.89).An external human baseline for the databases used in this work was unavailable in the literature.Source: Table 5 · p. 13, §4.4 · pp. 13–14
4.5. Analysis in the context of other automatic approaches
- Locally, the approach falls in the upper range of automatic-scoring results: κ = 0.78–0.83 versus 0.44–0.84 in other works.Across the full external dataset set, the new architecture improves on the authors’ earlier method, from κ = 0.50 to 0.63; local mean κ rises from 0.60 to 0.80.Source: Table 6 · p. 14, §4.5 · pp. 14–15
5. DISCUSSION
The discussion argues that external multi-database validation is essential because local performance does not reliably transfer, while reporting good general performance against human and automatic references. It also identifies gains from the CNN+LSTM design and local-model ensembles, alongside unresolved filtering effects and local–external performance gaps.
5. DISCUSSION
- Local testing risks source-specific overfitting; performance tends to decrease when a model encounters similar examples from an external database.The discussion therefore treats external multi-database prediction as a necessary validation step.Source: §5 · p. 15
- The validation included six independent databases, five freely available online, making it the largest study of its kind to the authors’ knowledge.The authors selected freely available databases to enhance reproducibility.Source: §5 · p. 15
- The architecture showed good general performance against human and automatic references, but cross-study comparisons require caution because protocols, recording counts, and patient selections differ.Only the authors’ earlier study used the exact same database benchmark.Source: §5 · pp. 15–16
- CNN_LSTM_5 performed best; sequence learning with an LSTM output block was more convenient than lengthening CNN-only inputs, with no benefit beyond five epochs.The authors also report improved scalability because the CNN input space is larger than the LSTM input feature space.Source: §5 · p. 16
- The proposed filtering pipeline had no consistent effect across tested datasets, leaving its value and the causes of inter-dataset variability unresolved.The authors call for more heterogeneous, independent samples from multiple external sources to support generalizable conclusions.Source: §5 · p. 16
- Ensembles of local models generalized better than individual local models, while offering scalability and flexibility.The discussion likens local models to source-specific experts whose outputs can contribute to consensus.Source: §5 · p. 16
- Ensembling improved generalization, but performance still degraded relative to local testing, and human-agreement references covered only local validation for HMC, SHHS, and ISRUC.The authors call for external human-agreement baselines and further study of the local–external gap.Source: §5 · p. 16
- Future work targets hyperparameterization and preprocessing, including unresolved effects of 100 Hz sampling homogenization and the selected signal derivations.The variability of results for the Dublin dataset with the proposed filtering pipeline also remains unclear.Source: §5 · p. 16