XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging

Huy Phan; Oliver Y. Chén; Minh C. Tran; Philipp Koch; Alfred Mertins; Maarten De Vos

Source-linked AI summary

XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging

Huy Phan, Oliver Y. Chén, Minh C. Tran, Philipp Koch, Alfred Mertins, Maarten De Vos

arXiv:2007.05492v4Jul 8, 2020eess.SPcs.LG

TL;DR

Automatic sleep staging must scale beyond labor-intensive manual scoring, but combining raw signals with time-frequency images is difficult because naïve multi-view learning can underperform. XSleepNet learns joint sequence-to-sequence features while adapting each view’s learning pace through generalization- and overfitting-aware gradient blending. Across five databases, it consistently outperformed single-view and naïve-fusion baselines and improved prior state-of-the-art results.

Problem
Manual sleep scoring is difficult to scale for millions affected by sleep deprivation or disorders, and effective learning from raw and time-frequency views remains poorly understood.
Method
XSleepNet is a sequence-to-sequence model that learns joint raw-signal and time-frequency representations while blending view gradients according to their generalization or overfitting behavior.
Results
XSleepNet consistently outperformed single-view and naïve-fusion baselines across five databases of different sizes and improved prior state-of-the-art results.
Takeaways & Limitations
Adaptive gradient blending retained the representation power of both views in joint features that represented the underlying data distribution better than the compared single-view and naïve-fusion approaches.
Takeaways & Limitations
Noisy minibatch loss curves can produce varying loss weights and unstable learning, particularly when the training data is small.

Abstract

from arXiv · show

Automating sleep staging is vital to scale up sleep assessment and diagnosis to serve millions experiencing sleep deprivation and disorders and enable longitudinal sleep monitoring in home environments. Learning from raw polysomnography signals and their derived time-frequency image representations has been prevalent. However, learning from multi-view inputs (e.g., both the raw signals and the time-frequency images) for sleep staging is difficult and not well understood. This work proposes a sequence-to-sequence sleep staging model, XSleepNet, that is capable of learning a joint representation from both raw signals and time-frequency images. Since different views may generalize or overfit at different rates, the proposed network is trained such that the learning pace on each view is adapted based on their generalization/overfitting behavior. In simple terms, the learning on a particular view is speeded up when it is generalizing well and slowed down when it is overfitting. View-specific generalization/overfitting measures are computed on-the-fly during the training course and used to derive weights to blend the gradients from different views. As a result, the network is able to retain the representation power of different views in the joint features which represent the underlying distribution better than those learned by each individual view alone. Furthermore, the XSleepNet architecture is principally designed to gain robustness to the amount of training data and to increase the complementarity between the input views. Experimental results on five databases of different sizes show that XSleepNet consistently outperforms the single-view baselines and the multi-view baseline with a simple fusion strategy. Finally, XSleepNet also outperforms prior sleep staging methods and improves previous state-of-the-art results on the experimental databases.

1 INTRODUCTION

Automatic sleep staging is needed to scale assessment and monitoring because sleep disorders affect millions, while manual PSG scoring is labor-intensive. Although raw signals and time-frequency images offer complementary input views, naïve multi-view learning can underperform single-view training, motivating XSleepNet.

Motivation: Millions of people experience sleep deprivation or disorders, creating demand for scalable automated sleep assessment, diagnosis, and longitudinal monitoring.Manual sleep scoring is difficult to scale because it is labor-intensive and time-consuming.
Motivation: Two hours of expert work for one overnight PSG recording contrasts with machine processing in a few seconds.
Prior work: Automatic sleep staging has advanced rapidly alongside larger annotated databases and deep-learning methods approaching sleep-expert performance.
Prior work: Existing systems primarily process either one-dimensional raw signals or two-dimensional time-frequency images derived from those signals.Time-frequency images are generally treated as higher-level representations, but neither view is universally superior.
Contribution: Simple concatenation of multiple input views is suboptimal and can make a multi-view network worse than the best single-view model.XSleepNet addresses this by adapting view contributions through gradient blending based on generalization and overfitting behavior.

2 DESIGN PRINCIPLES

XSleepNet uses two deliberately different network streams for raw signals and time-frequency images, then regulates their learning through generalization- and overfitting-aware gradient blending. This design addresses why simple concatenation can generalize worse than the best single-view stream.

Architecture: XSleepNet contains separate streams for raw signals and time-frequency images, with design principles targeting robustness to training-data size and view complementarity.
Robustness: The raw-data stream has 5.6×10^6 parameters, about 35 times more than the time-frequency stream’s 1.6×10^5 parameters.The differing footprints are intended to match network capacity to available training data.
Complementarity: The raw-signal stream uses a CNN and the time-frequency stream an RNN because their sleep-staging outputs show complementary strengths.On MASS, the raw-signal CNN favors N3, while the time-frequency RNN performs better for N1 and REM.
Adaptive training: Generalization-aware training accelerates a stream that generalizes well and discourages one that is overfitting.This requires access to stream-specific gradient flows rather than only concatenated joint features.
Adaptive training: Simple fusion can average validation losses and generalize worse than the best single-view stream because it cannot regulate each stream’s learning pace separately.

3 XSLEEPNET

XSleepNet combines raw-signal and time-frequency inputs through distinct network streams and sequence modeling, then adaptively blends their classification gradients during training. Its design targets robustness to training-data size, complementary representations, and improved generalization across views.

Architecture: XSleepNet encodes raw signals and time-frequency representations in separate streams before applying long-term inter-epoch sequential modeling.The raw input is handled by a fully convolutional feature map, while the time-frequency input uses learnable filterbanks, attention, and recurrent processing.
Architecture: The raw stream uses a CNN and the time-frequency stream an RNN to promote complementary representations and sleep-stage predictions.The paper notes that raw-signal CNNs and time-frequency RNNs can favor different sleep stages.
Architecture: Three classification branches produce view-specific and joint outputs, with the joint output used as the final prediction during evaluation.The branches operate on the raw-stream, time-frequency-stream, and concatenated joint features.
Adaptive gradient blending: XSleepNet adaptively weights branch losses using generalization and overfitting measures to regulate learning across network streams.The generalization measure captures gained information about the target distribution, while overfitting measures the gap between training-set and target-distribution gains.
Adaptive gradient blending: XSleepNet2 estimates loss behavior from tangent directions fitted robustly to noisy curves, addressing unstable weights and suboptimal learning in XSleepNet1.The tangent-based approximations are described as second-order, whereas the earlier approach is first-order.

4 EXPERIMENTS

Experiments evaluate XSleepNet and baselines across five databases of varying sizes and channel combinations. XSleepNet generally improves over separately trained single-view models, naive fusion, and prior methods, while revealing dataset-size effects on model performance.

4.2 Experimental setup: The experiments used established literature data splits and extracted time-frequency inputs from overlapping two-second windows using a 256-point FFT.Signals were resampled to 100 Hz, and time-frequency images were normalized before training and testing.
4.4 Experimental results: 1.6×10^5 versus 5.6×10^6 parameters explains part of the baseline reversal: ARNN+RNN often performs better on smaller databases, whereas FCNN+RNN commonly improves on larger ones.The authors also attribute discrepancies to differing CNN/RNN pattern preferences and the target cohort or channel combination.
4.4 Experimental results: Naive Fusion sometimes improves over single-view baselines, but can also fall below both, indicating that simple concatenation does not guarantee a performance gain.The authors suggest asynchronous learning behavior between views as a possible reason for these divergent results.
4.4 Experimental results: 0.9%, 0.6%, and 1.1% absolute accuracy gains were obtained by XSleepNet1 over Naive Fusion, ARNN+RNN, and FCNN+RNN; XSleepNet2 reached 1.2%, 0.9%, and 1.4%.The models outperformed the baselines and prior works across databases and channel combinations, except for XSleepNet1’s modest result on SleepEDF-20.
4.4 Experimental results: Class-wise improvements from XSleepNet1 and XSleepNet2 occurred across many databases and channel combinations and often spread across all five sleep stages.On SHHS, naive fusion showed behavior similar to XSleepNets because the large dataset strongly regularized training.

5 DISCUSSION

XSleepNet2 is more stable than XSleepNet1, especially on the smallest database, while both perform strongly on medium and large databases. The training behavior illustrates how adaptive weighting addresses view-specific overfitting, and the model remains smaller than DeepSleepNet.

Performance and stability: XSleepNet2’s superiority is ubiquitous on SleepEDF-20, outperforming all baselines and XSleepNet1, whereas XSleepNet1 falls 0.8% below ARNN+RNN on EEG·EOG.Both models perform remarkably well on medium or large databases, except MASS (EEG·EOG·EMG), where XSleepNet2 is not better than XSleepNet1.
Training behavior: FCNN+RNN converged and began overfitting after 105 training steps, while ARNN+RNN’s validation loss showed no overfitting signs in the MASS example.Under naive fusion, the overfitting stream can affect joint training, motivating adaptive gradient blending.
Training behavior: On SHHS, Naive Fusion performed consistently better than the single-view baselines because the largest database strongly regularized the networks against overfitting.The authors therefore anticipate that model size can be safely increased when data are sufficiently large.
Model footprint and scope: XSleepNet has 5.8×10^6 parameters, four times fewer than DeepSleepNet’s 22.9×10^6 parameters despite using two network streams.The authors also suggest that multiple EEG channels could further improve XSleepNet’s performance.
Training behavior: Figure 5 compares validation losses and adaptive weights for baselines and the three classification branches of XSleepNet1 and XSleepNet2 during one MASS fold.The XSleepNet2 adaptive weights were denoised with a 10-point moving-average filter before plotting.
Training behavior: Figure 6 compares validation-loss trajectories for the baselines and the three classification branches of XSleepNet1 and XSleepNet2 on SHHS.The figure provides the validation-loss view corresponding to the large-database behavior discussed above.

6 CONCLUSIONS

The paper presents XSleepNet as a sequence-to-sequence architecture that jointly learns from raw signals and time-frequency images. Adaptive gradient blending improves multi-view representation learning, with XSleepNet2 providing more stable performance than XSleepNet1 across five databases.

Contributions: XSleepNet uses two network streams to learn jointly from raw signals and time-frequency inputs for automatic sleep staging.The architecture is designed for robustness to training-data size, stream complementarity, and stream-specific generalization or overfitting.
Contributions: Adaptive weights derived from generalization/overfitting measures blend gradients so generalizing streams are encouraged and overfitting streams are discouraged.The two approximation approaches produce XSleepNet1 and XSleepNet2.
Results: Across five databases, XSleepNet outperformed the developed baselines and existing work, while XSleepNet2 delivered more stable performance than XSleepNet1.XSleepNet2 relies on a second-order approximation, whereas XSleepNet1 uses a first-order approximation.