PaperlayerView exact revision on arXiv

Source-linked AI summary

SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification

Huy Phan, Kaare Mikkelsen, Oliver Y. Chén, Philipp Koch, Alfred Mertins, Maarten De Vos

arXiv:2105.11043v3cs.LGeess.SP

TL;DR

Automatic sleep staging remains difficult to adopt clinically because deep-learning models are viewed as black boxes, motivating models that explain decisions and quantify uncertainty. SleepTransformer uses a transformer-only sequence-to-sequence architecture, self-attention visualizations, and entropy-based confidence estimation; it performs comparably to existing methods on databases of different sizes and can identify low-confidence epochs for review.

  • Problem

    Clinical adoption of automatic sleep staging is limited by skepticism toward black-box models, while sleep-stage ambiguity makes interpretability and uncertainty quantification important.

  • Method

    SleepTransformer is a transformer-based sequence-to-sequence model that uses self-attention for epoch- and sequence-level interpretability and entropy to quantify decision uncertainty.

  • Results

    SleepTransformer performs comparably to existing methods on the large SHHS and smaller SleepEDF-78 databases, while its uncertainty estimates identify low-confidence epochs enriched for misclassifications.

  • Takeaways & Limitations

    Attention visualizations expose attended EEG features and neighboring-epoch influence, while entropy can support deferring uncertain epochs to human experts.

  • Takeaways & Limitations

    The entropy-based uncertainty approach addresses knowledge uncertainty but does not cover all uncertainty types, including model uncertainty.

Abstract

from arXiv · show

Background: Black-box skepticism is one of the main hindrances impeding deep-learning-based automatic sleep scoring from being used in clinical environments. Methods: Towards interpretability, this work proposes a sequence-to-sequence sleep-staging model, namely SleepTransformer. It is based on the transformer backbone and offers interpretability of the model's decisions at both the epoch and sequence level. We further propose a simple yet efficient method to quantify uncertainty in the model's decisions. The method, which is based on entropy, can serve as a metric for deferring low-confidence epochs to a human expert for further inspection. Results: Making sense of the transformer's self-attention scores for interpretability, at the epoch level, the attention scores are encoded as a heat map to highlight sleep-relevant features captured from the input EEG signal. At the sequence level, the attention scores are visualized as the influence of different neighboring epochs in an input sequence (i.e. the context) to recognition of a target epoch, mimicking the way manual scoring is done by human experts. Conclusion: Additionally, we demonstrate that SleepTransformer performs on par with existing methods on two databases of different sizes. Significance: Equipped with interpretability and the ability of uncertainty quantification, SleepTransformer holds promise for being integrated into clinical settings.

I. INTRODUCTION

SleepTransformer targets clinical skepticism toward black-box automatic sleep staging by combining transformer-based modeling with interpretability and uncertainty quantification. It uses attention to explain decisions and entropy to identify uncertain epochs for expert review.

  • Motivation: Manual sleep scoring is labor-intensive, taking experts about two hours per overnight polysomnography recording.Automating 30-second epoch classification could complete the task in seconds.
  • Motivation: Clinical adoption remains limited because deep-learning sleep-staging models are viewed as black boxes.The paper identifies interpretability and uncertainty quantification as two obstacles to collaborative machine-assisted scoring.
  • Proposed approach: SleepTransformer is a convolution- and recurrent-free sequence-to-sequence model whose backbone uses transformer self-attention for epoch- and sequence-level modeling.The paper presents it as a transformer-only sequence-to-sequence model for automatic sleep scoring.
  • Interpretability: Self-attention scores support interpretability by highlighting sleep-relevant EEG features and showing neighboring epochs’ influence on a target epoch.The sequence-level view is intended to reflect the contextual reasoning used in manual scoring.
  • Uncertainty quantification: Entropy of the model’s multi-class output distribution provides a concrete uncertainty measure for its decisions.The distribution is most confident when concentrated on one class and least confident when flat.

II. MATERIALS

The experiments use the large SHHS database and the smaller SleepEDF-78 database, while the transformer backbone is built from multi-head attention and position-wise feed-forward modules. These modules include learnable projections, residual connections, and normalization layers.

  • Databases: The experiments use SHHS, a large-scale multi-center database containing 5,791 subjects from its first visit.The study uses C4-A1 EEG and R&K-based scoring with N3/N4 merged into N3 and movement or unknown epochs discarded.
  • Databases: SleepEDF-78 contains 78 healthy Caucasian subjects with two consecutive day-night polysomnography recordings for most subjects.Its epochs were manually scored by sleep experts using the R&K standard across eight categories.
  • Transformer backbone: The transformer encoder comprises multi-head attention and a position-wise feed-forward network.The encoder is used here rather than the decoder’s left-context-only generation architecture.
  • Transformer backbone: Multi-head attention applies parallel learnable linear projections, performs scaled dot-product attention, concatenates the heads, and linearly projects the result.The input is Z ∈ R^l×d, and the attentive output has the same stated dimensional form.
  • Transformer backbone: The position-wise feed-forward network uses two linear transformations with a ReLU activation, alongside residual and normalization layers.Its transformations use learnable weight matrices and biases.

IV. SLEEPTRANSFORMER

SleepTransformer is a sequence-to-sequence sleep-staging model that uses transformer-based processing for both individual EEG epochs and epoch sequences, without convolutional or recurrent components.

  • SleepTransformer processes sequences of 30-second EEG epochs for five-stage sleep staging.Each epoch is represented as a time-frequency image with 29 time frames and 128 frequency bins, while labels use five-class one-hot encoding.
  • The model uses transformers as the backbone for both intra-epoch and inter-epoch processing.This design replaces the convolutional and recurrent components commonly used in existing deep-learning sleep-staging models.
  • SleepTransformer follows a sequence-to-sequence sleep-staging framework.The supplied architecture figures illustrate the transformer encoder and the SleepTransformer model.

A. Epoch transformer

The epoch transformer converts each EEG epoch’s time-frequency representation into a compact feature vector using positional encoding, stacked transformers, and learned attention-weighted aggregation.

  • A. Epoch transformer: The epoch transformer treats a time-frequency image as a sequence of spectral columns and adds positional encodings to preserve order information.The positional encoding uses sine and cosine functions.
  • A. Epoch transformer: A heap of NE transformers models the positional-encoded spectral-column sequence.The resulting representation is reduced to a compact feature vector for epoch-wise modeling.
  • A. Epoch transformer: The final transformer output columns are combined through learned softmax attention weights to represent the input epoch.The derived feature vector x is in R^F, and the attention weights are learned by a softmax attention layer.
  • A. Epoch transformer: The attention layer uses a learnable weight matrix, bias, and trainable epoch-level context vector.The attention size is denoted by A.

B. Sequence transformer

The sequence transformer models interactions among epoch-wise feature vectors, preserving sequence order and producing per-epoch representations for final sleep-stage prediction.

  • B. Sequence transformer: Epoch-wise feature vectors are processed by a heap of NS transformers for inter-epoch modeling.This replaces the bidirectional recurrent networks typically used for sequence-level processing in related sequence-to-sequence models.
  • B. Sequence transformer: The sequence transformer adds sine-and-cosine positional encodings to the epoch feature sequence before transformer processing.The encoded sequence is represented as X̃, with positional encoding matrix Pseq.
  • B. Sequence transformer: The final sequence-transformer output provides one vector for each epoch in the input sequence.These vectors are subsequently presented to fully connected layers and a softmax layer to obtain the output sequence.
  • B. Sequence transformer: SleepTransformer uses two ReLU-activated fully connected layers followed by softmax classification and trains with sequence-level cross-entropy loss.

V. INTERPRETABILITY AND CONFIDENCE QUANTIFICATION

SleepTransformer uses self-attention scores to visualize which EEG features and neighboring epochs influence sleep-stage decisions, providing interpretability at both epoch and sequence levels.

  • V. INTERPRETABILITY AND CONFIDENCE QUANTIFICATION: Self-attention scores support two visualizations: an EEG attention heat map and an epoch-influence bar chart.The heat map identifies attended regions in the input EEG, while the bar chart represents neighboring-epoch contributions to a target epoch.
  • V. INTERPRETABILITY AND CONFIDENCE QUANTIFICATION: The EEG heat map aggregates attention scores from EpochTransformer heads to show which input regions contribute to learned representations.The resulting score vector is normalized to [0, 1].
  • V. INTERPRETABILITY AND CONFIDENCE QUANTIFICATION: The heat-map approach may obscure attention structures specific to individual attention heads.An alternative transforms the time-frequency input using attention score matrices immediately after the final EpochTransformer, omitting other nonlinear operations.
  • V. INTERPRETABILITY AND CONFIDENCE QUANTIFICATION: The epoch-influence chart uses SequenceTransformer attention scores to indicate how neighboring epochs contribute to representing a target epoch.This visualization is presented as resembling clinicians’ use of contextual neighboring epochs during manual scoring.

B. Entropy-based confidence quantification

SleepTransformer quantifies prediction confidence from the entropy of its five-class sleep-stage probability distribution, enabling low-confidence epochs to be deferred for manual review.

  • Entropy measures uncertainty in SleepTransformer’s five-class probability output, while normalized entropy converts uncertainty to a [0, 1] range.The output contains one probability for each of five sleep stages.
  • Confidence is highest when one sleep stage has probability 1 and lowest when the probability distribution is flat.For five-stage classification, confidence ranges from 0 for a uniform distribution to 1 for a one-hot distribution.
  • Low-confidence epochs can be selected by thresholding confidence or by retaining a predefined percentage with the lowest confidence for manual verification or correction.The method defines low-confidence epochs as those below a threshold or among the lowest-confidence fraction.
  • The model processes 30-second EEG epochs as normalized time-frequency images generated with overlapping STFT frames.Each epoch is decomposed into two-second frames with 50% overlap and transformed into 128 frequency bins.

B. Parameters

Experiments evaluated SleepTransformer on SHHS and SleepEDF-78 using cross-validation or subject-level train-test splits, reporting performance alongside confidence-based error filtering.

  • Parameters: SleepTransformer was evaluated separately on SHHS and SleepEDF-78 using subject-level validation and testing protocols.SleepEDF-78 used 10-fold cross-validation, while SHHS used a 70% training and 30% testing split with 100 validation subjects.
  • Parameters: The evaluation used accuracy, Cohen’s kappa, macro F1-score, average sensitivity, average specificity, and class-wise macro F1-score.Overall metrics and class-specific macro F1-score were used to assess sleep-staging performance.
  • Sleep staging performance:: 87.7% overall accuracy and κ of 0.828 were achieved on SHHS, with a 1.2% accuracy improvement and 0.017 κ improvement over SeqSleepNet.Performance was reported as on par with XSleepNets despite using a single time-frequency input view and a smaller, cheaper model.
  • Sleep staging performance:: SleepEDF-78 performance improved by 3.5% accuracy, 0.046 κ, and 4.5% macro F1 after initializing from an SHHS-trained model.The resulting 84.9% accuracy, κ of 0.789, and MF1 of 78.8 outperformed prior works except U-Sleep.
  • Confidence estimation:: At a 20% low-confidence selection, accuracy was around 60% and the selected epochs contained about 65% of all misclassified epochs.When the selected set reached 50% of epochs, it captured more than 90% of all misclassified epochs.
  • Confidence estimation:: Low-confidence epochs contained more transitioning epochs and were dominated by difficult-to-recognize N1 epochs.With a confidence threshold of 0.5, roughly 90% or more of nightly epochs had confidence above the threshold on SleepEDF-78 or SHHS.

3) Attention score visualization for interpretation:

SleepTransformer’s self-attention is visualized at epoch and sequence levels to show sleep-relevant EEG features and neighboring-epoch influence, while confidence estimates identify uncertain decisions.

  • Visualization setup: The visualization uses sequence length L = 11 and combines attention scores with predicted labels, ground-truth labels, and estimated confidences.Epoch-level and sequence-level attention are shown using the corresponding transformer components.
  • Epoch-level interpretation: Epoch-level heat maps suggest stronger attention to sleep-relevant features, including K-complexes, Delta waves, Alpha waves, and high-amplitude neural activity.The constructed EEGs also show distinguishable frequency distributions in the accompanying time-frequency representations.
  • Sequence-level interpretation: Sequence-level attention weights combine features across epochs, emphasizing context that helps classify target epochs when their own EEG contains limited stage-specific information.For N1, dispersed attention reflects similarities with Wake and N2, while other stages receive stronger weights from more relevant neighboring epochs.
  • Sequence-level interpretation: Near transition boundaries, the model places greater attention on distant epochs with more reliable features, using longer-term sleep structure for recognition.This behavior is illustrated for N2 and N3 around the transitions shown in Figure 8.
  • Clinical interpretation: The attention visualizations are intended to resemble human scoring and support manual verification and correction of low-confidence epochs.This creates a potential interaction pathway between practitioners and the model.

D. Discussion

The discussion examines model capacity, computational cost, uncertainty quantification, and interpretability limitations, while concluding that SleepTransformer performs comparably to state-of-the-art models on two databases.

  • Architecture and performance: SleepTransformer’s performance varies with the numbers of EpochTransformer and SequenceTransformer blocks, evaluated by varying each while fixing the other on SHHS.The investigation considers NE and NS values from 1 to 4.
  • Model size and cost: SleepTransformer has moderate model size and modest computational overhead even with NE = 4 and NS = 4.Compared with XSleepNet, its model size is two thirds as large and it trains 2.7 times faster.
  • Uncertainty limitations: The proposed entropy-based uncertainty method measures knowledge uncertainty but leaves data uncertainty from complexity, multimodality, and noise for future work.Alternative uncertainty measures and methods covering both uncertainty types are identified as future directions.
  • Interpretability limitations: The attention visualization is not necessarily the only or best interpretation method, and further expert interaction is needed before embedding it in daily sleep practice.The discussion also identifies exploration of more advanced transformer variants as future work.
  • Overall conclusion: SleepTransformer performs comparably to state-of-the-art models on both the large SHHS database and the smaller SleepEDF-78 database.Its attention scores support epoch- and sequence-level interpretation, while entropy-based confidence aligns with model mistakes and successes.

APPENDIX A TIME-FREQUENCY REPRESENTATION CORRESPONDING TO THE CONSTRUCTED EEGS

The appendix presents time-frequency representations of constructed EEGs, showing frequency bins 1–40 corresponding to the range (0, 15.5] Hz.

  • Time-frequency representations: The appendix figures show time-frequency representations corresponding to constructed EEGs from Figures 7 and 8.The representations correspond specifically to the constructed EEG examples used in the attention visualizations.
  • Displayed frequency range: Only frequency bins 1–40 are displayed, covering the frequency range (0, 15.5] Hz.This display restriction applies to both appendix representations.
Loading 2105.11043v3…