PaperlayerView exact revision on arXiv

Source-linked AI summary

SleepLM: Natural-Language Intelligence for Human Sleep

Zongzhe Xu, Zitao Shuai, Eideen Mozaffari, Ravi S. Aysola, Rajesh Kumar, Yuzhe Yang

arXiv:2602.23605v2cs.AI

TL;DR

SleepLM addresses sleep-analysis systems that are restricted to closed labels and language models that cannot directly handle continuous physiological data. It aligns multimodal PSG with language through multilevel captions, a large paired sleep-text dataset, and pretraining that combines contrastive alignment, caption generation, and signal reconstruction. Experiments report superior performance across established sleep tasks, alongside controllable insight generation and zero-shot generalization to novel concepts.

  • Problem

    Existing sleep models are predominantly confined to predefined stages or events, while LLMs are ill-equipped to process high-dimensional continuous physiological data.

  • Method

    SleepLM combines multilevel sleep captioning, a large paired sleep-text dataset, and compound pretraining with contrastive alignment, caption generation, and signal reconstruction.

  • Results

    SleepLM achieves superior performance across established real-world sleep understanding tasks and enables controllable insight generation and zero-shot generalization to novel clinical concepts.

  • Takeaways & Limitations

    SleepLM supports describing and interacting with raw sleep signals through natural language rather than limiting analysis to fixed sleep categories.

  • Takeaways & Limitations

    SleepLM is a research prototype not clinically validated for diagnosis, treatment, or medical decision-making, and evaluation covers five PSG cohorts from NSRR.

Abstract

from arXiv · show

We present SleepLM, a family of sleep-language foundation models that enable human sleep alignment, interpretation, and interaction with natural language. Despite the critical role of sleep, learning-based sleep analysis systems operate in closed label spaces (e.g., predefined stages or events) and fail to describe, query, or generalize to novel sleep phenomena. SleepLM bridges natural language and multimodal polysomnography, enabling language-grounded representations of sleep physiology. To support this alignment, we introduce a multilevel sleep caption generation pipeline that enables the curation of the first large-scale sleep-text dataset, comprising over 100K hours of data from more than 10,000 individuals. Furthermore, we present a unified pretraining objective that combines contrastive alignment, caption generation, and signal reconstruction to better capture physiological fidelity and cross-modal interactions. Extensive experiments on real-world sleep understanding tasks verify that SleepLM outperforms state-of-the-art in zero-shot and few-shot learning, cross-modal retrieval, and sleep captioning. Importantly, SleepLM also exhibits intriguing capabilities including language-guided event localization, targeted insight generation, and zero-shot generalization to unseen tasks. All code and data will be open-sourced.

1. Introduction

SleepLM addresses the gap between language-centered sleep interpretation and models limited to closed labels or poorly suited to continuous physiological data. It combines multilevel captions, a large paired sleep-text dataset, and unified pretraining to support broad sleep understanding and natural-language interaction.

  • Sleep physiology spans interacting brain, cardiac, and respiratory signals, while PSG records these dynamics through synchronized multimodal channels.
  • Existing sleep models mainly predict predefined stages or events, whereas general-purpose LLMs struggle with high-dimensional continuous physiology.
  • SleepLM introduces hierarchical captions spanning global summaries, local details, and channel-specific characteristics for raw sleep-signal interpretation.
  • ReCoCa jointly learns language and physiological time series through contrastive alignment, caption generation, and signal reconstruction.
  • Experiments report superior performance across established sleep tasks and capabilities including controllable insight generation and zero-shot generalization to novel concepts.
  • The paired sleep-text dataset contains over 100,000 hours from more than 10,000 individuals.

2. Related Work

Prior work provides strong unimodal sleep representations and multimodal alignment methods, but existing sleep foundation models remain focused on classification and lack natural-language interaction. SleepLM adapts multimodal pretraining to dense physiological time series with sleep-specific captions, paired data, and a dedicated architecture.

  • Existing sleep foundation models learn representations for staging and event detection but remain unimodal and optimized for classification.
  • General time-series foundation models address broad temporal data, but physiological signals differ in sampling rate, recording length, and fixed channel structure.
  • Vision-language methods such as CLIP and CoCa establish contrastive alignment and captioning patterns that have been extended to temporal and wearable signals.
  • ReCoCa jointly optimizes signal reconstruction, contrastive alignment, and caption generation while subsuming formulations such as CLIP, Cap, and CoCa.
  • SleepLM contributes a sleep-specific captioning pipeline, a large paired sleep-text dataset, and pretraining designed for dense physiological time series.

3. Human Sleep Captioning at Scale

SleepLM constructs large-scale sleep-text supervision from multimodal PSG by generating captions at channel, local, and global levels. The resulting dataset spans over 100,000 hours and supports descriptions of signal features, temporal events, and holistic physiological states.

  • The pretraining corpus aggregates five large-scale datasets from the National Sleep Research Resource.
  • PSG recordings are segmented into non-overlapping 30-second epochs using a standardized 12-channel montage grouped by physiological modality.
  • The captioning pipeline addresses the information bottleneck created by assigning only one coarse sleep-stage label to each epoch.
  • Channel captions verbalize clinically used signal features, while local captions describe transient changes and event onset and offset timestamps.
  • Global captions summarize sleep stage and overall autonomic state, with some descriptors derived from withheld signals as a masked-prediction proxy.
  • The paired dataset contains over 100,000 hours of PSG across more than 12,000 recording nights and over 10,000 individuals.

4. SleepLM

SleepLM’s ReCoCa framework jointly learns from physiological time series and text by combining contrastive alignment, caption generation, and signal reconstruction. Its architecture preserves channel structure, supports targeted physiological descriptions, and instantiates a configurable family of sleep-language models.

  • SleepLM family: The framework supports CLIP-, captioning-, and CoCa-style formulations as special cases by enabling or disabling objective components.The full ReCoCa configuration uses all three losses as the default model.
  • Pretraining framework: ReCoCa combines contrastive alignment, signal reconstruction, and caption generation in a unified pretraining objective for paired PSG epochs and captions.The final objective weights the three component losses, while varying their weights produces different SleepLM family members.
  • Sleep encoder: Channel-independent patch embeddings and interleaved temporal- and channel-attention blocks encode modality-specific PSG structure before cross-channel mixing.The design reflects that EEG, EOG, and respiratory-effort channels carry different meanings within a fixed montage.
  • Signal reconstruction: A reconstruction decoder predicts original signal patches from encoder latents, preserving waveform details that captions may omit and regularizing multimodal pretraining.This objective is intended to encourage physiologically grounded representations rather than relying only on caption alignment.
  • Caption generation: A modality-conditioned decoder samples one target system from Brain, Respiratory, Cardiac, and Somatic groups to generate controllable, targeted captions.The selected system is represented by a learnable token prepended to the decoder input.

5. Experiments and Results

SleepLM is evaluated across zero-shot recognition, retrieval, unseen-concept generalization, caption generation, transfer, and ablations. Across these experiments, it outperforms the reported baselines and supports clinically relevant language-grounded capabilities.

  • Zero-Shot Recognition: SleepLM consistently outperforms proprietary LLMs and fine-tuned VLMs across four zero-shot task categories.These include sleep staging, event localization, physiological inference, and explicit signal grounding.
  • Zero-Shot Cross Modal Retrieval: SleepLM substantially outperforms LLM baselines in bidirectional cross-modal retrieval, which are only slightly above random.It maintains strong performance on the full 2,000-sample validation set despite baseline context-length constraints.
  • Generalization to Unseen Concepts: Approximately 80% F1 and accuracy are achieved on held-out Mixed Apnea and Obstructive Apnea events, while LLM baselines perform at chance.UMAP places the unseen events near the related seen concept Central Apnea and separates them from Oxygen Desaturation.
  • Transfer and Representation Analysis: Caption-based supervision produces discriminative and transferable features that remain competitive across tasks and settings, including the unseen CFS dataset.The authors report strong performance across diverse tasks and matching or exceeding prior domain-specific state-of-the-art methods.
  • Sleep Caption Generation: SleepLM produces concise, clinically accurate captions covering sleep stage and localized-event timing, whereas Gemini 2.5 Pro often introduces incorrect associations.The learned retrieval space also shows a smooth semantic gradient in which similarity tracks query physiology.
  • Full-Night Reporting: Full-night aggregation derives diagnostic metrics such as AHI and WASO from sliding-window epoch predictions, with strong concordance to manual scoring.The analysis connects epoch-level outputs with full-night reporting on five SHHS subjects.
  • Ablation Analyses: Removing reconstruction degrades discriminative-task performance, while multilevel supervision outperforms global-only captions, supporting both reconstruction and low-level grounding.The results support reconstruction as a regularizer and grounded waveform descriptors as useful for broader physiological inference.
  • Scope and Limitations: The evaluation uses five PSG cohorts with fully held-out datasets for external validation, but the study does not clinically validate SleepLM for diagnosis or medical decisionmaking.The authors identify broader clinical variability, devices, and patient populations as areas for future robustness assessment.

A. Training Details

ReCoCa is trained with AdamW for 15 epochs using a cosine schedule, warmup, weighted multi-objective losses, and distributed H100 computation. Multimodal LLM baselines use a two-stage alignment and finetuning protocol with LoRA.

  • ReCoCa Training: ReCoCa uses AdamW with a 1e-4 learning rate, cosine annealing, 15 epochs, and 5,000-step linear warmup.Training typically converges within approximately 48 hours.
  • ReCoCa Training: The ReCoCa loss weights are λ_con = 1.0, λ_cap = 2.0, and λ_rec = 0.1, with gradients clipped at norm 1.0.The global batch size is 384 across four NVIDIA H100 GPUs.
  • Baseline Finetuning: Baseline finetuning first trains only projector and token-pooler layers while freezing the Sleep Encoder and LLM backbone.This alignment stage matches the sleep feature space to the LLM embedding space.
  • Baseline Finetuning: The second baseline stage unfreezes the projector, pooler, and Sleep Encoder and applies LoRA to the LLM backbone.LoRA uses rank 16, α = 32, and dropout p = 0.05.
  • Baseline Finetuning: Baseline experiments use batch size 8, gradient accumulation every 16 steps, 500 warmup steps, and one full finetuning epoch.They require 96–144 hours on the same 4×H100 setup.

B.1. Data Split & Preprocessing

The study separates training, internal evaluation, and external validation at the subject or dataset level, then standardizes heterogeneous PSG recordings before caption-statistic extraction. CFS and WSC are held out for generalization-focused evaluation.

  • Data Splits: SHHS and MrOS are partitioned by subject into pretraining and internal evaluation sets, CCSHS is training-only, and CFS and WSC are fully held out.Evaluation samples 2,000 epochs from each validation dataset to manage computational cost.
  • Preprocessing: Missing channels are zero-padded, all signals are resampled to 64Hz, and recordings undergo manual trimming of excessive wakefulness or non-wear.These steps address heterogeneous source formats and extreme sensor noise.
  • Preprocessing: Signals are normalized per night with z-scores, while respiratory channels use area-dependent z-score normalization for consistent amplitude scaling.The respiratory-specific procedure accounts for amplitude variation over time.
  • Cohort Composition: Cohort age distributions are reported as percentages, with demographic skew attributed to the higher prevalence of sleep disorders at advanced ages.The table reports age distributions across cohorts and train-validation splits.
  • Channel Captions: Channel captions use variable statistics selected for each channel to provide diverse, fine-grained, clinically relevant information.The specific channel-statistic assignments are provided in Table 5.

C.1. Zero-Shot Task Definitions

SleepLM’s zero-shot evaluation covers sleep staging, event localization and classification, implicit vital-sign inference, and explicit signal grounding. Additional experiments test few-shot transfer, data scale, model scale, reconstruction, and multilevel-caption design choices.

  • Zero-Shot Task Definitions: Zero-shot evaluation spans sleep staging, sleep-event localization and classification, implicit HR/SpO2 inference, and explicit channel-statistics grounding.The evaluation uses balanced accuracy, IoU, MAE, Recall, and sMAPE for the corresponding tasks.
  • Unseen Concept Classification: Held-out mixed-apnea and obstructive-apnea classification tests generalization beyond concepts present in the pretraining vocabulary.The protocol uses event and no-event text anchors, with the negative anchor recognized as noisy because negatives may contain other events.
  • Few-Shot Transfer: Few-shot linear probing freezes pretrained encoders and trains classifiers with K ∈ {1, 5, 10, 20, 50} labeled samples per class on staging and oxygen-desaturation tasks.Baselines include MAE, SimCLR, WideResNet, and ViT under approximately matched parameter counts.
  • Data Scale: Multisource pretraining improves performance on both external and internal datasets, indicating that the captioning pipeline extends beyond individual dataset boundaries.The comparison uses internal SHHS and external CFS settings.
  • Model Scale: Increasing SleepLM capacity from 38M to 410M parameters yields monotonic gains, especially for cross-modal retrieval and channel-statistics regression, without saturation at Base scale.The evaluated variants are SleepLM-T (38M), SleepLM-S (180M), and SleepLM-B (410M).
  • Ablations: Ablations evaluate sleep reconstruction and low-level channel captions on high-level staging and event-identification tasks.The reconstruction objective preserves fine-grained waveform morphology, while multilevel supervision links channel details to higher-level inference.

D.4. Fewshot Results

Few-shot evaluations test frozen SleepLM representations against self-supervised, supervised, domain-specific, and general time-series baselines. SleepLM consistently outperforms standard SSL and supervised representations and remains competitive across broader task settings.

  • Few-Shot Transfer: SleepLM consistently outperforms MAE, SimCLR, and supervised baselines across few-shot sleep staging and oxygen-desaturation detection.The evaluation freezes encoders and trains linear probes with 1, 5, 10, 20, or 50 labels per class.
  • Domain-Specific Comparisons: Frozen linear probing and full fine-tuning compare SleepLM with RobustSleepNet, U-Sleep, SleepFM, and Chronos-2 on the unseen CFS dataset.Models are evaluated under matched training splits, with final AUCs reported on the full test split.
  • Overall Findings: SleepLM remains competitive across tasks and settings, while foundation models generally outperform smaller specialized models.The authors associate this pattern with model scale and broader transfer of learned representations.

D.5. Classification Raw Results

Raw zero-shot results compare SleepLM and ReCoCa with proprietary LLMs, fine-tuned VLMs, and standard multimodal architectures across classification, retrieval, regression, and trend tasks. ReCoCa leads most reported metrics, including retrieval under a substantially larger candidate pool than proprietary LLMs.

  • Classification: ReCoCa achieves the top performance across most zero-shot sleep-stage and event-detection metrics and ranks second on the remainder.The comparison includes proprietary LLMs, fine-tuned VLMs, and CLIP-, Cap-, and CoCa-style baselines.
  • Retrieval: ReCoCa consistently reaches top-tier cross-modal retrieval performance despite evaluation with N=2000 candidates, whereas proprietary LLMs use N=100 and perform near chance.The larger pool makes the ReCoCa retrieval setting more challenging than the proprietary-LLM comparison.
  • Regression and Trends: Zero-shot regression separates explicit channel-statistics grounding from implicit HR/SpO2 inference, using MAE, sMAPE, scalar estimation, and trend Recall.HR and SpO2 are excluded from the encoder input for implicit inference.
  • Regression and Trends: ReCoCa maintains competitive HR/SpO2 inference while achieving a dominant margin on the Channel Stats metric.The analysis attributes strong proprietary-LLM vital-sign results to reasoning over established clinical algorithms, without claiming the same mechanism for ReCoCa.

D.8. Statistical Results for Full Night Metrics

Full-night evaluation aggregates epoch-level predictions into clinical metrics for five SHHS recordings and compares ReCoCa with a fine-tuned Qwen3-VL model. ReCoCa performs better on most sleep-architecture and respiratory measures, while Qwen3 retains isolated advantages on some arousal-related metrics.

  • Evaluation Setup: Full-night evaluation processes five SHHS recordings with sliding windows to estimate longitudinal clinical metrics from epoch-level predictions.Reported metrics include sleep architecture, respiratory indices, and arousal measures.
  • Sleep Architecture: ReCoCa dominates macro-structural measures including Sleep Efficiency, Sleep Latency, and Wake After Sleep Onset.These measures are reported in the comparison with fine-tuned Qwen3-VL-8B-Instruct.
  • Arousal Metrics: Qwen3 performs marginally better on aggregate Arousal Index and selected Central Apnea Index submetrics, but not consistently across broader respiratory reporting.The comparison therefore includes isolated baseline advantages alongside ReCoCa’s broader gains.
  • Overall Findings: The evaluation reports that ReCoCa translates epoch-level understanding into accurate, longitudinal clinical summaries and outperforms larger fine-tuned VLMs on consistent full-night reports.This conclusion is stated for the five-subject full-night comparison.

D.9. Full Generation Quality Results

The section presents SleepLM’s generated sleep descriptions, including stage, physiological, and localized-event information, alongside comparisons with language-model baselines. SleepLM’s targeted generation conditions decoding to produce descriptions for designated modalities.

  • Quality comparison: The compared language models generate marginally more accurate descriptions but retain significant misconceptions and insufficient granular spatial awareness for precise event localization.
  • Targeted generation: SleepLM can produce precise targeted generations for one or multiple designated modalities by prepending corresponding condition tokens during text decoding.
  • Quality comparison: SleepLM outperforms language-model and fine-tuned VLM baselines across all evaluated caption-quality metrics.The raw scores are reported in Table 20.
  • Generated descriptions: SleepLM generates descriptions spanning sleep stage, sleep events, brain, heart, respiratory, and somatic characteristics.Examples include stage labels, event timing, EEG and EOG features, heart-rate measures, respiration, and chin EMG characteristics.

E.4. More Retrieval Quality Examples

The retrieval examples vary sleep-event timing and type while preserving semantic sleep descriptions. Similarity scores are reported for perturbed captions and original examples, illustrating retrieval behavior across related descriptions.

  • Perturbation examples: Perturbation examples modify the timing of hypopnea, central apnea, and arousal events in otherwise related sleep captions.
  • Similarity scores: 0.4007 is the reported similarity for an original Stable Light Sleep (N2) caption containing central apnea and oxygen desaturation.
  • Similarity scores: Reported perturbation similarities range from 0.1243 to 0.3568 across altered arousal, hypopnea, and central-apnea descriptions.
  • Similarity scores: 0.3294 is the reported similarity for an original Stable Light Sleep (N2) caption containing an arousal event from 15.0 to 30.0 seconds.
Loading 2602.23605v2…