Source-linked AI summary
SensorLM: Learning the Language of Wearable Sensors
Yuwei Zhang, Kumar Ayush, Siyuan Qiao, A. Ali Heydari, Girish Narayanswamy, Maxwell A. Xu, Ahmed A. Metwally, Shawn Xu, Jake Garrison, Xuhai Xu, Tim Althoff, Yun Liu, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Cecilia Mascolo, Xin Liu, Daniel McDuff, Yuzhe Yang
TL;DR
Wearable sensor data are difficult to align with language because richly paired descriptions are scarce, limiting broad sensor-language modeling. SensorLM addresses this with hierarchical caption generation, a large paired dataset, and unified multimodal pretraining, achieving superior performance across zero-shot, few-shot, and retrieval tasks while showing broader scaling and generalization capabilities.
Problem
Rich paired descriptions for uncurated wearable sensor data are scarce, creating a central gap for sensor-language pretraining.
Method
SensorLM uses hierarchical statistical, structural, and semantic captions, a large paired dataset, and a framework combining contrastive and generative learning.
Results
SensorLM achieves superior performance in zero-shot, few-shot, and cross-modal retrieval tasks and demonstrates scaling and zero-shot generalization capabilities.
Takeaways & Limitations
SensorLM enables wearable sensor understanding and applications through natural language across diverse real-world tasks.
Takeaways & Limitations
SensorLM is not clinically validated, and its evaluation is limited to specific wearable devices and sensor data.
Abstract
from arXiv · showhide
We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks.
1. Introduction
SensorLM addresses the difficulty of aligning continuous wearable sensor streams with language by combining scalable caption generation, a large paired dataset, and a generic pretraining framework. The study evaluates these models across real-world tasks and reports broad performance and generalization capabilities.
- Over 59.7 million hours of multimodal wearable data from over 103,000 individuals underpin the SensorLM study.The study uses a large-scale wearable dataset to investigate sensor-language modeling.
- SensorLM targets the lack of large-scale, diverse sensor-language pairs needed for generalizable multimodal foundation models.Existing datasets are limited in scale and often use restrictive question-answering formats.
- Its three innovations are hierarchical caption generation, a 59-million-hour sensor-language dataset, and a generic framework integrating multimodal architectures.The pipeline captures statistical, structural, and semantic information from fine-grained sensor streams.
- Extensive experiments benchmark SensorLM against state-of-the-art methods across human activity analysis and metabolic health.The evaluation covers diverse real-world tasks and multimodal understanding.
- SensorLM shows scaling, data-efficiency, multimodal generation, and zero-shot generalization properties beyond its benchmark results.These analyses examine behavior on unseen tasks and concepts.
2. Related Work
Prior sensor-language work is constrained by scarce paired corpora and by approaches that often rely on summaries, tabular inputs, or specialized alignment modules. SensorLM extends multimodal pretraining ideas from vision-language modeling to wearable sensor data.
- Large-scale paired corpora from uncurated wearable data remain scarce despite their importance for cross-modal alignment.
- Prior approaches use pre-derived sensor summaries, raw sensor values as tabular LLM inputs, or specialized sensor encoders with alignment modules.
- Sensor foundation models have mainly focused on single-modal, multi-channel data with contrastive or reconstruction-based pretraining.
- SensorLM adapts vision-language pretraining paradigms, including prominent multimodal architectures, to scalable sensor-language modeling.
3. Sensor-Language Dataset Construction
SensorLM constructs a large paired dataset by generating hierarchical captions for multimodal wearable streams. The captions progress from quantitative summaries to temporal dynamics and contextual behavioral information.
- The dataset combines wrist-worn PPG, accelerometer, altimeter, skin-temperature, and electrodermal-activity signals.The collected streams support multimodal time-series modeling.
- Sensor-language pretraining is limited by the absence of paired descriptions for wearable data and by sparse coarse activity labels.
- The hierarchical strategy generates statistical, structural, and semantic captions at distinct abstraction levels.This design represents numerical summaries, temporal patterns, and high-level events or contextual states.
- Statistical Captions: Statistical captions summarize each sensor feature with measures such as mean, maximum, minimum, and standard deviation.
- Structural Captions: Structural captions identify increasing, decreasing, or stable trends and sharp spikes or drops using sliding windows.These descriptions encode temporal shape and behavioral dynamics.
- Semantic Captions: Semantic captions describe recognized activities, sleep periods, and timestamped user-reported moods.
- 2,489,570 person-days from 103,643 people across 127 countries form the curated paired pretraining dataset.The data were collected between March 1 and May 1, 2024, with de-identification, consent, and IRB review.
4. SensorLM
SensorLM unifies contrastive and generative sensor-language learning in a framework that extends established multimodal architectures. Its architecture encodes sensor sequences and text, while configurable objectives recover CLIP-, Cap-, and CoCa-like variants.
- SensorLM combines contrastive, generative, and hybrid multimodal pretraining paradigms within one sensor-language framework.
- SensorLM integrates complementary contrastive and generative strategies for sensor-language pretraining.
- Architecture: The model uses sensor, text, and multimodal decoder components to represent time-series data and generate multimodal text.The sensor encoder uses patched transformer processing, while the decoder integrates sensor features through cross-attention.
- Pretraining Objectives: The composite objective balances contrastive alignment and caption generation through weighted losses Lcon and Lcap.
- Representative Variants: CLIP, Cap, and CoCa are recovered by setting the contrastive and generative loss weights to specific configurations.CLIP uses λcap = 0, Cap uses λcon = 0, and CoCa uses λcon = λcap = 1.
5. Experiments & Results
SensorLM is evaluated across zero-shot classification, cross-modal retrieval, few-shot transfer, and analyses of scaling, caption types, and unseen-activity generalization. It performs strongly across these settings, including perfect retrieval on a 100-sample benchmark and AUROC 0.88 with 50 labels per class.
- Evaluation setup: SensorLM is evaluated on activity recognition, health-related prediction, and cross-modal retrieval using held-out and downstream datasets.The experiments include Activity and Metabolic datasets, with comparisons against LLM and self-supervised learning baselines.
- Zero-shot learning: SensorLM supports zero-shot classification across main activities, activity concepts, and fine-grained recognition tasks.Classification uses prompt-ensembled text embeddings and includes six activity-recognition tasks.
- Zero-shot cross-modal retrieval: SensorLM achieves perfect retrieval on a 100-sample benchmark and maintains high accuracy at 5k and 40k samples, while LLM baselines largely fail.The task covers both sensor-to-text and text-to-sensor retrieval; most baseline tasks are infeasible because of context limits.
- Zero-shot cross-modal retrieval: SensorLM retrieves semantically and temporally relevant captions, including similar activities at matching or different time frames.Qualitative examples show correct ground-truth retrieval alongside partially matching captions with high similarity scores.
- Few-shot transfer: AUROC 0.88 with 50 labels per class shows SensorLM outperforming baselines across few-shot activity-recognition settings.The few-shot evaluation varies training labels per class from 5 to 50 and also tests Hypertension and Anxiety prediction.
- Scaling analysis: Performance improves with training steps and data size, although some models show diminishing gains beyond 12 million hours and larger models can degrade when undertrained.These results support scaling behavior in sensor-language modeling while identifying compute requirements for larger models.
- Generalization and generation: SensorLM infers unseen activities from conceptual proximity in the learned embedding space and generates meaningful natural-language descriptions.The analyses examine semantic alignment across sensor and text embedding spaces and multimodal generation capabilities.
- Caption ablations: Semantic captions are crucial for zero-shot activity recognition, while semantic-plus-structural captions provide the best trade-off across tasks.Statistical captions reduce zero-shot activity-recognition performance when added to semantic captions but benefit Anxiety and Hypertension prediction.
6. Discussion
SensorLM uses aggregated wearable features and varied hierarchical caption templates, with evaluation spanning specific wearable datasets and demographic cohorts. The authors caution that it is not clinically validated and that generalization beyond the evaluated sensor settings remains open.
- Limitations: SensorLM is not a clinically validated diagnostic tool and is not intended for clinical diagnosis, treatment, or medical decision-making.Deployment for healthcare uses would require further analysis of applicable regulations.
- Limitations: Evaluation is limited to specific wearable devices and sensor data, so generalizability to other sensor types requires further study.The authors state that the method is general but do not establish this broader generalizability in the reported evaluation.
- Input representation: SensorLM does not use high-resolution raw waveforms directly, instead relying on minutely aggregated features because storage, battery, and computation costs make full-day raw data impractical.A 200Hz signal over 24 hours produces approximately 17 million time points per instance.
- Dataset scope: The pretraining cohort includes participants with valid sensor data for at least 20% of one day and at least one logged event.The cohort is 38% female, has mean age 41.7 years, and includes reported BMI categories.
- Caption construction: The model’s captions use statistical, structural, and semantic descriptions with varied rewrite templates to encourage linguistic diversity.The implementation includes 20 statistical, 15 structural, and 20 semantic rewrite templates.
A.4. Data Acquisition and Approval
The study uses consented historical wearable data and an IRB-approved observational dataset, while SensorLM combines sensor, text, and multimodal decoder components. Baselines and training procedures are standardized for comparison.
- Data acquisition: The pretraining corpus uses historical wearable data collected with participant consent for research and publication.
- Approval: The metabolic dataset comes from an IRB-approved observational study evaluating wearable-device data for metabolic-health algorithms.
- Model architecture: SensorLM processes one-day, minute-resolution inputs with a sensor encoder, text encoder, and multimodal text decoder.
- Model variants: The model family includes SensorLM-S, SensorLM-B, SensorLM-L, and SensorLM-XL with increasing model sizes.
- Baselines: Self-supervised baselines use the same ViT-2D sensor-encoder backbone and standardized time-series augmentations, while DINO omits scaling because of convergence issues.
B.5.1. Task definition for zero-shot classification
The zero-shot evaluation covers broad activity recognition, activity concepts, fine-grained categories, unseen activities, retrieval, and captioning. SensorLM uses prompt ensembling, while LLM baselines receive subsampled tabular sensor inputs.
- Task scope: The evaluation defines six zero-shot activity-classification tasks spanning main activities, activity concepts, and fine-grained recognition.
- Main activity classification: Main activity classification contains 20 activities, including walking, running, swimming, weightlifting, skiing, and kayaking.
- Activity concepts: Concept classification covers environmental context, cardio versus strength training, and locomotion versus stationary exercise.
- Fine-grained recognition: Fine-grained recognition distinguishes four gym-cardio activities and four outdoor-sport activities.
- Evaluation procedure: SensorLM performs zero-shot classification by comparing sensor embeddings with averaged text embeddings from 30 prompt variants, whereas LLM baselines use subsampled tabular inputs.
- Unseen classes: Unseen-class generalization pretrains on 20 activities and tests nine previously unseen classes using one-versus-all AUROC evaluation.
C.1. Scaling Behaviors
SensorLM exhibits predictable scaling with compute, data, and model size, although data gains saturate and larger models can underperform when undertrained. Linear probing and regression results show competitive downstream performance.
- Scaling behaviors: Compute scaling raises SensorLM-B AUROC from 0.66 at 17.4 TPU hours to 0.75 at 174 TPU hours.
- Scaling behaviors: Expanding training data consistently improves performance, but gains beyond 12 million hours diminish.
- Scaling behaviors: Larger models outperform smaller counterparts with sufficient compute and data, while SensorLM-XL underperforms at low training steps because of undertraining.
- Scaling behaviors: The reported trends indicate consistent and predictable scaling behavior for sensor-language modeling.
- Linear probing: SensorLM reaches AUROC 0.60 on Hypertension and 0.94 on Activity, while its Anxiety AUROC of 0.65 is comparable to MSN's 0.66.
- Regression: SensorLM improves performance over self-supervised baselines on Age and BMI regression tasks.
C.3. Cross-Modal Retrieval
SensorLM provides strong bidirectional cross-modal retrieval and outperforms general-purpose LLM baselines that are often constrained by context length. Qualitative examples also show retrieval of statistical, structural, and semantic sensor descriptions.
- Quantitative retrieval: SensorLM achieves 100% R@1, R@5, and R@10 on the 100-sample benchmark in both Sensor →Text and Text →Sensor retrieval.
- Quantitative retrieval: At the 40k scale, SensorLM reaches R@1 scores of 96.1% for Sensor →Text and 90.0% for Text →Sensor.
- Baseline comparison: LLM baselines cannot perform most retrieval tasks because of context-length limitations and perform substantially worse where results are available.
- Qualitative retrieval: Qualitative retrieval examples show matching of statistical summaries and structural descriptions involving trends, spikes, and temporal dynamics.
- Caption generation: SensorLM consistently outperforms both LLM baselines across the reported caption-generation metrics.
- Caption generation: Compared with Gemini 2.0 Flash, SensorLM generates coherent semantic events with more accurately localized timeframes.
C.5. Complete Ablation Results
The ablations show that caption type and pretraining objective materially affect SensorLM’s downstream behavior. Semantic and structural captions broadly help, while statistical captions create task-specific trade-offs; CoCa’s combined objectives outperform single-objective variants.
- Semantic captions are critical for zero-shot activity recognition, and adding structural captions further improves performance across tasks, including cross-modal retrieval.
- Statistical captions improve performance on “Anxiety” and “Hypertension” but slightly reduce accuracy on “Activity”.
- SensorLM (CoCa) consistently outperforms SensorLM (CLIP) and SensorLM (Cap) across key zero-shot classification and linear-probing metrics.The result supports integrating contrastive and generative objectives during pretraining.
D. Societal Impact
SensorLM is presented as an accessible natural-language interface for multimodal wearable signals, with potential relevance to proactive health management. Its use remains bounded by safety, privacy, clinical-validation, and evaluation-scope constraints.
- SensorLM translates multimodal sensor signals into natural language, making wearable-derived information more accessible to consumers and domain experts.
- This interface may support clearer, more actionable, proactive, personalized, and preventative health management.
- Health-data applications require attention to safety, fairness, privacy, responsible development, and participant confidentiality.
- SensorLM is a research prototype, not a clinically validated diagnostic tool, and clinical deployment would require rigorous validation and regulatory compliance.
- Current evaluation covers specific wearable devices and sensor modalities, so performance across broader ecosystems, modalities, and populations remains unassessed.