PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck, Narayan Schuetz, Thomas Kaar, Manuel Milling, Nicolas Zumarraga, Ning Wang, Kelly W. Zhang, Anders Johnson, Robert Jakob, Björn W. Schuller, Elgar Fleisch, Daniel Seung Kim, Allan Lawrie, Patrick Langer, Maxwell A. Xu, Euan Ashley, Paul Schmiedmayer

arXiv:2610.21274v1cs.LGcs.CL

TL;DR

Open paired wearable time-series captions remain limited in scale and modality coverage, while reproducibility and extensibility are also gaps. TimeCap applies deterministic, signal-based extractors to create extensible paired captions; caption-augmented OpenTSLM outperformed a label-only baseline on 29 of 32 user-level binary evaluations.

  • Problem

    Paired wearable time-series captions remain limited in open-corpus scale and modality coverage, with existing methods also leaving reproducibility gaps.

    Source: §1 · p. 2

  • Method

    TimeCap uses deterministic extractors on signals, cross-channel synthesis, and channel definitions to adapt captioning to new datasets.

    Source: §3 · p. 5, §3.1 · p. 6

  • Results

    29 of 32 user-level binary evaluations favored caption-augmented OpenTSLM over an architecturally identical label-only baseline.

    Source: §5 · p. 11

  • Takeaways & Limitations

    TimeCap provides a reproducible mechanism to regenerate, audit, and extend paired-caption resources alongside new wearable hardware.

    Source: §6 · p. 12

    OpenMHC is skewed toward white, male US participants (median age 39), and reported clinical AUROCs indicate population patterns rather than reliable individual predictions.

    Source: §5 · p. 11

Abstract

from arXiv · show

Time-series data generated continuously across industrial, environmental, and especially health domains produce multimodal signals whose scale and heterogeneity make their processing and interpretation particularly challenging. Time-Series Language Models (TSLMs) promise scalable interpretation of these signals, yet remain bottlenecked by a fundamental lack of text-annotated time series pairs that are signal-grounded, reproducible, and extensible across modalities and schemas. Existing paired corpora are either closed or limited in scale and modality coverage. Annotation with large language models could scale them up, but its outputs cannot be regenerated exactly, which prevents others from re-deriving, extending, or auditing the resulting corpus. To address this, we release TimeCap, an open deterministic captioning pipeline with three reusable extractor tiers and a cross-channel synthesis layer that operates directly on the signal with no language model in the loop, extensible to new schemas by writing channel definitions alone. This positions TimeCap as a reproducible mechanism for future dataset generation rather than a one-off release. Applying TimeCap to OpenMHC, the MyHeart Counts wearable dataset, yields MHCap, the largest open caption layer for multimodal wearable time-series data. MHCap spans 19 Apple HealthKit channels and 7,005,763 captions over 908,944 person-days. To showcase the impact of the corpus, we train OpenTSLM, a time-series language model, on MHCap and benchmark on health-outcome prediction. Caption-augmented training (OpenTSLM-Cap) outperforms a label-only baseline on 29 of 32 per-user binary evaluations across 8 tasks.

1 Introduction

Open wearable time-series captioning faces gaps in corpus scale and openness, reproducible methods, and schema extensibility. TimeCap and MHCap address these gaps, and caption-augmented OpenTSLM outperforms a label-only baseline on most evaluated health-outcome tasks.

Data gap

  • The largest paired wearable corpus was closed, while the largest open alternative had 5,648 question–answer pairs over 60 subjects and far fewer recorded hours.The closed corpus comprised 59.7 million hours from 103,731 participants; the open alternative was roughly four orders of magnitude smaller in recorded hours.Source: §1 · p. 2

Methods gap

  • Existing annotation methods rely on language models or synthetic data, while recent benchmarks find state-of-the-art LLMs fall short on health time-series reasoning.TimeOmni-1 human-curated 2,339 of 23,605 samples; ChatTS used synthetic time series.Source: §1 · p. 2

Extensibility gap

  • Existing pipelines are tied to particular sensor schemas, so adapting them to new wearable datasets typically requires redesign rather than reconfiguration.This constrains the growth of open paired data alongside new devices.Source: §1 · p. 2

Contributions

  • TimeCap is an open, deterministic pipeline with four composable extractor tiers that operate directly on signals and can extend to new wearable datasets.Its channel-schema abstraction allows captions to be regenerated from accessible signals and traced to rules and thresholds.Source: §1 · p. 2
  • MHCap contains 7,005,763 captions over 908,944 person-days across all 19 Apple HealthKit channels.The corpus covers 2,081 sharable OpenMHC participants; TimeCap code is released under the MIT license.Source: §1 · pp. 2–3

Evaluation using SoTA TSLMs

  • Caption-augmented OpenTSLM outperformed an architecturally identical label-only baseline on the majority of health-outcome tasks.The evaluation used the OpenMHC suite.Source: §1 · p. 3

2 Related Work

Prior sensor-language resources trade off openness, scale, reproducibility, and schema coverage. TimeCap extends template-based captioning with cross-channel synthesis and schema reconfiguration, while MHCap provides an open, large-scale paired caption layer for wearable data.

2.1 Captioning Pipelines for Sensor Time-Series

  • SensorLM [6] established population-scale template captioning with statistical, structural, and semantic tiers, but its pipeline and corpus remain closed.TimeCap extends this hierarchy with cross-channel synthesis and channel definitions that support new datasets without pipeline redesign.Source: §2.1 · pp. 3–4

2.2 Open paired Sensor-Language wearable time-series data

  • Existing paired-language datasets vary in scope and grounding; MHCap provides 7,005,763 deterministic captions over 908,944 person-days across all 19 Apple HealthKit channels.The resource comparison highlights that no existing corpus combines openness, deterministic generation, and schema extensibility.Source: §2.2 · pp. 4–5

3 TimeCap and Corpora

TimeCap standardizes heterogeneous sensor streams and applies composable signal-based extractors to produce grounded captions. It yields a large OpenMHC caption layer and is also instantiated on WESAD using dataset-specific channel definitions.

3 TimeCap and Corpora

  • TimeCap maps heterogeneous sensor streams into a shared Recording and Annotation schema, applying deterministic extractors directly to signals and adapting through channel definitions rather than pipeline redesign.The Recording stores a channel-by-time matrix and metadata; annotations retain text, channel references, time intervals, labels, and optional numeric values.Source: §3 · pp. 5–6, Figure 1 · p. 5

3.1 Captioning

  • Four extractor tiers produce channel statistics, localized spikes and trends, semantic activity intervals, and cross-channel narratives linking events with concurrent signal evidence.Cross-channel synthesis can enrich a labeled walking interval with synchronized heart rate, steps, distance, and active energy, and distinguish time in bed from time asleep.Source: §3.1 · pp. 6–7

3.2 MHCap

  • MHCap pairs OpenMHC’s 19 HealthKit channels with 7,005,763 captions across 908,944 retained person-days from 2,081 participants.Captions retain user and date identifiers for matching to signal records.Source: §3.2 · p. 7, Table 2 · p. 7

3.3 WESADCap

  • TimeCap also generates WESADCap from 20 chest- and wrist-worn physiological channels, pairing affective-state intervals with concurrent physiological and movement evidence.The dataset-specific instantiation uses the same four extractor tiers at the session level.Source: §3.3 · pp. 7–8

4 Experiments and Evaluation

The evaluation compares caption-augmented and label-only OpenTSLM on health-outcome prediction, reporting user-level binary results, per-week gains, and a caption-family ablation.

4 Experiments and Evaluation

  • The comparison covers 10 participant-level outcomes from a week of hourly sensor data; four held-out clinical conditions are excluded from fine-tuning, and the models share an architecture but differ in training mix.Per-user AUROC, AUPRC, and Spearman ρ are primary metrics; per-week metrics provide a finer-grained view.Source: §4 · p. 8

4.1 Results

  • OpenTSLM-Cap outperforms OpenTSLM on 29 of 32 per-user binary metric-scale tasks across the 1B and 3B models.The user-level comparison uses per-user AUROC and AUPRC for binary tasks.Source: Table 3 · p. 9, §4.1 · p. 9
  • At 3B, held-out clinical per-week macro-F1 gains are +3.4 for Cardiovascular Disease, +6.1 for Diabetes, +2.6 for Coronary Artery Disease, and +2.6 for Atrial Fibrillation; in-distribution ordinal QWK gains are +4.8 on BMI and +5.0 on Happiness over OpenTSLM.Source: §4.1 · p. 9, Table 4 · p. 10
  • Any single caption family raises mean per-user AUROC over OpenTSLM by 1.9 to 3.2 points, but only the full mix beats it on all eight binary tasks.The ablation uses one seed per arm.Source: §4.1 · p. 10, Table 5 · p. 10

5 Discussion

The discussion presents TimeCap and MHCap as reusable, auditable resources, reports gains on health-outcome prediction, and identifies boundaries in evaluation, linguistic diversity, and cohort coverage.

5 Discussion

  • TimeCap adds cross-channel synthesis and a channel-schema abstraction to template hierarchies, offering a grounded, schema-driven alternative to LLM-driven pipelines.Source: §5 · p. 11
  • Caption-augmented OpenTSLM outperforms an architecturally identical label-only baseline on 29 of 32 user-level binary evaluations, with the largest gains on zero-shot clinical tasks.Source: §5 · p. 11
  • MHCap supports training and evaluating TSLMs on real consumer-wearable data, while TimeCap’s deterministic captions can be regenerated and traced to signals for auditing.Source: §5 · p. 11
  • The comparison leaves LLM-caption and prompted-VLM baselines untested, and fixed template banks with 20 variants per type limit linguistic diversity.Additional zero-shot ordinal transfer to held-out blood-pressure and sleep-duration categories failed across configurations.Source: §5 · p. 11
  • The OpenMHC cohort skews toward white, male US participants with a median age of 39, limiting confidence that pretrained models generalize uniformly to underrepresented populations.Clinical-task per-user AUROC values of 59.4 to 70.5 indicate population-level patterns rather than reliable individual predictions; individual use would require independent prospective validation and safeguards.Source: §5 · p. 11
  • Future work could add a data-availability gate before structural detection, use constrained language-model post-processing, and apply the pipeline to new cohorts or non-health time series.High-frequency signals such as ECG would additionally need dedicated feature extraction.Source: §5 · p. 11

6 Conclusion

TimeCap and MHCap address paired wearable-caption data gaps with a deterministic pipeline and an open caption layer; caption-augmented OpenTSLM outperforms its label-only baseline on user-level binary evaluations.

6 Conclusion

  • 29 of 32 user-level binary evaluations favor caption-augmented OpenTSLM over its label-only baseline.Source: §6 · p. 12
  • TimeCap is a deterministic pipeline without a language model in generation, and MHCap is an open paired-caption layer demonstrated on WESAD.Source: §6 · p. 12
  • Held-out clinical gains suggest captions help OpenTSLM understand sensor data beyond the tasks used in training.Source: §6 · p. 12

A Training

OpenTSLM and OpenTSLM-Cap are fine-tuned for one epoch, with LoRA updates; training uses specified optimization settings and holds other run conditions constant to isolate caption supervision.

A Training

  • Both models are fine-tuned for one epoch, with all parameter updates routed through LoRA adapters.The adapters use r=16 and α=32.Source: A Training · p. 18
  • Training uses AdamW, linear warmup and decay, gradient clipping at 1.0, weight decay 10^-2, and batch size 8.Warmup spans the first 3% of steps; learning rates are 2×10^-4 for the time-series encoder and 1×10^-4 for the projector and LoRA parameters.Source: A Training · p. 18
  • Each configuration runs on one NVIDIA H100 80 GB for about 35 hours, totaling roughly 140 GPU-hours across four configurations.Source: A Training · p. 18
  • The two runs at each scale are identical in other respects, isolating the effect of caption supervision.Source: A Training · p. 18

B Evaluation Protocol

The evaluation uses participant-level splits and week-long wearable inputs, applies benchmark data-quality filters, and reports both user-level and weekly metrics. MHCap is intended for research on wearable time-series and language alignment, not diagnosis or individual decisions.

B.1 Splits

  • 2,081 of 11,894 sharable participants remain after filtering; a seeded 60/10/30 participant-level split leaves 604 in test and prevents participant overlap.Source: §B.1 · p. 18
  • Table 6 gives binary test-set composition: 604 unique users overall, with per-task counts varying by label availability; † marks a held-out clinical task.Source: Table 6 · p. 18
  • The model receives one 168-hour week of multichannel sensor data interleaved with text; Figure 2 presents the evaluation prompt template.Source: §B.1 · p. 19, Figure 2 · p. 19
  • Ordinal-task reliability depends on score spread: concentrated distributions such as Happiness have less effective signal than more even distributions such as BMI.Source: Table 7 · p. 20

B.2 Aggregation Conventions

  • Weekly predictions are aggregated into per-user metrics, with binary predictions averaged and ranked across users and ordinal QWK computed against rounded user means.Table 4 reports weekly metrics; Table 3 reports user-level metrics.Source: §B.2 · p. 20

B.3 Data Curation

  • A retained week requires five of seven days passing wear-time and variance filters; missing values remain explicit, with no imputation.Days require at least 12 hours estimated wear time; zero values are selectively set to missing under channel-specific rules.Source: §B.3 · p. 20

C Responsible Use and Release

  • MHCap is intended for time-series–language research; evaluated outcomes are self-reported, do not forecast future clinical events, and are not for diagnosis or individual decisions.Source: C Responsible Use and Release · p. 20

C.1 Cohort Composition

OpenMHC participants are drawn from digital-health-app users and are not representative of the general public; reported demographics describe the full cohort, while MHCap’s filtered subset may differ.

C.1 Cohort Composition

  • The cohort is not representative of the general public: 77.1% of participants reporting biological sex are male, and about 90% are from the US.These figures describe all 11,894 OpenMHC participants; MHCap’s 2,081 participants passed an additional data-quality filter and may differ.Source: §C.1 · p. 21

C.2 Data Availability and Access

OpenMHC data and precomputed MHCap captions are available to qualified researchers under research-use terms, while the TimeCap pipeline is publicly released under the MIT license. Caption records retain shifted dates and participant identifiers but contain only sensor-channel descriptions, not survey or demographic information.

C.2 Data Availability and Access

  • Access to OpenMHC and precomputed MHCap captions is restricted to qualified researchers under research-use terms; the TimeCap pipeline is MIT-licensed and authorized researchers can regenerate captions.The terms limit use to personal, non-commercial research, prohibit redistribution without written permission, and exclude diagnosis or patient care; agreements cannot technically prevent misuse.Source: §C.2 · p. 21

C.3 Privacy

  • MHCap adds no new data collection and describes only 19 sensor channels, without survey responses or demographic information.Records retain participant identifiers and dates, but dates are shifted by a random whole-week offset per participant, and caption text contains no calendar dates.Source: §C.3 · p. 21

D Dataset-Specific Configurations

The OpenMHC and WESAD configurations apply the same four-tier pipeline through dataset-specific channel mappings, aggregation, detection, and synthesis choices. The following section presents time-aligned caption examples from both datasets.

D.1 MHCap Configuration

  • OpenMHC continuous movement and energy channels receive mean, max, min, and standard-deviation summaries, while sleep and workout intervals are enriched with aligned physiological and movement evidence.Semantic intervals are drawn from binary channels; synthesizers add context such as heart rate, distance, step count, and active energy where available.Source: §D.1 · pp. 21–23

D.2 WESADCap Configuration

  • WESAD streams are converted to per-second channels, and its state intervals are enriched with synchronized physiology and movement evidence using state-specific synthesizers.Continuous channels use mean, max, min, and standard deviation; spike thresholds are conservative descriptive heuristics, not clinical or diagnostic cutoffs, and sensor-specific thresholds address differing scales and artifact-prone signals.Source: §D.2 · pp. 24–26

E Generated Caption

  • Time-aligned examples of captions generated by TimeCap for each dataset are presented in the following section.Source: E Generated Caption · p. 27

E.1 MHCap Captions

The appendix presents MHCap caption examples for sleep and multiple activities, including workouts, walking, and yoga, followed by WESADCap examples for 15 subjects.

E.1 MHCap Captions

  • Figures 3–12 show MHCap examples spanning sleep, cycling, elliptical, running, HIIT, walking, and yoga.Source: §E.1 · p. 27, Figure 3 · p. 27, Figure 4 · p. 27
  • One HIIT example identifies activity from minutes 696–735, while its paired caption reports a mean heart rate of 129 bpm and peak of 162 bpm.The paired caption also states the day's mean heart rate was 73 bpm.Source: §E.1 · p. 30

E.2 WESADCap Captions

Loading 2610.21274v1…