PaperlayerView exact revision on arXiv

Source-linked AI summary

Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Prompting

Arvind Pillai, Dimitris Spathis, Subigya Nepal, Amanda C Collins, Daniel M Mackin, Michael V Heinz, Tess Z Griffin, Nicholas C Jacobson, Andrew Campbell

arXiv:2502.07608v3cs.LGcs.HC

TL;DR

Long sensor time series are difficult to use with LLMs through text prompts, which can require many tokens and domain expertise. Time2Lang learns a direct mapping from TFM features into LLM representations, training on synthetic periodicity prediction. Across longitudinal health-sensing evaluations, it reports stable inference time across input lengths and outputs aligned with time-series autocorrelation.

  • Problem

    Text prompting for long sensor sequences can increase errors and token use, and effective prompts require domain expertise.

    Source: §1 · p. 2, §2.4 · p. 4

  • Method

    Time2Lang trains learnable mappings between frozen Chronos and LLaMA using synthetic data and periodicity prediction, then evaluates on longitudinal mental-health sensing tasks.

    Source: §1 · pp. 2–3, §3.2.1 · p. 4

  • Results

    Time2Lang maintains stable inference times across input lengths, and its mapped outputs capture time-series properties such as autocorrelation.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    The evaluations demonstrate that a TFM and an LLM can be integrated for health-related tasks while minimizing information loss and transferring performance across domains.

    Source: §1 · p. 3, §7 · p. 11

    The framework is evaluated with Chronos and LLaMA, while other tasks or modalities may benefit from alternative pretrained models.

    Source: §6.3 · p. 11

Abstract

from arXiv · show

Large language models (LLMs) show promise for health applications when combined with behavioral sensing data. Traditional approaches convert sensor data into text prompts, but this process is prone to errors, computationally expensive, and requires domain expertise. These challenges are particularly acute when processing extended time series data. While time series foundation models (TFMs) have recently emerged as powerful tools for learning representations from temporal data, bridging TFMs and LLMs remains challenging. Here, we present Time2Lang, a framework that directly maps TFM outputs to LLM representations without intermediate text conversion. Our approach first trains on synthetic data using periodicity prediction as a pretext task, followed by evaluation on mental health classification tasks. We validate Time2Lang on two longitudinal wearable and mobile sensing datasets: daily depression prediction using step count data (17,251 days from 256 participants) and flourishing classification based on conversation duration (46 participants over 10 weeks). Time2Lang maintains near constant inference times regardless of input length, unlike traditional prompting methods. The generated embeddings preserve essential time-series characteristics such as auto-correlation. Our results demonstrate that TFMs and LLMs can be effectively integrated while minimizing information loss and enabling performance transfer across these distinct modeling paradigms. To our knowledge, we are the first to integrate a TFM and an LLM for health, thus establishing a foundation for future research combining general-purpose large models for complex healthcare tasks.

1. Introduction

Behavioral sensing can support mental-health assessment, and LLMs offer promise for personalized health models. Time2Lang maps time-series foundation model features into LLM representations without converting signals into text, and is evaluated on longitudinal mental-health sensing tasks.

1. Introduction

  • Behavioral data from phones and wearables, including step counts and conversation patterns, can help detect conditions such as depression.LLMs’ in-context learning and ability to process modalities beyond text motivate their integration with behavioral sensing.Source: §1 · pp. 1–2
  • Text prompting for long sensor sequences can increase errors and token use, while constructing effective prompts requires domain expertise.Time-series foundation models can generate representations that capture fundamental time-series characteristics.Source: §1 · p. 2
  • Time2Lang maps features from frozen Chronos to frozen LLaMA through learned networks, and is evaluated on two longitudinal mental-health datasets.The tasks use step counts for daily depression prediction and conversation duration for flourishing analysis.Source: §1 · pp. 2–3
  • Time2Lang is trained with synthetic data and reports near-constant inference time across input lengths, with mapped outputs aligned to autocorrelation.The authors describe the framework as minimizing information loss while transferring performance between time-series and text models.Source: §1 · p. 3

2. Related Work

Prior work repurposes frozen models across domains and applies LLMs to healthcare and sensor data, often by turning sensor series into text. Time2Lang instead feeds fixed-size time-series features into an LLM through a learned mapping.

2.1. Model Reprogramming

  • Model reprogramming repurposes frozen pretrained models across domains using input transformation and output mapping; Time2Lang applies this to health with self-supervised periodicity learning.The paper contrasts this with prior supervised reprogramming and says it explicitly incorporates periodicity properties.Source: §2.1 · p. 3

2.2. LLMs and Healthcare

  • Healthcare LLM work includes medical-text models and mental-health prompting, where added health context can improve performance but requires specialized domain knowledge.Examples include Med-PaLM 2, MedAlpaca, and Me-LLaMA.Source: §2.2 · p. 3

2.3. LLMs and Sensor Data

  • Sensor-to-LLM approaches use text prompts, visual prompts, or modality-specific encoders to represent time series.Examples include LLMTime and PromptCast for text prompting, and visual prompts for vision-language models.Source: §2.3 · pp. 3–4

2.4. Limitations of integrating mobile sensing with LLMs

  • For long behavioral sequences, text prompting can raise prediction error and token use, and effective prompt construction needs domain knowledge.Time2Lang offers an alternative by directly inputting fixed-size TFM embeddings to the LLM, eliminating prompting while aiming to preserve TFM performance.Source: §2.4 · p. 4

3. Methods

Time2Lang uses synthetic Gaussian-process series with known periodicities to train a mapping between frozen Chronos and LLaMA. Learnable encoder and projection networks are optimized for periodicity prediction, while the implementation uses modest-sized adapters and frozen foundation models.

3.1. Synthetic Data Generation

  • Synthetic series combine sampled non-periodic kernels with a periodic kernel, producing length-1440 Gaussian-process samples that repeat at a selected interval.The periodic values are 30, 60, 90, 120, 150, or 180; non-periodic kernels model trends, noise, and local variation.Source: §3.1 · p. 4

3.2. Time2Lang

  • Time2Lang sequences a frozen TFM, learnable input encoder, frozen LLM, and learnable projection to map time-series features into LLM representations.A residual connection from TFM features to the LLM-side representation is used during training to improve positive knowledge transfer.Source: §3.2 · p. 4, Figure 3 · p. 5

3.2.1. Training

  • The encoder and projection are trained on synthetic data with periodicity prediction as the self-supervised pretext task.The objective is multiclass prediction over periodicity classes, optimized with cross-entropy; the input is processed by frozen Chronos and passed through LLaMA embeddings.Source: §3.2.1 · pp. 4–5

3.2.2. Implementation

  • Implementation uses 200K synthetic samples and trains only the encoder and projection while Chronos base and LLaMA 3.2 remain frozen.The models have 200M and 1B parameters, respectively; the encoder has about 300K parameters and the projection about 1.7M.Source: §3.2.2 · p. 6

4. Experimental Setup

The experiments test Time2Lang across clinical and student populations, sensing modalities, and mental-health tasks, using prompting and model-based baselines. Evaluation uses subject-wise held-out data and AUROC/AUPRC.

4. Experimental Setup

  • The study evaluates daily depression detection from wearable step counts and flourishing classification from mobile-phone conversation duration.The tasks span a clinical MDD population and a non-clinical student population.Source: §4.1 · p. 6

4.1.1. Major Depressive Disorder (MDD)

  • The 90-day MDD study enrolled adults with MDD and collected wearable data alongside PHQ-9 surveys three times daily.Participants were screened using the structured clinical interview for DSM-5 and excluded for specified comorbidities and conditions.Source: §4.1.1 · p. 6
  • Daily depression labels use averaged PHQ-9 scores, with scores of 10 or higher indicating depression, aligned to 1,440 minute-level step counts.Days with under 25% missing data were zero-filled; days with at least 25% missingness were excluded. The dataset contains 17,251 person-days from 256 participants.Source: §4.1.1 · p. 6

4.1.2. StudentLife

  • StudentLife tracks conversation duration and flourishing among students over a 10-week term.The analysis uses 46 students, variable-length hourly conversation series averaging 1,455 ± 193 data points, and high/low flourishing groups defined by a median split of Flourishing Scale scores.Source: §4.1.2 · pp. 6–7

4.2. Baselines

  • Baselines include random classification, Chronos, Chronos embeddings directly placed in LLaMA, and three-shot LLaMA prompting with or without domain knowledge.The prompting setup includes numerical formatting and health and temporal context; the domain-knowledge variant adds task-specific context.Source: §4.2 · p. 7

4.3. Evaluation

  • Extracted Time2Lang embeddings train logistic-regression or random-forest classifiers, selected by three-fold cross-validation on subject-wise training splits.Final evaluation uses a held-out subject-wise test set, AUROC and AUPRC, and five shuffled training splits.Source: §4.3 · p. 7

5. Results

Time2Lang generally outperforms most baselines across depression detection and flourishing classification, while retaining near-constant inference latency as input length grows. Ablations and embedding correlations examine pre-training choices, architecture, and preserved temporal structure.

5.1. Overall Performance

  • 0.57 AUROC and 0.73 AUPRC for depression detection; flourishing AUROC and AUPRC exceed prompting and Chronos + LLaMA baselines by 0.15–0.20 and 0.09–0.13, respectively.Against Chronos, flourishing classification reaches 0.71 AUROC (+0.01) and 0.74 AUPRC (+0.05); overall, Time2Lang outperforms most baselines.Source: §5.1 · pp. 7–8, Table 1 · p. 8

5.2. Efficiency Analysis

  • Time2Lang has lower inference times than prompting across tested sequence lengths and StudentLife samples, with latency relatively stable as input length increases.Prompting time rises proportionally with input length; Figure 4 compares sequence lengths and variable-length conversation-duration data.Source: §5.2 · pp. 8–9, Figure 4 · p. 8

5.3. Ablation Study: Pre-training Data and Periodicity Classes

  • 200K synthetic samples and six periodicity classes yield the best downstream performance; gains from 100K to 200K are marginal, and nine classes add no improvement.The authors attribute the plateau at nine classes to the greater network complexity exceeding trainable parameter capacity.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4. Ablation Study: Architecture

  • Residual connections generally improve results, and the wider Time2Lang-L reaches 0.80 AUROC and 0.85 AUPRC for flourishing, gains of +0.09 and +0.11 over Time2Lang.Greater encoder depth did not improve performance; LLaMA 3.2 performed best among evaluated LLMs in general.Source: §5.4 · p. 9

5.5. Case Study: Correlation of Embeddings and ACF

  • 174 conversation-embedding dimensions (∼68%) and 70 step-count dimensions (∼27%) have at least one ACF lag with correlation above 0.3.Figure 6 summarizes Spearman correlations between original-series ACF values and embeddings; stronger correlations occur at lags 7–9.Source: §5.5 · pp. 9–10, Figure 6 · p. 10

6. Discussion

Time2Lang outperforms prompting across two mental-health tasks and preserves time-series characteristics with stable inference times. The discussion frames it as a step toward personalized sensing applications while noting limits in modality, model choice, and reasoning.

6.1. Findings

  • Time2Lang outperforms prompting across both tasks and metrics, and achieves marginal and notable improvements over Chronos.For daily depression prediction, AUROC=0.57 and AUPRC=0.73; for flourishing prediction, AUROC=0.71 and AUPRC=0.74.Source: §6.1 · p. 10
  • Inference time stays steady as input length increases, while LLaMA Prompting slows with longer sequences; mapped outputs show moderate-to-high correlations with ACF.The authors attribute the fixed-time behavior to fixed-sized TFM embeddings and the ACF connection to periodicity-prediction pretraining on synthetic data, which generalizes to real-world datasets.Source: §6.1 · p. 10

6.2. Implications

  • Continuous sensing could identify individual-specific patterns and support dynamic, real-time recommendations, but efficient sensing-data processing is a prerequisite.The discussion presents Time2Lang as an initial step toward mental-health systems that provide user-specific insights.Source: §6.2 · pp. 10–11

6.3. Limitations & Future Work

  • Time2Lang processes a single time-series modality, and evaluating it on more diverse datasets remains future work.The authors describe extension to multimodal data as a promising direction and identify generalizability as a reason to expand evaluation.Source: §6.3 · p. 11
  • Chronos and LLaMA were selected for fixed-length embeddings, language-modeling relation, and practical utility, but other tasks or modalities may favor different pretrained models.The authors call for systematic evaluation of modality interactions and model suitability rather than a one-size-fits-all prescription.Source: §6.3 · p. 11
  • Time2Lang’s reasoning capabilities remain limited despite the goal of computational efficiency for behavioral sensing in mental health.The authors aim to extend the approach toward more complex reasoning and generative tasks.Source: §6.3 · p. 11
  • The authors position Time2Lang as a foundation for integrating time series into LLMs and aim to extend it to complex reasoning and generative tasks.Their longer-term goal is a personalized LLM that understands individual behavior through daily data.Source: §6.3 · p. 11

7. Conclusion

The conclusion presents Time2Lang as a direct mapping from Chronos to LLaMA that maintains stable inference times and captures autocorrelation. The ethical discussion notes participant safeguards and risks including demographic bias and misuse.

7. Conclusion

  • Time2Lang maps Chronos outputs directly to LLaMA representations and, across longitudinal mental-health datasets, maintains stable inference times and captures autocorrelation.The conclusion reports improved efficiency, accuracy, and time-series comprehension compared with traditional prompting methods.Source: §7 · p. 11

Appendix A. Ethical Considerations

  • The study reports a predominantly White, female, U.S.-based participant sample and flags risks including unauthorized surveillance, biased decisions, and exploitation of health information.The study describes IRB approval, participant support, automatic alerts for severe depression or suicidality, and restricted data access.Source: Appendix A · p. 15

Appendix B. Architecture & Hyperparameters

The appendix describes the input encoder and downstream classifiers used in evaluation, alongside tables naming the CNN architecture and block types.

Appendix B. Architecture & Hyperparameters

  • The input encoder has 13 convolutional layers, followed by a projection layer with 768 and 256 features.The encoder starts with 768 input channels and 32 base filters; layers use batch normalization and ReLU.Source: Appendix B · p. 15
  • Logistic regression performed best for step-count depression detection, while random forest was used for flourishing classification from conversation duration.The listed hyperparameter search includes logistic-regression penalties and C values, and random-forest estimator counts and depth choices.Source: Appendix B · p. 15
  • The architecture tables are identified as a ResNet-style CNN and two basic block types.The input-encoder description points to Table 2 and Tables 3 and 4 for these components.Source: Appendix B · p. 15, Table 2 · p. 16, Table 4 · p. 16

Appendix C. Data

The appendix describes the MDD sample and depression-label distribution, self-report instruments, and prompt templates for depression and flourishing. The prompt variants differ in whether they include domain knowledge.

C.1. MDD

  • The MDD study recruited 300 participants for 90 days; average EMA compliance was 82.07%, and 68% of days were classified as depressed.The sample averaged 40.13 years of age, and 84% of participants were women.Source: §C.1 · p. 16, Figure 7 · p. 16

D.1. Patient Health Questionnaire - 9 (PHQ-9) (Kroenke et al., 2001)

  • The depression prompt asks users about symptoms over the past four hours using the PHQ-9 instrument.The appendix identifies the instrument as the Patient Health Questionnaire-9 (Kroenke et al., 2001).Source: §D.1 · p. 16, Table 5 · p. 17

D.2. Flourishing Scale (Diener et al., 2010).

  • The flourishing scale asks respondents to rate each statement from 1 to 7, from strongly disagree to strongly agree.The appendix identifies the instrument as the Flourishing Scale (Diener et al., 2010).Source: §D.2 · p. 16, Table 6 · p. 17

E.1. Depression Prompt

  • The depression prompt uses temporal and health context and asks for a probability from 0 to 1 based on daily step counts.The prompt converts labels to probabilities, which the authors report performed better than directly predicting 0 or 1.Source: §E.1 · pp. 16–17

E.2. Depression Prompt with Knowledge

  • The domain-knowledge depression prompt states that higher step counts are associated with lower depression symptoms.It asks for a 0-to-1 depression probability from daily step-count time series.Source: §E.2 · p. 17

E.3. Flourishing Prompt

  • The flourishing prompt asks for a 0-to-1 probability based on conversation-duration time series.This version states that conversation duration is related to flourishing without specifying a direction.Source: §E.3 · pp. 17–18

E.4. Flourishing Prompt with Domain Knowledge

  • The domain-knowledge flourishing prompt states that higher conversation duration is linked to higher flourishing.It asks for a 0-to-1 flourishing probability using conversation duration in seconds.Source: §E.4 · p. 18

Appendix F. Efficiency Analysis

Time2Lang scales more steadily than prompting; residual connections improve most reported metrics, while supervised task-specific models outperform it in three of four cases.

Appendix F. Efficiency Analysis

  • Task-specific models lead in 3/4 cases: depression AUROC 0.60 vs 0.57 but AUPRC 0.72 vs 0.73; flourishing AUROC 0.75 vs 0.71 and AUPRC 0.82 vs 0.74.The comparison is not direct: task-specific models receive downstream supervision, whereas Time2Lang uses feature extraction and a random forest.Source: Appendix F · p. 18, Table 8 · p. 19
  • Time2Lang beats LLaMA prompting in latency and throughput, maintaining near-constant performance across 512–4096 input sizes while prompting loses efficiency.Latency is single-prediction time; throughput is predictions per second at batch size 16.Source: Appendix F · p. 18, Table 7 · p. 19

H.1. Effect of residual connection

  • Residual connections improve 3/4 cases: depression step-count AUROC 0.54 to 0.57 and AUPRC 0.70 to 0.73; flourishing conversation AUROC 0.70 to 0.71 and AUPRC 0.77 to 0.74.The authors associate the gains, particularly for depression detection, with preserving and propagating temporal patterns.Source: §H.1 · p. 19

H.2. Architecture ablations

The architecture study finds the strongest overall results with a wider encoder and larger projection embedding; adding depth does not yield consistent gains.

H.2. Architecture ablations

  • A 64-filter encoder with a 512-dimensional embedding achieves the highest AUPRC for depression (0.73) and flourishing (0.85), and the highest flourishing AUROC (0.80).Using 10 residual blocks does not improve performance consistently and tends to perform slightly worse or similarly across embedding sizes.Source: §H.2 · p. 19

H.3. LLM ablation

Backbone performance varies by task, with a larger difference on flourishing than depression. The discussion identifies model choice, multimodal integration, missing data, and interpretability as areas for further work.

H.3. LLM ablation

  • For flourishing, Gemma 3 scores AUROC 0.41 and AUPRC 0.48; broader architecture and hyperparameter comparisons are needed for more definitive conclusions.The authors suggest its 1152-dimensional decoder embedding may be incompatible with the projection or TFM, but present this as a possibility.Source: §H.3 · p. 20

Appendix I. Extended Discussion

  • The discussion points to physiological sensing, including PPG and ECG, as potential extensions of Time2Lang’s health-sensing evaluation.It also mentions heart rate, respiratory patterns, and ballistocardiography as possible future time-series data.Source: Appendix I · p. 20
  • The discussion calls for systematic TFM and LLM comparisons and notes that unifying healthcare’s diverse modalities through an LLM remains challenging.Specialized encoders for each modality, merged within the reprogramming framework, are proposed as a possible approach.Source: Appendix I · p. 20
  • For missing sensor values, Time2Lang averages by downsampling within a window, then zero-fills remaining gaps; masked-autoencoder-style reconstruction is proposed for future resilience.The discussion also notes that real-world modalities may be absent or misaligned.Source: Appendix I · pp. 20–21

H.3. LLM ablation

  • Depression AUROC ranges from 0.57 to 0.58 and AUPRC is 0.73 across backbones; for flourishing, LLaMA 3.2 leads with AUROC 0.71 and AUPRC 0.74.All evaluated backbones have approximately 1B parameters.Source: §H.3 · p. 19, Table 11 · p. 21

Appendix I. Extended Discussion

  • Future interpretability work could probe activations and layer-wise attributions, or use counterfactual and concept-based explanations to clarify learned embeddings.The discussion identifies transparency as important in healthcare applications.Source: Appendix I · p. 21

Related papers

Loading 2502.07608v3…