PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A. Xu, Winnie Chow, Martin Maritsch, Robert Jakob, Ning Wang, Juncheng Liu, Aradhana Verma, Brian Han, Daniel Seung Kim, Henry Chubb, Scott Ceresnak, Aydin Zahedivash, Alexander Tarlochan Singh Sandhu, Fatima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver Aalami, Filipe Barata, Paul Schmiedmayer

arXiv:2510.02410v3cs.LG

TL;DR

LLMs remain limited in reasoning over time-series data, despite the importance of temporal information in medicine. OpenTSLM extends pretrained LLMs with native time-series inputs through soft prompting or cross-attention, and its models outperform baselines across text–time-series reasoning tasks while Flamingo scales better for longer sequences.

  • Problem

    LLMs remain limited in reasoning over time-series data, although clinical decision-making depends on temporal patterns in health information.

  • Method

    OpenTSLM extends pretrained LLMs with native time-series inputs using either soft prompting or cross-attention for text–time-series reasoning.

  • Results

    OpenTSLM models outperform baselines across benchmarks, including 69.88 F1 on Sleep-CoT and 65.44 F1 on HAR-CoT, while Flamingo maintains stable memory as sequence lengths grow.

  • Takeaways & Limitations

    OpenTSLM enables small open-weight LLMs to reason over longitudinal time-series data while retaining language-generation capabilities.

  • Takeaways & Limitations

    SoftPrompt becomes impractical for longer or multiseries inputs because its memory requirements grow with sequence length and context usage.

Abstract

from arXiv · show

LLMs have emerged as powerful tools for interpreting multimodal data. In medicine, they hold particular promise for synthesizing large volumes of clinical information into actionable insights and digital health applications. Yet, a major limitation remains their inability to handle time series. To overcome this gap, we present OpenTSLM, a family of Time Series Language Models (TSLMs) created by integrating time series as a native modality to pretrained LLMs, enabling reasoning over multiple time series of any length. We investigate two architectures for OpenTSLM. The first, OpenTSLM-SoftPrompt, models time series implicitly by concatenating learnable time series tokens with text tokens via soft prompting. Although parameter-efficient, we hypothesize that explicit time series modeling scales better and outperforms implicit approaches. We thus introduce OpenTSLM-Flamingo, which integrates time series with text via cross-attention. We benchmark both variants against baselines that treat time series as text tokens or plots, across a suite of text-time-series Chain-of-Thought (CoT) reasoning tasks. We introduce three datasets: HAR-CoT, Sleep-CoT, and ECG-QA-CoT. Across all, OpenTSLM models outperform baselines, reaching 69.9 F1 in sleep staging and 65.4 in HAR, compared to 9.05 and 52.2 for finetuned text-only models. Notably, even 1B-parameter OpenTSLM models surpass GPT-4o (15.47 and 2.95). OpenTSLM-Flamingo matches OpenTSLM-SoftPrompt in performance and outperforms on longer sequences, while maintaining stable memory requirements. By contrast, SoftPrompt grows exponentially in memory with sequence length, requiring around 110 GB compared to 40 GB VRAM when training on ECG-QA with LLaMA-3B. Expert reviews by clinicians find strong reasoning capabilities exhibited by OpenTSLMs on ECG-QA. To facilitate further research, we provide all code, datasets, and models open-source.

1. Introduction

Clinical reasoning depends on how health signals evolve over time, but current LLMs remain limited in reasoning over time-series data. OpenTSLM addresses this gap by treating time series as a native modality and comparing implicit soft prompting with explicit cross-attention.

  • Clinical assessment, diagnosis, and treatment rely on temporal patterns in vital signs, medications, laboratory values, and disease progression.
  • Current multimodal LLMs can synthesize text, images, audio, and video, but time-series reasoning remains a critical blind spot.
  • Prior methods either tokenize time series as text or reprogram LLMs with fixed prediction heads, limiting performance or free-form text generation.
  • Soft prompting preserves generation by concatenating learned time-series tokens with text, but longer series increase context length and quadratic self-attention compute.
  • OpenTSLM integrates time series as a native modality and compares soft prompting with cross-attention to model temporal structure in pretrained LLMs.

2. Related Work

Prior work bridges continuous time series and discrete language through tokenization, soft prompting, or cross-attention, while time-series foundation models primarily target forecasting. These approaches differ in whether they preserve language generation and whether they adapt pretrained LLMs directly.

  • Time-series language-model research uses three main strategies: text tokenization, soft prompting, and cross-attention.
  • Tokenization lets LLMs forecast or infer health information from physiological and wearable data through text representations and prompting.
  • Soft-prompt methods encode time series into embeddings, but some use classification heads or task solvers that disable free-form text generation.
  • Cross-attention has been used to extract statistical summaries from a single sensor, but prior work trained new encoders and decoders rather than adapting a pretrained LLM.
  • Time-series foundation models use large-scale pretraining for zero-shot forecasting, including decoder-only architectures, patching, probabilistic objectives, and multivariate settings.

3. Methods

OpenTSLM supports multiple time-series inputs by extending pretrained LLMs with time-series encoders and either soft prompting or gated cross-attention. The architectures preserve temporal information while enabling interleaved text–time-series reasoning.

  • Architecture overview: OpenTSLM provides two architectures: Soft Prompting interleaves projected time-series tokens with text, while Flamingo integrates time-series through gated cross-attention.Both support interleaving multiple text and time-series segments; Flamingo uses a time-series encoder, Perceiver Resampler, gated cross-attention layers, and a frozen language model backbone.
  • Time-series encoder: The time-series encoder divides each input into non-overlapping patches, maps them to embeddings with 1D convolution and positional encoding, then processes the resulting sequence.For an input of length L and patch size p, the encoder produces N = L/p patches.
  • Soft prompting architecture: OpenTSLM-SP projects encoded patches into the LLM hidden space and fine-tunes the pretrained language model with LoRA adapters.Its three components are a time-series encoder, a projection layer, and a pretrained LLM.
  • Soft prompting architecture: Soft prompting forms inputs from pre-prompt text, interleaved projected time-series tokens and descriptions, and a post-prompt question.Each projected time-series embedding is aligned with the LLM hidden space and paired with a textual description as a text–time-series chunk.
  • Cross-attention architecture: OpenTSLM-Flamingo replaces Flamingo’s vision encoder with a time-series encoder and conditions the LLM through cross-attention triggered by special tokens.The ⟨TS⟩ token activates conditioning on the latent time-series representation, while ⟨endofchunk⟩ marks the end of its textual description.
  • Cross-attention architecture: The Flamingo-style design uses a Perceiver Resampler to produce a fixed-size latent representation before gated cross-attention layers integrate it into the LLM.The cross-attention layers are inserted every N transformer blocks, with a learnable gate controlling the time-series influence.

4. Experiments

OpenTSLM is evaluated on time-series question answering and reasoning tasks, comparing two architectures, baselines, memory use, rationales, and pretrained time-series encoders. Across benchmarks, OpenTSLM generally achieves the strongest performance, while Flamingo offers better memory scaling and pretrained encoders further improve results.

  • Quantitative results: OpenTSLM models achieve the highest reported performance across TSQA, HAR-CoT, Sleep-CoT, and ECG-QA-CoT compared with tokenized, plot-based, and text-only baselines.The comparison covers Llama and Gemma backbones alongside GPT-4o and multiple baseline variants.
  • Quantitative results: 97.50 F1 and 97.54 accuracy are achieved on TSQA by OpenTSLM-SoftPrompt with Llama3.2-1B, exceeding the strongest tokenized-finetuned baseline at 84.54 F1 and 82.06 accuracy.OpenTSLM-Flamingo variants are close, while GPT-4o with image inputs reaches 59.24 F1 and 62.10 accuracy.
  • Quantitative results: 69.88 F1 and 81.08 accuracy are reached on Sleep-CoT by OpenTSLM-SoftPrompt with Llama3.2-1B, versus 9.05 F1 and 24.19 accuracy for the best tokenized-finetuned baseline.On HAR-CoT, OpenTSLM variants reach 65.44 F1 and 71.48 accuracy, while OpenTSLM-Flamingo leads ECG-QA-CoT with 40.25 F1 and 46.25 accuracy.
  • Memory use: OpenTSLM-Flamingo maintains near-constant training memory across datasets, whereas SoftPrompt memory varies with time-series and text length.Across simulated sequence shapes, Flamingo remains constant as the number and length of series increase, while SoftPrompt scales with both dimensions and can become a bottleneck.
  • Qualitative results and expert evaluation: Expert cardiologist review finds correct or partially correct ECG interpretation in 92.9% of evaluated cases from OpenTSLM-Flamingo-Llama3.2-3B rationales.Positive assessments were 85.1% for clinical context integration, 65.5% for ECG pattern recognition, and 62.5% for clinical reasoning.
  • Pretrained time-series encoder: OpenTSLM-Flamingo with pretrained Chronos-2 generally improves performance across LLM backbones and datasets, including TSQA, Sleep-CoT, and ECG-QA-CoT.For example, Sleep-CoT improves from 49.33 to 64.57 F1 for Llama3.2-1B and from 51.38 to 68.33 for Gemma3-270M.

5. Discussion

OpenTSLM models outperform baselines across time-series reasoning tasks, while Flamingo offers more stable memory scaling than SoftPrompt for longer or multivariate sequences. Qualitative and expert evaluations further indicate strong temporal reasoning on ECG-QA.

  • OpenTSLM models consistently outperform baselines across the evaluated time-series reasoning tasks.Finetuned text-only models perform especially poorly on Sleep-CoT, while OpenTSLM variants remain competitive across task types.
  • Flamingo sustains stable memory as sequence length and the number of series increase, whereas SoftPrompt becomes impractical for long inputs.SoftPrompt exceeds 180GB in simulations with 10,000-length series, while Flamingo uses up to 60GB for five such series with Llama-3.2-3B.
  • OpenTSLM-Flamingo provides qualitative Chain-of-Thought rationales for HAR, sleep staging, ECG-QA, and M4 captioning.The examples are generated with OpenTSLM-Flamingo and Llama3.2-1B.
  • Expert evaluations with cardiologists find strong reasoning capabilities and temporal understanding on raw ECG sensor data.The discussion also reports that OpenTSLM correctly identified relevant ECG features in most cases, with missing context 7.1% of the time.
  • OpenTSLM reaches 40.25 F1 on ECG-QA-CoT while retaining language-generation capabilities unlike fixed-head alternatives.The paper contrasts its rationale-generating formulation with approaches that use fixed classification or forecasting heads.

6. Conclusion

OpenTSLM enables small-scale language models to outperform much larger text-only models on time-series tasks. The conclusion presents lightweight, domain-adapted architectures as effective without massive model scales.

  • Both OpenTSLM variants enable small-scale LLMs to outperform much larger text-only models on time-series tasks.The paper frames this result as evidence that lightweight, domain-adapted architectures can achieve strong performance without massive model scales.

A.1. Training details

The training data are organized across pretraining and task-specific Chain-of-Thought reasoning stages, using an 80/10/10 train, validation, and test split. Table 4 summarizes the datasets used.

  • The datasets use an 80/10/10 train, validation, and test split.
  • The training overview records each dataset's sample counts, number of series, sequence length, and sampling frequency.
  • Table 4 overviews datasets used for Stage 1 pretraining and Stage 2 task-specific Chain-of-Thought reasoning.

A.1.1. TRAINING CONFIGURATION

Training uses a linear learning-rate schedule with warmup, runs for up to 200 epochs with early stopping, and uses task-informed learning-rate choices. Chronos-2 is downloaded pretrained, while its Flamingo learning rates are trained from scratch.

  • The training configuration is introduced as the common setup for the models.
  • Training uses a linear learning-rate schedule with warmup covering 10% of total training steps.
  • Models train for up to 200 epochs with early stopping after five epochs of patience.
  • Learning-rate choices follow Chow et al. (2024), while pretrained Chronos-2 is downloaded directly and its OpenTSLM-Flamingo learning rates are trained from scratch.

A.2. Generation of multivariate time-series CoT datasets

The study introduces three multivariate time-series Chain-of-Thought datasets for human activity recognition, sleep staging, and ECG question answering. Rationales were generated with GPT-4o from plotted signals and correct labels, then checked for consistency.

  • Three datasets cover HAR, sleep staging, and ECG question answering with explicit reasoning chains.The datasets are HAR-CoT, SleepEDF-CoT, and ECG-QA-CoT.
  • GPT-4o generated each rationale from a signal plot and its correct label, prompting a chain of reasoning toward the answer.
  • The authors manually reviewed a subset of generated samples to check that rationales were consistent and sensible.Plots retained the original data without normalization, with multiple signals shown as separate subplots in one figure.

A.2.1. HUMAN ACTIVITY RECOGNITON (HAR) COT

HAR-CoT converts multivariate accelerometer recordings into challenging binary activity-recognition tasks with generated rationales. Samples use 2.56-second triaxial windows and pair each correct activity with a dissimilar alternative.

  • Data Processing: HAR-CoT uses 2.56-second windows of triaxial accelerometer data visualized across X, Y, and Z axes.The dataset merges multiple HAR sources and retains eight activity classes.
  • Prompt for CoT generation: GPT-4o rationales were prompted to analyze movement intensity and behavior before revealing the final activity label.The prompt required a single natural paragraph and prohibited mentioning class labels until the final sentence.
  • Answer Selection Strategy: Each sample presents one correct activity and one deliberately dissimilar alternative to require analysis of movement patterns.The dissimilarity-based selection strategy was designed to make binary classification challenging rather than rely on simple pattern recognition.

A.2.2. SLEEP STAGE CLASSIFICATION CHAIN-OF-THOUGHT (SLEEPEDF-COT)

SleepEDF-CoT and ECG-QA-CoT pair physiological time-series with binary or clinical questions to elicit explicit reasoning. The appendix also reports baseline-output failures and compares memory scaling across OpenTSLM variants.

  • SleepEDF-CoT: SleepEDF-CoT uses 30-second single-channel Fpz-Cz EEG windows covering Wake, N1, N2, N3, and REM stages.Prompts pair physiologically distinct stages and require reasoning from signal patterns before the final class.
  • ECG-QA-CoT: ECG-QA-CoT contains complete 12-lead recordings sampled at 100 Hz for rhythm, morphology, and diagnostic questions.The ECG prompts ask models to consider rhythm, rate, morphology, intervals, and abnormalities across all leads.
  • Baseline outputs: Some text-only baselines score 0% F1 because they fail to produce the required rationale-and-answer template.
  • Memory usage: Flamingo memory remains nearly constant as series count and length vary, whereas SoftPrompt grows with both and reaches out-of-memory conditions.For example, Llama-3B Flamingo remains near 60.7–61.1 GB, while SoftPrompt rises from 6.3 GB to 42.7 GB and reaches OOM for N≥3 at L=10,000.
Loading 2510.02410v3…