PaperlayerView exact revision on arXiv

Source-linked AI summary

SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition

Zechen Li, Shohreh Deldari, Linyao Chen, Hao Xue, Flora D. Salim

arXiv:2410.10624v4cs.CL

TL;DR

LLMs face computational and input-processing challenges when applied to motion-sensor time series. SensorLLM aligns sensor data with automatically generated trend descriptions, then tunes the aligned representations for HAR; the authors report competitive results across five datasets and generalization to datasets distinct from those used for alignment.

  • Problem

    LLMs face computational demands, limited and imbalanced labels, and challenges processing numerical, long, multichannel sensor inputs for HAR.

    Source: §1 · pp. 1–2

  • Method

    SensorLLM uses automatically generated trend descriptions to align sensor time-series with an LLM, then applies Task-Aware Tuning for HAR.

    Source: §1 · p. 2

  • Results

    SensorLLM matches or surpasses state-of-the-art models across five HAR datasets and maintains strong Task-Aware Tuning performance on datasets distinct from those used during alignment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that SensorLLM captures complex sensor patterns and generalizes across diverse datasets without dataset-specific alignment.

    Source: §6 · p. 9

    The alignment focuses on trend descriptions; frequency-domain features, periodicity, and higher-order patterns remain possible alternatives for future study.

    Source: §7 · p. 9

Abstract

from arXiv · show

We introduce SensorLLM, a two-stage framework that enables Large Language Models (LLMs) to perform human activity recognition (HAR) from sensor time-series data. Despite their strong reasoning and generalization capabilities, LLMs remain underutilized for motion sensor data due to the lack of semantic context in time-series, computational constraints, and challenges in processing numerical inputs. SensorLLM addresses these limitations through a Sensor-Language Alignment stage, where the model aligns sensor inputs with trend descriptions. Special tokens are introduced to mark channel boundaries. This alignment enables LLMs to capture numerical variations, channel-specific features, and data of varying durations, without requiring human annotations. In the subsequent Task-Aware Tuning stage, we refine the model for HAR classification, achieving performance that matches or surpasses state-of-the-art methods. Our results demonstrate that SensorLLM evolves into an effective sensor learner, reasoner, and classifier through human-intuitive Sensor-Language Alignment, generalizing across diverse HAR datasets. We believe this work establishes a foundation for future research on time-series and text alignment, paving the way for foundation models in sensor data analysis. Our codes are available at https://github.com/zechenli03/SensorLLM.

1 Introduction

SensorLLM addresses challenges in applying LLMs to motion-sensor data by aligning sensor time-series with descriptive text, then tuning the aligned representations for HAR. The authors report competitive performance across five datasets and strong performance on datasets distinct from those used for alignment.

1 Introduction

  • Applying LLMs to sensor data faces computational demands and limited, imbalanced labels for fine-tuning, while prompting faces numerical-tokenization, context-length, multivariate-input, and prompt-design challenges.Numerical tokenization can disrupt temporal dependencies, and long sequences may be truncated or costly to process.Source: §1 · pp. 1–2
  • SensorLLM aligns wearable sensor data with automatically generated trend descriptions using statistical analyses and predefined templates, without modifying the LLM or requiring manual annotations.The authors describe the approach as precise, scalable, and interpretable.Source: §1 · p. 2
  • In Sensor-Language Alignment, generated question-answer pairs and a pretrained encoder map sensor time-series into an LLM-interpretable space, while special channel tokens support multichannel dependencies.The alignment is described as mitigating issues associated with text-specific tokenization.Source: §1 · p. 2
  • Task-Aware Tuning uses aligned embeddings for HAR while keeping LLM parameters frozen, and the framework supports varying sequence lengths and arbitrary channel counts.These inputs include multivariate time-series data.Source: §1 · p. 2
  • SensorLLM matches or surpasses state-of-the-art models across five HAR datasets and maintains strong Task-Aware Tuning performance on datasets distinct from those used during alignment.The authors describe this as evidence of robustness and generalizability.Source: §1 · p. 2

2 Related Work

Related work covers representing time series as text, using multimodal LLMs for sensor data, and applying LLMs to HAR. These approaches use numerical strings or encoders, visual prompting or joint representations, and varied pipelines for activity recognition.

2 Related Work

  • The section organizes prior work into time-series-as-text modeling, multimodal LLMs for sensor data, and LLM-based HAR.A broader review of deep-learning HAR and additional LLM-based forecasting is located in Appendix A.1.Source: §2 · p. 2

LLMs for Time Series as Text

  • Time-series-as-text methods include PromptCast for zero-shot forecasting and LLMTime (Gruver et al., 2023) for numerical-string inputs; encoder-based methods map time-series embeddings to language-model spaces.The review notes that raw numerical strings may fail to capture temporal dependencies and repetitive patterns (Spathis and Kawsar, 2023).Source: §2 · pp. 2–3

MLLMs for Sensor Data

  • For sensor data, Yoon et al. (2024) use visual prompting, while IMU2CLIP aligns inertial streams with text and video in a joint representation space.The review says visualizing sensor signals can lower token costs compared with raw-text baselines.Source: §2 · p. 3

LLMs for Human Activity Recognition

  • HAR approaches include object-sequence prompting (Xia et al., 2023), synthesized text and virtual IMU training in IMUGPT (Leng et al., 2023), and agentic reasoning and retrieval in ZARA (Li et al., 2025).HARGPT (Ji et al., 2024) downsamples signals to 10Hz for zero-shot HAR and reports matching or surpassing traditional and deep-learning models on simple tasks.Source: §2 · p. 3

3 Methods

SensorLLM aligns sensor time series with automatically generated trend descriptions, then uses the aligned representations for HAR. Its design keeps the pretrained LLM and time-series embedder frozen while training lightweight alignment and classification modules.

3 Methods

  • Only the alignment MLP and classifier are trainable; the pretrained LLM and time-series embedder remain frozen.The trainable shares are 5.67% (535.9 M) of parameters in the first stage and 0.12% (10.5 M) in the second.Source: §3 · p. 3

3.1 Sensor-Text Data Generation

  • Predefined templates and statistical analyses generate diverse question-answer pairs that describe observed trends without manual annotation.Templates are randomly combined to capture trend changes while retaining accuracy and scalability.Source: §3.1 · p. 3

3.2 Sensor-Language Alignment

  • The alignment stage processes each sensor channel independently, segments its readings at random lengths, and maps time-series embeddings into the LLM’s text-aligned embedding space.Random segment lengths expose the model to short-term fluctuations and long-term trends.Source: §3.2 · p. 4
  • Special tokens mark channel boundaries, and the aligned representations are combined with instructions and questions for sensor-text generation.The model is trained to predict generated tokens from preceding sensor and text embeddings.Source: §3.2 · pp. 4–5

3.3 Task-Aware Tuning

  • Task-Aware Tuning uses overlapping sensor windows and the pretrained alignment module to map multichannel readings to activity labels.The final hidden state is pooled and passed through a fully connected layer to predict activity classes using cross-entropy loss.Source: §3.3 · p. 5

4 Experiments

SensorLLM was evaluated on five varied HAR datasets and against GPT-4o for sensor-trend description. It outperformed GPT-4o across the reported trend metrics and ranked first on four of five HAR datasets, while placing second on UCI-HAR.

4 Experiments

  • Experiments assess sensor interpretation and HAR classification using Chronos, which was not pretrained on motion sensors, with test data kept unseen during alignment.HAR results average five random runs; experiments report F1-macro and accuracy.Source: §4 · p. 5, §4.3 · p. 6

4.1 Datasets

  • The evaluation uses five public datasets that vary in subjects, sensor placement, sampling rates, channel configurations, and activity types, spanning laboratory and free-living settings.Except for UCI-HAR’s fixed split, datasets use subject-independent splits with test users unseen during training.Source: §4.1 · p. 5

4.2 Sensor Data Understanding

  • SensorLLM outperforms GPT-4o across all reported trend-description metrics, with human evaluation also favoring SensorLLM, particularly for shorter sequences.CAPTURE-24 results are weaker than those on other datasets; the authors suggest its longer sequences trained with the same parameters may contribute.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Human Activity Recognition

  • SensorLLM achieves the best performance on four of five datasets and ranks second on UCI-HAR.It reaches 48.6% F1-macro on CAPTURE-24, a +5.0% absolute gain over Attend (43.6%); Table 2 reports F1-macro and accuracy over five runs.Source: Table 2 · p. 7, §4.3 · p. 7

5 Ablation Studies

The ablations identify alignment and textual prompts as important to HAR performance, while showing that deeper alignment modules and smaller models involve trade-offs. Cross-dataset tests also report comparable performance after transferring alignment between datasets with shared sensor channels.

Removing Alignment Hurts

  • SensorLLM consistently outperforms the Task-only variant across five datasets, whether or not textual prompts are included.The Task-only model often performs comparably to or worse than traditional time-series baselines; Chronos embeddings alone are insufficient for optimal HAR performance.Source: §5 · pp. 7–8

Textual Prompts Enhance HAR

  • Textual prompts consistently improve F1-macro scores across all datasets, with a more pronounced effect in the full SensorLLM architecture.The comparison tests additional textual information, such as statistical features for each sensor channel.Source: §5 · p. 8

MLP Depth Trade-offs

  • A two-layer alignment MLP improves F1-macro on UCI-HAR and MHealth but slightly decreases it on PAMAP2; one hidden layer offers a stated accuracy-efficiency balance.The comparison is between one hidden layer (1024 → 2048 → 4096) and two (1024 → 2048 → 3072 → 4096).Source: §5 · p. 8, Figure 3 · p. 8

Smaller SensorLLM Still Compete

  • SensorLLM-3b performs slightly below SensorLLM-8b but remains competitive, outperforming Attend on USC-HAD and MHealth and closely trailing it on UCI-HAR.The smaller variant uses Chronos-base and LLaMA3.2-3b and is evaluated on USC-HAD, UCI-HAR, and MHealth.Source: §5 · p. 8, Figure 4 · p. 8

Special Tokens Improve Performance

  • Removing special tokens slightly lowers F1-macro on PAMAP2 and MHealth, with the gap tending to widen as the number of sensor channels increases.The tokens act as learned channel identifiers and provide structural cues for channel-wise dependencies.Source: §5 · p. 8, Table 4 · p. 8

Alignment Enables Generalization

  • Cross-dataset experiments report performance comparable to same-dataset training when alignment and task-aware tuning use different datasets with shared sensor channels.The tested datasets differ in sensor wearing position, sampling rates, and activity distributions.Source: §5 · p. 9, Table 5 · p. 9

6 Conclusions

SensorLLM aligns sensor data with automatically generated text at a human-perception level and reports strong HAR performance across diverse datasets. Its cross-dataset results support generalization without dataset-specific alignment, and the work presents a foundation for Sensor-Text MLLMs.

Contributions

  • SensorLLM aligns sensor data with automatically generated text at a human-perception level, beyond machine-level alignment.The framework is presented as the first multimodal approach of this kind.Source: §6 · p. 9

Results

  • Experiments across diverse datasets report robust performance with variable-length sequences, multi-channel inputs, and textual metadata.The conclusion also reports superior performance in HAR tasks.Source: §6 · p. 9

Implications

  • Cross-dataset results highlight generalizability without dataset-specific alignment, and the work establishes a foundation for Sensor-Text MLLMs with potential sensor-reasoning applications.The authors release code and a data-generation pipeline to support future time-series and text integration research, particularly in low-resource domains.Source: §6 · p. 9

7 Limitations

The paper identifies limitations in both its fixed-class downstream classifier and its trend-focused sensor-text alignment. It points to generative or zero-shot approaches and richer sensor descriptions as directions for future work.

Choice of Classifier

  • A fixed-class classifier may constrain adaptation to new activity categories and does not fully use the LLM’s reasoning potential.The framework is compatible with other downstream heads, but generative or zero-shot approaches are left for future work.Source: §7 · p. 9
  • Generative or zero-shot approaches could broaden applications to activity discovery and open-set recognition.Source: §7 · p. 9

Scope of Sensor-Text Alignment

  • Trend-focused alignment leaves frequency-domain features, periodicity, and higher-order patterns as unexplored alternatives for sensor descriptions.The paper suggests investigating these characteristics to improve time-series interpretation and applications beyond activity recognition.Source: §7 · p. 9

A.1 More related work

The related work traces HAR from hand-crafted features to deep learning, then reviews LLM applications to time series, especially forecasting. It also describes approaches that align sensor or time-series representations with language-model spaces.

Deep learning in human activity recognition

  • HAR research progressed from hand-crafted feature extraction and machine-learning classifiers to CNNs, LSTMs, and other deep models for temporal and multimodal sensor data.The review notes that optimized Activity Recognition Chain feature extraction could rival or outperform end-to-end deep learning.Source: §A.1 · p. 14

Large Language Models for Time-Series Forecasting

  • LLM-based forecasting methods encode numerical time series as text or tokens, adapt pretrained models, or combine language models with time-series techniques.The reviewed approaches include prompting, numerical-string encoding, pretrained-model adaptation, and trend-seasonality decomposition.Source: §A.1 · p. 14
  • Other reviewed methods map time-series or sensor embeddings into language-model spaces using encoders, contrastive alignment, supervised alignment, or text prototypes.The cited approaches include instance-wise, feature-wise, and text-prototype-aligned contrast.Source: §A.1 · p. 14

A.2 Sensor–Text Data Pair Generation

The section describes generating sensor-text QA pairs from readings using templates and dataset-tailored prompts. It distinguishes trend-analysis and trend-summary tasks for alignment, followed by multivariate activity classification using aligned representations.

A.2 Sensor–Text Data Pair Generation

  • Predefined sentence templates are randomly selected to create diverse QA pairs; GPT-4o generates synonymous variations, while numerical values or textual information replace placeholders to align each pair with sensor data.Source: §A.2 · p. 14
  • The system prompt provides dataset-specific attributes, including sensor frequency and sampling rate, to tailor responses to each dataset.Source: §A.2 · pp. 14–15
  • Table 7 summarizes training and testing sensor-text QA-pair counts for the two stages.Source: Table 7 · p. 15, §A.2 · p. 15
  • The alignment stage pairs variable-length univariate sequences with two QA tasks: describing within-window signal changes and summarizing overall window behavior.Source: §A.2 · p. 15
  • The task-aware stage uses multivariate sequences for activity classification, with statistical information from each sensor channel included in the aligned input representation.Source: §A.2 · p. 15

A.3 Chronos

Chronos converts real-valued time series into discrete tokens through normalization and quantization, then trains a model to predict tokens over a forecast horizon. Special tokens support variable-length inputs, and the approach is designed for integration with language models without architectural changes.

A.3 Chronos

  • Chronos tokenizes real-valued time series into discrete representations for language-model training using normalization followed by quantization.Source: §A.3 · p. 15
  • Mean scaling normalizes series, then quantization maps normalized values to tokens using bin centers and bin edges.The mapping assigns values to bins spanning the real line.Source: §A.3 · p. 15
  • PAD and EOS tokens handle padding and sequence endings, allowing Chronos to process variable-length inputs.Source: §A.3 · p. 15
  • Chronos trains a categorical token distribution with cross-entropy loss over the forecast horizon, conditioned on historical context.The context length is C, the forecast horizon is H, and pθ is the predicted token distribution.Source: §A.3 · pp. 15, 18
  • The approach integrates with language models without architectural modifications and supports distribution learning across time-series datasets.Source: §A.3 · p. 18

A.4 GPT-4o Prompt for Sensor Data Trend Analysis

This appendix describes the prompt and evaluation procedures for trend descriptions, the datasets and baselines used for HAR, and examples comparing SensorLLM with GPT-4o. Across the presented shorter and medium-length sequences, SensorLLM’s descriptions more closely capture trends and their durations.

A.4 GPT-4o Prompt for Sensor Data Trend Analysis

  • GPT-4o’s structured prompt standardizes how it analyzes sensor data and generates trend descriptions for comparison with SensorLLM.The evaluation also compares model descriptions with human evaluations and NLP metrics.Source: §A.4 · p. 18, Table 10 · p. 18

A.5 Evaluation Metrics for Sensor-Language Alignment Stage

  • Trend-description evaluation assesses SensorLLM outputs using lexical-overlap metrics (BLEU-1, ROUGE-1, ROUGE-L, METEOR), embedding-based SBERT semantic similarity, and GPT-4o scores for correctness, completeness, and coherence.BLEU-1 measures unigram precision; ROUGE-1 measures unigram recall and ROUGE-L the longest common subsequence; METEOR combines precision and recall with stemming and synonym matching. SBERT computes cosine similarity between sentence embeddings, while GPT-4o assigns scores from 1 to 5.Source: §A.5 · pp. 18–19

A.6 Datasets

  • The study evaluates five datasets with differing sensing setups, ranging from body-worn inertial sensors and smartphone sensing to CAPTURE-24’s free-living, 3-channel wrist accelerometry collected for over 24 hours per participant.CAPTURE-24 includes annotated data from 151 participants.Source: §A.6 · p. 19

A.7 Baselines for Task-Aware Tuning Stage

  • The task-aware benchmark includes HAR and time-series baselines, including Chronos+MLP as a simpler comparison for SensorLLM’s design.Chronos processes each channel separately in this baseline, whose embeddings are concatenated and passed to an MLP classifier.Source: §A.7 · pp. 20, 22

A.8 Evaluation Metrics for Task-Aware Tuning Stage

  • F1-macro averages per-class F1 scores equally, making it suitable for evaluating performance on imbalanced activity labels.Unlike micro F1, it does not give more weight to frequent classes.Source: §A.8 · p. 22

A.9 Sensor-Language Alignment Stage Output Examples

  • SensorLLM outperforms GPT-4o on the presented shorter and medium-length sequences, including trend changes and durations.The examples also show GPT-4o receiving a score of 3 for moderate errors in trend durations and shifts against ground truth.Source: §A.9 · pp. 23–24, Table 13 · p. 25, Table 14 · p. 26

Related papers

Loading 2410.10624v4…