PaperlayerView exact revision on arXiv

Source-linked AI summary

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, Thomas Plötz

arXiv:2402.01049v1cs.CV

TL;DR

Large labeled datasets are difficult to obtain for HAR, limiting access to annotated wearable data. This paper evaluates language-based cross-modality transfer and extends IMUGPT with motion filtering and diversity-based stopping. Its evaluation reports best performance with GPT-3.5 and T2M-GPT, and over 50% lower generation time and compute demands from diversity metrics.

  • Problem

    HAR depends on labeled wearable data, but large datasets are scarce because annotation is expensive, requires expertise, and raises privacy concerns.

    Source: §1 · p. 2

  • Method

    The paper evaluates language-based cross-modality transfer and extends IMUGPT with a filter for irrelevant motion and diversity metrics for deciding when to stop generation.

    Source: §1 · p. 3

  • Results

    GPT-3.5 and T2M-GPT yielded the best downstream performance, and diversity metrics reduced virtual IMU generation time and compute resources by over 50%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that the extensions and evaluation demonstrate language-based cross-modality transfer's effectiveness for deriving HAR systems in practical scenarios.

    Source: §1 · p. 3

    The initial IMUGPT prototype left unclear when to stop generation and whether generated virtual IMU data were relevant before downstream training and evaluation.

    Source: §3 · p. 7

Abstract

from arXiv · show

One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality (IMU). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which open up IMUGPT for practical use cases beyond a mere proof of concept.

1 INTRODUCTION

HAR models depend on labeled wearable data, but collecting and annotating large datasets is costly and raises privacy concerns. This paper evaluates language-based cross-modality transfer and adds filtering and diversity-based stopping to make virtual IMU generation more practical.

1 INTRODUCTION

  • Large labeled HAR datasets are scarce because annotation is expensive, requires domain expertise, and can raise privacy concerns.HAR commonly uses supervised learning to classify segmented data streams into activities.Source: §1 · p. 2
  • Cross-modality transfer converts data from other modalities into virtual IMU data, providing additional labeled training data for downstream classifiers.Prior approaches use sources such as video, motion capture, and optical data.Source: §1 · p. 2
  • A motion filter excludes irrelevant generated sequences, while diversity metrics track text and motion diversity to identify when data generation can stop.The proposed stopping points indicate diversity saturation, beyond which additional generation does not enhance dataset diversity or downstream classifier performance.Source: §1 · p. 3
  • GPT-3.5 and T2M-GPT yielded the best downstream performance, while diversity metrics reduced virtual IMU generation time and compute resources by over 50%.The evaluation compared different LLMs and motion synthesis models across a broader range of activities and practically relevant HAR scenarios.Source: §1 · p. 3
  • The paper presents its extensions and evaluation as evidence that language-based transfer can derive effective HAR systems while alleviating the shortage of large labeled training data.The authors frame this as addressing, if not circumventing, a major challenge for contemporary HAR.Source: §1 · p. 3

2 RELATED WORK

Related work frames wearable HAR as constrained by limited labeled data and describes cross-modality transfer as one approach to that shortage. It then reviews text-driven motion synthesis and the original IMUGPT pipeline that this paper extends.

2.1 Challenges with HAR using Wearables

  • Wearable HAR faces limited labeled data, annotation difficulty, conflicting variation across activities and users, and sensor noise.The review notes that collecting and labeling wearable data can be impractical and privacy-sensitive; cross-modality transfer is among several approaches to sparse data.Source: §2.1 · p. 4

2.2 Cross Modality Transfer

  • Cross-modality transfer creates virtual target-modality data from other sources, including motion capture, videos, and optical signals.Prior examples include motion-capture-to-IMU methods, video-derived simulated sensors, and optical-motion-field reconstruction.Source: §2.2 · pp. 4–5

2.3 Text-Driven Motion Generation

  • Text-driven motion synthesis models such as MotionGPT, T2M-GPT, MotionDiffuse, and ReMoDiffuse generate human motion sequences from text descriptions.The reviewed approaches include VQ-VAE-based motion tokenizers and language models, diffusion models, and hybrid retrieval.Source: §2.3 · pp. 5–6

2.4 IMUGPT

  • IMUGPT uses an LLM to generate diverse activity descriptions, motion synthesis to turn them into 3D sequences, and motion-to-IMU conversion to produce HAR training data.The original system used combinations of LLMs, motion synthesis, and signal processing, avoiding the need to search for videos.Source: §2.4 · pp. 6–7
  • Preliminary IMUGPT experiments on locomotion activities found that extracted virtual IMU data could significantly improve downstream HAR performance.The review presents this as a promising proof of concept.Source: §2.4 · p. 7

3 IMUGPT 2.0: TOWARDS PRACTICAL APPLICATIONS OF LANGUAGE-BASED CROSS MODALITY TRANSFER FOR HAR

The initial IMUGPT prototype leaves data-generation stopping and virtual-data relevance unresolved; IMUGPT 2.0 adds diversity-based stopping tools and a motion filter to address these practical gaps.

3.1 Diversity Metrics

  • Diversity is measured in embeddings using absolute scores for within-set diversity and comparative scores for differences between sets.The paper motivates text-based measurement because textual descriptions precede motion synthesis, potentially saving computation.Source: §3.1 · p. 8

3.1.1 Absolute Diversity Metrics.

  • Absolute diversity uses embedding spread, measured by axiswise standard deviations and distances from a centroid; higher scores indicate more diverse sets.The standard-deviation method treats embeddings as a high-dimensional cluster; the centroid method summarizes distances from one centroid for the set.Source: §3.1.1 · p. 9

3.1.2 Comparative Diversity.

  • Comparative diversity uses Gaussian-kernel MMD between embedding sets, with larger values indicating more different distributions.The authors apply this measure to compare diversity before and after adding generated descriptions.Source: §3.1.2 · pp. 9–10

3.1.3 Saturation Point Identification.

  • Saturation Point Identification adds batches proportional to the existing prompt pool and stops when comparative diversity no longer changes meaningfully, subject to a tolerance.The authors resample the smaller embedding set to match the larger set and average repeated MMD calculations, returning a standard deviation as well.Source: §3.1.3 · pp. 10–11

3.2 Filtering out incorrectly generated motion sequences

  • The motion filter captions synthesized sequences with MotionGPT, then asks an LLM whether each caption depicts the target activity and filters incorrect sequences.The filter addresses synthesis models that can produce sequences that do not accurately describe the intended activity.Source: §3.2 · p. 11

3.2.1 Motion Captioning.

  • MotionGPT encodes motion into discrete tokens and combines them with task text; decoded output tokens yield a textual motion caption.Its motion tokenizer uses a VQ-VAE-based encoder, while the unified vocabulary can represent text, motion, or both.Source: §3.2.1 · p. 12

3.2.2 Activity Filtering Using Large Language Models.

  • Activity filtering frames caption–activity matching as binary classification and uses zero-shot LLM judgments, returning yes or no for each caption.Prompts include the activity and 10 captions; the authors note that the LLM can mislabel captions.Source: §3.2.2 · pp. 12–13

4 ON THE EFFECTIVENESS OF LANGUAGE-BASED CROSS MODALITY TRANSFER FOR HAR

The evaluation tests language-based virtual IMU generation across datasets, language models, motion-synthesis models, and HAR classifiers, then examines diversity metrics and motion filtering. GPT-3.5 and T2M-GPT lead key comparisons, while benefits from virtual data vary by dataset and classifier.

4 ON THE EFFECTIVENESS OF LANGUAGE-BASED CROSS MODALITY TRANSFER FOR HAR

  • The study compares original IMUGPT across LLMs and motion-synthesis models, then evaluates diversity metrics, saturation-point identification, and motion filtering.The first part uses more activities than the earlier pilot study; the second builds on its results.Source: §4 · p. 13

4.1.1 Datasets.

  • Five public HAR datasets and three classifiers are evaluated using two-second windows with 50% overlap and dataset-specific cross-validation averaged over three random seeds.The datasets include IMU recorded at varying on-body locations; HAD-AW uses five-fold stratified cross-validation, while the other datasets use leave-one-subject-out.Source: §4.1.1 · pp. 13–14, §4.1.2 · p. 14

4.2 Impact of LLM on Textual Description Generation and HAR

  • The description-generation experiment compares GPT-3.5, GPT-4, Palm 2, Gemini, and LLaMa 2 using activity names and sample descriptions as prompts.It generated 1,000 descriptions per activity in batches of 50, then cleaned responses before motion synthesis and virtual IMU extraction.Source: §4.2.1 · p. 14

4.2.1 Experimental Setting.

  • Across the tested LLMs, GPT-3.5, GPT-4, and LLaMa 2 produce descriptions with minimal errors, while Palm 2 and Gemini sometimes describe unrelated entities or repeat descriptions.The problems were especially noticeable for complex MyoGym workouts; the authors expect the first three LLMs to yield better downstream performance.Source: §4.2.2 · p. 14

4.2.2 Qualitative Evaluation of Generated Text.

  • With Random Forest, GPT-3.5 improves over GPT-4 by 1.0%, LLaMa 2 by 0.9%, Palm 2 by 0.6%, and Gemini by 0.8% across datasets.Gemini and Palm 2 are within 1% of the other LLMs despite qualitatively worse descriptions; Table 2 reports Macro F1 and defines Real Data as the baseline without virtual IMU.Source: §4.2.3 · p. 15, Table 2 · p. 15

4.2.3 Results.

  • Virtual IMU significantly improves Random Forest performance on every dataset except HAD-AW, while deep-model gains are mixed and absent on USC-HAD, HAD-AW, and MyoGym.The authors attribute the deep-model pattern to overfitting to virtual IMU features that do not generalize to the test data.Source: §4.2.3 · p. 15

4.3.1 Experiment Setting.

  • The study converts GPT-3.5 descriptions into motion sequences with four synthesis models; VAE-based models vary sequence length, whereas diffusion models use a fixed six-second length.For diffusion models, some sequences ended prematurely or continued with repeated or miscellaneous motions; this was not observed with VAE-based models.Source: §4.3.1 · p. 16

4.3.2 Results.

  • With Random Forest, T2M-GPT improves downstream performance by 0.7% over MotionGPT, 1.2% over MotionDiffuse, and 2.2% over ReMoDiffuse across datasets.Differences are statistically significant; ReMoDiffuse performs better for MyoGym across all three classifiers, and HAD-AW results are worse than real data.Source: §4.3.2 · p. 16

4.4 Diversity as a Predictor for HAR Model Performance

The paper tests whether diversity predicts downstream HAR performance and evaluates a saturation-point method and motion filter. Saturation-point generation preserves classifier performance with substantially fewer descriptions, while filtering improves performance on some datasets but has no significant effect on others.

4.4.1 Correlations.

  • At each step, the analysis adds 5% more textual descriptions, computes text and motion MMD, and correlates both diversity measures with per-class F1 changes relative to models trained only on real IMU data until saturation.Source: §4.4.1 · p. 17
  • Text and motion diversity were highly correlated, while diversity had a negative moderate-to-strong correlation with downstream F1 change in every dataset except HAD-AW.Because lower MMD indicates higher diversity, the authors interpret higher diversity as indicating better downstream performance; RealWorld had the strongest correlation, while some activities in USC-HAD, PAMAP2, and MyoGym declined.Source: §4.4.1 · p. 17, Table 4 · p. 18

4.4.2 Saturation Point Identification Algorithm Evaluation.

  • The saturation evaluation compares classifiers trained with 1,000 descriptions against those using descriptions generated only up to each activity’s saturation point.Both sets are converted to virtual IMU data, combined with the corresponding real datasets, and used to train downstream classifiers.Source: §4.4.2 · pp. 17–18
  • Around 400–600 descriptions per activity preserve similar or better performance than 1,000, saving at least 50% of data-generation time and compute.The comparison spans activities, datasets, and classifiers; saturation points are computed separately for each activity.Source: §4.4.2 · p. 18, Table 5 · p. 18

4.5.1 Experimental Setting.

  • The motion-filter evaluation generated 50 GPT-3.5 descriptions for each of RealWorld’s eight activities and converted them to motion sequences with T2M-GPT.Sequences were manually judged for whether they accurately portrayed the target activity; GPT-3.5, GPT-4, and human annotators were then compared using captions.Source: §4.5.1 · pp. 18–19

4.5.2 Validation Results.

  • GPT-4 filtering achieved 0.901 precision versus 0.706 for GPT-3.5, reducing incorrectly generated sequences in the remaining dataset from 35.5% to 9.9%.GPT-3.5 precision was comparable to human performance.Source: §4.5.2 · p. 19, Table 6 · p. 19

4.5.3 Activity Recognition Performance.

  • Motion filtering improved downstream performance by 4.3% on HAD-AW and 4.1% on MyoGym, but did not significantly affect the other three datasets.The reported gains were statistically significant; filtering also tended to reduce sequence diversity, potentially contributing to performance decreases.Source: §4.5.3 · p. 19, Table 7 · p. 20

5.1 How much impact does context information have on downstream HAR performance?

The study tests whether adding age, physique, and weather context to GPT-3.5 activity descriptions changes downstream HAR performance. It finds no significant difference, and notes that the motion-training data may not represent those contextual distinctions.

5.1 How much impact does context information have on downstream HAR performance?

  • Adding age, physique, and weather context to GPT-3.5 descriptions produced no significant downstream performance difference on RealWorld with T2M-GPT.The authors suggest weather may be absent from indoor motion-capture training data, while HumanML3D descriptions lack age and physique information, so generated motions may not reflect those differences.Source: §5.1 · pp. 20–21, Table 8 · p. 21

5.2 Do multi-sensor setups benefit more from the use of virtual IMU data?

Virtual IMU data improves results more consistently on multi-sensor datasets, while filtering increases relevance but reduces diversity and does not significantly change downstream performance. The discussion attributes limitations to activity coverage and subtle movements that the motion-to-sensor method cannot capture.

5.2 Do multi-sensor setups benefit more from the use of virtual IMU data?

  • Virtual IMU data significantly improves downstream performance on RealWorld and PAMAP2 across all three classifiers, but benefits are less consistent on single-sensor datasets.HAD-AW does not improve; USC-HAD and MyoGym improve only with the Random Forest Classifier.Source: §5.2 · p. 21

5.3 How does the motion filter affect data diversity?

  • Filtering reduces motion-sequence diversity across datasets and metrics, counteracting the benefit of improved relevance so downstream performance does not change significantly.The comparison uses standard-deviation and centroid diversity metrics for unfiltered and retained sequences.Source: §5.3 · p. 21, Table 9 · p. 22

5.4 Limitations and Future Work

  • For HAD-AW, virtual IMU data does not improve downstream performance because some activities are missing from HumanML3D and IMUSim cannot capture subtle real-IMU movements.Some irrelevant sequences remain after filtering, and the mismatch between real and virtual IMU data can lead deep classifiers to overfit virtual-only features.Source: §5.4.1 · pp. 21–22, §5.4.2 · p. 22

5.4.1 Lack of Expressivity of Motion Synthesis Model.

  • HumanML3D-trained synthesis models cannot generate activities absent from that dataset, leaving some irrelevant sequences after filtering and worsening downstream performance.The passage names washing hands and drawing as HAD-AW examples not included in HumanML3D.Source: §5.4.1 · p. 22

5.4.2 Lack of Capturing Subtle Motions.

  • IMUSim misses subtle wrist movements in activities such as writing and typing, producing virtual-to-real distribution differences that can lead deep classifiers to overfit.Proposed future work is a multimodal motion-capture and sensor dataset to train a motion-to-virtual-IMU model for all body locations.Source: §5.4.2 · p. 22

6 CONCLUSION

The paper extends IMUGPT with motion filtering and diversity metrics, then evaluates the system across HAR datasets, LLMs, and motion-synthesis models. The evaluation identifies GPT-3.5 with T2M-GPT as the best-performing combination, while the extensions reduce data-generation effort by over 50%.

6 CONCLUSION

  • Over 50% reduction in data-generation time and computational demands follows from adding motion filtering and diversity metrics to IMUGPT.The metrics help determine how much virtual data to generate, while the filter targets relevance.Source: §6 · p. 22
  • GPT-3.5 with T2M-GPT leads to the best downstream performance in an evaluation spanning HAR datasets, LLMs, and motion-synthesis models.The conclusion presents this as the result of a comprehensive evaluation.Source: §6 · pp. 22–23
  • IMUGPT 2.0 highlights the potential for more complex and generalizable HAR systems as a response to the lack of large labeled datasets.Source: §6 · p. 23

A USING LLM FOR ACTIVITY FILTERING

The appendix presents the prompts used to have an LLM judge whether motion captions accurately describe a specified activity.

A USING LLM FOR ACTIVITY FILTERING

B DATASETS

The experiments use five benchmark datasets, summarized in Table 10. They span everyday and gym activities, varied participant counts, and different sensing setups.

B DATASETS

  • The evaluation uses five HAR benchmark datasets, with Table 10 summarizing dataset characteristics and activity-sample proportions.Source: B DATASETS · p. 28, Table 10 · p. 29
  • RealWorld includes 15 subjects, eight activities, and sensors at seven body locations, recorded under realistic conditions.Each activity was recorded for ten minutes except jumping, recorded for 1.7 minutes.Source: B DATASETS · p. 28
  • PAMAP2 records 12 activities from nine subjects using three IMUs and a heart-rate monitor, with over ten hours of data sampled at 100 Hz.Source: B DATASETS · p. 28
  • USC-HAD contains five trials of 12 activities from 14 subjects, recorded across days and indoor and outdoor settings with sensors at the front right hip.Source: B DATASETS · p. 28
  • HAD-AW uses right-wrist Apple Watch recordings from 16 subjects across 25 activities, excluding six gym activities covered by MyoGym.Subjects repeated each activity ten times, yielding approximately 160 samples per activity.Source: B DATASETS · p. 28
  • MyoGym covers 30 gym activities performed by ten subjects using a forearm Myo armband; experimental data were downsampled to 20 Hz to match virtual IMU data.Subjects performed ten repetitions of each activity using free weights targeting different muscle groups.Source: B DATASETS · p. 28

C REAL AND VIRTUAL DATASET SIZES

This section reports real and virtual IMU dataset sizes across different language models, motion-synthesis models, and use of the motion filter.

C REAL AND VIRTUAL DATASET SIZES

D SAMPLE GENERATED TEXT DESCRIPTIONS

This section gives examples of generated activity descriptions, including descriptions from different LLMs and contextual descriptions for walking.

D SAMPLE GENERATED TEXT DESCRIPTIONS

  • The prompt template asks for a short description of one person performing an activity and an adjective describing how they perform it.The prompt specifies fewer than 15 words and excludes descriptions of the environment.Source: D SAMPLE GENERATED TEXT DESCRIPTIONS · p. 31
  • Table 14 presents example generated textual descriptions from various LLMs.Source: Table 14 · p. 32
  • Table 15 presents GPT-3.5-generated descriptions of walking with context information.Source: Table 15 · p. 33

E SAMPLE GENERATED MOTION SEQUENCES

This section presents visualized example motion sequences for activities in the RealWorld dataset.

E SAMPLE GENERATED MOTION SEQUENCES

  • The section shows visualized motion-sequence examples for activities in the RealWorld dataset.Source: Figure 5 · p. 34

Related papers

Loading 2402.01049v1…