Source-linked AI summary
HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?
Sijie Ji, Xinzhe Zheng, Chenshu Wu
TL;DR
Whether LLMs can recognize human activities from raw sensor data in a zero-shot setting remains an open question amid doubts about their understanding of the physical world. HARGPT tests this by prompting LLMs with raw IMU data. Across two datasets, it reports zero-shot recognition that outperforms existing methods.
Problem
Whether LLMs can understand physical-world data remains debated, with concerns about their performance on time-series data and meaningful interaction with the physical world.
Source: §1 · p. 1Method
HARGPT feeds raw sensor data to LLMs with prompts that use IMU expertise and request step-by-step analysis.
Source: §1 · p. 1, §2.1.3 · p. 3Results
Across two datasets, LLMs performed zero-shot HAR and outperformed existing machine-learning and deep-learning methods on unseen users.
Source: §1 · pp. 1–2Takeaways & Limitations
The findings support LLMs’ potential to process IoT sensor data and perform zero-shot HAR without fine-tuning or domain-specific expertise-guided prompt engineering.
Source: §1 · p. 2Llama2-70b sometimes gives two possible answers, while GPT4 occasionally offers constructive opinions without answering, issues the authors say should be mitigated for real-time, stable CPS interaction.
Source: §3.1 · p. 6
Abstract
from arXiv · showhide
There is an ongoing debate regarding the potential of Large Language Models (LLMs) as foundational models seamlessly integrated with Cyber-Physical Systems (CPS) for interpreting the physical world. In this paper, we carry out a case study to answer the following question: Are LLMs capable of zero-shot human activity recognition (HAR). Our study, HARGPT, presents an affirmative answer by demonstrating that LLMs can comprehend raw IMU data and perform HAR tasks in a zero-shot manner, with only appropriate prompts. HARGPT inputs raw IMU data into LLMs and utilizes the role-play and think step-by-step strategies for prompting. We benchmark HARGPT on GPT4 using two public datasets of different inter-class similarities and compare various baselines both based on traditional machine learning and state-of-the-art deep classification models. Remarkably, LLMs successfully recognize human activities from raw IMU data and consistently outperform all the baselines on both datasets. Our findings indicate that by effective prompting, LLMs can interpret raw IMU data based on their knowledge base, possessing a promising potential to analyze raw sensor data of the physical world effectively.
1 INTRODUCTION
Foundation models attract interest for their reasoning and generalization, but LLMs’ ability to interpret the physical world remains contested. HARGPT tests zero-shot HAR from raw IMU data, reporting 80% average accuracy on unseen data and outperforming existing methods.
1 INTRODUCTION
- The paper examines LLMs amid concerns that text-and-image training leaves them with surface-level physical-world knowledge and subpar digital or time-series analysis.The introduction also notes difficulty interacting meaningfully with the physical world, such as generating precise control sequences.Source: §1 · p. 1
- Researchers’ interest in integrating LLMs with cyber-physical systems has grown, but their potential as physical-world models remains largely unexplored.Prior sensor examples relied on direct textual information and expert knowledge supplied through prompt engineering.Source: §1 · p. 1
- 80% average accuracy on unseen data; HARGPT reports that LLMs outperform existing methods on the same activities performed by unseen users.The study evaluates raw IMU data from datasets with distinct and similar activity patterns.Source: §1 · pp. 1–2, Figure 2 · p. 2
- HARGPT demonstrates zero-shot human activity recognition without fine-tuning or domain-specific expertise-guided prompt engineering.The authors also present this as evidence of LLM proficiency in processing IoT sensor data for physical-world tasks.Source: §1 · p. 2
- The authors discuss insights from LLMs handling IoT sensor data as groundwork for future research on LLM adoption in cyber-physical systems.The paper frames this as an exploration of current capabilities rather than a comprehensive assessment.Source: §1 · pp. 1–2
2 HARGPT: ZERO-SHOT HAR WITH LLMS
HARGPT tests prompted GPT-4 on activity-recognition tasks spanning distinct and similar movement patterns. GPT4-CoT outperforms trained baselines on unseen-user tests across both datasets.
2.1 Experiments
- The experiment contrasts activities with distinct patterns, such as sleep and walking, against highly similar upstairs and downstairs movements.Source: §2.1 · p. 2, §2.1.1 · p. 2
2.1.1 Dataset Setup.
- Capture24 covers four distinct wrist-acceleration activities; HHAR covers stairs using waist-carried phone IMU from nine users, with both datasets user-split 4:1:1:2 and LLM inputs downsampled to 10 Hz.Capture24 data are sampled at 100 Hz; HHAR phone data are sampled at 100 or 200 Hz.Source: §2.1.1 · p. 2
2.1.2 Baseline.
- The trained baselines are Random Forest, SVM, DCNN, and LIMU-LSTM, which processes input features in chronological order.The classical and deep-learning baselines require training data to acquire classification ability.Source: §2.1.2 · pp. 2–3
2.1.3 Prompt Structure.
- The prompt combines role-play and step-by-step guidance without an answer template, and asks about the collection details, down-sampled sequence, and possible activities.A direct-output prompt replaces the final step-by-step instruction to test the effect of CoT.Source: §2.1.3 · p. 3
2.2 Evaluation
- On Capture24 unseen users, GPT4-CoT averaged F1 0.795, 0.33 above GPT4-DO; on HHAR, accuracy was close to 80%, with top LIMU-LSTM nearly 10% lower.The overall results are reported in Table 1 for Capture24 and Table 3 for HHAR.Source: §2.2 · pp. 3–4, Table 1 · p. 4, Table 3 · p. 4
- In the walking example, GPT-4 uses expert motion patterns, then describes periodic up-and-down movement before judging the input most likely to be walking.Source: §2.2 · p. 4
3 DISCUSSION
The discussion presents LLMs as extending logical analysis beyond text to physical-world data, translating raw sensor signals into abstract representations. It also notes variation in LLM reasoning and occasional GPT-4 replies without answers.
- The authors describe LLM reasoning as extending beyond textual information to analysis of data from the physical realm.Source: §3.1 · pp. 4–5
- Figure 4 presents GPT-4's detailed step-by-step inference for a walking example.Source: Figure 4 · p. 5
- Figure 5 compares other LLMs' inference results for the walking scenario.Source: Figure 5 · p. 5
- LLMs can convert raw digitized sensor data into abstract linguistic representations such as periodic, stationary, or abrupt.Source: §3.1 · p. 5
- Unlike conventional machine- and deep-learning approaches, LLMs generate plausible assertions by emulating human thought rather than matching raw-data characteristics.Source: §3.1 · p. 5
- GPT-4 occasionally gives constructive opinions without corresponding answers, which the authors say must be mitigated for real-time, stable CPS interaction.Source: §3.1 · p. 6
4 FUTURE WORK AND CONCLUSIONS
The conclusion reports that LLMs can perform zero-shot HAR with high accuracy and robustness, while calling for broader evaluation to establish where and how well they work. It also identifies standardized procedures and more comprehensive benchmarks as future needs.
- The study concludes that LLMs can perform zero-shot HAR with high accuracy and robustness while processing IoT sensor data without expertise-guided examples.Source: §4 · p. 6
- Further study is needed to determine the extent and cases in which LLMs are effective, alongside standardized evaluation and more comprehensive benchmarks.Source: §4 · p. 6
- The findings suggest potential for LLMs to analyze raw sensor data in CPS and other real-world data, including WiFi Channel State Information.Source: §4 · p. 6