PaperlayerView exact revision on arXiv

Source-linked AI summary

SenseAgent: An LLM Agent for Adaptive Cross-Domain IMU Sensing

Tianya Zhao, Chuan Liu, Xuyu Wang

arXiv:2609.32000v1cs.AI

TL;DR

Cross-domain IMU models may become unreliable for new users, devices, or positions, while different shifts can require different adaptation actions. SenseAgent uses an LLM as a runtime planner over sensing evidence, tools, memories, and verifiers; across diverse deployment shifts, it improves cross-domain activity recognition while preserving auditable decisions.

  • Problem

    Models trained in one setting can perform poorly on new users, devices, or positions, and fixed adaptation methods apply the same procedure regardless of the target shift.

    Source: §1 · p. 1

  • Method

    SenseAgent uses an LLM as a structured runtime planner over local sensing evidence, source and target memories, and verifier-checked adaptation actions.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    SenseAgent improves cross-domain IMU activity recognition across diverse user, position, device, and compound shifts.

    Source: §9 · p. 13

  • Takeaways & Limitations

    The results support closed-loop agentic sensing with auditable decisions and practical deployment overhead across the evaluated shifts.

    Source: §9 · p. 13

    The planner is limited to a fixed report-to-action interface: it configures predefined routes and tools but cannot expand the sensing action space.

    Source: §8.1 · p. 12

Abstract

from arXiv · show

Deep learning has improved inertial measurement unit (IMU) sensing for mobile and wearable applications. However, an IMU model trained in one domain often becomes unreliable when it is used with a new user, device, or body position. Existing methods usually treat this problem as a static model-design task: they pretrain a stronger representation, add data augmentation, or select one adaptation method before deployment. In practice, the target domain is only gradually observed, labels are scarce, and different domain shifts require different sensing actions. This paper presents SenseAgent, an LLM-guided sensing agent for cross-domain IMU activity recognition. Instead of asking an LLM to classify raw IMU signals, SenseAgent uses the LLM as a runtime planner over sensing tools, source-domain experience memory, online target memory, and verifiers. The agent builds a label-free diagnosis report from the target stream and uses it to decide whether to keep raw inference or invoke specialized tools, including gravity-aware sensing, prototype transfer, and style normalization. Verifiers check source calibration, target-memory reliability, and no-harm criteria before accepting high-risk tool decisions. SenseAgent also supports scarce feedback without retraining the backbone or replacing the label-free route. This design converts cross-domain IMU sensing from a fixed inference pipeline into a closed-loop sensing process that diagnoses target shifts, selects suitable sensing actions, and rejects unsafe adaptations. We evaluate SenseAgent across multiple IMU datasets and deployment shifts. Results show that its verified route selection improves cross-domain sensing, especially under harder placement and compound shifts, and further benefits from limited user feedback.

1 Introduction

Cross-domain IMU models can become unreliable on new users, devices, or body positions, while fixed adaptation pipelines apply the same procedure across target shifts. SenseAgent instead uses an LLM as a runtime planner over sensing evidence, tools, memories, and verifiers.

1 Introduction

  • Models trained in one setting can perform poorly on new users, devices, or positions, while fixed adaptation pipelines apply the same procedure regardless of the observed shift.Existing approaches include self-supervised pretraining, augmentation, few-shot learning, and domain adaptation.Source: §1 · p. 1
  • User style, device rotation, and body placement can call for different actions, and applying the wrong action can cause negative transfer when target labels are scarce.Gravity-aware reasoning may help with orientation or placement shifts, while prototype transfer may suit class-consistent patterns with different amplitudes.Source: §1 · p. 1
  • Deployment adaptation requires inspecting unlabeled target evidence, choosing and verifying a sensing action, and updating target memory as data or scarce feedback arrives.The proposed framing treats adaptation as runtime decision-making over sensing data, diagnostics, and actions.Source: §1 · pp. 1–2
  • SenseAgent uses the LLM as a structured planner, not a raw-signal classifier, reasoning over compact diagnostics and source and target memories to select auditable actions.Verifiers check planned tool calls and memory updates using label-free stability signals and optional user feedback before predictions change.Source: §1 · p. 2
  • The work evaluates SenseAgent across multiple datasets and user, sensing-condition, and compound shifts, studying zero- and scarce-feedback settings, planner decisions, verifier overrides, and stronger planners.Its proposed tools include dynamic-anchored gravity reasoning, prototype experts, style normalization, and feedback-based propagation.Source: §1 · p. 2

2 Background and Motivation

LLM agents can plan, use tools, and revise decisions from feedback, but IMU adaptation needs sensing-grounded evidence. Preliminary experiments show that target difficulty, adaptation benefit, and gravity cues vary across domains and activities, motivating verified runtime decisions.

2.1 Background: LLM Agents

  • LLM agents combine planning, tool use, and feedback or memory, but IMU sensing requires domain-specific observations and verification rather than prompting over raw sensor streams.The related work describes ReAct, Toolformer, and Reflexion as examples of these agent capabilities.Source: §2.1 · p. 2

2.2 Preliminary Experiments

  • Raw PatchTST performance varies across TNDA target domains, and even domains in the same OOD category can be easy or difficult.An ankle-mounted target can be challenging while a back-mounted target under the same split is easier, so OOD category alone does not determine the response.Source: §2.2 · p. 3, Figure 1 · p. 3
  • One-shot prototype adaptation improves some hard targets but degrades some easier ones, so scarce examples should not be applied blindly.The examples include gains on Subject34-LeftAnkle and declines on Subject19-Back and Subject44-RightWrist.Source: §2.2 · p. 3
  • Dynamic windows provide a more concentrated gravity reference than static windows, whose dispersion is weakly negatively associated with raw static-class F1 (r = −0.21).Gravity can diagnose some static placement ambiguity, while dynamic activities require motion-pattern evidence beyond gravity.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Motivation

  • Together, the experiments indicate that target-specific difficulty, unsafe adaptation, and activity-dependent gravity cues make a single preselected adaptation rule inadequate.The system must inspect label-free target evidence, choose an action, verify its safety, and update memory as target data or feedback arrives.Source: §2.3 · p. 3
  • SenseAgent keeps the LLM from classifying windows directly, using structured diagnostics to plan constrained actions executed by local tools and checked by verifiers.The tools include gravity diagnostics, prototype matching, feedback integration, and conservative fusion.Source: §2.3 · p. 3

3 Problem Formulation

The problem is cross-domain IMU activity recognition with labeled source domains and a target stream observed at deployment. Adaptation uses source data, unlabeled target windows, and optional scarce user feedback without relying on target identities or hidden labels.

3 Problem Formulation

  • The setup provides labeled data from one or more source domains, while the target domain is observed only at deployment.The activity label space contains C classes.Source: §3 · p. 4
  • A domain can represent a user, device, body position, sensor placement, or another deployment condition, and each window is a length-L six-axis time series.The six axes comprise three-axis accelerometer and gyroscope readings.Source: §3 · p. 4
  • Target labels are unavailable except for explicit user feedback, and target-domain identities are not used for adaptation or prediction.The formulation includes zero-feedback adaptation (N = 0) and scarce-feedback adaptation (N > 0).Source: §3 · p. 4

4 SenseAgent Design

SenseAgent uses the LLM as a structured planner over sensing evidence, tools, memories, and verifiers rather than as an IMU classifier. Its design moves from source-only assets and online diagnosis through verified route selection, with an optional scarce-feedback path.

4.3 Source Experience Memory

  • Source experience memory provides offline assets, while online target memory caches unlabeled target windows and optional feedback anchors without backbone retraining.The source assets include the frozen backbone, source statistics, prototype banks, and source-calibrated reliability summaries.Source: §4.1 · p. 4, §4.3 · p. 5

4.1 System Overview

  • The system overview pairs offline source-asset construction with online target diagnosis, planner-proposed routes and memory actions, and verifier checks before predictions change.The LLM planner operates on compact structured evidence rather than raw IMU signals or direct activity labels.Source: §4.1 · p. 4

4.2 Unified Sensing Tool Interface

  • Four tools provide complementary evidence: raw inference, gravity-aligned views, raw- or gravity-space prototypes, and channel-statistics style normalization.These tools supply candidate outputs and evidence; they do not directly determine final predictions.Source: §4.2 · pp. 4–5

4.4 Online Target Memory

  • The online target memory maintains seed, stable, and proposed states, and a proposal is retained only when it preserves label-free action value and verifier reliability.The diagnosis report summarizes stable memory with raw-model entropy and top-class margin, alongside label-free shift diagnostics.Source: §4.4 · p. 6

4.5 Agent Planning and Verification

  • The planner chooses a prediction route, memory action, and optional feedback use from diagnosis reports, while deterministic local modules compute logits, statistics, and checks.The LLM is a constrained sensing-specific arbitration layer, not a free-form tool caller.Source: §4.5 · p. 6

4.6 Verified Prediction Routes

  • Verified route selection makes sensing actions executable only after the verifier checks no-harm conditions, rather than using fixed dataset-level adaptation rules.The planner proposes a route from evidence in verified memory; only an approved route can affect predictions.Source: §4.6 · p. 7

4.7 Scarce-feedback Adaptation

  • With scarce feedback, labeled anchors guide memory updates and optional logit-level residual corrections, while the verified label-free route remains in place.A feedback candidate is used only if it improves anchor utility without harming unlabeled stability.Source: §4.7 · p. 7

5 Experimental Settings

Experiments use four public IMU activity-recognition datasets, with domains defined by user identity and sensing condition. The setup evaluates user, sensing-condition, and compound deployment shifts.

5 Experimental Settings

  • Each IMU window contains 120 time steps and six accelerometer and gyroscope channels, and macro-F1 is used because class distributions can be imbalanced across domains.PatchTST is the raw backbone and Qwen3.5-9B the planner; the framework is described as backbone-flexible and LLM-agnostic.Source: §5 · p. 7

5.1 Datasets

  • The evaluation defines User OOD, Position/Device OOD, and Compound OOD shifts across four public datasets, testing reserved users, reserved sensing conditions, or both.Each target domain is evaluated independently.Source: §5.1 · p. 8
  • The four datasets cover varied activities, users, devices, and placements, with source and reserved target domains specified for HHAR, TNDA, PAMAP2, and Feng.Reserved conditions include LG devices in HHAR and unseen body positions in TNDA, PAMAP2, and Feng.Source: §5.1 · p. 8

6 Experimental Evaluation

Across target domains, SenseAgent uses selective, verified adaptation rather than one fixed route, with its strongest zero-feedback gains on placement and compound shifts. Limited feedback further improves performance when integrated conservatively, while route, planner, component, and deployment analyses examine how the system achieves these results.

6.1 Zero Feedback Performance

  • On PAMAP2, SenseAgent improves over PatchTST from 0.1531 to 0.2972 on position OOD and from 0.1910 to 0.4089 on compound OOD; on TNDA, it improves from 0.4473 to 0.5263 and from 0.4392 to 0.5518, respectively.The reported largest gains occur under severe placement and compound shifts.Source: §6.1 · p. 8

6.2 Scarce User Feedback Adaptation

  • SenseAgent achieves the best average macro-F1 under random-anchor feedback; with N=8 it leads on user, position/device, and compound OOD, and with N=1 it is competitive with or better than the stronger PTN 1-shot reference on average position/device and compound shifts.PTN 1-shot uses one class-balanced example per class, unlike SenseAgent’s randomly sampled anchors.Source: §6.2 · p. 9, Table 3 · p. 9
  • Increasing the feedback budget generally improves SenseAgent’s macro-F1, with more pronounced gains on PAMAP2 and TNDA and substantial HHAR gains on compound and device shifts.Feng improves more gradually, consistent with an already relatively reliable initial route.Source: §6.2 · p. 9

6.3 Runtime Decision Analysis

  • Route selection varies by dataset: Gravity is retained in 78.6% of PAMAP2 runs and 91.2% of TNDA runs, but rarely on Feng and never on HHAR; forcing Gravity is harmful in 98.1% of Feng runs, while the verifier overrides unsupported gravity actions in 93.7%.The analysis attributes Feng’s broader source coverage of body positions to raw-backbone exposure during training.Source: §6.3 · p. 10

6.4 LLM Planner Analysis

  • Compared with deterministic Prompt-Rule routing, SenseAgent obtains much larger gains on harder PAMAP2 and TNDA shifts and narrows the gap to the oracle selector.Replacing Qwen3.5-9B with GPT-5.4 changes final macro-F1 only marginally but consistently reduces verifier overrides, especially on HHAR.Source: §6.4 · pp. 10–11, Table 5 · p. 10, Table 6 · p. 10

6.5 Ablation Study

  • Fixed Gravity improves PAMAP2 and TNDA to 0.425 and 0.598 macro-F1 under N=0 but degrades Feng and HHAR to 0.576 and 0.702; without verification, Feng falls from 0.772 to 0.576 under N=0 and from 0.770 to 0.634 under N=1.Gravity components matter most on PAMAP2 and TNDA, while the gravity verifier has the largest effect on Feng and HHAR; target memory mainly helps PAMAP2 and TNDA.Source: §6.5 · p. 11, Table 7 · p. 11
  • The ablations indicate that gains depend on coordinating route selection, target memory, gravity evidence, and verification rather than on one universally best component.Source reliability has smaller and mixed effects, serving as a calibration signal rather than a standalone performance driver.Source: §6.5 · p. 11

6.6 Deployment Overhead

  • Window-level inference takes 7.82 ms on the raw route and 14.91 ms on the gravity route, while report-level overhead falls from 10.52 s for a cold planner call to 1.76 s with cached outputs.The evaluation excludes offline source construction; gravity is slower because it constructs gravity-view channels and runs the gravity-view encoder.Source: §6.6 · p. 11, Table 8 · p. 11

7 Related Work

Related work addresses deployment heterogeneity through robustness and cross-domain representation learning, while newer semantic and LLM-based sensing systems broaden sensor interpretation. SenseAgent is distinguished as a runtime planner for deployment-time IMU adaptation.

7.1 Deployment Heterogeneity in IMU Sensing

  • Prior IMU sensing work addresses deployment heterogeneity through shared representations, physics-informed augmentation, and methods for device, body-location, and sampling-rate variation.Examples include SenseHAR and UniHAR.Source: §7.1 · p. 12

7.2 Cross-domain IMU Representation Learning

  • Cross-domain representation learning uses unlabeled or synchronized sensor data for pretraining and evaluates robustness across source and target conditions.LIMU-BERT, ColloSSL [19], and CrossHAR illustrate these approaches.Source: §7.2 · p. 12

7.3 Semantic and LLM-based Sensing

  • Semantic and LLM-based sensing includes multimodal alignment, language-guided HAR, and sensor reasoning, while SenseAgent uses an LLM to plan runtime adaptation over structured target evidence.Table 9 is identified as a comparison with LLM-based sensing systems.Source: §7.3 · p. 12, Table 9 · p. 12

8 Discussion and Future Work

The discussion finds that LLM-guided routing helps on harder placement-heavy shifts, but the planner is limited to predefined tools and actions. It proposes learning planner and verifier decisions from logged deployment evidence and outcomes.

8.1 Leveraging Stronger LLMs

  • LLM-guided routing outperforms a deterministic prompt-rule router on harder placement-heavy shifts, while replacing Qwen3.5-9B with GPT-5.4 yields only marginal macro-F1 gains.The stronger planner reduces verifier overrides, but the fixed report-to-action interface constrains its action space.Source: §8.1 · p. 12
  • The planner can select and configure predefined sensing routes and tools, but cannot expand the sensing action space.The proposed direction is stronger sensing harnesses that let planners compose primitives or generate executable adaptation code while retaining structured reports, logs, and verifier gates.Source: §8.1 · p. 12

8.2 Learning the Planner and Verifier

  • Planner actions and verifier rules rely on sensing priors and calibrated source episodes rather than direct optimization from deployment outcomes.The discussion proposes learning from logged reports, actions, verifier decisions, sparse feedback, and delayed performance, including reinforcement learning or contextual bandits for the planner.Source: §8.2 · p. 12

9 Conclusion

SenseAgent uses an LLM for structured planning over sensing tools while keeping raw sensing computation local and using a verifier to guard against harmful adaptation. Experiments across diverse deployment shifts show improved cross-domain activity recognition with auditable decisions and practical deployment overhead.

9 Conclusion

  • SenseAgent keeps raw sensing computation local and uses an LLM to plan over sensing tools, with a verifier guarding against harmful adaptation.Its closed-loop design integrates source experience memory, target memory, gravity reasoning, prototype and style tools, and scarce-feedback adaptation.Source: §9 · pp. 12–13
  • Experiments across diverse user, position, device, and compound shifts show improved cross-domain IMU activity recognition.The conclusion also reports auditable decisions and practical deployment overhead.Source: §9 · p. 13
  • The paper frames verified LLM-guided planning as a potential foundation for adaptive sensing under real-world distribution shifts.SenseAgent replaces a fixed adaptation pipeline with closed-loop sensing decisions.Source: §9 · p. 13

A Planner Prompt Template

The prompt template frames the LLM as a structured planner rather than an IMU classifier. Its route is represented through tool actions and a gravity variant, while verifier rules govern feedback use.

A Planner Prompt Template

B Verifier Threshold Sensitivity

The route verifier is not highly sensitive to a single threshold setting across the middle range, but overly permissive static support reduces no-harm performance.

B Verifier Threshold Sensitivity

  • Zero-feedback planner proposals are replayed over saved diagnosis reports and materialized tool outputs to evaluate route-verifier sensitivity.The sweep varies the static-support threshold τ_s and raw-stability threshold τ_r.Source: B Verifier Threshold Sensitivity · p. 14
  • Overly permissive static support (τ_s = 0.35) accepts more unsupported gravity actions and reduces the no-harm rate.Macro-F1 remains stable across the middle threshold range.Source: B Verifier Threshold Sensitivity · p. 14
  • No-Harm measures the fraction of target domains where the replayed route is no worse than raw PatchTST.Source: Table 11 · p. 14
  • Similar macro-F1 and no-harm behavior across the middle threshold range suggests low sensitivity to a single verifier hyperparameter setting.Source: B Verifier Threshold Sensitivity · p. 14

Related papers

Loading 2609.32000v1…