PaperlayerView exact revision on arXiv

Source-linked AI summary

AegisAgent: An Autonomous Defense Agent Against Prompt Injection Attacks in LLM-HARs

Yihan Wang, Huanqi Yang, Shantanu Pal, Weitao Xu

arXiv:2512.20986v1cs.CR

TL;DR

LLM-HAR systems face prompt-injection threats that existing sensor-focused and text-only defenses do not adequately address. AegisAgent combines layered detection, memory-guided planning, and defensive execution; its full system shows the strongest and most stable robustness among the evaluated variants, while the paper’s threat model assumes prompt-manipulation capabilities.

  • Problem

    Prompt injection can manipulate LLM-HAR inputs and propagate across modalities, while existing sensor-focused and text-only defenses do not sufficiently address these threats.

    Source: §1 · pp. 1–2

  • Method

    AegisAgent combines layered perception and reasoning with a planning agent, memory hub, and execution agent to detect, correct, and recover from attacks.

    Source: §4.1 · p. 6

  • Results

    Across five target models, the full AegisAgent shows the strongest and most stable detection accuracy; removing its robust reasoner drops DA to 7–8%.

    Source: §5.3 · pp. 10–11

  • Takeaways & Limitations

    The paper concludes that automated defense is crucial to securing LLM-HAR systems against prompt injection and erroneous predictions in real-world scenarios.

    Source: §7 · p. 12

    The threat model assumes attackers can manipulate prompts and considers white-box attacks.

    Source: §3.1.2 · p. 4

Abstract

from arXiv · show

The integration of Large Language Models (LLMs) into wearable sensing is creating a new class of mobile applications capable of nuanced human activity understanding. However, the reliability of these systems is critically undermined by their vulnerability to prompt injection attacks, where attackers deliberately input deceptive instructions into LLMs. Traditional defenses, based on static filters and rigid rules, are insufficient to address the semantic complexity of these new attacks. We argue that a paradigm shift is needed -- from passive filtering to active protection and autonomous reasoning. We introduce AegisAgent, an autonomous agent system designed to ensure the security of LLM-driven HAR systems. Instead of merely blocking threats, AegisAgent functions as a cognitive guardian. It autonomously perceives potential semantic inconsistencies, reasons about the user's true intent by consulting a dynamic memory of past interactions, and acts by generating and executing a multi-step verification and repair plan. We implement AegisAgent as a lightweight, full-stack prototype and conduct a systematic evaluation on 15 common attacks with five state-of-the-art LLM-based HAR systems on three public datasets. Results show it reduces attack success rate by 30\% on average while incurring only 78.6 ms of latency overhead on a GPU workstation. Our work makes the first step towards building secure and trustworthy LLM-driven HAR systems.

1 INTRODUCTION

LLM-based HAR expands activity recognition into semantically rich, multimodal systems, but also introduces prompt-injection risks that existing sensor-focused and text-only defenses do not address. The paper presents AegisAgent and evaluates it across attacks, HAR systems, and datasets.

1 INTRODUCTION

  • LLM-HAR systems use LLMs to turn inertial measurements into semantically rich activity understanding, extending beyond traditional sensor-based classification.Examples include generating virtual IMU data from text, translating between text and motion, and recognizing activities from raw IMU data.Source: §1 · p. 1

1.1 Challenges and Contributions

  • Prompt injection can manipulate sensor-to-prompt descriptions and propagate across signal, text, and prompt layers, enabling compound attacks and hazardous misclassifications.Attackers may combine language manipulation, inference disruption, and sensor-level inconsistencies in practical settings.Source: §1 · p. 1, §1.1 · p. 2
  • AegisAgent is proposed as an automated defense that detects and corrects prompt attacks through input sanitization, cross-modal consistency verification, and robust reasoning.The framework is described as model-agnostic and training-free, with operation across diverse LLMs, datasets, and agent configurations.Source: §1.1 · p. 2
  • Standard text defenses only partially recover HAR accuracy after prompt injection, leaving performance below the original levels.Figure 2 compares accuracy before attacks, after attacks, and after text defenses.Source: Figure 2 · p. 3

2 PRELIMINARIES

The preliminaries describe how LLM-HAR maps inertial measurements into language-mediated activity understanding and use an IMUGPT-2.0 case study to examine prompt injection. The study finds substantial accuracy degradation and incomplete recovery from text-only defenses.

2.1 Primer on IMU-based HAR

  • LLM-HAR pipelines convert IMU representations into motion-aware prompts that support natural-language descriptions or activity-category predictions.This staged workflow connects low-level inertial measurements with high-level semantic concepts.Source: §2.1 · p. 3

2.2 Case Study

  • Combined prompt attacks reduce classification accuracy by over 30% across USC-HAD, UCI HAR, and PAMAP2 in the IMUGPT pipeline.The case study implements fifteen attacks spanning signal, text, and prompt path vulnerabilities.Source: §2.2 · p. 3
  • Prompt changes can destabilize generated joint velocities and produce anomalous motion trajectories that poison synthetic training data and lower classification accuracy.Examples include near-synonym substitutions, typos, and multi-label prompts that create conflicting trajectories.Source: §2.2 · pp. 3–4

2.3 Insufficiency of existing defenses

  • Text defenses recover only 5% to 10% of accuracy after attacks, remaining below original levels in the multimodal LLM-HAR pipeline.The reported preliminary case study uses IMUGPT-2.0; similar results in other LLM-HARs are noted but not included.Source: §2.3 · p. 4

3 ATTACK MODEL

The threat model treats LLM-HAR attacks as manipulations of signals, intermediate text, or prompts that redirect the system from its intended task. The paper formalizes these attack paths through a unified operator framework that also supports hybrid attacks.

3.1 Threat Model

  • The threat model spans attempts to disrupt IMU interpretation, redirect task objectives, and distort semantic cues guiding model reasoning.Attacks may target activity labels, reasoning trajectories, or multimodal descriptions.Source: §3.1.1 · p. 4

3.1.1 Attacker’s Goal.

  • Attackers seek to make activity labels, reasoning, or descriptions align with injected objectives rather than the intended task, under an assumed white-box setting.The model assumes attackers can alter signal interpretations, text, and prompt context.Source: §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.2 Attacker’s Capabilities.

  • The attacker is assumed to modify or append prompt-channel text, manipulate signal, text, or prompt layers, and know model internals in white-box attacks.The stated assumption includes access to architecture, parameters, and weights.Source: §3.1.2 · p. 4

3.1.3 Attack Methods.

  • Attacks target raw IMU or surrogate signals, intermediate descriptions, or full prompts, spanning signal, text, and instruction layers.Examples include noise, drift, temporal splicing, linguistic edits, task injection, role confusion, and instruction overwriting.Source: Figure 4 · p. 4, §3.1.3 · p. 5

3.2 Attack Formalization

  • The formalization represents benign prompts as instruction, generated motion description, and optional context, then models compromised inputs as producing an injected-task output.The attacker uses an operator to construct compromised signal, text, or prompt data; the framework presents a unified structure for evaluating and combining operators.Source: §3.2 · pp. 5–6
  • The framework catalogs representative operators by attack path and hybrid form, and supports composing signal, text, and prompt operations into new attacks.Its example hybrid combines multiple manipulations across all three layers.Source: §3.2 · pp. 5–6

4 AEGISAGENT DESIGN

AegisAgent combines sanitization, cross-modal consistency checks, and robust reasoning in an automated defense pipeline. Its workflow coordinates a planner, executor, and shared memory hub to detect, remediate, and verify potential attacks.

4.1 System Overview

  • AegisAgent’s three defense layers sanitize inputs, verify consistency across IMU modes, descriptions, and user intent, and use a coordinated control plane for robust reasoning.The planner, memory hub, and execution agent coordinate replanning and remediation when a layer raises an alert.Source: §4.1 · p. 6
  • The design is described as modular, verified before output, event-driven, training-free, and model-agnostic.Its sub-agents schedule tasks in response to anomalies or feedback, while verification agents check reasoning-step integrity.Source: §4.1 · pp. 6–7

4.2 AegisAgent Workflow

  • The workflow transforms raw IMU and potentially tampered prompts into sanitized inputs and a secure prediction through six coupled components.Input sanitization prepares an interaction tuple and delegates normalization and lexical filtering to the Memory Hub.Source: §4.2 · p. 7
  • Robust reasoning is triggered by semantic or temporal threshold violations and can combine structured reasoning, parallel inference paths, voting, and veto-based backtracking.Execution also upgrades to robust inference when intermediate-state similarity to benign prototypes falls below 0.85.Source: §4.2 · pp. 7–9
  • The Memory Hub stores purified defense operators and annotated trajectories, retrieves successful cases, and provides validation metrics and memories to the planner.The planner selects defense operators and execution order; the executor translates the symbolic plan into Hub and LLM calls.Source: §4.2 · pp. 8–9
  • Consistency verification checks semantic and temporal agreement, flags threshold violations, and supplies metrics and warnings to upstream planning.When only a corrupted prompt is available, the system can reconstruct an approximate benign prompt from memory prototypes to refine context.Source: §4.2 · pp. 7–8

5 EVALUATION

The evaluation tests AegisAgent’s defense effectiveness, component contributions, attack severity, baseline performance, and latency across five LLM-HAR models and three datasets. Results show stronger detection and recovery than the compared defenses, reduced attack success, and 78.6 ms average overhead.

5.1 Goals, Metrics, and Methodologies

  • Evaluation spans five LLM-HAR models, three datasets, and four downstream LLM classifiers, with DA, ASR, RR, SC, and HS as metrics.Experiments use target-model default configurations and are repeated five times; AegisAgent is training-free.Source: §5.1 · pp. 9–10

5.2 Defense Effectiveness (RQ1)

  • Across five models, AegisAgent averages 85% DA, reduces ASR by 30%, and attains 75% SC.The reported improvements persist across target models, datasets, and downstream LLM-based HAR classifiers.Source: Table 1 · p. 10, §5.2 · p. 10

5.3 Ablation Study (RQ2)

  • Without the sanitizer DA falls to 53–57%, without the verifier to 46–49%, and without the reasoner to 7–8%; the full system performs best.The study attributes the declines respectively to noisy inputs, cross-modal misalignment, and loss of task-intent reconstruction.Source: Figure 6 · p. 11, §5.3 · p. 11

5.4 Attack Severity (RQ3)

  • Hybrid attacks are most severe, with HS 3.8–4.1, compared with signal-path attacks at 3.5–3.6, prompt-path attacks at 2.9–3.1, and text-path attacks at 2.6–2.8.Source: §5.4 · p. 11

5.5 Comparison with baselines (RQ4)

  • 93.0% DA, 56.3% RR, and 27.2% ASR make AegisAgent stronger than the three defense baselines across all reported metrics.Text-only, classical HAR, and multimodal detection-only defenses report 44.1%, 58.7%, and 64.9% DA, respectively.Source: Figure 7 · p. 11, §5.5 · p. 12

5.6 Latency (RQ5)

  • 78.6 ms is AegisAgent’s average additional latency per query on an NVIDIA RTX 3090 GPU workstation.The paper characterizes this runtime as real-time responsiveness when deployed in a server.Source: Table 3 · p. 12, §5.6 · p. 12

6 RELATED WORK

Prior work explores LLM-based interpretation of IMU signals and prompt attacks, but leaves cross-modal prompt manipulation in LLM-HAR pipelines insufficiently addressed. AegisAgent targets this vulnerability with an automated end-to-end defense.

LLM-based HAR

  • LLM-HAR work explores activity descriptions, multimodal representations, and zero-shot generalization, generally assuming a benign sensor-to-text pipeline.Examples include IMUGPT-2.0, MotionGPT, LLaSA, HAR-GPT, and ContextGPT.Source: §6 · p. 12

Prompt attacks

  • Prior prompt-attack studies examine jailbreaks and reasoning disruptions but overlook attacks originating jointly from user commands and sensor-generated descriptions.The cited attack studies include jailbreak suffixes [22], adversarial prompts, semantic drift [15], and multimodal jailbreaks.Source: §6 · p. 12

Cross-modal defense

  • AegisAgent addresses cross-modal prompt manipulation affecting user instructions and IMU descriptions as an end-to-end LLM-HAR defense.The paper describes it as the first such defense mechanism designed specifically for multimodal LLM-HAR systems.Source: §6 · p. 12

7 CONCLUSION

The conclusion presents AegisAgent as an automated defense for LLM-HAR prompt injection that detects and repairs adversarial prompts without human intervention.

Conclusion

  • AegisAgent is an automated agent defense that detects and repairs adversarial prompts without human intervention in LLM-HAR systems.The conclusion says it demonstrates robustness against a wide range of adversarial perturbations.Source: §7 · p. 12

A. Attack Catalog

The appendix catalogs attack operators under a unified formula, organizing them by operation path and including representative hybrid perturbations.

A. Attack Catalog

  • The catalog groups operators into signal, text, and prompt paths and lists representative hybrid perturbations with definitions or examples.It complements the attack methodology in Section 3.2.Source: A. Attack Catalog · p. 15

B. End-to-End Pipelines: From Signals to Text (or Tokens) and to Prompt Construction

The appendix traces how five target LLM-HAR systems convert signal representations into text or tokens and embed them in prompts.

B.1 LLaSA: Signal →Text →Prompt

  • LLaSA turns short statistical descriptors from IMU windows into text strings for a JSON-style prompt template.Its example prompt asks for an activity label from a fixed set of classes.Source: §B.1 · p. 15

B.2 IMUGPT-2.0: Signal →Text →Prompt

  • IMUGPT-2.0 represents IMU statistical signals as compact feature strings and places them in an activity-classification prompt.The example input includes acceleration variances and step frequency.Source: §B.2 · p. 15

B.3 HARGPT: Signal →Text →Prompt

  • HARGPT prints the complete IMU sequence as text, embedding accelerometer and gyroscope arrays in a step-by-step reasoning prompt.The example prompt also identifies the device and collection location and lists activity classes.Source: §B.3 · p. 15

B.4 ContextGPT: Signal →Text →Prompt

  • ContextGPT combines contextual cues such as room surroundings with IMU summaries in a prompt asking what the user is doing.The example pairs a bedroom setting with step frequency and Z-axis acceleration information.Source: §B.4 · p. 15

B.5 MotionGPT: Signal →Motion Tokens → Prompt

  • MotionGPT converts 3D motion into discrete tokens with a VQ-VAE tokenizer and integrates those tokens with natural-language instructions.The examples include text-to-motion, motion-to-text, and motion question-answering prompts.Source: §B.5 · pp. 15–16
  • MotionGPT's prompt examples support both generating motion from text and describing or querying motion tokens.The examples include a waltz-generation request, a motion-caption request, and prompts to describe or extend motion.Source: §B.5 · p. 16

Related papers

Loading 2512.20986v1…