PaperlayerView exact revision on arXiv

Source-linked AI summary

CHAI: Command Hijacking against embodied AI

Luis Burbano, Diego Ortiz, Qi Sun, Siwei Yang, Haoqin Tu, Cihang Xie, Yinzhi Cao, Alvaro A Cardenas

arXiv:2510.00181v2cs.CRcs.AIcs.LG

TL;DR

Embodied AI’s ability to handle unforeseen situations through LVLM reasoning leaves intermediate text-based planning decisions underexplored as a security target. CHAI embeds optimized natural-language commands in visual scenes to hijack those decisions, and the authors report strong results across simulated tasks and a real-world robotic vehicle. The findings identify an attack surface in multimodal embodied systems and motivate defenses suited to it.

  • Problem

    The security of LVLM-driven embodied agents’ intermediate text-based planning decisions remains largely unexplored, despite prior attacks targeting perception or external text inputs.

    Source: §I · p. 1

  • Method

    CHAI jointly optimizes the semantic content and visual appearance of natural-language signs to induce malicious intermediate commands from embodied LVLM agents.

    Source: §I · p. 1, A. Optimization Problem · p. 4

  • Results

    CHAI outperforms state-of-the-art attacks across evaluated tasks, reaching up to 95.5% ASR in simulation and at least 87% ASR in real-world vehicle experiments.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    CHAI exposes a new attack vector against LVLM-driven embodied AI and motivates principled multimodal defenses.

    Source: D. Future Work · p. 13, §XI · p. 13

    Authentication-based defenses assume a closed set of trusted entities, limiting their fit for open settings such as public streets or disaster zones.

    Source: §XI · p. 13

Abstract

from arXiv · show

Embodied Artificial Intelligence (AI) promises to handle edge cases in robotic vehicle systems where data is scarce by using common-sense reasoning grounded in perception and action to generalize beyond training distributions and adapt to novel real-world situations. These capabilities, however, also create new security risks. In this paper, we introduce CHAI (Command Hijacking against embodied AI), a physical environment indirect prompt injection attack that exploits the multimodal language interpretation abilities of AI models. CHAI embeds deceptive natural language instructions, such as misleading signs, in visual input, systematically searches the token space, builds a dictionary of prompts, and guides an attacker model to generate Visual Attack Prompts. We evaluate CHAI on four LVLM agents: drone emergency landing, autonomous driving, aerial object tracking, and on a real robotic vehicle. Our experiments show that CHAI consistently outperforms state-of-the-art attacks. By exploiting the semantic and multimodal reasoning strengths of next-generation embodied AI systems, CHAI underscores the urgent need for defenses that extend beyond traditional adversarial robustness.

I. INTRODUCTION

LVLMs may help embodied agents handle unforeseen situations, but their intermediate text-based decisions create an underexplored security target. CHAI attacks that command layer with optimized visual instructions, outperforming prior attacks across simulated and real-world evaluations.

I. INTRODUCTION

  • LVLMs offer common-sense reasoning and adaptation to rare scenarios, while the security of their intermediate text-based planning decisions remains largely unexplored.Prior attacks target perception or external text inputs rather than commands bridging perception and actuation.Source: §I · p. 1

II. RELATED WORK

  • Prior typographic attacks, including SceneTAP, differ from CHAI in their optimization, image-specific scope, limited physical-world focus, and lack of CPS targeting.Earlier attacks often use one-shot generation or create a unique attack for each image.Source: §I · p. 1, §II · p. 3

I. INTRODUCTION

  • CHAI jointly optimizes the semantic content and visual realization of human-readable signs to induce malicious intermediate outputs from LVLM-driven agents.The attack operates across language and vision channels to target high-level physical decisions.Source: §I · p. 1
  • CHAI reaches up to 95.5% ASR on CloudTrack, 81.8% on DriveLM, and 72.8% on drone landing in simulation, with at least 87% ASR in real-world vehicle experiments.Compared with SceneTAP, it is up to 10× more effective in some use cases and generalizes to new scenes while maintaining the same success rates.Source: §I · pp. 1–2
  • CHAI also generalizes across English, Chinese, Spanish, and Spanglish, handles adverse weather, and can exploit task-specific prompts.The authors present these findings as evidence of a practical cross-modal jailbreak and motivate multimodal defenses.Source: §I · p. 2

III. MOTIVATION

Visual signs can inject commands into an LVLM-driven agent, but the sign’s text alone does not determine whether an attack succeeds. The paper frames the problem as finding practical, reusable combinations of textual and visual cues under a physical attacker model.

III. MOTIVATION

  • DriveLM stops correctly for signs reading “Go Onward” or “Proceed,” but changing the sign’s colors while retaining “Proceed” makes the attack succeed.This motivates searching for effective combinations of text, colors, and locations rather than relying on the correct words alone.Source: §III · p. 3

A. Challenges

  • A universal attack must work across multiple images when the attacker knows the general scene but not the exact image captured by the LVLM.The paper contrasts this with SceneTAP, which generates a unique attack per image and may fail when the scene or background changes.Source: A. Challenges · p. 3
  • CHAI must find effective words and jointly optimize visual attributes such as color, font, size, and placement for a single attack across several images.A one-shot generated phrase may fail, and poor visual choices may make the text unreadable or ineffective.Source: A. Challenges · pp. 3–4

IV. THREAT MODEL

  • The threat model allows an external attacker to display a physical prompt without contacting the vehicle or compromising its onboard systems.The attacker is black-box, may query the LVLM, and knows the task specification and system prompt but not internal model or control details.Source: §IV · p. 4

V. PROBLEM FORMULATION

  • The formulation represents the target LVLM as f, which takes a text prompt and one or more images and returns text output.The model combines image features from a vision encoder with prompt features from a tokenizer in a language-model backbone.Source: §V · p. 4

A. Optimization Problem

CHAI attacks the command layer by jointly optimizing the semantic content and perceptual appearance of visual prompts. Its black-box pipeline first reduces the vocabulary to a prompt dictionary, then jointly selects a dictionary prompt and perceptual features.

A. Optimization Problem

  • The attack maximizes the chance that the LVLM outputs the attacker’s target label when given an attacked image.The notation defines the target label, unmodified output label, image, and attacked output used in the optimization.Source: A. Optimization Problem · p. 5

B. Attack Pipeline

  • The pipeline first creates a prompt dictionary, then jointly optimizes its prompt choice and the attack’s perceptual features.The two-stage design makes the second-stage search operate over dictionary prompts and perceptual features.Source: Figure 3 · p. 5, B. Attack Pipeline · p. 5

A. Optimization Problem

  • CHAI jointly optimizes a visual prompt’s text content and perceptual realization, treating them as discrete text tokens and high-dimensional image-patch perturbations.The text content is the visual prompt; perceptual features include position, rotation, color, and font.Source: A. Optimization Problem · pp. 4–5

B. Attack Pipeline

  • The search space is combinatorially large and mixes categorical prompts with variables that lack an inherent order or well-defined distance.Even choosing one English word can involve hundreds of thousands of candidate visual prompts.Source: B. Attack Pipeline · p. 5
  • The method uses black-box input-output access, while its modular design can incorporate gradient-based refinements if white-box access becomes available.The authors focus on black-box LVLMs because many state-of-the-art models are accessible only through restricted APIs.Source: B. Attack Pipeline · pp. 5–6

VI. DICTIONARY CREATION

CHAI builds a dictionary of visual prompts by guiding an attacker LLM to propose and refine commands against a target LVLM. It combines prompt–image evaluation with a dictionary-based curriculum before joint optimization.

VI. DICTIONARY CREATION

  • The attacker LLM learns from target-LVLM refusals as the pipeline tests candidate prompts with initially naive visual characteristics and refines the attacker’s prompt using evaluation history.The refinement feedback encourages successful phrases and helps the LLM generate different prompts rather than repeating a term.Source: §VI · p. 6
  • The attacker LLM proposes short imperative commands, using task context and the target LVLM prompt to search the effectively unbounded space of visual cues.The attacker’s objective, capabilities, and constraints also inform its prompt-generation instructions.Source: §VI · p. 6
  • The resulting K-long dictionary may succeed on some training images, and the authors state that optimization can improve attack effectiveness.The prompt-generation pipeline is inspired by PAIR but adds cross-modal context, visual-prompt synthesis and evaluation, and a dictionary-based curriculum.Source: §VI · p. 6
  • Candidate phrases are rendered as visual prompts and evaluated with a joint prompt–image oracle, then successful prompts accumulate in a K-element dictionary.The dictionary is refined through query-efficient black-box optimization in the next stage.Source: §VI · p. 6

VII. JOINT SEMANTIC AND PERCEPTUAL OPTIMIZATION

Because the LVLM objective is black-box, CHAI uses stochastic optimization, adopting the cross-entropy method to search semantic and perceptual variables together. The method updates a parameterized search distribution using high-performing samples.

VII. JOINT SEMANTIC AND PERCEPTUAL OPTIMIZATION

  • The authors use stochastic optimizers because their black-box objective provides no information such as a Lipschitz constant, unlike deterministic methods that typically require structural assumptions.Stochastic methods do not provide the same rigorous optimality guarantees described for deterministic global optimizers.Source: §VII · p. 7
  • The cross-entropy method is chosen over Bayesian Optimization because BO’s scalability degrades as the number of variables increases.CE is population-based, query-efficient, modality-agnostic, and parallelizable; its sampling unifies dictionary entries and RGB patches.Source: §VII · p. 7
  • CE approximates the unknown optimal distribution Ω with a parameterized distribution pα by minimizing KL(Ω, pα).The ideal fit has KL(Ω, pα) = 0.Source: §VII · p. 7
  • Each iteration samples candidates, evaluates their objective values, and uses the top-performing samples to update α until a stopping condition is met.For a piecewise uniform distribution, α’s entries represent probabilities that the maximum lies in the corresponding regions.Source: §VII · p. 7

VIII. EXPERIMENTAL SETUP

  • The following experimental-setup section presents the attack applications, evaluation metrics, image collection, and implementation details.Source: §VIII · p. 7

A. Applications

The section instantiates CHAI across drone landing, autonomous driving, and aerial tracking, then describes its evaluation metric and held-out image sets.

A. Applications

  • Fig. 5's landing case uses a sign on the crowded roof to steer the drone toward it instead of the empty roof.The attack aims to make the LVLM planner select the unsafe landing site.Source: A. Applications · p. 7, Figure 5 · p. 8
  • DriveLM's crosswalk case changes benign braking into acceleration under a malicious sign, endangering pedestrians.DriveLM uses an LVLM to produce vehicle actions from camera images and questions.Source: A. Applications · p. 8
  • CloudTrack can be diverted from the patrol car to a civilian decoy bearing an image overlay.The system then locks onto and follows the wrong vehicle.Source: A. Applications · p. 8

B. Evaluation Metrics

  • Attack success rate (ASR) measures how often the LVLM output changes to the attacker's target label.The evaluation also measures ASR without an attack to account for stochastic model outputs.Source: B. Evaluation Metrics · p. 8

C. Datasets

  • Known Images are used for attack optimization, while held-out Transferability Images test generalization to images unseen by the optimizer.The paired sets contain diverse conditions.Source: C. Datasets · p. 9

D. Implementation Details

CHAI is evaluated against SceneTAP on known images, on held-out images, and with printed prompts on a real robotic vehicle. It consistently outperforms SceneTAP on known-image tests and transfers to unseen scenes, while printed prompts raise real-world attack success rates.

D. Implementation Details

  • The implementation tests GPT-4o and InternVL2.5 8B, uses SceneTAP as baseline, and searches Scenic attack spaces with VerifAI's modified cross-entropy optimizer.The robotic-vehicle implementation is based on the BARC project.Source: D. Implementation Details · p. 9

A. Attack Success on Known Images

  • Over 54% ASR across applications and LVLMs, CHAI beats SceneTAP throughout known-image tests; CloudTrack exceeds 92% for both tested models.The authors report up to an order-of-magnitude improvement over SceneTAP for GPT in landing.Source: A. Attack Success on Known Images · p. 9

B. Attack Success on Transferability

  • At least 50% average ASR across applications and both LVLMs transfers to held-out images; GPT exceeds 70% across applications, while InternVL remains above 50%.For CloudTrack, GPT remains close to 95% in both image sets, while InternVL drops from 92.50% to 66.50%.Source: B. Attack Success on Transferability · pp. 9–10, Table V · p. 9

C. Real World Deployment

  • GPT ASR rose from 4.28% without attack to more than 80% in both printed-sign scenarios; InternVL rose from 17.40% to 54% with a vehicle or 42% with an external sign.The experiments include printing artifacts, lighting variation, and sensor noise.Source: C. Real World Deployment · p. 10, Table VI · p. 11
  • With GPT on landing, ASR rises from 25.50% without perceptual optimization to 72.75% with it when using the selected word.The ablation reports that semantic selection has a significant impact, and perceptual optimization further improves performance.Source: A. Ablation Study · p. 11, Table VII · p. 11

B. Universal vs. Individual Attacks

Scene-specific prompts improve in-sample landing success but lose ground on held-out scenes. Further experiments show visual prompts can override driving decisions, work across languages, misdirect object tracking, and divert a drone to an unsafe landing site.

B. Universal vs. Individual Attacks

  • 84.35% in-sample ASR for scene-specific landing prompts exceeds the universal prompt's 72.75%, but held-out ASR falls to 48.44% versus 71% for the universal attack.The comparison uses GPT-4o and Emergency Landing frames.Source: B. Universal vs. Individual Attacks · p. 11

C. Insights from Experiments

  • At a crosswalk, the visual attack changes DriveLM's decision from braking to an unsafe left turn despite the model still identifying pedestrians and traffic signals.The section also reports successful attacks using Chinese and Spanish signs and the Spanglish prompt “Turn izquierda.”Source: C. Insights from Experiments · pp. 11–12, Figure 11 · p. 12
  • CloudTrack shifts its identification from an unmarked police cruiser to a civilian car when the overlay “POLICE SANTA CRUZ” is placed on the decoy.The attacker exploits the cue “SCPD patrol car.”Source: C. Insights from Experiments · p. 12, Figure 12 · p. 12
  • On held-out landing trajectories, a sign reading “Safe to land” diverted the drone to the hazardous roof in 92% of runs, versus 100% benign landings on the safer roof.The trajectory varied altitude, angle, and lighting.Source: C. Insights from Experiments · p. 12

D. Future Work

The paper proposes multimodal defenses for CHAI and identifies limits of simple defenses in open environments. It also notes that experiments use out-of-the-box models and that model stochasticity prevents exact reproducibility.

D. Future Work

  • The authors propose input/output or joint filters, safety alignment, and provable defenses as directions for defending against structured command hijacking.They also conclude that agents must judge inferred instructions against physical state, environment, and mission objectives rather than relying only on sensor filtering.Source: D. Future Work · pp. 12–13, §XI · p. 13

XI. CONCLUSIONS

  • The conclusion frames CHAI as a new attack vector against embodied LVLMs and calls for principled multimodal defenses before critical deployments.It argues that agents must assess whether an inferred instruction fits their physical state, environment, and mission objectives.Source: D. Future Work · p. 13, §XI · p. 13

D. Future Work

  • The study evaluates out-of-the-box LVLMs and leaves adversarial training, including its effects on agent and attack performance, for future work.The authors note that domain-specific training does not inherently address adversarial vulnerabilities.Source: D. Future Work · p. 13

XI. CONCLUSIONS

  • Authentication and suppressing multimodal inputs are limited defenses: open settings undermine trusted-entity assumptions, while suppression can block legitimate interactions and degrade task performance.The paper notes that vehicles may need to respond to safety-critical signs from flaggers.Source: §XI · p. 13

LLM USAGE CONSIDERATIONS

  • Exact reproducibility is not guaranteed because the models behave probabilistically; experiments also limit optimization iterations, decision variables, and the word set to reduce query costs.The word set is limited to ten; the authors say expanding the search or optimization could increase attack success but require more queries.Source: LLM USAGE CONSIDERATIONS · p. 14

APPENDIX A DATASETS

The appendix describes how the authors construct vehicle and drone image datasets and establish benign labels. Drone images use randomized simulated trajectories to capture varied poses and viewpoints.

1) Ground Vehicles:

  • For ground vehicles, the authors curate NuScenes urban frames where misclassified commands could risk collisions, then use the modal DriveLM output across eleven runs as each frame’s benign action.Comparing responses before and after adding a visual prompt measures the attack’s ability to override the baseline command.Source: 1) Ground Vehicles · p. 15

2) Drones:

  • Because standard drone datasets are unavailable, the authors create simulated drone data and use randomized LTI trajectories to capture camera frames for the Known and Transferability sets.The LTI model is used only for data collection, not mission control; randomized positions provide varied angles, altitudes, and distances.Source: 2) Drones · pp. 15–16
  • For emergency landing, the authors collect images of safe and unsafe rooftop settings, including two buildings in one city and buildings across varied cities for transferability testing.The varied-city set includes pedestrians on rooftops; the paper says it is used to test whether the attack still works on transferability images.Source: 2) Drones · pp. 15–16
  • For CloudTrack, the dataset includes streets with multiple actors and both dark, police-like cars and light cars lacking police-like features.The authors use the latter to test whether an attack can change the model’s decision even when vehicles do not resemble police cars.Source: 2) Drones · p. 16
  • At each simulated time step, the authors capture a ground photo and vary the camera angle to account for drone tilt; landing and CloudTrack labels are manually annotated.They also annotate where attacks can be placed, including vehicle positions and rooftops.Source: 2) Drones · p. 16

APPENDIX B ADDITIONAL RESULTS

Additional experiments show CHAI overriding a benign road sign and retaining effectiveness on rain-filtered transfer images. The appendix also describes a universal image-patch attack baseline.

APPENDIX B ADDITIONAL RESULTS

  • With a benign Stop sign, GPT-4o recommends stopping without an attack but suggests turning left when the CHAI attack is attached to a road structure.The result indicates that CHAI can override benign signs containing text.Source: Appendix B · p. 16
  • On rain-filtered Transferability Images, the attack achieves over 50% ASR across applications and reaches 80% in CloudTrack.The procedure places the attack before applying the rain filter, so the sign is affected by the weather transformation.Source: Appendix B · pp. 16–17

APPENDIX C CLASSICAL IMAGE PATCH ATTACK

  • The classical image-patch baseline searches for a universal continuous-value patch that minimizes language-modeling loss toward a target label across images, using cross-entropy for the label-space objective.The appendix presents this as a white-box patch attack baseline.Source: Appendix C · p. 17

A. Patch Attacks vs. CHAI

Classical patches have limited effects in the tested landing scenario, while CHAI is designed to guide an LVLM’s reasoning through visual prompts. White-box optimization yields essentially the same performance as black-box attacks.

A. Patch Attacks vs. CHAI

  • The patch attack raises Deception Rate (ASR) from 0% to 10%; increasing patch coverage from 5.5% to 64% raises ASR to 40%.The authors attribute limited performance partly to the patch’s small influence on the full image context and describe the larger, conspicuous patch as impractical for real-world use.Source: A. Patch Attacks vs. CHAI · p. 17
  • Hard-label patches target the final output while ignoring chain-of-thought reasoning, which the authors say makes them less effective against LVLMs using CoT.The cited discussion links step-by-step reasoning to greater robustness against generalized attacks [56].Source: A. Patch Attacks vs. CHAI · p. 17
  • CHAI instead aims to misguide the LVLM’s CoT process, using multimodal understanding to steer reasoning toward a target label.The paper contrasts this with forcing a final output through a hard-label method.Source: A. Patch Attacks vs. CHAI · p. 17

APPENDIX D OPTIMIZATION FOR WHITE-BOX MODELS

  • White-box optimization on InternVL performs essentially the same as the black-box attacks, so the paper emphasizes black-box results for their generality and performance.The white-box setup trains glyph RGB values and applies translation, rotation, and isotropic scaling.Source: Appendix D · pp. 17–18

APPENDIX E SYSTEM PROMPTS

The appendix provides system prompts for CloudTrack object detection, DriveLM autonomous driving, and drone emergency landing. The prompts specify the visual questions or decisions requested in each use case.

APPENDIX E SYSTEM PROMPTS

  • The appendix presents CloudTrack and DriveLM use-case questions drawn from the CloudTrack repository and DriveLM paper.The prompts are labeled for CloudTrack and DriveLM, respectively.Source: Appendix E · pp. 18–19
  • The CloudTrack prompt asks an object-detection assistant whether an image contains a Santa Cruz Police Department police car.Source: Appendix E · p. 19
  • The DriveLM prompt uses six vehicle photos to ask about scene objects, their motion and future states, and safe ego-vehicle actions.It also asks about object priority, visual descriptions, and actions that could lead to collisions.Source: Appendix E · p. 19
  • The drone emergency-landing prompt asks the assistant to identify buildings, assess rooftop people or obstacles, and select a safe landing rooftop.Source: Appendix E · p. 19
Loading 2510.00181v2…