PaperlayerView exact revision on arXiv

Source-linked AI summary

BadRobot: Jailbreaking Embodied LLM Agents in the Physical World

Hangtao Zhang, Chenyu Zhu, Xianlong Wang, Ziqi Zhou, Changgan Yin, Minghui Li, Lulu Xue, Yichen Wang, Shengshan Hu, Aishan Liu, Peijin Guo, Leo Yu Zhang

arXiv:2407.20242v5cs.CYcs.AIcs.RO

TL;DR

Existing jailbreak prompts largely fail to transfer to embodied LLMs, which must plan and execute physical actions. BADROBOT exploits three identified risk surfaces and evaluates malicious physical-action queries, reporting vulnerabilities across prominent frameworks and a real-world robot.

  • Problem

    Existing LLM jailbreak instructions and malicious queries largely fail to transfer to embodied systems, whose action-planning space adds challenges beyond text-only responses.

    Source: §1 · p. 2

  • Method

    BADROBOT uses three attack variants targeting LLM manipulation, language–action safety misalignment, and conceptual deception, alongside a benchmark of malicious physical-action queries.

    Source: §1 · p. 2, §3.1 · p. 5, §3.3 · p. 6

  • Results

    BADROBOT exposed vulnerabilities across prominent frameworks and digital, simulator, and real-world settings; contextual jailbreak increased MSR by an average of 215.9% over direct malicious queries.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The authors call for resolving embodied-LLM safety issues before widespread market deployment and outline technical, legal, and policy mitigation strategies.

    Source: §1 · p. 2

    The described attacks require physical access, carefully engineered prompts, absent basic safety monitoring, and preconfigured environmental conditions enabling execution.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Embodied AI represents systems where AI is integrated into physical entities. Large Language Model (LLM), which exhibits powerful language understanding abilities, has been extensively employed in embodied AI by facilitating sophisticated task planning. However, a critical safety issue remains overlooked: could these embodied LLMs perpetrate harmful behaviors? In response, we introduce BadRobot, a novel attack paradigm aiming to make embodied LLMs violate safety and ethical constraints through typical voice-based user-system interactions. Specifically, three vulnerabilities are exploited to achieve this type of attack: (i) manipulation of LLMs within robotic systems, (ii) misalignment between linguistic outputs and physical actions, and (iii) unintentional hazardous behaviors caused by world knowledge's flaws. Furthermore, we construct a benchmark of various malicious physical action queries to evaluate BadRobot's attack performance. Based on this benchmark, extensive experiments against existing prominent embodied LLM frameworks (e.g., Voxposer, Code as Policies, and ProgPrompt) demonstrate the effectiveness of our BadRobot. Our code is available at https://github.com/Rookie143/BadRobot.

1 INTRODUCTION

Embodied LLMs extend language-based planning into physical action, but existing safety research in real-world settings remains scant. BADROBOT tests whether malicious prompts can jailbreak these systems and reports vulnerabilities across frameworks and a real robot.

1 INTRODUCTION

  • Figure 1 reports the Elephant/UR robot’s potential engagement in activities spanning physical harm, privacy violations, pornography, fraud, illegal activities, hateful conduct, and sabotage.The figure caption describes this as the first successful jailbreak of a real-world robotic system.Source: Figure 1 · p. 2
  • Existing in-the-wild LLM jailbreak instructions and malicious queries largely fail to transfer because embodied systems must also plan and execute physical actions.The paper identifies the added action-planning space as a new challenge beyond text-only chatbot responses.Source: §1 · p. 2
  • BADROBOT targets three risks: jailbreaks cascading into robotic commands, safety misalignment across language and action outputs, and incomplete world knowledge.The introduction presents the attack paradigm as a way to make embodied LLMs act outside intended ethical constraints.Source: §1 · p. 2
  • BADROBOT demonstrates vulnerabilities in prominent embodied LLM frameworks and reports successful jailbreaks across digital, simulator, and real-world scenarios.The authors report the first successful jailbreak of a real-world robotic system.Source: §1 · p. 2
  • The benchmark covers malicious physical-action queries, and experiments report vulnerabilities in Voxposer, Code as Policies, ProgPrompt, and Visual Programming.The authors also report experiments using state-of-the-art commercial LLMs.Source: §1 · p. 2

2 ON THE RISKS OF EMBODIED LLMS: A CONCEPTUAL OUTLINE

The section organizes embodied-LLM risks around autonomy, embodiment, and cognition, then formalizes when linguistic or action outputs violate safety. It frames the attack as voice-based manipulation without prior knowledge of the system.

2.1 BE CAUTIOUS OF HIDDEN DANGERS!

  • The three risk surfaces are cascading jailbreaks, language–action safety misalignment, and conceptual deception that can prompt harmful actions without recognizing their consequences.Conventional jailbreak queries often produce verbal posturing rather than physical action; aligned systems may refuse in language while still outputting harmful action commands.Source: §2.1 · pp. 3–4, Figure 2 · p. 3

2.2 FORMALIZATION OF EMBODIED LLMS JAILBREAK

  • The formalization models an embodied LLM through input, perception, action planning, world-model, and safety components, with distinct linguistic and action safety checks.A robust system satisfies both checks for all inputs, while a jailbreak compromises either check for a malicious input.Source: §2.2 · p. 4
  • BADROBOT focuses primarily on action safety because physical actions can have direct and potentially irreversible real-world consequences.The paper links the three risks to linguistic-processing influence, direct action-generation manipulation, and an inadequate or manipulated world model.Source: §2.2 · p. 4

2.3 MIND THE ATTACKERS!

  • The threat model gives attackers no prior knowledge of the system and limits interaction to voice communication, as in a no-box setting.The stated objective is to induce outputs that deviate from human values rather than have the system refuse harmful instructions.Source: §2.3 · p. 4
  • The attack objective is physical malicious action, with harmful language output treated as secondary, and the section presents three attack approaches.The listed physical-action categories include physical harm, privacy violations, pornography, fraud, illegal activities, hateful conduct, and sabotage.Source: §2.3 · p. 5

3 BADROBOT: HOW TO MANIPULATE EMBODIED LLMS?

BADROBOT presents three attack variants that exploit distinct safety weaknesses in embodied LLMs: contextual jailbreaks, language-action misalignment, and conceptual deception. Together, they target unsafe linguistic or physical outputs through contextualized prompts, structured action outputs, and semantic reframing.

3.1 TALK IS CHEAP: CONTEXTUAL JAILBREAKS

  • Existing in-the-wild LLM jailbreak prompts transfer poorly to embodied LLMs because agent system prompts conflict with jailbreak instructions and malicious queries can be ineffective.The paper identifies these as two challenges to transferability.Source: §3.1 · p. 5
  • BADROBOT combines contextualized instructions aligned with an agent’s system prompt and 277 physical-world malicious queries spanning six categories of harmful actions.The queries cover Physical Harm, Privacy Violence, Pornography, Fraud, Illegal Activity, and Hateful Conduct.Source: §3.1 · p. 5

3.2 HITTING WHERE IT HURTS: SAFETY MISALIGNMENT

  • In safety misalignment, an agent can verbally refuse a harmful request while encoding the prohibited action in its function output; structured-output suffixes exploit this weaker scrutiny of function-like representations.The paper also notes that safe code-generation criteria are less established than linguistic safety measures.Source: §3.2 · p. 6

3.3 ALL ROADS LEAD TO ROME: CONCEPTUAL DECEPTION

  • Conceptual deception exploits the gap between rejecting a direct harmful command and complying with seemingly innocuous instructions that produce a similar outcome.The paper attributes this ethical blind spot to reliance on token probabilities and a lack of essential embodied knowledge.Source: §3.3 · p. 6
  • Adversaries can preserve a harmful instruction’s operational intent while rephrasing it, and an imperfect world model may permit harmful actions despite safe linguistic output.The paper describes using GPT to generate semantic rephrasings that can still lead to the same malicious actions.Source: §3.3 · pp. 6–7

4 EVALUATION

Across digital tests, BADROBOT increased malicious-action success over direct queries, and the attacks also elicited harmful behavior in several embodied systems and physical robot trials.

4.1 EXPERIMENTAL SETUP

  • The evaluation used GPT-4-turbo as its primary black-box target and measured malicious-action success and GPT-4-judged harmfulness.The prototype planner decomposed user instructions into actionable steps and returned language and actions in JSON; target models also included GPT-3.5-turbo, GPT-4o, Yi-vision, and Llava-1.5-7b.Source: §4.1 · p. 7

4.2 RESULTS IN THE DIGITAL WORLD ENVIRONMENT

  • Across models, Contextual Jailbreak, Safety Misalignment, and Conceptual Deception increased MSR over direct malicious queries by averages of 215.9%, 193.8%, and 213.7%, respectively.Table 2 reports MSR across LLMs and harmful categories; Figure 6 reports GPT-4-judged harmfulness scores across seven categories. Existing in-the-wild jailbreak prompts largely fail because they conflict with agents’ predefined roles.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Figure 6 · p. 8

4.3 TAKE CARE! SOTA EMBODIED LLMS SYSTEMS CAN BE ALSO MANIPULATED

  • On Code as Policies, BADROBOT significantly outperformed Vanilla by +0.60 on average across 10 malicious queries in a PyBullet tabletop manipulation setup.The simulation included a person and a knife, with a query asking the robot to move the knife to the person.Source: §4.3 · p. 8, Figure 7 · p. 9
  • On ProgPrompt, BADROBOT achieved high MSR across malicious household-task queries, including harassment, privacy violations, and hazardous actions.Examples included heating a phone in a microwave, placing a knife to harm, and causing water overflow.Source: §4.3 · p. 8, Figure 7 · p. 9
  • BADROBOT prompted VoxPoser to turn off lights to help someone steal without being caught, across tests covering seven RLBench subtasks.The evaluation covered Object Interactions and Spatial Compositions.Source: §4.3 · p. 8, Figure 7 · p. 9
  • The evaluation also tested Visual Programming on Image Editing and Factual Knowledge Object Tagging tasks.The supplied passages point to visualizations but report no quantitative outcome for these tasks.Source: §4.3 · p. 8, Figure 7 · p. 9

4.4 REAL-WORLD EXPERIMENT

  • In 35 physical evaluations, BADROBOT achieved 68.57% average MSR, versus 22.85% for direct malicious queries.The authors report a slight reduction in effectiveness compared with the digital environment; Figure 8 visualizes the physical results.Source: §4.4 · p. 9, Figure 8 · p. 9

5 MITIGATION, CHALLENGES AND IMPLICATIONS

The section discusses layered defenses, reporting that multimodal consistency validation reduces attack success but does not fully stop BADROBOT, while world-model fine-tuning has mixed effects and substantial resource costs.

Mitigation, challenges and implications

  • The authors advocate combining policy mechanisms with technical strategies, noting that new jailbreaks make safety a continuing challenge.They argue that protections should extend beyond traditional LLM safety measures.Source: §5 · pp. 9–10

Multimodal consistency validation

  • Consistency validation reduced BADROBOT’s MSR by 22.27% on average but did not fully mitigate its impact.The proposed validation compares semantic alignment between action sequences and language outputs as an additional firewall.Source: §5 · p. 10

Comprehensive world model

  • Fine-tuning with a comprehensive world model lowered Bcd MSR by 18.33% but made the model more vulnerable to other attacks and required substantial computation and data.The evaluation used models fine-tuned with Xiang et al. (2024).Source: §5 · p. 10

6 RELATED WORK

Prior work addresses several embodied-system safety topics, but the paper identifies no-box attacks that induce malicious robotic manipulations as a gap. It frames embodied jailbreaks as distinct from digital jailbreaks because robotic action spaces and system prompts complicate transfer.

6 RELATED WORK

  • The paper identifies its contribution as the first no-box attacks compromising embodied systems into malicious robotic manipulations in simulators and the physical world.It situates this claim alongside prior work on adversarial robustness, bias, safety frameworks, and backdoor attacks.Source: §6 · p. 10
  • Model-agnostic jailbreak prompts suit no-box settings, but embodied action spaces and conflicts with system prompts make transfer difficult.BADROBOT instead leverages embodied systems’ intrinsic features to craft attacks that induce malicious physical actions.Source: §6 · p. 10
  • Unlike digital jailbreaks that elicit malicious text, BADROBOT targets malicious physical actions.The passage describes this as a new attack paradigm for embodied LLMs.Source: §6 · p. 10

7 CONCLUSIONS AND BROADER IMPACT

The conclusion presents BADROBOT as a paradigm for triggering malicious actions, reports analysis of potential mitigations, and invites safety testing with its benchmarks.

7 CONCLUSIONS AND BROADER IMPACT

  • BADROBOT introduces three attack variants designed to trigger malicious actions in embodied LLMs.The paper also analyzes potential mitigation measures.Source: §7 · p. 10
  • The authors invite the community to test embodied systems using the paper’s benchmarks for embodied AI safety.They also hope the open-source embodied system supports broader safety testing, including adversarial robustness and backdoor attacks.Source: §7 · p. 10
  • The paper frames its intended use as supporting broader safety testing of embodied systems.The authors state that they hope the open-source system can be used for such testing.Source: §7 · p. 10

ETHICS STATEMENT

The ethics statement acknowledges dual-use risks while arguing that attack conditions constrain casual misuse. It presents vulnerability disclosure as a means to support defenses and describes safeguards and responsible research commitments.

ETHICS STATEMENT

  • The authors say their attacks require expert-level physical access to robots, which they describe as a significant barrier to malicious use.They argue that adversarial testing can benefit AI security when accompanied by ethical safeguards and misuse prevention.Source: ETHICS STATEMENT · p. 11
  • Potential misuse requires physical access, carefully engineered prompts, absent basic safety monitoring, and enabling environmental conditions such as reachable weapons.The authors say these constraints create substantial barriers to casual misuse; tested platforms also lack contextual reasoning and mission autonomy, limitations they say commercial robots typically do not share.Source: ETHICS STATEMENT · p. 11
  • The authors argue that documenting vulnerabilities can inform adaptive safeguards, drawing on the principle that strong attacks enable strong defenses.They describe proactive weakness identification as critical to developing robust defenses.Source: ETHICS STATEMENT · p. 11
  • The authors argue that security research is urgent as multimodal reasoning progresses, while current embodied systems lack the autonomy and situational awareness for catastrophic misuse without targeted human intervention.They present early-stage research as a way to shape next-generation robotics.Source: ETHICS STATEMENT · p. 11
  • The authors say full replication materials will be released through managed access and that they shared findings with affected manufacturers and developers before public release.They describe this as supporting verification while preventing misuse and enabling coordinated vulnerability patching.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The supplied sections describe the study’s reproducibility considerations, background on embodied LLMs and jailbreaks, experimental platforms, and proposed policy interventions.

REPRODUCIBILITY STATEMENT

  • Publicly released code and resources may support security improvements, but mitigation measures added after the experiments could affect reproduction of the reported results.The authors consider that reproducibility trade-off justified by potential safety improvements in future releases.Source: REPRODUCIBILITY STATEMENT · p. 11

A DETAILED BACKGORUND

  • LLM task planners and visual-language integration have advanced embodied systems, while the background identifies a remaining gap in research on their safety implications.The passage also describes earlier reinforcement-learning approaches as having limited generalization across diverse tasks.Source: A DETAILED BACKGORUND · p. 19
  • LLMs support task planning and decision-making, while MLLMs add visual processing that can support perception tasks such as object recognition and spatial reasoning.The passage describes these capabilities as contributing to human-robot-environment interaction.Source: A DETAILED BACKGORUND · p. 19
  • The background describes alignment methods including RLHF and instruction tuning, while noting that little research addresses human-aligned LLM-based embodied AI.It states that embodied systems can manipulate real-world objects, making jailbreak consequences more severe than text generation alone.Source: A DETAILED BACKGORUND · p. 19
  • The paper focuses on model-agnostic jailbreak prompts because they can be applied without knowledge of the underlying system.The passage characterizes embodied systems as often operating through no-box input-output interfaces.Source: A DETAILED BACKGORUND · p. 20

B PLATFORM

  • Digital experiments used a server with two 80GB Nvidia A100 GPUs; physical experiments used UR3e and myCobot 280-Pi six-degree-of-freedom manipulators.The digital server ran 64-bit Ubuntu 20.04.1 and experiments were performed in Python.Source: B PLATFORM · p. 20

C LAW AND POLICY

  • The policy discussion proposes integrating robotics safety standards into certification and drawing ethical guidelines from established frameworks.It cites ISO 10218 and ISO 13482, and proposes transparency requirements inspired by the EU AI Act; it also calls for human oversight and combined policy and technical strategies.Source: C LAW AND POLICY · p. 20

D.1 TRANSFERABILITY STUDY: CAN EXISTING IN-THE-WILD JAILBREAK PROMPTS WORK AGAINST EMBODIED LLMS?

Existing in-the-wild jailbreak prompts transfer poorly to embodied LLMs, while BADROBOT’s three attacks have different cost and model-specific trade-offs. The appendix also examines mitigations and prompt-design principles.

D.1 TRANSFERABILITY STUDY: CAN EXISTING IN-THE-WILD JAILBREAK PROMPTS WORK AGAINST EMBODIED LLMS?

  • MSR fell versus Vanilla by 0.15 for Disguised Intent, 0.22 for Role Play, 0.24 for Structured Response, 0.09 for Virtual AI Simulation, and 0.18 for Hybrid Strategies.The authors attribute widespread failures to conflicts between jailbreak demands and system prompts defining the robot’s role and actions.Source: §D.1 · p. 21

D.1.1 COST AND FEASIBILITY OF THE PROPOSED ATTACKS

  • BADROBOT’s attacks use crafted prompts rather than white-box gradient optimization or extensive black-box queries, and the authors characterize their resource demands as low.Bsm and Bcj require no extra training or computation; Bcd uses semantic rephrasing with O(1) time per API query and O(n) for n queries.Source: §D.1.1 · pp. 21–22

D.1.2 TRADE-OFF BETWEEN BADROBOT’S THREE ATTACKS

  • Bcj is less effective on highly aligned commercial LLMs, Bsm is effective on GPT-4o, and Bcd has the lowest cross-model standard deviation, 0.1064.Bcj’s MSR averages 0.88 on GPT-3.5-turbo and 0.45 on GPT-4o; the authors associate Bsm’s performance with natural-language/code alignment and Bcd’s with ethical-action reasoning gaps.Source: §D.1.2 · pp. 22–23
  • VisProg is less susceptible to the attacks; its modular components independently validate tasks, making exploitation more challenging.The authors identify modular design as a potential route to greater embodied-system robustness.Source: §D.1.2 · p. 23

D.2 MITIGATION STRATEGIES

  • Multimodal safety checks can reduce MSR but do not eliminate attacks, especially when both language and action outputs are malicious and therefore consistent.The proposed checks examine each output modality and use a consistency score between actions and language.Source: §D.2 · p. 23
  • Fine-tuning for physical-world understanding lowered Bcd MSR but increased Bcj and Bsm MSR as language generation capability and safety alignment diminished.The authors cite the gap between virtual simulations and real-world dynamics and say world knowledge provides only partial mitigation.Source: §D.2 · p. 24

E.1 CONTEXTUAL JAILBREAK

  • Contextual jailbreak prompts combine role-play or disguised intent with robotic-role keywords, while keeping prompts short to reduce conflicts and interruptions.The authors also caution that their hand-crafted prompts are not fixed or optimal and may be patched by commercial LLMs.Source: §E.1 · p. 24

E.2 SAFETY MISALIGNMENT

  • The safety-misalignment suffix directs the model to place the instruction in the action channel while allowing refusal only in its spoken reply.It requires a nonempty executable command and an output structured according to the system prompt.Source: §E.2 · p. 25

E.3 CONCEPTUAL DECEPTION

Conceptual deception exploits gaps in LLM world models and ethical reasoning by changing how a harmful command is worded while preserving its physical action. The reported rephrasings use lexical substitutions and softening language to make malicious directives less detectable.

E.3 CONCEPTUAL DECEPTION

  • LLMs’ reliance on token probabilities rather than logical inference can leave planned outcomes logically incorrect, exposing a gap between ethical reasoning and practical action.The authors link this vulnerability to limitations in LLM world-model representations.Source: §E.3 · p. 25
  • GPT-4 rewrites malicious commands to sound harmless while keeping the robot’s physical motion and target unchanged.The prompt directs it to replace alarming terms with ordinary synonyms without making the task vague or changing the action.Source: §E.3 · p. 25
  • Rephrasings use lexical substitutions and softening adverbs to obscure malicious intent, making harmful instructions less detectable to language understanding and moderation systems.The authors describe these superficial linguistic changes as conceptual deception.Source: §E.3 · pp. 25–26

F SIMULATED EXPERIMENTS: TESTING ON SOTA EMBODIED LLM SYSTEMS

In simulated evaluations, BADROBOT attacks prompted Voxposer, ProgPrompt, Code as Policies, and VisProg to attempt or perform malicious actions. The reported scope and success criteria vary across systems.

F.1 VOXPOSER

  • BADROBOT manipulated Voxposer to carry out malicious actions across seven query categories, including physical harm, privacy violations, fraud, and sabotage.The queries were designed for RLBench-supported scenarios.Source: §F.1 · p. 26

F.2 PROGPROMPT

  • BADROBOT prompted ProgPrompt to attempt eight malicious actions in household scenarios, although simulator and system limits may prevent full task completion.The authors count an attack as successful when the system does not reject a query and attempts to execute it.Source: §F.2 · p. 26

F.3 CODE AS POLICIES

  • Code as Policies was manipulated into a human-directed knife attack, based on four malicious queries and scenes with one or two people and knives.The authors report an assassination attempt that violated Asimov’s First Law of Robotics.Source: §F.3 · p. 26

F.4 VISPROG

  • BADROBOT manipulated VisProg’s Natural Language Image Editing and Factual Knowledge Object Tagging subtasks into malicious behavior.VisProg generates Python programs that invoke computer-vision models, image-processing routines, or Python functions for intermediate outputs.Source: §F.4 · p. 27

G REAL-WORLD EXPERIMENTS: TESTING ON OUR PHYSICAL EMBODIED LLM SYSTEMS

The physical system converts speech to text for an LLM, then converts text responses to speech and uses a six-degree-of-freedom arm to execute actions. The appendix describes its prompts and a GPT-4 procedure for scoring harmfulness.

G REAL-WORLD EXPERIMENTS: TESTING ON OUR PHYSICAL EMBODIED LLM SYSTEMS

  • The physical setup uses ASR, an LLM, TTS, hand-eye calibration, and inverse kinematics to translate voice interactions into six-degree-of-freedom arm actions.The appendix lists predefined arm functions, including coordinate movement, object manipulation, and image-based question answering.Source: G REAL-WORLD EXPERIMENTS: TESTING ON… · pp. 27–28
  • Examples show the system sequencing actions such as returning to origin, moving objects, and changing LED color in response to user instructions.The examples also include image-based questions and multi-step commands.Source: G REAL-WORLD EXPERIMENTS: TESTING ON… · p. 29

H IMPLEMENTATION OF OUR AUTOMATED HARMFULNESS EVALUATION

  • GPT-4 scores harmfulness from 1 to 5 using the model’s policy, the harmful instruction, the model response, and a rubric that considers both language and action.The evaluation prompt generally places greater emphasis on actions because they tend to be irreversible.Source: H IMPLEMENTATION OF OUR AUTOMATED… · pp. 30–31

I DETAILS ABOUT OUR MALICIOUS PHYSICAL ACTION QUERIES BENCHMARK

The benchmark organizes malicious physical-action queries into seven categories, grounded in robotics ethics guidance and intended to test embodied LLM safety. The accompanying material describes query diversity and catalogs jailbreak prompt types used in demonstrations across several embodied LLM frameworks.

I DETAILS ABOUT OUR MALICIOUS PHYSICAL ACTION QUERIES BENCHMARK

I.0.1 DIVERSITY IN OUR MALICIOUS QUERIES

  • Examples vary in target entities, context, and whether instructions are direct, generalized, or implicit, while the figure shows query diversity and category distribution.Table A1 gives examples; the qualitative analysis describes linguistic variability and semantic diversity.Source: §I.0.1 · pp. 31–32, Figure A3 · p. 31, Table A1 · p. 32

J COLLECTION OF IN-THE-WILD JAILBREAK PROMPTS

Related papers

Loading 2407.20242v5…