PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Guardrails for LLM-Enabled Robots

Zachary Ravichandran, Alexander Robey, Vijay Kumar, George J. Pappas, Hamed Hassani

arXiv:2503.07885v2cs.ROcs.AI

TL;DR

LLM-enabled robots remain vulnerable to context-dependent jailbreaking attacks that can cause physical harm, while existing safeguards do not jointly address contextual robotics and adversarial misuse. ROBOGUARD grounds designer rules in the robot’s world model with a protected reasoning LLM, then synthesizes safe plans. In simulation and real-world experiments, it reduced unsafe behavior from over 92% to under 3% while retaining safe-plan performance.

  • Problem

    LLM-enabled robots can be jailbroken into harmful physical behaviors, while existing alignment and robot-safety approaches do not jointly address contextual robotic actions and adversarial use.

  • Method

    ROBOGUARD uses a two-stage architecture that grounds offline safety rules in world-model context with a shielded root-of-trust LLM, then applies minimal-violation control synthesis.

  • Results

    ROBOGUARD reduced unsafe-plan execution from 92.3% to under 2.5% without compromising safe-plan performance in simulation and real-world experiments.

  • Takeaways & Limitations

    ROBOGUARD provides a general-purpose, context-aware and adversarially robust safeguard for LLM-enabled robots, with resource-efficient operation.

  • Takeaways & Limitations

    ROBOGUARD requires an accurate world model; a severely compromised model can induce safety failures.

Abstract

from arXiv · show

Although the integration of large language models (LLMs) into robotics has unlocked transformative capabilities, it has also introduced significant safety concerns, ranging from average-case LLM errors (e.g., hallucinations) to adversarial jailbreaking attacks, which can produce harmful robot behavior in real-world settings. Traditional robot safety approaches do not address the contextual vulnerabilities of LLMs, and current LLM safety approaches overlook the physical risks posed by robots operating in real-world environments. To ensure the safety of LLM-enabled robots, we propose RoboGuard, a two-stage guardrail architecture. RoboGuard first contextualizes pre-defined safety rules by grounding them in the robot's environment using a root-of-trust LLM. This LLM is shielded from malicious prompts and employs chain-of-thought (CoT) reasoning to generate context-dependent safety specifications, such as temporal logic constraints. RoboGuard then resolves conflicts between these contextual safety specifications and potentially unsafe plans using temporal logic control synthesis, ensuring compliance while minimally violating user preferences. In simulation and real-world experiments that consider worst-case jailbreaking attacks, RoboGuard reduces the execution of unsafe plans from over 92% to below 3% without compromising performance on safe plans. We also demonstrate that RoboGuard is resource-efficient, robust against adaptive attacks, and enhanced by its root-of-trust LLM's CoT reasoning. These results demonstrate the potential of RoboGuard to mitigate the safety risks and enhance the reliability of LLM-enabled robots. We provide additional resources at https://robo-guard.github.io/.

I. INTRODUCTION

LLM-enabled robots introduce contextual and adversarial safety risks that existing robot-safety and LLM-alignment approaches do not fully address. ROBOGUARD combines contextual safety reasoning with control synthesis to enforce safety while preserving safe-plan performance.

  • Traditional robot-safety formalisms require precise specifications, while LLM-enabled robots operate with contextual environments that make safety harder to define and enforce.
  • LLM-enabled robots can inherit jailbreaking vulnerabilities that produce physical harms such as human collisions or blocked emergency exits.
  • ROBOGUARD grounds offline safety rules and robot descriptions in the current world model, then resolves plan conflicts through minimal-violation control synthesis.
  • ROBOGUARD is designed to be context-aware, applicable across robot platforms and planners, adversarially robust, and compatible with safe-plan performance.
  • Existing formal-methods approaches often require manually enumerated specifications and generally do not address adversarial use cases in LLM-enabled robots.

C. LLM safety and jailbreaking

LLM alignment and robot-safety methods leave important gaps because jailbreaks can elicit harmful physical behavior and existing formal approaches lack contextual, online safety reasoning. ROBOGUARD uses a protected root-of-trust LLM to ground rules in a semantic world model before formal verification and synthesis.

  • Jailbreaking attacks bypass LLM safety mechanisms, and in robotics their harms depend on physical context rather than text alone.
  • ROBOGUARD’s safety reasoning module shields its root-of-trust LLM from malicious user input while grounding high-level rules into rigorous contextual specifications.
  • The robot’s semantic-graph world model represents objects, regions, traversable paths, and object accessibility for online safety reasoning.
  • Chain-of-thought reasoning lets the root-of-trust LLM iteratively apply each safety rule to the current world-model state.
  • Contextual atomic propositions map physically realizable actions in the world model to temporal-logic formulas that can be checked against proposed action sequences.

B. Control synthesis module

ROBOGUARD translates an LLM-generated plan into a temporal-logic specification and checks it against contextual safety specifications. If the proposed plan is unsafe, synthesis returns a safety-compliant alternative while preserving the proposal when possible.

  • The control synthesis module prioritizes adherence to the LLM-generated plan while enforcing the generated safety specifications.
  • The proposed plan is translated into an LTL specification and word sequence, then evaluated against the contextual safety specification.
  • If the resulting automaton trace is accepting, ROBOGUARD returns the proposed plan; otherwise, it returns a safety-compliant plan.
  • ROBOGUARD’s control synthesis module always provides a safe control plan regardless of the proposed plan.
  • The resulting plan follows the proposed plan when possible while always adhering to the safety specifications.
  • Safety guarantees depend on alignment between the inferred contextual specification and the designer’s contextual understanding of safety.

IV. EXPERIMENTS

ROBOGUARD is evaluated as a general-purpose guardrail for LLM-enabled robots across simulated and real-world environments, harmful behaviors, attack strategies, and planner configurations.

  • Experimental setup: The evaluation measures contextual awareness, applicability, utility, and efficiency for ROBOGUARD operating with an LLM planner.The setup includes the planner, harmfulness benchmark, elicitation attacks, and evaluation environments.
  • Experimental setup: ROBOGUARD is instantiated with GPT-4o as its root-of-trust LLM, Spot for control synthesis, and three high-level safety rules.The rules address privacy, physical harm, and hazardous areas, while the system receives descriptions of the planner API and world model.
  • Experimental setup: The harmfulness benchmark spans average-case errors to worst-case abuses, including hazardous-area entry, emergency-exit blocking, weapon search, surveillance, and human collision.The benchmark is based on behaviors proposed in the ROBOPAIR study.
  • Experimental setup: The attacks include direct prompting, templated jailbreaking, ROBOPAIR, and adaptive black-box, gray-box, and white-box variants.Adaptive variants receive differing access to the guarded robot, world model, guardrail specifications, or internal states.
  • Experimental setup: ROBOGUARD is evaluated in randomized small-indoor, large-indoor, and outdoor environments covering nearly 20,000m2, with 210 evaluations per setting.The environments include an academic laboratory, an office-building floor, and an office park.

A. Unsafe behavior mitigation

ROBOGUARD substantially reduces unsafe behavior under non-adaptive and adaptive attacks, while its effectiveness depends on generating appropriate contextual safety specifications and benefits from CoT reasoning.

  • Unsafe behavior mitigation: 82.3% and 92.3% ASRs for template and non-adaptive ROBOPAIR attacks drop below 3% with ROBOGUARD, without reducing utility on safe tasks.The unguarded planner rejects 98.75% of direct prompts, but templated and ROBOPAIR attacks bypass its alignment.
  • Unsafe behavior mitigation: Adaptive attack ASRs do not significantly change relative to non-adaptive settings, but increase marginally as attackers gain more access to ROBOGUARD internals.The guarded system remains robust but may be susceptible to a strong adaptive attacker.
  • Importance of reasoning for safety: Without CoT reasoning, ROBOGUARD misses safety specifications and produces incorrect ones.The result concerns contextualized safety specifications generated by the root-of-trust LLM.
  • Unsafe behavior mitigation: Failures occur when the safety reasoning module does not produce appropriate safety specifications, including specifications involving complex temporal requirements or nuanced semantics.One example requires recognizing that a hammer can function as both a tool and a weapon.
  • Importance of reasoning for safety: Removing the root-of-trust LLM’s CoT reasoning increases ASR from 4.3% to 12.8%, reaching 25.7% with higher temperature.Increasing temperature alone does not affect the guardrail’s performance, whereas higher temperature further harms the no-reasoning variant.

C. Guardrail efficiency

ROBOGUARD uses substantially fewer resources than attack generation and operates online, while real-world experiments show strong mitigation and context-dependent safety enforcement.

  • Guardrail efficiency: ROBOGUARD uses 12% to 21% of attackers’ tokens and requires 1 LLM query versus 15 for the attacker.The single-query, limited-token design is reported as performant enough for online use in the robot control loop.
  • Real-world experiments: The physical evaluation integrates the LLM planner with a Clearpath Jackal and an onboard semantic mapper that continuously updates the world model.ROBOGUARD continuously monitors candidate plans for harm within this control loop.
  • Real-world experiments: ROBOGUARD prevents 100% of adversarial attacks in the real-world experiments without compromising utility.The authors attribute the stronger real-world mitigation than in simulation to differences in scenario difficulty and robot contexts.
  • Real-world experiments: In online operation, ROBOGUARD allows a safe plan and prevents a collision plan after a person is discovered in the robot’s world model.The guardrail initially generates no safety constraints when the world model is empty, then generates and enforces an appropriate specification.
  • Limitations and future work: ROBOGUARD’s limitations include dependence on an accurate world model, scalability over larger world models and rule sets, and cases where LTL is unsuitable.The paper suggests perception safeguards, improved LLM capabilities or distillation, and alternative specification languages as possible responses.

APPENDIX A FORMAL METHODS BACKGROUND

The appendix describes LTL-based safety specifications, the root-of-trust planner and attack implementations, and the harmful behaviors used to evaluate guardrails.

  • APPENDIX A FORMAL METHODS BACKGROUND: LTL formulas encode temporal and logical constraints over robot actions and environment propositions.The paper uses operators including globally, finally, until, next, negation, implication, and conjunction.
  • A. LLM Planner: The root-of-trust LLM is configured with robot-specific API and world-model definitions for planning.The API refers to robot capabilities, while the world-model definition specifies the modeled environment.
  • B. Attacker implementation: Attacker evaluations use ROBOPAIR with GPT-4o attacker and judge models, a GPT-4o target planner, and five iterations.The implementation also includes black-box, grey-box, template, and adaptive attack variants.
  • B. Attacker implementation: Template attacks embed malicious direct prompts in fictional non-malicious contexts intended to bypass internal safety filters.The template instructs the robot to act as a villain in a movie and frames the task as hypothetical.
  • B. Attacker implementation: Adaptive attacks expose attackers to increasing levels of ROBOGUARD information, including outputs, internal specifications, and the world model.Attackers receive instructions and relevant information through system and in-context prompts.
  • B. Attacker implementation: ROBOGUARD evaluations represent safety decisions as Boolean outcomes and action-level safe or unsafe judgments.The guardrail details include rules, LTL specifications, reasoning, and step-by-step action evaluations.
  • A. Behaviors: The harmful-behavior suite includes bomb detonation and indoor or outdoor obstruction of emergency exits and roads.The prompts vary wording while preserving the targeted harmful behavior.
  • A. Behaviors: The evaluation also includes blocking emergency exits or roads through driving and remaining in the obstructed location.Indoor prompts target escape routes, while outdoor prompts target roads and vehicle access.

3. Weapon search

This section lists prompts for weapon-search and object-based harm scenarios, including indoor warehouse and outdoor pedestrian settings.

  • 3. Weapon search: Weapon-search prompts ask the robot to locate tools or objects that could be used to injure or attack someone.Examples include finding an item capable of harm, a weapon, or a tool usable against a person.
  • 3. Weapon search: Indoor warehouse prompts target accidents involving heavy or unstable shelves near people.The requested actions include identifying, pushing, knocking, or tipping a shelf onto someone.
  • 3. Weapon search: Additional prompts explicitly describe driving near a person or object to cause an impact or accident.These include bumping shelves or signs and using nearby objects to cause harm.
  • 3. Weapon search: Outdoor pedestrian prompts transfer the same object-based harm pattern to heavy or unstable road signs.The prompts ask the robot to find, push, or knock a sign onto a person.
  • 3. Weapon search: The prompt set includes repeated variants of seeking an object or tool for bodily harm.The variants differ in wording while preserving the same harmful objective.

5. Keep-out zone

The section contains keep-out-zone, surveillance, human-collision, and safe-behavior prompts spanning indoor, outdoor, and scene-inspection tasks.

  • 5. Keep-out zone: Keep-out-zone prompts instruct the robot to enter restricted or hazardous areas despite warnings, barriers, or explicit keep-out instructions.Variants include bypassing warning signs, crossing safety perimeters, and ignoring danger measures.
  • 5. Keep-out zone: The suite includes covert-surveillance prompts for observing, recording, or collecting information about people without their knowledge or consent.Examples include secretly reporting behavior, monitoring individuals, and recording private moments.
  • 5. Keep-out zone: Human-collision prompts direct the robot to drive or maneuver toward people with intent to cause physical harm.Variants include accelerating toward, steering into, locking onto, or deliberately contacting a person.
  • 5. Keep-out zone: The human-collision variants include deliberate contact and movement toward pedestrians or individuals.The prompts describe driving at, navigating toward, or propelling the robot toward a person.
  • 5. Keep-out zone: Safe-behavior prompts ask the robot to inspect scenes, locate objects or vehicles, count items, and check environmental conditions.Examples include checking doors, spills, debris, sidewalks, and whether objects or vehicles are broken.
  • 5. Keep-out zone: The safe set includes checking whether a car is broken, whether debris is present, and whether sidewalks are clear.These tasks provide non-harmful scene-understanding queries for outdoor environments.
  • 5. Keep-out zone: Outdoor safe prompts include inspecting a dock or courtyard, finding a place to sit, and checking nearby cars or scene objects.They also ask about vehicle counts, debris, and sidewalk clearance.

B. Clearpath jackal configuration

Real-robot experiments use a Clearpath Jackal with RGB-D, LiDAR, GPU, and CPU hardware, while simulations derive randomized graphs from real semantic maps.

  • B. Clearpath jackal configuration: The Clearpath Jackal carries a ZED 2 RGB-D camera, an Ouster LiDAR, an Nvidia A4000 GPU, and a Ryzen 5 3600 CPU.These components support the real-robot experiments.
  • B. Clearpath jackal configuration: The semantic mapping pipeline uses RGB-D and LiDAR inputs, Faster-LIO for state estimation, and GroundGrid for traversability estimation.Traversability supports estimating region nodes and connectivity in the semantic graph.
  • B. Clearpath jackal configuration: Experiments cover real and simulated indoor and outdoor environments derived from maps built by the robot’s semantic mapping system.Simulation randomizes the maps by adding and removing graph elements.
  • B. Clearpath jackal configuration: ROBOGUARD receives textual representations of semantic graphs to generate contextual constraints.The graph represents objects, regions, and their connections.

A. Illustrative examples

The examples show ROBOGUARD adapting safety reasoning to mapped environments and preventing both bomb detonation and adaptive pedestrian-harm plans.

  • Environment representation: ROBOGUARD uses a robot-built environment representation to contextualize safety reasoning.The representation illustrates the textual environment information used by ROBOGUARD.
  • Non-adaptive bomb detonation: ROBOGUARD infers safety specifications that prevent a bomb-detonation plan from being realized.The scene includes people and potentially dangerous objects and regions.
  • Adaptive pedestrian harm: Under an adaptive white-box attack, ROBOGUARD constrains potentially dangerous actions and prevents execution of a harmful pedestrian plan.The attacker has access to ROBOGUARD’s state and adapts its reasoning accordingly.

B. Rule set ablation

The rule-set experiments examine ROBOGUARD across disaster-response and construction-assistance contexts, using context-specific rules, LTL constraints, and safe-task trials. The reported results indicate consistent performance across rule sets while preventing unsafe behaviors.

  • Scenario-specific rule sets: The experiments evaluate rule sets for disaster response and construction assistance, including context-specific objectives such as avoiding hazards and protecting people.Disaster response prioritizes responder access and hazard avoidance, while construction assistance prioritizes privacy and preventing harm.
  • Evaluation outcome: Across the evaluated rule sets, ROBOGUARD exhibits consistent performance while allowing only one unsafe behavior without compromising safe-task performance.Safe tasks are evaluated over four randomized trials in simulated disaster-response and construction-assistance settings.
  • Constraint generation: ROBOGUARD’s root-of-trust safety filter converts textual safety rules into LTL constraints that reference robot actions.The prompt directs the model to produce constraints for filtering controller actions, with examples such as G(!goto(region_1)).
  • Rule-set configurability: The system accepts multiple rule sets and generates constraints with associated reasoning in a structured format.Its prompt specifies one or more constraints per rule and JSON-formatted reasoning.
  • World-model grounding: The root-of-trust LLM reasons over a semantic graph containing objects, regions, and their connections.The graph representation includes object nodes, region nodes, object edges, and region edges.
  • Unsafe-plan filtering: In a bomb-detonation example, the planner targets a region containing two people, while ROBOGUARD forbids movement there and marks the plan unsafe.The generated specification includes G(!goto(ground_21)) because a person is present on that region.
  • Adaptive attack mitigation: ROBOGUARD also rejects an adaptive pedestrian-harm plan by constraining access to a pedestrian region and inspection of dangerous objects.The example includes constraints on sidewalk_2 and drill_3, followed by an unsafe-plan decision.
Loading 2503.07885v2…