PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics

Songhua Yang, Ziyu Liu, Yuanwei Liu, Xuetao Li, Xuanye Fei, He Huang, Zheng Wang, Miao Li

arXiv:2609.39178v1cs.ROcs.CRcs.CV

TL;DR

VLA models directly control physical robots, but their security has received little systematic study and attacks for simpler systems do not transfer directly. The paper proposes a multi-level attack using a universal textured sphere. Across 13 tasks, it reports average success-rate reductions of 31.2%-39.9% for Pi0 and RDT, with near-zero success in complex settings.

  • Problem

    VLA models directly control physical robots, yet security evaluations have been scarce and attacks designed for simpler systems do not directly transfer to their multimodal, diffusion-based, temporal architectures.

    Source: §I · p. 1

  • Method

    The paper uses a universal textured sphere and a multi-level attack framework to disrupt trajectory planning, task execution, and motion control.

    Source: §I · pp. 1–2

  • Results

    Across 13 tasks, the adversarial object reduced average task success rates by 31.2%-39.9% for Pi0 and RDT, with near-zero success in complex settings.

    Source: §I · p. 2

  • Takeaways & Limitations

    The study reports vulnerability to physical adversarial objects across recent VLA models and successful transfer from simulation to real-world scenarios.

    Source: §V · p. 7

Abstract

from arXiv · show

Recently, Vision-Language-Action (VLA) models have revolutionized robotic manipulation by seamlessly integrating visual perception, language understanding, and action generation in an end-to-end learning framework. However, since these models are designed to interact directly with the physical world and humans, their security is critical, and even small vulnerabilities can lead to catastrophic failures. In this work, we propose the Universal Adversarial Object, a sphere with optimized surface texture that significantly degrades task success rates when placed within the robot's field of view. Specifically, our approach introduces a multi-level attack framework that jointly disrupts trajectory planning, task execution, and action control. We validate our method in both simulated and real-world robotic settings. Experimental results demonstrate that the adversarial object reduces the average task success rates by 31.2%-39.9% for two representative VLA models (Pi0 and RDT), with success rates dropping to near zero in complex scenarios. Index Terms--Vision-Language-Action models, adversarial attack, robotic security, universal adversarial object

I. INTRODUCTION

VLA models directly control physical robots, yet their security has received little systematic study, and attacks designed for simpler robotic or vision systems do not transfer directly. The paper introduces a multi-level universal-object attack and reports substantial success-rate reductions in simulation, alongside validation in real-world settings.

I. INTRODUCTION

  • VLA security research is scarce, while their direct control of physical robots interacting with objects and humans raises security risks.The introduction says most prior work focused on performance improvements rather than security vulnerabilities.Source: §I · p. 1
  • The attack is illustrated by adding an adversarial object to the robot status, image, and instruction inputs, turning a successful grasp into a failed attempt.The paper also characterizes its attack as a systematic analysis of physical vulnerabilities in mainstream VLA models.Source: Figure 1 · p. 1
  • Physical attacks on simpler robotic and vision systems cannot be directly transferred to VLA models because of multimodal fusion, diffusion-based generation, and temporal decision-making.These architectural features motivate attack strategies tailored to VLA models.Source: §I · p. 1
  • A universal adversarial object—a 3cm textured sphere—targets trajectory planning, task execution, and motion control while remaining physically realizable and visually inconspicuous.The reported effects include trajectory deviations, oscillations, and task failures.Source: §I · pp. 1–2
  • The attack reduces average task success rates by 31.2%-39.9% for Pi0 and RDT, with success rates near zero in complex environments.The authors also report validation in simulation and on real-world robotic platforms.Source: §I · p. 2

II. RELATED WORKS

Prior VLA work developed end-to-end models and increasingly joint action-generation approaches, while security analysis remained limited. The paper also situates VLA inputs and outputs in a robotic control setting.

II. RELATED WORKS

  • VLA research progressed from hierarchical frameworks using VLM perception and reasoning – to joint training that treats actions as an additional modality [7]–[11].These approaches aim to generalize across complex manipulation tasks.Source: §II · p. 2
  • Pi0 unifies manipulation tasks using large-scale demonstrations, while RDT uses diffusion models to generate trajectories through iterative denoising.Existing work on these models largely emphasizes capabilities rather than systematic security analysis.Source: §II · p. 2
  • Physical adversarial patches and objects have deceived vision models, [24],, but VLA architectures introduce attack surfaces that traditional methods do not address directly.Earlier robotic studies targeted imitation learning and reinforcement learning; Wang et al. [25] examined only specific VLA models and tasks.Source: §II · p. 2

A. Preliminary

  • At each timestep, the VLA receives language instructions, multi-view images, and proprioceptive state, then predicts an action sequence and task-completion indicator.The described platform is a 7-DoF dual-arm Aloha robot, with actions represented by relative position and rotation changes plus a binary gripper state.Source: A. Preliminary · pp. 2–3

B. Problem Definition

The paper defines a universal visual perturbation as a textured patch mapped onto a sphere in the robot workspace, intended to disrupt execution without changing model parameters, instructions, or robot state. Its broader attack framework jointly targets multiple levels of VLA decision-making.

B. Problem Definition

  • The universal adversarial object is formed by projecting a visual patch onto a sphere and placing it in the robot workspace, where it appears in camera views.The attack leaves model parameters, language instructions, and robot states unchanged.Source: B. Problem Definition · p. 3
  • The optimization objective seeks an adversarial trajectory that deviates from the clean trajectory and induces task failure, using losses for trajectory deviation, task failure, action perturbation, and visual naturalness.The patch is optimized through black-box access to model outputs rather than internal parameters or gradients.Source: B. Problem Definition · p. 3

C. Multi-level Attack Framework for VLA

  • The framework jointly attacks pixel-level perception, task-level goal understanding, and trajectory-level action planning for comprehensive adversarial impact.These levels reflect the progression from visual perception through goal understanding to action planning.Source: C. Multi-level Attack Framework for VLA · p. 3

1) Trajectory Deviation Attack:

The trajectory attack concentrates deviations at later, precision-critical stages of manipulation. The task-failure attack addresses the difficulty of directly optimizing binary success in diffusion-based VLA models.

1) Trajectory Deviation Attack:

  • Later timesteps receive exponentially increasing weight, concentrating trajectory-deviation attack energy on critical decision moments and late-stage precision control.The weight function is w_t = exp(α·t/T), with α controlling its growth rate.Source: 1) Trajectory Deviation Attack · p. 3

2) Task Failure Attack:

  • Binary success and inaccessible internal reward signals motivate a contrastive-inspired task-failure loss for diffusion-based VLA models.The success rate is the primary metric, but its discrete nature prevents direct gradient optimization.Source: 2) Task Failure Attack · p. 3
  • The loss targets originally successful tasks that fail under attack and penalizes unintended improvements, with β > 1 reflecting the attack's directional nature.The two terms address attack-induced failure and cases where failed tasks become successful.Source: 2) Task Failure Attack · p. 3

3) Action Perturbation Attack:

The action-perturbation attack targets directional changes between adjacent actions, while regularization constrains texture appearance and spatial variation.

3) Action Perturbation Attack:

  • The action-perturbation loss maximizes directional differences between adjacent timesteps to generate oscillatory patterns, perturbing direction rather than magnitude under velocity constraints.The directional difference ranges from 0 for linear action to 2 for a 180-degree reversal.Source: 3) Action Perturbation Attack · p. 3

4) Regularization Constraints:

  • Regularization constraints support physical realizability and inconspicuousness, while the infinity norm limits maximum pixel intensity within the standard RGB range.The passage introduces regularization constraints but does not specify their coefficient here.Source: 4) Regularization Constraints · p. 3
  • Total-variation regularization penalizes adjacent pixel differences, promoting smooth textures that look natural, reduce detection risk, tolerate printing errors and lighting, and avoid high-frequency local optima.These are the stated benefits of spatial smoothness.Source: 4) Regularization Constraints · p. 3

5) Overall Optimization Objective:

The overall objective combines four losses targeting different aspects of robot behavior, with task failure weighted most heavily. Optimization uses black-box gradient estimation to work within limited query budgets.

5) Overall Optimization Objective:

  • Task failure receives the highest loss weight (λ2 = 0.45), followed by trajectory deviation (λ1 = 0.30), action perturbation (λ3 = 0.20), and regularization (λ4 = 0.05).The framework combines four complementary losses, from high-level task failure to low-level motion perturbations, and maps the optimized texture onto a sphere.Source: 5) Overall Optimization Objective · p. 3, Figure 2 · p. 4

D. Optimization Strategy

1) Gradient Estimation:

  • SPSA estimates high-dimensional gradients with two function evaluations because VLA models' internal gradients are inaccessible; its step size decreases from exploration to fine-tuning.The perturbation-vector elements are independently sampled from a Bernoulli distribution.Source: 1) Gradient Estimation · p. 4

2) Data Augmentation:

Data augmentation is used to maintain attack effectiveness as camera viewpoints and object appearance change. The optimization alternates across models and tasks, filtering out tasks that initially fail.

2) Data Augmentation:

  • Augmentations address appearance changes and occlusion as camera viewpoints shift during manipulation.Random rotation, scale, cyclic shift, contrast, and brightness variation target orientation, distance, sphere rotation, and lighting stability.Source: 2) Data Augmentation · p. 4

3) Optimization Process:

  • For each model-task pair, optimization evaluates clean performance, uses SPSA to update the patch with two evaluations per iteration, and skips tasks that initially fail.Source: 3) Optimization Process · p. 4
  • The optimization is implemented in the RoboTwin simulator with PyOpenGL sphere texture mapping to generate patches intended to work across multiple VLA models and tasks.Source: 3) Optimization Process · p. 4

IV. EXPERIMENT AND ANALYSIS

The experiments evaluate the attack on Pi0 and RDT, using a sequence of setup, main-results, ablation, parameter, multi-view, and real-world analyses. The target models represent Transformer- and diffusion-based approaches to dual-arm manipulation.

IV. EXPERIMENT AND ANALYSIS

  • The experiment section proceeds from setup and main attack results through ablation, parameter, multi-view, and real-world evaluation.The section outline identifies the later analyses but does not report their outcomes.Source: §IV · pp. 4–5

1) Target Models:

  • Pi0 uses a Transformer-based architecture with a PaLI-Gemma backbone and generates continuous action trajectories using Flow Matching.It is selected as one of two target VLA models for dual-arm manipulation evaluation.Source: 1) Target Models · p. 5
  • RDT uses a 1.2-billion-parameter Diffusion Transformer pretrained on 46 multi-robot datasets and fine-tuned with 6,000 dual-arm operation data.The paper selects RDT alongside Pi0 as a target VLA model.Source: 1) Target Models · p. 5

2) Tasks and Environment:

The evaluation uses 13 representative RoboTwin tasks in clean and varied conditions, measures both task completion and trajectory deviation, and tests a fixed optimized texture on a spherical object.

2) Tasks and Environment:

  • The RoboTwin evaluation selects 13 of 50 dual-arm tasks, with Easy and Hard modes differing by added lighting, material, and irrelevant-object variation.The setup uses an Aloha robot with two 7-DoF arms and three fixed cameras; the sphere is randomly placed and visible in at least two views.Source: 2) Tasks and Environment · p. 5

3) Evaluation Metrics:

  • Task success rate measures completed tasks, while Trajectory Discrepancy measures deviations in end-effector positions relative to the clean trajectory.The denominator normalizes by total trajectory length, and each task is evaluated over 10 random seeds with the mean reported.Source: 3) Evaluation Metrics · p. 5

4) Implementation Details:

B. Main Results

A textured sphere substantially reduces VLA task success and perturbs trajectories, with larger effects in difficult scenes. Ablations and parameter analysis show that attack components interact and that patch size and regularization shape the impact.

B. Main Results

  • In Hard mode, success rates fall to 2.1% for RDT and 1.9% for Pi0, versus 25.7%–36.4% residual performance in Easy mode.Precision-demanding tasks show larger drops, including 70% for Pi0-Hard on Grab Roller and 68% for RDT-Hard on Adjust Bottle.Source: B. Main Results · p. 5
  • 31.2%–39.9% average success-rate drops and 19%–27% positional errors are reported across scenarios for the UAO attack on Pi0 and RDT.The results cover 13 representative manipulation tasks; the positional errors also occur in partially successful attempts.Source: B. Main Results · p. 5, Table I · p. 6

C. Ablation Study

  • Removing task-failure loss lowers the success-rate drop from 32.3% to 18.6%; single-loss attacks achieve only 8.2%–18.7% drops.Trajectory-deviation and action-perturbation losses also contribute, while regularization supports physical realizability and visual inconspicuousness.Source: Table II · p. 6, C. Ablation Study · p. 6

D. Patch Parameter Analysis

  • A 3 cm-radius patch maximizes attack impact, while increased regularization can reduce success-rate reduction yet increase trajectory deviation.Smaller patches have limited image coverage, while larger patches may exceed camera boundaries during manipulation.Source: D. Patch Parameter Analysis · p. 6

E. Multi-view Robustness Evaluation

The attack remains effective under limited camera visibility and transfers from simulation to physical-robot trials. The conclusion reports cross-task and cross-model generalization and calls for further attack and defense research.

E. Multi-view Robustness Evaluation

  • Moderate 3 cm patches maximize attack impact, while regularization above γ > 0.1 diminishes success-rate reduction but increases trajectory perturbation.The figure compares patch size and regularization strength against success-rate drop and trajectory deviation.Source: Figure 4 · p. 7
  • With visibility in one camera, success rates decrease by 14.3%–18.6%; visibility in two or three cameras raises the drops to 24.7%–31.2% and 31.2%–39.9%.Wrist-camera visibility yields larger drops than top-camera visibility.Source: E. Multi-view Robustness Evaluation · pp. 6–7

F. Real-world Validation

  • 81.4% for RDT and 82.2% for Pi0: these are the reported success-rate drops relative to simulation in five real-world tasks.The attack used simulation-optimized patches on a physical Aloha robot and remained effective despite printing, lighting, and calibration challenges.Source: F. Real-world Validation · p. 7

V. CONCLUSION

  • The conclusion reports generalization across tasks and models, including transfer from simulation to real-world scenarios.Future work is proposed on additional attack modalities and defenses such as adversarial training and real-time detection.Source: §V · p. 7
Loading 2609.39178v1…