PaperlayerView exact revision on arXiv

Source-linked AI summary

Phantom Menace: Exploring and Enhancing the Robustness of VLA Models Against Physical Sensor Attacks

Xuancun Lu, Jiaxiang Chen, Shilin Xiao, Zizhi Jin, Zhangrui Chen, Hanwen Yu, Bohan Qian, Ruochen Zhou, Xiaoyu Ji, Wenyuan Xu

arXiv:2511.10008v2cs.ROcs.AI

TL;DR

Physical sensor attacks against VLA models remain underexplored despite their reliance on cameras and microphones. The paper introduces Real-Sim-Real to simulate and validate eight attack types, finds broad vulnerability patterns, and proposes adversarial training that improves robustness while largely preserving clean performance.

  • Problem

    Physical sensor attacks against VLA models are underexplored, while existing security work primarily manipulates inputs digitally rather than through physical signals.

    Source: Introduction · pp. 1–2

  • Method

    Real-Sim-Real automatically simulates six camera attacks and two microphone attacks, evaluates VLA models across tasks and intensities, and validates selected attacks on real robotic systems.

    Source: Introduction · p. 2

  • Results

    VLA models are highly vulnerable to physical sensor attacks, while adversarial training improves robustness and can increase moderate-attack performance by around 60% for OpenVLA.

    Source: Conclusion · p. 7, Adversarial Training Defense · p. 7

  • Takeaways & Limitations

    Physical sensor robustness should be evaluated before real-world VLA deployment, with systematic benchmarks and mitigation strategies needed for security-critical applications.

    Source: Introduction · p. 2, Conclusion · p. 7

    The attack goal and evaluation scope concern inducing task failure through physical attacks on cameras or microphones in the specified robotic setting.

    Source: Attack Goal · p. 3, Threat Model · p. 3

Abstract

from arXiv · show

Vision-Language-Action (VLA) models revolutionize robotic systems by enabling end-to-end perception-to-action pipelines that integrate multiple sensory modalities, such as visual signals processed by cameras and auditory signals captured by microphones. This multi-modality integration allows VLA models to interpret complex, real-world environments using diverse sensor data streams. Given the fact that VLA-based systems heavily rely on the sensory input, the security of VLA models against physical-world sensor attacks remains critically underexplored. To address this gap, we present the first systematic study of physical sensor attacks against VLAs, quantifying the influence of sensor attacks and investigating the defenses for VLA models. We introduce a novel "Real-Sim-Real" framework that automatically simulates physics-based sensor attack vectors, including six attacks targeting cameras and two targeting microphones, and validates them on real robotic systems. Through large-scale evaluations across various VLA architectures and tasks under varying attack parameters, we demonstrate significant vulnerabilities, with susceptibility patterns that reveal critical dependencies on task types and model designs. We further develop an adversarial-training-based defense that enhances VLA robustness against out-of-distribution physical perturbations caused by sensor attacks while preserving model performance. Our findings expose an urgent need for standardized robustness benchmarks and mitigation strategies to secure VLA deployments in safety-critical environments.

Introduction

VLA systems rely on sensory inputs yet remain underexplored against physical sensor attacks. This paper introduces Real-Sim-Real evaluations and adversarial training to quantify vulnerabilities and improve robustness.

Motivation

  • Physical sensor attacks expose a gap in VLA security research, which has mainly studied digitally manipulated inputs rather than physical signals.Digital attacks may not capture vulnerabilities arising during real-world physical interactions.Source: Introduction · p. 1

Framework

  • The Real-Sim-Real framework evaluates eight attacks—six targeting cameras and two targeting microphones—using physical signals such as ultrasound, lasers, and electromagnetic waves.It simulates attack physics at three intensities and evaluates four VLA models across four datasets.Source: Figure 1 · p. 1, Introduction · p. 2

Findings

  • VLA models show inherent vulnerability to physical sensor attacks, with susceptibility varying by dataset, attack modality, and model architecture.Simulation-selected parameters are also tested on real VLA systems to validate the findings.Source: Introduction · p. 2

Defense

  • Adversarial training improves robustness to out-of-distribution sensor perturbations while preserving performance on clean datasets.The defense mixes attack data into training after initial clean-data training.Source: Introduction · p. 2

Contributions

  • The framework bridges digital attack simulation and resource-intensive physical experiments while supporting large-scale robustness evaluation and real-world validation.The authors identify systematic evaluation and defense as contributions for safer VLA deployment.Source: Introduction · p. 2

Related Work

Prior VLA security work emphasizes digital attacks, while robustness studies mainly test simulated scenario changes and sensor-attack studies isolate individual modules. This paper targets physical sensor attacks against complete embodied VLA systems.

Attacks on VLA Models

  • Existing VLA security studies examine adversarial patches, text jailbreaks, and backdoors, whereas this work studies physical sensor attacks.The paper positions itself as the first exploration of physical sensor attacks on VLA models.Source: Attacks on VLA Models · p. 2

VLA Robustness Evaluation

  • VLA robustness benchmarks mainly evaluate in-distribution scenario changes such as blur, noise, lighting, obstacles, camera poses, and unseen objects.This work instead examines out-of-distribution robustness from the sensor-attack perspective.Source: VLA Robustness Evaluation · p. 2

Physical Sensor Attacks

  • Prior sensor-level attacks typically study individual sensing modules through resource-intensive physical experiments rather than complete embodied AI systems.That experimental focus makes large-scale evaluation difficult.Source: Physical Sensor Attacks · p. 2

Background

VLA models convert visual and audio observations into robot actions through a vision-language model and action decoder. Their cameras and microphones transform physical signals into digital inputs that can be disrupted by several attack modalities.

Vision-Language-Action

  • A VLA receives camera images and microphone audio, uses a VLM to extract action tokens, and decodes them into physical robot actions.Audio instructions are commonly converted to text through ASR before action generation.Source: Vision-Language-Action · pp. 2–3, Figure 2 · p. 3

Vision-Language Model.

  • The VLM combines visual and textual embeddings through an LLM backbone to generate action tokens.Visual encoders may include CNNs, ViT, SigLIP, or DinoV2, while the LLM backbone converts multimodal embeddings into actions.Source: Vision-Language Model · p. 3

Action Decoder.

  • Action decoders map action tokens to robot actions through direct decoding, MLP mapping, diffusion decoding, or flow matching.These mechanisms differ in whether they autoregressively generate actions, map hidden states, denoise sequences, or learn distributional velocity fields.Source: Action Decoder · p. 3

Camera and attacks.

  • Cameras convert optical signals into electrical and digitized images through a transducer and image signal processor, making measurements susceptible to lasers and acoustic signals.Some cameras also use IMUs for stabilization.Source: Camera and attacks · p. 3

Microphone and attacks.

  • Microphones convert acoustic signals into electrical, filtered, and digitized signals, but ultrasonic signals and lasers can disrupt this process.The microphone pipeline consists primarily of an acoustic transducer and signal-processing circuit.Source: Microphone and attacks · p. 3

Methodology

The methodology models black-box attackers injecting physical signals into cameras or microphones, then simulates and reproduces six camera and two microphone attack types across intensities and hardware setups.

Threat Model

  • Attackers inject physical signals into cameras or microphones to induce unexpected or targeted incorrect robot actions, while lacking digital access to the VLA.The threat model covers robotic systems receiving voice instructions and camera-based environmental perception.Source: Attack Goal · p. 3, Threat Model · p. 3, Attacker Capability · p. 3

Model Knowledge.

  • Attackers have black-box access and do not know the model architecture, training data, parameters, sensor types, or sensor algorithms.This includes uncertainty about ASR and image stabilization systems.Source: Model Knowledge · p. 3

Camera-attack Design

  • The framework simulates eight attacks with six camera targets and two microphone targets across laser, light, acoustic, and electromagnetic signals at varying intensities.The attack instances are parameterized by intensity for systematic evaluation.Source: Figure 3 · p. 4

Microphone-attack Design

  • Microphone attacks either saturate the sensor with inaudible ultrasonic noise for voice denial of service or inject malicious instructions through modulated laser or ultrasound signals.Recorded malicious responses are superimposed on original audio to simulate physical attacks.Source: Microphone-attack Design · p. 4

Camera-attack Design

  • Camera attacks manipulate incoming light or sensor processing through laser blinding, projected images, color strips, electromagnetic interference, truncation, and ultrasound-induced blur.The simulations vary attack-specific parameters such as pattern weights, wavelengths, stripe geometry, truncation position, and blur amplitude.Source: Camera-attack Design · pp. 4–5
  • The real-world setup uses a Franka Panda with global and wrist cameras, a microphone, and dedicated EMI, projection, laser, and ultrasound attack platforms.The system is adapted to the environment using one hour of teleoperated block pick-and-place data.Source: Figure 4 · p. 5

Experiments

Experiments evaluate four fine-tuned VLA models across Libero tasks under simulated and physical sensor attacks, then test adversarial training as a defense. Results show attack- and architecture-dependent failures, real-world consequences, and improved robustness after training with attack data.

Simulator and Datasets.

Robustness Evaluation in the Simulator

  • Without attacks, the fine-tuned models reach up to 90% TSR on simpler Libero-Spatial and Libero-Object tasks, establishing strong benign baselines before robustness testing.OpenVLA-OFT retains good performance on long-horizon tasks.Source: Robustness Evaluation in the Simulator · p. 6
  • All evaluated VLA models are vulnerable to simulated sensor attacks, with degradation varying by architecture, attack type, intensity, and task horizon; strong attacks and long-horizon tasks can cause catastrophic collapse.The simulator evaluates robustness using TSR across multiple attack modalities and Libero task suites.Source: Evaluation metrics · p. 5, Robustness Evaluation in the Simulator · p. 6, Table 2 · p. 6
  • Camera attacks that disrupt visual information, including LB, ET, and UB, are more damaging than perturbations that mainly interfere with attention, while voice attacks depend on dataset-task correspondence and model semantics.OpenVLA and OpenVLA-OFT are especially vulnerable to voice spoofing, whereas pi0 and pi0-fast show greater resilience to visual attacks.Source: Robustness Evaluation in the Simulator · pp. 6–7

Robustness of VLA Models in Real World

Adversarial Training Defense

  • Adversarial training improves robustness to moderate-intensity sensor attacks while preserving clean-data performance, with OpenVLA gaining around 60% at most despite an average clean-data decline of approximately 3%.The defense mixes attack data into clean-data training and targets out-of-distribution physical perturbations.Source: Table 3 · p. 6, Adversarial Training Defense · p. 7

Conclusion

The paper identifies physical sensor attacks as a significant security threat to real-world VLA deployment. It introduces a Real-Sim-Real evaluation framework and adversarial training to assess and improve robustness.

  • Physical sensor attacks can severely degrade VLA performance, producing erroneous or hazardous behaviors in real-world applications.Source: Conclusion · p. 7
  • The Real-Sim-Real framework automatically simulates physics-based sensor attacks and validates them on real robotic systems.Source: Conclusion · p. 7
  • Adversarial training mitigates physical sensor attacks and supports secure VLA deployment.Source: Conclusion · p. 7

Appendix A: Sensor Attack Simulation Implementation Details

The appendix documents the implementation logic used to simulate six camera sensor attacks. It specifies each attack’s core algorithm and key parameters based on the provided code.

  • The appendix covers implementation details for six simulated camera sensor attacks.Source: Appendix A · p. 9
  • Each attack is described through its core algorithm.Source: Appendix A · p. 9
  • Key attack parameters are defined from the implementation code.Source: Appendix A · p. 9

Laser Blinding

The laser blinding attack simulates sensor overload from a bright laser by blending a laser pattern with the original image. Its blending factor controls attack intensity.

  • The laser blinding attack simulates a bright laser overwhelming the sensor through image blending.Source: Laser Blinding · p. 9
  • The attacked image is formed by combining the original image with a laser pattern, clipping pixel values to [0, 255].Source: Laser Blinding · p. 9
  • The blending factor α ranges from 0 to 1, with higher values assigning more weight to the laser pattern.Source: Laser Blinding · p. 9

Light Projection

The light projection attack simulates a malicious pattern projected onto a scene or camera lens. It overlays a watermark with adjustable transparency and position.

  • The light projection attack simulates projecting a malicious pattern onto the scene or directly onto the camera lens.Source: Light Projection · p. 9
  • The attack overlays a specified watermark image onto the original image with adjustable transparency.Source: Light Projection · p. 10
  • Transparency t controls opacity, while xoff and yoff specify horizontal and vertical placement offsets in pixels.Source: Light Projection · p. 10

Laser Color Strip

The Laser Color Strip attack exploits camera rolling-shutter behavior to inject colored horizontal bands, with intensity shaped by a 2D Gaussian-like function. Its implementation modifies RGB channels using channel-specific color percentages and clips the resulting image values.

Laser Color Strip

  • A rolling-shutter-based attack injects colored stripes into a horizontal image band.The simulated band’s color intensity is determined by a 2D Gaussian-like function.Source: Laser Color Strip · p. 10
  • The implementation adds channel-weighted perturbations to the original red, green, and blue values before clipping pixels to [0, 255].The attack uses channel percentages ρR, ρG, and ρB and applies the perturbation across the selected band.Source: Laser Color Strip · p. 10
  • The intensity function scales the attack by strength and varies it across pixel coordinates around an effect center.The function uses pixel coordinates, center coordinates, standard deviations, and coordinate correlation.Source: Laser Color Strip · p. 10

EM Color Strip

Three sensor attacks simulate distinct camera failures: electromagnetic color corruption, frame-buffer truncation, and IMU-induced motion blur. Together, they alter color decoding, image continuity, or motion compensation through parameterized image transformations.

EM Color Strip

  • The EM Color Strip attack simulates EMI transmission errors by applying a fixed incorrect color transformation to alternating horizontal stripes.The number of stripes controls how the image is divided for the transformation.Source: EM Color Strip · p. 10

EM Truncation

  • The EM Truncation attack simulates corrupted frame stitching by deleting a middle image section and appending rows from the original frame’s end.The truncation ratio determines both the starting position and the number of replaced lines.Source: EM Truncation · pp. 10–11

Ultrasound Blur

  • The Ultrasound Blur attack simulates camera-IMU resonance that causes incorrect motion compensation and blurred images.Each output pixel averages values along a motion path incorporating rotation, displacement, and scaling.Source: Ultrasound Blur · p. 11
Loading 2511.10008v2…