Source-linked AI summary
Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models
Jiawei Chen, Simin Huang, Jiawei Du, Shuaihang Chen, Yu Tian, Mingjie Wei, Chao Yu, Zhaoxia Yin
TL;DR
VLA models remain underexplored against physically realizable adversarial attacks, especially attacks attached to manipulated objects. Tex3D introduces FBD and TAAO to optimize adversarial 3D textures end to end within VLA simulation, achieving task failure rates up to 96.7% and exposing vulnerabilities that motivate robustness-aware training.
Problem
VLA robustness to physically realizable adversarial attacks is underexplored, while existing language and 2D visual attacks have limited physical realism or deployment representativeness.
Method
Tex3D uses FBD for differentiable dual-renderer texture optimization and TAAO to emphasize behaviorally critical frames across trajectories and views.
Results
Tex3D consistently achieves high task failure rates across multiple VLA models, manipulation suites, untargeted and targeted settings, reaching 96.7%.
Takeaways & Limitations
Object-level adversarial textures constitute a practical and effective attack surface for embodied agents, underscoring the need for systematic robustness evaluation and comprehensive adversarial training.
Takeaways & Limitations
Current VLA training corpora and benchmarks largely use clean, idealized environments with limited coverage of subtle appearance shifts and adversarial visual variations.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have shown strong performance in robotic manipulation, yet their robustness to physically realizable adversarial attacks remains underexplored. Existing studies reveal vulnerabilities through language perturbations and 2D visual attacks, but these attack surfaces are either less representative of real deployment or limited in physical realism. In contrast, adversarial 3D textures pose a more physically plausible and damaging threat, as they are naturally attached to manipulated objects and are easier to deploy in physical environments. Bringing adversarial 3D textures to VLA systems is nevertheless nontrivial. A central obstacle is that standard 3D simulators do not provide a differentiable optimization path from the VLA objective function back to object appearance, making it difficult to optimize through an end-to-end manner. To address this, we introduce Foreground-Background Decoupling (FBD), which enables differentiable texture optimization through dual-renderer alignment while preserving the original simulation environment. To further ensure that the attack remains effective across long-horizon and diverse viewpoints in the physical world, we propose Trajectory-Aware Adversarial Optimization (TAAO), which prioritizes behaviorally critical frames and stabilizes optimization with a vertex-based parameterization. Built on these designs, we present Tex3D, the first framework for end-to-end optimization of 3D adversarial textures directly within the VLA simulation environment. Experiments in both simulation and real-robot settings show that Tex3D significantly degrades VLA performance across multiple manipulation tasks, achieving task failure rates of up to 96.7\%. Our empirical results expose critical vulnerabilities of VLA systems to physically grounded 3D adversarial attacks and highlight the need for robustness-aware training.
1 Introduction
VLA models perform strongly in manipulation but remain vulnerable to physically grounded adversarial attacks. Tex3D addresses the missing differentiable and trajectory-level optimization path for adversarial 3D textures.
- VLA models map visual observations and language instructions to low-level control signals for complex manipulation tasks.
- Adversarial robustness is critical because attacks can induce incorrect, harmful, or unsafe VLA actions.
- Language attacks manipulate instructions, while 2D visual patches induce failures through perception-level manipulation but depend on viewpoint and pose alignment.
- Object-centric adversarial textures are more naturally integrated into manipulated objects and suitable for embodied interaction and physical deployment.
- 3D texture attacks require differentiable appearance optimization and sustained effectiveness across long trajectories, where frames contribute unequally to decisions.
- Tex3D combines FBD for differentiable texture optimization with TAAO for temporally consistent optimization over critical frames.
- Up to 96.7% task failure rates were achieved across simulation and real-robot evaluations spanning four manipulation task suites.
2 Related Work
Prior VLA attack research mainly targets language instructions or 2D visual inputs. These approaches expose interface and perception vulnerabilities but motivate attacks that are more object-centered and physically integrated.
- VLA models unify visual perception, language grounding, and action generation for robotic control.
- Representative VLA architectures include autoregressive, diffusion-based, and hybrid paradigms, including OpenVLA and π0.
- Existing VLA attacks primarily manipulate language instructions or apply adversarial patches to input images.
- Language-based attacks influence action generation through instruction perturbations, whereas patch-based attacks degrade performance through image perturbations and may transfer across settings.
3 Methodology
Tex3D optimizes adversarial object textures over trajectories and viewpoints within embodied simulation. Its pipeline combines dual-renderer compositing, differentiable texture gradients, and trajectory-aware frame selection.
- 3.1 Problem Formulation: Each simulated object is represented as a textured mesh, and rendered RGB observations drive VLA action prediction from visual input and language.
- Foreground–Background Decoupling: FBD decouples the adversarial object from the static background and aligns MuJoCo and Nvdiffrast renderings for differentiable texture optimization.
- Foreground–Background Decoupling: The composited adversarial observation is processed by the frozen VLA, while gradients from attack objectives flow back to the object texture.
- Trajectory-Aware Adversarial Optimization: TAAO identifies behaviorally critical frames through observation-sequence dynamics and concentrates optimization on them for long-horizon trajectories.
- 3.1 Problem Formulation: Tex3D optimizes adversarial textures jointly across task trajectories and multiple sampled views to sustain action deviation beyond a single observation.
Foreground–Background Decoupling
Tex3D combines a dual-renderer pipeline for differentiable texture optimization with trajectory-aware objectives that emphasize critical moments and constrain perturbations for transferability.
- Foreground–Background Decoupling: FBD keeps MuJoCo unmodified while rendering the target object in Nvdiffrast and relaying synchronized pose, viewpoint, and lighting parameters.The resulting gradients update the adversarial texture, which is then applied back to the MuJoCo object.
- Foreground–Background Decoupling: Scene compositing replaces the target object’s pixels with the differentiably rendered adversarial foreground while retaining the simulator-rendered background.This preserves a gradient chain from texture parameters through the observation and VLA action loss.
- Trajectory-Aware Adversarial Optimization: TAAO weights frames by latent velocity and acceleration so optimization concentrates on behaviorally critical transitions rather than isolated observations.The method uses normalized temporal feature changes and temperature-scaled frame weights.
- Trajectory-Aware Adversarial Optimization: Vertex-based parameterization restricts texture perturbations to a smooth, low-rank manifold, reducing the search space and improving transferability.The restriction follows from the number of optimized vertices being much smaller than the texture-map dimensionality.
- Attack Objectives: Targeted attacks optimize deviation toward semantically plausible, time-varying alternative actions, while untargeted attacks maximize weighted action deviation across sampled views.The targeted objective redirects the gripper toward an alternative target at each timestep.
- Physical Robustness: EoT complements multi-view optimization by modeling pose, viewpoint, distance, brightness, contrast, and blur variations for physical deployment.Multi-view optimization addresses simulated viewpoint changes, whereas EoT broadens the modeled transformations for sim-to-real transfer.
4 Experiments
Experiments evaluate Tex3D across LIBERO simulation tasks, victim VLA models, geometric variations, defenses, rendering consistency, and perturbation budgets. Tex3D produces high failure rates, remains more robust than 2D patches, and retains effectiveness under defenses and naturalness constraints.
- Experimental Setup: Tex3D is evaluated on four LIBERO task categories using four VLA models, with 50 independent trials per task and average Task Failure Rate as the metric.The evaluated models are OpenVLA, OpenVLA-OFT, 𝜋0, and 𝜋0.5.
- Main Results: 96.7% task failure rate is reached on OpenVLA’s Spatial task under targeted Tex3D attacks, rising from a 15.6% clean-task failure rate.On 𝜋0, the corresponding Spatial-task failure rate rises from 3.5% to 75.9%.
- Main Results: 88.1% and 90.5% average failure rates are achieved on OpenVLA under untargeted and targeted attacks, respectively, versus 24.1% without Tex3D.OpenVLA-OFT increases from 4.7% to 76.0% and 79.3% under the same settings.
- Additional Analysis: Tex3D remains more robust than 2D patch attacks under geometric variation, sustaining 80.8%–88.1% digital and 66.8%–67.6% physical failure rates.The physical-world baselines achieve 40.8%–50.8% for 2D patch and 49.8%–56.6% for 2D patch+EoT.
- Additional Analysis: 86.6%–87.3% failure rates persist across JPEG compression, additive noise, median blur, and bit-depth reduction defenses.Same-step MuJoCo and dual-renderer outputs also show SSIM scores consistently close to 1, with cross-rendering discrepancies of at most 1.2 points across task suites.
- Ablation Studies: L0 naturalness-constrained textures remain highly effective, while stronger perturbation levels from L1 to L3 generally produce higher failure rates and better transferability.L0 uses ε=64/255 with an additional MSE loss; L1–L3 use ε=16/255, 32/255, and 64/255, respectively.
5 Discussion and Conclusion
Tex3D demonstrates that adversarial 3D textures can remain effective across models, settings, geometric variations, and defenses, exposing a practical attack surface for embodied agents. The findings motivate more systematic robustness evaluation and training against physically grounded adversarial interactions.
- The findings highlight the need for systematic robustness evaluation and comprehensive training under persistent, physically grounded adversarial interactions.
- Tex3D consistently achieves high task failure rates under untargeted and targeted attack settings across task suites.
- The attack transfers effectively across VLA model architectures and remains robust under geometric variations and common defenses.
- These results identify object-level adversarial textures as a practical and highly effective attack surface for embodied agents.