Source-linked AI summary
Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation
Shihan Zhao, Henry Chen, Ya-Chuan Hsu, Minjune Hwang, Siddharth Srikanth, Freddie Liang, Akanksha Saran, Aaquib Tabrez, Stefanos Nikolaidis, Daniel Seita
TL;DR
VLAs often fail under minor environmental shifts despite strong in-distribution performance. Q-DEG searches for diverse, VLA-specific failure environments and fine-tunes on successful rollouts. Across three VLAs, it improves robustness to OOD adversarial environments and reports sim-to-real transfer benefits.
Problem
VLAs can fail under minor test-time changes in object positions, lighting, textures, and camera poses despite strong in-distribution performance.
Source: §I · p. 1Method
Q-DEG uses policy embeddings to guide a Quality Diversity search for high-failure environments, then fine-tunes the VLA on its successful rollouts.
Source: §I · p. 1, B. Feedback Metrics · p. 3, C. Quality Diversity (QD) Optimization · p. 4Results
Across three representative VLA models, Q-DEG improves adversarial-set success rates over the base VLA by up to 43.78% and improves robustness in simulated and real-world manipulation tasks.
Source: §VI · p. 6, §VII · p. 7Takeaways & Limitations
Q-DEG’s diverse adversarial environments and successful-rollout fine-tuning outperform the base VLA and baseline-data fine-tuning in the reported robustness evaluations.
Source: §VII · p. 7Q-DEG uses only successful rollouts for fine-tuning and applies PCA to policy embeddings, which may discard task-relevant information.
Source: §VII · p. 7
Abstract
from arXiv · showhide
A long-standing goal in robotics is to develop generalist robots that can perform tasks across a wide range of settings, flexibly specified with images and text. Vision-Language-Action (VLA) models offer a promising path toward this goal by training on large collections of robot demonstrations. However, while VLAs perform well in-distribution, they often fail under minor test-time shifts, such as changes in object positions, lighting, textures, and camera poses. To address this, we propose Quality Diversity for Diverse Environment Generation (Q-DEG), a method for improving VLA robustness by generating adversarial environments and fine-tuning the VLA on its own successful rollouts. Q-DEG uses Quality Diversity (QD) optimization to find a set of high-failure environments, while promoting diversity using policy embeddings extracted from the VLA. These environments expose vulnerabilities that can be mitigated through self-imitation fine-tuning, thereby improving the VLA's robustness. In simulation, we conduct experiments across three representative VLA models, and our results indicate that Q-DEG improves robustness to out-of-distribution (OOD) adversarial environments compared to the base VLA and VLAs fine-tuned using data collected from two red-teaming baselines. Furthermore, we demonstrate that improving VLA robustness through Q-DEG facilitates sim-to-real transfer. Our anonymous project website is at: https://qd-env.github.io/qdeg/.
I. INTRODUCTION
VLAs can fail under minor environmental shifts, motivating Q-DEG’s search for diverse, VLA-specific failure environments and use of successful rollouts to improve robustness. Unlike prior work focused on evaluation or other forms of red-teaming, Q-DEG searches interpretable simulator parameters and turns its findings into training data.
I. INTRODUCTION
- Minor shifts in object positions, textures, colors, lighting, and camera poses can expose failures despite VLAs’ strong in-distribution performance.Prior work characterized these vulnerabilities, but its hand-designed benchmark environments were not tailored to elicit failures in a specific VLA or improve its robustness.Source: §I · p. 1
- Q-DEG uses QD to find diverse environments that frequently induce VLA failures, then fine-tunes on the VLA’s own successful rollouts.It uses pre-action-layer policy embeddings for diversity, episode-level task success for feedback, and requires no action log-probabilities, supporting autoregressive and flow-based VLAs.Source: §I · p. 1
- The paper evaluates Q-DEG across three VLAs in LIBERO and reports advantages over baselines in environmental diversity, robustness improvement, and sim-to-real transfer.The contributions include self-imitation fine-tuning on successful rollouts saved during the QD search.Source: §I · p. 1
II. RELATED WORKS
- Unlike QD work that primarily characterizes policy behavior using specified diversity measures, Q-DEG searches diverse, challenging scenarios to produce VLA robustness-training data.Its diversity measure is the policy embedding induced by each environment.Source: §II · p. 2
- Q-DEG extends QD red-teaming from instructions to environments, searching simulator parameters for failures while keeping the instruction fixed.Rainbow Teaming generates prompts for LLM red-teaming, while Q-DIG generates instructions that accurately describe tasks but cause VLA failures.Source: §II · p. 2
- Unlike benchmarks whose environment variations are designer-specified for evaluation, Q-DEG searches for VLA-specific failure modes and converts successful rollouts into fine-tuning data.The related-work discussion also distinguishes Q-DEG from unsupervised environment design for reinforcement-learning agents, which does not target modern VLA policies.Source: §II · p. 2
III. PROBLEM FORMULATION AND ASSUMPTIONS
The problem setting assumes a VLA that can succeed on a default task and asks how to find diverse, feasible environments that challenge it, then improve its robustness to environment variation.
III. PROBLEM FORMULATION AND ASSUMPTIONS
- The base VLA maps visual observations and a language instruction to actions and is assumed capable of succeeding on the default task parameterization.The task is defined by an instruction and interpretable environment parameters; success is a binary indicator for a trajectory.Source: §III · p. 2
- Under a fixed instruction, the first problem is to generate diverse environments with high VLA failure rates while excluding environments with 100% failure.The setting prioritizes feasibly adversarial environments near the boundary of the VLA’s capabilities.Source: §III · pp. 2–3
- The second problem is to improve the base VLA’s robustness to environment variations within the considered parameter space.The overview depicts proposing and repairing environments, measuring failures and embeddings, archiving environments, and collecting successful trajectories for fine-tuning.Source: §III · p. 3, Figure 2 · p. 3
IV. METHOD: Q-DEG
Q-DEG proposes task-consistent simulator variations, evaluates the VLA in each environment, archives diverse adversarial cases, and fine-tunes on successful rollouts. Environment generation perturbs interpretable parameters while repairing proposals that would change task semantics.
IV. METHOD: Q-DEG
- Q-DEG’s pipeline proposes and repairs environments, measures feasibility-aware failure rates and policy embeddings, archives diverse adversarial cases, and fine-tunes on successful rollouts.The search uses the VLA’s rollouts in proposed environments to produce its feedback signals.Source: §IV · p. 3
A. Environment Generation
- Each batch perturbs object placements, camera pose, light-source position, table-background RGB, and reflective-specular RGB across a 25-dimensional continuous search space.The selected axes follow common policy-robustness variations.Source: A. Environment Generation · p. 3
- An MIQP repair enforces semantic constraints while minimizing edits to a candidate environment.For example, it can move objects minimally to preserve a task relation such as a bowl being close to a plate; repair is unnecessary for perturbations that do not affect task semantics.Source: A. Environment Generation · p. 3
B. Feedback Metrics
Q-DEG evaluates proposed environments with a feasibility-aware failure rate and policy embeddings that represent the VLA’s initial processing of each environment. It projects embeddings to a lower-dimensional space before QD search.
B. Feedback Metrics
- The empirical failure rate is set to zero when all rollouts fail, so prioritized environments retain successful trajectories for later self-imitation.Otherwise, the measure is one minus the fraction of successful rollouts.Source: B. Feedback Metrics · p. 3
- Policy embeddings are the VLA’s pre-action-layer activations at rollout start, mean-pooled across rollouts to represent its initial processing of each environment.The method uses the final hidden layer before the action decoder.Source: B. Feedback Metrics · p. 3
- Before QD search, Q-DEG projects high-dimensional policy embeddings into a lower-dimensional space using linear PCA.The passage motivates this projection by the curse of dimensionality in QD algorithms.Source: B. Feedback Metrics · p. 4
C. Quality Diversity (QD) Optimization
Q-DEG uses a quality-diversity archive to search for environments that cause high VLA failure rates across diverse policy-embedding regions. CMA-ES is guided by threshold-relative feedback, with repair penalties and archive-based restarts supporting the search.
C. Quality Diversity (QD) Optimization
- Q-DEG’s archive rewards high-failure environments across distinct policy-embedding regions, counting only the highest failure rate in each occupied cell.The archive is a grid over the embedding space, with each cell storing environments whose policy embeddings fall within that region.Source: C. Quality Diversity (QD) Optimization · p. 4
- CMA-ES samples candidate environment parameters and adapts its distribution using feedback on each candidate’s contribution to the archive objective.The natural incumbent-relative signal can be sparse because failure rates are estimated from few rollouts.Source: C. Quality Diversity (QD) Optimization · p. 4
- Threshold-relative improvement scores give CMA-ES denser feedback than sparse, frequently tied improvements over cell incumbents.Each candidate’s score is its failure rate minus its cell threshold; thresholds rise gradually as high-failure candidates are observed.Source: C. Quality Diversity (QD) Optimization · p. 4
- Q-DEG subtracts repair edit distance from feedback and restarts a stagnant CMA-ES search around an archive cell’s highest-failure environment.The restart uses that environment as a stepping stone within its embedding-space region.Source: C. Quality Diversity (QD) Optimization · p. 4
D. Self-Imitation Fine-Tuning
Q-DEG fine-tunes the base VLA through behavior cloning on its own successful trajectories from adversarial environments. It selects one short successful trajectory per eligible environment and adds original-task data to mitigate forgetting.
D. Self-Imitation Fine-Tuning
- The fine-tuning dataset uses the shortest successful trajectory from each environment with 0 < estimated failure rate < 1; fully successful or fully failing environments are excluded.The method excludes longer demonstrations from the same environment because multiple demonstrations can lead to action divergence that undermines imitation learning.Source: D. Self-Imitation Fine-Tuning · p. 4
- Q-DEG adds 10% of the VLA’s original default-task training data to the dataset to mitigate forgetting, then fine-tunes with standard behavior cloning.Source: D. Self-Imitation Fine-Tuning · p. 4
- The authors state that Q-DEG’s environmental diversity yields trajectory diversity in the dataset, helping prevent behavior collapse during self-imitation.Source: D. Self-Imitation Fine-Tuning · p. 4
V. EXPERIMENTS
The experiments ask whether Q-DEG generates diverse adversarial environments, improves robustness to OOD adversarial environments, and facilitates sim-to-real transfer. Two environmental red-teaming baselines provide comparisons for isolating the role of QD optimization.
V. EXPERIMENTS
- The experiments are designed to test Q-DEG’s environment diversity, robustness to OOD adversarial environments, and relevance to sim-to-real transfer.Source: §V · p. 4
A. Baselines
- Randomization samples environments from fixed Gaussian distributions centered on each LIBERO-Spatial task’s default parameters, without adapting distributions or performing QD search.Source: A. Baselines · p. 4
- CMA-ES adapts environment distributions to optimize failure rate alone, without QD’s diversity incentives; both baselines share a 25-dimensional parameter space and MIQP invalid-object-placement repair.Q-DEG’s initial distribution is centered on default LIBERO-Spatial parameters; its search uses four rollouts per environment, batches of eight, and a 100×100 archive over the top two PCA components.Source: A. Baselines · pp. 4–5
B. Simulation Experiments
The simulation experiments use LIBERO-Spatial and three VLA architectures to evaluate red-teaming diversity and fine-tuning robustness. They compare generated environments under a common rollout budget and assess base and fine-tuned models on default tasks and held-out environments.
B. Simulation Experiments
- Evaluation uses 10 language-conditioned LIBERO-Spatial pick-and-place tasks with π0-FAST, OpenVLA-OFT, and π0.5.Source: B. Simulation Experiments · p. 5
- Under a shared 3,200-rollout budget, red-teaming is assessed by the count of feasibly adversarial environments and their average pairwise DINOv2 embedding distance.Feasibly adversarial environments have estimated failure rates between zero and one; the distance uses first-step overview images.Source: B. Simulation Experiments · p. 5
- For robustness evaluation, each method’s successful trajectories are used for 10,000-step behavior-cloning fine-tuning, with 20% of adversarial environments held out.Success is evaluated on default tasks and held-out environments; trials per task are 50 for default tasks and four for held-out environments.Source: B. Simulation Experiments · p. 5
C. Real-World Experiments
Q-DEG finds more diverse adversarial environments and improves robustness over baseline tuning. In real-world bowl-grasp tasks, its fine-tuned VLA reaches toward target bowls more consistently.
C. Real-World Experiments
- The real-world evaluation tested two bowl-grasp tasks, using 20 trials per VLA-task combination after simulated red-teaming and additional fine-tuning.Trials were interleaved between the tested VLAs to control external-factor bias.Source: C. Real-World Experiments · p. 5
VI. RESULTS
- Across three VLA models, Q-DEG raises adversarial-set success over the base by up to 43.78% and often matches or beats baseline tuning on its own evaluation sets.Table I reports success rates across default and adversarial evaluation sets.Source: Table I · p. 6, §VI · p. 6
- Q-DEG finds more diverse adversarial environments across VLAs; for robust π0.5, that diversity also yields more feasible environments than failure-maximizing CMA-ES.CMA-ES tends to cluster around a few failure-inducing configurations, while diverse environments can serve as stepping stones toward niche configurations.Source: §VI · pp. 5–6, Figure 3 · p. 6
- On real-world JACO bowl tasks, Q-DEG fine-tuning sends the gripper more consistently toward target bowls, while base and baseline-tuned policies cluster at incorrect locations.Figure 4 marks successful trials in green and failed trials in red.Source: §VI · p. 6, Figure 4 · p. 6
- Adversarial fine-tuning slightly lowers default LIBERO-Spatial success for every method, but Q-DEG degrades less than the baselines.Source: §VI · p. 6
VII. CONCLUSION AND LIMITATIONS
Q-DEG red-teams VLAs with Quality Diversity search and fine-tunes them on their own successful rollouts. The authors report improved robustness, while identifying limits in trajectory use and policy-embedding dimensionality reduction.
VII. CONCLUSION AND LIMITATIONS
- Q-DEG searches for diverse, feasibly adversarial environments and fine-tunes the VLA on its own successful rollouts to improve robustness.Source: §VII · p. 7
- Fine-tuning uses successful rollouts only, leaving failed trajectories—and their information about actions to avoid—unused.Source: §VII · p. 7
- PCA before discretization-based CMA-MAE search may discard task-relevant information from policy embeddings.The authors suggest discretization-free QD as a possible route to diversification in the full embedding space.Source: §VII · p. 7
APPENDIX I POLICY EMBEDDING EXTRACTION AND COMPRESSION
The appendix describes how the three VLAs’ policy embeddings are extracted and compressed for QD search, and reports results for a uniform-sampling baseline. The first two principal components explain approximately 50% of embedding variance, while uniform sampling yields few feasibly adversarial environments for the weaker VLAs.
APPENDIX I POLICY EMBEDDING EXTRACTION AND COMPRESSION
- Policy embeddings come from model-specific activations preceding action generation, with extraction tailored to each VLA’s architecture.π0-FAST and OpenVLA-OFT activations are averaged across decoding or action dimensions; π0.5 activations are extracted before velocity prediction using a separate zero-initialized flow evaluation.Source: Appendix I · p. 7
- PCA compression addresses the prohibitively large QD archive required to cover high-dimensional raw embedding spaces.The paper uses the top M = 2 principal components for the low-dimensional subspace.Source: Appendix I · p. 7
- The first two principal components explain approximately 50% of policy-embedding variance, while subsequent components contribute substantially less.The method projects raw embeddings onto the top two components, despite discarding substantial information.Source: Appendix I · p. 7
APPENDIX II UNIFORM SAMPLING BASELINE
- Uniform sampling produced few feasibly adversarial environments for π0-FAST and OpenVLA-OFT, and its π0.5 checkpoint was only comparable to rand.After 3,200 rollouts, counts were 0±0, 29.0±7.9, and 179.3±13.5 for π0-FAST, OpenVLA-OFT, and π0.5, respectively.Source: APPENDIX II UNIFORM SAMPLING BASELINE · p. 7