VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

Motivation

• Existing benchmarks provide broad manipulation coverage but often use fixed complexity, separate robustness from extrapolation, and overlook safety constraints.These limitations hinder fine-grained analysis of capability degradation across visual, linguistic, and structural challenges.Source: §1 · p. 2

Benchmark

• VLA-Arena quantifies difficulty across orthogonal task structure, language command, and visual observation axes.Language and visual perturbations probe whether models use robust grounding or memorize training patterns.Source: §1 · pp. 2–3
• The task-structure axis contains 170 tasks in 11 suites spanning Safety, Extrapolation, Distractor, and Long Horizon dimensions.Each suite uses three difficulty levels, while language and visual axes provide graded diagnostic perturbations.Source: §1 · pp. 2–3

Framework

• VLA-Arena evaluates both task success and safety through formal task definitions and safety-aware metrics.Its framework includes an end-to-end toolchain from scene modeling or task definition through automated evaluation and standardized fine-tuning datasets.Source: §1 · pp. 2–3

Findings

• Evaluation reveals memorization over generalization, superficial visual perception, and neglected safety constraints across leading VLA architectures.These findings expose gaps between current capabilities and real-world deployment requirements.Source: §1 · p. 3

2. Structured Task Design

• VLA-Arena represents difficulty through task structure plus language and visual axes used as orthogonal probes.This design supports interpretable assessment of capability frontiers rather than a single undifferentiated task difficulty.Source: §2 · p. 3

2.1. Preliminary: Constrained BDDL

• CBDDL extends BDDL with dynamic entities, perturbations, and formal safety constraints for task specification.The framework enables explicit safety predicates and separates safety assessment from task success.Source: §2.1 · p. 3

2.2. Task Structure: Beyond Memorization

• The task-structure levels progress from familiar in-distribution skills at L0 to near-distribution variations at L1 and far-distribution compositional challenges at L2.L2 requires adaptation to novel workflows, object arrangements, environmental complexity, safety constraints, or object categories.Source: §2.2 · pp. 3–4

2.3. Language Command: Semantic Grounding Probes

• Language difficulty increases by substituting semantically related words in progressively more command slots while task structure remains unchanged.Substitutes are selected through WordNet synset proximity, producing W0-W4 command levels.Source: §2.3 · p. 4

2.4. Visual Observation: Visual Grounding Probes

• Visual difficulty follows a cumulative V0-V4 hierarchy from canonical views through lighting, color, viewpoint, and Gaussian-noise perturbations.The hierarchy maps visual failure boundaries and tests invariance to appearance, camera, and sensor changes.Source: §2.4 · p. 4

Evaluation

• Table 2 reports six-model performance across four dimensions and L0-L2, using success rate and cumulative cost for safety tasks.Color coding distinguishes difficulty levels, while zero success rates highlight failures and arrows mark non-increasing trends.Source: Table 2 · p. 5

Safety

• Safety suites test collision avoidance, safe grasping, hazard avoidance, state preservation, and dynamic-obstacle planning across L0-L2.Difficulty increases through more obstacles, hazardous contacts, closer hazards, fuller containers, and more complex obstacle motion.Source: §3 · p. 5

Distractor

• Distractor suites measure selective attention by increasing static clutter or dynamic distractor complexity around target manipulation.The dimension tests whether models filter irrelevant visual cues while maintaining performance in changing environments.Source: §3 · pp. 5–6

Extrapolation

• Extrapolation suites evaluate language-driven compositional reasoning and zero-shot recognition through novel spatial relations, workflows, and unseen objects.Difficulty progresses from familiar combinations to reconfigured tasks and entirely new object categories.Source: §3 · p. 6

Long Horizon

• Long Horizon tasks require chaining mastered atomic skills, with L1 composing two skills and L2 requiring interdependent multi-step workflows.Examples include opening a drawer, placing an object inside, and closing it.Source: §3 · p. 6

4.2. Analysis of Performance and Failure Modes

• Models overfit in-distribution L0 tasks, then degrade severely on L1 and L2 challenges while often accepting high cumulative cost to complete tasks.π0 generally outperforms other models, but severe degradation and safety trade-offs remain across models.Source: §4.2 · p. 7
• Static distractors sharply reduce performance through erroneous grasps, whereas moving distractors are less disruptive, possibly because they lack stable grasp affordances.Source: §4.2 · p. 7
• Language-based skill generalization fails when instructions require novel composition or reordering, with performance approaching zero beyond fixed semantic structures.Source: §4.2 · p. 7

4.3. Diagnosing Semantic and Visual Grounding

• Visual probes reveal shortcut-based grounding: models tolerate photometric shifts but collapse under viewpoint and noise perturbations, with π0 and OpenVLA-OFT retaining partial V4 functionality.Source: §4.3 · p. 7
• VLA policies lose 30.5% at V3 and 50.5% at V4, versus 6.7% for Qwen3-VL-8B, indicating fine-tuning can replace robust visual representations with pixel-specific patterns.Source: §4.3 · pp. 7–8

4.4. Disentangling Memorization from Generalization

• Rank reversals across L0-L2 show that memorization proficiency does not guarantee generalization, while safety remains an independent model capability axis.Source: Table 4 · p. 7, §4.4 · p. 8

4.5. Ablation Study

• Table 5 reports real-robot performance across progressive difficulty, evaluated on a physical Franka Research 3 robot.Source: Table 5 · p. 8
• Invalidating instructions causes a 52-64% collapse from a 79% baseline on VLA-Arena, compared with a 28% drop on LIBERO, supporting stronger language grounding.Source: §4.5 · p. 8, Figure 3 · p. 8

• 60.0% average success on L0 physical tasks drops to 26.6% on L1 and 3.3% on L2, while unsafe behaviors persist.Source: §5 · p. 8
• Real-world lighting and camera shifts produce degradation gradients that mirror simulated visual-perturbation trends.Source: §5 · pp. 8–9
• Physical deployment reproduces simulation’s failures: obstacle collisions, incorrect distractor grasps, lost grounding during composite instructions, and poor extrapolation to novel layouts.Source: §5 · p. 9, Figure 4 · p. 9

• VLA-Arena controls difficulty across task structure, language command, and visual observation, exposing memorization, superficial visual perception, and safety neglect.Source: §6 · p. 9
• The released framework includes an end-to-end toolchain, formal task definitions, and curated datasets intended to support research on generalizable, robust, and safe robotic agents.Source: §6 · p. 9

• VLA-Arena emphasizes safety metrics alongside robustness and generalization to support responsible development of embodied AI.Source: Impact Statement · p. 9
• The paper identifies fostering trust in autonomous systems and their beneficial domestic and industrial integration as a long-term impact.Source: Impact Statement · p. 9

A. Additional Experimental Results and Analysis

• Latest VLAs show strong in-distribution memorization but uniform performance degradation at L1 and L2, especially for Long Horizon and Extrapolation.Qwen2.5-VL-GR00T reaches 116.1 cumulative cost on CautiousGrasp L1, while π0.5 reaches 76.6 on StaticObstacles L2.Source: A. Additional Experimental Results and… · p. 14
• Adding L1 training data improves near-distribution performance but can degrade L2 generalization, showing that data composition does not guarantee extrapolatable skills under a fixed budget.The comparison includes focused +L0, combined +L0&L1, and broad +L0* training schemes.Source: A. Additional Experimental Results and… · p. 14, Table 10 · p. 17
• Models often trade safety for task success: UniVLA achieves 54% SR on StatePreservation L2 at cost 16.4, whereas π0 has 0 SR and cost 0.5 on CautiousGrasp L2.On HazardAvoidance L2, OpenVLA and OpenVLA-OFT incur costs of 15.73 and 14.71.Source: A. Additional Experimental Results and… · pp. 14–15
• Static distractors cause sharper failures than dynamic distractors, with OpenVLA-OFT and SmolVLA reaching 0% SR on StaticDistractors L1 while π0 maintains 70% SR on DynamicDistractors L1.The results indicate a selective-attention failure in cluttered scenes, while dynamic distractor performance decays more gradually.Source: A. Additional Experimental Results and… · p. 15

B. More Details of Related Work

• VLA-Arena extends prior benchmarks by testing task-level generalization through structurally novel tasks involving new semantic goals, instructions, or safety constraints.Its related-work position complements earlier benchmarks emphasizing manipulation breadth, long horizons, lifelong learning, world knowledge, or perceptual robustness.Source: B. More Details of Related Work · pp. 15–16

C. Details of Real-Robot Validation

• On a physical Franka Research 3 robot, performance decreases from L0 to L2 and reproduces simulation failure modes including incorrect grasps, object knockovers, weak extrapolation, and failed skill chaining.Physical lighting and camera shifts also preserve the simulated degradation ordering None →+Light →+L+Cam.Source: C. Details of Real-Robot Validation · p. 17, Table 11 · p. 18
• Evaluation of six additional physical models on Safety confirms that current VLAs fail to respect physical safety boundaries under generalization challenges.The physical study covers OpenPI and StarVLA families and focuses on the Safety dimension.Source: C. Details of Real-Robot Validation · p. 17

D.1. Attention Map Analysis on Generalization

• OpenVLA retains attention to the target plate on unseen L1 tasks, whereas OpenVLA-OFT lacks this fusion, implicating parallel decoding in degraded generalization.The comparison is based on cross-layer attention maps for the instruction to place a bowl on a plate.Source: §D.1 · p. 18, Figure 6 · p. 20
• π0.5 preserves object attention in layers 4–6 across seen and unseen instructions, unlike π0 and π0-FAST, consistent with stronger semantic retention and generalization.The analysis attributes this preservation to large-scale robot pretraining and gradient isolation.Source: §D.1 · pp. 18–19, Figure 7 · p. 21

D.2. Failure Modes in Dynamic Distractors

• Dynamic-distractor failures follow distinct stages: misplaced grasps occur at steps 30–50, failures to hold at steps 50–100, and policy collapse across steps 30–300.Policy collapse is associated with continuous out-of-distribution visual states caused by unpredictable motion.Source: §D.2 · p. 19

E. Constrained Behavior Domain Definition Language

• CBDDL extends BDDL with dynamic entities, visual perturbations, and explicit safety constraints to support more realistic robustness and safety evaluation.Its moving-object block defines autonomous motion independently of robot interaction.Source: E. Constrained Behavior Domain… · p. 19, §E.2 · p. 20

E.1. Preliminary: The BDDL

• BDDL specifies objects, initial literals, and goal conditions without prescribing action sequences, allowing diverse instances and multiple valid solutions.This process-agnostic formulation defines the scope of an activity through initial and goal conditions.Source: §E.1 · p. 19

E.2. Details of Dynamic Object Definition

• CBDDL supports linear, circular, waypoint, and projectile motion primitives with configurable trajectory parameters for moving objects.Waypoint motion interpolates positions linearly and orientations with SLERP, while circular motion uses a fixed pivot and period.Source: §E.2.1 · pp. 22–23

E.2.2. SIMULATION INTEGRATION

• Dynamic objects are driven through MuJoCo mocap joints, physically interact with the robot, and can be frozen after safety violations for reproducible failure analysis.The motion generator updates object poses each physics step and participates in cost evaluation.Source: §E.2.2 · p. 23

E.2.3. EXAMPLE SPECIFICATION

• The example CBDDL specification defines a toy motorbike with linear oscillation over 125 steps, 0.7 meters of travel, and motion along the Y-axis.This example illustrates a dynamic avoidance constraint for the robot.Source: §E.2.3 · p. 23

E.3. Details of Visual Perturbation Definition

• Visual perturbations operate exclusively on camera-based observations through image settings, noise, camera, and random-color blocks controlling enhancement, corruption, viewpoints, and materials.Enhancement includes brightness, saturation, contrast, and temperature; noise supports Gaussian and salt-and-pepper corruption, while viewpoints and scene colors are randomized.Source: §E.3 · p. 23, §E.3.1 · p. 23, §E.3.2 · p. 24

E.4. Details of Safety Constraints Definition

• CBDDL’s optional (:cost ...) block distinguishes successful trajectories from successful-and-safe ones by specifying undesirable simulator states and safety violations.Safety costs are formally listed in Table 13 and complement, rather than replace, the (:goal) condition.Source: Table 13 · p. 24, §E.4 · p. 25

E.4.1. SAFETY PREDICATES

• Safety predicates monitor contacts, distance, force, and object falls, while instantaneous costs accumulate per-step violations and terminal costs are scaled by α = 10.The predicate logic supports And, Or, and Not, enabling complex safety conditions.Source: §E.4.1 · p. 25, §E.4.2 · p. 25

F.1. Language Command Perturbation Principle

• Language perturbation replaces K ∈{1, 2, 3, 4} verbs or nouns with screened synonyms, hypernyms, or hyponyms, sampling original and variant instructions uniformly while keeping vision fixed.The procedure broadens linguistic expressions while preserving task logic and reducing reliance on specific lexical triggers.Source: §F.1 · p. 26

F.2. Visual Observation Perturbation Principle

• Visual randomization progresses through lighting at V1, object-color randomization at V2, camera-pose shifts at V3, and additive Gaussian sensor noise at V4.Lighting adjustment factors are sampled from U(−0.75, 0.75) and color temperature from U(3500, 8500) Kelvin; camera-coordinate offsets are sampled from U(−0.105, 0.105).Source: §F.2 · p. 27

G. VLA-Arena Task Suites Details

• The task suites operationalize capability challenges through graded scenarios including static and dynamic obstacles, cautious grasping, hazard avoidance, state preservation, and static distractors.Difficulty increases through added obstacles, object rotation or relocation, hazardous proximity, filled containers, moving obstacles, or visually similar clutter.Source: §G.1 · p. 27, §G.2 · p. 29, §G.3 · p. 30

G.5. DynamicObstacles

• Dynamic-obstacle tasks require temporal collision avoidance, progressing from stationary obstacles at L0 to linear motion at L1 and complex curvilinear motion with altered object positions at L2.CBDDL freezes a moving obstacle after the first temporal safety violation to standardize post-accident analysis.Source: §E.4.3 · p. 25, §G.5 · p. 33

G.7. DynamicDistractors

• DynamicDistractors progresses from stationary obstacles at L0 to linearly moving distractors at L1 and several distractors in complex curvilinear motion at L2.The suite uses pick-and-place tasks whose distractors become increasingly dynamic across difficulty levels.Source: §G.7 · p. 35

G.8. PrepositionCombinations

• PrepositionCombinations evaluates whether models understand spatial relationships by testing diverse object–relation combinations, recombinations, and familiar-object scenes with new configurations.L0 uses diverse combinations, L1 recombines relationships and objects from L0, and L2 places L0 relationships in new scenes with familiar objects.Source: §G.8 · p. 36

G.9. TaskWorkflows

• TaskWorkflows tests compositional reasoning by recombining familiar objects, destinations, and skills into novel workflows that challenge learned pairings.Its levels progress from canonical pairs (L0), to shuffled pairs (L1), to swapped object-destination roles (L2).Source: §G.9 · p. 37

G.10. UnseenObjects

• UnseenObjects measures zero-shot generalization by replacing trained objects with novel meshes, textures, and related categories across L0-L2.The suite separates specific-object manipulation from same-category appearance changes and new similar categories.Source: §G.10 · p. 38

G.11. LongHorizon

• LongHorizon evaluates temporal composition by increasing from atomic skills to consecutive actions and interdependent multi-skill workflows.This progression targets multi-step planning and chaining of previously mastered abilities.Source: §G.11 · p. 39

H.1. Vision-Language-Action Models

• The evaluated VLAs include autoregressive models such as OpenVLA, UniVLA, and π0-FAST, alongside continuous-action models including π0, OpenVLA-OFT, and SmolVLA.These models represent tokenized, latent-action, flow-matching, regression, and efficiency-oriented approaches.Source: §H.1 · p. 39

H.2. Simulator

• The experiments use RoboSuite with MuJoCo, a modular simulation platform providing reproducible manipulation tasks and standardized robot-learning interfaces.The framework supports diverse tasks and multiple robot platforms while abstracting environment setup.Source: §H.2 · p. 39

L0 Visual

• The cited examples include instructions to pick up and place apples, bananas, eggs, and tomatoes, as well as to open or close layers.Source: L0 Visual · p. 40, L1 Visual · p. 40

L2 Visual

• Higher-level examples require multiple objects, sequential drawer operations, and compound placement workflows rather than isolated actions.These examples include ordering constraints such as opening, picking, placing, and closing within one instruction.Source: L2 Visual · p. 40

H.3.1. OPENVLA TRAINING PARAMETERS

• OpenVLA uses LoRA fine-tuning across 8 GPUs with effective batch size 128 for 200k steps, while UniVLA uses batch size 16, learning rate 3.5 × 10−4, and trainable backbone components.OpenVLA uses rank r = 32 and learning rate η = 5.0 × 10−4; UniVLA uses rank r = 32 with a 16-entry codebook and 12-block encoder and decoder.Source: §H.3.1 · p. 41, §H.3.2 · pp. 41–42

H.3.3. π0-FAST TRAINING PARAMETERS

• π0-FAST and π0 are fine-tuned with LoRA for 60k steps using global batch size 32 and cosine-decayed AdamW optimization.π0-FAST uses FAST tokenization with 7-dimensional actions, horizon 10, and maximum sequence length 180; π0 uses Gemma-based VLM and action-expert variants.Source: §H.3.3 · p. 42, §H.3.4 · p. 42

H.3.5. OPENVLA-OFT TRAINING PARAMETERS

• OpenVLA-OFT uses LoRA, batch size 49, 150k steps, and an L1 regression objective, whereas SmolVLA trains up to 100k steps with its vision encoder frozen.SmolVLA predicts 50-step action chunks and uses two 256 × 256 images plus an 8-dimensional state vector.Source: §H.3.5 · p. 43, §H.3.6 · p. 44

H.4. Evaluation

• Each individual task is evaluated over 30 episodes, with 10 episodes per random seed, and variance is reported across seeds.This protocol is used for the benchmark’s evaluation results.Source: §H.4 · p. 44

H.5.1. PROMPT ENGINEERING FOR OBJECT LOCALIZATION

• Visual grounding is formulated as robotic planning that requires a VLM to output pick-and-place actions with bounding boxes for target objects and containers.The structured output tests both semantic understanding and spatial reasoning under perturbed observations.Source: §H.5.1 · pp. 44–45

H.5.2. GROUND TRUTH AND EVALUATION METRICS

• Grounding accuracy counts a prediction as correct only when the predicted and annotated boxes have Intersection over Union greater than 0.5.Ground truth boxes are manually annotated for targets and containers in the UnseenObjects evaluation suite.Source: §H.5.2 · p. 45

I. VLA-Arena Dataset Details

• 5750 validated trajectories cover 115 L0 and L1 tasks, with 50 human demonstrations collected per task.The dataset includes 60 L0 tasks and 55 L1 tasks after preprocessing and validation.Source: §I · pp. 45, 47
• Demonstrations are re-rendered at 256 × 256 resolution, use third-person and wrist-mounted cameras, and are filtered by replay success.The preprocessing also rotates third-person images by 180 degrees to correct the observed hardware orientation.Source: §I · p. 45
• Action cleaning retains selected no-operation actions around gripper transitions because removing them reduced playback success in this setup.The final trajectories preserve N ∈ {4, 8, 12, 16} no-operation actions only when replay validation succeeds.Source: §I · p. 45
• Each task level has Small, Medium, and Large dataset variants containing 10, 30, and 50 filtered trajectories per task, respectively.The main experiments use VLA-Arena-L0-L.Source: §I · p. 47

J. Diversity of Task Space

• Task-space diversity is measured with weighted tree-edit distances over CBDDL syntax trees and visualized using a force-directed graph.The distance accounts for node types and semantic similarity between subtrees.Source: J. Diversity of Task Space · p. 47
• Figure 9 shows VLA-Arena occupying a broader, more diverse space than LIBERO, while its four dimensions form distinct yet complementary clusters.The authors interpret this structure as evidence that the dimensions target different aspects of VLA capability.Source: Figure 9 · p. 46, J. Diversity of Task Space · p. 48