PaperlayerView exact revision on arXiv

Source-linked AI summary

StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing

StarVLA Community

arXiv:2604.05014v1cs.ROcs.AIcs.CV

TL;DR

VLA research lacks a unified basis for comparing and reproducing methods because architectures, codebases, training pipelines, and evaluation protocols differ substantially. StarVLA introduces a modular backbone–action-head framework with reusable training strategies and unified benchmark interfaces, achieving competitive and sometimes state-of-the-art performance across multiple benchmarks with both VLM and world-model backbones. Its results support viewing diverse VLA approaches within a common structural framework, while reinforcement-learning fine-tuning remains an ongoing integration effort.

  • Problem

    VLA methods use heterogeneous architectures, preprocessing, interfaces, and evaluation protocols, limiting fair comparison and reproducibility.

  • Method

    StarVLA uses interchangeable backbone and action-head modules, reusable cross-embodiment and multimodal co-training strategies, and unified interfaces spanning major benchmarks and deployment settings.

  • Results

    StarVLA achieves competitive and sometimes state-of-the-art performance across multiple benchmarks with both VLM and world-model backbones.

  • Takeaways & Limitations

    A shared backbone–action-head abstraction can accommodate VLM-based, generative-model-based, and dual-system VLA approaches within common data, training, and evaluation pipelines.

  • Takeaways & Limitations

    Reinforcement-learning fine-tuning is still an ongoing integration effort; the current public codebase focuses on supervised and co-training pipelines.

Abstract

from arXiv · show

Building generalist embodied agents requires integrating perception, language understanding, and action, which are core capabilities addressed by Vision-Language-Action (VLA) approaches based on multimodal foundation models, including recent advances in vision-language models and world models. Despite rapid progress, VLA methods remain fragmented across incompatible architectures, codebases, and evaluation protocols, hindering principled comparison and reproducibility. We present StarVLA, an open-source codebase for VLA research. StarVLA addresses these challenges in three aspects. First, it provides a modular backbone--action-head architecture that supports both VLM backbones (e.g., Qwen-VL) and world-model backbones (e.g., Cosmos) alongside representative action-decoding paradigms, all under a shared abstraction in which backbone and action head can each be swapped independently. Second, it provides reusable training strategies, including cross-embodiment learning and multimodal co-training, that apply consistently across supported paradigms. Third, it integrates major benchmarks, including LIBERO, SimplerEnv, RoboTwin~2.0, RoboCasa-GR1, and BEHAVIOR-1K, through a unified evaluation interface that supports both simulation and real-robot deployment. StarVLA also ships simple, fully reproducible single-benchmark training recipes that, despite minimal data engineering, already match or surpass prior methods on multiple benchmarks with both VLM and world-model backbones. To our best knowledge, StarVLA is one of the most comprehensive open-source VLA frameworks available, and we expect it to lower the barrier for reproducing existing methods and prototyping new ones. StarVLA is being actively maintained and expanded; we will update this report as the project evolves. The code and documentation are available at https://github.com/starVLA/starVLA.

1 Introduction

VLA research is fragmented across architectures, codebases, training assumptions, and evaluation protocols, making comparison and reproduction difficult. StarVLA addresses this with a modular backbone–action-head framework, reusable training strategies, broad benchmark integration, and a generalized perspective on VLA design.

  • Research fragmentation: VLA methods span incompatible action-decoding paradigms, system assumptions, and evaluation protocols, hindering systematic comparison, reproduction, and recombination.The fragmentation creates a “Tower of Babel” across architecture, system, and evaluation levels.
  • Unified framework: StarVLA decomposes VLA systems into interchangeable backbones and action heads, supporting VLM and world-model backbones with multiple action-decoding paradigms.Supported paradigms include autoregressive tokenization, parallel regression, flow-matching denoising, and dual-system reasoning.
  • Training strategies: StarVLA makes cross-embodiment learning and multimodal co-training reusable across supported paradigms through shared training infrastructure.The same infrastructure supports supervised action learning, web-scale vision-language co-training, and pretraining across heterogeneous robot datasets.
  • Benchmark integration: StarVLA integrates five benchmarks through a unified interface supporting controlled comparisons, simulation evaluation, and real-robot deployment without code changes.The benchmarks are LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K.
  • Results: StarVLA achieves competitive and sometimes state-of-the-art performance across multiple benchmarks with both VLM and world-model backbones.The framework also provides strong, easy-to-reproduce baselines.
  • Generalized VLA perspective: StarVLA’s shared abstraction accommodates diverse VLA approaches within common data, training, and evaluation pipelines, motivating a generalized VLA perspective.The perspective treats VLM-based and world-model-based methods as structural variations differing mainly in auxiliary learning signals.

2 Unified Framework for VLA Systems

StarVLA addresses fragmented VLA designs with a unified policy-centric formulation and modular backbone–action-head architecture. The framework supports diverse action-decoding strategies while standardizing interfaces across training and deployment.

  • Unified policy formulation: StarVLA models VLA systems as policies mapping multimodal observations and language instructions to future action chunks with optional auxiliary outputs.The observation history may include vision, depth, tactile feedback, proprioception, and other sensor modalities.
  • Unified policy formulation: The training objective decomposes into L = Laction + Laux, where action loss supervises predictions and auxiliary loss supplies inductive bias.Direct policies set Laux = 0, while VLM- and world-model-based methods use language or future-observation objectives.
  • Unified interfaces: The unified raw-observation I/O contract aligns training and inference inputs with deployment observations, reducing train/test distribution mismatch.Training uses forward() for losses, while predict_action() accepts observations and returns predicted action chunks.
  • Representative instantiations: Four representative decoding paradigms share the same backbone, base class, and forward/predict_action interfaces, differing only in action extraction.The spectrum includes autoregressive tokenization, parallel regression, flow-matching denoising, and dual-system reasoning.

3 Unified System Pipeline for Model Training and Testing

StarVLA connects heterogeneous data sources and pluggable dataloaders to a standardized model-forwarding interface. This shared pipeline supports diverse VLA architectures for end-to-end training and deployment.

  • Training and deployment: StarVLA supports supervised fine-tuning, multimodal co-training, and cross-embodiment co-training under a unified YAML configuration interface.Training loops use PyTorch with Accelerate and DeepSpeed for distributed execution.
  • Unified pipeline: The framework links heterogeneous data sources, pluggable dataloaders, and flexible data representations through a standardized model-forwarding interface.Figure 3 summarizes how data streams connect to the unified model framework.

3.1 Training Paradigms

StarVLA provides reusable training regimes spanning robot-only supervision, multimodal co-training, and heterogeneous-embodiment learning. Reinforcement-learning fine-tuning remains planned rather than available in the current public codebase.

  • Supervised learning: Robot-only supervised learning trains policies to predict continuous actions from observations and language instructions.The action modeling loss is returned by the framework forward() method.
  • Multimodal co-training: Co-training interleaves robot action learning with a vision-language loss on multimodal web data to preserve visual reasoning and language grounding.Two dataloaders produce separate VLA and VLM forward/backward passes.
  • Multimodal co-training: The VLM loss is scaled by trainer.loss_scale.vlm, controlling the trade-off between action learning and retention of VLM capabilities.The VLA pass obtains action_loss, while the VLM pass obtains language modeling loss.
  • Cross-embodiment learning: Cross-embodiment learning uses a unified mixture-dataset interface for heterogeneous robot datasets with different embodiments, action conventions, and camera setups.Named mixtures specify dataset names, sampling weights, and robot types.
  • Reinforcement-learning fine-tuning: Reinforcement-learning fine-tuning is an ongoing integration effort; the current public codebase focuses on supervised and co-training pipelines.The planned extension is being developed in collaboration with RLinf.

3.2 Evaluation and Deployment

StarVLA separates benchmark evaluation from model inference through a thin server–client interface. The same contract supports simulation, hosted benchmarks, and real-robot deployment while keeping model and environment dependencies separate.

  • Evaluation interface: A lightweight WebSocket policy server hosts checkpoints while benchmark evaluators interact through a small client, preserving proximity to official benchmark implementations.The model is loaded with baseframework.from_pretrained().
  • Inference interface: All framework variants expose Framework.predict_action(), receiving observation dictionaries containing images, language, and optional state or metadata.Payloads are serialized with msgpack before transmission.
  • Benchmark adapters: Benchmark-specific adapters translate raw observations into StarVLA’s common example format and convert returned actions into native control APIs.Adapters also handle tasks such as image resizing and checkpoint statistics.
  • Real-robot deployment: In real-robot deployment, the robot controller acts as the benchmark client, querying the unchanged policy server and executing returned actions on hardware.Control, safety logic, and device middleware remain outside the StarVLA model runtime.
  • Real-robot deployment: The same checkpoint can serve simulation and real-robot settings when clients provide the agreed observation dictionary and apply appropriate interface handling.The model stack can remain in a GPU inference environment while robot-side processes use vendor SDKs or ROS.

4 Multiple Benchmark Integration

StarVLA standardizes evaluation across heterogeneous embodied-manipulation benchmarks while preserving official benchmark workflows and reproducible policy interfaces. Its suite spans robustness, simulation, tabletop, bimanual, household, and long-horizon language-conditioned manipulation settings.

  • StarVLA addresses inconsistent preprocessing, policy interfaces, and evaluation protocols that hinder fair comparison and reproducibility.
  • Its benchmark integration uses a unified lightweight WebSocket policy interface while keeping benchmark-side evaluation close to official implementations.
  • The integrated suite includes LIBERO, LIBERO-Plus, SimplerEnv, RoboCasa-GR1, RoboTwin 2.0, BEHAVIOR-1K, and CALVIN, covering varied environments, embodiments, and protocols.
  • LIBERO: LIBERO evaluates instruction following and generalization across 130 tasks in Spatial, Object, Goal, and Long suites using standardized language-conditioned manipulation protocols.
  • SimplerEnv: SimplerEnv provides standardized simulation evaluations for WidowX and Google Robot platforms, including Visual Matching and Variant Aggregation settings.
  • Additional benchmarks: RoboCasa-GR1, RoboTwin 2.0, BEHAVIOR-1K, and CALVIN extend evaluation to humanoid-style tabletop, bimanual, household long-horizon, and sequential language-conditioned manipulation.

5 Single-Benchmark Training Examples

StarVLA provides transparent single-benchmark supervised fine-tuning recipes that use minimal data engineering and official evaluation workflows. Across LIBERO, SimplerEnv, RoboCasa-GR1, and RoboTwin 2.0, the baselines show strong and reproducible performance across backbones and action heads.

  • The single-benchmark experiments avoid VLA-specific pretraining, augmentation, and online refinement, using publicly released vision-language weights and benchmark-specific data.
  • LIBERO: LIBERO is presented as a fully reproducible worked example covering data loading, training, and evaluation across four task suites.
  • LIBERO: The LIBERO baseline uses native data with two RGB views, no proprioception, history stacking, or augmentation, and predicts chunked continuous end-effector actions.
  • LIBERO results: 96.6% average success is achieved by StarVLA-OFT with Qwen3-VL on LIBERO, compared with 97.1% for OpenVLA-OFT using 6× more steps and 23× more epochs.
  • SimplerEnv results: 65.3% average success is reached by StarVLA with Qwen3-VL-4B on SimplerEnv WidowX Visual Matching, while Cosmos-Predict2-2B reaches 61.6%.
  • RoboCasa-GR1 results: 48.8% average success is achieved by StarVLA-OFT on RoboCasa-GR1, exceeding StarVLA-GR00T at 47.8% and outperforming π0.5 by 11.8 points.
  • RoboTwin 2.0 results: All four StarVLA variants achieve strong average success rates when trained as one unified policy over 50 RoboTwin 2.0 tasks.

6 Multimodal Co-Training Examples

StarVLA supports multimodal co-training to preserve vision-language capabilities during action learning. Spatially guided co-training improves the balance between perception and manipulation compared with action-only or vanilla co-training strategies.

  • Multimodal co-training jointly optimizes robot action data with auxiliary tasks such as spatial grounding, visual question answering, and captioning.
  • Action-only fine-tuning can rapidly degrade grounding, instruction following, and scene understanding, while grounding co-training maintains perception-relevant gradient flow.
  • StarVLA supports heterogeneous action and vision-language datasets through shared configuration, tokenization, loss masking, and gradient accumulation infrastructure.
  • Study design: The spatially guided study compares action-only training, vanilla co-training, and co-training augmented with spatial pre-training and spatial prompting.
  • Perception–action dynamics: ∼70% of original grounding performance is maintained by spatially guided training while achieving strong WidowX manipulation success; vanilla action training falls near-random within 20K steps.
  • Main results: 84.6%/75.9% Google Robot VM/VA and 73.2% WidowX success are achieved alongside 71.2 IoU@0.5 RefCOCO-g grounding by the spatially guided variant.
  • Takeaways: Co-training improves over action-only fine-tuning by preserving multimodal understanding during policy learning, with broader treatment provided by ST4VLA.

7 Cross-Benchmark Training Examples

StarVLA evaluates a single jointly trained policy across multiple benchmarks and embodiments, rather than relying only on benchmark-specific specialists. The generalist model remains competitive overall and substantially improves RoboCasa-GR1.

  • Evaluation Protocol: A unified multi-benchmark evaluation tests whether one policy generalizes across diverse tasks, environments, and robot embodiments without benchmark-specific fine-tuning.The model is trained once on merged datasets and evaluated under each benchmark’s official protocol.
  • Baselines: The study compares specialist models trained per benchmark with a generalist model jointly trained across datasets.Baselines include specialist results and state-of-the-art methods such as π0.5 and GR00T-N1.6.
  • Main Results: 57.3% on RoboCasa-GR1’s 24-task average improves over the best specialist average of 48.8%.The comparison is reported for the jointly trained generalist model versus specialist models.
  • Main Results: The generalist model remains competitive across most evaluated benchmarks.These results support the feasibility of transfer across tasks and embodiments under unified training and evaluation.
  • Takeaways: The authors present all-in-one multi-benchmark training as a practical path toward cross-embodiment pretraining for future generalist VLA systems.This section is framed as a capability demonstration rather than an ablation analysis.

8 Computation Efficiency

StarVLA’s efficiency measurements show a trade-off between step latency and sample throughput, while multi-node scaling increases data-processing capacity despite communication overhead. Scaling efficiency stabilizes at large GPU counts.

  • Multi-Node Scaling: 79–80% parallel efficiency stabilizes beyond 32 GPUs despite distributed communication overhead.Figure 6 compares measured sample throughput with ideal linear scaling.
  • Efficiency Metrics: Step throughput measures seconds per step, whereas sample throughput measures global samples processed per second.These metrics can move in opposite directions during distributed scaling.
  • Single-Node Efficiency: Larger per-GPU batches increase sample throughput but sharply increase step latency on a single 8-GPU node.Latency rises from 0.703 s/step at batch 2 to 2.404 s/step at batch 24, while throughput rises from 22.7 to 79.9 samples/s.
  • Multi-Node Scaling: 0.735→0.93 s/step is the latency increase from 8 to 256 GPUs, while sample throughput rises from 87.0 to 2200.0 samples/s.Latency increases because of inter-node communication and then plateaus near 0.93 s/step.
  • Practical Guidance: Scaling out is most beneficial for data-volume-driven training, while fixed-step training does not become faster with more GPUs.A moderate per-GPU batch can balance latency and utilization; batch 24 reaches 96% utilization but increases latency 3.4×.

9 Authors and Contributors for StarVLA v1.0

StarVLA distinguishes its core authors from community contributors and describes both groups’ roles in maintaining and expanding the project.

  • Contributors: Jinhui Ye, Ning Gao, Yilun Chen, Weiyu Guo, and additional collaborators are listed as authors of StarVLA v1.0.The author list includes corresponding-author and institutional footnotes.
  • Community Contributors: Community contributors support StarVLA’s broader open-source ecosystem.The paper lists community contributors separately from the authors.
  • Authors: The author group forms StarVLA’s core team and maintains the foundational framework over the long term.Authors are responsible for continuously iterating on core features and providing ongoing support.
Loading 2604.05014v1…