PaperlayerView exact revision on arXiv

Source-linked AI summary

TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation

Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, Jian Tang

arXiv:2409.12514v5cs.ROcs.CV

TL;DR

Existing vision-language-action models are costly to train and slow to run, limiting practical robotic manipulation. TinyVLA combines compact multimodal backbones with a diffusion action head, outperforming OpenVLA in speed and data efficiency while achieving comparable or better performance.

  • Problem

    Existing VLA models face high inference latency and expensive, resource-intensive training, motivating more efficient robotic manipulation models.

  • Method

    TinyVLA uses compact pretrained vision-language models with a diffusion-based head for direct robot action prediction.

  • Results

    TinyVLA outperforms OpenVLA across simulation and real-world multi-task evaluations; its average success rate exceeds OpenVLA by 25.7%.

  • Takeaways & Limitations

    TinyVLA supports fast, data-efficient VLA development without large-scale robotic pretraining and generalizes across diverse task and environmental settings.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown remarkable potential in visuomotor control and instruction comprehension through end-to-end learning processes. However, current VLA models face significant challenges: they are slow during inference and require extensive pre-training on large amounts of robotic data, making real-world deployment difficult. In this paper, we introduce a new family of compact vision-language-action models, called TinyVLA, which offers two key advantages over existing VLA models: (1) faster inference speeds, and (2) improved data efficiency, eliminating the need for pre-training stage. Our framework incorporates two essential components to build TinyVLA: (1) initializing the policy backbone with robust, high-speed multimodal models, and (2) integrating a diffusion policy decoder during fine-tuning to enable precise robot actions. We conducted extensive evaluations of TinyVLA in both simulation and on real robots, demonstrating that our approach significantly outperforms the state-of-the-art VLA model, OpenVLA, in terms of speed and data efficiency, while delivering comparable or superior performance. Additionally, TinyVLA exhibits strong generalization capabilities across various dimensions, including language instructions, novel objects, unseen positions, changes in object appearance, background variations, and environmental shifts, often matching or exceeding the performance of OpenVLA. We believe that \methodname offers an interesting perspective on utilizing pre-trained multimodal models for policy learning. Our project is at https://tiny-vla.github.io.

I. INTRODUCTION

TinyVLA addresses VLA models’ slow inference and dependence on large-scale robotic pretraining with a compact architecture combining lightweight vision-language models and diffusion-based action generation. Experiments in simulation and real-world single-arm and bimanual settings show strong performance, data efficiency, and generalization.

  • Motivation: Existing VLA models achieve strong multitask learning and generalization but suffer from extremely slow inference and expensive, resource-intensive training.OpenVLA is pretrained on the 970K-sample OpenX dataset.
  • Method: TinyVLA attributes high inference latency to large vision-language backbones, often exceeding 7 billion parameters, and autoregressive discrete action-token generation.The latter requires repetitive inference for each degree of freedom.
  • Results: 25.7% higher success rate: TinyVLA-H surpasses OpenVLA in real-world experiments while using 5.5 times fewer parameters.The comparison is reported for real-world experiments.
  • Method: TinyVLA combines lightweight vision-language models with a diffusion model to enable fast inference, strong performance, and excellent generalization.The architecture is evaluated in both simulated and real-world settings, including single-arm and bimanual robot setups.
  • Results: TinyVLA demonstrates that strong VLA models can be trained without large-scale robotic datasets while retaining data efficiency and high performance.The paper presents this as a perspective for building VLA models for embodied control.

II. RELATED WORKS

Related work spans multimodal models, their use in robot learning, and multi-task robotic manipulation. TinyVLA builds on efforts to make multimodal models more efficient and to use vision-language models as robot action predictors.

  • Vision-language models: Vision-language models connect visual and linguistic inputs, extending large language models’ reasoning to multimodal processing.
  • Vision-language models: Many multimodal language models contain 7B–70B parameters, making inference costly, while recent studies explore models with fewer than 3B parameters.The smaller-model efforts target more efficient multimodal processing.
  • Vision-language models for robot learning: Robot-learning research uses vision-language models for high-level planning, task decomposition, and end-to-end robot action prediction.TinyVLA focuses on lightweight, fast vision-language models for robot action prediction.
  • Multi-task robot learning: Multi-task manipulation methods aim to execute complex tasks and generalize to novel scenarios, often relying on extensive interaction data.Examples include RT-1’s task-agnostic training and RT-2’s use of mixed robot data with image-text pairs.

III. METHOD · A. Building TinyVLA with Efficient Vision-Language Models

TinyVLA builds a compact policy from pretrained vision-language models, freezes most pretrained components during robot-data fine-tuning, and adds a policy decoder. Its VLM family spans 70 million to 1.4 billion parameters and uses Pythia with an LLaVA-style training pipeline.

  • III. METHOD: TinyVLA initializes its policy network with a pretrained vision-language model.
  • III. METHOD: During robot-data training, TinyVLA freezes the pretrained components and applies LoRA for parameter-efficient fine-tuning.
  • III. METHOD: 5% of the entire model’s parameters are trainable under the described LoRA fine-tuning setup.
  • III. METHOD: TinyVLA introduces a policy decoder concatenated to the pretrained multimodal model.
  • A. Building TinyVLA with Efficient Vision-Language Models: TinyVLA uses a compact VLM family ranging from 70 million to 1.4 billion parameters.
  • A. Building TinyVLA with Efficient Vision-Language Models: The VLMs use Pythia as their language-model backend and follow LLaVA’s training pipeline with its vision-language dataset.

B. Robot Data Finetuning for Manipulation

TinyVLA fine-tunes a multimodal backbone with parameter-efficient LoRA while using a diffusion policy decoder to learn continuous robot actions. Its pipeline encodes visual-language inputs into compact normalized features, and re-parameterization integrates LoRA for faster inference.

  • Frozen weights and low-rank adaptation: LoRA updates only low-rank attention adaptations while freezing the Transformer’s remaining weights.The weight update is parameterized as W0 + BA, with rank r much smaller than the original dimensions.
  • Frozen weights and low-rank adaptation: 5.0% of the Transformer’s parameters are trainable, preserving the language model’s intrinsic knowledge while retaining flexibility.After training, re-parameterization integrates LoRA into the standard language model to enhance inference speed.
  • Learning action with diffusion policy decoder: The model uses a diffusion policy decoder instead of discrete action tokenization, avoiding challenges associated with continuous or high-dimensional actions.The paper states that tokenization is difficult to train, data-intensive, and prone to converging to a single state.
  • Learning action with diffusion policy decoder: The training pipeline encodes observations and language instructions into multimodal embeddings, then adaptively pools and normalizes them into compact fixed-length features.These normalized features are subsequently concatenated with robot proprioceptive information.
  • Learning action with diffusion policy decoder: The VLM is fine-tuned with LoRA, whereas the diffusion-policy head undergoes full-parameter training.

IV. EXPERIMENTS · A. Experimental Setup

The experiments assess TinyVLA’s multitask success, instruction following, environmental generalization, and scaling behavior, while comparing three model sizes defined by multimodal-backbone scale.

  • IV. EXPERIMENTS: The experimental agenda is organized around questions about success rates, novel-instruction following, environmental robustness, and model scaling.These questions define the study’s experimental objectives.
  • IV. EXPERIMENTS: The experiments evaluate whether TinyVLA achieves higher multitasking robotic-manipulation success rates than baseline methods.This is the first stated experimental question.
  • IV. EXPERIMENTS: They test whether TinyVLA can interpret and follow novel instructions.This measures instruction-following capability beyond familiar commands.
  • IV. EXPERIMENTS: They examine generalization to unseen environments, new backgrounds, varying lighting, changed camera views, and novel distractors.The evaluation probes robustness across multiple environmental and visual shifts.
  • IV. EXPERIMENTS: They investigate whether TinyVLA follows a scaling law in which larger models improve performance and generalization.The question directly links model size with both capabilities.
  • A. Experimental Setup: TinyVLA is categorized into TinyVLA-S, TinyVLA-B, and TinyVLA-H according to the scale of its multimodal model.The three sizes correspond to Small, Base, and Huge configurations.

1) Simulation Benchmark: · 2) Real Robot Setup:

The simulation benchmark evaluates TinyVLA on MetaWorld’s 50 tasks against Diffusion Policy using multi-task training and repeated-seed success rates. Real-robot experiments cover single-arm and bimanual setups, teleoperated 7-dimensional action prediction, and comparisons with three baselines.

  • 1) Simulation Benchmark:: TinyVLA is evaluated on MetaWorld, whose 50 tasks span easy, medium, hard, and very hard difficulty levels.
  • 1) Simulation Benchmark:: The simulation comparison uses Diffusion Policy as the baseline and reports average success rates from multi-task training with 50 demonstrations.Evaluation uses three seeds, with each seed’s success rate averaged over five iterations.
  • 2) Real Robot Setup:: Real-robot evaluation includes a single-arm Franka Panda 7DoF setup and a bimanual setup with two UR5 arms.
  • 2) Real Robot Setup:: The single-arm scene uses two externally fixed ZED 2 stereo cameras, while the bimanual scene uses two wrist-mounted Realsense D435i cameras and one top camera.
  • 2) Real Robot Setup:: Single-arm tasks include drawer closing, cube stacking, box-lid opening, tennis-ball placement, and mug uprighting.
  • 2) Real Robot Setup:: The dataset is collected through teleoperation by recording RGB views and robot states, including joint positions and normalized gripper width.TinyVLA predicts 7-dimensional actions comprising position, rotation, and gripper width.
  • 2) Real Robot Setup:: Real-world comparisons evaluate TinyVLA against Diffusion Policy, Multimodal Diffusion, and OpenVLA with modifications intended to ensure fair camera-view usage.OpenVLA processes separate camera views through a shared visual backbone before concatenating visual tokens for the language model.

B. Experimental Results on Multi-Task Learning · C. Generalization to Unseen Instructions

TinyVLA outperforms Diffusion Policy and baselines in multi-task simulation and real-robot experiments, while TinyVLA-H generalizes to unseen instructions, object attributes, environments, and skill-object combinations. These results support the model’s effectiveness across increasingly difficult manipulation and generalization settings.

  • B. Experimental Results on Multi-Task Learning: TinyVLA’s success rate exceeds Diffusion Policy by 21.5% in simulation and is sixfold better on MetaWorld Hard.The performance disparity widens as task complexity increases.
  • B. Experimental Results on Multi-Task Learning: TinyVLA-H attained a 98.3% success rate in flipping a mug and stacking cubes, plus a 90% success rate in place tennis.Real-world evaluation used 20 trials per task in a single-arm setting, reporting mean and standard deviation across 3 checkpoints.
  • B. Experimental Results on Multi-Task Learning: Real-robot experiments show performance increases drastically from TinyVLA-S to TinyVLA-H, consistent with scaling the model.All models in the bimanual UR5 experiments were trained in multi-task settings and evaluated over 10 trials.
  • C. Generalization to Unseen Instructions: TinyVLA-H is investigated for generalization because it delivers the best performance in both real-world scenarios and simulations.Its pre-trained multimodal backbone is associated with embodied capabilities driven by implicitly stored world knowledge.
  • C. Generalization to Unseen Instructions: Instruction generalization tests used fixed “Pick the [object]” instructions with randomized unseen mugs, toy cars, and pink cubes.The experiments evaluated three object types not present in the training data.
  • C. Generalization to Unseen Instructions: TinyVLA successfully flipped a green mug whose color was unseen during training, demonstrating understanding of object attributes.The test contrasted two mugs with seen and unseen colors while instructing the model to flip the green mug.
  • C. Generalization to Unseen Instructions: TinyVLA successfully picked up a cube despite both the environment and instruction being absent from training, mapping text descriptions to physical objects.Both objects in this second-level test had appeared in the training data.
  • C. Generalization to Unseen Instructions: TinyVLA successfully executed “pick a toy car” and “place it into the box,” combining an unseen object with a new skill-object pairing.A pink cube beside the toy car tested whether the model could comprehend the instruction and altered object function.

D. More Real-World Experiments: Bimanual Robot · E. Experiments on Generalization

TinyVLA-H outperformed Diffusion Policy on a bimanual robot while OpenVLA failed all trials. Across generalization experiments, TinyVLA showed robustness to viewpoint, background, lighting, distractors, and unseen object locations.

  • D. More Real-World Experiments: Bimanual Robot: TinyVLA-H achieved a 44.5% average success rate on three bimanual UR5 tasks, surpassing Diffusion Policy’s 38.2%.The tasks were PlaceBread, StackCube, and PlaceTennisBag; OpenVLA failed in every trial.
  • D. More Real-World Experiments: Bimanual Robot: OpenVLA failed on the bimanual tasks, possibly because its OpenX pre-training used only single-arm robot data.
  • E. Experiments on Generalization: The pre-trained multimodal model with Diffusion Policy head was evaluated using one trial for each generalization setting.The integration is intended to enhance action output and adaptability across diverse environments.
  • E. Experiments on Generalization: TinyVLA showed greater robustness to viewpoint changes, whereas Diffusion Policy was highly sensitive and could fail after slight shifts.
  • E. Experiments on Generalization: TinyVLA accurately located objects and completed tasks across six background styles, including position-sensitive tennis-ball placement scenarios.Three backgrounds were tested on Task A and three on Task B.
  • E. Experiments on Generalization: TinyVLA remained unaffected when overhead lights were turned off or all lights were turned off.The experiment compared training illumination with two reduced-lighting conditions.
  • E. Experiments on Generalization: TinyVLA handled distractors at both difficulty levels in the StackCube task without data augmentation.The distractors included unrelated objects and identical cubes in different colors.
  • E. Experiments on Generalization: TinyVLA completed tasks at unseen object locations outside the training zone despite not being trained on those specific positions.The spatial evaluations used the position-sensitive place-tennis and flip-mug tasks.

G. Visual Generalization

TinyVLA addresses visual generalization to novel textures, colors, backgrounds, and lighting by interpreting their semantic cues without changing task structure. It generalizes to altered object colors similarly to OpenVLA, without training-time data augmentation.

  • Visual generalization: Visual generalization covers novel textures, background colors, object appearances, and ambient lighting while preserving object-target positioning.These changes require accurate interpretation of the semantic meanings associated with visual cues.
  • Appearance generalization: TinyVLA successfully generalizes to target objects with altered colors, similarly to OpenVLA.The evaluation changed the mug and lid colors from their initial brown-and-white appearance.
  • Appearance generalization: TinyVLA achieves appearance generalization without relying on training-time data augmentation.This result indicates that the model possesses the relevant generalization capability without augmentation during training.

V. ABLATION STUDY · A. Trade-off between size of VLM and TinyVLA’s Performance

The ablation study shows that TinyVLA follows a scaling law: larger VLM backbones improve average task success. Failure analysis attributes this trade-off to better language comprehension and more accurate positioning with increased model size.

  • A. Trade-off between size of VLM and TinyVLA’s Performance: TinyVLA’s average success rate across tasks improves as model size increases.The evaluated variants were TinyVLA-0.4B, TinyVLA-1.3B, and TinyVLA-3B.
  • A. Trade-off between size of VLM and TinyVLA’s Performance: The analysis evaluated TinyVLA-0.4B, TinyVLA-1.3B, and TinyVLA-3B to examine the size-performance trade-off.TinyVLA-0.4B and TinyVLA-1.3B were used in the main experiments, while TinyVLA-3B uses the pre-trained PaliGemma m…
  • A. Trade-off between size of VLM and TinyVLA’s Performance: VLM size significantly impacts task success.This conclusion was drawn from failure case analysis across the three TinyVLA variants.
  • A. Trade-off between size of VLM and TinyVLA’s Performance: TinyVLA-0.4B failed three times because it misinterpreted instructions.The failures were attributed likely to the smaller VLM’s limited language comprehension capabilities.
  • A. Trade-off between size of VLM and TinyVLA’s Performance: Increasing the model size to 1.3B resolved the instruction-misinterpretation issue.The passage links this improvement to increased model size and language comprehension.
  • A. Trade-off between size of VLM and TinyVLA’s Performance: Larger models mitigated failures caused by inaccurate positioning and incorrect target locations.The improvement was attributed to using models like PaliGem…

B. Choice of Policy Model · C. Which Part of TinyVLA makes it Fast? · VI. CONCLUSION

TinyVLA’s policy-model analysis examines diffusion against alternative policy heads and attributes its speed advantage to the lightweight architecture, while the conclusion emphasizes fast, data-efficient VLA learning with reduced training resources.

  • B. Choice of Policy Model: TinyVLA’s strong performance and generalization are attributed largely to integrating a pretrained vision-language model with a diffusion model.The paper questions how essential the diffusion component is and motivates comparison with alternative policy networks.
  • C. Which Part of TinyVLA makes it Fast?: TinyVLA-H achieves a higher average success rate than OpenVLA with 5.5 times fewer parameters and operating 20 times faster.The comparison identifies TinyVLA’s lightweight design as a central speed advantage over OpenVLA.
  • C. Which Part of TinyVLA makes it Fast?: The speed analysis replaces OpenVLA’s Prismatic-7B VLM backbone with the same architecture used in TinyVLA.This controlled comparison is intended to identify the primary contributor to TinyVLA’s inference-speed advantage.
  • C. Which Part of TinyVLA makes it Fast?: Inference latency is reported in milliseconds for single-action predictions from OpenVLA-1B, OpenVLA-7B, and TinyVLA-1B on one A6000 GPU.The table quantitatively measures the time required for each action prediction.
  • VI. CONCLUSION: The work explores pretrained multimodal models for robotic manipulation as a way to overcome limitations of previous approaches.The conclusion presents this direction as the basis of TinyVLA’s approach.
  • B. Choice of Policy Model: The ablation replaces TinyVLA-H’s diffusion model with ACT and a vanilla MLP head, evaluating success rates on five real-robot tasks.TinyVLA-H is used as the base model for this policy-choice study.
  • VI. CONCLUSION: TinyVLA enables fast inference while significantly reducing the computational resources required for training.The method’s effectiveness is demonstrated through simulation and real-world experiments.
  • VI. CONCLUSION: The authors characterize TinyVLA as a novel solution for building fast, data-efficient vision-language-action models.This concluding claim follows demonstrations in both simulation and real-world experiments.
Loading 2409.12514v5…