PaperlayerView exact revision on arXiv

Source-linked AI summary

Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting

Asher J. Hancock, Xindi Wu, Lihan Zha, Olga Russakovsky, Anirudha Majumdar

arXiv:2509.22195v1cs.RO

TL;DR

Fine-tuning VLMs for robotic control can erode their general-purpose reasoning and multimodal understanding, while co-training to mitigate knowledge loss is costly. VLM2VLA represents robot actions in natural language and uses LoRA to adapt the VLM; it reports retained VQA performance and generalization in real-world robotics experiments.

  • Problem

    Fine-tuning VLMs on robot data risks overwriting pretrained knowledge, while co-training to mitigate that loss requires costly VLM-scale data and a carefully constructed mixture.

    Source: §1 · p. 2

  • Method

    VLM2VLA aligns robot fine-tuning data with VLM representations by expressing low-level actions in natural language, enabling adaptation with LoRA.

    Source: §1 · p. 2

  • Results

    Over 800 real-world robotic experiments show generalization to novel tasks, and VLM2VLA retains over 85% of base-model performance across challenging VQA benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VLM2VLA’s retained world knowledge supports zero-shot generalization to novel tasks requiring open-world semantic reasoning.

    Source: §5 · p. 9

    A separate verifier transitions between subtasks and slows inference.

    Source: §5.1 · p. 10

Abstract

from arXiv · show

Fine-tuning vision-language models (VLMs) on robot teleoperation data to create vision-language-action (VLA) models is a promising paradigm for training generalist policies, but it suffers from a fundamental tradeoff: learning to produce actions often diminishes the VLM's foundational reasoning and multimodal understanding, hindering generalization to novel scenarios, instruction following, and semantic understanding. We argue that this catastrophic forgetting is due to a distribution mismatch between the VLM's internet-scale pretraining corpus and the robotics fine-tuning data. Inspired by this observation, we introduce VLM2VLA: a VLA training paradigm that first resolves this mismatch at the data level by representing low-level actions with natural language. This alignment makes it possible to train VLAs solely with Low-Rank Adaptation (LoRA), thereby minimally modifying the VLM backbone and averting catastrophic forgetting. As a result, the VLM can be fine-tuned on robot teleoperation data without fundamentally altering the underlying architecture and without expensive co-training on internet-scale VLM datasets. Through extensive Visual Question Answering (VQA) studies and over 800 real-world robotics experiments, we demonstrate that VLM2VLA preserves the VLM's core capabilities, enabling zero-shot generalization to novel tasks that require open-world semantic reasoning and multilingual instruction following.

1 Introduction

VLM2VLA addresses the risk that adapting VLMs for robot control erodes their general-purpose knowledge, a loss often mitigated with costly co-training. It aligns robot actions with the VLM’s language representations so LoRA can adapt the model while retaining reasoning and generalization capabilities.

1 Introduction

  • Robot-data fine-tuning can overwrite VLM world knowledge, weakening generalization to novel objects, linguistic variations, distractions, and concepts beyond training.The paper links this risk to adapting the VLM for robotic control through full-parameter fine-tuning on robot imitation data.Source: §1 · p. 2
  • Co-training with non-robotic data can mitigate knowledge loss, but requires costly VLM-scale datasets and a carefully constructed data mixture.The paper frames preserving foundational knowledge without co-training as its goal.Source: §1 · p. 2
  • VLM2VLA represents low-level robot actions as natural-language descriptions, aligning fine-tuning data with the VLM’s pretrained representations.The paper identifies mismatch between robotic action spaces and image-text pretraining data as a motivation for this data-level alignment.Source: §1 · p. 2
  • The alignment lets LoRA adapt the VLM for robotic control without significantly perturbing pretrained weights, avoiding major architecture changes or complex co-training.The paper also reports that the backbone assigns higher probabilities to language-based actions than to actions mapped to arbitrary tokens.Source: §1 · p. 2, Figure 3 · p. 2
  • Over 800 real-world robotic experiments show generalization to novel tasks, while VLM2VLA retains over 85% of base-model performance across challenging VQA benchmarks.The novel tasks include objects and language instructions unseen during training.Source: §1 · p. 3

2 Related Work

Related work describes VLA strategies for generating actions and organizing reasoning, alongside approaches to retaining VLM knowledge during robot fine-tuning. VLM2VLA’s distinction is to represent low-level actions in language and use LoRA without explicit action decoders.

2.1 Vision-Language Action (VLA) Models

  • VLA models transfer pretrained VLM knowledge to action prediction, but must bridge discrete tokens and continuous robot-control representations.The paper describes fine-tuning pretrained VLM backbones as the dominant paradigm for generalist robot policies.Source: §2.1 · p. 3

2.1.1 Action Representation in VLAs

  • Prior VLAs discretize continuous actions into tokens or add action heads for continuous generation; VLM2VLA instead uses natural-language actions.The action-head approach can enable faster prediction, but adds randomly initialized parameters that can corrupt pretrained representations.Source: §2.1.1 · p. 3

2.1.2 Reasoning Models

  • VLA reasoning methods use task-level plans or subgoals and structured mid-level representations such as trajectory sketches, code, or spatial affordances.VLM2VLA uses staged inference, generating subtasks, trajectory plans, then actions, within a single VLM.Source: §2.1.2 · pp. 3–4
  • VLM2VLA represents low-level actions as text, uses no explicit action decoder, and trains solely with LoRA, unlike prior VLAs and embodied-reasoning models.The paper contrasts this with concurrent work, which uses full-parameter fine-tuning and co-training.Source: §2.1.1 · p. 3, §2.1.2 · p. 4

2.2 Averting Catastrophic Forgetting

  • Fine-tuning on narrow robotics data can cause catastrophic forgetting of a base VLM’s general-purpose knowledge and impair generalization beyond that training scope.The paper describes this as compromising the VLM’s visual and semantic understanding.Source: §2.2 · p. 4

2.2.1 Co-Training for Knowledge Retention

  • Co-training mixes robot teleoperation with non-robotic data to preserve general reasoning and visual understanding, but requires costly tuning of the data mixture.Finding a useful robotics-to-web-data ratio requires training runs and real-world evaluation, which can be impractical for billion-parameter models.Source: §2.2.1 · p. 4

2.2.2 Advanced Training Schemes

  • Other retention strategies use stop-gradient operators to shield VLMs from action-module updates or train with frozen VLM weights and an action expert.VLM2VLA argues that language-based actions and LoRA can avoid co-training and major architectural changes.Source: §2.2.2 · p. 4

3 Methodology

VLM2VLA aligns robot teleoperation data with a VLM’s representational space by translating it into natural-language descriptions, then trains the policy with LoRA. Its methodology organizes action prediction hierarchically, verifies progress at inference, and constructs the language-annotated dataset from robot trajectories.

3.1 Actions as Language: A Hierarchical Representation

  • Action prediction uses three language-based stages: predict a subtask, describe a spatial motion plan, then generate a variable-length action chunk for execution.The plans specify directional end-effector movements, and this work represents only translational degrees of freedom in action chunks.Source: §3.1 · p. 5

3.1.1 Inference-Time Procedure

  • At inference, the policy fixes all subtasks from the initial observation and uses a verifier after each action cycle to retry or advance.Gemini 2.5 Pro serves as the verifier.Source: §3.1.1 · p. 5

3.2 Data Curation: Translating Robot Trajectories into a Hierarchical Format

  • The data-curation process assumes instructed human teleoperation trajectories but not joint-angle or absolute-position state, then uses Gemini to annotate each step with an observation, subtask, motion plan, and action chunk.Gemini receives the robot base-frame coordinate system to ground relative end-effector movement descriptions; Figure 4 depicts this annotation pipeline.Source: Figure 4 · p. 5, §3.2 · p. 6

3 Methodology

  • The pipeline converts teleoperated trajectories into image-text training examples and fine-tunes Gemma-3-12B-IT with LoRA to support action prediction while preserving VLM world knowledge.Gemini annotates trajectories with subtasks, motion plans, and action chunks; the methodology posits that this data transformation minimizes distribution shift.Source: §3 · p. 4, §3.2 · p. 6

4 Experiments

VLM2VLA largely preserves multimodal understanding and shows its clearest advantages on out-of-distribution manipulation tasks. Its language-based action representation outperforms the token-based ablation as task complexity increases.

4 Experiments

  • Experiments test VQA retention, in-distribution manipulation, and out-of-distribution reasoning against OpenVLA and ECoT, including a same-data action-token ablation.The baselines were selected because they run on the authors’ hardware without additional fine-tuning.Source: §4 · p. 6

4.1 Evaluation of Multimodal Understanding

  • VLM2VLA has only minor losses across VQA benchmarks, outperforms MolmoAct on all reported benchmarks, and the results show co-training does not guarantee retention.The authors note model size confounds the MolmoAct comparison; π0.5 performs lower on most VQA benchmarks.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Evaluating Robotic Manipulation

  • Real-world evaluations cover two in-distribution tasks, one borderline in-distribution task, and two out-of-distribution tasks involving pick-up or pick-and-place.The tasks probe action prediction over short and long horizons and generalization beyond fine-tuning data.Source: §4.2 · p. 8

4.2.1 Results of Baselines

  • OpenVLA performs best on in-distribution pick-up and pick-and-place, while VLM2VLA achieves comparable success rates.The authors attribute OpenVLA’s performance likely to its larger Open-X-Embodiment training set and describe VLM2VLA’s results as competitive in a specific robot embodiment.Source: §4.2.1 · p. 8
  • VLM2VLA significantly outperforms OpenVLA and ECoT on multilingual pick-up, and is the only model with meaningful success on the Ash Ketchum task.The multilingual example shows it identifying the instructed object amid distractors; the Ash Ketchum task combines open-world semantic knowledge and spatial reasoning.Source: Figure 5 · p. 7, Figure 6 · p. 8, §4.2.1 · p. 9

4.2.2 Results of Ablations

  • 60% versus 30% success rate in the Ash Ketchum task for VLM2VLA and VLM2VLA-AT, respectively; the token-based variant degrades as task complexity increases.The authors report VLM2VLA-AT also struggles with multilingual commands despite retaining strong VQA capabilities.Source: §4.2.2 · p. 9

5 Conclusion and Discussion

VLM2VLA uses language-grounded actions and LoRA to adapt VLMs for robot control while retaining foundational capabilities, with zero-shot generalization to novel tasks requiring open-world semantic reasoning. The discussion identifies boundaries in inference speed, dexterous and cross-embodiment control, verifier use, and training scale.

5 Conclusion and Discussion

  • VLM2VLA grounds action prediction in natural language so LoRA can train manipulation policies without significantly degrading the backbone VLM’s foundational capabilities.The approach avoids major architectural changes and expensive co-training; retained world knowledge supports zero-shot generalization to novel tasks requiring open-world semantic reasoning.Source: §5 · p. 9

5.1 Limitations and Future Work

  • Autoregressive action generation is slow: one action-generation cycle took a median 6.1 seconds, with high variance.The authors suggest exploring faster decoding that preserves pretrained representations.Source: §5.1 · pp. 9–10
  • The focus on translational end-effector control suits simple pick-and-place tasks but precludes dexterous manipulation requiring nuanced actions such as rotation.The motion-plan granularity is also coarse; the authors identify fine-grained language annotations as a next step.Source: §5.1 · p. 10
  • Training was limited to one robot embodiment, so the policy does not readily adapt to robots using different low-level controls such as joint angles.The authors suggest language-based relabeling could potentially support cross-embodiment training.Source: §5.1 · p. 10
  • A separate verifier transitions between subtasks and further slows inference; future work could train the base VLM to verify or reason about completion from observations and task descriptions.Source: §5.1 · p. 10
  • The authors are working to scale training with larger robotics datasets and hypothesize that this may strengthen zero-shot generalization, open-world semantic reasoning, and instruction following.Source: §5.1 · p. 10

A Data Curation

The data-curation appendix describes prompting Gemini to turn robot trajectories into language-annotated steps with associated action sequences. Coordinate conventions and trajectory context provide spatial grounding for the annotations.

A.1 Prompting

  • Gemini receives robot observations and actions, the task instruction, and robot coordinate conventions as context for generating language-annotated trajectories.The prompt specifies a WidowX 250S embodiment and translational degrees of freedom; its invariant text is shared across trajectories while designated phrases vary with each trajectory.Source: §A.1 · p. 16
  • The prompt asks Gemini to reason through action phases, then output a structured JSON sequence pairing each step description with its actions.The example decomposes opening a pot into steps such as grasping and lifting the lid, with reasoning and movements associated with each step.Source: §A.1 · pp. 16–18
  • The base-frame coordinate system grounds relative end-effector movements and helps Gemini create annotations and choose subtrajectory boundaries.The authors manually inspected a random subset for accuracy in subtask decomposition, motion planning, and action prediction.Source: §A.1 · pp. 16–17
  • The authors initially used Gemini 2.5 Pro, then switched to Gemini 2.5 Flash to reduce costs; generating the dataset cost roughly $900.Source: §A.1 · p. 17

A.2 Data Preparation

Data preparation chunks actions to produce larger-magnitude predictions and adds verifier and movement-direction training examples. The action-token ablation instead maps digits to Gemma-3’s least-likely tokens.

A.2 Data Preparation

  • Chunking actions along one dimension to increase their magnitude addressed negligible action predictions despite sensible motion plans.The post-processing uses a 2.5-centimeter threshold along a single dimension without sign change and a 5-centimeter absolute threshold.Source: §A.2 · p. 19
  • The data includes positive and negative subtask-completion examples, but the model could not reliably perform verification; a direction-prediction auxiliary signal improved downstream performance.Direction labels include left, right, or none and use a 2.5-centimeter movement threshold; this signal is not used at run time.Source: §A.2 · p. 20

A.3 Actions as Tokens

B VQA Experiment Details

The VQA experiments use a standardized evaluation framework and compare VLM2VLA with several VLA and original VLM baselines. Multi-view models are evaluated on single-image benchmarks using only the primary image input.

B VQA Experiment Details

  • The evaluation pipeline uses lmms-eval for standardized multimodal benchmarks, metrics, and inference wrappers.Source: B VQA Experiment Details · p. 21
  • The reported models include Gemma-3-12B-finetuned VLM2VLA, OpenVLA and ECoT with Prismatic-7B backbones, and original instruction-tuned Gemma-3-4B and Gemma-3-12B.Source: B VQA Experiment Details · p. 21
  • For single-image VQA benchmarks, MolmoAct and π0.5 use only the primary image input, with their other image inputs masked; π0.5 experiments use the π0.5-base checkpoint.Source: B VQA Experiment Details · p. 21

C.1 Experiments and Manipulation Evaluation Metrics

The appendix evaluates policies on in-distribution, borderline in-distribution, and out-of-distribution manipulation tasks, with scoring designed to account for planning and non-planning outputs.

C.1 Experiments and Manipulation Evaluation Metrics

  • Policies were evaluated on two in-distribution tasks, one borderline in-distribution task, and two out-of-distribution tasks, with 30 roll-outs per task.The in-distribution tasks include picking up a carrot and placing it on a yellow plate; the borderline task combines eggplant placement with lifting a fish.Source: §C.1 · pp. 21–22
  • Scoring gives planning policies credit for correct task decomposition as well as execution, while non-planning policies receive credit for execution milestones.For the eggplant–pan–fish task, the rubric awards planning credit for the correct primitives and then scores progress through object contact, placement, and lifting.Source: §C.1 · p. 22
  • The out-of-distribution tasks test multilingual carrot instructions and identifying an object above an unseen Ash Ketchum image.The multilingual commands are Spanish, Mandarin, or Hindi; distractors are included in the multilingual task, and target positions vary across trials.Source: §C.1 · p. 22
  • For the multilingual carrot and Ash Ketchum tasks, planning-policy partial credit requires naming the correct target and making contact; non-planning partial credit requires contact.The carrot task awards partial credit for carrot contact, while the Ash Ketchum task requires contact with the correct object.Source: §C.1 · pp. 22–23

C.2 Hardware Details

  • Real-world evaluations used a 6-DoF WidowX 250S arm, a side-mounted Realsense D435 camera, varied object starting conditions, and a safety filter.The toy-kitchen setup used Bridgev2 objects; the filter blocked further actions that would drive the robot into the counter or wall.Source: §C.2 · p. 23

C.3 Task Decomposition Evaluation Metrics

The appendix specifies keyword-based scoring for task decomposition, a staged prompting process for actions, and a verifier that checks whether each subtask succeeds sufficiently for the next one.

C.3 Task Decomposition Evaluation Metrics

  • Reasoning succeeds when a model's textual output contains the required task keywords, with case-insensitive matching and manual checks for fairness.For the eggplant–pan–fish task, success requires all three keywords or synonyms; the Ash Ketchum task requires naming the correct object for that trial.Source: §C.3 · p. 23
  • VLM2VLA sometimes used the Hindi word “gajar” for carrot, but this direct translation behavior was not observed for Spanish or Mandarin inputs.The multilingual task's reasoning score required a carrot synonym in the model output.Source: §C.3 · p. 23

C.4 Test-Time Prompting

  • Test-time prompting proceeds from instruction decomposition to spatial reasoning for a subtask, then to low-level action prediction using the motion plan and current observation.The action prompt encodes directional conventions and gripper states; generation uses top-p sampling with stage-specific temperatures.Source: §C.4 · pp. 23–24

C.5 Verifier Prompting

  • A closed-loop verifier evaluates before-and-after observations alongside current and next subtasks, and its judgment determines the next action-generation cycle.The verifier is Gemini 2.5 Pro, though the authors note the model itself could be trained to perform this step.Source: §C.5 · p. 24
  • The verifier judges whether a move establishes the precondition for the next step, requiring alignment for grasping rather than merely being close enough.A successful grasp also requires the gripper to be centered above the object's grasp point in the preceding move.Source: §C.5 · p. 25

C.6 Inference Latency

Inference latency was measured over 30 runs on an NVIDIA A100; the median was low, but variability and long outliers point to runtime bottlenecks.

C.6 Inference Latency

  • 6.1 seconds median inference latency was reported across 30 runs, while the standard deviation was 14.3 seconds.A single inference cycle includes both the mid-level reasoning trace and low-level action prediction.Source: §C.6 · p. 25, Table 3 · p. 25
  • Approximately 10% of cases triggered retries for bad output formatting, and a small subset of trials took more than 45 seconds.The authors identify these cases as contributors to runtime variability and suggest a computational bottleneck.Source: §C.6 · p. 25
  • The authors identify faster decoding schemes for low-latency real-world performance as future work.Source: §C.6 · p. 25

D Training Details

VLM2VLA is fine-tuned from Gemma-3-12B-IT using LoRA, with a single-epoch training run on four A100 GPUs.

D Training Details

  • Gemma-3-12B-IT was fine-tuned using parameter-efficient LoRA updates.Training used the TRL library, Accelerate, and DeepSpeed ZeRO Stage 2 for multi-GPU memory efficiency.Source: D Training Details · p. 26
  • The final checkpoint used one epoch of training on a single node with four NVIDIA A100 GPUs and required approximately 300 GPU hours.The authors report that one epoch was sufficient for this work.Source: D Training Details · p. 26, Table 4 · p. 26

Related papers

Loading 2509.22195v1…