CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models

Wenxuan Song; Han Zhao; Fuhao Li; Ziyang Zhou; Xi Wang; Jing Lyu; Pengxiang Ding; Yan Wang; Donglin Wang; Haoang Li

Source-linked AI summary

CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models

Wenxuan Song, Han Zhao, Fuhao Li, Ziyang Zhou, Xi Wang, Jing Lyu, Pengxiang Ding, Yan Wang, Donglin Wang, Haoang Li

arXiv:2605.10903v1May 11, 2026cs.CVcs.RO

TL;DR

Pretrained VLA models may adapt insufficiently under standard SFT, while auxiliary-objective finetuning adds substantial computational overhead. CapVector separates general capability gains from task-specific action fitting through parameter differences and model merging, then uses orthogonal regularization during downstream training. The resulting approach is reported to match auxiliary-objective SFT's performance and efficiency across diverse models, tasks, environments, and embodiments.

Problem
Pretrained VLA models often adapt insufficiently to complex downstream tasks with standard SFT, whereas auxiliary objectives add computational overhead.
Method
CapVector extracts the parameter difference between auxiliary-objective and standard SFT models, merges it with pretrained parameters, and applies orthogonal regularization during downstream training.
Results
CapVector achieves performance and training efficiency comparable to auxiliary-objective SFT across multiple downstream tasks, diverse VLA architectures, and SFT strategies.
Takeaways & Limitations
The method provides a strategy for transferring finetuning gains into pretrained models while retaining standard SFT's simplicity and computational efficiency.
Takeaways & Limitations
The extraction assumes task-relevant parameter changes are approximately the same between auxiliary-objective and standard SFT under consistent finetuning settings.

Abstract

from arXiv · show

This paper proposes a novel approach to address the challenge that pretrained VLA models often fail to effectively improve performance and reduce adaptation costs during standard supervised finetuning (SFT). Some advanced finetuning methods with auxiliary training objectives can improve performance and reduce the number of convergence steps. However, they typically incur significant computational overhead due to the additional losses from auxiliary objectives. To simultaneously achieve the enhanced capabilities of auxiliary training with the simplicity of standard SFT, we decouple the two objectives of auxiliary-objective SFT within the parameter space, namely, enhancing general capabilities and fitting task-specific action distributions. To deliver the goal, we only need to train the model to converge on a small-scale task set using two distinct training strategies, resulting in two finetuned models. The parameters' difference between the two models can then be interpreted as capability vectors provided by auxiliary objectives. These vectors are then merged with pretrained parameters to form a capability-enhanced meta model. Moreover, when standard SFT is augmented with a lightweight orthogonal regularization loss, the merged model attains performance comparable to auxiliary finetuned baselines with reduced computational overhead. Internal and external experiments demonstrate that our capability vectors (1) are effective and versatile across diverse models, (2) can generalize to novel environments and embodiments out of the box.

1 Introduction

CapVector addresses the computational cost of auxiliary-objective finetuning by separating general capability gains from task-specific action fitting in parameter space. It extracts capability vectors from paired finetuned models, merges them with pretrained parameters, and supports efficient downstream adaptation.

Standard SFT often fails to adapt pretrained VLA models quickly or outperform training from scratch on complex downstream tasks.
Auxiliary objectives improve foundational capabilities and downstream performance, but extra modules and forward passes create prohibitive computational overhead as tasks and data scale.
CapVector interprets the parameter difference between auxiliary-objective SFT and standard SFT on the same task as a transferable capability vector.
Merging capability vectors with pretrained parameters produces a capability-enhanced meta model intended to transfer general capabilities to downstream tasks.
Extensive experiments report effectiveness and efficiency across tasks, environments, models, VLA architectures, and SFT strategies, including generalization to novel environments and embodiments.
Adding a lightweight orthogonal regularization loss to standard SFT preserves the simplicity of standard SFT while targeting the performance of auxiliary-objective finetuning.

2 Capability Vectors (CapVector)

CapVector separates capability gains from task-specific action fitting by comparing auxiliary-objective and standard SFT models, then transfers the difference into a pretrained model. Orthogonal regularization helps retain these properties during downstream finetuning with slight overhead.

Capability-vector transfer: These vectors are merged with pretrained parameters to form a capability-enhanced meta model that provides a better initialization for new tasks.The meta model is constructed before downstream training by transferring the extracted capability vectors.
Capability-vector extraction: The method obtains capability vectors by subtracting a standard-SFT model from an auxiliary-objective-SFT model trained under consistent settings.The task-relevant parameter changes are assumed approximately equal, so their difference isolates auxiliary-objective capability changes.
Orthogonal regularization: Standard SFT can damage transferred properties because its parameter updates occur in the same parametric space, especially with more training steps.Without auxiliary supervision, downstream updates may interfere with the capability vectors.
Orthogonal regularization: Orthogonal regularization constrains downstream task updates to remain orthogonal to capability vectors, preventing interference during finetuning.The method treats parameter changes as model update directions and defines orthogonality between capability and task vectors.
Orthogonal regularization: The total loss combines action loss with a weighted orthogonality loss, while LoRA tuning computes the latter only on matrix A.The added overhead is described as slight; matrix A represents the update direction, whereas matrix B provides linear weighting coefficients.

3 Experiments

Experiments show that CapVector transfers auxiliary capabilities across tasks, models, environments, and embodiments while retaining efficient standard-SFT adaptation. Orthogonal regularization preserves these injected capabilities, and diverse task-irrelevant visual features improve vector quality.

3.2 In-distribution (ID) Study (RQ1): CapVector matches or exceeds Spatial Forcing across training steps and tasks, while outperforming OpenVLA-OFT after only 5k training steps.These results support parameter merging as a way to transfer auxiliary spatial capabilities without auxiliary-objective training.
3.2 In-distribution (ID) Study (RQ1): Orthogonal loss mitigates capability degradation during extended standard finetuning and keeps CapVector superior to Spatial Forcing at 150k steps.Without the loss, injected capabilities are updated and reduced during regular finetuning; the loss constrains updates along new directions.
3.3 Out-of-distribution (OOD) Study (RQ2): CapVector transfers capabilities to unseen distributions, improving RoboTwin success rates from 6.7% to 31.8% with OpenVLA-OFT and approaching Spatial Forcing.The experiments cover different architectures, extraction datasets, and merging strategies, supporting task-agnostic transfer across simulated environments.
3.4 Versatility Study (RQ3): CapVector transfers diverse capabilities across auxiliary methods and architectures, including geometric comprehension, multimodal reasoning, autoregressive models, and flow-matching models.On StarVLA, it outperforms standard StarVLA and performs comparably to full LaRA-VLA; on π0.5, merging both VLM and action-expert parameters performs relatively better.
3.5 Determinants of Capability Vector Quality (RQ4): Randomized backgrounds produce higher-quality capability vectors than clean backgrounds under matched data scale and task count, whereas task-relevant visual cues can induce shortcut learning.The reported success-rate ordering across datasets is SRSpatial > SRLong > SR90 despite comparable pairs per task, implicating data disparity as an additional determinant.
3.6 Real-world Study (RQ5): Simulation-derived capability vectors improve real-world performance across tasks and embodiments, including raising four-test-tube transfer success from 0.36 to 0.53.The vectors capture fundamental geometric cues rather than domain-specific textures, enabling transfer from scalable simulation data without real-world auxiliary objectives.

4 Related Work

Prior work improves VLA adaptation through optimized SFT, auxiliary objectives, and model merging, but auxiliary objectives often add substantial computation. CapVector instead avoids this overhead through capability extraction and model merging.

SFT Strategies for VLAs: Auxiliary-objective methods strengthen foundational capabilities such as spatial alignment and multimodal reasoning, but typically require additional forward passes.
Model Merging: CapVector avoids auxiliary-objective training overhead by extracting capability vectors and merging them into pretrained models.
Model Merging: Model merging combines parameters from distinct models to reuse knowledge, improve robustness, combine skills, and mitigate distribution shifts.

5 Conclusion

The paper introduces capability vectors and a pipeline for integrating them into pretrained VLA models. Experiments report effectiveness, versatility, computational efficiency, and real-world generalization across models, strategies, environments, and embodiments.

Capability vectors represent gains in general capabilities acquired during finetuning and are integrated with pretrained models to construct capability-enhanced meta models.
CapVector combines the simplicity and computational efficiency of standard SFT with the high performance of auxiliary-objective SFT.
Experiments analyze capability-vector effectiveness, versatility, and mechanisms across diverse VLA models and SFT strategies.
Real-world experiments validate practical applicability and generalization, including transferring finetuning gains into pretrained models.

A More Ablations.

The ablation studies examine the capability-vector merging weight and report the setting used in subsequent experiments. The best reported performance occurs at α = 1.1.

α controls how strongly the extracted general capabilities γao are integrated into the pretrained backbone θpt to form θmeta.
The model achieves optimal performance at α = 1.1, which is selected as the default for subsequent experiments.
Table S1 reports success rates for varying capability-vector merging weights, using LIBERO-Spatial for extraction and LIBERO-Long for downstream evaluation.

B Extra Overhead Induced by Orthogonal Loss

The paper evaluates the orthogonal loss overhead using training FLOPs and GPU memory. The added cost is negligible while preserving capability-vector properties during downstream finetuning.

Table S2 reports training FLOPs and GPU memory for the orthogonal loss under LoRA tuning of OpenVLA-OFT.
The orthogonal loss preserves capability-vector capabilities during downstream finetuning by regularizing updates in orthogonal subspaces.
0.3G (<0.002%) additional training FLOPs are introduced by the orthogonal loss.
Approximately 0.5G (<0.8%) additional GPU memory is required by the orthogonal loss.

C Extra Overhead Comparison between Auxiliary-objective SFT Methods and Ours

Table S3 compares the computational overhead of auxiliary-objective SFT with CapVector-based training. Spatial Forcing adds substantial cost, whereas CapVector has negligible overhead while maintaining comparable or better performance.

Spatial Forcing adds 5.0T training FLOPs and 10.9G GPU memory to the OpenVLA-OFT baseline.These increases correspond to 28% more FLOPs and 17% more GPU memory.
CapVector introduces negligible computational overhead compared with auxiliary-objective finetuning.
CapVector achieves performance comparable to, or exceeding, full auxiliary-objective finetuning across diverse tasks.

D Real-world Setup.

The real-world evaluation uses three hardware platforms to assess task performance and cross-embodiment generalization. The ARX Lift 2 and AgileX Cobot setups provide distinct bimanual manipulation environments.

The experiments deploy models across three hardware platforms to evaluate real-world performance and cross-embodiment generalization.
The real-world robot and task setups for two external collaborators are shown in Figure S1.
ARX Lift 2: ARX Lift 2 uses a 6-DoF dual-arm system with D405 cameras for complex, multi-stage manipulation tasks.Its workspace includes a test tube rack, test tubes, toolbox, tray, and power strip.
AgileX Cobot: AgileX Cobot uses teleoperation-ready master and puppet 6-DoF dual arms for everyday dexterous tasks.Its workspace includes a dish rack, plate, and sponge.

E Limitations.

The study focuses on capability vectors obtained through supervised fine-tuning and leaves reinforcement-learning settings for future work.

Capability vectors are investigated only in supervised fine-tuning settings, while reinforcement-learning capability vectors remain future work.

F Baselines.

The baselines span optimized finetuning, open-world VLA modeling, modular VLA infrastructure, spatial alignment, and latent multimodal reasoning. Together, they represent diverse approaches to improving robotic control and generalization.

OpenVLA-OFT: OpenVLA-OFT combines parallel decoding, action chunking, continuous actions, and L1 regression to improve task adaptation and inference efficiency.FiLM enhances OFT for tasks requiring fine-grained language comprehension.
OpenVLA-OFT: OpenVLA-OFT reports a 97.1% success rate and 26× faster action generation, with up to 15% absolute average success-rate improvement on dexterous manipulation tasks.
π0.5: π0.5 uses heterogeneous robot, semantic, web-scale multimodal, and language data within a hierarchical architecture for open-world manipulation generalization.
StarVLA: StarVLA provides a modular backbone-action-head framework for interchanging vision-language or world-model backbones with action-decoding paradigms.
Spatial Forcing: Spatial Forcing aligns intermediate visual embeddings with geometric representations from pretrained 3D foundation models to improve spatial reasoning and action precision.It does not require explicit 3D sensors or depth estimators.
LaRA-VLA: LaRA-VLA trains continuous latent representations through a curriculum that transitions from explicit multimodal chain-of-thought supervision to pure latent reasoning.