EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

Yiyang Du; Zhanqiu Guo; Xin Ye; Liu Ren; Chenyan Xiong

Source-linked AI summary

EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong

arXiv:2604.20012v1Apr 21, 2026cs.CVcs.AIcs.CL

TL;DR

Most VLAs start from general-purpose VLMs whose training distributions are not tailored to embodied action generation, leaving a gap between VLM pretraining and VLA fine-tuning. EmbodiedMidtrain measures this gap, selects VLA-aligned VLM samples with a learnable proximity estimator, and mid-trains before VLA fine-tuning; across three benchmarks, it consistently improves performance and provides a stronger initialization.

Problem
Most VLAs inherit general-purpose VLM representations whose broad pretraining distribution is mismatched with robotic manipulation data and embodied action generation.
Method
EmbodiedMidtrain uses a lightweight proximity estimator to rank VLM samples by closeness to the VLA domain and mid-trains the VLM on the selected mixture.
Results
Across three manipulation benchmarks, proximity-based mid-training consistently improves performance across backbones, with results competitive with larger models and expert VLAs.
Takeaways & Limitations
Mid-training provides a stronger initialization whose advantage appears from the earliest VLA fine-tuning stages and grows throughout training.

Abstract

from arXiv · show

Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and mid-trains the VLM on this curated mixture before downstream VLA fine-tuning. Experiments on three robot manipulation benchmarks show that mid-training consistently improves performance across different VLM backbones, achieving results competitive with expert VLAs and off-the-shelf VLMs trained with larger model scale and training budgets. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.

1 Introduction

EmbodiedMidtrain addresses the mismatch between broad VLM pretraining and embodied VLA learning by selecting VLA-aligned VLM data for mid-training. Across three manipulation benchmarks and multiple backbones, this produces stronger VLA initialization and consistent performance gains.

Motivation: Most VLAs use general-purpose VLM backbones whose broad pretraining does not specifically target embodied action generation.VLM data include captioning, visual question answering, and document understanding, whereas VLA training uses robotic manipulation trajectories.
Motivation: VLA data form compact clusters separated from broad VLM distributions, although some VLM samples are more aligned with the embodied domain.This heterogeneous alignment motivates selecting individual samples rather than treating all VLM data uniformly.
Results: Mid-training yields consistent performance gains across Calvin ABC-D, SimplerEnv Bridge, and Libero-10, including results competitive with substantially larger models.The reported gains hold across different VLM backbones and benchmarks.
Results: Mid-training advantages appear from the earliest VLA fine-tuning steps and widen over time, indicating a stronger initialization rather than a transient head start.The learned proximity estimator also outperforms hand-crafted alternatives by capturing fine-grained alignment.
Approach: EmbodiedMidtrain learns proximity to the VLA domain and selects top-ranked VLM candidates to construct a distribution-aligned mid-training mixture.The proximity-based data engine adapts the VLM before downstream VLA fine-tuning.

2 Related work

Prior work extends VLMs toward embodied capabilities through VLA architectures, VLM mid-training, and embodied-oriented data or model adaptation. However, improvements from embodied VLM fine-tuning do not reliably transfer to downstream VLA performance.

Vision-Language-Action Models: VLAs extend VLMs to generate robot actions while differing in backbone choice and action-generation mechanism.Token-based approaches discretize robot actions into tokens for autoregressive generation.
VLM Mid-training: VLM mid-training is an intermediate training stage that adapts foundation models toward desired domains or capabilities before final post-training or task-specific fine-tuning.This paper studies mid-training as a way to bridge general VLM pretraining data and VLA fine-tuning data.
Embodied-oriented VLMs: Embodied-oriented VLM research combines embodied dataset construction with model-level adaptation, but reported gains do not reliably transfer to downstream VLA tasks.This transfer problem motivates a more targeted adaptation strategy.

3 Data distribution gap between VLMs and VLAs

The VLM and VLA training distributions are substantially separated: VLA datasets are compact and concentrated, while VLM data are broader and more dispersed. Alignment is nevertheless heterogeneous within the VLM distribution, motivating sample-wise selection toward the VLA domain.

Distribution analysis: VLM and VLA data are analyzed in a shared representation space to characterize their distribution gap quantitatively and qualitatively.The analysis uses VLM last hidden states as sample features.
Distribution analysis: Pairwise MMD distances are generally smaller within VLM and within VLA groups than across groups, confirming a clear distributional mismatch.The study reports globally normalized pairwise MMD scores across dataset pairs.
Distribution analysis: VLA datasets form compact clusters mostly detached from the main VLM regions, with only a small subset of VLM data nearby.t-SNE visualization further illustrates the separation and local proximity.
Heterogeneity: Alignment varies substantially across VLM sources and local regions, making the VLM–VLA relationship a spectrum rather than a binary distinction.Some VLM sources lie noticeably closer to VLA domains despite global separation.
Implication: These findings motivate sample-wise selection within datasets instead of only coarse dataset-level mixture adjustment.The proposed strategy prioritizes VLM samples most compatible with the target embodied domain.

4 Data engine for EmbodiedMidtrain

EmbodiedMidtrain selects VLM samples that are most aligned with the VLA domain, using a lightweight proximity estimator trained on frozen VLM features, then mid-trains on the curated subset. This preserves VLM-data diversity while shifting the training distribution toward embodied adaptation.

Selection combines general-purpose and embodied-oriented VLM sources while operating at the individual-sample level.These design choices aim to preserve diversity and capture variation in alignment within datasets.
The data engine selects VLM samples whose distribution best aligns with the target VLA domain, rather than training indiscriminately on the full candidate pool.The curated subset preserves useful VLM diversity while concentrating on samples most relevant to embodied adaptation.
The method frames proximity scoring as a domain-membership problem and uses a binary classifier on frozen VLM features to distinguish VLA from VLM samples.At optimality, the classifier recovers the density ratio pVLA(x)/pVLM(x), so classifier-output ranking matches density-ratio ranking.
The proximity estimator applies a learnable scoring function to frozen VLM representations, followed by a sigmoid, and is trained with VLA positives and VLM negatives.Binary cross-entropy is used as the training objective.
After scoring all candidates, the engine retains the top-K samples to form a targeted mid-training corpus for subsequent VLM adaptation.This procedure converts the broad candidate pool into a distribution-shifted corpus while preserving useful diversity.

5 Experiments

Experiments evaluate proximity-based mid-training across three simulated robot-manipulation benchmarks and multiple VLM and VLA baselines. Mid-training consistently improves downstream performance, competes with larger systems, and transfers across VLM backbones.

5.1 Setup: The candidate pool combines diverse general-purpose and embodied-oriented VLM datasets for proximity-based selection and mid-training.The reported pipeline mid-trains InternVL3.5-1B and Qwen3VL-2B before VLA fine-tuning.
5.1 Setup: Experiments test the data engine across Calvin ABC-D, SimplerEnv Bridge, and Libero-10 using multiple VLM backbones and VLA baselines.The benchmarks cover unseen-scene generalization, randomized tabletop manipulation, and long-horizon tasks.
5.2 Baselines: Baselines include expert VLAs such as OpenVLA and π0, alongside VLAs fine-tuned from off-the-shelf VLMs spanning architectures and scales.Model size and total samples seen are reported to assess performance and training efficiency.
5.3 Main results: Mid-training produces consistent and substantial improvements across all three benchmarks, strengthening VLM readiness for downstream action learning.The main results are reported in Table 1.
5.3 Main results: The mid-trained InternVL3.5-1B surpasses both expert VLA baselines on Calvin ABC-D and matches much larger VLMs on SimplerEnv-Bridge and Libero-10.It also outperforms several off-the-shelf VLMs that are 3–8× larger while using a fraction of their training budgets.
5.3 Main results: The same data selected with InternVL3.5-1B features yields clear gains for Qwen3VL-2B across all three benchmarks.This indicates that the selected mixture captures alignment properties not specific to one backbone.

6 Analysis

The analysis shows that proximity-based selection, rather than random sampling or hand-crafted proximity measures, drives mid-training gains. Benefits appear early in VLA fine-tuning and widen over time, while selected data favor embodied visual reasoning and preserve diversity.

6.1 Ablations: Random selection underperforms the learned estimator across all three benchmarks, showing that additional unfiltered mid-training is insufficient.The results identify proximity-based retention of VLA-aligned VLM samples as critical to the benefit of mid-training.
6.1 Ablations: Hand-crafted proximity measures are less consistent than the learned estimator, only partially capturing VLA alignment.The alternatives include feature-space average distance, VLA-conditioned perplexity, and delta perplexity.
6.2 Training dynamics: Mid-trained models outperform the original backbone from early fine-tuning checkpoints, with the performance gap widening throughout training.Training loss remains highly similar between initializations, so loss alone does not fully reflect initialization quality.
6.3 Analysis of selected VLM data: Proximity scores vary across eight VLM datasets and within datasets, enabling both dataset-level preferences and fine-grained sample selection.RefSpatial has the highest average scores, while VCR has the lowest.
6.3 Analysis of selected VLM data: High-scoring samples require spatial grounding and reasoning, whereas low-scoring book-cover samples rely on text-only visual question answering.The estimator therefore distinguishes reasoning patterns relevant to embodied manipulation from less transferable text-centric content.
6.3 Analysis of selected VLM data: Selection shifts the proximity distribution toward higher scores while retaining diversity close to that of the original general VLM pool.The selected mixture removes part of the low-score mass without collapsing into a narrow embodied-data region.

7 Conclusion

The conclusion presents EmbodiedMidtrain as a proximity-based bridge between VLM and VLA data distributions. Across three manipulation benchmarks, it consistently improves performance, strengthens VLA initialization from the earliest steps, and preserves data diversity.

7 Conclusion: EmbodiedMidtrain bridges the VLM–VLA distribution gap through proximity-based selection using a lightweight estimator on frozen VLM features.The selected samples form the mid-training set used before downstream VLA fine-tuning.
7 Conclusion: Across three manipulation benchmarks, proximity-based mid-training consistently improves performance, reaching results competitive with larger-scale models and training budgets.The selected data also transfer to Qwen3VL-2B despite being curated with InternVL3.5-1B features.
7 Conclusion: Mid-training yields a stronger VLA initialization whose advantage appears early and grows throughout fine-tuning.Selected data show dataset- and sample-level preferences while preserving the diversity of the original VLM dataset.

A.1 Implementation Details for VLM Mid-training

The VLM mid-training implementation uses a lightweight proximity classifier over frozen VLM representations, followed by full-parameter supervised fine-tuning of InternVL3.5-1B.

A.1 Implementation Details for VLM Mid-training: The proximity estimator is a linear scoring function trained on frozen VLM representations with balanced VLM-candidate and target-VLA data.Training uses batch size 128 and typically stops after 75–100 steps through early stopping.
A.1 Implementation Details for VLM Mid-training: VLM mid-training uses LLaMA-Factory for full-parameter supervised fine-tuning of InternVL3.5-1B.The vision encoder, multimodal projector, and language model are all unfrozen.
A.1 Implementation Details for VLM Mid-training: Table 3 provides the key hyperparameters for VLM mid-training.

A.2 Implementation Details for VLA Fine-tuning

The VLA fine-tuning setup follows the VLM4VLA design and evaluates end-to-end models using single-view images without robot-state inputs.

A.2 Implementation Details for VLA Fine-tuning: Downstream VLA adaptation follows the VLM4VLA design and evaluation protocol.Models use a single-view image and are fine-tuned end-to-end.
A.2 Implementation Details for VLA Fine-tuning: Baseline results for expert VLAs and off-the-shelf VLMs are taken directly from VLM4VLA rather than rerun.Model-specific training details and hyperparameters are referred to that work.
A.2 Implementation Details for VLA Fine-tuning: The VLA maps multimodal VLM representations to latent control features and decodes them into chunked robot actions with an MLP action head.Continuous arm actions use Huber loss, while binary gripper actions use binary cross-entropy.

A.3 Detailed Experimental Results

The experiments analyze alternative proximity measurements, diversity preservation, mixture composition, and the effect of mid-training on the VLM’s original capabilities. Results indicate that selected data remains diverse, is concentrated in particular source datasets, and selectively reorients benchmark performance.

Proximity measurements: Learned proximity estimation is compared with feature-space distance, VLA-conditioned perplexity, and delta perplexity as alternatives for selecting VLM samples.The supplied passages identify these three alternative measurements and note that their formal definitions are provided for comparison with the learnable estimator.
Diversity preservation: Selected VLM data achieves diversity 1.93, close to the full general VLM pool at 1.96 and above embodied-oriented VLM data at 1.62 and VLA data at 1.26.The diversity metric uses feature representations from the frozen VLM’s last hidden state space, with t = 2 in the calculation.
Selected-mixture composition: LAION-400M contributes 32.0% of the selected mixture, followed by LLaVA-Instruct-665k at 20.2% and RoboPoint at 19.9%.RefSpatial contributes 14.7%, Robo2VLM 9.2%, and CC-12M 3.7%; EmbSpatial-Bench and VCR contribute 0.1% and 0.0%.
Selected-mixture composition: LAION-400M’s dominance reflects its scale: sample-level selection finds a useful subset from a massive source rather than discarding the source through dataset-level filtering.This distinguishes source-level composition from the alignment of every sample in that source.
VLM capability profile: Mid-training largely preserves the original VLM capability profile while selectively improving VisuLogic and 3DSRBench, leaving POPE nearly unchanged and moderately decreasing BLINK and SpatialEval.The reported results are from InternVL3.5-1B evaluated on VLM benchmarks before and after mid-training.