Source-linked AI summary
Towards Zero-Shot Transfer Across Embodiments For Driving VLAs
Caio Azevedo, Stefano Sabatini, Sascha Hornauer, Fabien Moutarde
TL;DR
Driving VLAs have limited evidence for zero-shot geometric transfer across unseen datasets and camera rigs, and adding datasets does not always improve seen-embodiment performance. The paper studies multi-dataset training and BEV-Forcing, which transfers occupancy-based spatial supervision into the VLA backbone; it finds stronger benefits with few training embodiments and diminishing auxiliary-task gains as rig diversity increases.
Problem
Autonomous-driving VLAs are rarely evaluated for zero-shot transfer across unseen datasets and camera rigs, while multi-dataset additions do not consistently improve performance within seen embodiments.
Method
The paper combines multi-dataset driving-VLA training with BEV-Forcing, supervising backbone image representations using occupancy maps from a specialized BEV model.
Results
BEV-Forcing improves in- and out-of-dataset planning performance with limited camera-rig diversity, but its benefits diminish as training-embodiment diversity increases.
Takeaways & Limitations
Evaluating robustness methods across different numbers of camera rigs can reveal whether their gains persist when training diversity scales.
Takeaways & Limitations
The paper focuses on representation learning with image embeddings and leaves reasoning traces, action representation, architecture, and richer BEV teachers for future work.
Abstract
from arXiv · showhide
Vision-Language-Action models (VLAs) have shown strong potential in autonomous driving by leveraging multimodal pretraining for instruction following, visual reasoning, and scene-level generalization. In robotic manipulation, scaling VLA fine-tuning across multiple robot setups--especially when unifying representations across embodiments--has been shown to improve in-dataset performance and cross-embodiment generalization; in autonomous driving, however, VLAs remain largely trained on individual datasets and are rarely evaluated for zero-shot transfer to unseen datasets and camera rigs; furthermore naively adding more datasets to the training data does not necessarily lead to better performance within seen embodiments. To address these problems, we study multi-dataset training for the driving task and BEV-Forcing, an auxiliary objective that transfers ground-plane object-layout information from a specialized Bird's-Eye-View model into the VLA backbone. By encouraging the model to represent object position through a shared BEV spatial interface, we show that an auxiliary task such as BEV-Forcing can improve both in-distribution and out-of-distribution performance when training on a small number of camera rigs. As the number of training embodiments increases, however, the benefits of the auxiliary task are reduced; we present this as evidence that new techniques in the literature may see their benefits diminish when simply scaling up training diversity, which motivates presenting results taking into account data scaling.
1. Introduction
The paper studies why driving VLAs struggle to transfer spatially grounded trajectories across camera rigs and evaluates multi-embodiment training with BEV-Forcing as an auxiliary spatial objective. It argues that BEV-Forcing can improve performance with limited rig diversity, while motivating evaluation across scaling regimes.
- Motivation: Driving VLAs may retain semantic understanding across datasets yet fail to transfer 3D spatially grounded trajectories across different camera setups.This geometric-transfer problem remains underexplored in autonomous driving, where effective zero-shot transfer could facilitate adaptation to new embodiments and prototype iteration.
- Approach: BEV-Forcing transfers ground-plane object-position awareness into the VLA backbone through an auxiliary objective based on teacher occupancy maps.The approach is presented as inspired by Spatial Forcing and uses a specialized BEV model to provide supervision.
- Research questions: The study investigates seen-rig performance and zero-shot transfer as the number of training embodiments increases.It hypothesizes that transfer may improve when models learn a unified spatial representation from multi-view images across varying camera rigs.
- Contributions: BEV-Forcing can improve in-dataset and out-of-dataset performance when training uses a limited number of camera rigs, without adding inference-time cost.Its auxiliary prediction head can be removed in production, and it does not require annotated maps.
- Contributions: The paper scales training across multiple public datasets and evaluates planning and VQA models on seen embodiments and zero-shot transfer settings.It also reports ablations examining interactions with image augmentation and calibration-parameter inputs.
2. Related Work
Related work distinguishes semantic transfer from geometric cross-embodiment transfer and motivates camera-only spatial supervision for autonomous driving. The paper positions BEV-Forcing as a low-data-cost approach compatible with unified multi-dataset training.
- Spatial awareness for VLAs: Driving VLAs often learn spatial awareness implicitly, which can support strong in-dataset planning but leaves challenging edge cases.Other approaches add LiDAR-aligned inputs or 3D-supervised objectives, increasing modality, data, or training-stage requirements.
- Spatial awareness for VLAs: The paper uses camera inputs only and avoids requiring 3D labels or extra training stages to preserve vision-language alignment and support camera-only datasets.This choice addresses the scarcity and collection cost of 3D data such as LiDAR.
- VLA generalization: Robotic VLA research emphasizes diverse embodiment pretraining, while autonomous-driving research has comparatively limited geometric zero-shot-transfer literature.Driving alternatives such as action-aligned world modeling can improve transfer but require high computation during training and inference.
- VLA generalization: Autonomous driving offers a unified task with similar planned-trajectory outputs across embodiments, potentially supporting multi-dataset scaling despite fewer public camera-rig variations.The paper uses this setting to study transferability across driving embodiments.
3. Methodology
The method adapts a pretrained vision-language model for trajectory prediction and adds a capacity-limited BEV head that supervises image representations with occupancy maps. Training combines the BEV loss with next-token prediction across unified planning and VQA datasets.
- Backbone and Action Representation: The backbone is a pretrained Qwen 3.5 2B vision-language model, while trajectories are represented as text waypoint pairs and fine-tuned with LoRA.Waypoints are sampled at 1 Hz and resampled with cubic splines toward the target rate.
- BEV-Forcing: The BEV head uses randomly initialized spatial-grid queries that attend to image embeddings and output vehicle occupancy maps.Its low capacity encourages planning-relevant image features to encode surrounding-object positions.
- BEV-Forcing: Image embeddings from a target backbone layer are projected to the BEV-head dimension, cross-attended by spatial queries, and linearly mapped to occupancy logits.The head is intentionally capacity-limited, for example using cross-attention instead of a full transformer block, so image tokens retain spatial information.
- Training: BEV occupancy predictions are trained with binary cross-entropy against SimpleBEV teacher occupancy maps, including for datasets without ground-truth maps.The auxiliary BEV loss is added to the standard next-token prediction loss during supervised fine-tuning.
- Training: VQA co-training applies next-token prediction to answers while passing target-layer image states through the same BEV head.Samples from planning and VQA datasets are loaded through a unified contract with task-specific inputs, prompts, and supervision.
4. Experiments
The experiments evaluate driving VLAs across multiple datasets and camera embodiments, measuring both in-dataset planning and zero-shot transfer. BEV-Forcing improves transfer and in-dataset performance most with limited training diversity, while its benefits diminish as more embodiments are added.
- Experimental Setup: Experiments scale training from WOD-E2E to combinations including NAVSIM and nuScenes-QA, evaluating planning on WOD-E2E, Physical AI, and KITScenes.Metrics include ADE, FDE, RFS, and MMS across in-dataset and zero-shot settings.
- Experimental Results: A 10.1% decrease in ADE results from applying BEV-Forcing when training only on WOD-E2E for zero-shot Physical AI transfer.The BEV task provides a geometry-aware auxiliary signal when training uses few camera rigs.
- Experimental Results: On KITScenes LongTail validation, MMS rises from 3.90 to 4.64 with BEV-Forcing for WOD-E2E-only training, but falls from 5.13 to 4.84 when all three datasets are used.The authors attribute the changing effect to increased data variability overshadowing the auxiliary task.
- Experimental Results: On WOD-E2E validation, BEV-Forcing lowers FDE from 5.946 to 5.831 for single-dataset training while slightly increasing RFS from 7.961 to 7.971.Across configurations, the auxiliary task maintains or improves in-dataset planning without an apparent validation trade-off.
- Experimental Results: Adding datasets can reduce the effectiveness of planning techniques, motivating evaluation across multiple camera rigs and at zero-shot transfer rather than only single-dataset training.The experiments also examine linguistic supervision and robustness techniques, including image augmentation and calibration inputs.
- Ablation Studies: Image augmentations and calibration inputs improve zero-shot transfer on Physical AI-1k only when paired with BEV-Forcing.The combination retains in-dataset improvements, although gains over the base model remain modest.
5. Conclusion
The paper presents multi-dataset training and BEV-Forcing for driving VLAs, showing improved planning and zero-shot transfer under limited dataset variability. It also identifies representation learning and spatial-awareness design as areas for future work.
- BEV-Forcing generally improves in-dataset planning and zero-shot transfer when trained on limited dataset variability.The technique acts as a geometry-aware regularizer in this setting.
- BEV-Forcing transfers unified multi-view camera understanding into the VLA backbone through BEV occupancy-map supervision.The auxiliary objective supervises image hidden states at one target VLM layer using occupancy maps from a specialized BEV model.
- The method also improves the effectiveness of common robustness-training techniques.
- The paper recommends evaluating planning methods across varying numbers of camera rigs or embodiments, including zero-shot transfer.This is proposed as an additional way to assess the applicability of techniques for end-to-end driving.
- Future work includes reasoning traces, action representations, architectural choices, and higher-quality BEV teacher occupancy maps.The authors specifically mention maps that include drivable area as a possible improvement.