Source-linked AI summary
4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
Jiahui Zhang, Yurui Chen, Yueming Xu, Ze Huang, Yanpeng Zhou, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, Xingyue Quan, Hang Xu, Li Zhang
TL;DR
Diverse robotic datasets are difficult to exploit efficiently because incomplete observations produce coordinate system chaos and state chaos in conditional action distributions. 4D-VLA integrates spatial and temporal information through sequential RGB-D inputs and memory bank sampling, and MV-Bench evaluates spatial generalization across viewpoints. The model outperforms existing methods in simulated, real-world, and multi-view evaluations.
Problem
Incomplete observations in robotic pretraining can produce dispersed, nonsmooth, high-variance, or multimodal conditional action distributions that hinder robust and generalizable behavior.
Method
4D-VLA encodes sequential RGB-D inputs with spatial coordinates and samples informative historical frames through memory bank sampling.
Results
4D-VLA outperforms existing methods in simulated LIBERO, real-world evaluations, and MV-Bench multi-view evaluation.
Takeaways & Limitations
The results support 4D-VLA as a pretraining approach with stronger spatiotemporal reasoning, spatial perception, and generalization to novel viewpoints.
Abstract
from arXiv · showhide
Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset's action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.
1 Introduction
4D-VLA addresses incomplete observations in robotic pretraining by integrating spatial and temporal information into visual inputs. The framework also introduces memory bank sampling and MV-Bench to improve efficiency and evaluate spatial generalization.
- Incomplete observations can produce nonsmooth, high-variance, or multimodal action distributions that hinder robust and generalizable behavior.
- Single RGB images and textual instructions create coordinate system chaos and state chaos by omitting spatial, temporal, or contextual cues.
- 4D-VLA combines spatial coordinate embeddings, a 3D-aware module, and historical frames to align robot and scene coordinates and improve temporal reasoning.
- Memory bank sampling selects key historical frames according to similarity, improving model efficiency.
- MV-Bench evaluates spatial understanding and generalization across diverse viewpoints, while 4D-VLA improves generalization to novel scenarios and outperforms baselines.
2 Related works
Related work spans vision-language models, vision-language-action control, and pretraining on heterogeneous robotic datasets. Existing approaches differ in input history, action prediction, and dataset-specific adaptation.
- Vision-language models: Vision-language models align text and image feature spaces, with newer approaches supporting multi-image inputs for more complex tasks and scenarios.
- Vision-language-action models: Vision-language-action models predict trajectories, closed-loop actions, or future states using action heads and world models.
- Robotic pretraining: Robotic pretraining methods use single images, historical context, diffusion action heads, or dataset-specific parameters to improve generalization and efficiency.
3 Method
4D-VLA processes sequential RGB-D observations with spatial and temporal encoding to produce spatiotemporal tokens for action prediction. Its design combines 3D coordinate-aware visual features, adaptive historical-frame sampling, temporal encoding, and an action-learning objective, with MV-Bench evaluating multi-view generalization.
- Overall architecture: 4D-VLA converts sequential RGB-D images into spatial vision tokens, combines them with task-specific text tokens, and decodes actions through a VLM Transformer and action head.The pipeline uses a pretrained VLM backbone and an action head for control prediction.
- Spatial-aware visual tokens: Depth values are back-projected using camera intrinsics and extrinsics into world or robot coordinates, then encoded as learnable positional embeddings and added to visual features.The resulting spatial vision tokens are generated by the VLM projector from the fused features.
- 4D representation with multi-frame encoding: Historical spatial vision tokens extend single-frame encoding into a coherent 4D spatiotemporal representation and are decoded jointly with instruction tokens.Uniformly sampled frames provide the baseline multi-frame construction, with temporal positions represented separately.
- Memory bank sampling: Memory bank sampling adaptively selects a small set of informative historical frames because dense sampling increases memory use and reduces inference speed.The method returns k timestamps from a temporal window using a feature extractor and similarity-based memory bank procedure.
- Action prediction and loss: The action head predicts translation, rotation, and gripper state from the Transformer’s last-token features, while training combines translation, rotation, grip, and directional losses.The directional loss emphasizes action direction because translation magnitudes are often small.
- MV-Bench: MV-Bench evaluates control-policy learning across viewpoints using In-View and Cross-View settings, including testing on unseen viewpoints.The dataset samples six training and six testing viewpoints within a 270° front-facing range.
4 Experiments
The experiments evaluate 4D-VLA through simulation and real-world closed-loop manipulation, including LIBERO, MV-Bench, and targeted real-world settings. Results report gains over established baselines and examine spatial, temporal, viewpoint, and layout generalization.
- Experimental setup: 4D-VLA is pretrained on DROID and fine-tuned with simulation and real-world trajectories for closed-loop evaluation.The evaluation reports task performance across diverse environments.
- Real-world evaluation: Real-world tasks test spatial generalization, distractor robustness, precise placement, and instruction following using task-specific evaluation procedures.The experiments use a Franka arm, while Table 3 incrementally adds pretraining, coordinate encoding, and memory-bank-selected historical frames.
- LIBERO evaluation: LIBERO evaluation measures task success rate across subtasks, object layouts, and random initial-state seeds.Each task uses 1,500 simulation tracks, with means and standard deviations computed from three seeds.
- LIBERO evaluation: 12.1% higher average success rate than OpenVLA is reported on LIBERO, with additional gains across LIBERO-SPATIAL, LIBERO-OBJECT, LIBERO-GOAL, and LIBERO-LONG.The reported margins are 5.2% over OpenVLA on LIBERO-SPATIAL, 2.7% over DiffusionPolicy on LIBERO-OBJECT, 6.3% over Octo on LIBERO-GOAL, and 25.4% over OpenVLA on LIBERO-LONG.
- MV-Bench evaluation: MV-Bench evaluates In-View and Cross-View success across diverse and unseen viewpoints, with 81.0% success reported for 4D-VLA in In-View.The model is also reported to achieve the best performance in Cross-View evaluation.
- Real-world evaluation: Coordinate encoding improves spatial grounding and action accuracy, while memory bank sampling supplies temporal context for multi-step reasoning under latent state ambiguity.Matching downstream inputs to the pretraining setting improves transfer, although the model also outperforms baselines with mismatched inputs.
5 Discussion
The discussion examines how historical context, temporal encoding, and sampling choices affect 4D-VLA’s effectiveness and efficiency. Historical window size matters more than the number of sampled frames, while temporal encoding addresses ambiguity from non-uniform sampling.
- Discussion: The analysis studies historical-context utilization, architectural components, and the effect of coordinate-system chaos on model performance.The experiments use an InternVL-4B model with an MLP action head and a single RGB image as the initial vision input.
- Historical information: Figure 6 compares historical-image sampling configurations, with larger points indicating lower efficiency.The figure supports comparison of performance and efficiency across historical-window and sampling choices.
- Historical information: The historical window size n has a larger performance effect than the sampled-frame count k.The study varies k while fixing n, then varies n while fixing k.
- Historical information: Memory bank sampling reduces redundancy and improves performance with a smaller k, increasing the efficiency of historical-context use.The findings suggest uniform sampling can introduce excessive redundancy that diminishes efficiency.
- Temporal encoding: Non-uniform memory-bank sampling motivates explicit temporal encoding because sampled historical frames otherwise become temporally ambiguous.The paper evaluates additive and concatenated temporal encodings in an ablation study.
6 Conclusion
4D-VLA addresses coordinate-system and state chaos by encoding sequential RGB-D inputs with 3D coordinates and sampling informative historical frames. It outperforms existing methods in LIBERO and MV-Bench, while requiring RGB-D hardware.
- Conclusion: 4D-VLA encodes sequential RGB-D images with corresponding 3D coordinates to align the robot coordinate system with the scene.This design supports spatiotemporal reasoning with minimal training overhead.
- Conclusion: Memory bank sampling extracts informative and diverse key frames from sequences to improve efficiency.The strategy is presented as part of the model’s 4D information integration.
- Conclusion: 4D-VLA outperforms existing methods in the simulated LIBERO environment and shows superior spatial perception and novel-view generalization on MV-Bench.The conclusion reports these outcomes across robotic control and multi-view evaluation.
- Conclusion: The approach relies on RGB-D input, introducing a hardware restriction.This is stated as a limitation of the method.
7 Appendix section
The appendix details 4D-VLA’s lightweight RGB-depth-coordinate design, evaluation settings, ablations, memory-bank sampling, and coordinate-chaos experiments. Results favor memory-bank sampling and 3D information, while showing that coordinate chaos degrades non-3D performance.
- Method: 4D-VLA uses RGB inputs augmented with depth-aligned spatial coordinates, mapping vision-language inputs directly to low-level actions without point-cloud encoders.The representation uses 2D patches plus (x, y, z) embeddings and remains VLM-compatible.
- Implementation: The streaming memory bank keeps 5 history frames within a 20-frame context, while the current step uses front and wrist views.History stores only the front view, balancing efficiency and spatial coverage.
- Ablations: The MLP action head provides the highest inference speed with a relatively high success rate, while the autoregressive head is slower because it predicts multi-token actions.These action-head comparisons are reported on LIBERO-LONG.
- Ablations: Proprioceptive tokens hurt performance, likely encouraging overfitting to dataset-specific motion patterns instead of generalizable visuomotor features.The cited explanation links proprioceptive states such as joint positions and velocities to action labels.
- Memory bank sampling: MBS achieves the highest success (0.866) with 160.0 ms latency and 8,682.9 MB peak memory on Libero-Spatial.Single-frame inference is faster at 76.5 ms but has lower success (0.738).
- Coordinate system chaos: Coordinate chaos is generated through random translations and rotations across 30 viewpoints, then evaluated by comparing models with and without 3D information.The controlled experiment uses transformed action and camera information during training.
- Coordinate system chaos: As rotation-induced chaos increases, the non-3D model’s performance drops sharply, whereas 3D information alleviates the degradation and yields lower variance across viewpoints.Chaos levels use 0°, 15°, 30°, and 90° random z-axis rotations, with translations sampled within 0.5.