PaperlayerView exact revision on arXiv

Source-linked AI summary

BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation

Peiyan Li, Yuze Zhu, Yixiang Chen, Qisen Ma, Yuan Xu, Jiabing Yang, He Guan, Yan Huang, Hongtao Wu, Xiao Ma, Tao Kong, Liang Wang, Tieniu Tan

arXiv:2608.05042v1cs.RO

TL;DR

Existing 3D VLA methods can be insufficient when current observations cannot track completed sub-goals or use task-relevant geometry occluded during execution. BridgeVLA++ equips BridgeVLA with unified temporal interaction-history and persistent spatial-geometry memories for deciding what to do next and where to act. BridgeVLA++ achieves state-of-the-art results on two memory-dependent benchmarks, matches or improves BridgeVLA on original benchmarks, and raises real-world memory-dependent success from 20.0% to 93.3%.

  • Problem

    Existing 3D VLA methods can be insufficient when current observations cannot track completed sub-goals or use task-relevant geometry occluded during execution.

  • Method

    BridgeVLA++ equips BridgeVLA with unified temporal interaction-history and persistent spatial-geometry memories for deciding what to do next and where to act.

  • Results

    BridgeVLA++ achieves state-of-the-art results on two memory-dependent benchmarks, matches or improves BridgeVLA on original benchmarks, and raises real-world memory-dependent success from 20.0% to 93.3%.

  • Takeaways & Limitations

    The framework supports data-efficient, generalizable, memory-aware manipulation across spatial tasks, memory-dependent settings, and multiple robot embodiments.

Abstract

from arXiv · show

Leveraging pre-trained vision-language models (VLMs) to construct vision-language-action (VLA) models has emerged as a promising paradigm for 3D robot manipulation. However, existing 3D VLA methods remain data-hungry, exhibit limited generalization under distribution shifts, and lack explicit memory of past observations. These limitations hinder their application to data-scarce, open-world, and memory-dependent manipulation scenarios. Our previous work, BridgeVLA, improves data efficiency and generalization by preserving the input--output alignment of a pre-trained VLM during 3D action learning: raw point clouds are projected into multi-view images, and intermediate heatmaps are predicted before generating robot actions. In this work, we develop BridgeVLA++ by equipping BridgeVLA with a unified spatio-temporal memory architecture that models persistent spatial context and temporal interaction history. The resulting memory-augmented framework can reason over observation histories while preserving BridgeVLA's data efficiency and generalization capabilities. Extensive experiments show that our framework achieves strong performance on spatial manipulation tasks while exhibiting robust generalization. BridgeVLA++ further achieves state-of-the-art performance on two challenging memory-dependent manipulation benchmarks without sacrificing the data efficiency and generalization of the original BridgeVLA. In addition, BridgeVLA++ performs effectively in bimanual manipulation settings and is validated on an additional real-world robotic platform, demonstrating its scalability across tasks, environments, and robotic platforms. These results establish BridgeVLA++ as a unified 3D vision-language-action framework that simultaneously supports data-efficient learning, robust generalization, and effective memory-aware robot manipulation. Project website: https://bridgevla-plus.github.io/.

I. INTRODUCTION

The introduction frames 3D VLA as a way to combine VLM semantic generalization with 3D geometric efficiency, then presents BridgeVLA++ as a memory-augmented extension for history-dependent manipulation. It summarizes state-of-the-art simulation results, real-world gains, bimanual capability, and cross-embodiment scalability.

  • Motivation: Existing 3D VLAs, lose spatial observation-action correspondence through autoregressive action tokens, face a 2D-to-3D modality gap, and lack memory for occluded or history-dependent tasks.These limitations hinder data efficiency, VLM prior transfer, explicit 3D geometry use, and correct action selection when current observations are insufficient.
  • BridgeVLA: BridgeVLA aligns VLM pre-training with 3D manipulation by projecting point clouds into multi-view images and predicting 2D heatmaps that back-project to 3D end-effector positions.A language-conditioned object-grounding pre-training strategy transfers heatmap knowledge to downstream robot action prediction.
  • Simulation results: BridgeVLA achieves state-of-the-art performance on RLBench, COLOSSEUM [14], and GemBench, while BridgeVLA++ reaches state-of-the-art results on memory-dependent RMBench [16] and MemoryBench.BridgeVLA++ matches or improves upon BridgeVLA on the original benchmarks while retaining data efficiency and generalization.
  • Real-world validation: On real robots, BridgeVLA outperforms a strong baseline by 32% on average, while BridgeVLA++ raises memory-dependent success from 20.0% to 93.3% over BridgeVLA.Experiments span Franka Research 3 and Dobot CR5A, including visual perturbations, unseen categories and instructions, and memory-independent and memory-dependent tasks.
  • BridgeVLA++: BridgeVLA++ introduces unified spatio-temporal memory: temporal history determines what to do next, while persistent spatial geometry determines where exactly to act.Spatial memory re-renders earlier, less-occluded scenes and can be shared across two arms with arm-specific action heads.

II. RELATED WORK … A. Problem Formulation

The paper reviews language-conditioned 2D and 3D vision-language-action models, emphasizing their progression toward 3D reasoning and the need for memory when current observations are insufficient. It then presents BridgeVLA’s multi-view heatmap formulation and defines the keyframe-based, memory-free manipulation problem that BridgeVLA++ extends with retained interaction information.

  • A. Language-Conditioned Visuomotor Policies: 2D VLA policies commonly use transformers to map visual inputs and language conditions to 3D manipulation actions, often leveraging pre-trained VLMs for complex skill learning,,, –.
  • B. 3D Vision-Language-Action (VLA) Models: 3D VLA research remains relatively underexplored, with methods incorporating 3D reasoning, robotic representations, point-cloud fusion, or 3D positional encoding into vision-language policies,, [31],,.
  • C. Memory-Dependent Manipulation: Most existing VLA and 3D manipulation policies, including BridgeVLA, are Markovian and predict actions from the current observation alone, which is inadequate when tasks require temporal context or occluded spatial geometry.
  • III. BRIDGEVLA: BridgeVLA aligns 3D manipulation inputs and outputs in a shared 2D space by rendering observed point clouds into orthographic views and predicting language-conditioned multi-view heatmaps.
  • III. BRIDGEVLA: The predicted heatmaps localize a 3D waypoint, after which the point cloud is cropped, magnified, and re-rendered before heatmaps are back-projected to recover the next end-effector translation.
  • A. Problem Formulation: The problem uses expert demonstrations of language-conditioned multi-task 3D manipulation, where each RGB-D observation is paired with an expert action at decision step t.
  • A. Problem Formulation: At sparse keyframe decision points, the single-arm action specifies target translation, rotation, gripper state, and optionally a collision-avoidance flag,,.The collision flag is omitted for benchmarks that do not provide this action component.
  • A. Problem Formulation: After execution by a motion planner or low-level controller, the observation is refreshed and the policy predicts the next keyframe until task completion or a step limit; BridgeVLA++ conditions this process on retained earlier interactions.

B. 2D-Heatmap Pre-Training

BridgeVLA++ adds a pre-training stage that converts a VLM’s unstructured token generation into language-conditioned 2D spatial localization through heatmap prediction. The resulting backbone and decoder transfer to 3D action fine-tuning and can use datasets with spatially convertible annotations.

  • Motivation and contribution: The additional pre-training stage teaches the VLM to ground language-specified objects through 2D heatmap prediction, aligning pre-training outputs with spatially precise robot action learning.This replaces an output interface based on unstructured token generation with spatially structured localization.
  • Heatmap supervision: Using the 120K object-detection split of RoboPoint [48], each image-prompt sample is converted into spatially truncated Gaussian maps for the prompted objects.For multiple target objects, the maps are averaged and normalized into one ground-truth heatmap.
  • Model architecture: PaliGemma [49], combining a SigLIP vision encoder with a Gemma transformer [51], rearranges image tokens into a 2D patch grid before convex upsampling decodes a full-resolution heatmap.Convex upsampling predicts spatially varying interpolation weights, enabling finer localization than fixed bilinear or nearest-neighbor interpolation.
  • Transfer and scalability: The heatmap-pre-trained VLM backbone and decoder are transferred to 3D action fine-tuning, while the formulation can use datasets whose annotations become object centers, keypoints, or segmentation regions.This provides a scalable alternative to representing robot actions as token sequences [11],.

C. 3D Action Fine-Tuning

BridgeVLA fine-tunes a pre-trained VLM through multi-view heatmaps and explicit 3D geometry, predicting precise robot actions without proprioceptive inputs. Its coarse-to-fine interface motivates BridgeVLA++’s stage-specific temporal and spatial memory while preserving the original action design.

  • 3D Action Fine-Tuning: RGB-D observations are reconstructed as colored point clouds, rendered into top, front, and right orthographic views, and processed with language to predict translational heatmaps.The VLM remains purely image-and-language based, reducing distribution shift between 2D heatmap pre-training and 3D policy fine-tuning.
  • Translation prediction: Candidate 3D workspace locations are scored by aggregating projected multi-view heatmap values, and the highest-scoring candidate becomes the next keyframe’s end-effector translation.This preserves geometric correspondence between observations, heatmap outputs, and 3D translational actions.
  • Rotation, gripper, and collision prediction: Rotation, gripper state, and collision avoidance are jointly predicted from global multi-view context and local features around the selected translation.Rotation uses a continuous 6D representation, while gripper and collision outputs use two-class softmax predictions.
  • Coarse-to-fine refinement: Coarse-to-fine refinement first localizes translation across the full workspace, then uses a shared-parameter VLM on a magnified crop to produce the final precise position.The coarse stage supports global localization, while the fine stage improves precision for fine-grained manipulation.
  • BridgeVLA++ motivation: BridgeVLA++ adds temporal memory to coarse-stage representations and spatial memory to fine-stage representations, creating a memory-conditioned policy while preserving BridgeVLA’s heatmap interface and action parameterization.The coarse stage can use interaction history and completed sub-goals, whereas the fine stage can use spatial context in locally occluded geometry.

IV. BRIDGEVLA++ · A. Overview · B. Temporal Memory for Coarse-Stage Reasoning

BridgeVLA++ extends BridgeVLA from a memory-free policy to a memory-conditioned framework that uses complementary temporal and spatial memories. These memories support coarse-stage decisions from interaction history and fine-stage localization when target geometry is occluded, while operating in the VLM’s visual token space.

  • A. Overview: BridgeVLA++ extends the memory-free BridgeVLA policy with an episode memory available at each decision step.The extension addresses cases where the current observation cannot reveal completed sub-goals or task-relevant occluded geometry.
  • A. Overview: The episode memory combines temporal memory summarizing interaction history with spatial memory preserving previously observed scene geometry.Together, these components provide complementary information beyond the current observation.
  • A. Overview: Temporal memory guides coarse-stage decisions about what to do next, whereas spatial memory supports fine-stage decisions about where to act under partial occlusion.This division complements BridgeVLA’s coarse-to-fine action prediction.
  • A. Overview: Both memories are represented and processed in the VLM’s visual token space, enabling integration without modifying BridgeVLA’s original heatmap-based pathway.The supplied passage states that integration does not modify the original heatm…; the pathway description is therefore limited to the cited heatmap-based formulation.
  • B. Temporal Memory for Coarse-Stage Reasoning: The coarse stage predicts an approximate target region and provides features for rotation, gripper-state, and collision-avoidance prediction.Because these decisions depend on recent interactions and overall task progress, temporal memory augments the coarse-stage representation.
  • B. Temporal Memory for Coarse-Stage Reasoning: Temporal memory stores initial anchor views, recent neighboring keyframes, and adaptively selected sub-goal keyframes as coarse-stage visual tokens.These components respectively preserve an initial-scene reference, short-term transition context, and longer-term evidence of completed sub-goals.

1) Initial Anchor Views: … C. Spatial Memory for Occlusion-Robust Fine Localization

BridgeVLA++ combines fixed initial anchor views, dynamic temporal keyframes, adaptive sub-goal retention, and a persistent spatial point-cloud reference to support progress reasoning and occlusion-robust fine localization. These memories preserve global context, interaction history, informative milestones, and previously visible geometry while complementing current observations.

  • 1) Initial Anchor Views:: Initial anchor views A0 provide a consistent global reference for identifying scene changes and reasoning about task progress throughout execution.The initial point cloud is rendered into the coarse stage’s three orthographic views, whose virtual cameras remain fixed.
  • 2) History Keyframes:: Dynamic history memory combines recent neighboring keyframes with representative sub-goal observations to encode immediate transitions and longer-term task progress.The neighboring buffer stores the most recent n = 2 executed keyframes, while sub-goal keyframes record informative milestones selected adaptively.
  • 3) Adaptive Sub-Goal Keyframe Selection:: An adaptive selector retains a keyframe only when its predicted retention probability exceeds a predefined threshold, reducing redundancy in sub-goal memory.A learnable query summarizes memory-conditioned image tokens, and a small MLP predicts the retention probability.
  • 3) Adaptive Sub-Goal Keyframe Selection:: The selector evaluates memory-conditioned tokens, allowing it to retain observations that add information beyond what temporal memory already represents.This selection occurs after temporal-memory integration rather than on pre-memory tokens.
  • C. Spatial Memory for Occlusion-Robust Fine Localization: Spatial memory stores the initial colored point cloud P0 as a less-occluded geometric reference for fine localization when the current crop is obstructed.P0 is re-rendered using the same viewpoint and zoom configuration as the subsequent fine-stage crop after the coarse stage predicts a waypoint.
  • C. Spatial Memory for Occlusion-Robust Fine Localization: Spatial memory complements rather than replaces the current observation by combining its latest state with persistent geometry from the canonical initial point cloud.This adaptive zoom alignment allows one P0 reference to support changing fine-stage crops.
  • C. Spatial Memory for Occlusion-Robust Fine Localization: Applying the predicted waypoint and zoom transformation to both current and initial point clouds produces geometrically aligned views for fine-stage comparison.The aligned reference is rendered through fine-stage virtual cameras and encoded into visual tokens.
  • C. Spatial Memory for Occlusion-Robust Fine Localization: View-specific attention lets each current-view token attend only to the corresponding reference-view memory tokens, preserving aligned spatial correspondence.The current observation captures the latest scene state, while spatial memory supplies previously visible geometry that may now be occluded.

D. Memory Integration … 1) Training:

BridgeVLA++ integrates temporal and spatial memories through lightweight attention modules while preserving BridgeVLA’s downstream processing pipeline. Its scene-level memory design supports bimanual manipulation, and training constructs consistent memories and supervises action and sub-goal prediction.

  • D. Memory Integration: Temporal memory conditions the coarse-stage representation, while spatial memory conditions the fine-stage representation through compact attention modules.Each memory-injection block uses current visual tokens as queries and memory tokens as keys and values, followed by self-attention and feed-forward updates.
  • D. Memory Integration: Encoded visual-token grids are cached instead of historical projection images, allowing reuse throughout an episode without re-encoding.Each cached observation has shape R_V × N × d, with V orthographic views, N visual tokens per view, and d the token dimension.
  • D. Memory Integration: The memory-injection modules preserve token-grid shape, so BridgeVLA’s convex-upsampling modules and action-prediction heads remain unchanged.The temporal and spatial injection modules add approximately 168M and 84M parameters, while adaptive sub-goal selection adds approximately 18M parameters.
  • E. Bimanual Extension: Scene-level temporal and spatial memories are shared across arms, while duplicated convex-upsampling modules and MLP action heads produce arm-specific bimanual actions.The VLM backbone, episodic memories, and adaptive selection module are shared, avoiding duplication of the most computationally expensive components.
  • F. Training and Inference Details: During training, memories are built from preceding expert-demonstration observations, using the initial observation as the temporal anchor and spatial reference.Neighboring observations and annotated sub-goal keyframes are selected from earlier execution steps.
  • 1) Training:: Random rigid-body augmentation is applied consistently to current observations, memory observations, and ground-truth actions to preserve geometric consistency.The objective combines BridgeVLA’s action-prediction loss with binary cross-entropy supervising whether the current keyframe should be retained as a sub-goal keyframe.
  • 1) Training:: For bimanual tasks, the base loss includes both arms’ action losses, while the shared adaptive selection module receives one sub-goal supervision loss.This shares sub-goal selection supervision across the two arm-specific action predictions.

2) Inference: … B. COLOSSEUM & GemBench: Generalization

BridgeVLA++ preserves BridgeVLA’s out-of-distribution robustness while adding spatio-temporal memory, matching or improving BridgeVLA on COLOSSEUM and GemBench. Evaluations also cover memory construction and general 3D manipulation on RLBench.

  • 2) Inference:: At episode start, inference builds temporal anchor views and a spatial point-cloud reference P0, while remaining temporal-memory slots receive zero padding.After each action, current-observation image tokens enter the temporal buffer as neighboring keyframes, and adaptive selection may retain sub-goal keyframes.
  • V. EXPERIMENTS: Experiments evaluate learning, distribution-shift robustness, architectural components, cross-platform deployment with 10 demonstrations, and memory-dependent manipulation.The evaluation spans simulation and real-world environments and assesses both BridgeVLA and BridgeVLA++.
  • A. RLBench: General 3D Manipulation: RLBench evaluates 18 multi-task manipulation settings using a Franka Panda with four RGB-D cameras, including non-prehensile, pick-and-place, and precision insertion tasks.The benchmark uses CoppeliaSim and follows prior evaluation protocols.
  • A. RLBench: General 3D Manipulation: BridgeVLA is compared with state-of-the-art 2D and 3D baselines, including voxel-space PerAct, point-scoring Act3D, and SAM2Act.SAM2Act is identified as the previous state-of-the-art method on RLBench and integrates SAM2 with a multi-view transformer.
  • A. RLBench: General 3D Manipulation: BridgeVLA achieves a 90.5% average success rate across 18 RLBench tasks, outperforming SAM2Act by 3.7 percentage points and establishing a new state-of-the-art.The improvement is especially pronounced on precision-critical tasks such as Stack Cups, supporting dense per-view heatmap localization.
  • B. COLOSSEUM & GemBench: Generalization: COLOSSEUM tests 12 unseen perturbation axes plus original and jointly perturbed RLBench settings, while GemBench tests hierarchical generalization to novel objects and compositions.COLOSSEUM varies object and receptacle properties, backgrounds, lighting, distractors, and camera pose.
  • B. COLOSSEUM & GemBench: Generalization: BridgeVLA++ achieves 65.2% versus BridgeVLA’s 64.0% on COLOSSEUM and 51.1% versus 50.0% on GemBench, preserving generalization.The comparison covers 14 COLOSSEUM conditions and GemBench’s hierarchical generalization to novel rigid and articulated objects and unseen object–color compositions.

C. RMBench: Memory-Dependent Bimanual Manipulation … 2) Memory-Augmented Manipulation:

BridgeVLA++ achieves strong memory-dependent manipulation performance in bimanual, single-arm, and real-world settings while preserving the base model’s general manipulation capabilities. Its targeted spatio-temporal memory enables episodic reasoning with high data efficiency and robust transfer across tasks and platforms.

  • C. RMBench: Memory-Dependent Bimanual Manipulation: 96.0% overall success on RMBench lets BridgeVLA++ outperform MemoryWAM by 13.0 points and Mem-0 by 54.0 points, ranking best or tied-best on eight of nine tasks.RMBench requires short- and long-term episodic reasoning from 50 demonstrations per task, whereas memory-free BridgeVLA achieves only 18.9%.
  • D. MemoryBench: Single-Arm Memory Validation: 99.7±0.3% success on MemoryBench confirms BridgeVLA++ generalizes episodic-memory handling beyond bimanual coordination to single-arm scenarios.MemoryBench provides single-arm memory-dependent evaluations, with per-task comparisons reported in Appendix I, Table XII.
  • E. Real-World Experiments: Real-world experiments deploy BridgeVLA and BridgeVLA++ on Franka and Dobot platforms to test sample efficiency, generalization, physical memory transfer, and preservation of base manipulation competence.The evaluation includes a 7-DoF Franka with a static ZED 2i camera and a 6-DoF Dobot CR5A with an external RGB-D camera.
  • 1) General Manipulation:: 95.4% success with only three demonstrations per task shows BridgeVLA’s sample efficiency, while it consistently outperforms RVT-2 across all seven real-world generalization settings.The settings include distractors, lighting, background, height, combination, and category shifts.
  • 2) Memory-Augmented Manipulation:: 93.3% average success on three real Dobot memory-dependent tasks gives BridgeVLA++ a threefold improvement over SAM2Act+ at 30.0%, while BridgeVLA reaches only 20.0%.The comparison covers Cover Blocks, Press Button, and Swap Eggplant in the basic setting.
  • 2) Memory-Augmented Manipulation:: Targeted spatio-temporal memory outperforms SAM2Act+ by avoiding indiscriminate storage and fixed-window retrieval that dilute critical historical information.BridgeVLA++ combines selective spatial and temporal memory with a pre-trained VLM backbone for robustness against visual disturbances.
  • 2) Memory-Augmented Manipulation:: On two memory-free Dobot tasks, BridgeVLA++ matches or exceeds BridgeVLA in success rate and generalization across all settings, showing that memory integration preserves foundational manipulation skills.The memory-free tasks are Put in Drawer and Put on Shelf.

F. Ablation Studies · VI. CONCLUSION AND FUTURE WORK

Ablations identify heatmap-based prediction, input alignment, and temporal memory as the largest contributors, while other components provide targeted gains. The conclusion presents BridgeVLA and BridgeVLA++ as alignment-based, memory-augmented 3D VLA frameworks built on pre-trained VLMs.

  • F. Ablation Studies: Replacing heatmap prediction with direct target-position regression collapses RLBench average success from 90.5% to 31.4% and requires a threefold larger batch size.Heatmaps provide denser supervision, a spatial prior from 3D-to-2D projection, and alignment with the VLM’s image-space representations.
  • F. Ablation Studies: Adding per-pixel 3D positions to the VLM backbone lowers success from 90.5% to 56.2%, indicating that preserving the pre-trained image-feature distribution is important.The 3D convolutional fusion module injects spatial cues but shifts image features away from the VLM pre-training distribution.
  • F. Ablation Studies: Removing 2D heatmap pre-training prevents BridgeVLA from matching RVT-2 in language-related real-world settings, whereas the full model performs best, especially on Combination.The pre-training teaches language grounding directly in heatmap space, which limited robot trajectories alone cannot provide.
  • F. Ablation Studies: Replacing continuous 6D rotation with discretized Euler-angle classification reduces RLBench average success from 90.5% to 88.2%, especially on high-precision orientation tasks.The 6D representation remains robust near-vertical gripper poses and avoids Euler-angle gimbal lock.
  • F. Ablation Studies: Removing spatial memory S lowers BridgeVLA++’s RLBench average from 93.7% to 92.0%, with Sort Shape falling from 72.0% to 60.8% under occlusion-heavy precision demands.The effect is concentrated in tasks requiring fine-grained geometric alignment during arm-induced occlusions.
  • F. Ablation Studies: Removing temporal memory T drops RMBench success from 96.0% to 21.3%, near the memory-free base model’s 18.9%, while also reducing RLBench performance from 93.7% to 91.9%.Temporal memory supports both memory-dependent sequencing and general manipulation through stable reference views and local motion cues.
  • VI. CONCLUSION AND FUTURE WORK: The conclusion presents BridgeVLA and BridgeVLA++ as efficient 3D VLA models built on a pre-trained VLM, using multi-view 2D rendering, image-space heatmaps, and scalable heatmap pre-training.BridgeVLA++ extends BridgeVLA’s alignment principle with memory augmentation while retaining the shared 2D image-space formulation.

APPENDIX … E. Evaluation Protocol

The appendix specifies BridgeVLA++’s memory architecture, pre-training and fine-tuning procedures, training-data preparation, and benchmark-specific evaluation protocols. These settings cover spatial-temporal memory construction, staged optimization, keyframe and sub-goal supervision, rendering and augmentation constraints, and evaluation across simulation and real-robot platforms.

  • A. Network and Memory Architecture: The memory architecture uses three two-layer, eight-head injection blocks in the backbone’s 2048-dimensional patch-token space for anchor-view, dynamic-keyframe, and spatial memory.The anchor block concatenates three views to track scene changes across views, while the other memory blocks use the corresponding cached context.
  • B. Pre-Training: All fine-tuning runs warm-start from one 2D-heatmap pre-training run on RoboPoint’s 120K object-detection split, while memory blocks and the sub-goal gate train from scratch.The pre-training setup renders ground-truth heatmaps from detection boxes and retains heatmap predictions after fine-tuning.
  • C. Fine-Tuning Details: Fine-tuning uses a two-phase schedule that first freezes PaliGemma while adapting external modules, then jointly trains the unfrozen backbone with those modules.The initially trainable components include convex upsampling, MLP action heads, memory injection blocks, and the sub-goal gate.
  • C. Fine-Tuning Details: Memory-enabled training disables random in-plane image augmentation and centers the workspace on fixed scene bounds to preserve pixel correspondence with cached memory tokens.SE(3) augmentation is instead applied jointly to the current observation and memory inputs.
  • C. Fine-Tuning Details: Memory capacity is K=12 on RMBench, holding neighboring keyframes and up to ten sub-goal slots, but K=2 elsewhere without sub-goal annotations.Every executed keyframe occupies a neighboring slot, while gated keyframes enter sub-goal slots after leaving the neighboring window; other rendering uses three 224 × 224 orthographic views.
  • D. Training Data Preparation: Training data preparation converts demonstrations into consecutive-keyframe transitions using a PerAct heuristic, with a bimanual variant for RMBench and segment-level sub-goal labels.Positive sub-goal labels mark the last keyframe of each language segment, while repeated identical segments remain separate.
  • E. Evaluation Protocol: The Dobot suite evaluates five tasks and seven instructions using shared ten-demonstration training sets across memory-free BridgeVLA, memory-augmented BridgeVLA++, and SAM2Act+ [17].The comparison separates memory effects from backbone and retrieval differences because BridgeVLA shares BridgeVLA++’s backbone without memory, whereas SAM2Act+ [17] uses memory with a different backbone and retrieval scheme.

F. Computational Cost … K. General Manipulation on the Franka Platform

Across computational, generalization, memory, ablation, and real-world evaluations, BridgeVLA++ preserves BridgeVLA’s robustness while adding memory-dependent capabilities and maintaining practical inference cost. Its temporal memory is critical for history-dependent tasks, while real-world results support data-efficient manipulation from input–output alignment.

  • F. Computational Cost: End-to-end latency averages 0.35 seconds per prediction step for BridgeVLA and 0.57 seconds for BridgeVLA++ over 100 trials.Evaluation uses one GPU per run, and real-world deployment uses a single NVIDIA RTX 4090; both latency figures are small relative to the remainder of the control loop.
  • G. Per-Task Results on COLOSSEUM: BridgeVLA ranks best among prior methods in 13 of 14 COLOSSEUM settings, with its largest lead under appearance shifts.The lead reaches 11–15 points for table texture, table color, light color, and receptacle texture.
  • H. Per-Task Results on GemBench: BridgeVLA++ achieves the best GemBench L2 result at 68.9% and raises L4 performance from BridgeVLA’s 0.0% to 8.2%.BridgeVLA remains competitive on L1 and leads L3, while BridgeVLA++ gives up a few points on L1 and L3; its L4 gain comes almost entirely from the memory extension.
  • I. Per-Task Results on MemoryBench: BridgeVLA++ nearly solves MemoryBench, with its Table XII margin over SAM2Act-based baselines reflecting a handful of failed episodes rather than run-to-run instability.MemoryBench tests three tasks in which pressing a button erases visual evidence needed later, and compares against RVT-2, SAM2Act, and SAM2Act+.
  • K. General Manipulation on the Franka Platform: The Franka suite evaluates 13 tasks spanning simple pick-and-place through long-horizon drawer manipulation, with trajectories containing 3–9 keyframes.Demonstrations are collected by kinesthetic teaching at expert trajectory keypoints, followed by the stated training and evaluation protocol.
  • K. General Manipulation on the Franka Platform: At 10 demonstrations, BridgeVLA exceeds π0.5 by over 75 points on the Franka suite, isolating input–output alignment rather than the shared backbone as the source of sample efficiency.The comparison uses heatmap prediction in projected views for BridgeVLA versus a separate flow-matching action expert for π0.5.

L. Memory-Dependent Manipulation on the Dobot Platform … O. Preservation of Object Grounding after Fine-Tuning

BridgeVLA++ is evaluated on memory-dependent tasks using a second Dobot embodiment, while baseline analyses, reduced-demonstration training, and fine-tuning visualizations assess robustness, efficiency, and preserved object grounding. Across these evaluations, the supplied passages report reliable task performance, failures of competing methods, at least 7/10 success with three demonstrations, and retained grounding after fine-tuning.

  • L. Memory-Dependent Manipulation on the Dobot Platform: BridgeVLA++ transfers RMBench’s three memory-dependent task families to a real Dobot platform, where current observations alone underdetermine the next action.The setup uses a 6-DoF Dobot CR5A with a ChangingTek CTAG2F90-C parallel-jaw gripper and one static ZED 2i stereo camera.
  • M. Real-Robot Baseline Failure Modes: SpatialVLA fails on nearly all tasks with 10 trajectories per task; increasing training to 50 raises performance from 3.1% to 28.5%, but it remains far behind BridgeVLA.The passage notes that SpatialVLA often does not even move toward the correct target and remains especially weak on harder tasks.
  • M. Real-Robot Baseline Failure Modes: BridgeVLA remains reliable across evaluated tasks, whereas π0.5 [2] succeeds occasionally on simple pick-and-place but fails on complex long-horizon tasks with unstable, premature-gripper motions.The passage gives Put Zebra in Drawer as an example of π0.5’s complex-task failure.
  • M. Real-Robot Baseline Failure Modes: ACT generalizes poorly near workspace boundaries, consistent with its Gaussian action prior assigning low probability to peripheral regions.Its failures are concentrated outside regions densely covered by demonstrations.
  • M. Real-Robot Baseline Failure Modes: RVT-2 is the strongest Franka baseline but remains less robust than BridgeVLA, with imprecise grasps, inaccurate placements, and a wider gap under generalization.The comparison is based on the reported failure modes across the Franka suite and generalization settings.
  • M. Real-Robot Baseline Failure Modes: SAM2Act+ [17] fails on Dobot memory tasks because its fixed-size history window loses early block colors and lacks explicit counting of completed button presses.In Cover Blocks, about 11 accumulated history steps are dominated by near-duplicate frames, obscuring the initial block colors.
  • N. Sample Efficiency with 3 vs. 10 Demonstrations: With three rather than ten demonstrations per task, BridgeVLA stays at or above 7/10 on every task and matches the 10-demonstration model on most tasks.Table XVIII lists the corresponding per-task success counts.
  • O. Preservation of Object Grounding after Fine-Tuning: After fine-tuning on robot action data, BridgeVLA retains object grounding from 2D-heatmap pre-training, whose detection targets are truncated Gaussians centered on annotated boxes.Figures 15 and 16 visualize target construction and fine-tuned predictions on repeated multi-view-like inputs.

P. Real-Robot Generalization Settings

The real-robot evaluation defines six generalization settings spanning visual disturbances, novel object–skill combinations, and unseen object categories. It reports per-task results across four GemBench levels, memory ablations, and validations on Franka and Dobot platforms.

  • Generalization Settings: Six settings test distractors, lighting changes, background variants, elevated objects, 13 unseen object–skill pairings, and 7 objects from unseen categories.Distractor objects resemble targets; Lighting turns lights off; Background changes the tablecloth; Height raises objects onto a drawer or box.
  • GemBench Evaluation: Per-task GemBench results cover novel placements, novel rigid objects, novel articulated objects, and novel long-horizon tasks.Tables XIII–XVI report success rates as mean±std over five random seeds with 20 trials per task variation, including baselines quoted from [15].
  • Memory Evaluation: The evaluation includes memory ablations on RMBench, comparing full BridgeVLA++ with two single-memory variants and memory-free BridgeVLA.Table XVII reports success rates over 100 episodes per task.
  • Real-Robot Platforms: Real-robot validation spans Franka experiments with 3 versus 10 demonstrations and Dobot experiments covering memory-dependent and memory-free instructions.Table XVIII reports Franka success counts over 10 trials per task, while Table XIX reports Dobot success counts over 10 trials per language instruction and setting.
Loading 2608.05042v1…