PaperlayerView exact revision on arXiv

Source-linked AI summary

VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulation

Boyao Han, Chen Shi, Jingjing Qian, ZhuoTan Tian, Li Jiang

arXiv:2610.12451v1cs.CVcs.RO

TL;DR

Camera-configuration changes can shift VLA inputs and degrade policies, while many policies fix their camera count during training. VersaCamVLA learns fixed-size scene tokens from posed RGB views and conditions a pretrained VLA with them; it reports consistent improvements across benchmarks and camera configurations.

  • Problem

    VLA policies often fix camera count during post-training, and changes in camera count or pose can alter visual-token distributions and degrade policies.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    VersaCamVLA learns fixed-size scene tokens from variable sets of posed RGB views through multi-signal target-view prediction and WAPS, then adds them as supplementary conditioning to a pretrained VLA.

    Source: §1 · pp. 2–3

  • Results

    VersaCamVLA consistently outperforms prior VLA methods and direct multi-view baselines across RoboTwin, LIBERO, and a real-robot platform under varying camera counts and poses.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    The reported findings show that a unified scene-token interface makes pretrained VLA policies more robust to practical camera-configuration changes.

    Source: §1 · p. 2

    VersaCamVLA assumes calibrated camera intrinsics and extrinsics and focuses on tabletop bimanual setups with manual camera placement, without autonomous viewpoint selection.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as powerful foundations for robotic manipulation, but their reliance on fixed camera configurations during training makes them brittle to changes in camera count or pose during deployment. To overcome these limitations, we propose VersaCamVLA, a camera-configurable framework that decouples camera-set representation from action learning. VersaCamVLA learns a unified scene-token interface that maps an arbitrary, variable set of posed RGB views into fixed-size latent scene tokens. This is achieved via multi-signal target-view prediction and Wrist-Augmented Pose Sampling (WAPS), which leverages natural wrist-camera motion for free pose diversity. At deployment, a lightweight spatial encoder injects these compact scene tokens into a pretrained base VLA as a supplementary visual condition, requiring no explicit 3D sensing or novel-view rendering. Experiments on RoboTwin, LIBERO, and a real-robot platform demonstrate that VersaCamVLA consistently outperforms prior VLA methods and direct multi-view baselines, maintaining robust performance across varying camera counts and unseen camera poses.

1 Introduction

Camera changes can shift the visual inputs on which fine-grained manipulation policies rely. VersaCamVLA addresses this with a fixed-size scene-token interface and reports consistent gains across benchmarks and camera configurations.

1 Introduction

  • The framework targets camera-configurable manipulation because changing camera count or pose alters visual-token distributions and can degrade policies, while task-preferred views vary.The paper notes that useful views may be unavailable or suboptimal on a given robot platform.Source: §1 · p. 1
  • VersaCamVLA decouples camera-set representation from action learning by mapping any non-empty set of posed RGB views into fixed-size scene tokens.The policy-facing visual format remains unified across camera configurations.Source: §1 · pp. 2–3
  • Stage I learns scene tokens through target-view prediction of RGB, semantic, and edge signals, while WAPS uses wrist-camera motion to provide pose diversity without extra multi-pose demonstrations.Source camera subsets vary while the decoder reconstructs available views, yielding self-reconstruction and cross-view prediction signals.Source: §1 · pp. 2–3
  • Stage II freezes the scene encoder and supplies its tokens as supplementary visual conditioning to a pretrained VLA, retaining the base policy's native RGB input and action-learning objective.A lightweight spatial encoder compresses the tokens; deployment requires posed RGB views, not explicit 3D reconstruction or novel-view rendering.Source: §1 · pp. 2–3
  • VersaCamVLA consistently outperforms prior VLA methods and direct multi-view baselines on RoboTwin, LIBERO, and a real-robot platform across changing camera counts and poses.The reported results include unseen camera poses.Source: §1 · pp. 2–3

2 Related Work

VLA policies use pretrained vision-language models for robot control, but camera-configuration flexibility remains an underexplored issue. Related approaches broaden viewpoints through augmentation, geometry-aware representations, or multi-camera inputs.

2.1 Vision-Language-Action Models

  • VLA policies adapt pretrained vision-language models to robot control, but often fix camera count during post-training and can be sensitive to camera-configuration shifts.The cited work identifies viewpoint variation and limited camera-pose awareness as concerns.Source: §2.1 · p. 3

2.2 Viewpoint Generalization for Vision-Language-Action Policies

  • Viewpoint-generalization methods use multi-pose demonstrations, novel-view synthesis, or geometric priors, with collection cost and synthesis fidelity noted as constraints.The paper contrasts these approaches with compact scene tokens learned from posed RGB views, while retaining a calibrated-intrinsics-and-extrinsics requirement.Source: §2.2 · p. 3

2.3 Multi-Camera Perception for Vision-Language-Action Policies

  • Direct multi-camera policies concatenate image features or tokens from multiple views to improve scene coverage, reduce occlusion, and support fine-grained manipulation.VLA-style methods similarly append external-view tokens to policy inputs.Source: §2.3 · p. 3

3 Methodology

VersaCamVLA separates camera-set representation learning from action learning, mapping variable sets of posed RGB views to fixed-size scene tokens that condition a pretrained VLA. It learns the interface through target-view prediction and wrist-camera pose diversity, then uses a compact scene-token pathway for policy learning and deployment.

3.1 Camera-Configurable VLA Setting

  • Camera-configurable policy learning requires variable numbers of posed views to map to fixed-size scene tokens, independent of camera count and pose.VersaCamVLA augments rather than redesigns the pretrained VLA backbone.Source: §3.1 · p. 4

3.2 Unified Scene-Token Interface Learning

  • Target-view prediction trains the scene encoder to produce a fixed-size representation that supports RGB, semantic-map, and edge-map reconstruction beyond its input views.The scene decoder queries scene tokens with target-camera rays; semantic and edge signals supplement RGB reconstruction.Source: §3.2 · pp. 5–6

3.3 Wrist-Augmented Pose Sampling

  • Wrist-Augmented Pose Sampling varies source-camera subsets and uses wrist-camera motion to provide diverse cross-view targets without extra data-collection cost.Static views are randomly sampled, wrist views are independently included with probability 0.5, and the full camera pool supplies targets.Source: §3.3 · pp. 6–7

3.4 Scene-Token-Conditioned VLA Policy Learning

  • The frozen scene encoder feeds tokens through a convolutional spatial encoder that compresses them into a compact supplementary representation for the VLA.The spatial encoder and VLA are trained jointly with action loss, while the scene encoder remains frozen.Source: §3.4 · p. 7
  • At deployment, any available subset of posed camera observations maps to the same fixed-length representation without novel-view rendering, depth, point clouds, or explicit 3D reconstruction.The scene decoder and prediction heads are discarded from the inference path.Source: Figure 2 · p. 4, §3.4 · p. 8

4 Experiments

Experiments compare VersaCamVLA on manipulation benchmarks and camera-configuration shifts, including real-robot tasks, and examine design choices through ablations.

4.1 Experimental Setup

  • Evaluation spans RoboTwin 2.0 bimanual and four-suite LIBERO benchmarks, camera-pose and camera-count shifts, and three real-robot bimanual tasks.Real-robot tasks use 20 trajectories each; pose tests vary azimuth, distance, and pitch, while camera-count tests evaluate VersaCamVLA with 3–6 views.Source: §4.1 · p. 8

4.3 Ablation Study

  • 192 scene tokens yield 55.63% average success, versus 50.06% with 48 tokens and 47.50% with 432.The token ablation is on RoboTwin 2.0 Clean.Source: Table 5 · p. 9, §4.3 · p. 9
  • WAPS raises unseen-pose average success from 48.79% to 52.63%; pretrained LVSM initialization reaches 55.63%, versus 51.25% from scratch.The WAPS result is a 3.84-point gain; the initialization ablation is on RoboTwin 2.0 Clean.Source: Table 7 · p. 9, Table 8 · p. 9, §4.3 · p. 10
  • Convolutional spatial-encoder compression reaches 55.63% average success, compared with 50.13% for pooling.The spatial-encoder ablation is on RoboTwin 2.0 Clean.Source: Table 6 · p. 9, §4.3 · p. 10

4.2 Main Results

  • VersaCamVLA achieves RoboTwin 2.0 averages of 52.56% on Clean and 23.00% on DR, plus 95.9% average success on LIBERO versus π0.5’s 95.8%.On RoboTwin 2.0, it records the best average in both settings; on LIBERO, it is substantially above Diffusion Policy (72.4%) and OpenVLA-OFT (94.5%).Source: §4.2 · p. 9
  • Under matched four-view camera-pose perturbations, π0.5 drops 16.0 pp while VersaCamVLA loses 0.8 pp.The comparison covers RoboTwin 2.0 and real-robot tasks; the LIBERO counterpart is deferred to Appendix B.2.Source: §4.2 · p. 9
  • With matched 4-view inputs, VersaCamVLA exceeds π0.5 by 11.7 percentage points at seen poses and 45.0 points at unseen poses.The real-robot evaluation covers three bimanual tasks; the same VersaCamVLA model is also tested with one extra camera at test time.Source: §4.2 · p. 9

4.3 Ablation Study

  • On RoboTwin 2.0 DR, adding edge or semantic supervision raises average success from 20.94% to 24.75% or 25.69%; combining both reaches 29.50%.The combined result is an 8.56-point improvement over RGB-only supervision.Source: Table 9 · p. 10, §4.3 · p. 10

5 Conclusion

VersaCamVLA augments pretrained policies with a unified scene-token interface and combines multi-signal target-view prediction with Wrist-Augmented Pose Sampling. Experiments report consistent improvements across camera configurations, while the method remains bounded by calibrated-camera tabletop setups and requires further safety-critical validation.

5 Conclusion

  • VersaCamVLA augments pretrained policies with a unified scene-token interface and introduces multi-signal target-view prediction and Wrist-Augmented Pose Sampling.Wrist-camera motion supplies pose diversity, and compact scene tokens condition the pretrained VLA without explicit 3D sensing.Source: §5 · p. 10
  • Experiments on RoboTwin 2.0, LIBERO, and a real-robot platform show consistent improvements over prior VLAs and direct multi-view baselines under camera-pose perturbations and varying camera counts.Source: §5 · p. 10
  • The method assumes calibrated camera intrinsics and extrinsics and focuses on tabletop bimanual setups with manual camera placement; safety-critical deployment needs further validation under large configuration shifts.Uncalibrated cameras, online active perception, and mobile workspaces remain future-work directions.Source: §5 · p. 10

A Benchmark Details

The appendix describes camera pools and evaluation protocols for RoboTwin 2.0 and LIBERO, with benchmark-specific embodiments, task suites, training data, and trial counts.

A Benchmark Details

A.1 RoboTwin 2.0

  • RoboTwin 2.0 uses an Aloha-AgileX bimanual embodiment and 16 selected fine-grained tasks, with 50 Clean-setting demonstrations per task and evaluation in Clean and domain-randomized settings.Each method is evaluated for 100 trials per task per setting, with success rates averaged across the 16 tasks.Source: §A.1 · p. 15

A.2 LIBERO

  • LIBERO follows its four-suite protocol on a Franka Emika Panda arm, using 50 unmodified human demonstrations per task and 500 evaluation trials per suite.The suites probe spatial relationships, object-instance generalization, language-conditioned goals, and long-horizon manipulation.Source: §A.2 · p. 15

A.3 Real-Robot Platform

The real-robot evaluation uses a dual-arm platform for three bimanual tasks, while additional experiments cover camera-pose robustness and other design evaluations.

A.3 Real-Robot Platform

  • The Cobot Magic dual-arm platform evaluates sequential coordination, long-horizon multi-object manipulation, and handover-and-insertion using a pool of four environment and two wrist cameras.Tasks include cube transfer into a bowl, stacking three cubes, and handing a test tube between arms for insertion.Source: §A.3 · p. 15
  • Each real-robot task uses 50 human-teleoperated demonstrations, with 20 evaluation trajectories per task per condition.Source: §A.3 · p. 15
  • Figure 4 contrasts original camera poses with four randomly perturbed pose configurations for two training views on RoboTwin 2.0 Stack Blocks Three.Source: Figure 4 · p. 16

B Additional Experimental Results

  • Additional results cover unseen viewpoints, calibration noise, camera-count and real-robot evaluations, plus per-task ablations and inference-latency measurements.Table 10 presents per-task RoboTwin 2.0 success rates at seen and unseen poses under matched four-view inputs.Source: B Additional Experimental Results · p. 16, Table 10 · p. 16

B.1 Camera-Pose Robustness on RoboTwin 2.0

On RoboTwin 2.0, VersaCamVLA's average success rate changes less between seen and unseen camera poses than π0.5's.

B.1 Camera-Pose Robustness on RoboTwin 2.0

  • Success rate decreases by 0.81 percentage points for VersaCamVLA and 16.00 percentage points for π0.5 at unseen versus seen poses.Source: §B.1 · p. 16
  • The comparison evaluates both methods with matched four-view inputs at trained camera poses and perturbed poses unseen during training.Source: §B.1 · p. 16
  • Table 10 provides the per-task results underlying the RoboTwin 2.0 averages reported in Table 3.Source: §B.1 · p. 16

B.2 Camera-Pose Robustness on LIBERO

VersaCamVLA retains performance across the evaluated unseen camera poses on LIBERO and RoboTwin 2.0, including large azimuth shifts. The RoboTwin evaluation stratifies results by shift magnitude to examine this robustness.

B.2 Camera-Pose Robustness on LIBERO

  • On LIBERO, VersaCamVLA’s average success decreased from 95.70% at seen poses to 95.15% at unseen poses, a drop of 0.55 percentage points.The evaluation used four-view inputs across the four LIBERO suites; π0 and π0.5 are reported at seen poses for reference.Source: §B.2 · p. 16, Table 11 · p. 17

B.3 Generalization to Challenging Unseen Camera Pose Shifts

  • VersaCamVLA’s maximum drop was 1.31 percentage points, while π0.5 fell from 48.88% to 30.06% (18.82 points) at [60◦, 90◦], widening the gap from 4.56 to 23.32 points.The stratified evaluation groups RoboTwin 2.0 results into [0◦, 30◦], [30◦, 60◦], and [60◦, 90◦] absolute azimuth offsets, with matched four-view inputs.Source: §B.3 · p. 17, Table 12 · p. 17

B.4 Robustness under Camera Calibration Noise

VersaCamVLA degrades gradually under the evaluated camera-calibration noise and maintains average success across three to six input views. The supplied real-robot passage describes the evaluation setup but gives no outcome values.

B.4 Robustness under Camera Calibration Noise

  • At σt = 20 mm and σr = 2◦, average success fell from 52.56% to 52.13%; at 50 mm and 5◦, it was 50.38%, down 2.18 points from clean.Each camera received independently sampled Gaussian translation and rotation perturbations fixed throughout the episode; σt and σr are per-axis standard deviations.Source: §B.4 · pp. 17–18, Table 13 · p. 17

B.5 Camera-Count Flexibility on RoboTwin 2.0

  • A single VersaCamVLA model maintained average success between 52.56% and 55.63% across three, four, five, and six input views, with its highest average at six views.π0.5 baselines were trained separately at three and four views, reaching 35.88% and 48.88%; five- and six-view baselines were not evaluated.Source: §B.5 · p. 18, Table 14 · p. 18

B.6 Real-Robot Camera-Pose and Camera-Count Results

  • The real-robot per-task results evaluate one VersaCamVLA model at seen and unseen poses with four views and with a fifth view added at test time.Evaluation used 20 trajectories per task and condition; π0.5 baselines were trained separately for three- and four-view configurations.Source: §B.6 · p. 18, Table 15 · p. 18

B.7 Per-Task Ablation Results

The appendix reports per-task ablations of scene-token count, spatial-encoder architecture, and supervision, alongside latency measurements as input-view count varies. It also identifies the implementation details specified in the following section.

B.7 Per-Task Ablation Results

  • The per-task ablations compare compressed scene-token counts, pooling versus convolutional spatial encoders, and RGB-only versus full multi-signal supervision.The supervision comparison is on RoboTwin 2.0 DR; full supervision combines RGB, semantic, and edge targets.Source: §B.7 · p. 18, Table 17 · p. 19

B.8 Latency and Efficiency

  • From four to six views, VersaCamVLA added 7.06 ms per inference step versus 29.47 ms for π0.5, leaving a 22.14 ms latency gap at six views.At four views, inference took 231.20 ms for VersaCamVLA and 230.93 ms for π0.5; measurements used one NVIDIA 4090 at batch size 1.Source: §B.8 · p. 19

C Implementation Details

  • The following implementation section specifies the architecture, two training stages, and camera-pose sampling protocol used in the experiments.The supplied passage describes the section’s scope.Source: C Implementation Details · p. 19

C.1 Architecture Hyperparameters

The appendix details the scene-token architecture, its two training stages, multi-signal targets, and evaluation-time camera-pose sampling. It also includes technical-appendix submission guidance.

C.1 Architecture Hyperparameters

  • The scene encoder produces 3,072 tokens in a 3 × 32 × 32 layout, while the spatial encoder compresses them to 192 tokens in a 3 × 8 × 8 representation.The spatial encoder maps channel dimension 768 to 2048; the appendix also includes per-task ablations of pooling versus convolutional compression and RGB-only versus full multi-signal supervision.Source: §C.1 · pp. 19–20, Table 18 · p. 20, Table 19 · p. 20

C.2 Stage I: Scene-Token Interface Learning

  • Stage I trains the scene encoder and decoder to predict RGB, semantic, and edge signals at target views, using a combined multi-signal loss.Training uses AdamW for 100,000 steps on eight H100 GPUs; source-view subsets are sampled per frame, while predictions cover available target views.Source: §C.2 · p. 21

C.3 Multi-Signal Supervision Targets

  • Semantic targets come from simulator segmentation in simulation and a fine-tuned SAM3 segmenter on real-robot data; Canny detection supplies edge targets.Figure 6 shows RGB observations, semantic masks, and edge maps across four states of the real-robot test-tube insertion task.Source: §C.3 · p. 21, Figure 6 · p. 22

C.4 Stage II: Scene-Token-Conditioned Policy Learning

  • Stage II freezes the scene encoder and jointly trains the spatial encoder with the base VLA under a flow-matching action loss.The action prediction uses language, native RGB, supplementary scene tokens, and proprioceptive state to predict an action chunk with horizon 50.Source: §C.4 · pp. 21–22

C.5 Camera-Pose Perturbation Sampling

  • Evaluation samples each camera’s azimuth, radial scale, and pitch independently, reorients it toward the workspace, and shares each trial’s configuration across methods.The stratified evaluation restricts absolute azimuth offsets to specified intervals while retaining the other sampling settings.Source: §C.5 · p. 22

D Technical appendices and supplementary material

  • The technical-appendix guidance describes supplementary materials as optional reviewer reading and states that critical supporting experiments should not appear only in the appendix.It also permits technical appendices with no page limit and allows ZIP uploads for videos or code.Source: D Technical appendices and… · p. 22

Related papers

Loading 2610.12451v1…