PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models

Hyeongjun Heo, Seungyeon Woo, Sang Min Kim, Junho Kim, Junho Lee, Yonghyeon Lee, Young Min Kim

arXiv:2603.05868v2cs.RO

TL;DR

Fine-tuned VLAs can be sensitive to camera viewpoint changes, limiting their robustness across configurations. AnyCamVLA uses feed-forward novel-view synthesis to adapt test-time observations to the training camera viewpoint without policy fine-tuning or architectural changes. It improves viewpoint robustness on LIBERO and in real-world settings, while its synthesis quality, latency, and memory needs bound its use.

  • Problem

    Fine-tuned VLAs can overfit to their training camera configurations, making viewpoint changes a challenge for robust deployment.

    Source: §I · p. 1

  • Method

    AnyCamVLA uses feed-forward novel-view synthesis to transform test-time observations into training-view images for a frozen VLA, without additional demonstrations or architectural changes.

    Source: §I · pp. 1–2, B. Zero-Shot Camera Adaptation via… · p. 3

  • Results

    AnyCamVLA achieves the best task success rate over LIBERO baselines under extrinsic shifts up to 15 cm and 60° rotation, and reports minimal performance degradation.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework offers plug-and-play camera adaptation for RGB-based VLAs while preserving their original visual input modality and pre-trained capabilities.

    Source: B. Zero-Shot Camera Adaptation via… · p. 3

    Performance can fail when novel-view reconstruction degrades, including with a single distant source view or large occlusions; synthesis adds about 30 ms per frame and requires additional GPU memory.

    Source: §V · p. 7

Abstract

from arXiv · show

Despite remarkable progress in Vision-Language-Action models (VLAs) for robot manipulation, these large pre-trained models require fine-tuning to be deployed in specific environments. These fine-tuned models are highly sensitive to camera viewpoint changes that frequently occur in unstructured environments. In this paper, we propose a zero-shot camera adaptation framework without additional demonstration data, policy fine-tuning, or architectural modification. Our key idea is to virtually adjust test-time camera observations to match the training camera configuration in real-time. For that, we use a recent feed-forward novel view synthesis model which outputs high-quality target view images, handling both extrinsic and intrinsic parameters. This plug-and-play approach preserves the pre-trained capabilities of VLAs and applies to any RGB-based policy. Through extensive experiments on the LIBERO benchmark, our method consistently outperforms baselines that use data augmentation for policy fine-tuning or additional 3D-aware features for visual input. We further validate that our approach constantly enhances viewpoint robustness in real-world robotic manipulation scenarios, including settings with varying camera extrinsics, intrinsics, and freely moving handheld cameras. Project Page: https://heo0224.github.io/AnyCamVLA/

I. INTRODUCTION

Fine-tuned VLAs are sensitive to camera changes, yet existing robustness approaches can demand more training data or architectural changes. AnyCamVLA instead synthesizes test-time observations into the training viewpoint, preserving the frozen policy while improving performance under viewpoint shifts.

I. INTRODUCTION

  • Fine-tuned VLAs can overfit to training camera configurations, and a 3 cm wrist-camera shift can halve success rate.Even small hardware changes can require collecting new demonstrations and fine-tuning the entire model.Source: §I · p. 1
  • Viewpoint augmentation requires more data and training, while geometry-aware or view-invariant inputs can require architectural changes and limit use of pre-trained RGB priors.These trade-offs motivate adapting observations at test time instead.Source: §I · p. 1
  • AnyCamVLA adapts viewpoints without additional demonstrations, policy fine-tuning, or architectural modifications.It uses a feed-forward novel-view synthesis model to generate training-view images while retaining the VLA’s frozen latent space.Source: §I · pp. 1–2
  • Up to 15 cm translation and 60° rotation in camera extrinsics cause minimal performance degradation with AnyCamVLA, which achieves the best task success rate over LIBERO baselines.The approach also reports consistent performance across real-world camera configurations, including handheld capture.Source: §I · p. 2

II. RELATED WORK

The related work describes camera viewpoint sensitivity in VLAs, reviews representation- and augmentation-based responses, and situates feed-forward novel-view synthesis as an option for online robotics.

A. Vision-Language-Action Models

  • VLAs can show substantial performance degradation under camera perturbations, with reported success rates dropping from over 90% to below 30%.The section attributes viewpoint overfitting to operation on 2D RGB observations without explicit spatial supervision or camera-pose awareness.Source: A. Vision-Language-Action Models · p. 2

B. Viewpoint Robust Robotic Policies

  • Point-cloud approaches can canonicalize representations across viewpoints, but open-source large-scale robotic datasets lack point-cloud data, limiting scalability.GeoAwareVLA instead uses implicit geometry-aware features from a pre-trained multi-view vision model; augmentation methods still require costly policy retraining.Source: B. Viewpoint Robust Robotic Policies · p. 2
  • AnyCamVLA uses novel-view synthesis to transform observations at test time without depth sensors, architectural modifications, or additional policy training.This contrasts with prior viewpoint augmentation that trains on augmented demonstrations.Source: B. Viewpoint Robust Robotic Policies · p. 2

C. Novel View Synthesis for Robotics

  • Optimization-based NeRF and 3D Gaussian Splatting require per-scene training, whereas feed-forward models predict novel views or 3D representations in one pass.The single-pass approach enables real-time inference suited to online robotic manipulation.Source: C. Novel View Synthesis for Robotics · p. 2

III. METHOD

The method formulates camera adaptation for a VLA trained with known camera parameters and evaluated under a different camera configuration. Its goal is robust inference without retraining or fine-tuning.

III. METHOD

  • Training cameras are specified by parameters that include both extrinsics, such as pose, and intrinsics, such as focal length and principal point.The policy maps images from the training cameras and a language instruction to actions.Source: A. Problem Setup · p. 2
  • At inference, the test camera parameters differ from the training parameters, and the method assumes both sets are known in a shared coordinate frame.The paper notes that camera calibration can be performed once during system setup.Source: A. Problem Setup · pp. 2–3
  • The objective is to maintain robust policy performance across camera changes without retraining or fine-tuning, preserving the VLA’s pre-trained capabilities.Source: A. Problem Setup · p. 3

B. Zero-Shot Camera Adaptation via Feed-Forward Novel View Synthesis

The camera adaptation module synthesizes test-time observations as images from the training camera configuration, then feeds them to the unchanged VLA. A feed-forward novel-view synthesis model makes this approach suitable for real-time, plug-and-play use.

B. Zero-Shot Camera Adaptation via Feed-Forward Novel View Synthesis

  • The module F transforms test-time observations to match the training camera configuration, and the resulting images are passed to the frozen policy.The camera adaptation module can accept and produce different numbers of views.Source: B. Zero-Shot Camera Adaptation via… · p. 3
  • A feed-forward novel-view synthesis model generates photorealistic views from source images and camera parameters in a single forward pass.Unlike NeRF, it does not require per-scene optimization, making it suitable for real-time robotic control.Source: B. Zero-Shot Camera Adaptation via… · p. 3
  • The pipeline captures current-camera images, synthesizes training-view images, feeds them to the frozen policy, and executes its action.LVSM synthesizes two novel views from two input views at 256×256 resolution with 36.55 ms latency on an NVIDIA RTX 4090 using BF16 mixed precision, corresponding to 27 FPS.Source: B. Zero-Shot Camera Adaptation via… · p. 3
  • The framework requires no additional demonstrations or policy fine-tuning and applies to RGB-based VLAs without architectural modifications.It preserves the original visual input modality and the pre-trained visual-linguistic reasoning capabilities.Source: B. Zero-Shot Camera Adaptation via… · p. 3

IV. EXPERIMENTS

The experiments test whether feed-forward novel-view synthesis improves robustness to camera changes, and whether zero-shot adaptation compares favorably with policy fine-tuning in efficiency and flexibility. They also examine the view-adaptation approach and real-world manipulation performance.

  • The experiments ask how feed-forward novel-view synthesis affects robustness to camera changes compared with baselines.Source: §IV · p. 3
  • They compare zero-shot adaptation with policy fine-tuning on efficiency and flexibility.Source: §IV · p. 3
  • They examine whether feed-forward novel-view synthesis is the best approach for view adaptation.Source: §IV · p. 3
  • They test whether the method performs effectively in real-world robot manipulation scenarios.Source: §IV · p. 3

A. Simulation Benchmark

The simulation benchmark evaluates viewpoint robustness on LIBERO, whose two input images come from agent and wrist cameras. Camera perturbations isolate changes to either camera, and the baselines include unperturbed VLAs, viewpoint-augmented fine-tuning, and a geometry-aware representation method.

A. Simulation Benchmark

1) Baselines:

  • The unperturbed base policies are OpenVLA-OFT and π0.5, both trained on LIBERO without camera perturbations.Source: 1) Baselines · p. 4
  • The data-augmentation baseline fine-tunes π0.5 on rendered viewpoints, sampling 50 agent-camera and 15 wrist-camera views per perturbation.Source: 1) Baselines · p. 4
  • GeoAware-VLA replaces the RGB encoder with VGGT-derived 3D-aware features, while the proposed method adds an LVSM decoder-only visual adapter to frozen base policies.Source: 1) Baselines · p. 4
  • LVSM is fine-tuned on 491 LIBERO-Plus-derived scenes with 64 viewpoint variations per scene, using multi-view images without robot action data.Source: 1) Baselines · p. 4

2) Viewpoint Robustness Evaluation:

The proposed adaptation method achieves the strongest reported performance across LIBERO suites under agent-camera perturbations and outperforms data augmentation and GeoAwareVLA on wrist-camera perturbations in LIBERO-Long. The section also discusses why wrist-camera changes challenge policies and geometry-aware features differently.

2) Viewpoint Robustness Evaluation:

  • 94.5% average success rate is achieved by Ours-π across agent-camera perturbation levels, with the methods reported as best across LIBERO suites.The base policies degrade as camera variation increases: OpenVLA-OFT drops from 85.2% (small) to 46.2% (large), and π0.5 from 92.4% to 39.9%.Source: 2) Viewpoint Robustness Evaluation · p. 4
  • 88.6% average success rate on LIBERO-Long wrist-camera perturbations exceeds both data augmentation and GeoAwareVLA.GeoAwareVLA falls below 10% across wrist-camera perturbation levels despite 86.1% on agent-camera perturbations.Source: 2) Viewpoint Robustness Evaluation · p. 4
  • π0.5 is more sensitive to wrist-camera changes than agent-camera changes, which the authors link to reliance on close-range geometric and contact information.Its reported success drops to 28.6% under wrist-camera changes versus 49.0% under agent-camera changes.Source: 2) Viewpoint Robustness Evaluation · p. 4
  • Figure 3 plots LIBERO-Long success against fine-tuning steps, separating original-view performance from performance across three unseen views.Source: Figure 3 · p. 5
  • The proposed method synthesizes canonical-view RGB images, whereas wrist-camera perturbations can misalign GeoAwareVLA’s wrist-centric geometric reference frame.Source: 2) Viewpoint Robustness Evaluation · p. 5

3) Zero-Shot Adaptation vs. Policy Fine-Tuning:

The fine-tuning comparison varies task coverage and tracks success on novel and original viewpoints over training. Fine-tuning can fail to transfer across tasks and degrade original-view performance, whereas zero-shot visual adaptation keeps the policy frozen and avoids those fine-tuning costs and forgetting risks.

3) Zero-Shot Adaptation vs. Policy Fine-Tuning:

  • The comparison tests π0.5 fine-tuned with 1, 5, or 10 LIBERO-Long tasks, tracking novel-view and original-view success throughout training.Source: 3) Zero-Shot Adaptation vs. Policy… · p. 5
  • Fine-tuning on one task decreases novel-view performance on the other nine tasks, and original-view performance declines across all tested task-coverage settings.The reported decline persists even when fine-tuning uses all 10 tasks, consistent with catastrophic forgetting.Source: 3) Zero-Shot Adaptation vs. Policy… · p. 5
  • Viewpoint generalization across diverse tasks requires demonstrations from many tasks, while fine-tuning risks degrading performance on original viewpoints.Source: 3) Zero-Shot Adaptation vs. Policy… · p. 5
  • Qualitative results indicate that fine-tuning LVSM reduces the domain gap for simulation input views.Source: Figure 4 · p. 5
  • Zero-shot adaptation keeps the policy frozen and adapts only visual input, requiring no additional robot demonstrations or policy training and avoiding forgetting risk.Source: 3) Zero-Shot Adaptation vs. Policy… · p. 5

4) Ablation on Viewpoint Adaptation Methods:

The ablation compares feed-forward novel view synthesis with geometric adaptation methods and finds that the full method outperforms them across perturbation levels. Fine-tuning LVSM helps it produce coherent views for LIBERO and retain near-original task performance.

  • The comparison includes homography, depth-based projection, pre-trained LVSM without fine-tuning, and the full method with LVSM fine-tuned on a custom multi-view dataset.Homography assumes a planar table-height scene; depth-based projection reprojects a point cloud, and geometric baselines use Telea inpainting for unobserved regions.Source: 4) Ablation on Viewpoint Adaptation… · pp. 5–6
  • 88.6% vs. 92.4% on the original view, our method maintains near-original performance and outperforms geometric baselines across perturbation levels.The comparison covers image synthesis quality and task success rate.Source: 4) Ablation on Viewpoint Adaptation… · p. 6
  • Without fine-tuning, LVSM produces blurry or geometrically inconsistent views because its real-world training data and camera trajectories differ from LIBERO’s synthetic images and dual-camera setup.Fine-tuning on the custom multi-view dataset bridges the domain and camera-configuration gaps.Source: 4) Ablation on Viewpoint Adaptation… · p. 6
  • LVSM’s one-time fine-tuning uses multi-view images without robot action labels and is more efficient than fine-tuning a much larger VLA.LVSM has 171M parameters, compared with 7B for OpenVLA-OFT and 3.3B for π0.5.Source: 4) Ablation on Viewpoint Adaptation… · p. 6

B. Real-World Experiments

The real-world experiments evaluate whether zero-shot camera adaptation improves viewpoint generalization in manipulation. Across the tested tasks, the method maintains success rates comparable to those with the training viewpoint despite novel viewpoints.

1) Experimental Setup:

  • Four manipulation tasks use 20 demonstrations each, with a ZED2 stereo camera providing two training views and a different-position camera replacing the right-side view at test time.The π0.5 base model is fine-tuned with LoRA for 10,000 steps; the original RealEstate10K-trained LVSM decoder-only checkpoint is used at inference.Source: 1) Experimental Setup · p. 6

2) Performance Improvement on Novel Viewpoints:

3) Robustness on Various Handheld Cameras:

Handheld-camera experiments report real-time operation under moving cameras and variation in camera parameters. The conclusion also identifies reconstruction quality, latency, memory, and target-view selection as boundaries or open challenges.

3) Robustness on Various Handheld Cameras:

  • With three freely moved handheld cameras, the method operates in real time while generalizing across extrinsics, intrinsics, and image characteristics.The cameras are a ZED2 stereo camera, an Intel RealSense D435, and an iPhone 17 Pro; their poses are estimated using table-mounted ArUco markers.Source: 3) Robustness on Various Handheld… · p. 7

V. CONCLUSION

  • The framework enables viewpoint generalization without additional data collection, policy fine-tuning, or architectural modification of pre-trained VLAs.Feed-forward novel view synthesis serves as the camera-adaptation module while maintaining the policy inference loop.Source: §V · p. 7
  • Reconstruction can fail with a single source camera and a distant target view or large occlusions; synthesis adds about 30 ms per frame and requires extra GPU memory.Latency may pose challenges in extremely dynamic scenarios.Source: §V · p. 7
  • Automatic target-view selection remains an open challenge when training camera configurations vary across demonstrations.Source: §V · pp. 7–8

Related papers

Loading 2603.05868v2…