PaperlayerView exact revision on arXiv

Source-linked AI summary

GS-VLA: Plug-and-Play Viewpoint Canonicalization for Frozen VLA Policies via Gaussian Splatting

Yechan Park, HyunJin Kim

arXiv:2608.19066v1cs.CVcs.AI

TL;DR

Viewpoint shifts can severely degrade frozen VLA policies because deployment observations may differ from training views. GS-VLA adds a lightweight Gaussian-based canonicalizer before the frozen policy and improves success across policies, task suites, and perturbation scales without policy retraining.

  • Problem

    VLA viewpoint robustness remains an open challenge because deployment camera views may differ from training views, while observation-space adaptation is underexplored.

  • Method

    GS-VLA uses a lightweight 3D-Gaussian canonicalizer before a frozen VLA policy, reformulating bounded viewpoint shifts as localized observation-space disocclusion.

  • Results

    +39.8 pp 4-suite mean improvement is achieved across four policies, four LIBERO suites, and εt ∈[5, 200] cm without policy updates.

  • Takeaways & Limitations

    A single lightweight canonicalizer can improve frozen VLA viewpoint robustness across policy architectures, task suites, and perturbation scales.

  • Takeaways & Limitations

    Because the canonicalizer uses a single source frame and calibrated depth, it cannot recover scene content entirely unobserved by that camera.

Abstract

from arXiv · show

This paper proposes a lightweight, plug-and-play framework that improves robustness to viewpoint shifts in Vision-Language-Action (VLA) policies without policy retraining. To our knowledge, this is the first approach to directly leverage 3D Gaussian-based novel-view synthesis for observation-space adaptation in VLA policies. Current VLA performance relies on the implicit assumption that training and deployment camera configurations are identical. Our experiments show that even a small displacement of the camera mount can reduce the success rate on the LIBERO benchmark from about 90% to about 10% in the worst case. Prior approaches, such as large-scale fine-tuning or generative data augmentation, are computationally expensive and risk catastrophic forgetting. To address this, viewpoint shifts are reformulated as a localized novel-view synthesis problem. Under a Locality assumption, that camera perturbations remain within a small bounded region relative to the workspace, viewpoint normalization reduces to a scene- and policy-independent disocclusion task. Our work implements this idea with a 4M-parameter 3D-Gaussian canonicalizer prepended to a frozen VLA policy. Without modifying policy weights, GS-VLA improves performance across three orthogonal axes: (1) Policy architectures, (2) Unseen task suites, and (3) Perturbation scales. These results show that a lightweight visual module can recover a large fraction of the performance lost under viewpoint shift, without policy retraining.

1 Introduction

VLA policies can fail sharply under small deployment-camera viewpoint shifts because observations no longer match training conditions, even when high-level reasoning and control remain capable. GS-VLA addresses this mismatch through Locality-based, lightweight observation-space canonicalization before a frozen policy, avoiding policy retraining.

  • Problem: Small camera-viewpoint deviations can cause substantial VLA performance degradation because deployment observations differ from those seen during training.The introduction identifies viewpoint mismatch as a deployment-robustness problem for VLA policies that otherwise perform strongly on benchmarks and real-world robots.
  • Motivation: Viewpoint-shift failures may reflect observation mismatch rather than failures in high-level reasoning or control.This motivates adapting the observation space instead of relying entirely on policy-parameter learning for viewpoint robustness.
  • Locality assumption: Locality assumes camera perturbations are small relative to the workspace, typically ranging from a few to a few tens of centimeters, so only a limited image region requires geometric recovery.Under this premise, viewpoint adaptation becomes a local disocclusion problem while most pixels remain recoverable.
  • Approach: A lightweight canonicalization module placed before a frozen VLA policy restores aligned observations without policy retraining across policy architectures, task suites, and perturbation scales.The plug-and-play design avoids requiring a large view-synthesis model for arbitrary scenes and camera baselines.
  • Contributions: The paper formulates frozen-policy viewpoint adaptation in observation space and introduces the first Gaussian-based canonicalization module for VLA policies.The formulation uses Locality to reduce viewpoint adaptation to local disocclusion handling.

2 Related Works

Prior work spans frozen VLA foundation policies, camera-robust learning, feed-forward novel-view synthesis, Gaussian splatting, inpainting, and robustness benchmarks. GS-VLA distinguishes itself by applying a lightweight, observation-space canonicalizer to frozen policies for localized camera perturbations.

  • VLA Foundation Policies: VLA foundation policies unified multimodal inputs with large Transformer backbones trained on massive cross-embodiment datasets, achieving ≥90% success on LIBERO under static training camera setups.GS-VLA instead treats these multi-billion-parameter models as frozen, black-box policies.
  • Camera-Robust Robot Learning: Standard camera-robustness methods retrain policies with augmented camera distributions, while SPARTN uses NeRF to generate novel-view trajectories and AnyCamVLA freezes the policy but evaluates only narrow perturbations.AnyCamVLA relies on an LVSM synthesizer with dual-view inputs.
  • Feed-Forward Novel-View Synthesis: Per-scene radiance fields provide high-fidelity novel views but require scene-specific optimization, whereas feed-forward predictors generalize across scenes and diffusion generators use tens to hundreds of millions of parameters.Diffusion-based generators can synthesize plausible views under large baselines.
  • 3D Gaussian Splatting Variants: Feed-forward 3D Gaussian Splatting variants predict per-pixel Gaussians from one or more views, while GS-VLA specializes the pixel-aligned approach to in-distribution camera perturbations.The canonicalizer uses a U-Net to shape per-pixel Gaussians.
  • Image Inpainting: ε-bounded camera shifts create O(ε)-thin holes along depth discontinuities, so GS-VLA avoids semantic hallucination with a 4M-parameter U-Net rather than large image inpainters.State-of-the-art inpainters require tens of millions of parameters for free-form holes.
  • VLA Robustness Benchmarks: LIBERO-Plus identifies camera viewpoint as a major source of VLA fragility, motivating GS-VLA’s observation-space remedy evaluated across policy, task-suite, and perturbation-scale axes.The remedy is orthogonal to internal policy modifications and can be combined with them.

3 Proposed method

GS-VLA canonicalizes shifted observations to a frozen policy’s training viewpoint under a bounded Locality assumption. A compact U-Net uses geometry-aware Gaussian rendering to address only thin disocclusion bands, while operating frame-by-frame without policy updates.

  • Locality reduction: Under Locality, viewpoint normalization reduces to transporting visible content and inpainting a thin disocclusion band rather than synthesizing a general novel view.The newly exposed fraction grows linearly to leading order as η(ρ) = αpar ρpar + αfr ρfr + O(ρ2) for ρ ≤1 in matched units.
  • Canonicalization objective: The canonicalizer predicts the canonical-view image from a source image, depth map, and camera parameters, then feeds it to the frozen policy.The policy parameters remain unchanged, and canonicalization uses neither temporal aggregation, multi-view input, nor per-scene optimization.
  • Locality assumption: The Locality assumption bounds deployment translation and rotation around the canonical pose using tolerances εt and εR.The rotation tolerance is approximated as εR ≈ εt/dmin, allowing overall perturbation to be controlled by εt.
  • Canonicalizer architecture: The canonicalizer is a compact FiLM-modulated U-Net that takes the concatenated source image and depth map and predicts Gaussian descriptors for canonical-pose rasterization.Pose-pair information is embedded as zpose = MLP(Cs, C⋆) ∈R128, while the Gaussian renderer resolves occlusion and fills the disocclusion band.
  • Efficiency and training: 4M parameters make the canonicalizer roughly 840 × smaller than finetuning π0.5 and 42 × smaller than AnyCamVLA.Training uses a standard 3DGS 0.8:0.2 L1/SSIM reconstruction loss and deliberately avoids specialized training techniques.

4 Experiments

Experiments evaluate whether one frozen-policy GS-VLA checkpoint recovers viewpoint-shift performance across policies, LIBERO suites, perturbation scales, and deployment-realistic conditions. GS-VLA consistently improves success rates, with especially large gains where unprotected policies fail most severely.

  • Experimental setup: GS-VLA is a 4M-parameter canonicalizer evaluated against NO CANON and FWD-WARP while all tested VLA policies remain frozen.Experiments use four LIBERO suites and success rate over multiple episodes; the main policy is π0.5, with OpenVLA-OFT, RynnVLA-002, and XVLA for cross-policy transfer.
  • Three-axis generalization: A single GS-VLA checkpoint consistently improves success rates across policy, suite, and perturbation-magnitude generalization, with largest gains where NO CANON fails most severely.The checkpoint is trained once on libero_spatial at εt=100 cm and transfers without retraining across policies, suites, and perturbation scales.
  • Cross-policy generalization: 1.4% to 81.0% (+79.6 pp) on XVLA and 19.8% to 81.6% (+61.8 pp) on OpenVLA-OFT demonstrate cross-policy transfer without fine-tuning.The benefit anti-correlates with each policy’s intrinsic robustness.
  • Cross-suite generalization: 9.3% to 72.1% (+62.8 pp) on π0.5×libero_10 shows zero-shot transfer to an unseen long-horizon suite.The paper attributes the large improvement to reduced error accumulation from keeping the inpainted region small at each step.
  • Perturbation-scale robustness: 42.6% to 86.8% (+44.2 pp) at εt=100 cm and 35.7% to 78.5% (+42.8 pp) at εt=200 cm show robustness across perturbation scales.Across εt ∈ {5, 15, 30, 50, 100, 200} cm, the libero_10 gain peaks at +62.8 pp at εt=100 cm, while ∆(εt) is approximately linear for εt≤100 cm.
  • Deployment realism: ∆ remains within [+21, +42] pp across joint translation-and-yaw conditions, while calibration error within roughly 3 cm and 3° causes a three-suite mean drop of at most 5 pp.Larger yaw narrows ∆, attributed to undersampling of large rotations during training; degradation is more pronounced at 5 cm and 5° calibration noise.

5 Ablations

The ablations show that GS-VLA’s compact design and modest data requirements are sufficient, while replacing calibrated metric depth substantially harms policy performance. Increasing capacity improves reconstruction but not success rate, and training saturates at roughly 10k pairs.

  • Capacity: 15 M parameters raise reconstruction PSNR from 26.84 to 28.4 dB but lower SR to 85.2%, showing that extra capacity does not improve policy success.Beyond about 4M parameters, added capacity produces texture detail that the policy does not use.
  • Data: 84–87% SR across 10k, 25k, and 50k training pairs indicates that performance saturates at about 10k pairs.The saturation is consistent with Locality’s prediction that the small inpainting region has low intrinsic learning complexity.
  • Depth substitution: 44.5% SR with zero-shot DepthAnything V2, even after per-image scale/shift calibration, shows that metric-aligned depth is critical.Per-frame calibration cannot recover the absolute scale required by (5).

6 Limitation

The results are bounded by single-view observability, limited evaluation of combined translation and rotation, and reliance on simulator experiments. Multi-view extensions and real-robot replication remain future directions.

  • Single-view input: Single-view input prevents recovery of scene content entirely unobserved by the source camera.The canonicalizer consumes one source frame and calibrated depth; multi-view fusion is proposed as a future extension.
  • Locality boundary on rotation: εt=100 cm relative to a ∼0.7 m workspace already creates a rotation-sweep boundary because additional rotation can move the camera off the workspace.The paper could not run a clean rotation sweep beyond the joint (εt, θ) cells reported in Table 3.
  • Real-world evaluation: All experiments run inside the LIBERO simulator, leaving real-robot validation as a preliminary and uncompleted evaluation step.Real-world validation requires an external metric-depth source and engineering effort to instrument a physical rig, which the project could not provide.

7 Conclusion

GS-VLA is a 4M-parameter 3D-Gaussian canonicalizer that improves frozen VLA policies’ camera-viewpoint robustness without policy updates. Its design reduces bounded-perturbation canonicalization to a scene- and policy-independent disocclusion problem over an O(ρ)-area region.

  • Locality reduction: Under bounded perturbations, viewpoint canonicalization becomes a scene- and policy-independent disocclusion problem over an O(ρ)-area region.This reduction is identified as the Locality reduction.
  • Validation: The η(ρ) linearity link is confirmed at R2 ≥0.99, while the full ε→η→residual→∆ chain closes at R2 ∈{0.99, 0.81, 1.00}.These reported fits support the Locality-based reduction.

A Appendix · A.1 Proposition 1 of locality reduction

Under the Locality assumption, viewpoint canonicalization reduces to closed-form recovery plus learned disocclusion of a thin depth-edge band. Proposition 1 characterizes this band’s linear dependence on translation, depth, rotation, and image geometry, and connects it to policy success-rate degradation and same-domain transfer.

  • A.1 Proposition 1 of locality reduction: Locality partitions the canonical view into a closed-form depth-warp region and a thin disocclusion band of fractional area η along depth edges.The disocclusion band has no defining input pixel and must be filled by a learned prior.
  • A.1 Proposition 1 of locality reduction: Proposition 1 decomposes viewpoint displacement as ρ = ρpar + ρfr, with ρpar = ∥ts − t⋆∥/dmin and ρfr = ∥log(R⋆⊤Rs)∥ for ρ ≤ 1.The proposition assumes piecewise-smooth depth d ∈ [dmin, dmax] and silhouette edges meeting view rays transversely.
  • A.1 Proposition 1 of locality reduction: The parallax contribution scales with translation and depth contrast, whereas the frame contribution scales with rotation and image perimeter.px denotes canonical-view silhouette arc-length, Ppx the image-frame perimeter, Apx = HW the image area, and clow, C2 depend only on scene-regularity bounds.
  • A.1 Proposition 1 of locality reduction: η = ηpar + ηfr, with pure rotation giving ηpar = 0 while pure translation still has ηfr ≠ 0 because translation induces image-plane motion.ηpar and ηfr denote the parallax and frame contributions to the disocclusion area.
  • A. Appendix: R2 ≥ 0.99 validates the predicted linear regime to ρ ≈ 0.5, while matching upper and lower bounds certify that the linear rate cannot be improved.The upper bound integrates a parallax shadow band along each silhouette, and the lower bound uses Cauchy–Schwarz on the silhouette-normal field.
  • A.1 Proposition 1 of locality reduction: Because αpar and αfr depend only on silhouette–depth and image-frame geometry, a canonicalizer trained on one suite transfers to others in the same visual domain.These coefficients do not depend on texture, lighting, or policy.
  • A.1 Proposition 1 of locality reduction: The success-rate drop is ∆(ρ) = Lπ · crecon · (αparρpar + αfrρfr) + O(ρ2), coupling locality-induced area, reconstruction residual, and policy sensitivity.Here Lπ is the policy’s local Lipschitz constant on canonical inputs, and crecon is the residual–η slope of fϕ; Appendix A.2 numerically closes the chain.

A.2 Empirical validation of proposition 1

Controlled synthetic experiments validate Proposition A.1’s Locality predictions at both the geometric and end-to-end levels. The measured relationships support linear perturbation growth, silhouette-localized disocclusion, and propagation from reconstruction residuals to success-rate loss.

  • Linearity: η(ρ) is linear for ρ ≤0.5, with R2 ≥0.99 across perturbation modes and suites, while η(ρ)/ρ ∈[0.27, 1.09].Departure from linearity begins around ρ ≈0.5, matching the proposition’s predicted validity boundary.
  • Scene decomposition: Rotation slopes are nearly scene-invariant at 1.084 on spatial versus 1.089 on object, whereas translation slopes vary by about 60%.The rotation-slope discrepancy is 0.4%, consistent with the predicted dependence on image-plane perimeter rather than scene silhouette length.
  • Silhouette hugging: Occluded pixels lie 5–7 px from the nearest depth edge, versus 27–33 px for uniform-random pixels, yielding roughly 5× tighter localization.This concentration along depth silhouettes matches the geometric prediction of Proposition A.1.
  • End-to-end chain: Pearson r = 0.90 and R2 = 0.81 describe the residual–η link across 200 pairs, while composing ε→η→residual→∆ implies L ≈213 pp per L1-residual unit.The full chain closes numerically with R2 ∈{0.99, 0.81, 1.00}, and the estimated policy Lipschitz behavior is log Q_l(1 + L_l) = 389 across 421 weight matrices.

A.2.1 Locality slopes by mode × suite … A.4.7 Depth-substitution ablation

The appendix validates Locality across perturbation modes, policies, suites, and scales, while detailing a compact frozen-policy implementation and ablations on capacity, data, intrinsics, and depth. GS-VLA maintains gains within suite-specific Locality boundaries, with calibration and depth substitution identifying deployment limitations.

  • A.2.1 Locality slopes by mode × suite: Locality slopes are fit separately for translation, rotation, and joint perturbations across spatial and object suites, with parallax, frame, and total slopes reported.Silhouette-hugging occluded pixels average 5.05/6.78 px from depth edges versus 26.77/33.10 px for uniform-random pixels, a 5× tighter concentration.
  • A.3.1 Implementation detail: The implementation updates only the canonicalizer, initializes it at the closed-form forward warp, and leaves the downstream policy untouched.No discriminator or perceptual GAN loss is used.
  • A.3.1 Implementation detail: Canonicalizer cost is amortized across the policy’s native action-chunk schedule, including 50 actions per frame for π0.5.Softmax splatting is not used in the released checkpoint.
  • A.3.2 Data setup: Training uses 49,800 pairs and 200 validation pairs from 50,000 libero_spatial source→canonical pairs, with cross-suite evaluation remaining zero-shot.Pairs use canonical-camera simulator rollouts re-rendered under perturbed cameras; no colour jitter, multi-suite mixing, or per-task balancing is used.
  • A.3.3 Software framework: The software stack comprises PyTorch 2.4, CUDA 12.1, gsplat 0.1.x, robosuite 1.4, and lerobot.Released code bundles random seeds, hyperparameter dumps, and checkpoint-producing command lines.
  • A.4.1 Full cross-policy × suite matrix: Every policy–suite cell improves at εt=100 cm, with gains reaching +79.6 pp for XVLA, +62.8 pp for π0.5 on libero_10, and +61.8 pp for OpenVLA-OFT on object-centric tasks.The largest gains occur where baselines are closest to total failure.
  • A.4.2 GS-VLA absolute SR across suites (ε-sweep): Object and goal suites remain within ∼2 pp of their εt=5 ceilings through εt=100 cm, while the main drop at εt=200 cm marks their Locality boundary.Under joint position and look-at scaling, the canonicalizer decays more slowly than NO CANON and the utility gap widens until that boundary.
  • A.4.4 Capacity ablation: SR peaks at C=32 despite improving reconstruction PSNR with larger widths, while the signal saturates near 25k pairs and 50k provides only ∼2 pp over 10k.Extra capacity primarily adds texture detail that the policy does not consume.
Loading 2608.19066v1…