PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRe-VLA: Preserving Cross-View Coordination in VLAs under Camera Shifts

Tianhang Pan, Xuanhao Wang, Yiwen Pang, Bo Zhou, Jun Yang, Min-Ling Zhang, Shimin Di

arXiv:2609.37150v1cs.RO

TL;DR

Camera shifts can substantially reduce VLA task success, while existing approaches may require extra data collection or training. CoRe-VLA restores a training-time viewpoint using reconstructed point clouds and image restoration, and substantially improves viewpoint robustness across mainstream VLAs under camera shifts.

  • Problem

    Camera shifts substantially degrade VLA task success, while existing fine-tuning and visual-adaptation approaches can require additional data preparation, multi-view supervision, or training costs.

    Source: §1 · p. 1

  • Method

    CoRe-VLA reconstructs a scene point cloud and renders the observation from the training viewpoint, using R2C to reduce rendering artifacts and ETCA for asynchronous control.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    CoRe-VLA substantially improves viewpoint robustness across mainstream VLAs under camera shifts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The work identifies cross-view coordination breakdown under camera shifts and shows it can be mitigated by restoring observations familiar to the VLA during training.

    Source: §6 · p. 9

    CoRe-VLA relies on reasonably accurate camera depth measurements, and reprojection or R2C cannot reliably recover regions newly revealed by large viewpoint changes or occlusions.

    Source: §6 · p. 10

Abstract

from arXiv · show

VLAs combine pretrained vision-language representations with action generation to enable language-guided control across diverse tasks, becoming a mainstream paradigm in embodied intelligence. However, multiple studies have reported VLA's substantial declines in task success under camera shifts, revealing a key vulnerability that limits reliable deployment. To address this vulnerability, existing methods collect paired observations of the same scene from different viewpoints to fine-tune the VLA or train visual adaptation modules. Unfortunately, they require additional data collection and VLA training costs. In this paper, we first identify \emph{cross-view coordination breakdown} under external camera shifts: the robot may rely too heavily on wrist-view cues and consequently execute subtasks in the wrong order when losing global view. Motivated by this, we propose CoRe-VLA, a plug-and-play framework requiring neither additional multi-view data collection nor VLA fine-tuning, which can incorporate with exsiting VLAs. It reconstructs a scene point cloud and renders the observation from the VLA's training viewpoint to restore cross-view coordination. In CoRe-VLA, Render-to-Camera (R2C) Restoration reduces rendering-induced visual degradation, while Execution-Trajectory-Conditioned Alignment (ETCA) reduces robot idle time and mitigates motion conflicts during asynchronous execution. Experiments on 5 real-robot tasks, LIBERO-100 and LIBERO-Plus demonstrate CoRe-VLA substantially improves task success across mainstream VLAs under camera shifts. For example, CoRe-VLA raises PI0.5's success rate from 13.3% to 83.3% at a 1.6m camera shift in real-robot environment.

1 INTRODUCTION

Camera shifts sharply reduce VLA task success, and can disrupt coordination between wrist-view cues and global task state. CoRe-VLA restores a familiar external viewpoint without additional multi-view data collection or VLA retraining.

1 INTRODUCTION

  • At a 0.4 m camera shift, π0.5’s success rate falls from 96.7% to 66.7%.Task success declines as external-camera displacement increases across the five real-robot tasks.Source: §1 · p. 2
  • Existing viewpoint-robustness approaches can require additional data preparation, multi-view supervision, or VLA training costs.The paper contrasts CoRe-VLA with fine-tuning and visual adaptation approaches.Source: §1 · p. 2
  • Under shifted views, robots may over-rely on wrist-view cues and execute subtasks in the wrong order despite successful local manipulation.The paper’s example is grasping a tape measure while the basket lid remains closed.Source: §1 · p. 2
  • Render-to-Camera Restoration reduces rendering artifacts, while ETCA supports asynchronous control to reduce idle time and motion conflicts.R2C uses degraded image pairs constructed from existing demonstration videos; ETCA aligns new action chunks with the recorded execution trajectory.Source: §1 · pp. 2–3
  • CoRe-VLA is a plug-and-play approach that improves viewpoint robustness without additional multi-view data collection or VLA retraining.It reconstructs a scene point cloud and renders the observation from the VLA’s training viewpoint.Source: §1 · pp. 2–3

2 RELATED WORK

Related work documents VLA sensitivity to viewpoint changes, including in world action models. Existing reconstruction approaches trade scene-specific fitting or multi-view capture against limitations in image quality and latency.

VLAs and Viewpoint Robustness

  • Viewpoint perturbations are reported to degrade VLA performance, and lightweight visual adaptation can recover performance in one study.LIBERO-Plus identifies viewpoint perturbations as a vulnerability; VLA-Arena (Zhang et al., 2025) reports sharp degradation, while Li et al. (Li et al., 2026) attribute failures primarily to spatial representation misalignment.Source: §2 · p. 3

World Action Models

  • World action models remain sensitive to camera shifts, and adopting a WAM alone does not resolve viewpoint sensitivity.Selective Cross-View Consistency (Huang et al., 2026a) addresses the issue through training on same-state viewpoint pairs.Source: §2 · p. 3

Point Cloud Reconstruction

  • Neural point rendering may require scene-specific fitting, while Gaussian splatting pipelines may require multi-view capture and per-scene optimization.Direct RGB-D reprojection avoids scene-specific optimization but depends on depth accuracy and observation coverage, and can degrade texture, color, and sharpness.Source: §2 · p. 3

3 UNDERSTANDING VLA FAILURES UNDER EXTERNAL-CAMERA SHIFTS

Across five real-robot tasks, increasing external-camera displacement reduces VLA success, with task-state inconsistency becoming the dominant failure mode. The diagnostic findings motivate restoring the nominal external view to support coordination between global context and wrist-level cues.

Real-World Diagnostic Setup

  • The five-task benchmark covers basic manipulation, long-horizon execution, and spatial reasoning using a UR7e robot with external RGB-D and wrist cameras.Three models are evaluated after fine-tuning on 100 demonstrations per task at the nominal external-camera pose; the external camera is then shifted by 0.4, 0.8, 1.2, or 1.6 m.Source: §3 · p. 4
  • Failure Type 2 captures locally successful manipulation that conflicts with the global task state, such as grasping the tape measure before opening the lid.The paper also describes grasping a cup based on wrist-view cues despite a global spatial constraint.Source: §3 · p. 4

Quantitative and Qualitative Failure Analysis

  • π0.5 success falls from 98.0% at the nominal viewpoint to 13.3% at 1.6 m, while Type 2 failures rise from 0% to 76.7% of trials.Under nonzero displacements, Type 1 local execution failures account for 10.0% to 13.3% of trials.Source: §3 · p. 4
  • The findings motivate restoring the nominal external view so the VLA can coordinate global task context with wrist-level visual cues.The paper identifies over-reliance on wrist-view cues as a reason subtasks may be executed in the wrong order.Source: §3 · p. 4

4 METHODOLOGY

CoRe-VLA restores the VLA’s training viewpoint through geometric reprojection, then addresses rendering artifacts and asynchronous action-chunk switching with R2C and ETCA.

4 METHODOLOGY

  • CoRe-VLA combines geometric viewpoint restoration, R2C, and ETCA without view-transformation training or paired multi-view data.Source: §4 · p. 5, §4.1 · p. 5

4.1 GEOMETRIC VIEWPOINT RESTORATION

  • Direct RGB-D reprojection transforms valid-depth pixels into the training camera frame and rasterizes their source colors into a familiar-view image.The resulting image provides a basis for restoring coordination between global task context and wrist-level visual cues.Source: §4.1 · p. 5

4.2 RENDER-TO-CAMERA RESTORATION

  • R2C uses a lightweight NAFNet trained with perceptual, color, and pixel losses to recover texture, color consistency, and detail from rendered images.Its training pairs are constructed from existing 2D task demonstration videos, avoiding additional multi-view collection; the VLA remains frozen.Source: §4.2 · pp. 5–6

4.3 EXECUTION-TRAJECTORY-CONDITIONED ALIGNMENT

  • ETCA matches the robot’s recorded motion to a prefix of the incoming action chunk, then resumes execution from the remaining suffix.It checks path and terminal-state errors, using motion direction and elapsed inference time as auxiliary guidance; rejected chunks trigger inference from the latest observation.Source: §4.3 · pp. 6–7, Figure 4 · p. 6

5 EXPERIMENT

CoRe-VLA improves viewpoint-shift performance across real-robot and simulation evaluations while keeping VLA parameters frozen. The experiments also assess R2C restoration and ETCA’s effect on execution efficiency.

5.1 EXPERIMENTAL SETUP

  • The evaluation keeps VLA parameters frozen and focuses on external-camera position shifts across five real-robot tasks, LIBERO, and LIBERO-Plus.The real-robot comparison uses matched camera positions, object layouts, and control settings; wrist images remain unchanged.Source: §5.1 · p. 7, §5.2 · p. 7, §5.3 · p. 8

5.2 EVALUATION IN REAL-ROBOT ENVIRONMENT

  • At a 1.6 m shift, CoRe-VLA raises π0.5 success from 13.3% to 83.3% and GR00T success from 6.7% to 80.0% on real-robot tasks.Across evaluated displacements, restored observations maintain substantially higher success than raw observations.Source: Figure 5 · p. 8, §5.2 · p. 8
  • At 1.6 m, Type 2 failure proportions fall from 76.7% to 13.3% for π0.5 and from 80.0% to 13.3% for GR00T.The passage identifies Type 2 as task-state inconsistency and describes it as the dominant failure pattern.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 EVALUATION ON LIBERO

  • At 0.9 m on LIBERO-100, CoRe-VLA raises π0.5 success from 13.8% to 71.5% and OpenVLA-OFT success from 5.9% to 55.1%.The evaluation spans all 100 tasks and camera displacements up to 0.9 m; shifts of 1 m or more were not evaluated because LIBERO scenes are limited in size.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4 EVALUATION ON LIBERO-PLUS

  • On LIBERO-Plus Camera, CoRe-VLA raises π0.5 success from 72.0% to 89.1%, a gain of 17.1 percentage points without VLA fine-tuning or additional multi-view data.Anchor-Align reports a higher score but adapts the VLA, whereas CoRe-VLA leaves policy parameters unchanged.Source: Table 3 · p. 8, §5.4 · p. 9

5.5 ABLATION STUDY

  • Without R2C, real-robot success falls from 96.7% with raw images to 83.3% at zero displacement, while R2C raises it to 93.3%.At 1.6 m, success is 50.0% without R2C and 83.3% with it, a gain of 33.3 percentage points.Source: §5.5 · p. 9, Figure 7 · p. 9
  • ETCA reduces CoRe-VLA mean completion time from 64.5 to 43.7 s (32.2%) and idle time from 39.1 to 13.8 s (64.7%).The passage attributes the time reduction to asynchronous inference and execution rather than faster individual inferences.Source: §5.5 · p. 9, Table 4 · p. 9

6 CONCLUSION AND LIMITATIONS

CoRe-VLA addresses cross-view coordination breakdown through point-cloud reconstruction and re-rendering, without additional data collection or VLA fine-tuning. The paper notes depth and unseen-region constraints, and describes its AI use and appendices.

6 CONCLUSION AND LIMITATIONS

  • The paper presents cross-view coordination breakdown as a major cause of task failure and says CoRe-VLA mitigates it by restoring training-familiar joint observations without extra data or VLA fine-tuning.The approach uses point-cloud reconstruction and re-rendering.Source: §6 · p. 9
  • CoRe-VLA relies on reasonably accurate camera depth measurements, and reprojection or R2C cannot reliably recover regions newly exposed by large viewpoint changes or occlusions.The paper identifies hardware-independent robustness under large camera displacements as future work.Source: §6 · p. 10

AI USE STATEMENT

  • Generative AI tools were used to polish manuscript language, preliminarily screen literature, and discuss method-design feasibility, but not to implement CoRe-VLA or deploy experiments.The authors state that they reviewed AI-assisted material and take responsibility for the work.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendix covers real-robot task success criteria, LIBERO benchmark details, R2C training-pair construction, qualitative execution analysis, a visual ablation, and asynchronous-execution methods.The passage identifies Appendices A through F as covering these materials.Source: APPENDIX · p. 13

A REAL-ROBOT TASK SUCCESS CRITERIA

Sequential real-robot tasks require completing every instructed step in order.

A REAL-ROBOT TASK SUCCESS CRITERIA

  • For sequential tasks, success requires completing all required steps in the instructed order.Table 5 specifies the success criteria for the tasks defined in Table 1.Source: A REAL-ROBOT TASK SUCCESS CRITERIA · p. 13

B INTRODUCTION OF SIMULATION BENCHMARKS

The paper uses LIBERO and LIBERO-Plus as simulation benchmarks for language-conditioned robot manipulation and robustness evaluation.

B INTRODUCTION OF SIMULATION BENCHMARKS

  • LIBERO provides language-conditioned manipulation tasks with workspace and wrist-camera images, while LIBERO-100 contains 100 tasks and LIBERO-Plus adds controlled perturbations across seven dimensions.The LIBERO-Plus Camera subset contains 1,599 instances across four suites, with camera-distance, position, and orientation perturbations.Source: B INTRODUCTION OF SIMULATION BENCHMARKS · p. 13

C CONSTRUCTION OF R2C TRAINING PAIRS

R2C training pairs use existing 2D task-demonstration RGB frames, avoiding depth maps, camera calibration, and additional views. Synthetic image degradation approximates gaps from point-cloud re-rendering rather than creating a geometrically accurate new view.

C CONSTRUCTION OF R2C TRAINING PAIRS

  • Training pairs match degraded versions of existing 2D task-demonstration RGB frames to their original images.The procedure requires no depth maps, camera calibration, or additional views.Source: C CONSTRUCTION OF R2C TRAINING PAIRS · p. 13
  • Severity-controlled projective warps create uncovered regions, while sampled contour gaps and sparse missing-pixel clusters add further degradation.Uncovered and missing pixels are set to black, and warp parameters and masks vary across frames.Source: C CONSTRUCTION OF R2C TRAINING PAIRS · p. 13
  • The degradation procedure approximates visible point-cloud re-rendering gaps, not a geometrically accurate new viewpoint.Source: C CONSTRUCTION OF R2C TRAINING PAIRS · p. 13

D QUALITATIVE EXECUTION EXAMPLES

Qualitative executions show that shifted external views can disrupt task success and cross-view coordination, while restoring a familiar external view supports task order and spatial constraints. Figure 8 compares nominal, shifted, and restored-view executions, with arrows indicating action order and symbols marking outcomes.

D QUALITATIVE EXECUTION EXAMPLES

  • In Task 1, the shifted view causes a pick-and-place failure despite visible relevant objects, while nominal and restored-view executions succeed.Source: D QUALITATIVE EXECUTION EXAMPLES · pp. 13–14, Figure 8 · p. 14
  • In Task 3, the shifted-view robot acts on the locally visible tape measure without first opening the lid; in Task 5, it selects the wrist-visible cup instead of the object closest to the basket.Source: D QUALITATIVE EXECUTION EXAMPLES · pp. 13–14
  • Restoring an external view familiar from training helps align wrist-level cues with intended task order and spatial constraints despite residual image artifacts.The examples indicate that local cues can remain actionable while global context is no longer used effectively for task-state interpretation and target selection.Source: D QUALITATIVE EXECUTION EXAMPLES · p. 14

E VISUAL ABLATION OF R2C RESTORATION

The visual ablation finds that adding perceptual loss yields the closest restoration match in the example. The asynchronous-execution comparison describes ETCA as matching predicted action prefixes to measured joint trajectories before executing a suffix.

E VISUAL ABLATION OF R2C RESTORATION

  • Adding perceptual loss produces the closest match to the original image in the visual example, with clearer objects and a more coherent robot appearance.Color loss alone, or color plus pixel losses, leaves much of the corruption visible.Source: E VISUAL ABLATION OF R2C RESTORATION · p. 15

F COMPARISON WITH ASYNCHRONOUS EXECUTION METHODS

  • ETCA matches candidate prefixes of an already predicted action chunk to measured joint trajectories, then executes the corresponding suffix after quality checks.It is suited to CoRe-VLA’s variable observation-processing latency without changing the frozen VLA or its action-generation procedure.Source: F COMPARISON WITH ASYNCHRONOUS… · p. 15
Loading 2609.37150v1…