PaperlayerView exact revision on arXiv

Source-linked AI summary

See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

Byungkun Lee, Dongyoon Hwang, Dongjin Kim, Hojoon Lee, Minho Park, Jaegul Choo

arXiv:2607.11498v1cs.ROcs.AI

TL;DR

Vision-language-action models observe scenes in camera coordinates but predict actions in the robot’s coordinate frame, making viewpoint variation challenging. This paper introduces robot-centric pointmaps that preserve image-grid structure while encoding robot-frame 3D geometry, improving both pretrained VLAs and widening gains at unseen camera placements.

  • Problem

    VLAs observe scenes in camera coordinates while predicting robot-frame actions, creating a mismatch that becomes harder to handle across diverse viewpoints.

  • Method

    Robot-centric pointmaps encode each scene point’s 3D coordinates in the robot frame while preserving the dense image grid for integration with pretrained VLAs.

  • Results

    +7.6 and +4.2 points improve π0.5 and SmolVLA on RoboCasa, while the real-robot RGB advantage widens to +11.7 points at an unseen camera placement.

  • Takeaways & Limitations

    When calibration is available, expressing observations in the robot’s action coordinate frame can improve pretrained VLAs across simulation and real-robot experiments.

  • Takeaways & Limitations

    Pointmaps require calibrated camera intrinsics and extrinsics at training and test time, and the study does not cover camera-count or field-of-view changes.

Abstract

from arXiv · show

Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.

1 Introduction

Robot-centric pointmaps address the mismatch between camera-frame observations and robot-frame actions by providing robot-frame 3D geometry in the dense image layout expected by pretrained VLAs. On RoboCasa, the method improves π0.5 and SmolVLA while outperforming representative camera-viewpoint and 3D-aware baselines.

  • VLAs predict robot-frame actions from camera-frame observations, requiring metric 3D reasoning about target objects relative to the robot.
  • Diverse camera viewpoints break the consistent observation-to-action mapping available under a fixed viewpoint, increasing the required viewpoint generalization.
  • Robot-centric pointmaps provide robot-frame 3D geometry while preserving the dense H × W image structure expected by pretrained 2D VLAs.
  • Each pointmap pixel stores the corresponding scene point’s 3D coordinates in the robot frame after lifting RGB-D observations and transforming them from the camera frame.
  • +7.6 points for π0.5 and +4.2 points for SmolVLA on RoboCasa, with improvements over representative camera-viewpoint and 3D-aware baselines.

2 Related Work

Related work addresses viewpoint variation and 3D perception in VLAs through camera-view inputs, viewpoint cues, action reparameterization, synthesized views, or dedicated geometric modules. These approaches generally leave the scene unexpressed in the robot frame or require architectural additions and may not preserve metric 3D at deployment.

  • Viewpoint variation: Camera viewpoint variation degrades VLA performance in both multi-view training datasets and unseen-camera evaluation settings.This work primarily targets training-time viewpoint variation and additionally tests generalization to unseen evaluation cameras.
  • Viewpoint-aware methods: Existing methods supply viewpoint information through explicit camera or 3D cues, camera-frame action reparameterization, or synthesized views.The camera-view input remains unchanged in the first line of methods.
  • Viewpoint-aware methods: Camera-view-based methods do not express the scene in the robot frame where actions are defined.Their visual input stays a camera-view image even when viewpoint information is supplied around it.
  • 3D-aware VLA methods: 3D-aware VLA methods either add dedicated 3D encoders or geometric experts, or forgo metric 3D at deployment.Dedicated modules cannot inherit the VLA’s pretrained visual weights.

3 Method

The method lifts RGB-D observations into robot-centric, end-effector-centered pointmaps so scene geometry and actions share a coordinate origin. A separate pointmap encoder fuses this geometry with RGB tokens while preserving the VLA’s existing visual stream.

  • Robot-centric pointmap: Each RGB-D pixel is lifted into 3D and transformed from the camera frame into the robot base frame using camera calibration and depth.The resulting pointmap retains one 3D coordinate per pixel on the original H × W grid.
  • Robot-centric pointmap: The same physical scene point receives the same robot-frame coordinate across camera viewpoints, even when its image pixel changes.
  • Robot-centric pointmap: End-effector centering expresses scene geometry relative to the current end effector, giving observation and action space a common origin.The paper ablates this centering choice in Section 4.3.
  • Pointmap-RGB fusion: A separate pointmap encoder, initialized from the RGB encoder, maps the pointmap into tokens matching the RGB-token shape.The pointmap tokens are added element-wise to corresponding RGB tokens.
  • Pointmap-RGB fusion: The fused tokens replace the original RGB tokens as VLA visual input without a point cloud encoder, voxel module, or additional token sequence.

4 Design Choices for Robot-Centric 3D Observations in VLAs

Controlled RoboCasa studies show that precomputing robot-frame geometry, preserving it as an image-form pointmap, and centering it on the end effector improves viewpoint robustness. The resulting design is more effective than camera-information inputs, point-cloud representations, and robot-base-centered coordinates.

  • 4.1 Camera information versus robot-centric geometry: 34.7 success rate with RGB + Pointmap exceeds RGB + Plücker + Depth’s 31.6, despite identical depth, intrinsics, and extrinsics.The comparison isolates precomputing robot-centric geometry as the key difference.
  • 4.2 Image-form pointmap versus point cloud: 34.7 success rate from element-wise RGB-pointmap fusion exceeds concatenation’s 30.7 and point-cloud baselines at 24.2.Element-wise addition preserves spatial correspondence between pointmap and RGB tokens while reusing the image-based VLA pathway.
  • 4.3 Coordinate origin: 36.9 success rate with end-effector-centered pointmaps exceeds robot-base-centered pointmaps’ 34.7 under fixed evaluation viewpoints.Under randomized viewpoints, the robot-base-centered variant drops 2.0 points (34.7 →32.7), while the end-effector-centered variant drops 0.3 points (36.9 →36.6).
  • 4.4 Training-time viewpoint variation: 1.8-point loss for RGB + pointmap under increasing training-time viewpoint variation is smaller than RGB’s 9.6-point loss.RGB changes from 34.5% to 24.9%, whereas RGB + pointmap changes from 37.6% to 35.8%.
  • Design selected for subsequent experiments: The selected design precomputes robot-frame geometry, retains it as an image-form pointmap, and re-centers it at the current end-effector position.These choices are used in the remaining experiments.

5 Experiments

Experiments evaluate pointmaps with pretrained VLAs on RoboCasa and a real Franka setup under fixed, seen, and unseen camera viewpoints. Pointmaps improve both tested VLA backbones and provide larger gains when the camera placement is unseen during training.

  • 5.1 Simulation: RoboCasa evaluates π0.5 and SmolVLA on 24 tasks using 50 demonstrations and 50 evaluation episodes per task, with randomized training viewpoints and a fixed evaluation viewpoint.For each backbone, RGB and RGB+Pointmap variants differ only in visual input.
  • 5.1 Simulation: Adding pointmaps raises π0.5’s 24-task average from 55.3 to 62.9 and SmolVLA’s from 37.2 to 41.4, with π0.5 gains across all five task categories.The improvement occurs across backbones differing in scale and action expert.
  • 5.2 Real World: The real-robot study uses a Franka Research 3 with fixed wrist and repositionable external cameras, collecting 15 demonstrations per task across four tasks and three training placements.Evaluation uses seen and unseen external-camera placements to test viewpoint robustness.
  • 5.2 Real World: 78.3% versus 73.3%: at a seen camera placement, pointmaps improve π0.5 by +5.0 over RGB-only π0.5 and outperform DP3 at 63.3.All models are evaluated 15 times under seen and unseen configurations across four manipulation tasks.
  • 5.2 Real World: At an unseen camera placement, RGB-only π0.5 drops from 73.3 to 55.0, while the pointmap advantage grows.Pointmaps preserve scene-point coordinates in the robot frame as the camera is repositioned, avoiding viewpoint-specific observation-to-action mappings.

6 Limitations

The study leaves open how pointmaps should be integrated and how their point-cloud comparison changes with sampling budget, while requiring camera calibration during training and testing.

  • Open questions: The best way to inject pointmaps relative to the action expert and pretraining recipe remains unsettled.The study does not ablate either design choice or their interaction.
  • Open questions: A larger point-cloud sampling budget could narrow the reported comparison gap.The comparison uses a single sampling budget.
  • Calibration requirements: Pointmaps require calibrated camera intrinsics and extrinsics during both training and testing, restricting applicable setups.The passage states that calibration requirements limit where pointmaps can be used.

7 Conclusion

The paper addresses viewpoint-induced frame mismatch by expressing observed scene points directly in the robot frame as image-form pointmaps. Experiments show that this simple representation improves pretrained VLAs in simulation and real-robot settings.

  • 7 Conclusion: Robot-centric pointmaps express each observed scene point directly in the robot frame, avoiding reliance on policies to infer camera-to-robot relationships from RGB, depth, and calibration.The pointmaps are image-form observations centered on the end effector.
  • 7 Conclusion: Robot-frame, image-form pointmaps centered on the end effector provide a simple and effective input for pretrained VLAs.They retain the image-form structure expected by pretrained vision-language-action models.
  • 7 Conclusion: Robot-centric pointmaps improve pretrained VLAs in both simulation and real-robot experiments.The results support expressing observations in the coordinate system where the robot acts when calibration is available.

Supplementary Material · A Details of Training-Time Camera Viewpoint Variation · B Implementation Details and Experimental Setup

The supplementary material defines training-time third-person camera viewpoint variation for RoboCasa and describes how increasing jitter broadens the viewpoints seen by the policy. It also documents the camera configuration and visualizations of these distributions and rendered episodes.

  • A Details of Training-Time Camera Viewpoint Variation: RoboCasa uses two third-person cameras for left and right views plus a wrist camera.The two third-person cameras are varied during training, while the wrist camera remains fixed.
  • A Details of Training-Time Camera Viewpoint Variation: At each demonstration’s start, zero-mean Gaussian noise jitters the third-person cameras’ positions and orientations.This variation changes both camera placement and orientation across demonstrations.
  • A Details of Training-Time Camera Viewpoint Variation: No variation applies no jitter, so every demonstration uses the canonical camera pose.This is the fixed-viewpoint setting.
  • A Details of Training-Time Camera Viewpoint Variation: Low variation uses position and orientation standard deviations of (5cm, 3◦), matching RoboCasa’s default per-demonstration camera randomization.The setting changes both position and orientation with the listed standard deviations.
  • A Details of Training-Time Camera Viewpoint Variation: Figure 9 provides qualitative examples of an episode rendered under each camera-variation level.The examples cover No variation, Low, and High.
  • A Details of Training-Time Camera Viewpoint Variation: High variation doubles the jitter relative to Low, reaching (10cm, 6◦) and widening the policy’s viewpoint range.The position and orientation jitter grow from zero under No variation to (10cm, 6◦) under High.
  • A Details of Training-Time Camera Viewpoint Variation: Figure 8 visualizes per-demonstration third-person camera poses as left-blue and right-pink frustums in the robot-base frame.The end-effector wrist camera is shown in black and is never jittered.

B.1 Backbones and training … B.5 Evaluation metric and protocol

The appendix evaluates robot-centric pointmaps across controlled VLA backbones, RoboCasa simulation, and real-robot settings, using matched training and evaluation protocols. It also details baseline implementations and reports task success from single final checkpoints.

  • B.1 Backbones and training: Controlled RoboCasa studies use a π-style model with a base PaliGemma checkpoint, a scratch-trained action expert, 24 atomic tasks, 50 demonstrations per task, and 50 evaluation episodes.The studies run for 30k training steps and isolate input design rather than relying on the robot-pretrained π checkpoint.
  • B.2 Simulation setup (RoboCasa): RoboCasa covers 24 environments across doors, drawers, coffee, pick-and-place, and turn objects, using two third-person cameras plus one wrist camera with randomized training viewpoints.Unless otherwise stated, evaluation uses a fixed reference viewpoint.
  • B.3 Real-robot setup: Real-robot experiments use an FR3 with wrist D405 and external D435i cameras, collecting 15 demonstrations per task across four tasks and three external-camera placements.This yields 60 demonstrations per configuration and 180 total; stereo depth and one-time hand-eye calibration introduce sensor noise.
  • B.4 Baseline implementation details: RoboCasa comparisons reimplement GeoVLA, PointVLA, KYC, and OC-VLA on π0.5 with shared data and evaluation, while FP3 remains a standalone point-cloud policy.The shared setup uses 24 tasks, 50 demonstrations each, batch size 64, and 50 evaluation episodes; FP3 uses public weights.
  • B.4 Baseline implementation details: Baseline designs span geometric anchor tokens, action-expert point-cloud injection, Plücker-ray conditioning, camera-frame action labels, and RGB-token concatenation of point-cloud features.Robot-frame geometry is also tested with DP3 and point-cloud encoders, while pointmaps preserve pixel correspondence for element-wise RGB fusion.
  • B.5 Evaluation metric and protocol: Performance is reported as task success rate in percent, evaluated over 50 simulation episodes or 15 real-robot rollouts per camera condition from one final checkpoint.Methods are not selected by their best checkpoint.

C Full RoboCasa Results

On fixed-viewpoint RoboCasa evaluations, pointmaps improve or match RGB-only π0.5 on 22 of 24 tasks, with especially large gains on precise spatial-reasoning tasks. π0.5 with pointmaps also achieves the highest average in all five task categories, with the same pattern reported for SmolVLA.

  • Fixed-viewpoint RoboCasa results: Table 8 reports success rates for 24 RoboCasa environments under the fixed evaluation viewpoint.It breaks down the category averages reported in Table 4 into individual tasks.
  • Fixed-viewpoint RoboCasa results: 22 of 24 tasks improve or match RGB-only π0.5 when a pointmap is added.The pointmap improves or matches the RGB-only π0.5 on 22 of the 24 tasks.
  • Fixed-viewpoint RoboCasa results: 46 →76 on CoffeePressButton and 48 →64 on CoffeeServeMug demonstrate large pointmap gains on precise spatial-reasoning tasks.These are representative task-level improvements from the RGB-only π0.5 to π0.5 with pointmap.
  • Fixed-viewpoint RoboCasa results: π0.5 with pointmap achieves the highest average in each of the five task categories, and the same pattern holds for SmolVLA.All methods except FP3 use the π0.5 backbone; PMP denotes the added pointmap.

D RoboCasa Results under Randomized Evaluation Viewpoints

Under independently randomized third-person evaluation viewpoints, pointmaps preserve the fixed-viewpoint conclusions: they improve both pretrained backbones and keep π0.5 above camera-aware and 3D-augmented baselines. The evaluation covers 24 RoboCasa tasks across five categories.

  • Evaluation setup: Evaluation randomizes the third-person viewpoint independently in each episode for policies trained with randomized demonstration viewpoints.These results supplement the main fixed-viewpoint evaluation.
  • Results: 54.2 to 60.3: Adding pointmaps raises the 24-task average for π0.5 under randomized evaluation viewpoints.The results are reported as success rates over 24 tasks.
  • Results: 35.9 to 38.3: Adding pointmaps raises the 24-task average for SmolVLA under randomized evaluation viewpoints.The improvement carries over to the second pretrained backbone.
  • Results: π0.5 with pointmap remains above all camera-aware and 3D-augmented baselines under randomized evaluation viewpoints.Table 9 reports success rates over 24 tasks in five categories; methods use the π0.5 backbone except standalone FP3.
Loading 2607.11498v1…