PaperlayerView exact revision on arXiv

Source-linked AI summary

Imagination at Inference: Synthesizing In-Hand Views for Robust Visuomotor Policy Inference

Haoran Ding, Anqing Duan, Zezhou Sun, Dezhen Song, Yoshihiko Nakamura

arXiv:2509.15717v1cs.RO

TL;DR

Physical in-hand cameras can be challenging to deploy, although in-hand views provide useful information for manipulation. The paper fine-tunes a pose-conditioned diffusion model to synthesize those views from agent-view observations, and reports stronger performance than external-view-only policies, approaching real-camera policies across simulation and real-robot evaluations.

  • Problem

    Physical in-hand cameras can pose hardware, system-complexity, and cost challenges despite the value of in-hand views for manipulation.

    Source: §I · p. 1

  • Method

    The approach fine-tunes ZeroNVS with LoRA and uses relative camera pose to synthesize in-hand views from external observations for policy inference.

    Source: §I · p. 1, A. Experimental Setup · p. 4

  • Results

    Synthesized-view policies outperform external-view-only policies and closely match real wrist-camera policies across simulation and real-robot evaluations.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results suggest NVS can serve as a lightweight, scalable alternative to additional sensing hardware for vision-conditioned policies under partial observability.

    Source: §V · p. 7

Abstract

from arXiv · show

Visual observations from different viewpoints can significantly influence the performance of visuomotor policies in robotic manipulation. Among these, egocentric (in-hand) views often provide crucial information for precise control. However, in some applications, equipping robots with dedicated in-hand cameras may pose challenges due to hardware constraints, system complexity, and cost. In this work, we propose to endow robots with imaginative perception - enabling them to 'imagine' in-hand observations from agent views at inference time. We achieve this via novel view synthesis (NVS), leveraging a fine-tuned diffusion model conditioned on the relative pose between the agent and in-hand views cameras. Specifically, we apply LoRA-based fine-tuning to adapt a pretrained NVS model (ZeroNVS) to the robotic manipulation domain. We evaluate our approach on both simulation benchmarks (RoboMimic and MimicGen) and real-world experiments using a Unitree Z1 robotic arm for a strawberry picking task. Results show that synthesized in-hand views significantly enhance policy inference, effectively recovering the performance drop caused by the absence of real in-hand cameras. Our method offers a scalable and hardware-light solution for deploying robust visuomotor policies, highlighting the potential of imaginative visual reasoning in embodied agents.

I. INTRODUCTION

In-hand viewpoints can provide complementary, task-relevant visual information, but physical wrist cameras bring deployment challenges. This work synthesizes in-hand views from agent views at inference and reports improved policy performance across simulation and real-robot settings.

I. INTRODUCTION

  • Combining external and in-hand views can improve manipulation by supplying complementary information and reducing ambiguity during close-proximity tasks.Physical wrist cameras can cause mechanical interference, restrict range of motion, add cost and cabling, and be occluded by the gripper or object.Source: §I · p. 1
  • The approach repurposes pose-conditioned novel view synthesis for online inference, generating in-hand views from a single image without dense multiview supervision or explicit geometry.Unlike offline augmentation, synthesized views provide live visual information at runtime.Source: §I · p. 1
  • Synthesized in-hand views outperform external-view-only policies and closely match policies using real wrist-camera input across simulation and real-robot evaluations.The evaluations include RoboMimic, MimicGen, and a Unitree Z1 strawberry-picking task.Source: §I · p. 2
  • The contributions include adapting pretrained ZeroNVS with parameter-efficient LoRA fine-tuning and integrating synthesized views into inference without modifying or retraining the policy.The paper describes the framework as compatible in principle with learning paradigms beyond its imitation-learning experiments.Source: §I · pp. 1–2

II. RELATED WORK

Prior work uses multiple views to improve visuomotor robustness and uses novel view synthesis in robotics. This paper builds on those directions by synthesizing in-hand views from external observations at test time rather than requiring a physical in-hand camera.

A. Visuomotor Policy Learning with Multi-View Observation

  • Multi-view policy research uses viewpoint-invariant features, camera fusion, and view dropout or distillation to address occlusion, ambiguity, and missing views.Reported benefits include better generalization to unseen viewpoints and improved learning speed, especially with egocentric views.Source: A. Visuomotor Policy Learning with… · p. 2
  • This work differs from approaches requiring physical in-hand cameras at deployment by synthesizing egocentric views from external observations at test time.The stated aim is robust deployment without additional hardware complexity.Source: A. Visuomotor Policy Learning with… · p. 2

B. Novel View Synthesis for Robotics Manipulation

Robotics research has used novel view synthesis for perception and policy robustness, including NeRF-based methods and synthesized auxiliary policy views. This work instead uses diffusion-based synthesis to provide egocentric observations during deployment without an in-hand camera.

B. Novel View Synthesis for Robotics Manipulation

  • Robotics applications of novel view synthesis include camera-pose estimation and NeRF-based grasp-surface estimation and sequential grasping in visually challenging or cluttered scenes.The cited work uses NVS to address depth-sensing limitations in real-world manipulation.Source: B. Novel View Synthesis for Robotics… · p. 2
  • Prior policy-learning approaches synthesize off-demonstration or auxiliary views, while this work uses diffusion-based NVS to synthesize egocentric observations during deployment.The stated goal is to bypass in-hand cameras while maintaining strong visuomotor performance.Source: B. Novel View Synthesis for Robotics… · p. 2

III. METHODOLOGY

  • The method uses camera pose-conditioned NVS to generate in-hand observations from agent views and known camera transformations, retaining in-hand perception without additional test-time sensing hardware.The synthesized views complement external inputs for policy deployment.Source: §III · p. 3, Figure 2 · p. 3

A. Problem Definition

The problem is to synthesize a training-time-aligned in-hand observation from an agent-view image and the relative camera transformation, then use that generated view alongside the agent view at policy test time.

A. Problem Definition

  • Training observations pair an agent-view RGB image with an aligned in-hand RGB image, with their spatial relationship given by a calibrated relative camera transformation.The paper names the fixed third-person camera the agent view (AV) and the wrist-mounted egocentric camera the in-hand view (IH).Source: A. Problem Definition · p. 3
  • A pose-conditioned generative model uses the agent-view image and relative transformation to synthesize an in-hand view.The training data are described as triplets, though the supplied passage fragment does not fully enumerate them.Source: A. Problem Definition · p. 3
  • At policy test time, the synthesized in-hand view is paired with the external observation as input to the visuomotor policy, which outputs a control action.This formulation avoids reliance on additional sensing hardware at deployment.Source: A. Problem Definition · p. 3

B. Novel View Synthesis via Pose-Conditioned Diffusion

The paper uses pose-conditioned diffusion to synthesize an in-hand view from an agent-view image and the relative camera pose. The model encodes the image and pose, then denoises and decodes a latent representation into the target view.

B. Novel View Synthesis via Pose-Conditioned Diffusion

  • ZeroNVS synthesizes a target in-hand image from one agent-view image and the relative camera pose, without requiring 3D supervision or multiview inputs.Its single-image, pose-conditioned setup suits deployment with monocular RGB and pose information.Source: B. Novel View Synthesis via… · p. 3
  • The model encodes the agent-view image into a latent, embeds the 6-DoF relative pose with an MLP, and combines them in a U-Net denoiser through cross-attention.After iterative denoising, the latent is decoded into the target in-hand image.Source: B. Novel View Synthesis via… · p. 3
  • ZeroNVS implicitly learns 3D scene structure from 2D pose-labeled data and produces viewpoint-conditioned outputs using only an image and pose.The paper describes it as faster at inference and easier to integrate into robotics pipelines than NeRF-based methods.Source: B. Novel View Synthesis via… · p. 3

C. LoRA Fine-tuning

The paper adapts ZeroNVS with LoRA while freezing its other parameters, then supplies synthesized in-hand observations to policies at test time. It evaluates the approach across six simulation tasks and a real-world strawberry-picking task.

C. LoRA Fine-tuning

  • LoRA fine-tunes ZeroNVS by adding trainable low-rank matrices to U-Net attention projections while keeping all other model parameters frozen.The setup uses rank 128, scaling factor 256, and dropout 0.1; training uses paired RGB views and their relative SE(3) camera transformation.Source: C. LoRA Fine-tuning · p. 4

D. Integration into Policy

  • At test time, the policy receives a synthesized in-hand image in place of the physical wrist-camera observation used during training.Separate image encoders process the agent and in-hand views, whose features are fused for action prediction; the modular design supports multiple policy structures.Source: D. Integration into Policy · p. 4

IV. EXPERIMENT

  • Evaluation covers LIFT, CAN, and SQUARE in RoboMimic, STACK, COFFEE, and MUG CLEANUP in MimicGen, plus real-world strawberry picking with a Unitree Z1.The three compared inputs are agent view only, agent view with ground-truth in-hand view, and agent view with synthesized in-hand view.Source: §IV · p. 4

A. Experimental Setup

The experiments use Flow Matching Policies with visual features and robot proprioception as inputs. The view-synthesis model is LoRA-fine-tuned, and synthesized views are used as policy inputs at test time.

A. Experimental Setup

  • The experiments use Flow Matching Policies trained with external-camera inputs alone or with both external and in-hand views.The paper describes FMP as a flow-matching trajectory predictor, with improved training stability and faster inference than Diffusion Policy.Source: A. Experimental Setup · p. 4
  • Each view is encoded with ResNet-18, and its features are concatenated with end-effector position, orientation, and gripper-state signals before entering a UNet policy.UNet size varies by task according to scene complexity.Source: A. Experimental Setup · p. 4
  • ZeroNVS is fine-tuned with LoRA in the denoiser’s self- and cross-attention projections using paired RGB images and relative camera poses from RoboMimic and MimicGen.The stated LoRA rank is 128 and scaling factor is 256; the model is trained with MSE denoising loss on four NVIDIA A100 GPUs.Source: A. Experimental Setup · p. 4
  • At policy test time, the synthesized in-hand image is generated from the agent view and known camera transformation, with images resized to 84 × 84 for simulation or 180 × 320 for the real-robot experiment.Training uses random cropping at a 90% crop percentage.Source: A. Experimental Setup · p. 4

B. Simulation Results: Robomimic and Mimicgen

Synthesized in-hand views improve success rates over agent-view-only input across all six simulation tasks, with the largest stated benefits on tasks involving occlusion and fine-grained spatial reasoning. The experiments compare synthesized-view policies with agent-only and ground-truth in-hand-view configurations.

B. Simulation Results: Robomimic and Mimicgen

  • Figure 3 qualitatively compares ground-truth and synthesized in-hand views across six tasks: three RoboMimic tasks and three MimicGen tasks.Each image pair places ground truth on the left and the synthesized result on the right.Source: Figure 3 · p. 5
  • The evaluation reuses 50 simulator seeds across the three input configurations and reports means and standard deviations over three repeats per seed.On AV failures, the same trials are rerun with Syn-IH to count additional recoveries.Source: B. Simulation Results: Robomimic and… · p. 5
  • AV+Syn-IH consistently improves success rates over AV across all six tasks, with substantial gap closure toward AV+GT-IH on SQUARE, MUG CLEANUP, and CAN.LIFT is near-saturated across configurations, indicating that the contribution of synthesized views varies by task; Fig. 4 compares success rates by task and input configuration.Source: B. Simulation Results: Robomimic and… · p. 6, Figure 4 · p. 6
  • Synthesizing one in-hand view takes approximately 16.5 s per frame on an NVIDIA RTX 4060 Ti, and the authors note this latency may limit closed-loop responsiveness.The authors identify inference efficiency as important when integrating generative models into control pipelines.Source: B. Simulation Results: Robomimic and… · p. 6

C. Real-World Evaluation: Strawberry Picking

On the Unitree Z1 strawberry-picking task, synthesized in-hand views improved success over agent-view-only input, though they did not reach the ground-truth in-hand-view result. The evaluation used calibrated paired views and varied strawberry positions within the training distribution.

C. Real-World Evaluation: Strawberry Picking

  • The Unitree Z1 task used magnetically attached plastic strawberries, with success defined as clean detachment and a stable hold after a short lift.Failures included missed grasps, slips after contact, or incomplete detachment.Source: C. Real-World Evaluation: Strawberry… · p. 6
  • Ten teleoperated demonstrations provided synchronized agent and in-hand images, robot states, and calibration data for the camera relationship.The wrist camera was calibrated relative to the gripper, and the cross-camera alignment was measured at the initial frame.Source: C. Real-World Evaluation: Strawberry… · pp. 6–7
  • The calibration yielded a time-varying relative pose between views; policy training used paired images resized to 180 × 320 and random crops at a 0.9 scale factor.Policies used one observation frame and a 32-step prediction horizon.Source: C. Real-World Evaluation: Strawberry… · p. 7
  • Testing covered 10 strawberry positions varying in depth, lateral offset, and orientation while remaining within the training distribution.Target poses were approximately matched across policy configurations, with background and lighting held constant.Source: C. Real-World Evaluation: Strawberry… · p. 7
  • 3/10 agent-view-only trials succeeded, compared with 6/10 using synthesized in-hand views and 8/10 using ground-truth in-hand views.Success required clean detachment and a stable hold after lifting.Source: C. Real-World Evaluation: Strawberry… · pp. 6–7

D. Effect of ZeroNVS Lora Finetuning

The qualitative comparison shows that LoRA fine-tuning makes ZeroNVS outputs more closely match task-relevant in-hand views than the pretrained model. The comparison covers CAN and COFFEE, with improvements especially apparent near contact events and object boundaries.

D. Effect of ZeroNVS Lora Finetuning

  • Fine-tuned ZeroNVS produces sharp, geometrically consistent in-hand views, while pretrained outputs are often blurred or severely distorted.The comparison shows ground-truth, pretrained, and fine-tuned views for CAN and COFFEE; the pretrained model often fails to capture scene structure.Source: D. Effect of ZeroNVS Lora Finetuning · p. 7, Figure 6 · p. 7
  • The visual improvements are especially pronounced near contact events and object boundaries, which are critical for precise manipulation.Source: D. Effect of ZeroNVS Lora Finetuning · p. 7
  • Fine-tuning recovers task-relevant geometry in CAN and COFFEE views that pretrained ZeroNVS often fails to represent meaningfully.The figure compares ground-truth views with pretrained and fine-tuned outputs.Source: Figure 6 · p. 7

V. CONCLUSION

The paper concludes that synthesized in-hand views improve policy robustness and task success, narrowing the gap toward policies with physical wrist cameras. The current ZeroNVS implementation has noticeable inference latency and still depends on in-hand views for policy training and model fine-tuning.

V. CONCLUSION

  • Across simulated and real-world manipulation experiments, synthesized in-hand views at inference consistently improve policy robustness and task success, narrowing the gap toward physical wrist-camera policies.The evaluation includes RoboMimic, MimicGen, and Unitree Z1 strawberry picking.Source: §V · p. 7
  • The current ZeroNVS implementation introduces noticeable inference latency that can hinder smooth closed-loop execution.The pipeline also still requires in-hand views during policy training and ZeroNVS fine-tuning.Source: §V · p. 7
Loading 2509.15717v1…