PaperlayerView exact revision on arXiv

Source-linked AI summary

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields

Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, Xiaolong Wang

arXiv:2308.16891v3cs.ROcs.CVcs.LG

TL;DR

Robotic manipulation requires agents that generalize across diverse tasks from visual observations despite limited demonstrations and complex 3D scenes. GNFactor jointly learns a generalizable neural feature field and a language-conditioned Perceiver policy over shared volumetric features, improving results across simulated and real-robot evaluations. Its main practical limitation is the need for multiple views during GNF training.

  • Problem

    Limited-demonstration multi-task manipulation requires robots to generalize across objects, tasks, and environments while understanding 3D scene structure and semantics.

  • Method

    GNFactor jointly trains a Generalizable Neural Feature Field and a Perceiver Transformer over a shared 3D volumetric representation, using foundation-model features for semantic supervision and language instructions for action prediction.

  • Results

    GNFactor achieves strong results across 10 RLBench tasks and 3 real-robot tasks, outperforming PerAct by 1.55x and 1.57x in the reported simulation evaluations.

  • Takeaways & Limitations

    The results support GNFactor’s potential for real-world multi-task manipulation and generalization across simulated and real scenes.

  • Takeaways & Limitations

    GNFactor requires multiple views for GNF training; the current system uses three fixed cameras, limiting scalability in real-world data collection.

Abstract

from arXiv · show

It is a long-standing problem in robotics to develop agents capable of executing diverse manipulation tasks from visual observations in unstructured real-world environments. To achieve this goal, the robot needs to have a comprehensive understanding of the 3D structure and semantics of the scene. In this work, we present $\textbf{GNFactor}$, a visual behavior cloning agent for multi-task robotic manipulation with $\textbf{G}$eneralizable $\textbf{N}$eural feature $\textbf{F}$ields. GNFactor jointly optimizes a generalizable neural field (GNF) as a reconstruction module and a Perceiver Transformer as a decision-making module, leveraging a shared deep 3D voxel representation. To incorporate semantics in 3D, the reconstruction module utilizes a vision-language foundation model ($\textit{e.g.}$, Stable Diffusion) to distill rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3 real robot tasks and perform detailed ablations on 10 RLBench tasks with a limited number of demonstrations. We observe a substantial improvement of GNFactor over current state-of-the-art methods in seen and unseen tasks, demonstrating the strong generalization ability of GNFactor. Our project website is https://yanjieze.com/GNFactor/ .

1 Introduction

GNFactor addresses limited-demonstration multi-task manipulation by combining 3D scene structure with semantic information in a language-conditioned policy. It uses generalizable neural feature fields and achieves improved performance across real-robot and simulated evaluations.

  • Limited demonstrations make it difficult for robots to generalize across unseen objects, tasks, and environments while understanding 3D scene structure and semantics.
  • 2D visual representations improve sample efficiency but are mainly applied to simpler manipulation tasks, motivating 3D representations for geometric reasoning and occlusion handling.
  • GNFactor distills foundation-model features into Generalizable Neural Feature Fields and encodes a single RGB-D observation into a 3D semantic volumetric feature.
  • A Perceiver Transformer combines the volumetric scene representation with language instructions to embed tasks and predict manipulation actions.
  • GNFactor outperforms PerAct by 1.55x on 10 RLBench tasks and 1.57x on 6 designed generalization tasks, alongside improvements in three real-robot tasks.

2 Related Work

Related work advances multi-task manipulation, 3D representations, and neural fields, but the cited approaches differ in data requirements, representation choices, and generalization properties. The section situates this work among task-agnostic training, voxel-based policies, and neural-field scene representations evaluated in simulation and on a real robot.

  • Multi-Task Robotic Manipulation: Recent multi-task manipulation methods improve complex-task execution and generalization, often using extensive interaction data for training.RT-1 is cited as evidence for the benefits of task-agnostic training in real-world robotic tasks.
  • Experimental Context: The evaluation context includes 10 RLBench simulation tasks, sampled views for GNF, and a real robot setup.These settings are depicted as separate components of the experimental environment and hardware configuration.
  • Multi-Task Robotic Manipulation: Perceiver Transformer policies encode language goals and voxel observations for action prediction in real robot experiments.This work retains the PerAct action-prediction framework while targeting better generalization under limited data.
  • 3D Representations for Reinforcement/Imitation Learning: 3D representations are increasingly used in reinforcement and imitation learning because they support more complex manipulation tasks than 2D representations alone.Prior work includes a deep voxel-based 3D autoencoder that improves sample efficiency relative to 2D representation learning.
  • Neural Radiance Fields: Neural fields support novel-view synthesis and scene representation learning, but NeRF requires per-scene optimization and is difficult to generalize.The section describes subsequent methods as attempts to address limitations of standard neural-field approaches.

3 Method

GNFactor learns a shared deep 3D volumetric representation for reconstruction and language-conditioned robot action prediction. Its jointly trained neural field and Perceiver modules combine RGB, vision-language, geometric, proprioceptive, and task information for multi-task manipulation.

  • 3 Method: GNFactor shares a deep volumetric representation between a Generalizable Neural Feature Field for reconstruction and a 3D policy module for manipulation.The reconstruction module learns RGB and vision-language feature rendering, while the policy module predicts robot actions.
  • 3 Method: RGB-D observations are encoded into a 100^3 voxel grid whose 128-dimensional features form the volumetric scene representation.The voxel encoder produces v ∈ R100^3×128, and additional views support volumetric reconstruction.
  • 3.1 Problem Definition: Behavior cloning is reformulated as discretized keyframe prediction, with translation, rotation, gripper openness, and collision avoidance predicted for subsequent keyframes.The action representation uses discretized rotation bins and delegates intermediate motion to simulation or real-world motion planners.
  • 3.2 Learning Volumetric Representations with Generalizable Neural Feature Fields: GNF samples trilinearly interpolated 3D features and maps points, view directions, and features to density, RGB color, and 512-dimensional vision-language embeddings.Ray rendering integrates these quantities over camera rays to reconstruct images and foundation-model features.
  • 3.3 Action Prediction with Volumetric Representations: A Perceiver Transformer combines the 3D volume, robot proprioception, and CLIP language features before producing voxel-aligned action predictions.The transformed voxel representation is shared across action heads for translation, rotation, gripper openness, and collision avoidance.
  • 3.3 Action Prediction with Volumetric Representations: GNFactor jointly optimizes action and reconstruction objectives, balancing them with λ_recon without pre-training.The shared representation is trained simultaneously for GNF reconstruction and 3D policy prediction.

4 Experiments

GNFactor is evaluated in simulation and real-robot settings for multi-task performance, unseen-task generalization, and component contributions. It consistently outperforms PerAct, including under challenging task, scene, and camera conditions.

  • Experiment Setup: GNFactor outperforms PerAct by an average of 1.55x on 10 RLBench multi-task tests and 1.57x on unseen-task generalization.The evaluations use 25 episodes per checkpoint across 10 tasks and 20 episodes per unseen task, respectively, across 3 seeds.
  • Simulation Results: GNFactor achieves 28.0% success on sweep to dustpan versus 0% for PerAct, while open drawer reaches 76.0% versus 54.7%.The long-horizon sweep task requires picking up a broom and sweeping dust into a dustpan; open drawer is comparatively simpler.
  • Simulation Results: GNFactor generalizes better than PerAct on 5 of 6 altered-task conditions, although both methods struggle with the smaller-block slide variation.The tested changes include distractors, object size, and position.
  • Ablations: Ablations identify GNF reconstruction, RGB reconstruction loss, Stable Diffusion features, depth-guided sampling, and balanced rendering/action losses as important components.The RGB loss promotes 3D-consistent features, while depth-guided sampling helps the representation focus on object surfaces.
  • Real Robot Experiments: On real robots, GNFactor outperforms PerAct across almost all tasks, including teapot manipulation where PerAct records zero success across two kitchens.The real-robot evaluation covers three tasks across two kitchens using limited demonstrations.
  • Real Robot Experiments: GNFactor maintains performance when moving to the smaller, more challenging second kitchen, unlike PerAct, whose performance drops more substantially.The second kitchen requires greater manipulation accuracy because of its smaller size.
  • Real Robot Experiments: Grad-CAM visualizations show that the policy attends clearly to target objects despite receiving a Q-value training signal for only a single voxel.The visualization uses gradients and the 3D feature map after the Perceiver Transformer.

5 Conclusion and Limitations

GNFactor uses a shared 3D volumetric representation for visual behavior cloning and achieves strong results across simulated and real-world tasks. Its main limitation is the need for multiple views during GNF training, which complicates real-world scaling.

  • GNFactor uses a Generalizable Neural Feature Field for 3D volumetric representation and shares it with the action prediction module.Stable Diffusion features supervise GNF training alongside RGB features.
  • GNFactor achieves strong results across 10 RLBench tasks and 3 real robot tasks.
  • Multiple views are required for GNF training, creating a challenge for scaling GNFactor in real-world settings.The current system uses three fixed cameras; random camera collection would require camera-pose estimation.

A Visualizations

The visualizations examine GNFactor’s view synthesis and policy attention. Action-loss training produces blurrier renderings, while Grad-CAM visualizations show attention on target objects despite voxel-level supervision.

  • Action-loss training produces relatively blurred view-synthesis visualizations compared with rendering without the action loss.Inference uses one feedforward pass for feature extraction rather than per-step rendering optimization, contributing to noisy rendering.
  • GNFactor synthesizes both RGB images and feature representations in additional novel-view visualizations.The action loss is removed in these visualizations to improve rendering quality.
  • Grad-CAM visualizations show that the policy attends clearly to target objects.The training supervision is the Q-value for a single voxel.

B Task Descriptions

The evaluation includes varied language-conditioned RLBench tasks, additional scene-based generalization tasks, and three real-robot manipulation tasks across kitchens.

  • Simulated tasks: The RLBench evaluation uses 10 language-conditioned tasks with at least two variations each.The variations change colors, sizes, counts, placements, and object categories, totaling 166 variations.
  • Generalization tasks in simulation: Six additional simulation tasks modify the original training environments to test generalization.These tasks are summarized in Table 6.
  • Real robot tasks: The real-robot evaluation includes door, faucet, and teapot tasks, plus versions with distracting objects.The tasks require precise coordination, intricate motor control, and locating and moving a randomly placed teapot.

C Implementation Details

GNFactor combines a voxel encoder, a Generalizable Neural Field, and a Perceiver Transformer to build and use a shared 3D representation for robotic action prediction.

  • Voxel encoder: A lightweight 3D UNet encodes a 100^3 × 10 input voxel into a 100^3 × 128 deep volumetric representation.The input contains RGB features, coordinates, indices, and occupancy.
  • Voxel encoder: The voxel encoder uses convolution, batch normalization, LeakyReLU, and skip connections across progressively changing spatial resolutions.The pseudo-code shows resolutions from 100^3 through 13^3 and back to 100^3.
  • Generalizable Neural Field: GNF uses NeRF-style positional encoding and combines coordinates, view directions, and voxel features as input.The concatenated input has 170 dimensions, and the network outputs RGB, density, and Diffusion features.
  • Generalizable Neural Field: The GNF network projects inputs to 512 dimensions, processes them with five ResnetFCBlocks, and projects them to 516 output dimensions.The output contains RGB, density, and Diffusion features.
  • Percevier Transformer: A Perceiver Transformer processes 3D volume, language, and proprioception, then predicts voxelized translation Q-values and other robot actions.It uses a small set of latents for efficient long-sequence processing and aggregates 3D features for openness, collision avoidance, and rotation.

D Demonstration Collection for Real Robot Tasks

Real-robot demonstrations are collected by mapping tracked human hand movements onto an xArm robot, alongside RGB-D observations recorded with a RealSense camera.

  • An HTC VIVE controller and basestation track human hand 6-DOF poses for demonstration collection.The triad-openvr package maps these operations onto the xArm robot in a real kitchen.
  • The system records xArm poses and 640 × 480 RGB-D observations using pyrealsense2.Images are downscaled to 80 × 60 for neural rendering.
  • The same input voxel shape is used in simulation and real-world experiments despite different image sizes.This keeps the algorithm consistent across both settings.

E Detailed Data

Detailed evaluations report checkpoint-selection results, successful-trajectory counts, and a fairness-adjusted comparison in which GNFactor uses one view while PerAct uses four.

  • GNFactor surpasses PerAct by a large margin when evaluated using the best single checkpoint on RLBench.This setting selects the saved checkpoint with the highest success rate, although it is not used for the main results for fairness.
  • Table 8 reports successful-trajectory counts for novel-task generalization over 20 episodes across 3 seeds using the final checkpoint.
  • GNFactor still outperforms PerAct with more views even though GNFactor takes a single view as input.PerAct is enhanced with four predefined camera views designed to cover the entire scene.

G Hyperparameters

GNFactor’s hyperparameters specify ray sampling and loss weights, while the stronger PerAct comparison uses four predefined camera views covering the scene.

  • Hyperparameters: GNF training uses a ray batch size of 512, with λfeat = 0.01 and λrecon = 0.01.The stated weighting maintains major focus on action prediction.
  • Hyperparameters: Real-world experiments set reconstruction-loss weight to 1.0 and action-loss weight to 0.1.The passage attributes this choice to observations about reducing action-loss weight and increasing reconstruction emphasis.
  • PerAct Baseline: PerAct receives four predefined camera views whose coverage spans the entire scene.These views are used to strengthen the baseline comparison.
  • Ray Sampling: The coarse network samples 64 points per ray, while the fine network samples 32 depth-guided and 32 uniformly sampled points.
Loading 2308.16891v3…