Source-linked AI summary
FP3: A 3D Foundation Policy for Robotic Manipulation
Rujia Yang, Geng Chen, Chuan Wen, Yang Gao
TL;DR
Robot policy foundation models often rely on 2D observations, despite the importance of 3D geometry for spatial perception and reasoning. FP3 addresses this gap with a diffusion-transformer policy pretrained on 3D point-cloud data and adapted to downstream tasks. With only 80 demonstrations, it achieves over 90% success in novel environments with unseen objects, while its base model still has limited zero-shot performance.
Problem
Most robot foundation models rely on 2D image observations and lack 3D geometric information needed for perceiving and reasoning about spatial relationships.
Method
FP3 is a 1.3B diffusion-transformer policy that integrates point-cloud observations, language, and proprioceptive state, pretrained on large-scale robotic data and fine-tuned for downstream tasks.
Results
With only 80 demonstrations, FP3 learns new tasks with over 90% success in novel environments with unseen objects and significantly outperforms existing robot policies.
Takeaways & Limitations
3D point-cloud pretraining provides FP3 with an effective initialization for data-efficient and generalizable fine-tuning of new robotic manipulation tasks.
Takeaways & Limitations
FP3's base model has limited zero-shot performance, possibly because its DROID pretraining dataset is smaller than other 2D robotics datasets.
Abstract
from arXiv · showhide
Following its success in natural language processing and computer vision, foundation models that are pre-trained on large-scale multi-task datasets have also shown great potential in robotics. However, most existing robot foundation models rely solely on 2D image observations, ignoring 3D geometric information, which is essential for robots to perceive and reason about the 3D world. In this paper, we introduce FP3, a first large-scale 3D foundation policy model for robotic manipulation. FP3 builds on a scalable diffusion transformer architecture and is pre-trained on 60k trajectories with point cloud observations. With the model design and diverse pre-training data, FP3 can be efficiently fine-tuned for downstream tasks while exhibiting strong generalization capabilities. Experiments on real robots demonstrate that with only 80 demonstrations, FP3 is able to learn a new task with over 90% success rates in novel environments with unseen objects, significantly surpassing existing robot foundation models.
I. INTRODUCTION
FP3 addresses weak generalization and high demonstration demands in robot policies by introducing a large-scale foundation policy that uses 3D point clouds alongside language and visuomotor inputs. Its pretraining and architecture support efficient adaptation to new tasks and improved performance across novel objects, environments, and conditions.
- Motivation: Existing learned policies often generalize poorly to unseen scenarios, objects, and distractors, while single- or few-task methods usually require about 200 expert episodes for a new task.
- Motivation: 2D-only policy foundation models lack 3D observations, although geometric information is important for perceiving environments and reasoning about spatial relationships.Prior work indicates that 3D representations can improve sample efficiency and generalizability, with point clouds reported as particularly effective.
- Contribution: FP3 introduces the first 3D point cloud-based language-visuomotor policy foundation model for robotic manipulation, combining point-cloud, language, and proprioceptive inputs with diffusion-based action prediction.The model uses a pre-trained Uni3D encoder and an encoder-decoder Diffusion Transformer.
- Results: With only 80 post-training trajectories, FP3 achieves about 90% success on novel objects and environments, while DP3 and OpenVLA almost completely fail in this setting.
- Results: FP3 reports average performance improvements of around 60% in-domain and 80% in-the-wild over strong baselines with two-hour, single-GPU fine-tuning.
B. Robotic Manipulation with 3D representations
FP3 extends diffusion-policy foundation models by using large-scale 3D point-cloud representations rather than only 2D images. Its multimodal encoder-decoder predicts future action chunks from geometric observations, language, and robot state.
- Robotic Manipulation with 3D representations: FP3 differs from prior 3D manipulation methods by scaling a point-cloud encoder to 300M parameters and the full foundation model to 1.3B parameters on large multi-task data.It is designed for efficient and generalizable fine-tuning to new tasks.
- FP3 model: FP3 takes point-cloud observations, language, and proprioceptive state as input and predicts chunks of future actions with a diffusion-based policy.The observation includes camera point clouds, language, and robot state, while the action output is a temporal chunk.
- FP3 model: FP3 uses point clouds because they contain rich semantic and geometric information and are reported as more suitable for policy learning than other 3D representations.Language is encoded with a frozen CLIP model, while low-dimensional inputs and noise levels use two-layer MLPs.
- FP3 model: A Transformer encoder fuses multimodal embeddings, while a Transformer decoder denoises noisy action chunks using adaptive Layer-Norm conditioning.The denoiser uses temporal causal masking, and DDIM accelerates inference.
B. Pre-training
FP3 is pretrained on DROID, a large robotic dataset with depth observations, to provide the multi-task 3D data needed for a foundation policy. The preprocessing converts synchronized RGB-depth views into cropped, downsampled point clouds, while training fine-tunes the point-cloud encoder.
- Pre-training: FP3 uses 60k DROID demonstrations for pretraining from a dataset containing 86 tasks and 76k demonstrations with depth observations.
- Pre-training: Each of two camera views is converted from RGB and depth into a point cloud, transformed into a shared world frame, cropped around the operated object, and downsampled.
- Pre-training: The Uni3D ViT encoder is fine-tuned during pretraining rather than frozen, and random point dropping is used for augmentation with a dropout rate sampled from 0 to 0.8.
- Pre-training: The base model is pretrained for 3M steps with batch size 128 on eight NVIDIA A800 GPUs, taking about 48 hours.Fine-tuning the same model on one A800 takes approximately two hours.
- Pre-training: Two stacked frames, including one step of observation history, are used to compensate for missing dynamic information under partial observation.
C. Post-training
FP3 uses post-training to adapt its pretrained base model to downstream tasks with a small, diverse set of high-quality demonstrations. The collection strategy varies environments and objects rather than concentrating demonstrations in one scenario.
- Post-training: Post-training adapts the pretrained base model to specific downstream tasks using a small amount of high-quality data.This follows a pretraining-and-post-training pattern used in modern large language models.
- Post-training: Each downstream task uses 80 teleoperation demonstrations collected as 10 demonstrations in each of 8 environments with 8 unique objects.The strategy emphasizes environmental and object diversity rather than merely increasing demonstrations in one scenario.
IV. EXPERIMENTS
Experiments evaluate FP3 and three baselines on four real-robot manipulation tasks, measuring success rate across task types and environmental conditions. The setup compares small 2D, small 3D, and large 2D foundation policies against FP3.
- Evaluation questions: The experiments ask whether FP3 learns new tasks efficiently, generalizes to unseen objects and scenes, withstands perturbations, and follows language instructions.
- Tasks: Four downstream tasks—Fold Towel, Clean Table, Stand up Cup, and Pour Water—form the real-robot evaluation suite.
- Baselines and metrics: Success rate is averaged over 20 evaluation trials to compare FP3 with three baselines.The baselines are Diffusion Policy, DP3, and OpenVLA.
- Baselines and metrics: The baselines represent a small 2D policy, a small 3D point-cloud policy, and a large 2D vision-language-action foundation policy.
B. Efficient and generalizable fine-tuning for new tasks
With 80 demonstrations, FP3 is evaluated for efficient task adaptation and generalization from seen-object environments to unseen objects and novel environments. It substantially outperforms the baselines in both settings, with pre-training and 3D observations supporting its robustness.
- In-domain Performance: Over 90% success rates are achieved by FP3 across in-domain tasks after fine-tuning on only 80 demonstrations.The 80 demonstrations comprise 10 demonstrations per environment-object pair across 8 pairs.
- In-domain Performance: Baseline failures commonly involve imprecise grasping or pouring, whereas FP3 predicts smoother and more precise actions for complex tasks.
- In-the-Wild Performance: More than 80% average success is achieved by FP3 in novel environments with unseen objects, while baseline policies often approach zero performance.The comparison includes FP3-Scratch and baselines without pre-training.
- In-the-Wild Performance: FP3’s point-cloud observations capture geometric information needed for cross-domain generalization, while large-scale pre-training covers diverse scenes and objects.
- Failure Analysis of Baselines: OpenVLA performs poorly because it often gets stuck and interacts inaccurately, potentially due to missing action chunking, observation history, and restricted third-person-view coverage.
- Failure Analysis of Baselines: After an initial failure, only OpenVLA and FP3 commonly produce reasonable subsequent attempts; policies without pre-training often become stuck.The authors relate this behavior to limited fine-tuning data and out-of-distribution states after failure.
C. More experiments on generalization
Additional Clean Table experiments test robustness to altered object appearances, backgrounds, lighting, camera views, and distractors. FP3 generally maintains the strongest performance, although distractors remain challenging for every method.
- Experimental scope: The Clean Table task provides a common testbed for evaluating FP3 across different environments and robot setups.
- Generalization to different object appearances, backgrounds, and lighting conditions: FP3 surpasses the baselines under changes to object appearance, background texture, and lighting, benefiting from pre-training and 3D geometry understanding.DP3 remains stable for lighting and object-color changes but is limited by its in-domain performance.
- Generalization to new camera views: With camera views shifted approximately 30 degrees, FP3 maintains high performance because point clouds are converted to consistent coordinates when the camera is properly positioned.
- Generalization to distractors: FP3 remains the highest-performing and most stable method with random distractors, although all methods sometimes attempt to grasp interfering objects.
D. Instruction following
FP3 follows language instructions in a shared starting context and generalizes across environments, views, distractors, objects, and backgrounds. Ablations associate its strongest performance with 3D point clouds, larger models, and larger-scale pre-training.
- D. Instruction following: FP3 executes different tasks from different language instructions in the same initial state, whereas baselines fail or are distracted by objects from other tasks.
- E. Ablations: FP3 generalizes to unseen environments and new objects, while Diffusion Policy often misses the target object or position.
- E. Ablations: FP3 achieves outstanding performance across generalization settings spanning lighting, camera view, distractors, objects, and backgrounds.
- E. Ablations: FP3-Scratch performs poorly both in-domain and in-the-wild, indicating the importance of pre-training.
- E. Ablations: FP3-Base-Image matches FP3-Base in-domain but drops substantially in-the-wild, highlighting the effectiveness of 3D point-cloud representation.
- E. Ablations: FP3-Base and FP3-Base-30k perform similarly, and both trail the final FP3 model.
- E. Ablations: The ablation table reports best performance with 3D point-cloud input, a larger model, and larger-scale pre-training data.
V. LIMITATIONS
FP3 supports efficient and generalizable fine-tuning, but its base model has limited zero-shot performance and several representation limitations. The authors identify larger 3D datasets and stronger language and vision integration as future directions.
- The base model has limited zero-shot performance, possibly because the DROID pre-training dataset is smaller than large 2D robotics datasets such as OXE.
- Simple CLIP language conditioning is insufficient for complicated and dynamic information, motivating integration with VLMs.
- FP3 does not use robust pre-trained 2D vision encoders, leaving 2D–3D feature fusion as future work.
- FP3 learns new tasks with over 90% success rates in novel environments with unseen objects, significantly outperforming existing robot policies.
APPENDIX
The appendix evaluates policies across in-domain and unseen environments using post-training data collected in eight environments. Four environments are reserved for each evaluation condition.
- For all four tasks, policies are trained in 8 environments and evaluated in 4 in-domain and 4 unseen environments.
B. Object Visualization
The appendix documents the task objects, hardware, and training configurations used for evaluation. Each task includes eight training objects and four unseen objects, with varied object properties and randomized placements.
- B. Object Visualization: Each task uses 8 training objects and 4 unseen objects, which are visualized for evaluation.
- B. Object Visualization: Training and fine-tuning hyperparameters are listed in Tables III and IV, with all models trained on 8 NVIDIA A800 GPUs.
- B. Object Visualization: Fold Towel varies towel center position, orientation within ±30 degrees, and towel color, texture, and material.
- B. Object Visualization: Clean Table randomizes spitball and trash-bin positions while varying bin color, texture, and size.
- B. Object Visualization: Stand up Cup randomizes cup placement and orientation within a 180-degree range while requiring insertion, grasping, lifting, and upright placement.
- B. Object Visualization: The hardware setup is documented separately for the experiments.