Source-linked AI summary
Transferring Foundation Models for Generalizable Robotic Manipulation
Jiange Yang, Wenhui Tan, Chuhao Jin, Keling Yao, Bei Liu, Jianlong Fu, Ruihua Song, Gangshan Wu, Limin Wang
TL;DR
Robotic manipulation agents trained on collected data can struggle to generalize to unseen objects and environments, while gathering such data is costly. This paper conditions a two-stream policy on language-reasoning masks generated using foundation models. The authors report improved sample efficiency and generalization, especially to unseen objects and backgrounds.
Problem
Collecting large-scale robot data is costly, and limited data diversity constrains generalization to unseen objects and environments.
Source: §1 · p. 1Method
The method uses foundation models to turn language instructions into object masks, then conditions an end-to-end two-stream policy on masks, images, and proprioception.
Source: §1 · pp. 1–2Results
The authors report improved sample efficiency and generalization, especially for unseen objects and backgrounds.
Source: §6 · p. 8Takeaways & Limitations
The reported experiments support using language-reasoning masks and a two-stream policy for generalizable robotic manipulation within the evaluated setting.
Source: §6 · p. 8The paper leaves scaling individual components and improving coordination among system components as future exploration.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Improving the generalization capabilities of general-purpose robotic manipulation agents in the real world has long been a significant challenge. Existing approaches often rely on collecting large-scale robotic data which is costly and time-consuming, such as the RT-1 dataset. However, due to insufficient diversity of data, these approaches typically suffer from limiting their capability in open-domain scenarios with new objects and diverse environments. In this paper, we propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models, to condition robot manipulation tasks. By integrating the mask modality, which incorporates semantic, geometric, and temporal correlation priors derived from vision foundation models, into the end-to-end policy model, our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning, including new object instances, semantic categories, and unseen backgrounds. We first introduce a series of foundation models to ground natural language demands across multiple tasks. Secondly, we develop a two-stream 2D policy model based on imitation learning, which processes raw images and object masks to predict robot actions with a local-global perception manner. Extensive realworld experiments conducted on a Franka Emika robot arm demonstrate the effectiveness of our proposed paradigm and policy architecture. Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
1. Introduction
The paper targets costly, data-limited robotic learning that struggles with unseen objects and environments, proposing language-reasoning masks as a condition for sample-efficient generalization. Real-world experiments report generalization to unseen objects, backgrounds, and distractors.
1. Introduction
- The motivating gap is that collecting large real-world robot datasets is costly, while limited data diversity constrains generalization to unseen objects and environments.The cited RT-1 dataset comprises approximately 130,000 demonstrations collected over 17 months with 13 robots.Source: §1 · p. 1
- The approach conditions an end-to-end policy on language-reasoning masks that carry semantic, geometric, and temporal priors from internet-scale vision foundation models.The paper presents masks as a more specified and unambiguous condition representation than unprocessed human language.Source: §1 · pp. 1–2
- The policy follows a closed-loop 2D pipeline that maps raw images to continuous actions without requiring depth calibration or completely accurate masks.The paper contrasts this with perception-and-motion-planning pipelines that require accurate masks and depth to construct point clouds.Source: §1 · p. 2
- The system grounds instructions with GPT-4, locates and segments target objects, then uses a two-stream policy to predict actions from images, masks, and proprioception.The policy uses a deeper branch for global RGB information and a shallower branch for local object-related RGB-mask features.Source: §1 · p. 2
- Experiments report improved performance and generalization to unseen objects, new backgrounds, and more distractors, with transfer to additional manipulation skills.The additional skills include drawer opening, placing objects inside or on top of another object, folding, and stacking.Source: §1 · p. 2
2. Related Work
Related work spans language-based task planning, vision-based robot learning, and language-conditioned control. The paper distinguishes its approach through object masks and a closed-loop policy that maps observations to actions.
Pre-trained Foundation Models for Robotics
- Prior language-planning work uses off-the-shelf LLMs [21] [22] [23] [24] [33] [34] [36] [64] [69] or fine-tunes them with in-domain data [3] [12] [27] [50] [53] [56].Other examples include masked-autoencoder visual-language encoding in InstructRL [35], CLIP feedback in PAFF [14], and object-centric representations in MOO [65].Source: §2 · p. 2
- Unlike approaches that use a pixel-centered representation or build point clouds for separate grasp and motion planners [22] [33] [64], this method conditions an end-to-end closed-loop policy on detected, tracked, and segmented object masks.The paper states that its policy does not require depth, camera calibration, or completely accurate masks.Source: §2 · p. 3
Vision-based Robot Learning
- Vision-based work explores pretrained visual models, temporal cues, dense rewards, human-video skill priors, and point-cloud grasping [29] [43] [44] [46] [51] [54] [58] [75] [81].The paper describes its work as orthogonal to these contributions and notes that direct comparisons with point-cloud grasping methods are usually not performed because the problem formulations differ.Source: §2 · p. 3
Language-Conditioned Robotics Control
- Language-conditioned control includes history encoding [18], RGB-D voxel representations [63], and scaling real-world data collection [26] [4].The cited discussion says limited quantity and diversity of robot data constrain generalization, especially to unseen object categories [74].Source: §2 · p. 3
3. Approach
The approach maps language instructions and robot observations to continuous actions through a foundation-model mask generator and a two-stream policy. It uses GPT-4 reasoning, object detection and tracking, SAM segmentation, and behavior-cloned policy learning.
3.1. Problem formulation
- The system maps proprioception, visual observations, and language instructions to continuous robot actions using a mask generator followed by a two-stream policy.The approach is illustrated primarily with pick-and-place, while the same manner is stated to apply to other skills.Source: §3.1 · p. 3, §3 · p. 3
3.2. Mask Generation with Foundation Models
- GPT-4 reasons target objects from instructions; open-vocabulary detection and tracking produce bounding boxes, which prompt SAM to generate target-object masks.Detection supplies the initial box, then tracking supplies subsequent boxes to handle occlusion, disturbance, and distractors.Source: §3.2 · pp. 3–4
3.3. Two-stream Policy Model Architecture
- The TPM combines a shallow RGB-mask branch for local object features with a deeper RGB branch for global scene relations, then fuses view features and proprioception to predict actions.The model uses spatial attention and multi-view feature fusion; behavior cloning trains a deterministic policy with cross-entropy for gripper state and mean squared error for end-effector movement.Source: §3.3 · p. 5
4. Experiment
The experiments evaluate imitation learning on a 1,000-episode, multi-view pick-and-place dataset and test generalization across objects, backgrounds, and distractors. Results report robustness to unseen objects and more distractors, while additional comparisons examine model components and transfer to other manipulation skills.
4.1. Dataset
- The dataset contains 1,000 pick-and-place trajectories across 40 objects, five typical shapes, three tabletop backgrounds, and 0–2 distractors per demonstration.Instructions are annotated, and GroundingDINO and SAM produce language-conditioned masks; images are captured from frontal and lateral perspectives.Source: §4.1 · p. 5
4.2. Implementation Details
- The implementation trains TPM for 500,000 iterations on 224×224 images, using Adam, a batch size of 24, and an NVIDIA RTX A6000 GPU.The learning rate decays from 5·10^-5 to 5·10^-6 with a cosine schedule; no data augmentation is used.Source: §4.2 · p. 5
4.3. Experimental Setup
- Evaluation holds out 20 objects from the 40 training objects and tests seen or unseen objects, a complex-textured new background, and scenes with 3–6 distractors.Pick-and-place success requires grasping A and placing it within 2 inches of B; experiments use a Franka arm with fixed frontal and lateral cameras.Source: §4.3 · p. 5, Figure 3 · p. 6
4.4. Experiment results
- The results report robustness to unseen objects and more distractors, a slight performance decrease on a new background, and performance exceeding RT-1's limit.The paper attributes the reported performance to including language-reasoning segmentation masks from foundation models for action prediction.Source: §4.4 · p. 6
- Replacing segmentation masks with bounding boxes reduces the reported result from 75.625 to 36.25, while frame-by-frame detection lowers success and inference speed relative to detection followed by tracking.The paper reports that masks provide geometry and shape priors and greater robustness to complex textures and distractors; tracking helps with occlusion and disturbances.Source: §4.4 · p. 7
- Multi-view fusion shows a 65.0 versus 20.0 result in the new-background setting compared with a single front view.The authors suggest multi-view disparity helps estimate depth and may explain the difference.Source: §4.4 · p. 7
- The authors describe a separate RGB branch as beneficial and hypothesize it could improve depth generalization; they report a 50.0 versus 65.0 disparity between two models when the background changes.The supplied passage does not identify which model corresponds to each value.Source: §4.4 · p. 7
4.5. Extension to other skills
- Transfer to a low-cost dual-arm robot uses 50 new demonstrations for folding and stacking, with success rates of 85% and 65% for seen and unseen folding, and 50% and 30% for seen and unseen stacking.The dual-arm setup does not require additional motion-planning time; drawer opening achieves approximately 50% success over 40 trials after co-fine-tuning on 100 demonstrations.Source: §4.5 · pp. 7–8
5. Discussion and Limitations
The discussion identifies prompt design, contact-rich skills, component coordination, object detection, and inference latency as areas for further work. It also describes measures that may improve the system’s performance and deployment efficiency.
5. Discussion and Limitations
- Contact-rich skills still require extensive demonstrations, and the system requires manually designed complex prompt templates.The authors leave improving prompt generation to future work, such as better code generation.Source: §5 · p. 8
- Improving component performance through scaling laws and coordinating the system’s components remain directions for future exploration.The authors frame the multi-model paradigm as a way to combine internet-scale models with imitation learning’s potential to capture multimodal action distributions.Source: §5 · p. 8
- The authors identify the connection between language reasoning and detection as the system’s main performance bottleneck because current detectors lack visual concepts.They suggest adding shape and color descriptions to LLM prompts; the system does not require completely precise masks, which are represented in its training data.Source: §5 · p. 8
- Foundation-model integration introduces noticeable latency, motivating offline language models and distilled lightweight vision models for deployment.GPT-4 performs best in the reported reasoning comparison, while DetGPT is described as a good lightweight dedicated-model option.Source: §5 · p. 8
6. Conclusion
The paper proposes conditioning robotic manipulation on masks derived from internet-scale vision foundation models and using a two-stream policy with local-global perception. Experiments report generalization particularly to unseen objects and backgrounds, and the policy can be conditioned for new skills.
6. Conclusion
- Masks carrying semantic, geometric, and temporal correlation priors improve sample efficiency when conditioning the manipulation policy.The priors come from internet-scale vision foundation models.Source: §6 · p. 8
- The two-stream policy model uses a local-global perception manner and performs excellently, according to the authors.Source: §6 · p. 8
- Experiments demonstrate the paradigm’s generalization capabilities, especially for unseen objects and backgrounds.The policy can also be conditioned by assigning different values to object masks for different skills, which the authors describe as making the system scalable to new skills.Source: §6 · p. 8