Source-linked AI summary
Point Bridge: 3D Representations for Cross Domain Policy Learning
Siddhant Haldar, Lars Johannsmeier, Lerrel Pinto, Abhishek Gupta, Dieter Fox, Yashraj Narang, Ajay Mandlekar
TL;DR
Robot learning lacks large-scale real-world manipulation data, and simulation’s scalability is limited by the visual gap between simulation and reality. Point Bridge uses unified domain-agnostic point representations with VLM-guided extraction and transformer-based policies to transfer synthetic-data training to real robots. It reports up to 44% gains in zero-shot transfer and up to 66% with limited real data across single-task and multitask settings.
Problem
Robot foundation-model progress is constrained by scarce large-scale real-world manipulation datasets, while simulation is limited by the visual domain gap between simulation and reality.
Method
Point Bridge uses unified domain-agnostic point representations, VLM-guided point extraction, transformer-based policy learning, and inference-time pipelines for synthetic-data sim-to-real transfer.
Results
39% and 44% improvements are reported for single-task and multitask zero-shot transfer, while limited real-data co-training improves over prior works by 61% and 66%, respectively.
Takeaways & Limitations
Point Bridge supports zero-shot and limited-real-data sim-to-real policy learning across single-task and multitask settings using synthetic simulation data.
Takeaways & Limitations
The framework depends on vision models, requires camera-pose alignment between simulation and reality, discards scene context in clutter, and currently operates at lower control frequency than image-based baselines.
Abstract
from arXiv · showhide
Robot foundation models are beginning to deliver on the promise of generalist robotic agents, yet progress remains constrained by the scarcity of large-scale real-world manipulation datasets. Simulation and synthetic data generation offer a scalable alternative, but their usefulness is limited by the visual domain gap between simulation and reality. In this work, we present Point Bridge, a framework that leverages unified, domain-agnostic point-based representations to unlock synthetic datasets for zero-shot sim-to-real policy transfer, without explicit visual or object-level alignment. Point Bridge combines automated point-based representation extraction via Vision-Language Models (VLMs), transformer-based policy learning, and efficient inference-time pipelines to train capable real-world manipulation agents using only synthetic data. With additional co-training on small sets of real demonstrations, Point Bridge further improves performance, substantially outperforming prior vision-based sim-and-real co-training methods. It achieves up to 44% gains in zero-shot sim-to-real transfer and up to 66% with limited real data across both single-task and multitask settings. Videos of the robot are best viewed at: https://pointbridge3d.github.io/
1 INTRODUCTION
Point Bridge addresses scarce real-world robot data and the simulation-to-reality visual gap with unified point-based representations, enabling synthetic-data policy transfer. It combines VLM-guided extraction, scalable policy learning, and inference pipelines, reporting gains in zero-shot and limited-real-data settings.
- Robot learning remains constrained by scarce real-world interaction data, while simulation offers scalable synthetic demonstrations but faces a visual domain gap.
- Point Bridge uses unified, domain-agnostic point-based representations to enable zero-shot sim-to-real policy transfer from synthetic simulation data.
- The framework is evaluated on six real-world tasks and includes a systematic analysis of its key design choices.
- Its components include a VLM-based point extraction pipeline and multiple inference-time pipelines for different performance and throughput needs.
- 39% and 44% improvements are reported for single-task and multitask zero-shot sim-to-real transfer, respectively.
- 61% and 66% improvements over prior works are reported with small real-data co-training in single-task and multitask settings, respectively.
2 RELATED WORK
Related work addresses robot-learning data bottlenecks, structured representations, behavioral cloning, and sim-to-real transfer. Existing approaches reduce some challenges but retain dependence on human input, restricted generalization, or substantial tuning.
- Structured representations include bounding boxes, object poses, and point clouds, each trading off instance generalization, model specialization, or spatial-relationship learning.
- Teleoperated and tool-assisted collection methods expand demonstrations but still rely on human input, while simulation-centric generation automates data creation.
- Behavioral cloning learns policies offline from demonstrations, but image-based approaches can limit generalization to new objects and uncontrolled environments.
- Domain randomization improves sim-to-real robustness through simulated variability but requires careful tuning and substantial human effort to define effective ranges.
3 PREREQUISITES
The prerequisites formulate imitation learning from demonstrations and expand simulated data for zero-shot deployment, with optional real-data co-training and multitask learning. They assume structured observations, task actions, and calibrated camera geometry.
- Imitation learning trains a behavior policy from expert demonstrations containing observations and actions over episode horizons.
- Behavioral cloning trains the policy by maximizing the log-likelihood of expert actions.
- The source simulation dataset is expanded with synthetic generation into a larger dataset for policies intended for zero-shot real-world deployment.
- A small real-world demonstration set can support joint training with simulated data, and one policy can be trained across multiple tasks.
- Simulation assumes continuous end-effector pose commands, a discrete gripper command, manipulable objects, and observable object poses before contact.
- Real-world experiments assume calibrated camera intrinsics and extrinsics with all 3D observations aligned to the robot arm’s base frame.
4 POINT BRIDGE
POINT BRIDGE uses a unified point-based scene representation to support sim-to-real policy transfer with minimal alignment. It expands small simulated demonstration sets, extracts task-relevant 3D points, and trains policies that can use multiple deployment and sensing pipelines.
- Framework overview: POINT BRIDGE converts simulated and real observations into compact point-based representations for policy learning and sim-to-real transfer.The framework combines a unified scene representation with synthetic data generation and optional real-world co-training.
- Data collection and synthetic data generation: A small set of simulated demonstrations is expanded into a larger dataset by adapting demonstration segments to novel object configurations.MimicGen applies constant SE(3) transformations that preserve relative end-effector and object geometry.
- Point extraction: Simulation samples points from object meshes while reproducing real camera intrinsics, extrinsics, and viewpoint-dependent visibility.Mesh points are projected using the corresponding camera parameters to better match the observations available from real cameras.
- Policy learning: Robot end effectors are represented as keypoints and combined with object points for PointNet encoding and decoder-only multi-task transformer policy learning.Language embeddings can be included for multitask learning, and action chunking with exponential temporal averaging smooths predicted tracks.
- Policy inference: POINT BRIDGE supports deployment with stereo depth, commodity RGB-D sensors, or multiview triangulation while retaining the same trained policy.The inference pipeline tracks and triangulates keypoints as needed, enabling deployment across diverse real-world setups.
5 EXPERIMENTS
Experiments evaluate POINT BRIDGE across simulated and real manipulation settings, including zero-shot transfer, co-training, multitask learning, soft and articulated objects, and system design choices.
- Zero-shot sim-to-real transfer: POINT BRIDGE outperforms the strongest baseline by 39% in single-task and 44% in multitask zero-shot sim-to-real transfer.The method operates despite substantial differences in table appearance, backgrounds, and lighting.
- Zero-shot sim-to-real transfer: POINT BRIDGE handles diverse object instances and visually challenging transparent or reflective objects through FoundationStereo depth estimation.RGB-D depth sensing typically struggles with transparent or reflective items.
- Multitask transfer: Multitask POINT BRIDGE policies are language-conditioned and achieve comparable or better performance than single-task policies.The filtered point-cloud representation supports scaling across diverse tasks.
- Co-training: Adding 45 real demonstrations to 1200 simulated demonstrations per task improves performance by up to 30%, while outperforming image-based co-training by 61% in single-task and 66% in multitask settings.Co-training uses an 80–20 simulation-to-real ratio across three tasks.
- Soft and articulated objects: POINT BRIDGE achieves an 85% success rate across real tasks involving soft towels and articulated drawers and ovens, using 20 demonstrations per task without simulation data.These results extend the representation beyond rigid-object manipulation.
- System analysis: FoundationStereo provides the best depth-estimation performance at 5 Hz and remains robust on reflective surfaces, whereas alternative methods are noisier, slower, or less accurate.Accurate depth estimation is critical for policy inference.
- System analysis: Training with camera-aligned simulated points significantly improves sim-to-real transfer over uniformly sampled points.Randomizing camera views during data generation can reduce the need for matched camera views.
6 LIMITATIONS & CONCLUSION
POINT BRIDGE uses domain-agnostic point representations for minimally aligned sim-to-real transfer, real-data co-training, and multitask learning, while retaining several practical limitations.
- Conclusion: POINT BRIDGE enables zero-shot sim-to-real transfer with minimal visual alignment, supports real-world co-training, and facilitates multitask policy learning.The conclusion frames these capabilities as the framework’s central contributions.
- Limitations: POINT BRIDGE depends on VLMs and other vision models, making it vulnerable to their failures.The authors expect robustness to improve as these models advance.
- Limitations: The method requires camera pose alignment, discards critical scene context in cluttered environments, and currently operates at lower control frequency than image-based baselines.The authors propose diverse simulated viewpoints and hybrid representations as possible remedies.
A.1 COMPARISON WITH POINT POLICY
POINT BRIDGE and Point Policy both use keypoint-based cross-domain learning, but differ in transfer target, point extraction, representation, and task scope.
- Transfer setting: Point Policy primarily targets zero-shot human-to-robot transfer, whereas POINT BRIDGE mainly targets sim-to-real transfer.
- Point extraction: POINT BRIDGE replaces Point Policy’s per-task manual annotations with VLM-based automated point extraction for scaling to novel tasks.
- Point extraction: Point Policy uses point tracking and multiview triangulation, while POINT BRIDGE uses 2D segmentation tracking with SAM-2 for faster point handling.SAM-2 runs at 20 Hz on 512 × 512 images, with throughput unaffected by the number of object points.
- Architecture: POINT BRIDGE encodes the entire 3D point cloud as one PointNet embedding instead of representing each point-track history as an individual transformer token.
- Task scope: Point Policy is limited to single-task training, whereas POINT BRIDGE functions in multitask settings.
A.2 EXPERIMENTS
Additional experiments describe the simulation data scale, real-task coverage, hardware constraints, and boundaries imposed by unrealistic articulated-object assets.
- Simulation data generation: MimicLabs defines three atomic tasks with four object-instance pairs each, and MimicGen scales five demonstrations per pair to 300.POINT BRIDGE uses this synthetic-data scale for sim-to-real transfer.
- Simulation boundaries: The experiments primarily transfer pick-and-place tasks from simulation, while articulated tasks remain difficult because simulated asset dynamics can differ from real objects.The authors leave more realistic assets and broader object variants for future work.
- Policy learning considerations: FoundationStereo runs up to 10 Hz in TensorRT-optimized form, but the overall control frequency is 5 Hz because depth processing uses a separate machine and Ethernet communication.
- Task coverage: Real-task evaluation includes substantial spatial and object variation, with 1200 simulated demonstrations for three transfer tasks and 20 real demonstrations for tasks lacking aligned simulation assets.
A.2.1 HYPERPARAMETERS
This section provides the hyperparameters used for Point Bridge.
- Table 5 lists the hyperparameters used by Point Bridge.
- The reported hyperparameters support reproducibility of the Point Bridge experiments.
- The section identifies Table 5 as the location of the implementation settings.
A.2.2 ADDITIONAL EXPERIMENTS AND SYSTEM ANALYSIS
Additional experiments analyze baseline comparisons, calibration, distractors, object generalization, representation choices, robustness, and system latency. Together, they characterize Point Bridge’s transfer behavior and operating conditions.
- Calibration changes: Approximately 47% average success is obtained across three tasks when camera viewpoints are randomized between simulation and deployment.Performance drops relative to matched viewpoints, motivating robustness to viewpoint-dependent 3D point-distribution differences.
- Background distractors: POINT BRIDGE maintains distractor-free performance with background clutter, unlike the unfiltered point-cloud baseline.The comparison evaluates BAKU-PCD and Point Bridge with and without background distractors.
- Generalization to held-out objects: 76% and 80% success rates are achieved on novel real-world objects in single-task and multitask zero-shot transfer, respectively.Co-training raises success to 98% for single-task and 100% for multitask settings on objects present in the real dataset.
- Generalization to held-out objects: 97% multitask success is retained on held-out objects absent from both simulated and real training sets, versus 100% for encountered objects.Failures mainly occur for bowls substantially larger than those in training.
- Number of points: All point-count configurations exceed 86% success, while 64 points per object performs best among 10, 64, and 128 points.This indicates similar performance across sparse and dense point-cloud regimes.
- Action representation: Pose regression and point-track prediction show comparable performance for Point Bridge.The authors suggest that the relative benefit of point-track supervision may be reduced by the larger dataset scale used here.
- System analysis: The VLM-guided scene-filtering pipeline achieves high success rates despite foundation-model failures, with approximately 9 seconds of one-time initialization overhead.The robustness analysis evaluates bowl-on-plate, mug-on-plate, and stack-bowls tasks; runtime analysis separates initialization from per-step execution.