Source-linked AI summary
FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects
Ben Eisner, Harry Zhang, David Held
TL;DR
Articulated-object manipulation requires perception that can handle varied geometries and kinematic structures without class-specific articulation knowledge. FlowBot3D learns dense 3D articulation flow from point clouds and uses it for grasp selection and motion planning. The system demonstrates strong generalization across simulated and real environments, including unseen object categories.
Problem
Robots must manipulate articulated objects despite large variation in their geometries, kinematics, and categories, which makes class-specific articulation knowledge limiting.
Method
FlowBot3D predicts dense per-point 3D articulation flow from a static point cloud and uses the predicted field to select grasp points and guide robot motion.
Results
FlowBot3D successfully manipulates articulated objects in simulated and real environments and shows strong sim-to-real transfer generalization.
Takeaways & Limitations
A single learned 3D perception module can support articulation across seen and entirely unseen object categories without explicitly encoding articulation parameters.
Abstract
from arXiv · showhide
We explore a novel method to perceive and manipulate 3D articulated objects that generalizes to enable a robot to articulate unseen classes of objects. We propose a vision-based system that learns to predict the potential motions of the parts of a variety of articulated objects to guide downstream motion planning of the system to articulate the objects. To predict the object motions, we train a neural network to output a dense vector field representing the point-wise motion direction of the points in the point cloud under articulation. We then deploy an analytical motion planner based on this vector field to achieve a policy that yields maximum articulation. We train the vision system entirely in simulation, and we demonstrate the capability of our system to generalize to unseen object instances and novel categories in both simulation and the real world, deploying our policy on a Sawyer robot with no finetuning. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments.
I. INTRODUCTION
FlowBot3D addresses the challenge of generalizing articulated-object manipulation by learning dense 3D part-motion predictions and using them to guide action. Its representation and vision-based policy are designed to avoid class-specific articulation knowledge while supporting simulated and real-world evaluation.
- I. INTRODUCTION: The approach targets variation in object geometry and kinematics without explicitly estimating rotational or translational articulation axes.Prior methods often depend on class-specific articulation structures, limiting operation on novel categories.
- I. INTRODUCTION: FlowBot3D predicts dense per-point 3D articulation flow and uses it to produce manipulation actions across articulated objects.The system separates affordance learning from motion planning and trains a single 3D perception module across object categories.
- I. INTRODUCTION: The paper evaluates FlowBot3D through simulated experiments on PartNet-Mobility objects and real-world deployment on a Sawyer robot.These experiments test articulation across varied objects, including generalization and real-world feasibility.
- I. INTRODUCTION: Unlike approaches that learn motion through intermediate articulation parameters or demonstrations, FlowBot3D predicts part motion without those requirements.The related-work comparison characterizes the method as learning without demonstrations and without relying on intermediate articulation parameters.
A. An Idealized Policy Based On Dynamics and Kinematics
The paper derives optimal force application for prismatic and revolute joints, then motivates 3D articulation flow as a compact, hierarchy-free alternative to explicit articulation parameters.
- For prismatic joints, maximum acceleration is obtained by applying force parallel to the joint’s motion direction at any surface point.All points move in parallel, so contact location does not affect the maximum acceleration.
- For revolute joints, maximum acceleration comes from grasping farthest from the rotation axis and applying force tangent to the point’s circular trajectory.The optimal force direction is parallel to r × ω.
- Explicit articulation parameters become difficult for objects with multiple joints or stationary points because they require hierarchical link assignments or six parameters per point.These representations are difficult to recover from raw observations without prior structural knowledge.
- 3D articulation flow assigns each point a normalized motion-direction vector, avoiding explicit kinematic trees while remaining equivalent to parameter-based representations.The representation is bounded within the unit sphere and can be applied directly to point clouds.
- 3D articulation flow is translation- and scale-invariant and supports arbitrary discrete or continuous geometric representations, including depth-derived point clouds.For a point cloud, each point is associated with a flow vector whose magnitude is at most one.
C. Predicting 3D Articulation Flow from Vision
The system predicts 3D articulation flow directly from depth-camera point clouds and uses those predictions in a closed-loop policy for grasp selection and repeated articulation.
- The task assumes a depth camera observing a single articulated object and aims to articulate a specified part through its full range of motion.
- A neural network is trained with supervised L2 loss to predict ground-truth 3D articulation flow from point-cloud observations.Each scene configuration has a corresponding ground-truth flow determined by the articulation-flow formulation.
- The policy first predicts flow for all observed points, selects the highest-magnitude contact location, and plans a suction grasp aligned against its flow direction.The selected point is intended to maximize actuation efficiency through acceleration.
- During execution, the robot repeatedly observes the object, predicts visible flow vectors near the contact point, and moves in the direction with greatest magnitude.The loop continues while maintaining suction contact and is designed to handle multiple articulated parts.
E. FlowBot3D: A Robot Articulation System
FlowBot3D deploys the flow-based policy on a Sawyer robot using RGB-D point clouds, adapting contact selection to the physical constraints of suction grasping.
- FlowBot3D combines an RGB-D camera, Sawyer arm, and pneumatic suction gripper to execute the generalized articulation policy on a tabletop workspace.The system uses an Azure Kinect camera fixed above the workspace.
- Because suction cannot seal reliably on high-curvature locations, the robot filters edge-adjacent and highly curved points before selecting contact.The practical exclusion distance is d = 2cm, matching the suction-tip radius.
- The deployed system follows two phases: select a suction contact from predicted flow, then repeatedly follow the strongest visible flow direction during articulation.Flow vectors are shown as connections from current points to predicted positions, with vectors downsampled for clarity.
F. Training Details
FlowBot3D trains a single ArtFlowNet model on synthetic point-cloud and ground-truth-flow pairs across training categories, then evaluates it with multiple baselines in simulation. The evaluation includes normalized distance and success rate, with results indicating strong performance across novel test categories.
- Training Setup: FlowBot3D trains one ArtFlowNet model across categories using synthetically generated observations paired with ground-truth articulation flow.Training randomizes object states and uses PartNet-Mobility geometry and kinematics to generate supervision.
- Simulation Setup: The simulation uses ManiSkill with a suction-gripper interface to evaluate objects from PartNet-Mobility across training and entirely unseen categories.The split contains 11 training categories and 10 unseen categories, with 499 training, 128 testing, and 238 unseen objects.
- Metrics: Evaluation measures normalized distance after rollout and binary success, where success means articulating a part for more than 90% of its range of motion.Success thresholds normalized distance at δ = 0.1.
- Baselines: The strongest behavioral-cloning baseline, DAgger Oracle + F, fully articulates objects in only 33% of trials.The baseline uses ground-truth flow for contact selection and as an input, yet remains weak in the reported comparison.
- Simulation Results: FlowBot3D achieves a high simulated success rate across completely novel test categories, indicating that ArtFlowNet learns salient geometric features for articulated points.Its predicted flows recognize doors, lids, drawers, and other large articulated features; performance improves when objects are partially open.
B. Real-World Experiments
FlowBot3D was evaluated on varied household articulated objects with a Sawyer robot, using five trials per object and metrics covering contact, motion, distance, and overall success. It transferred from simulation without retraining, substantially outperforming DAgger while failures mainly arose from flow prediction and contact errors.
- Experimental setup: The real-world protocol used five trials per object across doors, drawers, hinges, jars, and other household articulations placed at random workspace positions.Objects were selected for workspace fit, suction compatibility, and visibility to the depth camera.
- Quantitative analysis: The DAgger baseline achieved only 14% overall success, while its contact rates were comparable and its predicted motions were usually unsuccessful.Both methods used the same FlowBot3D contact selection and execution policy for the contact-success comparison.
- Qualitative analysis: FlowBot3D transferred from simulation without retraining, with ArtFlowNet producing high-fidelity flow predictions on varied real objects despite occlusions.The experiments used a single simulation-trained model and viewpoint augmentation for the real camera configuration.
- Failure modes: Most real-world failures came from incorrect flow predictions or contact failures, including excessive robot occlusion, missed articulations, and poor suction points.The authors suggest multiple viewpoints, temporal filtering, recurrent policies, improved gripper compliance, and stronger contact prediction as possible mitigations.
C. Simulation Ablations
Ablations examined articulated-part masks and viewpoint augmentation in ArtFlowNet. Removing the mask caused only a small performance decrease, while viewpoint augmentation had little simulated effect but improved sim-to-real transfer.
- Including a part mask: Removing the articulated-part mask decreased performance only slightly, although masks improved predictions in ambiguous coplanar configurations.The mask was most helpful when a closed door was coplanar with its parent link.
- Applying viewpoint augmentations during training: Viewpoint augmentation had little effect in simulation but improved performance during sim-to-real transfer.The augmentation randomized camera viewpoints during synthetic dataset generation.
V. CONCLUSION
The paper presents 3D Articulation Flow and FlowBot3D as a visual representation and manipulation policy that work in simulated and real environments. It reports strong sim-to-real generalization while identifying substantial room for improvement and several future directions.
- Conclusion: 3D Articulation Flow and FlowBot3D successfully manipulate articulated objects in both simulated and real environments.The conclusion frames the representation and policy as the paper’s central contributions.
- Conclusion: Strong sim-to-real transfer generalization was observed, but the method still has substantial room for improvement across object classes.The authors propose more compliant control, better contact prediction, online adaptation, learning from demonstrations, and more complex manipulation settings as future directions.
APPENDIX
The appendix describes the real-world sensing, calibration, point-cloud preprocessing, and suction-contact procedures used to deploy FlowBot3D on a Sawyer robot. These procedures compensate for differences between idealized simulation contact and real suction-grasping conditions.
- Hardware and calibration: Real-world deployment used a Sawyer robot, Azure Kinect depth camera, 3 cm suction cup, and a calibrated workspace with collision geometry.Camera-to-world calibration used an ArUco marker and approximately 30 robot-pose samples.
- Point-cloud preprocessing: Point clouds were cleaned by removing tabletop points below 0.015 m and filtering robot points using a rendered URDF model plus statistical outlier removal.The robot-point filter assumes perfect camera calibration, so residual points may remain when calibration is slightly misaligned.
- Suction contact: Because real suction contact is not guaranteed as in simulation, the system selects low-curvature interior points and rejects high-curvature maximum-flow points when necessary.If the maximum-flow point exceeds the curvature threshold, the nearest low-curvature point at least 2 cm away is selected.
F. Grasp Selection Details
The real-world grasp-selection procedure uses predicted articulation flow to choose a contact point, orient the end effector, and plan an approach trajectory.
- FlowNet predictions and a curvature heuristic select the max-flow point when it is sufficiently low-curvature, otherwise choosing a nearby low-curvature point at least 2 cm away.The selected point is used as the contact point for grasp selection.
- The robot computes a quaternion difference from the chosen flow direction and applies it to the starting orientation to obtain the goal end-effector orientation.The orientation is then converted to Euler angles for angular-speed control.
- Because the robot does not contact the selected point directly, it first plans to a point 10 cm along the chosen flow direction before switching to velocity control for the approach.The initial pose is obtained through position control and an RRTConnect-based inverse-kinematics solver.
H. Network Architecture
FlowBot3D generates training flow labels analytically from simulated kinematic chains and predicts them with a PointNet++-based regression network.
- ArtFlowNet retains the PointNet++ architecture but replaces its segmentation head with a regression head for articulation-flow prediction.The network is implemented with PyTorch-Geometric and trained with L2 loss using Adam.
- Ground-truth 3D Articulation Flow is generated by parsing simulated URDF files into kinematic chains and analytically computing point locations after small displacements.Part-specific masks isolate the points belonging to the part of interest.
- The training environment modifies ManiSkill and SAPIEN with diverse PartNet-Mobility objects drawn from the UMPNet training and testing object lists.Some phone and door objects were filtered because of mesh-collision problems during simulator loading.
K. Hyperparameters
The supplied implementation and evaluation details cover simulation settings, real-world object trials, and comparisons in the UMPNet environment and on physical experiments.
- Real-world experiments use 14 objects, whose identities and articulation types are listed in Table V and illustrated individually in Figure 8.The object set includes jars whose lids function like kitchen-pot lids despite not technically being kitchen pots.
- The direct UMPNet evaluation uses PyBullet rather than SAPIEN, disables most gripper-object collisions, and imposes a hard contact constraint.FlowBot3D’s original environment uses SAPIEN, full contact, and softer spring-like contact behavior.
- The UMPNet evaluation compares methods using normalized distance to the target articulation joint angle, where lower values are better.The metric is evaluated after a full rollout in the official UMPNet environment.
- Overall, FlowBot3D and UMPNet perform similarly in the UMPNet environment, although ArtFlowNet was not significantly tuned for that different task distribution.FlowBot3D was retrained on point clouds generated in PyBullet for this evaluation.
- Full-trial results in Tables VIII and IX average metrics over all five trials for each object.The tables report real-world trials for FlowNet and a DAgger Oracle.
- Success rate is defined as the fraction of trials whose normalized distance is below 0.1, with higher values preferred.This comparison is also performed after a full rollout in the official UMPNet environment.