Source-linked AI summary
Revisiting Articulated Parts Perception in Robot Manipulation
Xiaoqian Wu, Yejie Guo, Xiaoyang Chen, Lixin Yang, Cewu Lu, Yong-Lu Li
TL;DR
Articulated-part perception needs representations that avoid the annotation cost of pose methods and the noise and ambiguity of tracked affordance flow. This paper introduces GPS and a portable VR-GPS system for scalable geometry annotation, then evaluates a generalized GPS model in robot manipulation, achieving a 73% success rate without in-domain fine-tuning.
Problem
Pose-based articulated-part representations require high-cost annotation, while affordance-based flow suffers from tracking errors and ambiguity.
Method
GPS represents articulated geometry through explicit motion axes and part points, collected with a portable VR system using the grasping hand as a motion proxy.
Results
73% success rate was achieved across 270 initial states for 9 objects without in-domain fine-tuning using predicted GPS and a heuristic manipulation policy.
Takeaways & Limitations
VR-GPS collected 41K frames for 234 objects across six part classes, and the learned GPS model outperformed pose-based and flow-based methods in articulated-parts understanding.
Takeaways & Limitations
Affordance-based flow remains limited by tracking inaccuracies from self-occlusion or camera movement and by sensitivity to object scale and temporal execution variations.
Abstract
from arXiv · showhide
We are surrounded by various objects with movable, articulated parts, e.g., box, handle, door. An accurate and generalizable perception of articulated parts is essential to enhance robotic manipulation capabilities. Building on this need, recent efforts in articulated parts perception have followed two main directions: One line of work uses pose-based representation, which requires high manual cost; in parallel, affordance-based methods extract future object motion from point tracking without additional manual efforts, but suffer from low-quality data. In this paper, we propose a new representation of articulated parts, Geometric Primary Structure (GPS), an abstraction of the part geometry structure to balance scalability and quality. For efficient and scalable data collection, GPS is integrated with a portable Virtual Reality (VR) device and requires only one minute to annotate one object sequence. This direct human annotation provides higher quality than the estimated affordance. With this efficient VR-GPS system, we collect 41K frames for 234 objects across six part classes, and train a generalizable GPS model with a single RGB-D object image as input. For object manipulation, we deploy a heuristic policy based on GPS prediction. Without any in-domain fine-tuning, our method achieves an 73% success rate, covering 270 initial states for 9 objects. Our code, data and reusable tool are available at https://enlighten0707.github.io/gps.
1. Introduction
Articulated-part perception remains difficult because pose-based representations are costly to annotate and affordance-based flow is noisy and ambiguous. The paper proposes GPS with VR annotation and demonstrates generalized perception for robot manipulation.
- Pose-based methods require costly synthetic modeling or real-world scanning, frame-wise annotation, and remain vulnerable to category, processing-time, and real-world-error limitations.
- Affordance-based methods reduce manual annotation by extracting point trajectories, but tracking errors and flow ambiguity make predictions sensitive to occlusion, object scale, and execution speed.
- GPS explicitly annotates motion axes with a VR device and uses the grasping hand as a motion proxy to reduce tracking noise and flow ambiguity.
- 41K RGB-D frames covering 234 objects across six part classes were collected with VR-GPS, requiring one minute per video without manual post-processing.
- 73% success rate was achieved across 270 initial states for 9 objects without in-domain fine-tuning using a heuristic policy based on predicted GPS.
- The paper contributes GPS, a VR-collected geometry-rich dataset, and a generalizable GPS model for real-world manipulation of daily objects.
2. Related Work
Prior articulated-object perception methods are broadly pose-based or affordance-based. Pose-based approaches estimate structured part geometry, while affordance-based approaches predict manipulation contacts and future motion with coarser supervision.
- Pose-based methods estimate part segmentation and 6-DoF pose, often using Normalized Part Coordinate Space for each object category.
- Pose-based training data comes from synthetic assets, professionally captured real-world scans, and associated pose and camera annotations.
- Affordance-based methods identify where and how to manipulate objects through contact points and future trajectories.
- VRB uses hand-object contact and hand motion as 2D affordance guidance, but 2D affordance provides only coarse supervision.
3. Definition
GPS represents articulated geometry using axis-defining points and a part point, with formulations for both rotational and translational motion. The representation replaces difficult-to-benchmark contact points with looser geometric constraints.
- Part Rotation: For revolute parts, the structure is constrained by a unit axis direction u, an anchor point q, and a contact point m.
- Part Rotation: The contact point m follows a rotation-angle trajectory defined by the revolute axis and its skew-symmetric matrix.
- Part Rotation: GPS represents rotational parts as {q1, q2, p}, where q1 and q2 define the axis and p is a part point replacing the non-unique contact point.
- Part Rotation: The part point p lies on the plane defined by the axis points and contact point, and loose constraints make it more generalizable for grasp proposal selection than m.
- Part Translation: For prismatic parts, motion follows a unit translation axis u and offset δ, with GPS using q1, q2, and p to encode axis direction and a centered axis.
- Part Translation: For translational parts, p is constrained to a plane defined by the axis direction and contact point m.
4. Data Collection
VR-GPS uses a portable headset-based workflow to annotate articulated-part geometry directly during RGB-D interaction. The system supports diverse views and efficient collection without expensive scanning or motion-capture equipment.
- System Design: Annotators place virtual axis points before interaction and attach a virtual part point to their fingers during RGB-D recording.The points remain fixed or move with the annotated interaction, enabling real-time GPS capture.
- System Design: Multiple RGB-D videos are recorded from different headset views, then transformed and segmented into object data with GPS annotations.The pipeline repeats axis placement and recording across object poses and camera views.
- System Design: AR annotation places points anywhere in 3D space and enables corrections during interaction, including points not visible on the camera-facing surface.This addresses difficult annotations such as revolute axes on thin objects and supports portability beyond laboratory settings.
- Data Analysis: The VR-GPS device costs 800 dollars, and each video takes one minute to annotate on average.The setup records three videos per object with different camera views and avoids expensive MoCap or 3D-scanning systems.
- Data Analysis: The dataset contains 41K frames from 234 objects spanning six part classes.The classes include Lid, Lid-thin, Lid-book, Handle, Door, and Drawer.
5. Geometric Structure Learning
The GPS model predicts geometric structure from RGB-D features and is evaluated through axis and part errors on external datasets. It outperforms pose-based and flow-based comparisons across the reported benchmarks, while flow remains sensitive to tracking and training-data diversity.
- Model Design: The model concatenates semantic RGB features with corresponding 3D points to predict GPS parameters {q1, q2, p}.RGB features have dimension d_s = 480 and are extracted from the RGB image alongside the depth-derived point cloud.
- Model Design: Training losses separately supervise axis direction, axis offset, and part direction or offset for rotational and prismatic motion.The rotational formulation uses L = L_ad + L_ao + L_pd, while translation uses L = L_ad + L_ao + L_po.
- Benchmark: Evaluation measures Average Axis Direction Error, Average Axis Offset Error, Average Part Direction Error, and Average Part Offset Error.The maximum offset error is 2 when the point cloud is normalized into a unit cube.
- Performance Comparison: GPS outperforms CAPNet across all five categories on the HOI4D out-of-domain benchmark.The authors attribute the comparison advantage partly to real-world training data mitigating the sim-to-real gap.
- Performance Comparison: The GPS-based heuristic policy selects grasp proposals and generates manipulation waypoints from predicted geometric structure.The policy uses predicted axis and part points to guide grasping and subsequent motion.
- Performance Comparison: GFlow exhibits large errors on the out-of-domain RGBD-Art benchmark, which the authors attribute to flow sensitivity and limited training-data diversity.Using the collected data for flow training still leaves Ours-Flow below Ours-GPS because of inaccurate tracking and flow ambiguity.
6. Real Robot Experiments
The experiments use GPS predictions to select initial grasps and generate axis-guided waypoints, evaluating manipulation across diverse objects and viewpoints. GPS achieves 73% success without in-domain fine-tuning, while GPS-GT reaches 91% and the baselines perform substantially worse.
- Initial Grasp Evaluation: 91% success rate over 270 trials verifies that the GPS representation can support initial grasp selection and waypoint execution.This GPS-GT evaluation excludes learned GPS prediction error.
- Performance Comparison: 73% average success rate is achieved by the learned GPS model without any in-domain fine-tuning.The evaluation covers 9 objects and 270 trials, with each object tested across different states and camera views.
- Performance Comparison: 33% success rate is obtained by CAPNet, whose inaccurate part recognition or articulation axis can produce invalid grasps and motions.The reported failures include misrecognizing a closed door and bending a box lid instead of opening it.
- Performance Comparison: 35% success rate is obtained by GFlow despite using a manually given contact point, with predicted flows deviating from the required object motion.Its policy selects nearby query points and uses singular value decomposition to align end-effector motion over four execution steps.
- Ablation: 58% success rate is obtained when GPS predicts contact point m instead of part point p, supporting the use of looser geometric constraints.A folder failure occurred because the predicted contact point was far from easily graspable edges.
- Failure Case Analysis: Failure cases arise from point-cloud noise, incorrect grasp selection despite accurate GPS, and a scoring function that cannot flexibly balance grasp confidence with geometry.The authors suggest fine-tuning the grasping model with GPS input or integrating GPS with diffusion or vision-language-action policies.
7. Conclusion
The paper presents GPS as an affordance representation for articulated-part estimation that balances data scalability with annotation quality. A VR-based data-collection system supports a dataset with rich object geometry knowledge, and the learned GPS model facilitates manipulation of daily objects through a heuristic policy.
- GPS balances data scalability with annotation quality for articulated-part estimation.
- A data-efficient VR system produces the VR-GPS dataset with rich object geometry knowledge.
- The learned GPS model achieves better perception performance and facilitates robot manipulation of daily objects via a heuristic policy.
9. Detailed Comparison with Existing Works
The paper contrasts GPS with post-processing approaches that reconstruct articulated objects from scans or visual inputs. Existing methods can be time-consuming, category-restricted, or unreliable on real-world scenes.
- Post-processing methods reconstruct articulated objects from visual inputs but have distinct computational and generalization limitations.
- RSRD requires about 40 minutes for reconstruction on one 3090 GPU and 10 minutes of pose estimation for each interaction sequence or new environment and camera view.
- Ditto is trained on only 8 categories and requires a separate network for each category, limiting processing of many VR-GPS objects.
- PARIS and ArtGS perform well on synthetic objects but fail to estimate joints in real-world scenes and require an extra 20 minutes to manually align two states.
10. Detailed Dataset Statistics
The VR-GPS dataset uses a Meta Quest 3-based system and covers six articulated-part classes. It contains annotations from multiple volunteers across 234 objects, while most evaluated robot tasks involve object geometric structure.
- VR-GPS maps virtual point coordinates into the camera frame using headset, world-frame, and RealSense transformations.
- 70% of 89 complex tasks in RH20T require knowledge of object geometric structure.
- Eight volunteers annotated the dataset and three additional volunteers checked it.
- The dataset contains six part classes: Lid, Lid-thin, Lid-book, Handle, Door, and Drawer.The class counts are 89, 21, 32, 34, 33, and 25 objects, respectively.
11. Geometric Structure Learning
This section describes evaluation datasets, flow-to-GPS transformations, and the GPS-based heuristic policy for planning robot trajectories.
- Evaluation Setup: External evaluation uses HOI4D and RGBD-Art datasets spanning five articulated-part categories.HOI4D and RGBD-Art provide frames for Laptop, Trashcan, Safe, Bucket, and Drawer objects.
- Transform Flow into GPS: Flow-to-GPS comparison samples 1024 surface points, retains K = 256 trajectories with largest displacements, and derives GPS from their motion.The procedure filters static parts before estimating revolute or prismatic axis geometry.
- Transform Flow into GPS: For revolute objects, the rotation-axis direction is computed by PCA over motion vectors.The axis direction minimizes the summed squared projections of motion vectors onto a unit vector.
- Transform Flow into GPS: The axis point q is estimated by averaging per-trajectory axis-position candidates obtained from common intersections of internal perpendicular-motion lines.Each selected trajectory contributes one candidate before the final mean is computed.
- Transform Flow into GPS: For prismatic joints, the axis direction uses the largest-eigenvalue eigenvector, while axis position averages trajectory centers projected onto the perpendicular plane.This reflects motion parallel to the prismatic axis.
- Heuristic Policy: GPS is computed from predicted part segmentation and NPCS using RANSAC, Umeyama alignment, and bounding-box coordinates.The resulting GPS is used in the policy pipeline.
- Heuristic Policy: The heuristic policy takes an object point cloud and time step as input and returns a planned robot trajectory.It updates poses using rotation for revolute joints and translation for prismatic joints.
12. Real Robot Experiments
The real-robot experiments evaluate GPS-guided grasp selection and trajectory planning across diverse articulated objects, with an additional diffusion-policy integration study.
- Real-Robot Evaluation: The experiments cover nine objects spanning lid, handle, door, drawer, lid-book, and lid-thin part classes.The objects include Box, Document-Box, Bucket, Door, Drawer, Notebook, Folder, Lamp, and Clapperboard.
- Heuristic Policy: GPS predictions score grasp proposals using geometry-specific criteria combined with original grasp confidence scores.Revolute joints use a plane-angle criterion, whereas prismatic joints use distance to a plane.
- Real-Robot Evaluation: The system provides robot manipulation videos as supplementary material.
- GPS-Policy with RISE: GPS-Policy conditions the RISE diffusion policy on predicted GPS for a rotation-lid closing task.GPS is extracted from the initial frame, and future GPS predictions guide action generation at each frame.