Source-linked AI summary
ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory
Ajinkya Jain, Rudolf Lioutikov, Caleb Chuck, Scott Niekum
TL;DR
Existing visual methods for articulation-model estimation require textured objects or prior knowledge of the articulation category. ScrewNet uses screw theory to estimate models from depth images without that prior category knowledge. It reports performance at par with or better than the baseline across object model categories, including generalization to novel object classes within an articulation category.
Problem
Visual articulation-model methods require textured objects or prior knowledge of the articulation category, which may not be available for novel objects encountered by robots.
Source: §I · p. 1Method
ScrewNet represents one-degree-of-freedom rigid, revolute, prismatic, and helical models as screw displacements along a common axis and estimates the model, parameters, and configurations from segmented depth-image sequences.
Source: B. ScrewNet · p. 3Results
ScrewNet performed at par or better than the baseline across object model categories and generalized to novel object classes within the same articulation category, where the baseline failed.
Source: C. Across articulation model category · p. 5, 3) Experiment 2: Same articulation… · p. 9Takeaways & Limitations
The results support estimating articulation models from depth images without requiring a priori category knowledge, including across articulation model categories.
Source: §I · p. 1, §VI · p. 6
Abstract
from arXiv · showhide
Robots in human environments will need to interact with a wide variety of articulated objects such as cabinets, drawers, and dishwashers while assisting humans in performing day-to-day tasks. Existing methods either require objects to be textured or need to know the articulation model category a priori for estimating the model parameters for an articulated object. We propose ScrewNet, a novel approach that estimates an object's articulation model directly from depth images without requiring a priori knowledge of the articulation model category. ScrewNet uses screw theory to unify the representation of different articulation types and perform category-independent articulation model estimation. We evaluate our approach on two benchmarking datasets and compare its performance with a current state-of-the-art method. Results demonstrate that ScrewNet can successfully estimate the articulation models and their parameters for novel objects across articulation model categories with better on average accuracy than the prior state-of-the-art method. Project webpage: https://pearl-utexas.github.io/ScrewNet/
I. INTRODUCTION
Articulation models matter for robot manipulation, but existing visual methods rely on object texture or prior knowledge of the articulation category. ScrewNet uses screw theory to estimate models from depth images without that category knowledge, and the paper evaluates it on benchmark datasets and real objects.
I. INTRODUCTION
- Robots need articulation properties to manipulate diverse objects safely, while learning policies through interaction may not cover that diversity within limited interaction time.Estimating models from observations offers an alternative for subsequent planning or model-based reinforcement learning.Source: §I · p. 1
II. RELATED WORK
- Prior visual methods estimate articulation by tracking relative motion with fiducial markers or feature tracking on textured objects [9] [11], while deep networks automatically extract relevant features from raw images [13].Interactive-perception approaches also require textured objects to estimate articulation models from raw images [20].Source: §I · p. 1, §II · p. 2
I. INTRODUCTION
- ScrewNet uses screw theory to unify rigid, revolute, and prismatic motion as constrained motion about a fixed screw axis, enabling category-independent estimation from depth images.The unified approach also supports helical models without changes to network architecture or training procedure.Source: §I · p. 1
- ScrewNet is evaluated on two benchmark datasets and three real-world objects against a state-of-the-art method and three ablations, outperforming all baselines by a significant margin.The evaluated objects are a microwave, drawer, and toaster oven.Source: §I · p. 1
II. RELATED WORK
- Other related approaches estimate articulation using known CAD models or categories [25] [26], or use fiducials to track object movement, unlike ScrewNet's depth-image approach.Demonstration-based methods have also incorporated natural-language descriptions.Source: §II · p. 2
III. BACKGROUND
Screw theory represents a rigid-body displacement as rotation and translation about an axis. Plücker coordinates represent the axis as a line, whose distance and frame transformation are defined for comparing displacements.
III. BACKGROUND
- A screw displacement encodes rigid-body motion as an axis line, rotation θ, and translation d, with d = hθ for pitch h.The axis line uses Plücker coordinates (l, m), where m = p × l, with ∥l∥ = 1 and ⟨l, m⟩ = 0.Source: §III · p. 2
- The distance between Plücker lines is zero when they intersect and uses ∥l1 × (m1 − m2)∥ when they are parallel.The distance expression distinguishes these cases from the general line-distance formula.Source: §III · p. 2
- The frame-displacement matrix transforms a Plücker line from frame FA to frame FB using the rotation R and translation t between the frames.The translation's skew-symmetric matrix is [t]×.Source: §III · p. 2
A. Problem Formulation
ScrewNet estimates an articulated object's model, parameters, and configurations from depth-image sequences without prior knowledge of its articulation category. It represents the supported one-degree-of-freedom models through screw displacements along a shared axis.
A. Problem Formulation
- For a sequence of depth images, the goal is to estimate the articulation model, its parameters, and the configurations describing relative motion between two parts.The formulation assumes motion is governed by a single articulation model with at most one degree of freedom.Source: A. Problem Formulation · p. 2
B. ScrewNet
- ScrewNet represents rigid, revolute, prismatic, and helical motion as screw displacements along a common axis, distinguished by their rotation and displacement components.Rigid motion has neither component; revolute and prismatic motion have only rotation or displacement, while helical motion has both.Source: B. ScrewNet · p. 3
- The shared representation uses six parameters for the screw axis and 2n configuration parameters, allowing one network to estimate models across categories and infer the category.The paper notes that a single network can share training data across categories and can represent helical motion without architectural or training-procedure changes.Source: B. ScrewNet · p. 3
- ResNet extracts image features, an LSTM encodes their sequence, and an MLP predicts screw displacements with a shared axis.A decision tree infers the model category from the predicted displacements.Source: B. ScrewNet · p. 3, Figure 2 · p. 3
- Training minimizes a weighted multi-objective loss over screw-axis orientation and distance, axis constraints, and configuration errors.The total loss combines the four terms shown in Eq. (3).Source: B. ScrewNet · p. 3
- The configuration loss compares rotations and displacements in ways intended to keep predictions consistent with screw-displacement geometry.The rotation loss uses rotation-matrix properties, while the displacement loss compares displacements along the target and predicted axes.Source: B. ScrewNet · p. 3
- Training images are rendered in Mujoco with random frame skipping and pixel dropping to simulate real-world sensor noise.The training data draws on simulated object classes and PartNet-Mobility categories, including revolute and prismatic examples.Source: B. ScrewNet · pp. 3–4
- Training labels are relative screw displacements between temporally separated poses, expressed in a frame attached to the base object.The configurations remain unchanged when the screw axis is transformed into that frame.Source: B. ScrewNet · p. 4
V. EXPERIMENTS
The experiments assess generalization to unseen instances, novel objects within familiar articulation categories, cross-category estimation, and transfer from simulation to real objects. Inputs are segmented depth images, and the baseline is adapted to use the same object-centric frame.
V. EXPERIMENTS
- The evaluation covers unseen instances from trained classes, novel objects in familiar categories, cross-category estimation, and simulation-to-real transfer.ScrewNet is compared with the method of Abbatematteo et al.; real-world tests use three objects.Source: §V · p. 4
- Inputs are assumed to be semantically segmented depth images containing only the two objects, and predictions use an object-centric frame centered on an object's bounding box.For a fair comparison, the baseline is modified to predict parameters in the same object-centric frame.Source: §V · p. 4
- Training labels are generated from screw displacements between temporally displaced object poses and expressed in the base object's frame.Figure 3 illustrates this label-generation procedure.Source: Figure 3 · p. 4
A. Same object class
For unseen instances of trained object classes, ScrewNet estimates axis position or displacement and observed configurations better than the baseline on both datasets, while its orientation errors remain higher but low.
A. Same object class
- <5° orientation errors on both datasets; ScrewNet beat baseline on joint-axis position and observed configurations on the first, and axis displacement and configurations on PartNet-Mobility, while baseline orientation errors were lower.The evaluation used 10,000 training examples per object class and 1,000 withheld instances.Source: A. Same object class · p. 4
B. Same articulation model category
ScrewNet generalizes across object classes within an articulation category and across articulation categories. Its cross-category results are competitive with class-specific baselines despite using half the training examples.
B. Same articulation model category
- The same-category tests trained revolute models on ovens and microwaves before testing dishwashers, and prismatic models on multi-column before single-column storage furniture.The reported errors cover joint-axis orientations, positions, and configurations for 1,000 test objects per class.Source: B. Same articulation model category · p. 4, Figure 5 · p. 5
- For unseen classes within a fixed articulation category, ScrewNet generalized where the baseline failed, with baseline joint-axis-position errors an order of magnitude higher for both categories.Evaluation used PartNet-Mobility because the simulated dataset lacked enough same-category object-class variety.Source: B. Same articulation model category · pp. 4–5, Figure 5 · p. 5
C. Across articulation model category
- On PartNet-Mobility, ScrewNet had lower joint-axis-position and configuration errors but higher orientation errors; on the first dataset, cabinet orientation error was also higher.The paper attributes the cabinet orientation result as possibly stemming from left- and right-opening configurations differing by 180° in axis orientation.Source: C. Across articulation model category · p. 5, Figure 6 · p. 6
- Across articulation categories, ScrewNet matched or beat the class-specific baseline across object categories and achieved better average performance while using half the training examples.The cross-category experiment used 5,000 examples per object class in the mixed training set.Source: C. Across articulation model category · pp. 5–6, Figure 6 · p. 6
- Across the ablations, ScrewNet outperformed the L2-error and two-images versions; NoLSTM was comparable overall but failed to generalize reliably across categories.NoLSTM sometimes predicted NaNs and had higher errors on drawers and across all classes in PartNet-Mobility.Source: C. Across articulation model category · p. 6
D. Real world images
Trained only on simulated data, ScrewNet produced reasonable articulation estimates for real-world objects. Its estimates indicate that retraining on real-world data would be needed for better performance.
D. Real world images
- With simulation-only training, ScrewNet’s real-world estimates had mean errors of ∼20° in axis orientation and ∼1.5cm in axis position.The estimates were for a microwave, drawer, and toaster oven; Fig. 7 shows microwave poses.Source: D. Real world images · p. 6, Figure 7 · p. 6
- Fig. 7 overlays ground-truth joint axes in blue and predicted axes in red for different microwave poses.The figure provides a qualitative view of the real-world predictions.Source: Figure 7 · p. 6, D. Real world images · p. 6
- Although simulation-only training yielded reasonable real-world estimates, better performance would require retraining on real-world data.Source: D. Real world images · p. 6
VI. CONCLUSION
ScrewNet uses screw theory to estimate articulation models and parameters from depth images without prior knowledge of the articulation category. The paper reports successful cross-category estimation with better average performance than baselines while using half the training data.
VI. CONCLUSION
- ScrewNet estimates articulation models from depth images without requiring prior knowledge of the model category, and the reported results show better average performance than baselines using half the training data.Source: §VI · p. 6
- At present, ScrewNet predicts only 1-DOF articulation models.For multi-DoF objects, the described procedure requires segmentation to mask non-relevant parts and can be repeated across object-part pairs.Source: §VI · p. 6
- Future work would directly estimate multi-DoF objects by learning segmentation with ScrewNet and predict parameters in the robot’s camera frame.The paper states camera-frame predictions can help robots learn articulation models through active learning.Source: §VI · p. 6
APPENDIX
The experiments test ScrewNet on simulated and PartNet-Mobility objects, both within object classes and on novel classes sharing an articulation category. ScrewNet improves joint-axis position and configuration estimation over the baseline, and generalizes to novel same-category classes where the baseline fails.
1) Dataset:
- The simulated dataset covers cabinets, drawers, microwaves, and toasters; PartNet-Mobility covers dishwashers, ovens, microwaves, and single- and multi-column drawers.PartNet-Mobility includes 11 microwaves, 36 dishwashers, 9 ovens, 26 single-column drawers, and 14 multi-column drawers; train/test splits are specified for each class.Source: 1) Dataset · p. 9, Figure 8 · p. 9, Figure 9 · p. 9
2) Experiment 1: Same object class:
- For both datasets, ScrewNet outperforms the baseline on joint-axis position and observed configurations, while the baseline has stronger orientation accuracy: 0.08° versus ScrewNet’s <0.5° on most simulated classes and <2.5° on PartNet-Mobility.On simulated cabinets, ScrewNet’s mean orientation error is approximately 5°, which the passage describes as small relative to the 180° difference between opening configurations.Source: 2) Experiment 1: Same object class · p. 9
3) Experiment 2: Same articulation model category:
- On novel object classes within the same articulation category, ScrewNet generalizes while the baseline fails; their orientation and configuration errors are comparable, but baseline joint-axis position errors are an order of magnitude higher.The comparison uses PartNet-Mobility test objects.Source: 3) Experiment 2: Same articulation… · p. 9
4) Ablation studies:
The ablation studies compare alternative loss, recurrent-layer, and image-sequence choices. Tables IV and V report errors across the simulated and PartNet-Mobility datasets, but the supplied passages do not state the ablation outcomes.
4) Ablation studies:
- Three ablations test raw L2 loss, replacing the LSTM with a parameter-matched fully connected layer, and estimating articulation from two images instead of a sequence.Each version, including ScrewNet, uses a single network.Source: 4) Ablation studies · p. 9
- Tables IV and V compare mean joint-axis orientation, joint-axis position, and configuration errors across 1,000 test objects per class in the simulated and PartNet-Mobility datasets.The captions note that the baseline uses a separate network for each object class, unlike ScrewNet and its ablations, which use one network.Source: Table IV · p. 11, Table V · p. 12