PaperlayerView exact revision on arXiv

Source-linked AI summary

GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion

Cheng Chi, Shuran Song

arXiv:2104.05177v2cs.CVcs.LGcs.RO

TL;DR

Garments’ near-infinite degrees of freedom, self-occlusion, and thin geometry make full configuration estimation from partial observations challenging. GarmentNets completes garment shape in a shared canonical space and uses a winding number field representation. The authors report generalization to unseen instances and significantly better performance than alternative approaches.

  • Problem

    Garments have near-infinite degrees of freedom, and their severe self-occlusion and thin, non-watertight geometry make full configuration estimation from partial observations challenging.

    Source: §1 · p. 1, §3 · p. 2

  • Method

    GarmentNets maps observed garment surfaces to a category-level canonical space, completes the shape there, and represents thin cloth with a winding number field.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    GarmentNets generalizes to unseen garment instances in real-world and simulated images and achieves significantly better performance than alternative approaches.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The approach enables category-level full-configuration estimation from partial observations, including for novel garment instances and real-world images.

    Source: §1 · p. 2

    Training requires substantial high-quality data with detailed labels, including pre-aligned 3D meshes and dense correspondence labels that are difficult to obtain for real-world data.

    Source: §6 · p. 8

Abstract

from arXiv · show

This paper tackles the task of category-level pose estimation for garments. With a near infinite degree of freedom, a garment's full configuration (i.e., poses) is often described by the per-vertex 3D locations of its entire 3D surface. However, garments are also commonly subject to extreme cases of self-occlusion, especially when folded or crumpled, making it challenging to perceive their full 3D surface. To address these challenges, we propose GarmentNets, where the key idea is to formulate the deformable object pose estimation problem as a shape completion task in the canonical space. This canonical space is defined across garments instances within a category, therefore, specifies the shared category-level pose. By mapping the observed partial surface to the canonical space and completing it in this space, the output representation describes the garment's full configuration using a complete 3D mesh with the per-vertex canonical coordinate label. To properly handle the thin 3D structure presented on garments, we proposed a novel 3D shape representation using the generalized winding number field. Experiments demonstrate that GarmentNets is able to generalize to unseen garment instances and achieve significantly better performance compared to alternative approaches.

1. Introduction

Garment pose estimation must recover a near-infinite-dimensional configuration from partial observations despite severe self-occlusion and thin, non-watertight geometry. GarmentNets addresses these challenges through category-level canonical-space completion and a winding-number representation.

1. Introduction

  • Prior garment-perception methods often assume full visibility, a known instance mesh, or known initial-state information, limiting generalization to unseen instances.These assumptions can reduce the task to instance-level tracking.Source: §1 · p. 1
  • A category-specific normalized coordinate space, defined using a canonical human pose, supports semantic correspondences across garment styles, shapes, and configurations.This offers a shared representation for garments whose configurations have near-infinite degrees of freedom.Source: §1 · pp. 1–2
  • Completing the observed garment in canonical space allows full configuration estimation despite an incomplete visible surface.The approach targets severe self-occlusion, especially in folded or crumpled garments.Source: §1 · pp. 1–2
  • The proposed winding number field represents thin cloth with strong surface gradients and smooth, continuous values elsewhere.Garments are thin and often non-watertight, unlike the solid shapes targeted by typical occupancy-grid or signed-distance representations.Source: §1 · pp. 1–2
  • Experiments show GarmentNets generalizes to novel garment instances and real-world images.The authors report category-level full-configuration estimation from partial observations.Source: §1 · p. 2

2. Related Work

Prior work addresses rigid and articulated pose estimation, shape completion, instance-level cloth perception, and on-body reconstruction, but each leaves challenges for crumpled, deformable garments. GarmentNets uses category-level semantic priors and canonical-space completion to address these gaps.

Pose estimation

  • Rigid-object category pose methods use normalized canonical spaces, but garments’ near-infinite degrees of freedom violate their piecewise-rigid assumptions.Articulated-object methods add canonical joint configurations but remain unsuitable for garments.Source: §2 · p. 2

Shape completion

  • Rigid-object completion representations do not directly address garments’ high intra-class variation and thin geometry; the paper proposes a winding number field for these concerns.Prior representations include occupancy grids, distance functions, point clouds, and implicit functions.Source: §2 · p. 2

Instance-level cloth perception

  • Instance-level reconstruction can miss regions occluded throughout a sequence, whereas GarmentNets uses category-level semantic priors to infer occluded cloth.Earlier pose-estimation work also assumes known initial states, instance meshes, or visual markers; this approach instead uses simple robot interactions.Source: §2 · p. 2

On-body cloth perception

  • On-body reconstruction models can fail to express highly crumpled garments, while canonical-space completion is described as more robust to large deformation and occlusion than task-space completion.The cited alternatives include body-model-based reconstruction, registration without completion, and task-space completion.Source: §2 · p. 2

3. Approach

GarmentNets estimates garment pose by mapping partial observations into a shared category-level canonical space, completing the geometry there, and warping the completed mesh back to task space. Its approach combines robot-assisted visibility, canonical-coordinate feature scattering, and winding-number shape completion.

3. Approach

  • The approach formulates garment pose estimation as canonical-space shape completion, producing a completed mesh whose vertices carry canonical-coordinate labels.The shared canonical space provides semantically meaningful correspondence across garment instances.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1. Pick First then Recognize

  • The robot lifts and shakes a crumpled garment, lets gravity settle it, then captures RGB-D observations from the gripper frame.The pick-first setup reduces the configuration space and increases surface visibility.Source: §3.1 · p. 3

3.2. Normalized Canonical Space for Garments

  • Garments are normalized into a category-level canonical space based on a human T-pose, with each category’s largest bounding-box dimension scaled to a unit cube.A PointNet++ network predicts per-point coordinates using independent 64-bin classifications, which handle symmetry-related bimodal predictions better than regression.Source: §3.2 · pp. 3–4

3.3. Shape Completion with Winding Number

  • Winding number fields represent thin, non-watertight cloth surfaces while distinguishing openings that TSDF zero-crossings incorrectly include as surfaces.The network predicts the field from query points and a dense feature volume, then uses the field’s spatial-gradient magnitude to distinguish surface from openings.Source: §3.3 · pp. 4–5, Figure 4 · p. 4

3.4. Canonical to Task Space Mapping

  • A learned implicit warp field maps each completed canonical-space mesh vertex back to its task-space location.The alternative physical-simulation mapping performs worse quantitatively and assumes known physical parameters.Source: §3.4 · p. 5

4. Evaluation

Evaluation uses simulated garments with instance-disjoint splits and measures both surface reconstruction and pose correspondence. Qualitative examples show predictions on unseen simulated garments at each stage.

4. Evaluation

  • Training, validation, and test sets are disjoint by garment instance; six CLOTH3D categories are each simulated 21 times with randomly sampled gripping points.Blender generates RGB-D images, UV maps, object masks, and ground-truth images from canonical-pose meshes.Source: §4 · p. 5
  • Figure 5 shows input and stage outputs for unseen simulated garments, with ground-truth and predicted grasp points marked as red spheres.For Dress and Trousers, the predicted grasp point is on the wrong side, but the final pose is still predicted correctly through warp-field prediction.Source: Figure 5 · p. 6
  • Symmetric Chamfer Distance (Dc) measures surface accuracy and completeness, while Correspondence Distance (Dn) measures pose-estimation accuracy using nearest canonical-label correspondences.Dc is measured in task space for Table 1 and canonical space for Table 2; both metrics use 10k sampled points and a KD-tree.Source: §4 · p. 6

5. Experimental Results

Experiments compare alternative pose-estimation approaches, canonical-coordinate classification and regression, shape representations, and ways to map completed meshes back to task space. Results include improved shape-completion metrics and qualitative generalization to unseen simulated and real-world garments.

Qualitative Results

  • Figures 5 and 10 show qualitative GarmentNets results on unseen garments for simulated and real data; following sections discuss qualitative results and ablation studies.Source: §5 · p. 6

Comparison to Alternative Approaches

  • Nearest-neighbor retrieval has comparable Chamfer distance (Dc) but significantly higher pose error (Dn), while canonical-space completion outperforms direct task-space completion.Similar crumpled surfaces can hide configuration mismatch; canonical space supplies a garment-configuration-invariant shape prior.Source: §5 · p. 6, Table 1 · p. 7, Figure 6 · p. 7

NOCS Prediction: Classification v.s. Regression

  • 0.14 v.s. 0.16 (in NOCS space) and 0.06 v.s. 0.11 if we consider symmetry: classification yields lower NOCS prediction error than regression.For selected left-sleeve points, classification predicts a bimodal distribution, whereas regression predicts coordinates near the mean of two possible locations.Source: §5 · pp. 6–7, Figure 7 · p. 7

Shape Representation

  • 26.7% over TSDF, 18.7% over TDF, and 32.2% over occupancy grid: the winding number field improves shape-representation performance overall.It predicts garment openings; TDF performs better on thin strips in Tops, which the winding number field tends to miss.Source: §5 · p. 7, Table 2 · p. 7, Figure 8 · p. 8

Learned Warp Field v.s. Physics Simulation

  • The learned warp field often yields more accurate task-space mappings than physics simulation, correcting NOCS prediction errors using raw task-space point coordinates.Figure 9 illustrates self-correction of mirrored gripping-point predictions in both examples.Source: §5 · p. 7, Table 3 · p. 8, Figure 9 · p. 8

Testing with Realworld Data

  • Models trained on simulated data complete geometry and estimate pose for unseen real-world garments in grasped states captured with a UR5 and iPhone 12 Pro Max.The real-world garments in this experiment were not included in training; Figure 10 shows qualitative results.Source: §5 · p. 8, Figure 10 · p. 8

6. Conclusion

GarmentNets estimates garment pose from partial point clouds and generalizes to unseen simulated and real-world instances. Its real-world training remains constrained by the need for costly, detailed labels.

6. Conclusion

  • GarmentNets estimates category-level garment pose from partial point clouds and generalizes to unseen instances in both simulated and real-world images.The authors report significantly better performance than alternative approaches.Source: §6 · p. 8
  • Training requires substantial high-quality data with detailed labels, including pre-aligned 3D meshes and dense correspondence labels, which are difficult to obtain for real-world data.The authors identify self- or weakly supervised training on real-world data as future work.Source: §6 · p. 8

A. Network Architecture Details

The architecture predicts canonical coordinates per point, scatters features into a volume for completion, and uses MLPs to predict the winding number field and warp field.

Canonical Coordinate Prediction Network

  • A PointNet++ network predicts per-point canonical coordinates as 3 × 64-way classification using cross-entropy loss.It uses three Set Abstraction layers and three Feature Propagation layers, followed by a three-layer MLP.Source: A. Network Architecture Details · p. 11, Table 4 · p. 11

Feature Completion Network (3D CNN)

Shape Completion Network

Warp Field Network

B. Additional Results

Additional results evaluate category classification, show correspondence error tracking canonical-coordinate error, and present real-world and simulated outputs alongside failure cases.

Additional results

  • The unseen-instance examples include real-world point clouds collected with an iPhone 12 Pro Max and simulated results.Figures 12 and 13 show the real-world and simulation results, respectively.Source: B. Additional Results · p. 11

Garment Category Classification

  • The input category is assumed known, with an RGB-image classifier proposed for mixed garment piles.The classifier uses an ImageNet-pretrained ResNet-50 backbone and a three-layer MLP for six-way classification.Source: B. Additional Results · p. 11
  • 93.85% test accuracy is achieved by majority-ensembling predictions from four independently classified views.The confusion matrix is reported in Table 5.Source: B. Additional Results · p. 11, Table 5 · p. 11

Failure mode analysis

  • Simulation failures include missing shoulder faces from low-sharpness winding-number predictions, over-smoothed warp fields, and a missing Dress strap.The strap failure is attributed to the winding number field's inability to represent wire-like structures.Source: B. Additional Results · p. 11

Error distribution and correlation

  • Correspondence error is highly correlated with canonical-coordinate error, suggesting joint optimization may improve performance.Figure 11 plots correspondence distance against canonical-coordinate prediction error for Dress and shows the correspondence-error histogram.Source: B. Additional Results · p. 11, Figure 11 · p. 11

C. Limitations and Future Work

GarmentNets shows real-world results despite training only on synthetic data, but end-to-end training is blocked by gradient-flow limitations and costly real-world correspondence annotation. The section also presents qualitative results and failure cases on unseen garments.

C. Limitations and Future Work

  • End-to-end training is prevented because gradients cannot propagate from shape completion and warp-field prediction to canonical-coordinate prediction.The authors note that correspondence error is highly correlated with canonical-coordinate error, suggesting joint optimization might improve performance, but manually defining dense real-world correspondences is expensive.Source: C. Limitations and Future Work · p. 12
  • Real-world results are reported even though GarmentNets is trained only on synthetic data.Figure 12 presents qualitative results on unseen real-world garment instances.Source: C. Limitations and Future Work · p. 12, Figure 12 · p. 13
  • Qualitative results are also shown for unseen garment instances in simulation, including failure cases.Figures 13 and 14 show simulation results and failure cases, respectively.Source: Figure 13 · p. 14, Figure 14 · p. 15

Related papers

Loading 2104.05177v2…