PaperlayerView exact revision on arXiv

Source-linked AI summary

ClothesNet: An Information-Rich 3D Garment Model Repository with Simulated Clothes Environment

Bingyang Zhou, Haoyu Zhou, Tianhai Liang, Qiaojun Yu, Siheng Zhao, Yuwei Zeng, Jun Lv, Siyuan Luo, Qiancai Wang, Xinyuan Yu, Haonan Chen, Cewu Lu, Lin Shao

arXiv:2308.09987v1cs.ROcs.AIcs.CV

TL;DR

Robotic learning for clothes-related tasks faces high-dimensional state representations and complex dynamics, while direct real-world training can be costly and unsafe. ClothesNet combines annotated 3D garment models with perception benchmarks and simulated manipulation environments. The paper reports perception results, simulation tasks, and a real-world dual-arm folding experiment.

  • Problem

    Clothes robotics is challenging because of high-dimensional state representations and complex dynamics, and direct real-world training can be costly and unsafe.

    Source: §1 · p. 1

  • Method

    ClothesNet provides 4,400 annotated 3D garment meshes across 11 categories, benchmarks perception tasks, and develops simulated manipulation environments using DiffClothAI.

    Source: §1 · p. 2

  • Results

    The paper establishes benchmarks for clothes classification, boundary line segmentation, and keypoint detection, develops four simulated manipulation tasks, and demonstrates real-world dual-arm folding.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ClothesNet is presented as a resource for computer vision and robot interaction tasks, with a real-world dual-arm folding experiment demonstrating its usefulness.

    Source: §5 · p. 9

    In the hanging task, performance is sensitive to small perturbations and decreases when agent training continues.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

We present ClothesNet: a large-scale dataset of 3D clothes objects with information-rich annotations. Our dataset consists of around 4400 models covering 11 categories annotated with clothes features, boundary lines, and keypoints. ClothesNet can be used to facilitate a variety of computer vision and robot interaction tasks. Using our dataset, we establish benchmark tasks for clothes perception, including classification, boundary line segmentation, and keypoint detection, and develop simulated clothes environments for robotic interaction tasks, including rearranging, folding, hanging, and dressing. We also demonstrate the efficacy of our ClothesNet in real-world experiments. Supplemental materials and dataset are available on our project webpage.

1. Introduction

ClothesNet addresses the need for large-scale clothing assets and simulation environments for robot learning, alongside challenges from clothes’ high-dimensional states and complex dynamics. It combines an annotated 3D garment dataset with perception benchmarks, simulated manipulation tasks, and real-world folding experiments.

1. Introduction

  • A dual-arm robot folds clothes in a real-world experiment, demonstrating the dataset’s usefulness.The paper also reports comprehensive experiments in simulation and real-world settings.Source: §1 · p. 2
  • ClothesNet contains 4,400 3D mesh models across 11 categories, annotated with clothes features, boundary lines, and keypoints.The authors describe it as the first large-scale dataset with rich annotations for clothes-centric robot vision and manipulation, to their knowledge.Source: §1 · p. 2
  • The authors benchmark clothes classification, boundary line segmentation, and keypoint detection to demonstrate the dataset’s usefulness.The introduction also identifies experiments in simulation and the real world.Source: §1 · p. 2
  • The paper develops simulated clothes manipulation tasks, including folding, hanging, rearranging, and dressing, using DiffClothAI.Clothes activities pose robotics challenges due to high-dimensional state representation and complex dynamics, while direct real-world training can be costly and unsafe.Source: §1 · pp. 1–2

2. Related Work

Related work spans garment datasets, differentiable cloth simulation, task suites, clothes perception, and robotic manipulation. The discussion positions ClothesNet and its simulated environments alongside limitations and capabilities of prior resources and methods.

2. Related Work

  • The related-work review covers 3D clothes datasets, simulation task suites, robotic perception, and manipulation, comparing them with the paper’s work.The authors identify these areas as the scope of their review.Source: §2 · p. 2

2.1. 3D Garment Datasets

  • ClothesNet Asset contains 4,400 mesh models across 11 categories, with a subset processed for differentiable cloth simulation.The paper notes that this processing transforms static clothes meshes into deformable clothes; prior scanned models may be unsuitable for deformation-heavy robotic simulation because of representation or mesh quality.Source: §2.1 · p. 2

2.2. Differentiable Cloth Simulations

  • DiffClothAI supports intersection-free frictional contact and two-way coupling between cloth and articulated bodies, unlike several earlier simulators.DiffPD and DiffCloth lack two-way cloth-rigid-body coupling, while Qiao et al.’s simulation supports coupling with rigid bodies but not articulated bodies.Source: §2.2 · p. 2

2.3. Clothes Simulation Task Suite

  • The authors build folding, hanging, rearranging, and dressing environments on DiffClothAI, extending the range of cloth-focused simulated tasks described in related work.The reviewed suites include SoftGym tasks for ropes and a rectangular cloth, while AssistiveGym includes a hospital-gown dressing task.Source: §2.3 · p. 3

2.4. Robotic Perception for Clothes

  • ClothesNet annotates garment attributes, boundary lines, and keypoints, building on work on feature lines and 3D keypoint detection.The paper describes boundary edges such as necklines, cuffs, and waists as potential clues for identifying garments and grasping or insertion tasks.Source: §2.4 · p. 3

2.5. Robotic Manipulation for Clothes

  • Prior robotic manipulation work includes garment rearrangement, grasp-point search, bimanual folding, and dressing controllers; the authors position their simulated environments as facilitating cloth-related manipulation learning.The reviewed work also includes wrapping fabric around a cylinder and haptic-feedback control for robot-assisted dressing.Source: §2.5 · p. 3

3. ClothesNet

ClothesNet contains around 4,400 textured 3D garment meshes across 11 categories, with category, feature, boundary-line, and keypoint annotations. ClothesNetM is a simulation-ready subset, and the section describes how its meshes and physical materials support cloth simulation.

3. ClothesNet

  • ClothesNet contains around 4,400 textured mesh models across 11 categories: tops, dresses, gloves, masks, scarf/ties, skirts, socks, hats, one-piece garments, trousers, and underpants.The models are cleaned and converted to triangular meshes; disconnected components are merged during processing.Source: §3 · pp. 3–4
  • ClothesNetM contains 3,051 models that can be loaded into DiffClothAI, with connected triangle meshes and no non-manifold edges to support large-deformation simulation.The mesh conditions address splitting during large deformation and the solver’s need for a single dihedral-angle constraint.Source: §3 · pp. 4–5
  • The dataset annotates garment categories, selected design features, open boundary lines, and keypoints.Feature tags cover sleeve, hood, gallus, collar, and garment-length attributes; keypoints are generated with Skeleton Merger on sampled surface points.Source: §3 · p. 5
  • Boundary annotations mark open edges such as necklines, cuffs, and waistlines, which provide cues for garment recognition and manipulation.Vertices on boundary edges are labeled using a filter.Source: §3 · p. 5
  • Keypoints are produced by sampling mesh-surface points and applying the self-supervised Skeleton Merger algorithm to the resulting point clouds.The section motivates keypoints as an effective representation for clothes-related tasks given the high-dimensional state of clothes.Source: §3 · pp. 5–6
  • Different physical-material parameters produce different simulated garment deformation; sweatpants deform more than jeans under the illustrated waistband grasps and gravity.The differentiable simulator also supports automatically updating physical parameters.Source: §3 · p. 6, Figure 4 · p. 6

4. Tasks and Benchmarks

ClothesNet benchmarks clothing classification, boundary segmentation, and keypoint detection, and supports simulated manipulation tasks plus real-world evaluation. Results include strong classification and segmentation scores, learned keypoints, and real-world classification accuracy.

4.1. Classification and Segmentation

  • ResNet50 achieves 93.8% classification accuracy on 2D pictures rendered from 2,455 3D garment meshes; PointNet and PointNet++ achieve 80.4% and 87.5% classification accuracy, respectively.The 2,455 meshes were rendered into 9,820 images at four camera poses; 80% of the images were used for training and 20% for testing.Source: §4.1 · p. 6
  • 0.724 mIoU for PointNet and 0.797 mIoU for PointNet++ on boundary-line segmentation; the authors report that the models learn to identify annotated boundaries.Each mesh is represented by 2048 sampled surface points, labeled according to proximity to an annotated boundary line.Source: §4.1 · p. 6, Table 2 · p. 7

4.2. Keypoint Detection

  • Skeleton Merger predicts ten keypoints per garment and produces visualizations the authors describe as reasonable.The detector uses PointNet++ for point-cloud processing.Source: §4.2 · p. 6

4.3. Manipulation Tasks

  • Folding targets a desired garment configuration, while rearrangement targets placing a garment in a washing machine; both use the folding task's state and action spaces.Folding uses 20 observed vertices and eight control vertices; the task is complete when the observed vertices are within 3 centimeters of their targets.Source: §4.3 · pp. 7–8
  • Hanging controls seven garment vertices, and performance decreases with continued training because the task is sensitive to small perturbations; dressing targets putting a garment on a human model.The hanging task uses vertex positions and velocities as the reinforcement-learning state.Source: §4.3 · p. 8
  • The simulated environments support rearranging, folding, hanging, and dressing, with differentiable cloth simulation coupled to articulated rigid bodies.These tasks include long-horizon dressing, which requires complex planning.Source: §4 · p. 6, §4.3 · pp. 7–8

4.4. Real-world Experiments

  • In real-world tests, classification accuracy is 82% for 50 Internet images and 98% for 50 Deep Fashion3D point clouds; boundary segmentation is visualized as reasonable.The folding experiment uses a dual-arm Kinova MOVO, an RGB-D point cloud, color segmentation, and detected keypoints to guide grasping.Source: §4.4 · pp. 8–9, Figure 10 · p. 9

5. Conclusion

ClothesNet combines a large annotated 3D clothing dataset with perception benchmarks and simulated robot-interaction environments. The paper also reports real-world experiments demonstrating the dataset's effectiveness.

5. Conclusion

  • ClothesNet contains about 4400 3D clothing models across 11 categories, annotated with clothes features, boundary lines, and keypoints.The authors present it as a dataset for computer-vision and robot-interaction tasks.Source: §5 · p. 9
  • The paper establishes benchmark tasks for clothes classification, boundary-line segmentation, and keypoint detection.Source: §5 · p. 9
  • The paper develops simulated environments for rearranging, folding, hanging, and dressing, and reports real-world experiments demonstrating ClothesNet's effectiveness.Source: §5 · p. 9

Related papers

Loading 2308.09987v1…