Source-linked AI summary
FoldNet++: a Large-Scale Synthetic Dataset for Robotic T-Shirt Folding and Unfolding
Yuxing Chen, Zhiyuan Wei, Bowen Xiao, Zhizheng Zhang, He Wang
TL;DR
Garment policies must handle highly deformable cloth and long-horizon tasks, while real demonstrations are costly and prior synthetic pipelines often assume flattened garments. FoldNet++ builds a large synthetic dataset with keypoint-based, rule-generated demonstrations across embodiments. Synthetic-only policies achieve over 90% real-world end-to-end success from arbitrary initial states, while the dataset remains limited to T-shirts and a single folding style.
Problem
Garment policies must handle highly deformable cloth and long-horizon subtasks, while real demonstrations are expensive and prior synthetic pipelines often assume flattened garments.
Source: §1 · p. 1Method
FoldNet++ builds a large synthetic T-shirt folding and unfolding dataset using semantic keypoints and rule-based demonstrations across robot embodiments.
Source: §1 · p. 2Results
Over 90% end-to-end task success rates were achieved by synthetic-only policies in real-world evaluations from arbitrary initial garment states, comparable to prior large-scale-real-data approaches.
Source: §1 · p. 2Takeaways & Limitations
Models trained on FoldNet++ transfer from simulation to real-world settings and generalize across unseen embodiments, T-shirts, and environments.
Source: §5 · p. 9The dataset currently covers only T-shirts and a single folding style; long sleeves and features such as zippers and buttons remain open challenges.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Due to the highly deformable nature of garments, training a generalizable policy for robotic T-shirt folding and unfolding remains a significant challenge. In this work, we present a large-scale synthetic dataset for robotic T-shirt folding and unfolding, covering 6 robotic embodiments, 1K T-shirts, 1K environmental assets, and 120K episodes with rich annotations, which can be used to train a wide range of manipulation policies. We first follow the FoldNet pipeline to generate a large-scale dataset of physically simulatable T-shirts with diverse appearances and annotated semantic keypoints. Based on these semantic keypoints, we then generate manipulation demonstrations for different robotic embodiments through a unified rule-based framework. We use these demonstrations to train visuomotor policies, and experimental results demonstrate that models trained solely on our synthetic data can achieve over 90\% end-to-end task success rates when directly deployed to unseen real-world environments and previously unseen T-shirts from arbitrary initial configurations. Project URL: https://pku-epic.github.io/FoldNetXX/.
1 Introduction
Garment manipulation is difficult because cloth deforms substantially and tasks involve long sequences, while available real and synthetic demonstrations have important scale and initial-state limitations. FoldNet++ provides a large synthetic dataset and rule-based demonstrations, with synthetic-only policies achieving over 90% real-world task success from arbitrary initial states.
1 Introduction
- Real-world demonstration collection is expensive and time-consuming, while prior synthetic pipelines typically begin with flattened garments rather than arbitrary, highly crumpled states.Changes in robot platforms, environments, or target garments typically require new datasets.Source: §1 · pp. 1–2
- FoldNet++ contains 6 robot embodiments, 1K T-shirts, 1K diverse environments, and 120K annotated episodes for folding and unfolding.Its unfolding component can flatten T-shirts for downstream tasks such as hanging or ironing.Source: §1 · p. 2
- The dataset uses semantic-keypoint T-shirt meshes and a rule-based framework to generate folding and unfolding demonstrations across robot platforms from arbitrary initial configurations.The mesh-generation method is adopted from [6].Source: §1 · p. 2
- Over 90% end-to-end task success rates were achieved by synthetic-only policies in real-world evaluations from arbitrary initial garment states, comparable to prior large-scale-real-data approaches.Evaluations covered multiple policy architectures and in-distribution and out-of-distribution embodiments under zero-shot and few-shot conditions.Source: §1 · p. 2
- Compared with FoldNet, FoldNet++ adds unfolding-to-folding from crumpled states, a six-embodiment dataset, and evaluations of cross-embodiment transfer, policy categories, and real-data fine-tuning.Source: §1 · p. 3
2 Related Works
Prior garment-manipulation work includes action primitives, dynamics-based planning, and learned policies, but faces challenges in fine-grained control, model accuracy, data cost, and generalization. FoldNet++ addresses these gaps with synthetic-data policies and a higher-fidelity simulation and demonstration pipeline.
2.1 Garment Manipulation
- Action primitives can offer efficient training and stable control but struggle with fine-grained interactions, while dynamics-based planning is sensitive to modeling inaccuracies and simulation errors.The cited approaches include [14] [15] [16] [17] [18] [19] [20] and [21] [22] [23] [24].Source: §2.1 · p. 3
- End-to-end policies trained on real-world datasets can require costly collection and generalize poorly to unseen settings, whereas FoldNet++ reports strong generalization from synthetic data while retaining fine-grained, long-horizon manipulation.The prior learned-policy approaches include [3] [4], world models [26] [27], and reinforcement learning [28].Source: §2.1 · p. 3
2.2 Garment Simulation Environment
- Prior simulators have sim-to-real gaps under complex deformations, simplified folding, and limited unfolding demonstrations; FoldNet++ uses a high-fidelity simulator and scalable rule-based unfolding demonstrations.Prior synthetic pipelines and folding procedures are described in [6] [7].Source: §2.2 · p. 3
3 FoldNet++ Dataset
FoldNet++ generates demonstrations from semantic keypoints and inverse kinematics, with rules for folding flattened garments and unfolding arbitrary initial configurations. Its simulation setup combines garment and environment assets, multiple embodiments, and domain randomization.
3 FoldNet++ Dataset
- Semantic-keypoint T-shirt meshes are imported into a physics simulator, where keypoint coordinates and robot inverse kinematics support rule-based motion generation.Each annotated keypoint corresponds to a garment vertex; the dataset section then describes the robotic system, folding rules, unfolding rules, and simulation environment.Source: §3 · pp. 3–4
3.1 Robot Setup
- Robots begin from arbitrary garment configurations by grasping and releasing a random garment point from a height, allowing it to fall freely.The procedure is used in simulation and followed similarly in the real world.Source: §3.1 · p. 4
3.2 Folding Rules
- The folding sequence performs two midline folds with intervening drags, then a left-to-right fold, flip, and final drag; rules adjust for collar orientation.Alternative folding styles can be implemented by adding or removing steps, but are beyond this work's scope.Source: §3.2 · p. 4
3.3 Unfolding Rules
- Unfolding first uses a fling chosen from manipulable garment vertices to spread and flatten the garment, then pick-and-place actions align its orientation and position.The later actions move the garment toward the table center and align the collar left or right with the world frame.Source: §3.3 · p. 4
3.4 Simulation Environment
- The setup combines Style3D FEM cloth simulation, Blender rendering, 1,000 T-shirts, six embodiments, environment assets, and camera, robot-pose, and cloth-property randomization.Open-gripper state perturbations are applied at 5% of timesteps, each 0.2 seconds, to encourage reliance on visual observations.Source: §3.4 · p. 5
4 Experimental Results
The experiments test performance across robot embodiments, transfer to real-world settings, model architectures, and real-data fine-tuning. Across these evaluations, the results describe cross-embodiment transfer, zero-shot real-world deployment, and effects of different training strategies.
4.1 Experimental Setup
- Trials count as failures after 1,000 steps and use unseen evaluation backgrounds and garments; real-world success requires expert judgment of a neatly folded result.Simulation success is assessed using garment keypoints relative to a target folded configuration, a criterion reported as largely consistent with human judgment.Source: §4.1 · p. 6
4.2 Testing on Different Embodiments
- After cross-embodiment pretraining, unseen embodiments show zero-shot generalization; fine-tuning improves performance and reduces post-training data needs from 20K to approximately 5K episodes.Performance also improves as pretraining embodiments increase, though that comparison combines data scale and embodiment diversity; ARX’s lower performance than Galbot is attributed to its smaller workspace.Source: §4.2 · p. 7
4.3 Sim-to-Real Testing
- Real-world tests show zero-shot cross-embodiment capability with pretraining alone, and further gains after embodiment-specific fine-tuning.The model was deployed on Galbot and ARX without real-world fine-tuning; the out-of-distribution ARX pretraining excluded ARX.Source: §4.3 · p. 7
4.4 Comparison with Different Models
- π0 ScratchExpert performs strongly without large-scale multi-task robotic pretraining, while UWM and DP achieve strong success rates in simulation and real-world tests.The comparison uses 20K Galbot training episodes and includes representative VLA, WAM, and VA models.Source: §4.4 · p. 7
4.5 Fine-tuning with Real Data
- Uniform full-trajectory real-data fine-tuning performs worse than simulation-only training, while grasp-focused sampling improves action stability before grasp execution.The study collects 1K Galbot teleoperation trajectories and compares sampling full trajectories with sampling short pre-grasp segments; the authors attribute uniform sampling’s weaker result to inconsistent intermediate behaviors.Source: §4.5 · pp. 7–9, Table 1 · p. 9
5 Conclusion
FoldNet++ is a large-scale synthetic dataset for T-shirt folding and unfolding, with experiments validating transfer from simulation to real-world settings and generalization across unseen embodiments, T-shirts, and environments.
5 Conclusion
- FoldNet++ supports direct transfer from simulation to the real world, with strong generalization across unseen embodiments, T-shirts, and environments.Extensive experiments in simulation and real-world settings validate the dataset's effectiveness.Source: §5 · p. 9
- The conclusion presents FoldNet++ as a dataset for both T-shirt folding and unfolding.Source: §5 · p. 9
- Experiments in both simulation and real-world settings validate the effectiveness of the proposed dataset.Source: §5 · p. 9
6 Limitation
The current dataset covers T-shirts and one folding style; extending it to other garments and procedures is a future direction, while sleeve folding and fine-grained handling of zippers and buttons remain open problems.
6 Limitation
- The current dataset is limited to T-shirts and a single folding style.Source: §6 · p. 9
- The rule-based framework can be extended to other garment types and alternative folding procedures in future work.Source: §6 · p. 9
- Long sleeves require additional sleeve-folding manipulations, while zippers and buttons require fine-grained dexterous operations that remain open problems.Source: §6 · p. 9
A.1 Unfolding Rules
The unfolding rules use garment keypoints and vertex affordances to flatten arbitrary configurations and prepare for folding. The appendix also describes error-recovery demonstrations, trajectory generation, simulation settings, and the scale and cost of demonstration generation.
A.1 Unfolding Rules
- Vertex manipulability depends on whether the neighborhood grasped with a vertex lies within a threshold distance in the rest shape; stacked regions may therefore be unmanipulable.The rules use keypoints and affordance maps to flatten arbitrary garment states, then test flattening and orientation before switching to folding.Source: §A.1 · pp. 14–15, Figure 8 · p. 14
- When predefined keypoint pairs are unavailable, the rules score sampled manipulable vertex pairs by estimated post-fling area, with workspace and gripper-distance constraints.Pairs are prioritized by their ability to flatten the garment and adjust orientation; fallback pairs use the larger estimated enclosed area.Source: §A.1 · p. 15
- The cosine-shaped fling trajectory performs better than straight-line and elliptical-arc alternatives and can flatten garments under about 0.4 s of inference latency.Source: §A.1 · p. 16
A.2 KG-DAgger
- Offline KG-DAgger injects noise during data generation to teach recovery from prediction errors, and generates regrasping demonstrations.The paper reports that KG-DAgger enables reattempts after grasp failures and substantially improves model robustness.Source: §A.2 · p. 16
A.3 Trajectory Generation
- Fling trajectories use a cosine profile, while other actions interpolate linearly between end-effector poses before inverse kinematics computes joint angles.Source: §A.3 · p. 16
B.1 Physical Parameters
- The physical-parameter table uses U(a, b) to denote uniform sampling over [a, b].Source: §B.1 · p. 17
B.2 Camera Parameters
- Simulation randomizes camera intrinsics and extrinsics; camera resolutions and focal lengths vary by camera type and robot.Intrinsics are perturbed by U(−16, +16), while extrinsics vary within 0.02 m translation and 0.1 radians rotation.Source: §B.2 · p. 17
B.3 Other Parameters
- The setup randomizes table pose, initial end-effector pose, and evaluation base poses for dual-arm platforms within specified ranges.Source: §B.3 · p. 17
C.1 Shared Training Setup
The shared setup defines a 24-dimensional action space and flow-matching batch size N, while the appendix points to model-specific training hyperparameter tables for π0, UWM, and diffusion policy.
C.1 Shared Training Setup
- The shared action space has 24 dimensions: 9 per arm pose, 2 gripper actions, and 4 leg actions, with leg dimensions zero-padded for robots without legs.Source: §C.1 · p. 18
- Flow Matching Batch Size N replicates each target action within the action expert and draws N independent timestep and noise samples for the loss.Only one forward pass outside the action expert is performed per target action.Source: §C.1 · p. 18
C.2 π0 Model
- The appendix lists π0 training hyperparameters and keeps all other settings the same as in the original model.Source: §C.2 · p. 18
C.3 Unified World Model
- The appendix lists UWM training hyperparameters and keeps all other settings the same as in the original model.Source: §C.3 · p. 19
C.4 Diffusion Policy
- The appendix lists diffusion policy training hyperparameters and keeps all other settings the same as in the original model.Source: §C.4 · p. 19
C.5 Real data Finetuning
This section reports real-data fine-tuning configurations and implementation details, then describes test garments and the kinds of execution failures observed. Closed-loop recovery can address many intermediate failures, though the authors suggest more diverse training data may improve efficiency and reduce their frequency.
C.5 Real data Finetuning
- “FTRD-Uniform” and “FTRD-Grasp” are fine-tuned from “Only Sim Data” at a 1:1 synthetic-to-real sampling ratio for 2K steps at 2.5e-6 and 20K steps at 2.5e-5, respectively.Training durations were selected through empirical evaluations across multiple checkpoints and correspond to the best-performing configurations.Source: §C.5 · p. 19
D Control Detail
- IK uses Newton iterations and the end-effector Jacobian; initialization toward a reference pose improves robustness for redundant Galbot arms, while gripper actions use three discrete bins.Training-Time RTC enables asynchronous inference; gripper values are 0 (open), 0.5, and 1.0 (closed).Source: D Control Detail · p. 19, Figure 9 · p. 20
E Test Garment Details
- The tested real T-shirts measure 49–62 cm long and 60–73 cm wide; synthetic shirts follow FoldNet’s short-sleeve distribution with geometry and texture variation.No garment-specific failure pattern was observed across the tested shirts.Source: E Test Garment Details · p. 20
F Failure Mode
- Unsuccessful cases mainly involve unsafe behavior or exceeding the step limit; most suboptimal intermediate failures are temporary, and closed-loop control can attempt recovery.The authors suggest more diverse training data may improve efficiency and reduce failure frequency.Source: F Failure Mode · p. 20
G Additional Experiments
Additional experiments examine embodiment diversity, fine-tuning choices, wrist cameras, and transfer to a held-out robot. They also demonstrate simulated folding and unfolding of shorts, while noting that strong sim-to-real generalization to a new garment category requires scaling its dataset.
Fixed-data-budget ablation
- With 20K pretraining episodes fixed, success rises from 59% to 64% to 65% as embodiment count increases from 1 to 3 to 5, showing an additional diversity benefit.The authors identify overall pretraining scale as the dominant factor.Source: Table 8 · p. 21, G Additional Experiments · p. 21
Real-only baseline and fine-tuning hyperparameters
- Under an aggressive shared fine-tuning configuration, “FTRD-Uniform” reaches 4/10 success and “FTRD-Grasp” 9/10; a more conservative configuration raises “FTRD-Uniform” to 6/10.The authors attribute the lower aggressive-configuration result to simulation-to-real distribution mismatch and overfitting on limited real data.Source: Table 9 · p. 21, G Additional Experiments · p. 21
Wrist camera and physics engine ablation
- Removing wrist-mounted cameras reduces success from 96% to 58%; Style3D is necessary for valid demonstrations, but its contribution is difficult to ablate directly.PyFlex could not reliably complete the more complex unfolding-to-folding pipeline used in this task.Source: Table 10 · p. 21, G Additional Experiments · p. 21
Galbot as OOD embodiment
- 87% for Galbot versus 63% for ARX when each is held out of pretraining, under the same 20K-episode pretraining budget and target-embodiment fine-tuning.The authors suggest ARX’s smaller workspace is a primary factor in its lower performance.Source: G Additional Experiments · p. 21, Table 11 · p. 22
Extension to other garment types
- The pipeline generated simulated shorts demonstrations after one day of category-specific rule design, but strong sim-to-real generalization requires scaling the new category’s dataset.The authors leave that dataset scaling for future work.Source: G Additional Experiments · p. 22, Figure 11 · p. 22