Source-linked AI summary
SpeedFolding: Learning Efficient Bimanual Folding of Garments
Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kröger, Ken Goldberg
TL;DR
Garment folding is difficult because garments have complex dynamics and high-dimensional configuration spaces. SpeedFolding uses learned bimanual primitives to smooth and fold garments from arbitrary configurations. It achieves 30 to 40 FPH compared with prior works’ 3 −6 Folds Per Hour (FPH).
Problem
Garments’ complex dynamics and high-dimensional configuration spaces make reliable and efficient folding a long-standing robotic manipulation challenge.
Source: §I · p. 1Method
BiMaMa-Net predicts paired gripper poses for bimanual primitives that move garments from crumpled configurations toward folded goals.
Source: A. Action Primitives · p. 3, §VI · p. 8Results
30 to 40 FPH versus prior works’ 3 −6 Folds Per Hour (FPH), with 93 % success and under 120 s average folding time.
Source: §VI · p. 8Takeaways & Limitations
Additional action primitives let the robot reach sufficiently smoothed configurations beyond the fling primitive used in Ha et al..
Source: §VI · p. 8Generalization is to similar unseen garments; textile patterns may be harder to classify correctly, and generalization to different shapes may be limited.
Source: E. System Limitations · p. 7
Abstract
from arXiv · showhide
Folding garments reliably and efficiently is a long standing challenge in robotic manipulation due to the complex dynamics and high dimensional configuration space of garments. An intuitive approach is to initially manipulate the garment to a canonical smooth configuration before folding. In this work, we develop SpeedFolding, a reliable and efficient bimanual system, which given user-defined instructions as folding lines, manipulates an initially crumpled garment to (1) a smoothed and (2) a folded configuration. Our primary contribution is a novel neural network architecture that is able to predict pairs of gripper poses to parameterize a diverse set of bimanual action primitives. After learning from 4300 human-annotated and self-supervised actions, the robot is able to fold garments from a random initial configuration in under 120s on average with a success rate of 93%. Real-world experiments show that the system is able to generalize to unseen garments of different color, shape, and stiffness. While prior work achieved 3-6 Folds Per Hour (FPH), SpeedFolding achieves 30-40 FPH.
I. INTRODUCTION
Garment folding is difficult because garments have complex dynamics and high-dimensional configuration spaces, while smoothing and folding require a long sequence of steps. SpeedFolding learns bimanual actions to smooth initially crumpled garments and fold them to user-defined configurations.
I. INTRODUCTION
- The workflow unfolds with fling actions, smooths with a drag, repositions for reachability, and then executes the user-defined folds.BiMaMa-Net selects a smoothing primitive and predicts a pair of end-effector poses; folding then uses a sequence of folding primitives.Source: Figure 1 · p. 1, Figure 2 · p. 2
- SpeedFolding combines smoothing from crumpled configurations with folding, which prior work often approached as separate tasks.Garment handling is challenging because of complex configuration spaces and nonlinear deformable-object dynamics.Source: §I · p. 1
- Over 90 % success on known and unseen garments, with t-shirt folding under 120 s on average, improving baselines by 30 to 47 % and prior works by 5 to 10 ×.Unseen garments differed in color, shape, and stiffness.Source: §I · p. 1
- BiMaMa-Net predicts two corresponding planar gripper poses without spatial restrictions and uses automated calibration to account for robot reachability constraints.The system learns smoothing through self-supervision before folding along user-defined lines.Source: §I · p. 1
II. RELATED WORK
Prior garment manipulation methods often address smoothing or folding separately, while dual-arm manipulation adds planning complexity. SpeedFolding formulates the task as transferring a garment from an arbitrary observed configuration to a user-defined goal using paired planar gripper poses.
II. RELATED WORK
- Recent work mainly addressed smoothing from arbitrary configurations or folding garments assumed to be initially flattened; SpeedFolding combines smoothing and folding from crumpled starts.Dual-arm systems extend workspace and support more complex behavior, but add planning complexity from extra degrees of freedom and self-collisions.Source: §II · p. 2
- Earlier smoothing methods used feature detection or quasi-static learned actions, while Ha et al. used a 4 DoF fling that achieved ∼80 % coverage within 3 actions but had limited scope and grasp failures.The fling parameterization was limited to fling actions, did not fully smooth garments, and induced grasp failures in more than 25 % of actions.Source: §II · p. 2
- Earlier folding approaches included heuristic methods with 10 to 20 min cycle times and learned goal-conditioned policies; SpeedFolding compares folding from smoothed garments with folding from non-smooth garments using dimension priors.Source: §II · p. 2
III. PROBLEM STATEMENT
- The task is to compute and execute an action from a visual observation that transfers the garment from its current configuration to a user-defined goal invariant to workspace position and orientation.The observation assumes a calibrated overhead view and a garment distinguishable from the workspace.Source: §III · p. 2
- Each discrete action primitive is parameterized by two planar gripper poses, one per arm, and assumes a flat, obstacle-free workspace with collision-free dual-arm trajectories.The architecture maps images to a primitive type and predicts joint values for paired gripper poses.Source: §III · p. 2, Figure 3 · p. 3
IV. METHOD
SpeedFolding uses BiMaMa-Net to smooth a crumpled garment to a sufficiently smooth configuration, then applies an instruction-based folding pipeline. Its action primitives combine learned and heuristic behaviors to handle unfolding, local corrections, repositioning, and folding.
IV. METHOD
- SpeedFolding seeks a small set of quasi-static and dynamic primitives that can move arbitrary garment configurations to folded goals with fewer action steps.BiMaMa-Net performs learned smoothing before the instruction-based folding pipeline.Source: A. Action Primitives · p. 3, §IV · p. 3
A. Action Primitives
- Every primitive uses a shared grasp approach: each gripper moves 4 cm above the grasp pose, rotates 8◦, then moves 1 cm toward the higher fingertip.This motion improves grasping, particularly at garment edges.Source: A. Action Primitives · p. 3
- Fling stretches the garment to a force threshold, then flings it forward and backward; it can increase coverage in a few steps but often does not fully smooth the garment.Its motion parameters are fixed because the fling was found robust to changes in velocity and trajectory parameters.Source: A. Action Primitives · p. 3
- Pick-and-place uses one arm to move a garment point while the other presses along the pick-to-place line; drag uses workspace friction to smooth wrinkles or corners.Pick-and-place can fix local faults such as folded corners or sleeves, while drag pulls a fixed distance away from the garment-mask center.Source: A. Action Primitives · p. 3
- The fold primitive uses simultaneous bimanual pick-and-place, while move drags the garment center toward a target, often toward the robot to improve reachability.Source: A. Action Primitives · p. 3
- BiMaMa-Net classifies whether a garment is sufficiently smoothed instead of relying on coverage, which can miss wrinkles or faults that impair subsequent folding.The learned smoothness value provides the transition from smoothing to folding.Source: A. Action Primitives · p. 3
B. BiMaMa-Net for Bimanual Manipulation
BiMaMa-Net predicts paired gripper poses by combining individual pose values with learned correspondence information. Its shared image encoder and value-map decoders support selection of bimanual actions.
B. BiMaMa-Net for Bimanual Manipulation
- Unconditioned pose-value maps can have multiple peaks and lack information about correspondences between candidate poses.BiMaMa-Net addresses this by using correspondence descriptors to condition the value of pose pairs.Source: B. BiMaMa-Net for Bimanual Manipulation · p. 4
- Each pose descriptor combines a learned pixel embedding with the unconditioned value, positional encodings, and action primitive type.The final conditioned value depends on a pair of these descriptors.Source: B. BiMaMa-Net for Bimanual Manipulation · p. 4
- A shared ResNext-50 encoder predicts the manipulation primitive, while a U-Net decoder produces value maps for discrete end-effector orientations.The U-Net is chosen to support high-resolution inputs needed to detect garment edges and wrinkles.Source: B. BiMaMa-Net for Bimanual Manipulation · p. 4
- For inference, poses are sampled with probabilities emphasizing high unconditioned values, then descriptor pairs are combined to produce conditioned action values.Training samples instead allow negative examples to better estimate the distribution of action values.Source: B. BiMaMa-Net for Bimanual Manipulation · p. 4
- When pose roles must remain distinct, separate decoders predict the two value maps.Pick-and-place actions are given as an example of poses with a required relation.Source: B. BiMaMa-Net for Bimanual Manipulation · p. 4
- The network predicts a primitive and two poses at runtime, while its training pipeline iterates from manual annotations to self-supervised data collection.After execution, the robot folds if the garment is classified as sufficiently smoothed; otherwise, it repeats the process.Source: Figure 4 · p. 4
C. Reachability Calibration
Reachability calibration creates separate action-space masks for the two arms and uses them to constrain pose selection. Collision checking then rejects unsafe pose pairs.
C. Reachability Calibration
- A one-time boundary search over x, y, and orientation computes separate reachable-space masks for the left and right arms.The search follows the reachability boundary at a constant height above the table.Source: C. Reachability Calibration · p. 4
- The masks constrain BiMaMa-Net's action sampling, and masked value maps are used to ensure each arm's mask contains at least one pose.The approach forms masked maps from the arm-specific value maps and reachability masks.Source: C. Reachability Calibration · p. 4
- Orientation-specific calibrated masks significantly reduce false negatives compared with reachability approximations such as circular masks.After pose selection, the system checks for inter-arm collisions and tries the next-best action if needed.Source: C. Reachability Calibration · p. 5
D. Training for Smoothing
BiMaMa-Net learns primitive types and paired action poses from overhead images through self-supervised real-world training. The reward combines garment coverage change with confidence that the garment is sufficiently smoothed.
D. Training for Smoothing
- BiMaMa-Net is trained through self-supervised real-world learning to predict the primitive and paired action poses from an overhead garment image.The learning process is designed to scale interactions with minimal human intervention.Source: D. Training for Smoothing · p. 5
- The reward combines change in garment coverage with the classifier's Sufficiently Smoothed confidence, using tuned weights.The reward is scaled to [-1, 1] and clipped to non-negative values, so no change gives zero reward.Source: D. Training for Smoothing · p. 5
- For continuous training, the robot resets the garment by randomly grasping its mask and dropping it from a fixed height.Training and execution are interleaved, with actions explored by uniform sampling from the N_s best actions.Source: D. Training for Smoothing · p. 5
- Human annotations kickstart exploration, with fixed rewards, Gaussian-decay pose targets, and classifier training restricted to examples above a reward threshold.These adjustments distinguish annotated data from self-supervised data during training.Source: D. Training for Smoothing · p. 5
E. Folding Pipeline
SpeedFolding compares instruction-based folding, a fast t-shirt heuristic, and fling-to-fold. The instruction-based method follows user-specified lines on a smoothed garment, while fling-to-fold merges a fling and fold action.
E. Folding Pipeline
- The instruction-based method registers a template to the current image, then folds a smoothed garment along user-specified lines.The instruction comprises a binary template mask and folding lines relative to that template.Source: E. Folding Pipeline · p. 5
- Figure 7 compares normalized coverage and learned Sufficiently Smoothed classifier predictions across action steps for different baselines, while showing generalization to unseen garments with different colors, patterns, and materials.Source: Figure 7 · p. 6
- The instruction-based fold selects two pick points that maximize the area of the polygon formed with the folding-line intersections, then mirrors them across the line for placement.Gripper orientation is set using the normal at each pick point.Source: E. Folding Pipeline · p. 6
- The “2-second” heuristic uses only a few folding steps but is specific to t-shirts.It is described as a bimanual method for a smooth garment.Source: E. Folding Pipeline · p. 6
- Fling-to-fold merges the fling and first fold because they grasp the same sleeve and non-diagonal bottom corner, reducing imaging and motion time.The approach requires prior knowledge of t-shirt dimensions to adjust arm height before folding.Source: E. Folding Pipeline · p. 6
V. EXPERIMENTS
The experiments evaluated SpeedFolding’s garment smoothing and folding performance on a known t-shirt and two garments unseen during training.
V. EXPERIMENTS
A. Experimental Setup
Experiments used a single t-shirt for training and compared smoothing methods using coverage and a learned sufficient-smoothing classification. BiMaMa-Net reached that classified state in fewer steps than the baselines, while the FlingBot baseline did not reach 80% coverage in this physical setup.
A. Experimental Setup
- Training used one t-shirt and a 4,300-action dataset assembled from manual annotation, self-supervised collection, and re-annotation.The 600 initial scenes were manually annotated in 1 hour; the robot collected 2,200 actions in 16 hours, and 1,500 actions were copied and re-annotated in 3 hours.Source: A. Experimental Setup · p. 6
- Each experiment averaged 15 trials; a trial failed if folding failed by majority reviewer vote or exceeded the 10-action horizon.Motion-planning-error early terminations were excluded.Source: A. Experimental Setup · p. 6, Table I · p. 7
B. Sufficiently Smoothed
- The smoothing evaluation measured overhead-image garment coverage and a binary Sufficiently Smoothed classification from a learned classifier.Baselines included Max Value Map, Only Flings, and pretrained Flingbot.Source: B. Sufficiently Smoothed · pp. 6–7
- For Table I, fold durations average successful folds, whereas cycle time and folds per hour average successful and unsuccessful folds.Source: Table I · p. 7
- BiMaMa-Net reached a Sufficiently Smoothed configuration in ∼3 fewer steps than baselines requiring ∼5.Only Flings achieved similar coverage increases but reached the classified state later or sometimes failed to reach it.Source: B. Sufficiently Smoothed · p. 7
- FlingBot failed to reach 80% coverage in this setup, with frequent grasp failures presumed to reflect physical-setting differences.The authors also observed higher coverage when a stretching motion preceded a fling.Source: B. Sufficiently Smoothed · p. 7
C. Folds per Hour
End-to-end results favored Fling-to-fold in speed when garment dimensions were known, while generalization experiments showed adaptation to unseen garments with differing color, stiffness, and shape. The faster “2-second” fold had low success and orientation sensitivity, and towel classification required additional training examples.
C. Folds per Hour
- Instruction-based folding was 30% faster than the Max Value Map baseline, while the “2-second” fold added 10.4% speedup when successful but had low fold success and t-shirt-orientation sensitivity.The “2-second” fold is suited to t-shirts only.Source: C. Folds per Hour · p. 7, Figure 8 · p. 7
- With prior knowledge of t-shirt dimensions, Fling-to-fold used 40% fewer smoothing actions and imaging time and achieved over 25% speedup, reaching 40.9 folds per hour on average.This combined primitive does not require a smoothed state before folding, but assumes prior garment knowledge.Source: C. Folds per Hour · p. 7, Figure 8 · p. 7
D. Generalization to Unseen Garments
- Instruction-based folding generalized to a different-color, different-stiffness t-shirt without model or template changes and to a different-color rectangular towel after 20 sufficiently-smoothed towel images were added for retraining.Before those additions, the system failed to classify a sufficiently smoothed towel configuration.Source: D. Generalization to Unseen Garments · p. 7
E. System Limitations
The system’s practical limits include grasp failures during stretching and restricted generalization to similar unseen garments. The discussion reports 93% folding success in under 120 seconds on average and contrasts SpeedFolding’s 30–40 folds per hour with prior work’s 3–6.
E. System Limitations
- Grip loss, often during stretching before a fling, can sharply reduce garment coverage; force or visual feedback may mitigate this failure.Top-down grasps in the first “2-second” fold step may require different gripper jaws.Source: E. System Limitations · p. 7
- Generalization is to similar unseen garments; patterns may challenge detection, and different shapes may require additional sufficiently-smoothed examples.The paper describes data augmentation for patterns and added towel examples for shape adaptation.Source: E. System Limitations · p. 7
VI. CONCLUSION AND DISCUSSION
- SpeedFolding folded garments in under 120 s on average with 93% success after learning from 4,300 human-annotated or self-supervised actions.Source: §VI · p. 8
- Prior end-to-end cloth-folding work achieved 3–6 folds per hour, compared with SpeedFolding’s 30–40 folds per hour.The paper attributes its approach to adding primitives beyond the fling to reach a sufficiently smoothed state.Source: §VI · p. 8