Source-linked AI summary
Folding Deformable Objects using Predictive Simulation and Trajectory Optimization
Yinxiao Li, Yonghao Yue, Danfei Xu, Eitan Grinspun, Peter Allen
TL;DR
Robotic garment folding requires finding trajectories that account for deformable cloth and its interaction with the table. The paper optimizes trajectories using predictive simulation and executes them with a Baxter robot. Experiments report comparable simulated and real folding outcomes across several garment types, with performance varying by garment.
Problem
Finding effective garment-folding trajectories is challenging because deformable garments respond differently to trajectories, cloth properties, and table friction.
Source: §I · p. 1Method
The method optimizes trajectory parameters using predictive thin-shell simulation and a quadratic objective that penalizes path length and dissimilarity from the desired folded shape.
Source: §I · p. 1, A. Trajectory parametrization · p. 4Results
The Baxter robot produced real garment deformations and final folding outcomes comparable to simulation; most garments took about 1.5 minutes to fold.
Source: C. Garment manipulation and folding · p. 6Takeaways & Limitations
Optimized trajectories supported folding sweaters, pants, and towels, though performance was less stable for jeans and pants with higher surface shear resistance.
Source: §I · p. 1, C. Garment manipulation and folding · p. 6Folding was less stable for jeans and pants when surface shear resistance was relatively high, sometimes leaving small wrinkles in successful folds.
Source: C. Garment manipulation and folding · p. 6
Abstract
from arXiv · showhide
Robotic manipulation of deformable objects remains a challenging task. One such task is folding a garment autonomously. Given start and end folding positions, what is an optimal trajectory to move the robotic arm to fold a garment? Certain trajectories will cause the garment to move, creating wrinkles, and gaps, other trajectories will fail altogether. We present a novel solution to find an optimal trajectory that avoids such problematic scenarios. The trajectory is optimized by minimizing a quadratic objective function in an off-line simulator, which includes material properties of the garment and frictional force on the table. The function measures the dissimilarity between a user folded shape and the folded garment in simulation, which is then used as an error measurement to create an optimal trajectory. We demonstrate that our two-arm robot can follow the optimized trajectories, achieving accurate and efficient manipulations of deformable objects.
I. INTRODUCTION
The paper addresses garment folding as the remaining stage of robotic garment manipulation and seeks trajectories that account for differences in cloth and table conditions. It combines automatic garment-keypoint detection, predictive simulation, and robot experiments to produce folding trajectories for several garment types.
I. INTRODUCTION
- The method automatically detects garment key points, including sleeve ends, collar, and waist corners, for folding-plan generation.The introduction describes this detection algorithm as fast and robust.Source: §I · p. 1
- An optimization algorithm learns manipulation trajectories by combining mathematical-model evolution with predictive thin-shell simulation.Source: §I · p. 1
- The approach adjusts the simulation environment to the robot's working environment and scales trajectories for similar garments of different sizes.Source: §I · p. 1
- Baxter robot experiments report successful folding trajectories for sweaters, pants, and towels, with real outcomes comparable to simulation.The pipeline assumes the garment is initially flat on the table and uses a predefined folding plan based on detected key points.Source: §I · p. 1
- The paper targets garment folding, the remaining manipulation stage after prior work addressed grasping, recognition, regrasping, unfolding, and placing garments flat.Different trajectories can produce different results as cloth properties and table friction vary.Source: §I · p. 1
II. RELATED WORK
Prior garment-manipulation work addressed unfolding, shape parsing, and folding, but not trajectory optimization for folding. The paper uses simulation to model garment folding along computed trajectories.
A. Folding Pipeline in Simulation
- The folding pipeline iteratively compares the simulated garment contour with a template, then exports and adapts the resulting trajectories for a real robot.The paper's figure shows this process repeated until the garment is folded in simulation.Source: Figure 2 · p. 2
II. RELATED WORK
- Earlier work addressed garment unfolding, state recognition, shape parsing, and folding, while trajectory optimization for folding remained unaddressed.Improper trajectories can shift the garment or add deformation that accumulates.Source: §II · pp. 1–2
A. Folding Pipeline in Simulation
- The simulator assumes one garment lies flat on a virtual table and models its movement and deformation with a physics engine.Each folding step picks up part of the mesh, moves it along a Bézier-curve trajectory, and places it on the table; wide folds may use both arms.Source: A. Folding Pipeline in Simulation · pp. 2–3
B. Parameter Adaptation
The simulation adapts fabric shear resistance to match how the real garment elongates under gravity. The procedure measures the garment in hanging and spread conditions, then matches the simulated shear fraction to the real one.
1) Material properties:
- Shear resistance is calibrated by matching simulated garment elongation under gravity to the real garment's elongation.The paper identifies shear resistance as the most important garment property for its simulation.Source: 1) Material properties · p. 3
- The real garment's shear fraction is measured using the distance from a picked extremity to the lowest point when hanging and when maximally spread.The hanging measurement is L1 and the spread-condition measurement is L2; the passage refers to a shear-resistance fraction but does not provide its formula.Source: 1) Material properties · p. 3
- The virtual garment is hung in Maya, and its shear parameter is adjusted until its shear fraction matches the real garment's.Figure 4 illustrates diagonal-length measurement, zoomed regions, and the garment hanging under gravity.Source: 1) Material properties · p. 3, Figure 4 · p. 3
2) Frictional forces:
Table-surface conditions create friction variability, so matching simulated friction to the real table is important for trajectory optimization. The authors calibrate virtual-table friction against the real garment’s sliding threshold and report similar manipulation results in simulation and reality.
2) Frictional forces:
- Matching simulated friction to the real table is crucial because garment shifts can impair folding and require additional repositioning.Source: 2) Frictional forces · p. 3
- They measure the real garment’s sliding threshold by tilting the table, then reduce virtual-table friction until the garment slides at the corresponding angle.Source: 2) Frictional forces · p. 3
- Using the calibrated virtual friction produces similar manipulation results for the simulated and real garments.Source: 2) Frictional forces · p. 3
IV. TRAJECTORY OPTIMIZATION
The folding task is specified by the desired folded shape and grasp-point start and target positions; the method optimizes a trajectory whose simulated fold matches that shape while limiting path length.
IV. TRAJECTORY OPTIMIZATION
- The task seeks a trajectory that produces the specified folds from the grasp point’s start position to its target position, with the two-end-effector case treated after the single-end-effector case.The desired outcome is specified by the garment’s initial and folded shapes and the grasp point’s start and target positions.Source: §IV · p. 4
A. Trajectory parametrization
- A cubic Bézier curve fixes its endpoints at the grasp-point start and target positions and adjusts two intermediate control points to define the trajectory.The paper uses a third-order curve for simplicity and describes its intermediate points as knot points.Source: A. Trajectory parametrization · p. 4, Figure 5 · p. 4
- The objective balances trajectory length against dissimilarity between the desired folded shape and the shape predicted by cloth simulation.The trajectory-length term favors efficiency in time and energy; α weights the shape-matching term.Source: A. Trajectory parametrization · p. 4
- Shape dissimilarity measures the area-normalized distance between corresponding points on the desired and simulated folded garment surfaces.For triangle meshes, the implementation approximates this with distances between corresponding triangle barycenters, weighted by target-triangle area.Source: A. Trajectory parametrization · p. 4, Figure 6 · p. 4
- The curve length is approximated by recursively subdividing the Bézier curve until chord-length deviation is sufficiently small, then summing the subdivided chord lengths.The subdivision uses De Casteljau’s algorithm.Source: A. Trajectory parametrization · p. 4
B. Optimization.
The optimizer estimates the cost gradient numerically and uses a secant Levenberg–Marquardt procedure to iteratively seek an improved trajectory, stopping when improvement becomes sufficiently small.
B. Optimization.
- The cost gradient is estimated by sampling trajectories perturbed along each orthonormal basis direction, using δ = 10−1.The samples are trajectories x + δe_j.Source: B. Optimization · p. 4
- The secant Levenberg–Marquardt procedure iteratively builds a local quadratic approximation of the squared cost and steps toward an improved state.Its step combines steepest-gradient-descent and conjugate-gradient directions.Source: B. Optimization · p. 4
- The iterative procedure terminates when the improvement in the squared cost becomes sufficiently small.The approach follows the procedure described by Madsen et al..Source: B. Optimization · p. 4
C. Multiple arms.
For multiple arms, the method assigns each arm its own trajectory and extends the optimization state; the resulting dual-arm trajectories are in 3D space.
C. Multiple arms.
- Each arm receives an individual trajectory, and the state variable is extended to include the trajectories for all arms.The remaining optimization procedure is the same as in the single-arm case.Source: C. Multiple arms · p. 5
- Both single-arm and dual-arm trajectories are in 3D space, and dual-arm optimization can find a solution that overcomes failures shown in Figure 3 bottom.Source: C. Multiple arms · p. 5
V. APPLICATION TO GARMENT FOLDING
Garment key points are localized by registering a category-specific template contour to a segmented garment mask, while regularizing contour deformation and preserving semantic feature points.
V. APPLICATION TO GARMENT FOLDING
- After color-based foreground–background segmentation, category recognition selects a garment template, which is registered to the captured mask using a distance field.The paper uses marker-based watershed segmentation for homogeneous-color tables and notes GrabCut as an option for more complicated scenes.Source: A. Key Points Localization · p. 5
- Contour registration minimizes fitting, stretching, and bending energies to match the target mask while limiting and regularizing deformation to preserve angle features.The stretching and bending terms are weighted by user-specified coefficients κ and β.Source: A. Key Points Localization · p. 5
- Registration iteratively subdivides long contour segments and merges short adjacent segments without feature points, then retrieves semantic feature locations by their unique IDs.Iterations stop when the reduction in E_T(S, S̄) becomes sufficiently small.Source: A. Key Points Localization · p. 5
VI. EXPERIMENTAL RESULTS
The experiments tested the method on long-sleeve T-shirts, pants, and towels across multiple trials. The garments required both single-arm and dual-arm folds.
VI. EXPERIMENTAL RESULTS
- The evaluation covered long-sleeve T-shirts, pants, and towels over multiple trials.The tests are shown in Figure 8.Source: §VI · p. 6
- The tested garments required both single-arm and dual-arm folds.Source: §VI · p. 6
- Figure 8 left shows the garments used in the evaluation.Source: §VI · p. 6
A. Robot Setup
The experiments pair robot execution with measured garment and table properties and optimized folding trajectories. Folding was stable for long-sleeve shirts and towels, while pants and jeans were less stable; most garments took about 1.5 minutes to fold.
A. Robot Setup
- The Baxter robot has two seven-degree-of-freedom arms, a calibrated Prime Sense Xtion range sensor, and tactile gripper sensors.Source: A. Robot Setup · p. 6
B. Measurement of parameters
- The authors manually measure each garment’s stretch resistance and table friction; for common garments, column averages are suggested as simulator initialization values.Source: B. Measurement of parameters · p. 6
C. Garment manipulation and folding
- The robot follows optimized trajectories for each step of a multistep folding plan, with real-garment deformation and final outcomes comparable to simulation.Key-point detection from the binary mask takes 3–5 seconds on a regular CPU.Source: C. Garment manipulation and folding · p. 6
- About 1.5 minutes was the average time to fold most garments; long-sleeve T-shirts and towels were stable, while jeans and pants were less stable.The reported instability was associated with high surface shear resistance and difficulty bending; some successful pants and jeans folds had small wrinkles.Source: C. Garment manipulation and folding · p. 6
- Contour fitting is performed once, assuming later folded shapes remain close to the simulation; fitting at every step would take more time and computation.The authors report that the single-fitting approach saves time and works well for predefined folding steps.Source: C. Garment manipulation and folding · p. 6
- Some failures stem from Baxter motor-control drops or jitter, which can increase garment pull forces and cause drift or inaccurate folding.The authors also identify the difference between simulated mesh-point motion and moving a grasped garment area as a simulation–reality gap.Source: C. Garment manipulation and folding · p. 6
VII. CONCLUSION
The paper optimizes garment-folding trajectories in a simulation calibrated to approximate the real setting. The Baxter robot executes the mapped trajectories, and the reported experiments demonstrate accurate and efficient manipulation.
VII. CONCLUSION
- An iterative quadratic objective measures dissimilarity between the simulated folded shape and a user-specified shape to obtain an optimized trajectory.Source: §VII · p. 6
- The optimized trajectory is mapped to a real robot and executed, with experiments demonstrating efficient and accurate garment manipulation.Source: §VII · p. 6
- Successful simulation and real-world folding examples include a long-sleeve shirt, pants, and a medium-size towel.Source: Figure 9 · p. 7