Source-linked AI summary
FoldBack: Self-Correcting Masked Generative Policy for Long-Horizon Garment Folding
Lipeng Zhuang, Shiyu Fan, Yingdong Ru, Zhuo He, Florent P. Audonnet, Paul Henderson, Gerardo Aragon Camarasa
TL;DR
Long-horizon folding policies can continue after a missed or slipped grasp, even though the garment may not be in the intended configuration. FoldBack aligns refinement and grasp verification with pick-and-place events, rolls back failures, and selectively repairs editable trajectory actions. Across 33 garments from six categories, it achieves 75.2% final folding success and 0.837 final-mask IoU, outperforming the strongest prior baseline.
Problem
Long-horizon policies can fix executed actions without verifying their intended physical outcomes, allowing folding to advance after a missed or slipped grasp.
Source: §I · p. 1Method
FoldBack uses an editable masked full-trajectory policy that aligns refinement and verification with key actions, preserves verified history, and repairs failed grasp windows after rollback.
Source: §I · p. 1, A. Key-Action-Aligned Inference (KAAI) · p. 3, B. Grasp-outcome monitoring and… · p. 4Results
75.2% final folding success and 0.837 mean final-mask IoU, exceeding the strongest prior baseline by 29.5 percentage points and 0.148, respectively.
Source: §I · p. 1Takeaways & Limitations
Across six garment categories, FoldBack improves stage completion, recovery success, and final folding performance over action-chunking and full-trajectory baselines without recovery demonstrations or base-policy retraining.
Source: §I · p. 1, §V · p. 8Future work will extend FoldBack to more challenging initial configurations that require a separate flattening phase before folding.
Source: §V · p. 8
Abstract
from arXiv · showhide
We present FoldBack, a self-correcting masked generative policy for long-horizon garment folding. Existing long-trajectory policies may continue after a missed or slipped grasp even when the garment has not reached the intended configuration. We structure FoldBack's recovery mechanisms around three inference-time decisions: when to refine and verify, how to roll back, and where and how to retry. FoldBack aligns refinement and grasp verification with pick-and-place events, returns the robot to a retryable pre-grasp configuration while preserving successful grasps, and selectively regenerates the failed segment and selected future actions while avoiding previous failed grasp locations. To our knowledge, FoldBack is the first editable full-trajectory policy to unify these decisions, enabling failed interactions to be detected, undone, and repaired before execution continues, without recovery demonstrations or base-policy retraining. Across 33 real garments from six categories, FoldBack achieves 75.2% final folding success and 0.837 final-mask IoU, versus 45.7% and 0.689 for the strongest prior baseline.
I. INTRODUCTION
FoldBack targets long-horizon folding failures by keeping grasp-critical actions editable until success is verified. It combines event-aligned refinement, rollback, and trajectory repair, and reports stronger final folding results than the strongest prior baseline.
I. INTRODUCTION
- A missed or slipped grasp can invalidate subsequent folding actions, so reliable long-horizon folding also requires detecting failures and recovering before execution advances.Existing long-trajectory policies may fix executed tokens without verifying physical outcomes; runtime monitors detect failures but do not revise the policy trajectory.Source: §I · p. 1
- FoldBack keeps a grasp-critical window tentative until all required grasps are verified, leaving failed windows editable for recovery.Its three inference-time decisions are when to refine and verify, how to roll back, and where and how to retry; it needs no recovery demonstrations or base-policy retraining.Source: §I · p. 1
- 75.2% final folding success and 0.837 mean final-mask IoU, exceeding the strongest prior baseline by 29.5 percentage points and 0.148, respectively.The reported evaluation spans 33 garments from six categories and single-arm and bimanual tasks.Source: §I · p. 1
II. RELATED WORK
- FoldBack uses editable full-trajectory generation, event-aligned refinement, verified rollback, and selective repair, unlike prior approaches that rely on fixed history or additional recovery machinery.The related-work discussion contrasts these choices with recovery demonstrations, policy adaptation, and auxiliary LLM/VLM reasoning.Source: §II · pp. 1–2
III. METHOD
FoldBack builds on masked generative policy to represent and refine a complete action trajectory. It distinguishes verified history from tentative grasp-critical actions and editable future actions so failures can be repaired without retraining the backbone.
III. METHOD
- FoldBack represents the full action trajectory as discrete tokens and uses a masked generative transformer to predict masked tokens in parallel from the observation.The MGP backbone uses a VQ-VAE tokenizer to map continuous trajectories to discrete tokens; the initial trajectory is generated from a fully masked sequence.Source: §III · p. 2
- Unlike the original MGP inference procedure, FoldBack keeps a grasp-critical window tentative until its grasp outcome is verified.Successful windows enter grasp-verified history, while failed windows remain editable and the unexecuted future can be refined.Source: §III · p. 2
- The policy is trained on Flat’n’Fold using the original two-stage procedure and requires no additional policy training or recovery demonstrations.The tokenizer is trained and frozen before the masked generative transformer is trained to recover masked or replaced action tokens.Source: §III · p. 2
A. Key-Action-Aligned Inference (KAAI)
Key-Action-Aligned Inference schedules refinement around pick-and-place transitions rather than uniform execution intervals. It updates future tokens using new observations while preserving the planned action order and arm assignments.
A. Key-Action-Aligned Inference (KAAI)
- KAAI refines the remaining trajectory before and after each key-action window, aligning updates with pick-and-place transitions rather than uniform intervals.The method uses two mask-refinement iterations at each update.Source: A. Key-Action-Aligned Inference (KAAI) · p. 3
- Each key action has a window spanning four actions before through three actions after its transition; a PICK window is grasp-critical.For bimanual manipulation, the two arm-specific PICK transitions form one key action, with a window spanning the first gripper closure through the joint lift.Source: A. Key-Action-Aligned Inference (KAAI) · p. 3
- After refinement, KAAI recomputes unexecuted transition times, windows, and token sets while keeping PICK/PLACE order and arm assignment fixed.The timing of those actions may shift as the trajectory is refined.Source: A. Key-Action-Aligned Inference (KAAI) · p. 3
B. Grasp-outcome monitoring and Verified Rollback
FoldBack verifies grasps after lifting and commits grasp-critical actions to trajectory history only after success. On failure, verified rollback restores the pre-grasp configuration while preserving successful grasps in asymmetric bimanual trials.
B. Grasp-outcome monitoring and Verified Rollback
- The local monitor compares pre-pick and post-lift point clouds in gripper frames and uses a PointNet binary classifier to assess grasp success.The local representation focuses on garment-geometry changes, largely excludes the gripper, and requires no garment segmentation; its threshold is τ_mon = 0.50.Source: B. Grasp-outcome monitoring and… · p. 3
- Monitor training approximates missed grasps by pairing a pre-pick local cloud with a nearby frame where the garment remains on the table.One classifier is shared across single-arm and bimanual stages.Source: B. Grasp-outcome monitoring and… · p. 3
- A grasp-critical window enters verified history only when all required grasps succeed; otherwise the history stays unchanged and the failed window remains editable.Neither the failed attempt nor rollback motion enters history; in bimanual windows both arms’ grasps must be verified.Source: B. Grasp-outcome monitoring and… · p. 4
- After a failed grasp, FoldBack reverses recorded end-effector motion to restore the pre-grasp configuration and preserves a successful grasp by keeping that gripper closed.For bimanual manipulation, both arms return in coordination; the procedure requires no base-policy retraining.Source: B. Grasp-outcome monitoring and… · p. 4
C. Structured Trajectory Plan Repair
FoldBack repairs a failed grasp by making the failed window and selected future actions editable, rather than trusting actions generated under an unsuccessful-grasp assumption. It regenerates failure-sensitive segments and selects retry candidates while preserving successful grasps.
Distance-guided remasking
- Because future actions assume the failed grasp succeeded, FoldBack remasks the failed window and more of the nearby future, with masking decreasing toward the baseline farther away.It remasks the lowest-confidence tokens in each segment and regenerates them from the post-rollback observation, leaving the verified prefix and unselected future tokens fixed.Source: C. Structured Trajectory Plan Repair · p. 4
Failure-aware candidate resampling
- Eight repair candidates are sampled, rejecting failed-arm picks within 0.04 m of remembered failed locations; asymmetric failures preserve the successful arm’s grasp.FoldBack tries up to three candidate batches; if none is valid, the recovery episode ends unsuccessfully at the restored pre-grasp configuration.Source: C. Structured Trajectory Plan Repair · p. 4
Candidate selection
- For single-arm failures or failures of both grasps, FoldBack executes the valid candidate with the highest mean confidence over regenerated positions.After all required grasps succeed on retry, the re-encoded bimanual subsequence enters verified history and the remaining trajectory is refined from the post-lift observation.Source: C. Structured Trajectory Plan Repair · pp. 4–5
- For asymmetric bimanual failures, candidates combine the successful arm’s stored motion with regenerated failed-arm motion; infeasible candidates are discarded before scoring.Candidates are time-aligned, and those violating inverse kinematics or joint limits are discarded.Source: C. Structured Trajectory Plan Repair · p. 4
IV. EXPERIMENTS
The experiments assess long-horizon folding, recovery from controlled grasp slips, generalization, and inference-time design choices in real-world trials using policies trained on Flat’n’Fold demonstrations.
Real-world setup
- The real-world evaluation uses a Baxter dual-arm robot, one ZED2 RGB-D camera, and 33 garments across six categories, with category-specific policies trained jointly on demonstrated strategies.Trials generally start with flattened garments; evaluated strategies may alternate between single-arm and bimanual manipulation.Source: §IV · p. 5
Model variants
- The model variants disable recovery altogether or replace structured repair with confidence-based remasking and a single repair candidate, while retaining specified FoldBack components.FoldBack without Rollback-and-Repair retains KAAI; FoldBack without Structured Repair retains KAAI, grasp-verified history, and rollback.Source: §IV · p. 5
Baselines
- Comparisons include DP3, SKIL, MGP-Long, and DP3 with verified rollback; the first three baselines do not explicitly detect grasp failures or initiate retries.Baselines infer every 12 executed actions, whereas FoldBack refines at key-action boundaries.Source: §IV · p. 5
Evaluation metrics
- Recovery success measures stage completion within up to three rollback-and-retry attempts, while final folding success requires every stage to complete and final-mask IoU to reach 0.80.The evaluation also defines first-retry success, stage completion, and final-mask IoU averaged over all trials, including incomplete ones.Source: §IV · pp. 5–6
A. Long-Horizon Garment Folding
FoldBack achieves the strongest reported long-horizon garment-folding results, with gains over MGP-Long that are larger on evaluated five-stage strategies.
Long-horizon garment folding
- 75.2% final folding success and 0.837 final-mask IoU are reported for FoldBack, which has the best final success in all six categories.MGP-Long is the strongest baseline without recovery.Source: A. Long-Horizon Garment Folding · p. 6
- KAAI alone raises average final success from 45.7% to 53.3%, while structured repair raises it from 62.9% to 75.2%.The passage reports these improvements as average final success and final success, respectively.Source: A. Long-Horizon Garment Folding · p. 6
- Against MGP-Long, stage-completion gains are 13.4–15.0 points on evaluated two-stage folds and 31.9–37.3 points on three five-stage garment categories.The evaluation includes 30 napkin trials and 15 trials for each other category.Source: A. Long-Horizon Garment Folding · p. 6
B. Recovery under Controlled Grasp-Slip Perturbations
Controlled one- or two-corner slips test recovery during bimanual towel folding. FoldBack combines grasp monitoring, editable failed windows, and rollback to repair failures while preserving successful grasps.
Controlled grasp-slip recovery
- The controlled bimanual towel-folding trials inject one- or two-corner slips during lifting, with one perturbation and no further intervention per trial.Fifteen trials were run for each method-condition pair.Source: B. Recovery under Controlled… · p. 6
- Across 96 failed and 352 successful grasps, the monitor achieves 92.7% failed-grasp recall and a 4.5% arm-wise false-positive rate.Recovery success is the fraction of detected episodes whose affected stage completes within three attempts.Source: B. Recovery under Controlled… · pp. 6–7
- 28.6 and 37.1 percentage-point recovery-success gains over FoldBack without Rollback show the benefit of physical rollback under one- and two-corner slips.Keeping the failed window editable rather than fixing it after execution raises recovery success by 14.3 and 13.3 points, respectively.Source: Table II · p. 6, B. Recovery under Controlled… · p. 7
- 85.7% and 80.0% recovery success; full FoldBack improves final folding success by 20.0 and 33.3 points over FoldBack without Structured Repair.For detected asymmetric failures, it preserves the successful grasp in 100% of cases, versus 71.4% without Structured Repair.Source: Table II · p. 6, B. Recovery under Controlled… · p. 7
C. Generalization
FoldBack is evaluated on unseen pants orientations, held-out garment instances, and non-canonical napkin and towel starts. It outperforms MGP-Long in the reported final-success comparisons.
Initial-orientation generalization
- FoldBack reaches 80.0%, 73.3%, and 60.0% final success at the training, moderate-rotation, and large-rotation pants orientations, versus 66.7%, 60.0%, and 40.0% for MGP-Long.Each unseen rotation range contains 15 trials per method.Source: C. Generalization · p. 7
Held-out garment instances
- FoldBack achieves 66.7% average final success on held-out garment instances, versus 36.7% for MGP-Long and 28.3% for DP3, and performs best in all six categories.Each category has one physical garment held out; ten trials per instance yield 60 trials per method.Source: C. Generalization · p. 7
Non-canonical initial configurations
- FoldBack achieves 80.0% final success on napkins and 73.3% on towels in non-canonical starts, versus 53.3% and 46.7% for MGP-Long.These near-flat starts include local wrinkles or a partially folded corner and require flattening before folding.Source: C. Generalization · p. 7
D. Ablation Studies
Towel-folding ablations examine event-aligned refinement, remasking strategy, and candidate selection. The strongest reported recovery and final-success results come from distance-guided remasking and failure-aware resampling.
Remasking strategy
- Distance-guided remasking achieves the highest recovery success (85.7%) and final folding success (93.3%).It fully remasks the failed window, then progressively lowers the masking ratio over subsequent segments, prioritizing low-confidence tokens.Source: D. Ablation Studies · p. 7
Candidate selection
- Failure-aware candidate selection achieves the highest first-retry, recovery, and final success rates: 71.4%, 85.7%, and 93.3%.It rejects candidates whose failed-arm pick falls within the exclusion region around previous failed grasps; resampling takes 58 ms per recovery event versus 37 ms for one candidate.Source: D. Ablation Studies · p. 7
Refinement strategy
- KAAI achieves the highest stage completion and final success, and its comparison with Uniform isolates event alignment using the same number of refinements.KAAI refines before and after each key-action window; Fixed-12 refines every 12 executed actions.Source: D. Ablation Studies · p. 7
V. CONCLUSION
FoldBack keeps failed grasp windows editable and repairs them after rollback. The conclusion reports improved folding performance over action-chunking and full-trajectory baselines, while identifying harder starts as future work.
Conclusion
- FoldBack maintains a grasp-verified trajectory history so failed grasp windows can be updated rather than becoming fixed history.After failure, it returns to a retryable pre-grasp configuration, selectively repairs the trajectory with distance-guided remasking, and resamples while avoiding previously failed grasp locations.Source: §V · pp. 7–8
- Across six garment categories, FoldBack improves stage completion, recovery success, and final folding performance over action-chunking and full-trajectory baselines.Future work will extend FoldBack to more challenging initial configurations that require a separate flattening phase before folding.Source: §V · p. 8