FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

• Cloth is difficult to manipulate because it has a high-dimensional configuration space, can be partially occluded, and does not transform rigidly.Scripted approaches are typically slow and do not generalize to arbitrary cloth goals.Source: §1 · p. 1
• Image-conditioned policies must jointly reason about the relationship between current and goal images and the action, while prior learning-based goal-conditioned methods were single-arm only.The introduction also identifies reward functions as weak supervision for complex cloth manipulation.Source: §1 · p. 1
• FabricFlowNet uses optical flow to represent observation-goal correspondence as policy input and to determine place points after the policy estimates picks.The method is trained with supervised simulation data from random actions, without expert demonstrations or reinforcement learning.Source: §1 · pp. 1–2
• FFN performs single- and dual-arm actions, switching between them according to the desired goal.The method significantly outperforms extended single-arm approaches on bimanual tasks.Source: §1 · p. 2
• FFN outperforms prior approaches in its test suite and generalizes to other cloth colors and shapes without training on those variations.The test suite encompasses and expands on goals used in earlier work [11].Source: §1 · p. 2

Bimanual Manipulation

• Earlier bimanual cloth methods include hard-coded flattening policies, while Tanaka et al. use a voxel dynamics model for goal-conditioned folding.Optimizing that model can slow inference compared with the model-free approach described here.Source: §2 · p. 2

Learning for Cloth Manipulation

• Prior cloth-learning approaches use hand-defined representations, learned pixel-wise embeddings, or goal-conditioned transporter networks, and some rely on expert demonstrations [11] [30] [8].FabricFlowNet uses test-time sub-goal states rather than expert actions.Source: §2 · pp. 2–3
• Compared with model-based cloth methods, FabricFlowNet is model-free and does not require online simulation or time-expensive CEM planning.The paper reports faster inference and significantly improved performance against state-of-the-art cloth approaches.Source: §2 · p. 3

Optical Flow for Policy Learning

• Prior optical-flow work uses successive observations to represent motion or identify failed actions, whereas FabricFlowNet matches cloth pose between observation and goal images to choose folding actions.Unlike visual servoing with demonstration trajectories [40], it learns which folding actions to take.Source: §2 · p. 3

3.1 Problem Definition

• A folding task is a sequence of visual subgoals, each achievable by one possibly bimanual pick-and-place action; the sequence is unavailable during training.Subgoals are used because a single goal image may not describe a highly self-occluded folded cloth.Source: §3.1 · p. 3

3.2 Overview

• Using flow as both policy input and place-point representation significantly improves performance over prior work and naturally supports transitions between single- and dual-arm actions.PickNet predicts pick points; the corresponding place points are computed from the flow image.Source: §3.2 · p. 4

3.3 Estimating Flow between Observation and Goal Images

• The flow network estimates pixel correspondences between current and goal depth images, trained with simulator ground-truth flow and endpoint error supervision.Because simulated cloth particles are sparser than depth-image pixels, supervision is masked to pixels aligned with particles.Source: §3.3 · p. 4

3.4 Learning to Predict Pick Points

• PickNet predicts two conditionally dependent pick points using separate networks, with the second conditioned on the first pick point.The first network predicts a heatmap from flow; the second receives flow plus a Gaussian centered on the first pick.Source: §3.4 · p. 5

3.5 Estimating the Place Points from Flow

• Place points are obtained by querying flow at each pick point and mapping it to its corresponding goal-image location.This assumes a picked point does not shift after gripper release; either pick-point or place-point proximity below 30 pixels triggers a single-arm action.Source: §3.5 · p. 6

3.6 Implementation Details

• Training data come from random actions in SoftGym, and the authors report transferring the trained policy to a real-world robot.The simulator models cloth as particles connected by springs, with random grasping actions biased toward cloth corners.Source: §3.6 · p. 6

4.1 Simulation Experiments

• Simulation evaluation covers 40 one-step and 6 multi-step folding goals, with one pick-and-place action allowed per subgoal.The protocol assumes cloth smoothing before folding and measures average particle position error against the goal configuration.Source: §4.1 · p. 6

4.1.1 Simulation Results

• FFN achieves the lowest mean particle distance error across all goals and the fastest inference time in simulation.Table 1 reports error for one-step goals, multi-step goals, and both sets combined.Source: §4.1.1 · p. 6, Table 1 · p. 7
• 6.62 mean error with iterative refinement versus 7.14 without refinement across all goals.The policy takes multiple actions per subgoal and advances when average flow reaches a minimum threshold.Source: §4.1.1 · p. 7

4.1.2 Ablations

• Flow as an action representation improves performance by 32.4%, while removing flow entirely yields a 60.4% improvement for FFN over NoFlow.The ablations also report 35.5% better performance with corner-biased picks and similar performance with the alternative loss formulation.Source: §4.1.2 · p. 7, Table 2 · p. 7

4.2 Real World Experiments

• Real-robot trials qualitatively show successful achievement of many goals, using a two-arm system evaluated on a 30x30 cm towel.Simulation-to-real transfer uses depth alignment and cloth masking; FFN uses depth images rather than RGB.Source: §4.2 · pp. 7–8, §4.2.1 · p. 8, Figure 4 · p. 8

4.2.1 Real World Results

• FFN achieves 0.80 mean IoU over 3 square-cloth trials, compared with 0.53 for NoFlow.IoU on achieved cloth masks is used as a proxy because true cloth-position error is unavailable in the real world.Source: §4.2.1 · p. 8
• Trained only on a square cloth in simulation, FFN generalizes to rectangular fabric and a T-shirt across shapes, textures, and colors.The tested fabrics are thinner than the square towel used in the real-world experiments.Source: §4.2.1 · p. 8, Figure 5 · p. 8

• FabricFlowNet uses flow to represent observation-goal correspondence and as an action representation for goal-conditioned fabric folding.The method is trained entirely using random data in simulation.Source: §5 · p. 8
• Separating correspondence learning from policy learning improves performance across an extensive suite of single- and dual-arm folding goals in simulated and real environments.Source: §5 · p. 8
• Experiments demonstrate generalization to different fabric shapes, textures, and colors.Future work could incorporate actions beyond pick and place, including parameterized trajectories or dynamic actions.Source: §5 · p. 8

A.1 FabricFlowNet Implementation Details

• The training dataset contains 20k samples from 4k episodes, each with five dual-arm pick-and-place actions and initial and next depth observations plus cloth-particle positions.Pick locations favor cloth-mask and true corners; place points move both arms in the same inward, orthogonal direction, with distances uniformly sampled from 25 to 150 px and truncated near image edges.Source: §A.1 · pp. 12–13, Figure S1 · p. 13
• FlowNet is trained on stacked initial and next depth images, using corresponding simulator cloth particles for sparse ground-truth flow and a masked loss.Training uses 20k random-action samples, spatial translation and rotation augmentation, and Adam with learning rate 1e-4, weight decay 1e-4, and batch size 8.Source: §A.1 · p. 13
• PickNet1 and PickNet2 use fully convolutional encoder-decoder architectures, with PickNet2 additionally receiving PickNet1’s first pick as a Gaussian input.Both output a 20 x 20 spatial grid; off-mask predictions are projected to the nearest mask pixel, and training uses Adam with learning rate 1e-4 and batch size 10.Source: §A.1 · p. 13

A.2 Additional Simulation Results for FabricFlowNet

• FFN achieved the cloth configurations shown for both one-step and multi-step goals, with one simulation trial because the policy and simulation are near-deterministic.The configurations are shown in Fig. S2b and Fig. S2f, respectively.Source: §A.2 · p. 13

A.3 Additional Real World Details and Results for FabricFlowNet

• Because FFN takes depth images as input, the network can transfer to cloths of different colors and patterns.Color-based table masking is used to obtain the cloth mask in the real world.Source: §A.3 · pp. 13–14, Figure S3 · p. 15
• NoFlow performs considerably worse than FFN on real cloth folding, with mean IOU reported for square towels, rectangular cloths, and T-shirts.The best-achieved-IOU trial is shown for each qualitative example; multi-step examples show only the final goal.Source: §A.3 · p. 14, Table S1 · p. 16
• Fixed pick-and-place primitives may not suit every cloth type, size, or fold, and can lead to cloth flopping back or unwanted secondary folds.The authors identify learning better primitives as a possible future direction.Source: §A.3 · pp. 14–15
• Slight flow-prediction inaccuracies can cause fold-distance overestimation or underestimation.Source: §A.3 · p. 15
• For sequential sub-goals, allowing only one action per sub-goal can produce poor predictions when the observed image differs from the demonstration’s starting image.The authors suggest recalculating flow after actions and allowing multiple actions before advancing as a possible improvement.Source: §A.3 · p. 16

B.1 Fabric-VSF [4] Implementation Details

• The Fabric-VSF comparison setup uses 7,115 trajectories of 15 random pick-and-place actions each, totaling 106,725 data points.This is 5× the 20,000 samples used to train FFN; Fabric-VSF takes three context frames and predicts seven target frames.Source: §B.1 · p. 16

B.2 Additional Fabric-VSF [4] Results

• The best Fabric-VSF variant beats FFN on overall and one-step error but is slightly worse on multi-step error, despite using 5× more training data.Its CEM planning takes ∼7 minutes per action, versus ∼0.007 seconds for an FFN forward pass; the authors call this impractical for real-world folding.Source: §B.2 · p. 17, Table S2 · p. 17
• For single-arm Fabric-VSF, large actions always outperform small actions, while dual-arm variants tend to perform worse than single-arm variants.The authors suggest that doubling action dimension increases CEM planning complexity and makes optimal actions harder to find.Source: §B.2 · p. 17

C.1 Lee et al. [9] Implementation Details

• Lee et al. uses a Q-network over discrete rotations and action lengths, selecting the action with the highest Q-value across image locations.The dual-arm extension produces a heatmap for each arm and selects actions using their averaged maximum Q-values.Source: §C.1 · pp. 17–18

C.2 Additional Lee et al. [9] Results

• The Lee et al. comparison varies arm count, image input, corner bias, and camera height, with all variants trained on 20,000 examples.FFN is also trained on discrete-action and continuous-action datasets; Lee et al. cannot train on continuous actions or actions outside its discrete space.Source: §C.2 · p. 18
• All Lee et al. variants perform poorly compared with FFN, particularly on one-step goals; FFN also outperforms variants trained on either discrete or continuous action data.Continuous-action training improves FFN on one-step goals, but the discrete-action dataset also performs fairly well, so the advantage cannot be explained solely by that distinction.Source: §C.2 · p. 18, Table S3 · p. 18
• Lee et al. has similar one-step performance with and without sub-goals, but performs worse on multi-step goals without them.The comparison uses the best Lee et al. variant at test time.Source: §C.2 · p. 18, Table S4 · p. 19

D.1 Ablation Implementation Details

• NoFlowIn replaces flow input with observation and goal depth images, while retaining flow-based place-point querying.Source: §D.1 · p. 19
• NoFlowPlace predicts place points from a flow image using a network with the PickNet architecture; NoFlow combines this with NoFlowIn.Source: §D.1 · p. 19
• NoCornerBias samples pick actions uniformly over the visible cloth mask instead of biasing data collection toward corners.The actions still constrain both arms to the same direction and distance from their pick points and point toward the frame center.Source: §D.1 · p. 19
• NoSplitPickNet uses one PickNet for both arms, producing two pick-point heatmaps from the flow image.Source: §D.1 · p. 19

D.2 Additional Ablation Results

• Ablation results are grouped by single-step, multi-step, and all goals.Source: §D.2 · p. 19
• The supplied passages do not specify which ablation performs best or give table values.Source: §D.2 · p. 19, Table S5 · p. 19
• Table S5 reports mean particle distance error for the ablations.Source: Table S5 · p. 19

E Additional Results on Unseen Cloth Shapes

• FFN outperforms Fabric-VSF and Lee et al. by a large margin on unseen rectangular cloth and T-shirt folding.Table S6 reports quantitative results for both shapes; Fig. S5 provides a qualitative comparison on rectangular cloth.Source: E Additional Results on Unseen Cloth… · pp. 19–20, Table S6 · p. 20, Figure S5 · p. 20

F End-to-End Variants of FFN

• JointFFN scores 9.28 versus 7.14 for FFN, while JointPredictPlace scores 22.88 versus 7.14 and 10.56 for PredictPlace on all goals.The authors report that end-to-end variants perform worse and suggest that pick-loss gradients into FlowNet add noise; removing the intermediate flow loss also worsens both variants.Source: F End-to-End Variants of FFN · pp. 20–21, Table S7 · p. 20

G FFN Performance with Crumpled Starting Configurations

• With slightly crumpled starts, FFN has slightly higher particle distance error but still produces actions very close to intended goals across 46 evaluation goals.The configurations are generated by a random pick-and-place action with a maximum translation of 10 pixels.Source: G FFN Performance with Crumpled… · p. 21, Table S8 · p. 21, Figure S6 · p. 21

H FFN Performance with Iterative Refinement

• Iterative refinement can improve performance, particularly on multi-step goals, using up to three actions per subgoal and an average-flow threshold to recognize completion.The policy compares the current observation with the goal and advances when the minimum average-flow threshold is achieved.Source: H FFN Performance with Iterative… · pp. 21–22, Table S9 · p. 21, Figure S7 · p. 22

I FlowNet Performance

• FlowNet achieves 1.0268 average endpoint error on simulated test goals unseen during training.Figure S8 shows arrow and RGB-channel flow visualizations; FlowNet takes depth images as input and is trained on sparse ground-truth flow.Source: I FlowNet Performance · p. 22, Figure S8 · p. 22