PaperlayerView exact revision on arXiv

Source-linked AI summary

DefFlow: Goal-Conditioned Deformable Object Manipulation using 3D Point Flows

Gaurav Singh, Sergio Orozco, Ryan Duong, Mete Tuluhan Akbulut, George Konidaris, Srinath Sridhar

arXiv:2610.22827v1cs.RO

TL;DR

Goal-conditioned deformable manipulation must handle diverse 3D configurations that can call for different manipulation behaviors as well as precise motions. DefFlow learns joint robot–object point flows from demonstrations, and outperforms action-only and robot-flow-only baselines across rope, granular-media, and cloth settings.

  • Problem

    As initial and target configurations broaden, deformable-object manipulation requires diverse goal-dependent behaviors in addition to precise continuous actions.

    Source: §I · p. 1

  • Method

    DefFlow learns from demonstrations by jointly predicting goal-conditioned 3D point flows for the robot and deformable object, then uses robot flows as the policy.

    Source: §I · p. 2, §VI · p. 8

  • Results

    Across rope, granular-media, and cloth environments, DefFlow outperforms action-only and robot-only flow baselines; it is more than 2× better than the next-best baseline on RopeEnv L2 and Succ@3cm.

    Source: §V · p. 7

  • Takeaways & Limitations

    Joint robot–scene prediction better captures manipulation decisions such as grasping, releasing, and re-grasping alongside the continuous motions needed for diverse goal configurations.

    Source: §I · p. 2

Abstract

from arXiv · show

Manipulating deformable objects toward desired 3D configurations is a fundamental capability for robots operating in unstructured physical environments. However, as the range of possible initial and target configurations grows, goal-conditioned deformable manipulation requires a robot to infer increasingly diverse manipulation behaviors in addition to precise continuous actions. We introduce DefFlow, a framework for goal-conditioned manipulation of deformable objects toward a broad range of 3D configurations. DefFlow learns from demonstrations by jointly predicting the goal-conditioned 3D point flow of the robot and deformable object. We show that this formulation better captures both the manipulation decisions, such as when and where to grasp, release, or re-grasp, and the precise continuous motions required for goal reaching, rather than predicting only the robot flow. Across rope, granular-media, and cloth manipulation settings with diverse initial and target configurations, DefFlow substantially outperforms goal-conditioned action-prediction and robot-flow-only baselines on success rates and goal-distance metrics. We further demonstrate deployment of our framework on a real robot using real point-cloud observations and unseen goal configurations.

I. INTRODUCTION

Deformable-object manipulation must handle complex 3D shape changes and goal-dependent behaviors, but existing approaches have limited behavior or configuration coverage. DefFlow predicts joint robot–object flows to represent both manipulation choices and continuous motions, and is evaluated across diverse settings and on a real robot.

I. INTRODUCTION

  • As initial and target configurations broaden, a robot may need different behaviors even for the same object, such as dragging for one rope goal or twisting, releasing, and re-grasping for another.Deformable objects also undergo broad shape changes governed by complex dynamics, and their motion is coupled with the robot’s.Source: §I · p. 1
  • Prior goal-conditioned point-cloud approaches cover narrower configuration distributions or isolated primitives, while robot-only dense-flow prediction omits deformable-object flow.Examples include single-grasp servoing, pinching or squeezing, and rigid tool-flow prediction.Source: §I · pp. 1–2
  • DefFlow jointly predicts goal-conditioned robot and deformable-object 3D point flows, using the resulting robot flows as a policy for diverse goal-reaching behaviors and continuous actions.The joint prediction associates robot behavior with its corresponding object deformation, helping distinguish behaviors by their effects on the scene.Source: §I · p. 2
  • The paper evaluates DefFlow across varied rope, granular-media, and cloth configurations and reports real-robot deployment on unseen goal shapes.The introduction also describes outperforming baselines that predict only robot actions or robot flow.Source: §I · p. 2

A. 3D Point Flows for Manipulation

Prior point-flow methods mainly model forward 3D dynamics, while goal-reaching policies use predicted flows to recover actions; existing approaches often restrict the action space or flow to rigid tools.

A. 3D Point Flows for Manipulation

  • Point-flow representations model 3D dynamics as particles moving through space, often for action-conditioned world models that predict future state evolution.Such models can support model-predictive control, but goal-reaching still requires action selection.Source: A. 3D Point Flows for Manipulation · p. 2
  • Existing point-flow approaches simplify goal-reaching through restrictions such as pre-grasped starts, tabletop pushing or grasp-and-drag, or parameterized gripping and lifting.Source: A. 3D Point Flows for Manipulation · p. 2
  • ToolFlowNet predicts dense 3D flows for rigid tools or end-effectors under a rigidity constraint, whereas this paper predicts full-scene flow conditioned on a goal configuration.Source: A. 3D Point Flows for Manipulation · p. 2

B. Goal-Conditioned Deformable Object Manipulation

Goal-conditioned deformable manipulation spans varied tasks, but prior methods often constrain either the action setting or the behaviors available, leaving diverse 3D configurations challenging.

B. Goal-Conditioned Deformable Object Manipulation

A. Problem Formulation

The formulation conditions robot actions on observation history and a desired deformable-object configuration. DefFlow predicts future joint scene flow from scene and goal point clouds, then uses robot flow to recover executable actions.

A. Problem Formulation

  • DefFlow processes the scene, including the robot, together with goal points in a unified 3D space and predicts future scene-point flows over a horizon.The architecture uses a PTv3 backbone and an MLP; robot flows are used to recover executable robot actions.Source: Figure 2 · p. 3, A. Problem Formulation · p. 3
  • The policy objective is to choose robot actions from observation history and a goal specifying the desired 3D object configuration, so the object moves from its initial state toward that goal.Source: A. Problem Formulation · p. 3
  • The model’s per-point MLP predicts future displacements for both robot and scene features, producing joint robot and scene flow.Source: A. Problem Formulation · p. 3

B. Action Extraction via Inverse Kinematics.

DefFlow converts predicted robot point positions into executable actions by solving for robot joint configurations that reproduce them. The resulting sequence can be sent to existing robot controllers without a separate learned action head.

B. Action Extraction via Inverse Kinematics.

  • Damped least-squares inverse kinematics recovers robot joint configurations whose forward kinematics reproduce the predicted point positions.The solution is computed for future timesteps using robot point clouds derived from the URDF, including gripper finger prismatic joints.Source: B. Action Extraction via Inverse… · p. 3
  • The resulting action sequence can be sent to existing robot controllers without a separate learned action head.Source: B. Action Extraction via Inverse… · p. 3

IV. EXPERIMENTS

The experiments benchmark deformable-object goal reaching across rope, granular-media, and cloth simulations, with varied configurations and interaction sequences. They describe demonstration collection, adapted baselines, evaluation metrics, and qualitative results.

Simulation Benchmark and Data Curation

  • RopeEnv tests multi-stage grasping, dragging, lifting, twisting, and releasing, including transitions to re-grasping, with continuously varying motion magnitudes.It is described as the most challenging environment because it tests behavior sequences across a broad space of rope configurations.Source: §IV · p. 4
  • GranularEnv tests non-prehensile shaping through paddle sweeps whose direction, distance, number, and order vary with the target shape.The same basic sweeping primitive can reshape the pile into substantially different forms.Source: §IV · p. 4
  • ClothEnv tests edge-grasping and dragging around a rigid obstacle, where contact affects how the cloth bends, wraps, and slides.The robot subsequently releases the cloth; the policy must infer behavior for the target configuration while accounting for contact-induced deformation.Source: §IV · p. 4

Collecting Demonstrations

  • Nearly 7,000 randomized tasks are generated from procedural play and hindsight relabeling, with held-out tasks differing in shapes and behaviors from training.The final object state of each episode is treated as its goal; demonstrations compose continuous interaction primitives in varying numbers.Source: §IV · p. 5

Baselines

  • Baselines include robot-flow ablations and adapted goal-conditioned action policies, with training data, goal distributions, environments, and input/output hyperparameters held fixed for adapted comparisons.The robot-flow-only ablation removes deformable-scene-flow supervision; the rigid-flow variant applies an SVD-based rigid-motion projection. ClothEnv’s Grasp-Servo baseline is not a direct fair comparison because its action primitives are provided.Source: §IV · p. 5
  • An MPC + PointWorld baseline was attempted but did not converge to plausible results because action sampling was the bottleneck, despite a working forward model.The authors state that MPC had not been successfully shown on deformable tasks as complex as these with multiple behaviors to search and chain.Source: §IV · p. 5

Qualitative results

  • In rope examples, DefFlow predicts precise grasping and swishing, and different actions for unseen start–goal combinations; granular results also include unseen goal and letter shapes.The rope comparisons show baselines making grasp errors or missing the precise swish; the granular examples require different sweep actions and orderings.Source: Figure 5 · p. 6, Figure 6 · p. 6

Metrics

  • Rope and cloth use mean per-particle L2 distance and Succ@3cm, while granular shaping uses Chamfer distance, Earth Mover’s Distance, and voxelized 3D IoU.Succ@3cm counts an episode as successful when final-state mean per-particle L2 distance is below 3 cm; granular metrics avoid particle correspondence.Source: §IV · p. 6

V. RESULTS AND ABLATIONS

DefFlow improves deformable-object goal reaching across rope, granular-media, and cloth tasks, with gains that are more pronounced in broader configuration spaces. Qualitative results and real-robot demonstrations also show goal-directed manipulation across varied and unseen configurations.

Qualitative results

  • DefFlow predicts coupled scene evolution and robot actions for rope swishing and granular sweeping, with actions accurately reaching the goal state.The coupling helps distinguish similar training actions with different deformations, supporting more accurate robot motion at inference.Source: §V · pp. 6–7

Quantitative results

  • DefFlow outperforms action-only and robot-only flow baselines across all three environments; on RopeEnv it is >2× better than the next best baseline on L2 and Succ@3cm, and on ClothEnv it achieves a 1.5× higher success rate than the strongest baseline.For GranularEnv, DefFlow gets the best IoU compared with other baselines.Source: §V · p. 7

Effect of Configuration Diversity

  • As cube-pose and stack-location perturbations increase, DefFlow’s success rate degrades more gracefully than GC-DP3 and Robot-Flow, which drop quickly.The demonstrations cover each setting’s task distribution but become increasingly sparse as the configuration space grows.Source: §V · p. 7, Figure 7 · p. 7

Real-Robot Deployment

  • On a physical robot, pretrained policies handled multiple rope starts and two goals, unseen granular letter shapes, and unseen cloth goals without task-specific fine-tuning.The demonstrations use a Franka Emika Panda arm and real point-cloud observations.Source: §V · pp. 7–8

VI. CONCLUSION

DefFlow targets diverse 3D start and goal configurations by learning joint robot–scene point flows from demonstrations. Across rope, granular-media, and cloth tasks, joint prediction improves goal reaching; a controlled toy study shows an advantage as configuration spaces broaden and demonstrations become sparser.

VI. CONCLUSION

  • DefFlow learns goal-dependent robot behavior and scene evolution by jointly predicting robot and deformable-scene point flows from demonstrations collected with procedural play data and hindsight goal relabeling.Across rope, granular-media, and cloth manipulation, joint prediction improves goal-reaching performance; a controlled toy study shows an advantage as configuration space broadens and demonstrations become sparser.Source: §VI · p. 8

Related papers

Loading 2610.22827v1…