Online Material Estimation for Conditioned Diffusion Policy in Shaping Deformable Linear Objects

I. INTRODUCTION

• Different DLO materials deform differently under identical actions, making material-aware manipulation important even when the desired shape is unchanged.The paper identifies stiffness, elasticity, and surface friction as relevant material properties.Source: §I · p. 1
• Imitation-learning policies commonly target a fixed object, while property-conditioned approaches use a simulator; practical settings may lack a simulator and known object identity.Source: §I · p. 1
• An LSTM-based estimator predicts material from multi-view images and robot joint-state histories, supplying its label to the diffusion policy at every inference step.Source: §I · p. 1
• At inference, the framework needs no simulator, reference demonstration, or dedicated probing action, instead re-estimating material from the task’s own observations.Source: §I · p. 1
• Online-estimated material labels achieved performance comparable to ground-truth-conditioned labels across 480 demonstrations, four materials, and three groove-placement tasks, without prior material information.The pronounced failure case was associated with persistent material confusion.Source: §I · p. 1
• Rollout analysis indicates that the estimator extracts material-related information, the policy responds to its conditioning signal, and persistent confusion accompanies the pronounced failure case.Source: §I · p. 1

A. Online Adaptation of DLO Manipulation

• Model-based approaches represent DLOs with Cosserat rods or mass–spring systems, but require identification for each object.Source: A. Online Adaptation of DLO Manipulation · p. 1
• Online adaptation estimates deformation-model parameters or robot-motion/deformation relationships, adapting models or controllers rather than imitation-learned policies.Source: A. Online Adaptation of DLO Manipulation · pp. 1–2

B. Learning-Based Manipulation of DLOs

• Behavior cloning learns rope visuomotor policies, hierarchical imitation supports long-horizon cable routing, and vision-language models guide routing with reinforcement-learning skills [10].Source: B. Learning-Based Manipulation of DLOs · p. 2

C. Estimating Physical Properties through Interaction

• Interactive perception recovers object properties through interaction, including from multi-step dynamics and in-hand tactile exploration in SwingBot.The cited work also describes properties observable only through interaction.Source: C. Estimating Physical Properties… · p. 2
• GenDOM conditions a policy on inferred deformable-object parameters, but relies on a simulator; SwingBot and GenDOM use probing or a reference demonstration and identify only once.Source: C. Estimating Physical Properties… · p. 2
• The proposed simulator-free method updates estimates from task observations, using a discrete material label instead of continuously supervised parameters; its estimator need not be perfectly accurate.Source: C. Estimating Physical Properties… · p. 2

III. PROBLEM SETTING

• The task places a grasped DLO along a target groove despite unknown material, with consistent initial shapes across object types and RGB-camera and robot-state observations.The robot acts through target joint angles and gripper commands; the material estimate and task label condition the policy.Source: §III · p. 2, Figure 1 · p. 2

IV. PROPOSED METHOD

• The estimator runs alongside the diffusion policy, predicting material from interaction history so the conditioning label is refreshed throughout execution.Source: §IV · p. 2

A. Material–Task Conditioned Diffusion Policy

• The diffusion policy is conditioned on material and task labels because DLOs with different material properties require different strategies for the same target shape.Source: A. Material–Task Conditioned Diffusion… · p. 2
• ResNet18 visual features from environment and hand cameras are combined with joint states and one-hot labels, then aggregated into a global conditioning feature.Source: A. Material–Task Conditioned Diffusion… · p. 2
• A 1D U-Net generates action sequences through iterative denoising; training uses ground-truth material labels, while execution uses estimated labels.Source: A. Material–Task Conditioned Diffusion… · p. 2

B. Online Material Estimation Network

• Materials with similar visual appearances can be distinguished through their different deformation responses to manipulation.The estimator therefore predicts material from a sequence of observations rather than a single image.Source: B. Online Material Estimation Network · p. 3
• The estimator encodes camera images and robot joint states at each of the latest 10 time steps, processes the concatenated features with an LSTM, and aggregates its outputs with attention pooling.An MLP maps the pooled feature to probabilities over material labels.Source: B. Online Material Estimation Network · p. 3
• The material-label probabilities are trained against labeled episodes using cross-entropy loss.The network's prediction is a probability for each material label.Source: B. Online Material Estimation Network · p. 3

V. EXPERIMENTS

• The experiments assess material estimation, material-conditioned policy performance, and how the estimator affects generated actions.The rollout analysis characterizes both the online estimator and its effect on actions.Source: §V · p. 3

A. Setup and Data Collection

• The real-robot setup uses a 6-degree-of-freedom UR5e and two RGB cameras, one environment-mounted and one near the robot hand.Source: A. Setup and Data Collection · p. 3
• The setup includes Silicone, Thick acrylic, Thin acrylic, and Cotton, tested on crank, vertical-S, and horizontal-S groove-placement tasks.The vertical- and horizontal-S tasks use the same S-shaped board with different initial configurations.Source: A. Setup and Data Collection · p. 3
• The groove cross-section is designed so excessive tension can pull a DLO away from the groove, testing whether policies adapt to material-dependent deformation.The groove depth accommodates the thickest DLO.Source: A. Setup and Data Collection · p. 3
• The dataset contains 480 successful teleoperated demonstrations: 40 for each of the 12 material–task combinations.Source: A. Setup and Data Collection · p. 3

B. Compared Policies and Evaluation Protocol

• The four comparisons are a per-material–task Specialist policy, a shared Task-only policy, an Oracle policy with ground-truth material labels, and an Estimated policy that substitutes online estimates into the Oracle policy.The Task-only and Oracle policies are trained on all 480 demonstrations; Specialist policies use 40 demonstrations per material–task pair.Source: B. Compared Policies and Evaluation… · pp. 3–4
• Table I reports material-estimation accuracy for different training datasets.Source: Table I · p. 4
• A trial counts as successful when the released DLO lies within the groove outline from above, regardless of its height above the board.Source: B. Compared Policies and Evaluation… · p. 4
• Each policy is evaluated on 120 trials, with 10 trials for each of the 12 material–task combinations and approximately matched initial configurations and fixed-end poses.Random seeds are identical across policies.Source: B. Compared Policies and Evaluation… · p. 4

C. Evaluation of the Material Estimation Network

• Estimator accuracy was measured on 120 held-out Oracle-policy rollouts spanning four materials and three tasks, using the latest 10 observations to predict each material label.Source: C. Evaluation of the Material… · p. 4
• The compared estimator training sources were human teleoperation, Task-only policy rollouts, and Specialist policy rollouts.All estimators used the same network architecture and were evaluated on Oracle-policy rollouts.Source: C. Evaluation of the Material… · p. 4
• The Specialist rollout dataset outperformed the other training datasets by more than 5 percentage points and exceeded human demonstrations and Task-only rollouts.Specialist rollout data was then used to train the final estimator.Source: C. Evaluation of the Material… · p. 4
• The smallest tested estimator model generalized best, reaching 96.26% material-estimation accuracy, and was used throughout.Variants removed attention pooling, increased dropout from 0.15 to 0.25, or reduced encoder and LSTM dimensions.Source: C. Evaluation of the Material… · p. 4

• 60.8% average success for Estimated was comparable to Oracle at 60.0%, despite no prior material information.Both are overall average success rates across material–task combinations.Source: D. Manipulation Results · p. 4
• Both Oracle and Estimated significantly outperformed Specialist at 41.7% and Task-only at 45.8%.Source: D. Manipulation Results · p. 4
• Estimated outperformed Oracle by 10 to 17 percentage points for Thick acrylic, Thin acrylic, and Cotton, but scored 40.0% versus Oracle’s 73.3% on Silicone.Source: D. Manipulation Results · p. 4

• Estimator errors were systematic, mainly within Silicone–Thin acrylic and Thick acrylic–Cotton pairs; separate unlabeled k-means clustering assessed material grouping across episode fifths.The representation examined was the vector pooled from per-step features before the final classification layer.Source: E. Analysis of Estimator and Policy… · pp. 4–5, Figure 7 · p. 5
• On Silicone/SV, Estimated scored 20% versus 100% for Oracle and 70% for both Specialist and Task-only; persistent confusion accompanied eight failures in ten trials.The estimator assigned Silicone an average probability of 0.32 in failed trials, versus 0.81 in the two successful trials, and assigned 0.66 to Thin acrylic; the sample was too small for a statistical test.Source: Figure 6 · p. 5, E. Analysis of Estimator and Policy… · pp. 5–6
• Commanded velocity changed by 0.82 mm/step with no condition movement, 0.97 mm/step without a top-probability change, and 1.78 mm/step when the top material changed.The latter changes were 1.17 and 2.17 times the no-movement change, respectively.Source: Figure 8 · p. 5, E. Analysis of Estimator and Policy… · p. 5

• On a real-robot groove-placement benchmark with four materials and three tasks, Oracle improved average success from 45.8% to 60.0% over Task-only, while Estimated reached 60.8% without prior material information.Source: §VI · p. 6
• Post-hoc analysis indicates that the estimator extracts material-related information from observations, the policy changes actions with the conditioning signal, and persistent confusion is associated with the pronounced failure case.Source: §VI · p. 6
• Future work includes generalizing to unseen materials and replacing discrete labels with interpretable material representations, such as friction and elasticity dimensions.Source: §VI · p. 6