Source-linked AI summary
PS-DLO: Property- and Spatially Aware Deformable Linear Object Manipulation Policy
Mengdi Li, Boxian Yao, Xiyu Zhu, Yifan Yang, Yunfan Lou, Xuan Han, Tao Liu, Zhibo Pang
TL;DR
DLO manipulation requires locating interaction-relevant regions while adapting actions to object-dependent physical properties. PS-DLO combines categorical property conditioning with distillation of SAM3-guided future spatial features into the action expert. Across nine real-world tasks, it achieves a 68.9% average success rate, 20.0 percentage points above π0.5.
Problem
DLO’s slender geometry, crossings, and self-occlusions complicate localization, while action-only VLA supervision leaves spatial and physical cues largely implicit.
Source: §I · p. 1Method
PS-DLO conditions action generation on categorical DLO property descriptions and distills SAM3-guided features from future observations into the policy’s action expert.
Source: §I · p. 1Results
68.9% average success rate across nine real-world tasks exceeds π0.5’s 48.9% by 20.0 percentage points.
Source: C. Main Results and Analysis · p. 5Takeaways & Limitations
PS-DLO achieves higher success rates than the baseline in six of seven tested butterfly-knot visual-shift conditions, matching it in the remaining condition.
Source: §I · p. 2Elastic-rope hooking is the only task where PS-DLO does not achieve the highest success rate, reaching 20.0% compared with DreamZero’s 30.0%.
Source: C. Limitations on Elastic-Rope Hooking · p. 8
Abstract
from arXiv · showhide
Deformable linear objects (DLO) pose a coupled perception and control challenge: their slender geometry makes precise localization difficult, while their deformations depend on object-specific physical properties. To address this challenge, we propose PS-DLO, a property- and spatially aware DLO manipulation policy. First, contrastive image-text learning maps the initial observation to categorical descriptions of DLO properties, providing explicit physical and structural priors for action generation. Second, SAM3 extracts mask-guided features from future observations, which are distilled into the policy's action expert to encode task-relevant DLO regions and their subsequent configurations. We evaluate PS-DLO on nine real-world tasks across single-arm and bimanual platforms, training a shared multi-task policy for each embodiment. PS-DLO achieves an average success rate of 68.9%, exceeding the strongest baseline by 20.0 percentage points. In generalization experiments, PS-DLO achieves higher success rates than the baseline in six of seven visual-shift conditions on butterfly-knot untying. These results demonstrate that combining object-property priors with future spatial supervision improves multi-task DLO manipulation and robustness to visual variation.
I. INTRODUCTION
PS-DLO addresses DLO manipulation’s coupled localization and property-adaptation challenges by combining object-property conditioning with future spatial representation distillation. Across nine real-world tasks, it outperforms the strongest baseline by 20.0 percentage points on average.
I. INTRODUCTION
- Action-only VLA supervision leaves spatial and physical cues largely implicit, motivating explicit guidance to attend to interaction-relevant regions and incorporate object-dependent physical information.DLO slenderness, crossings, and self-occlusions make relevant segments difficult to distinguish, while property differences can require different actions for the same goal.Source: §I · p. 1
- Categorical property descriptions inferred from initial observations provide explicit physical and structural information for conditioning action generation.Contrastive image–text learning aligns observations with property descriptions, which are combined with the task prompt throughout the episode.Source: §I · pp. 1–2
- SAM3-guided features from future observations are adapted by an action-supervised teacher and distilled into learnable DLO queries in the student policy’s action expert.A teacher assistant is optimized with a separate auxiliary action head, and an EMA of its encoder updates the teacher; these components and future observations are required only during training.Source: §I · pp. 1–2
- Across nine real-world DLO manipulation tasks, PS-DLO achieved an average success rate of 68.9%, exceeding the strongest baseline by 20.0 percentage points and posting the highest success rate on eight tasks.The policy was learned separately for bimanual and single-arm platforms.Source: §I · p. 2
- PS-DLO outperforms the baseline in six of seven butterfly-knot visual-shift conditions and matches it in the remaining condition.With rope color, lighting, and background all changed, PS-DLO achieves 10.0% success versus 0.0% for the baseline.Source: §I · p. 2
II. RELATED WORK
Prior DLO work spans task-specific geometric, physical, tactile, and planning approaches, while VLA research explores multi-task control and spatial or predictive representations. PS-DLO builds on these directions with mask-guided future supervision and categorical physical-semantic conditioning.
A. DLO Manipulation
- Prior DLO methods use geometric descriptors, deformation models, tactile primitives, cable tracing, and property adaptation, while simulation platforms support broader task coverage and multi-task VLA training.Examples include property-conditioned insertion and hierarchical routing that combines VLM planning with learned and predefined skills.Source: A. DLO Manipulation · p. 2
B. Representation Learning for VLA Policies
- VLA policies adapt pretrained vision-language representations for control, with approaches incorporating object geometry or task-conditioned affordances through visual representation supervision.π0 and π0.5 differ in how they incorporate proprioceptive state: action-expert tokens versus a textual input prefix.Source: B. Representation Learning for VLA… · p. 2
- Predictive-representation methods use future masks, video-expert distillation, or alignment with action-aware future embeddings; PS-DLO combines next-frame SAM3 supervision with CLIP-derived DLO property descriptions.The cited approaches differ in whether future-prediction branches remain at deployment or target embeddings are computed at inference.Source: B. Representation Learning for VLA… · p. 2
III. METHOD
The problem formulation treats DLO manipulation as multi-task control with a shared policy for each robot embodiment. From demonstrations, the policy learns to act across object-property and configuration variations.
III. METHOD
- The policy takes multiview RGB observations, a language task instruction, and the robot’s proprioceptive state as inputs.The number of camera views is experiment-dependent.Source: A. Problem Formulation · p. 2
- PS-DLO combines DLO property extraction and future spatial knowledge distillation, with three training stages: property-extractor adaptation, teacher pretraining, and joint teacher–student alignment.The property extractor conditions the policy on categorical descriptions; SAM3-guided features from future observations provide spatial supervision to the action expert through a teacher–assistant scheme.Source: Figure 2 · p. 3
- Each embodiment uses a shared vision-language-conditioned policy trained from demonstrations to perform diverse DLO tasks across property and configuration variations.The policy predicts an action chunk containing H consecutive actions.Source: A. Problem Formulation · p. 3
B. Framework Overview
PS-DLO combines property conditioning with spatially guided action learning. Its property extractor selects a categorical description from the initial observation and uses it to condition action generation.
B. Framework Overview
- PS-DLO adds property prompts and learnable DLO queries with a representation alignment head to a flow-matching VLA backbone.Training has three stages: cross-modal adaptation, teacher pretraining, and joint training.Source: B. Framework Overview · p. 3
- Contrastive image-text learning aligns initial images with property prompts, grounding object-specific descriptions for action conditioning.The extractor is frozen after adaptation during teacher pretraining and joint training.Source: B. Framework Overview · p. 3
- During teacher pretraining, mask-guided SAM3 features and task and property prompts condition an auxiliary action DiT, which is trained with flow-matching loss.The inputs also include proprioceptive state and a noisy action chunk.Source: B. Framework Overview · p. 3
- In joint training, the student learns demonstration actions while its intermediate DLO-query representations are aligned with stop-gradient teacher query representations.The TA encoder and auxiliary action DiT continue to receive flow-matching supervision, while the Teacher Encoder is updated only by EMA.Source: B. Framework Overview · p. 3
C. DLO Property Extraction
- Image-text similarity selects a categorical property description from candidates encoding object count, length, thickness, stiffness, extensibility, and initial configuration.The selected description is computed once per episode, remains fixed, and is combined with the task instruction and discretized state to condition actions; it is not a continuous measurement or dynamic state estimate.Source: C. DLO Property Extraction · pp. 3–4
D. Spatial Knowledge Distillation
The spatial distillation scheme uses future DLO observations during training to supervise the policy’s action expert. A teacher and assistant encode mask-guided spatial features and provide alignment targets for student representations.
D. Spatial Knowledge Distillation
- Future observations provide explicit spatial supervision: frozen SAM3 extracts DLO masks and visual features, and mask-covered patches retain their positions through positional embeddings.The features are combined with task and property text before passing through fusion Transformer blocks and a Q-former to produce 256-dimensional teacher tokens.Source: D. Spatial Knowledge Distillation · p. 4
- The Teacher Encoder and an eight-block conditional action DiT are pretrained for 20,000 updates to predict flow velocity for noisy demonstration action chunks.The DiT is conditioned on teacher tokens and current state.Source: D. Spatial Knowledge Distillation · p. 4
- Joint training initializes the TA from the pretrained Teacher Encoder and uses EMA-updated teacher targets while jointly optimizing the TA and auxiliary action DiT.The teacher follows the TA with EMA momentum m = 0.995 and provides temporally smoothed distillation targets.Source: D. Spatial Knowledge Distillation · p. 4
- The student optimizes action and future-alignment losses, while samples lacking an available future frame or valid DLO mask retain action supervision only.The future-alignment weight is λ = 0.1; the TA and auxiliary DiT receive gradients only from the auxiliary loss.Source: D. Spatial Knowledge Distillation · p. 4
A. Experiment Setup
The experiments evaluate shared multi-task policies on two robot embodiments and five DLO property dimensions. The dataset and evaluation protocol cover demonstrations, task-specific success criteria, and a common baseline training procedure.
A. Experiment Setup
- Experiments use a shared policy per embodiment across five property dimensions: bending stiffness, topology, extensibility, length, and thickness.The bimanual PiPER handles the first three categories; the single-arm Franka Research 3 handles length and thickness tasks.Source: A. Experiment Setup · p. 4
B. Data and Evaluation Protocol
- After cleaning, the dataset contains 500 demonstrations, with 100 each for straightening, knot untying, elastic-rope hooking, long-rope wrapping, and thick-rope threading followed by lifting.Knot-untying demonstrations are evenly divided across four knot types.Source: B. Data and Evaluation Protocol · p. 4
- All baselines use the same training data as PS-DLO, are fine-tuned for 10,000 optimization steps, and are evaluated using the checkpoint with lowest validation loss.Performance is monitored on the validation set during training.Source: B. Data and Evaluation Protocol · p. 4
- Trials last at most 60 seconds, and success is judged by task-specific criteria including complete knot removal, secure hook engagement, and stable lifting for 10 seconds.Other criteria include specified straightening angles and the required S-shaped wrapping path.Source: B. Data and Evaluation Protocol · pp. 4–5
- Representative sequences show five tasks across both platforms, with frames ordered chronologically from left to right.The upper three rows show PiPER tasks; the lower two show Franka Research 3 tasks.Source: Figure 3 · p. 5
C. Main Results and Analysis
PS-DLO leads the nine-task benchmark overall, but performance varies by task; in the displayed examples, its action attention is more concentrated around DLO structures and knots.
Attribute recognition
- 99.4% attribute-recognition accuracy was measured on 500 held-out images, and the metric covers categorical properties rather than continuous physical estimates.A prediction counted as correct only when all predicted attributes matched the annotation.Source: C. Main Results and Analysis · p. 5
Straightening, knot-untying and elastic-rope hooking
- On PiPER, PS-DLO reaches 60.0% and 80.0% on cable and rope straightening, and averages 80.0% across four knot-untying tasks versus π0.5’s 57.5%.Its success rates on single-knot, bow-tie, butterfly-knot, and figure-eight-knot untying are 75.0%, 70.0%, 90.0%, and 85.0%.Source: C. Main Results and Analysis · p. 5
Overall comparison
- 68.9% average success rate puts PS-DLO 20.0 percentage points above π0.5, the strongest baseline evaluated on all nine tasks.PS-DLO has the highest success rate on eight of nine tasks.Source: C. Main Results and Analysis · p. 5, Table I · p. 6
Straightening, knot-untying and elastic-rope hooking
- Elastic-rope hooking is the only task where PS-DLO is not best: its 20.0% success rate trails π0.5 at 25.0% and DreamZero at 30.0%.The gains are not uniform across tasks.Source: C. Main Results and Analysis · p. 6
Wrapping and threading tasks
- On Franka, PS-DLO reaches 80.0% on long-rope wrapping and 60.0% on thick-rope threading and lifting, exceeding π0.5 by 5.0 and 30.0 percentage points.π0.5 already achieves 75.0% on long-rope wrapping.Source: C. Main Results and Analysis · p. 6
Action attention visualization
- In the displayed examples, PS-DLO’s action attention is more concentrated around cables and knots than π0.5’s scattered high-response regions.The comparison uses the same observations and SAM3 masks as cable-location references.Source: C. Main Results and Analysis · p. 6, Figure 4 · p. 6
D. Ablation Study
The ablations compare property-prompt integration, SAM3-based spatial supervision, and teacher-target updates against π0.5 and other configurations.
Property fusion strategy
- Among property-only variants, suffix conditioning reaches 56.1%, versus 50.0% for prefix and 49.4% for joint text conditioning, indicating sensitivity to integration strategy.The baseline is 48.9%; suffix conditioning improves success by 7.2 percentage points.Source: D. Ablation Study · p. 6
SAM3-based spatial supervision
- Future-frame SAM3 supervision reaches 57.2%, 8.3 percentage points above baseline; adding current-frame SAM3 supervision to joint text conditioning reaches 61.1% versus 49.4% for joint text conditioning alone.The full method uses the next recorded observation Ot+1, whereas the current-frame variant uses Ot.Source: D. Ablation Study · pp. 6–7
Teacher-target update strategy
- The full teacher-update design reaches 68.9%, exceeding the fixed pretrained-teacher variant’s 63.3% by 5.6 percentage points, but this comparison does not isolate EMA smoothing.The full method combines online TA optimization through auxiliary action supervision with EMA updates to the Teacher Encoder.Source: D. Ablation Study · p. 7
E. Generalization Study
PS-DLO is compared with π0.5 on butterfly-knot untying under visual shifts, and a separate teacher-update comparison tests a combined online-TA and EMA design.
A. Teacher–Assistant Design
- The full teacher-update design reaches 68.9% versus 63.3% with a fixed pretrained teacher, evaluating online TA optimization and EMA together rather than isolating EMA smoothing.The TA refines representations through action prediction while the teacher follows its parameters through EMA.Source: A. Teacher–Assistant Design · p. 7
E. Generalization Study
- PS-DLO has higher success rates than π0.5 in six of seven shifted conditions, matching it under background-only change; under all three changes, it succeeds in 2 of 20 trials (10.0%).The study varies rope color, lighting, and background individually and in pairwise and three-factor combinations; success rates are low under the three-factor condition.Source: Table III · p. 7, E. Generalization Study · p. 7, Figure 5 · p. 7
B. Interpretation of Generalization Results
PS-DLO retains an advantage under most tested visual shifts, but performance remains sensitive to harder changes and elastic-rope hooking. The conclusion reports a 20.0-percentage-point average-success-rate gain over the best baseline across nine tasks.
B. Interpretation of Generalization Results
- PS-DLO outperformed π0.5 in six of seven butterfly-knot visual-shift conditions, with 15.0–20.0-percentage-point gains under pairwise color, lighting, and background changes.The proposed explanation is that SAM3-based spatial supervision may help retain DLO-relevant features despite scene-appearance changes.Source: B. Interpretation of Generalization… · p. 7
- Under changes to all three visual factors, PS-DLO succeeded in only 2 of 20 trials; under background-only change, it fell from 90.0% to 70.0%, matching π0.5 at 70.0%.Higher success rates do not necessarily mean smaller performance drops; the authors conclude that PS-DLO remains sensitive to visual variation.Source: B. Interpretation of Generalization… · p. 8
C. Limitations on Elastic-Rope Hooking
- On elastic-rope hooking, PS-DLO reached 20.0% success versus DreamZero’s 30.0%, and the comparison does not isolate the effect of future prediction.The task also requires handling elastic recoil and maintaining hook engagement after release; the authors suggest extending spatial supervision to predict deformation and contact transitions.Source: C. Limitations on Elastic-Rope Hooking · p. 8
VI. CONCLUSION
- 68.9% average success across nine real-world tasks exceeded the best baseline by 20.0 percentage points, although elastic-rope hooking and combined visual changes remained challenging.The paper identifies future work on these failure modes and evaluation on more diverse tasks and object properties.Source: §VI · p. 8