PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly

Jeon Ho Kang, Igal Tamarkin, Ethan Niu, Ian Novales, Satyandra K. Gupta

arXiv:2609.17714v1cs.RO

TL;DR

Long-horizon disassembly requires selecting ordered skills across multiple objects, while raw-observation imitation faces combinatorial configuration diversity and prior task-context evaluation lacked autonomous selection and unseen-configuration testing. The paper combines language-grounded imitation with hierarchical VLM task selection, and reports substantially better performance than existing methods alongside generalization across diverse configurations and held-out connector mechanisms.

  • Problem

    Raw-observation imitation faces combinatorial data demands across disassembly configurations, and prior task-context work relied on human online labels and a fixed assembly setup.

    Source: §I · p. 1

  • Method

    The framework combines counterfactual language-to-object grounding in a task-context-aware policy with hierarchical VLM-based online subtask selection.

    Source: §I · p. 1, A. Disambiguating Task Target in… · p. 3, C. Online Task Selection for… · p. 4

  • Results

    The method performs substantially better than existing methods and generalizes across diverse assembly configurations and held-out connector mechanisms.

    Source: B. End-to-End Task Success Rate… · p. 6, §VII · p. 8

  • Takeaways & Limitations

    Language-conditioned visual grounding and counterfactual supervision support target selection without explicit object annotations.

    Source: F. Key Findings · p. 7

    The current approach relies on carefully designed prompts and close-range visual inputs.

    Source: §VII · p. 8

Abstract

from arXiv · show

Real-world robotic disassembly requires long-horizon execution, where robots must perform ordered sequences of manipulation tasks across multiple parts within a single scene. Multiple valid task goals and diverse assembly configurations make it difficult for imitation policies to infer the intended skill from raw observations alone, particularly when training data cannot cover the combinatorial diversity of real-world configurations and part geometries. We show that incorporating task context through language alleviates these challenges by providing explicit structure for skill selection and associating language-specified tasks with their corresponding manipulation targets in the visual scene. The proposed framework combines hierarchical task selection with task-context-aware imitation learning to ground language instructions in spatial visual representations for robotic disassembly. The resulting framework generalizes across diverse connector geometries and assembly configurations without requiring explicit object annotations. Our method improves end-to-end task success by 35 percentage points over the baseline diffusion policy and by 75 percentage points over the previous task-context-aware baseline.

I. INTRODUCTION

Real-world disassembly poses long-horizon skill-selection challenges across multiple objects and assembly configurations, while raw-observation learning can require combinatorially diverse data. The paper combines language-grounded imitation learning with hierarchical VLM task selection to address these challenges without object-level annotations.

I. INTRODUCTION

  • Raw visual and force observations make training-data needs grow combinatorially with possible configurations, while long-horizon behavior cloning can absorb idle motions and cause temporal freezing.Removing pauses during data collection or filtering them manually is costly and can remove meaningful waits.Source: §I · p. 1
  • Prior work used human-provided online subtask labels and evaluated on a fixed NIST-board setup, leaving autonomous selection and generalization to unseen configurations unassessed.The paper adds automatic skill labeling and online subtask selection during execution.Source: §I · p. 1
  • Language-grounded imitation learning uses counterfactual supervision to associate task descriptions with target objects without explicit object annotations, including in held-out assembly configurations.A hierarchical VLM planner also selects semantic tasks for held-out configurations and connector mechanisms using the existing skill library.Source: §I · pp. 1–2
  • The evaluation covers connector assemblies with varied types, spatial arrangements, and held-out parts, extending beyond the settings considered in.The framework targets contact-rich disassembly in which manipulation primitives progressively release mechanical constraints.Source: §I · pp. 1–2

II. RELATED WORK

Prior work spans language-guided imitation, hierarchical planning, and counterfactual learning, but the paper identifies limited coverage of contact-rich disassembly and task-conditioned target disambiguation. It formulates multi-task disassembly as selecting skills for ordered subtasks and learning a policy conditioned on multimodal observations and language.

II. RELATED WORK

  • Language-guided imitation, diffusion policies, and vision-language-action models provide the related foundations for learning manipulation from demonstrations and incorporating semantic task information.The cited examples include OpenVLA and π0.5 [12].Source: §II · p. 2
  • Hierarchical manipulation methods – mainly target household tasks, while keypoint grounding is less applicable to small, visually similar, out-of-distribution industrial components.The paper identifies few studies of semantic task selection and language-guided object grounding for disassembly without explicit annotations.Source: §II · p. 2
  • Counterfactual methods – and contrastive language-conditioned learning do not address target disambiguation among visually similar components in contact-rich manipulation.This work adds counterfactual supervision to the diffusion objective without generating additional counterfactual trajectories.Source: §II · p. 2

III. PROBLEM FORMULATION

  • A parent task T_p is an ordered sequence of subtasks, each requiring a specific skill that the policy must select to complete the task.The policy is conditioned on proprioception, visual input, force measurements, and a language descriptor specifying the task object and subtask skill.Source: §III · p. 2

IV. METHOD

The method grounds task language in spatial visual features and uses counterfactual supervision to discourage incorrect language–object associations. Its policy combines attended visual features with subtask and force inputs for action prediction.

A. Disambiguating Task Target in Multi-task Scene with Target-object Grounding

  • Cross-attention aligns task language with spatial DINOv2 patches to ground domain-specific instructions among small, visually similar disassembly components.The architecture feeds attended visual features, subtask embeddings, and force inputs as tokens to the DiT for action prediction.Source: A. Disambiguating Task Target in… · p. 2, Figure 1 · p. 3
  • During training, correct and mismatched action–language MSE diverge while the counterfactual hinge loss converges toward zero.This loss behavior is presented as associating specialized manipulation skills with the intended target object.Source: Figure 2 · p. 3
  • Counterfactual supervision compares correct and incorrect object-conditioned predictions for the same action chunk, penalizing the incorrect-command prediction unless its error is worse than the correct-command prediction.The counterfactual context swaps only the object stream, leaving the subtask, image, force, and proprioception streams intact.Source: A. Disambiguating Task Target in… · p. 3
  • The counterfactual term is applied only to configured subtasks, defaulting to Approach, because later subtasks are physically committed to the target and do not exhibit the language-ignoring failure mode.The hinge reaches zero only when changing the object stream measurably degrades prediction; the margin prevents a trivial equal-prediction solution.Source: A. Disambiguating Task Target in… · p. 3

B. Task-Context-Conditioned Diffusion Transformer Action Head

The action head combines task and force representations through cross-attention, while causal masking restricts each action prediction to nonfuture information. Its attention patterns vary with task phase, with greater force attention during contact-rich pulling.

B. Task-Context-Conditioned Diffusion Transformer Action Head

  • Causal masking makes each action prediction at step t attend only to action steps ≤t and non-future state tokens.The mask applies over both the noisy action sequence and encoder–decoder attention.Source: B. Task-Context-Conditioned Diffusion… · p. 3
  • Without the proposed grounding mechanism, approach-phase attention maps under different language commands attend to seemingly unrelated regions.Source: Figure 3 · p. 4
  • Contact-rich pulling exhibits higher attention weights toward force inputs than the approach phase.Source: Figure 4 · p. 4
  • Cross-attention learns the relative importance of subtask and force representations, rather than using manually specified modality weights for each subtask.Prior work manually specified the relative importance of vision and force modalities.Source: B. Task-Context-Conditioned Diffusion… · p. 4

C. Online Task Selection for Long-Horizon Execution

A VLM selector proposes subtasks asynchronously while the low-level policy continues at a fixed control frequency. Its visual prompt can include structured robot-state information, and the framework evaluates cloud-based and locally deployed planners.

C. Online Task Selection for Long-Horizon Execution

  • The VLM planner is queried asynchronously while the low-level policy runs at fixed frequency, amortizing planning latency across multiple policy executions.Source: C. Online Task Selection for… · p. 4
  • The VLM-based selector uses semantic reasoning to choose subtasks for previously unseen components while reusing the existing skill library.Source: C. Online Task Selection for… · p. 4
  • The evaluated planners include cloud-based GPT-5 and locally deployed Qwen3-VL-8B; structured robot-state information in the visual prompt improves subtask prediction and enables failure recovery.The prompt can include force/torque measurements, gripper width, end-effector pose, and incremental motion encoded as natural language.Source: Figure 5 · p. 4

A. Testbed

The evaluation uses a custom robotic testbed and varied connector configurations, with complete disassembly demonstrations labeled by hierarchical task language. Results are organized around four questions about planner performance, grounding, and generalization.

A. Testbed

  • The testbed uses a KUKA LBR IIWA14 arm and custom setup inspired by NIST Task Board #1, with varied configurations including two identical connectors on one board.Source: A. Testbed · p. 4

B. Data Collection and Training

  • Demonstrations are collected through kinesthetic teaching while connector positions and surrounding configurations vary to promote generalization.Source: B. Data Collection and Training · p. 4
  • The dataset contains 102 USB, 118 D-sub, 115 terminal-block, and 121 BNC demonstrations; each trajectory runs from approach through connector extraction and is labeled with hierarchical language instructions.Training uses frozen CLIP and DINOv2 encoders, batch size 128, and approximately 22 hours on one RTX 4080 GPU.Source: Figure 6 · p. 5

VI. RESULTS

  • The results address VLM selection, end-to-end success and robustness, language-specified target identification without object annotations, and generalization to held-out connectors and assemblies.Source: §VI · p. 5

A. Evaluation of Vision–Language Models for Task Selection

The evaluation compares VLM task-selection accuracy and end-to-end policy variants across diverse configurations. GPT-5 achieves the highest reported online proposal accuracy, while the proposed method performs substantially better than existing methods in end-to-end comparisons.

A. Evaluation of Vision–Language Models for Task Selection

  • Offline labeling is evaluated on 694 samples from 10 trajectories across four connector types; GPT-5 is selected for its superior accuracy, while two Gemini models offer competitive accuracy with lower latency.Source: A. Evaluation of Vision–Language… · p. 5
  • GPT-5 achieves the highest online next-subtask proposal accuracy at 97.3%.Across 25 trajectories and 150 proposal queries, other models showed early or late completion predictions or difficulty advancing to pulling.Source: A. Evaluation of Vision–Language… · p. 5, Figure 7 · p. 5

B. End-to-End Task Success Rate Comparison

  • End-to-end success requires connector removal within 50 action steps, and multi-task policies must also select the instructed parent task correctly.Source: B. End-to-End Task Success Rate… · p. 6
  • The evaluation varies spatial arrangements, multiple identical connectors unseen during training, and connector geometry; held-out mechanisms and assemblies are tested only with CNN and VLM-V2 planners.Source: B. End-to-End Task Success Rate… · p. 6
  • The proposed method performs substantially better than existing methods in end-to-end comparisons across the evaluated configurations.The comparison includes the prior task-context-aware method, whose flattened language embedding provides no spatial correspondence to the scene; the authors attribute its low success to target-grounding failures when similar connectors coexist.Source: B. End-to-End Task Success Rate… · p. 6
  • Misalignment and incorrect target selection are the primary failure causes, while rule-based planning is especially prone to premature transitions or getting stuck on a subtask.Source: B. End-to-End Task Success Rate… · p. 6

C. CNN vs. VLM Planner Study

The study contrasts a low-latency CNN task selector with VLM semantic planning, finding their strengths diverge when connector mechanisms are held out. It also examines generalization and failure recovery using the existing skill library.

C. CNN vs. VLM Planner Study

  • The CNN classifies short multimodal observation sequences with a fine-tuned ResNet34 and temporal Conv1D classifier.The comparison characterizes prediction accuracy, inference latency, and generalization in in-distribution and held-out settings.Source: C. CNN vs. VLM Planner Study · p. 6
  • On held-out connector mechanisms, the VLM-based planner outperforms the CNN by 48 percentage points on average.This contrasts with their comparable in-distribution performance.Source: C. CNN vs. VLM Planner Study · p. 6
  • For in-distribution objects, the CNN achieves 92% subtask prediction accuracy; the planners score 0.93 vs. 0.94, while CNN inference latency is substantially lower.The CNN can be queried at the action-policy frequency, whereas the VLM planner is queried less frequently.Source: C. CNN vs. VLM Planner Study · p. 6
  • The CNN is limited to trained object categories and task transitions, whereas the VLM reasons over visual observations and structured robot state for semantic planning.The paper presents this as planning beyond closed-world task classification.Source: C. CNN vs. VLM Planner Study · pp. 6–7
  • The VLM-V2 and CNN were tested on four held-out connector mechanisms; failures mainly involved approach-phase gripper alignment on out-of-distribution geometries, not task selection.The held-out connectors could be manipulated with the existing skill library; connectors needing entirely new skills are outside the work's scope.Source: C. CNN vs. VLM Planner Study · pp. 6–7
  • Across ten induced failure cases, the VLM identified the failure condition and selected a recovery primitive, returning the manipulator to its previous pose before replanning.The VLM uses visual observations and structured robot-state information; heuristic recovery relies on manually specified rules.Source: C. CNN vs. VLM Planner Study · p. 7

D. Object Grounding Ablation

The ablation links better object selection to counterfactual supervision and cross-attention, while the task-switching comparison finds fewer transition steps with classifier-based policies than with the regular policy.

D. Object Grounding Ablation

  • Object-selection performance improves progressively with counterfactual supervision and cross-attention, isolating gains to the grounding mechanism rather than the frozen pretrained encoders.Without these components, the policy may move between objects or execute the commanded skill while grounding to the wrong target.Source: D. Object Grounding Ablation · p. 7

E. Number of Execution Prediction Steps Required: Regular Policy vs. Task-Context-Informed Policy

  • Table I reports success rates over 20 trials per connector and identifies BNC as most challenging; Table II reports held-out task success rates over 10 trials per connector.Source: Table I · p. 7, Table II · p. 7
  • The VLM-based method produces the fewest delays in predicting the next subtask; both classifier-based methods require fewer switching steps than the baseline, while the CNN requires slightly more steps than the VLM.The baseline tends to remain idle before committing to the next task.Source: Figure 8 · p. 7, E. Number of Execution Prediction… · p. 7

F. Key Findings

The key findings emphasize VLM planning and language-conditioned grounding for robust disassembly across configurations. The conclusion also identifies prompt design and close-range visual input as current constraints.

F. Key Findings

  • VLM-based planning improves task selection across held-out assemblies and connector mechanisms while supporting failure recovery through joint visual and robot-state reasoning.Source: F. Key Findings · p. 7
  • Language-conditioned visual grounding with counterfactual supervision supports target selection without explicit object annotations.The conclusion describes the combined framework as improving robustness and end-to-end task success across diverse configurations and held-out connector mechanisms.Source: F. Key Findings · p. 7, §VII · p. 8

VII. CONCLUSIONS

  • The current approach relies on carefully designed prompts and close-range visual inputs, partly because existing VLMs have limited visual reasoning capability.Future work will investigate larger skill libraries and generalization to novel language commands for autonomous skill discovery.Source: §VII · p. 8

Related papers

Loading 2609.17714v1…