PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation

Teli Ma, Jiaming Zhou, Zifan Wang, Ronghe Qiu, Junwei Liang

arXiv:2406.09738v1cs.ROcs.CV

TL;DR

Language-guided manipulation requires robots to distinguish task instructions and visual cues, despite limited simulated rewards and real-world data. Σ-agent integrates contrastive imitation learning and multi-view querying into an end-to-end agent, improving results across 18 RLBench tasks and achieving 62% success across five real-world tasks.

  • Problem

    Language-guided manipulation requires agents to distinguish linguistic commands and visual cues, while training is constrained by limited simulated rewards and real-world data.

    Source: §1 · p. 1, §3.4 · p. 5

  • Method

    Σ-agent integrates end-to-end contrastive imitation learning to refine feature extraction and vision-language interaction, using MVQ-Former for representative semantic aggregation.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Σ-agent improves average success rate by 5.2% and 5.9% over 18 RLBench tasks with 10 and 100 demonstrations, respectively, and achieves 62% average success across five real-world tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Contrastive IL also improves PolarNet and RVT, supporting the paper's claim of applicability across models and environments.

    Source: §1 · p. 2, §4.1 · p. 7

    The policy-learning setup relies on expert-policy demonstrations and behavior cloning rather than specific rewards.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Developing robots capable of executing various manipulation tasks, guided by natural language instructions and visual observations of intricate real-world environments, remains a significant challenge in robotics. Such robot agents need to understand linguistic commands and distinguish between the requirements of different tasks. In this work, we present Sigma-Agent, an end-to-end imitation learning agent for multi-task robotic manipulation. Sigma-Agent incorporates contrastive Imitation Learning (contrastive IL) modules to strengthen vision-language and current-future representations. An effective and efficient multi-view querying Transformer (MVQ-Former) for aggregating representative semantic information is introduced. Sigma-Agent shows substantial improvement over state-of-the-art methods under diverse settings in 18 RLBench tasks, surpassing RVT by an average of 5.2% and 5.9% in 10 and 100 demonstration training, respectively. Sigma-Agent also achieves 62% success rate with a single policy in 5 real-world manipulation tasks. The code will be released upon acceptance.

1 Introduction

The paper targets language-guided 3D manipulation, where agents must distinguish task instructions and visual cues amid limited simulated rewards and real-world data. It introduces end-to-end contrastive imitation learning and Σ-agent, reporting gains on RLBench and real-world tasks.

1 Introduction

  • Prior 3D manipulation methods emphasize perception but do not discuss aligning visual representations with language or differentiating among multiple tasks.The paper notes work on transferring 2D pre-trained representations, while 3D vision learning dominates instruction-guided manipulation to preserve geometric detail.Source: §1 · p. 1
  • Contrastive IL adds a training branch to behavior cloning that refines feature extraction and vision-language interaction by aligning positive pairs.The paper reports that contrastive IL produces more obvious separation between features belonging to different tasks in a t-SNE visualization.Source: §1 · p. 2, Figure 1 · p. 2
  • Σ-agent follows RVT, uses re-rendered virtual images from RGB-D reconstruction, and introduces MVQ-Former to aggregate representative semantic information with fewer tokens.The contrastive learning modules do not change the inference process.Source: §1 · p. 2
  • Σ-agent improves average success rate by 5.2% and 5.9% over 18 RLBench tasks with 10 and 100 demonstrations, respectively, and averages 62% success across five real-world tasks.The RLBench setting uses one policy for 18 tasks with 249 variations; the real-world result also uses a single policy.Source: §1 · p. 2

2 Related Work

Prior language-conditioned manipulation work develops 3D representations and policies across diverse benchmarks, while contrastive-learning research has connected representation objectives to reinforcement learning. The paper positions its approach against these lines of work and GRIF.

2.1 Language-conditioned Robotic Manipulation

  • RLBench offers hundreds of challenging tasks and variants spanning object poses, shapes, colors, sizes, and categories for evaluating agents from RGB-D cameras.The paper focuses on this benchmark among several language-conditioned manipulation benchmarks.Source: §2.1 · p. 3
  • Prior RLBench methods use voxel, point-cloud, or re-rendered virtual-view representations; Σ-agent follows RVT's virtual-view image approach.Examples include coarse-to-fine voxel methods, point-cloud policies, and RVT's Transformer processing of re-rendered views.Source: §2.1 · p. 3

2.2 Contrastive Learning in RL

  • Contrastive RL work combines representation learning with reward-updated reinforcement learning; GRIF instead uses contrastive learning to align linguistic representations with collected trajectory transitions.The paper identifies GRIF as a particularly similar work.Source: §2.2 · p. 3
  • Unlike GRIF's two-phase pretraining and single-policy RGB setting, contrastive IL trains end-to-end for 3D multi-task manipulation and refines both feature extraction and vision-language interaction.GRIF contrasts state-goal pairs with language instructions; contrastive IL refines feature extraction and vision-language interaction.Source: §2.2 · p. 3

3 Method

Σ-agent combines behavior cloning with contrastive objectives that refine visual and joint vision-language representations. Its architecture uses re-rendered views, language features, and multi-view queries to predict actions, with contrastive training disabled at inference.

3.1 Preliminaries

  • Behavior cloning learns a language-conditioned policy from expert transitions, using aligned RGB and depth inputs from front, shoulder, and wrist views.The agent uses RVT-style re-rendered virtual images from RGB-D inputs.Source: §3.1 · pp. 3–4

3 Method

  • The Σ-agent pipeline adds contrastive IL to imitation learning to refine visual representation ϕ and joint vision-language representation δ during training.The contrastive module does not change inference, and the future-state visual encoder shares parameters with the current-state encoder.Source: §3 · p. 3, Figure 2 · p. 4

3.2 Visual and Language Encoder

  • The visual encoder processes five re-rendered virtual views, while a frozen pre-trained CLIP language encoder projects text tokens for cross-attention with visual tokens.Each visual view contains RGB, depth, and coordinate channels, and is split into 20 × 20 patches for a two-layer self-attention Transformer.Source: §3.2 · p. 4

3.3 Multi-View Query Transformer (MVQ-Former)

  • MVQ-Former uses five learnable queries, one per virtual view, to aggregate visual information and reduce tokens before feature fusion and action decoding.The queries and visual and language tokens pass through four self-attention layers; the resulting context features feed the action decoder.Source: §3.3 · pp. 4–5

3.4 Contrastive Imitation Learning

  • Contrastive IL aligns state-language pairs and joint state-language representations with future goals, refining feature extraction and vision-language interaction.Positive goal states follow target-policy transitions; negative samples include other tasks in the batch, and the contrastive module is disabled during inference.Source: §3.4 · pp. 5–6

4 Experiments

Simulation experiments compare Sigma-agent with state-of-the-art methods and test contrastive-learning components; real-world experiments evaluate a single policy across five tasks.

4.1 Simulation Experiments

  • 3.3% and 2.3% higher success rates than Act3D and ChainedDiffuser, respectively, with 5x less training time.These results use 256×256 inputs and 100 evaluation episodes per task.Source: Table 2 · p. 6, §4.1 · pp. 6–7
  • 5.2% and 5.9% average success-rate gains over previous methods in RLBench’s 18-task, 10- and 100-demonstration settings, respectively.The comparison evaluates success rates on 25 episodes per task.Source: §4.1 · p. 6, Table 1 · p. 7
  • Adding contrastive IL improves PolarNet and RVT average success rates by 2.8% and 1.8%, respectively, across 18 tasks.Most tasks improve: 13 of 18 for PolarNet and 11 of 18 for RVT, with a largest margin of 13.9%.Source: Table 3 · p. 7, §4.1 · p. 7
  • Current-observation language and goal contrastive learning improves performance and early convergence; larger batches add negatives, and λ=0.5 is relatively optimal.The tested λ values from 0 to 1 produced slightly different results.Source: §4.1 · pp. 7–8, Figure 3 · p. 8

4.2 Real-world Experiments

  • A single policy trained with 10 demonstrations per task achieves 62% average success across five real-world tasks and nine variants.The evaluation uses a 6-DoF UR5 arm; the authors note aiming limitations from a single front-view camera and grasp-pose errors that worsen collisions.Source: Table 4 · p. 8, §4.2 · p. 8

5 Conclusions and Limitations

The paper presents contrastive IL and Σ-agent for language-guided multi-task 3D manipulation, while identifying sim-to-real discrepancy and behavior-cloning demonstration demands as limitations.

5 Conclusions and Limitations

  • Contrastive IL is a plug-and-play strategy that refines feature extraction and interaction within imitation learning for language-guided multi-task 3D manipulation.The end-to-end Σ-agent uses re-rendered virtual images from RGB-D input and is reported effective in simulated and real-world experiments.Source: §5 · p. 8
  • A large RLBench-to-real-world discrepancy limits the meaning of simulated training and leads to sim-to-real transfer failure.Source: §5 · p. 8
  • Behavior-cloning policies require human demonstrations, which are especially complicated to collect in real-world experiments and limit scaling to many tasks.Source: §5 · p. 8

A.1 RLBench Tasks

The supplied material describes the RLBench and Ravens evaluation settings, visualizes language-observation alignment, and specifies the real-robot setup.

A.1 RLBench Tasks

  • RLBench evaluation follows PerAct and RVT on 18 tasks with 249 variations, with task examples paired to human instructions.Source: §A.1 · p. 14, Figure 4 · p. 15

A.2 Effectiveness in Other Simulated Environments.

  • Σ-agent and contrastive IL improve over CLIPort on all six Ravens tasks.The Ravens evaluation uses 100 demonstrations and 100 evaluation instances per task.Source: §A.2 · p. 14

A.3 Visualization.

  • Contrastive IL amplifies alignment between language instructions and local visual observations, where the no-contrastive agent confuses close-jar instructions with screw-bulb and sort-shape instructions at the third keyframe.The visualization compares cosine similarities between trajectory observations and task instructions.Source: §A.3 · pp. 14–15, Figure 5 · p. 16

B.1 Hardware Setup

  • The real-world setup uses a 6-DoF UR5 arm with an overhead, forward-facing Orbbec Femto Bolt RGB-D camera, calibrated to the arm.The camera streams at 1280 × 960 and images are resized to 640 × 480 for model input.Source: §B.1 · pp. 15–16, Figure 6 · p. 16

B.2 Data Collection

The real-world data collection uses manually taught robot demonstrations across five manipulation tasks, with ten demonstrations per task.

B.2 Data Collection

  • Human demonstrations are collected by manually dragging the arm to predefined keypoints while recording visual observations and gripper poses.Source: §B.2 · p. 16
  • The five real-world tasks are Stack cups, Put fruit in plate, Hang mug, Put item in barrel, and Put tennis in mug.Source: §B.2 · p. 16
  • The dataset contains 10 manually collected demonstrations for each task.Source: §B.2 · p. 16

B.3 Training and Evaluating

The section reports the real-world training and evaluation setup and details how the agent decodes end-effector actions from fused visual features.

B.3 Training and Evaluating

  • Σ-agent is trained from scratch on 50 demonstrations across five tasks, with translation and yaw perturbations, and evaluated for 10 episodes per task.The real-world task set has five tasks and nine variants in total.Source: §B.3 · p. 16, Table 7 · p. 17

C.1 Action Prediction

  • The decoder predicts end-effector translation from the highest-scoring 3D point projected from five-view heatmaps, and predicts rotation, gripper state, and collision permission from fused context features.Source: §C.1 · p. 17
  • Rotation is classified into 216 categories using 5° Euler-angle bins, while gripper and collision states are binary predictions.Action prediction uses cross-entropy for translation and rotation, binary classification losses for gripper and collision states, and contrastive loss for representation learning.Source: §C.1 · pp. 17–18

C.2 Q-Function Analysis

The analysis relates the action-value Q-function to the probability of reaching the next state under the language-conditioned policy. A contrastive representation objective can serve as an extra critic to support policy imitation.

C.2 Q-Function Analysis

  • Q(at|st, l) is equivalent to the transition probability from state st under instruction l to the next state st+1 under the discounted state occupancy measure.The paper uses this relationship to connect action-value estimation with state transitions.Source: §C.2 · p. 18
  • Training fδ,φ brings matching language-state and next-state pairs closer while pushing negative pairs apart, aiming to maximize Pπ(st+1|st, l).The positive pair is [(st, l), st+1].Source: §C.2 · p. 18
  • The representation function fδ,φ can act as an extra critic that facilitates policy π mimicking target policy π+ at the representation-learning level.The passage links this role to maximizing Qπ(at|st, l) through the next-state transition objective.Source: §C.2 · p. 18

D Related Multi-Task Learning in RL

Multi-task learning faces conflicting task representations and gradients, while language instructions can help differentiate task representations in policy learning. This work uses contrastive learning to amplify the distinguishing role of linguistic hints.

D Related Multi-Task Learning in RL

  • Conflicting representations and gradients among tasks are identified as a central challenge in multi-task learning.Prior strategies include knowledge transfer, representation sharing, and gradient surgery.Source: D Related Multi-Task Learning in RL · p. 18
  • Language instructions are described as complementary hints for differentiating task representations in policy learning.The passage places this approach alongside the advent of large vision-language models and LLMs.Source: D Related Multi-Task Learning in RL · p. 18
  • The paper leverages contrastive learning to amplify the distinction function of linguistic hints.Source: D Related Multi-Task Learning in RL · p. 18
Loading 2406.09738v1…