PaperlayerView exact revision on arXiv

Source-linked AI summary

CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

Oier Mees, Lukas Hermann, Erick Rosete-Beas, Wolfram Burgard

arXiv:2112.03227v4cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robots need to connect unconstrained human language with perception and action while composing diverse long-horizon manipulation skills. CALVIN provides an open-source simulated benchmark spanning multimodal sensing, continuous control, varied environments, and language instructions. Its MCIL baseline performs poorly on long-horizon tasks, exposing substantial room for improved language-conditioned agents.

  • Problem

    Existing robots typically learn individual tasks one at a time and lack robust generalization from language to diverse, long-horizon manipulation behaviors.

  • Method

    CALVIN is an open-source simulated benchmark linking unconstrained language to robot skills, multimodal observations, continuous control, and manipulation across four environments.

  • Results

    MCIL performs poorly on CALVIN’s long-horizon tasks, achieving 0.08% success on five-instruction chains despite stronger short-horizon performance.

  • Takeaways & Limitations

    CALVIN provides a modular benchmark for developing and validating agents that relate language to perception and action across long-horizon manipulation tasks.

Abstract

from arXiv · show

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this paper, we present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments and objects. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

I. INTRODUCTION

CALVIN addresses the challenge of composing long-horizon robot manipulation tasks from unconstrained language by providing an open-source simulated benchmark. Its baseline evaluation shows that multi-context imitation learning struggles with these tasks.

  • Natural-language task descriptions can abstract across unseen entities and specify complex tasks as instruction sequences, capabilities current robots typically lack.
  • CALVIN links human language to robot motor skills, behaviors, and objects while requiring agents to execute arbitrary combinations of subtasks in sequence.The benchmark supports training, prototyping, and validation across four indoor manipulation environments, with approximately 24 hours of teleoperated play data and 20K language directives.
  • 53.9% success is achieved by MCIL on short-horizon tasks, but the approach performs poorly on CALVIN’s long-horizon manipulation setting.The paper presents MCIL as a baseline using relabeled imitation learning to distill reusable behaviors into a goal-directed policy.
  • The benchmark is intended to help standardize evaluation of language-driven robotics and support reproducible research on language-conditioned continuous-control policies.The authors describe CALVIN as a modular framework for training, prototyping, and validation.
  • CALVIN targets language grounding across general-purpose robot skills, extending beyond prior work focused on grounding objects or spatial relations.
  • Compared with ALFRED, CALVIN uses more subtasks, longer evaluation sequences, broader sensor support, and zero-shot evaluation in unseen environments and language settings.CALVIN contains 34 subtasks versus ALFRED’s seven predefined action primitives and evaluates five-instruction sequences versus four.

III. CALVIN

CALVIN defines a benchmark for long-horizon language-conditioned continuous control, organized around feasible instruction sequences, an environment suite, and a challenge framework.

  • CALVIN evaluates agents that follow consecutive unconstrained language instructions while solving long-horizon manipulation tasks in closed-loop continuous control.
  • The benchmark restricts evaluation to feasible task sequences achievable from a predefined initial environment state.
  • CALVIN consists of three components: the CALVIN Environment, CALVIN Dataset, and CALVIN Challenge.
  • A. The CALVIN Environment: Four structurally related environments contain a 7-DOF Franka Emika Panda arm, household-style manipulation fixtures, and colored geometric blocks.Textures and static-element positions vary across environments, while the desk, robot, and static camera remain in the same positions.

1) Observation and Action Space:

CALVIN supports multimodal observations and continuous robot actions for language-directed manipulation, with automatic detection of completed tasks.

  • CALVIN provides RGB-D observations from fixed and gripper cameras, vision-based tactile sensing, and continuous proprioceptive signals.
  • The agent performs closed-loop continuous control while following unconstrained language instructions for complex manipulation tasks.Actions are sent to the robot at 30 Hz, and the benchmark supports different action-space choices.
  • CALVIN defines 34 tasks that can be completed in each environment and automatically detects task completion from environment-state changes.The completion detector can provide sparse rewards for reinforcement learning and label variable-length offline sequences.

1) Unstructured Demonstrations:

CALVIN builds diverse training data from unstructured teleoperated play and procedurally assigns language instructions to meaningful skills.

  • Twenty-four hours of teleoperated play data were collected across four environments to cover the state space.
  • The play dataset contains approximately 2.4M interaction steps and 40M short-horizon windows for relabeled goal-conditioned imitation learning.The windows span 1–2 seconds and are generated from roughly six hours of data collection per environment.
  • Unstructured play covers a multimodal range of possible solutions, but some task instances occur less often when they depend on completing another task first.
  • CALVIN collects over 400 crowd-sourced instructions for more than 34 tasks and labels episodes procedurally from recorded environment states.Only sequences displaying meaningful skills receive language annotations under this scheme.
  • The benchmark combines open-ended manipulation with language conditioning, requiring agents to connect words such as object colors and locations to multimodal observations and action sequences.

1) Training and Test Environments:

CALVIN evaluates policies across three training–test environment combinations, culminating in zero-shot transfer to an unseen environment with changed textures and object locations.

  • Training and Test Environments: CALVIN offers single-environment, multi-environment, and zero-shot multi-environment training–test combinations with increasing generalization demands.The zero-shot setting trains in three environments and tests in the fourth unseen environment.
  • Training and Test Environments: Single Environment trains and evaluates the policy in the same environment, matching the setting of Lynch et al..
  • Training and Test Environments: Multi Environment trains in all four environments and evaluates in one, requiring generalization across textures and object locations while providing more data.
  • Training and Test Environments: Zero-Shot Multi Environment tests in an unseen room whose textures and static-element positions differ from training environments.The scene elements remain present across training environments but appear in different locations or with somewhat different visual appearances.

2) Evaluation Metrics:

CALVIN measures language-conditioned control across 34 manipulation tasks and five-task instruction chains, including novel language descriptions and sequential subgoal transitions.

  • Evaluation Metrics: Multi-Task Language Control evaluates generalization to 34 manipulation tasks using novel language instructions absent from training.Each task is evaluated with 10 rollouts from different starting states after resetting to a valid unseen demonstration state.
  • Evaluation Metrics: Long-Horizon MTLC evaluates five sequential tasks as subgoals, requiring the policy to transition between different language-specified subgoals.Only feasible sequences from a predefined initial state are retained after filtering cycles, redundancies, and similarities.
  • Evaluation Metrics: Each manipulation task is defined by a success criterion, such as rotating a block clockwise more than 60° around the z-axis without excessive x- or y-axis rotation.
  • Evaluation Metrics: Long-horizon evaluation conditions the policy on the current instruction and advances to the next subgoal only after the environment state indicator confirms success.

3) Sensor Combinations:

CALVIN supports sensor configurations for visuomotor control, combining RGB-D cameras, proprioception, and vision-based tactile sensing to study language-conditioned manipulation.

  • Sensor Combinations: The benchmark targets agents that relate language to onboard sensor observations while performing manipulation tasks in settings intended to capture real-world challenges.
  • Sensor Combinations: CALVIN supports RGB-D images from static and gripper cameras, proprioceptive information, and vision-based tactile sensing.The benchmark evaluates baseline agents using different combinations of these sensors.

IV. BASELINE MODELS

The baseline models language-conditioned manipulation as goal-directed imitation from play data, using contextual plans to produce actions from state and goal information. The described MCIL framework adapts this approach toward language-conditioned policies, while highlighting the difficulty of relabeling language goals.

  • Baseline Models: CALVIN agents jointly reason over perceptual and language input to produce sequences of low-level motor commands.
  • Baseline Models: MCIL learns a goal-reaching policy from unstructured teleoperated play data without predefined tasks by relabeling visited states as reached goals.The relabeled data pairs trajectories with goal states for imitation learning.
  • Baseline Models: The imitation objective trains a policy πθ(at | xt, xg) from short-horizon goal-image-conditioned demonstrations.
  • Baseline Models: MCIL addresses multimodal play behavior by encoding contextual demonstrations into a latent plan space with a sequence-to-sequence CVAE.Its decoder reconstructs actions conditioned on state, goal, and inferred plan, which is then followed in closed loop at test time.
  • Baseline Models: Language-conditioned policies cannot directly relabel visited states as natural-language goals because language and observation spaces are not equivalent.Prior work instead paired a small number of random windows with after-the-fact language instructions.

B. Implementation Details

The evaluation establishes MCIL baseline performance across sensor suites and environment settings, revealing severe failures on long-horizon and cross-environment generalization. The authors identify context dependence, perceptual grounding, and distribution shift as challenges requiring further methods.

  • Baseline evaluation: 53.9% success rate is achieved by the static-camera MCIL baseline in same-environment MTLC evaluation, with comparable performance after adding gripper-camera, depth, or tactile sensing.The evaluation covers all 34 manipulation tasks trained and tested on the same environment.
  • Failure analysis: MCIL performance depends significantly on the robot’s initial position, suggesting reliance on context rather than disentangling initial states from tasks.The authors associate this observation with possible causal confusion between proprioceptive information and target actions.
  • Long-horizon evaluation: 0.08% success rate is achieved on five-instruction chains by the best MCIL model in same-environment Long-Horizon MTLC evaluation.The model solves the first subtask from a neutral position in 48.9% of cases.
  • Failure analysis: Block manipulation is sometimes executed correctly, but the policy confuses red and blue block colors, exposing a difficult perceptual-grounding problem.The authors suggest aligning visual and language representations with auxiliary losses as one possible remedy.
  • Generalization: General performance drops significantly in multi-environment and zero-shot multi-environment settings that violate the usual independent-identically-distributed task assumption of imitation learning.The authors suggest domain adaptation, data augmentation, and greater use of depth inputs as possible ways to improve zero-shot generalization.

VI. CONCLUSION

CALVIN is presented as a public, modular benchmark combining language, multimodal perception, continuous control, and long-horizon manipulation across seen and unseen environments. Evaluation shows that a model competent on some subgoals still has poor overall success, highlighting the benchmark’s difficulty and the need for further agent development.

  • Benchmark contribution: CALVIN combines natural-language conditioning, multimodal high-dimensional inputs, 7-DOF continuous control, and long-horizon object manipulation in seen and unseen environments.The benchmark is intended to support training, prototyping, validation, reproducibility, and future task expansion.
  • Evaluation conclusion: The evaluated conditional sequence-to-sequence variational autoencoder is relatively competent at some subgoals but achieves poor overall success rates on CALVIN.The paper identifies diverse skill acquisition, object detection, referring-expression and action grounding, and task-agnostic continuous control as subproblems.
  • Implications: CALVIN targets agents that relate human language to perception and actions while generalizing abstract concepts to unseen entities.The paper presents this as a direction for future development of language-driven robotics.

APPENDIX

The appendix specifies how CALVIN tasks are detected and how language annotations are generated from recorded trajectories. It also lists task examples and their state-change success criteria.

  • Task evaluation: Task success is determined by comparing simulator states at the first and final frames of a sequence against task conditions.The environment is reset to those states to evaluate whether task conditions are fulfilled.
  • Task criteria: Rotate-right tasks require clockwise rotation exceeding 60° around the z-axis while x- and y-axis rotation remains below 30°.The listed tasks include red, blue, and pink blocks.
  • Task criteria: Rotate-left tasks require counterclockwise rotation exceeding 60° around the z-axis while x- and y-axis rotation remains below 30°.The listed tasks include red, blue, and pink blocks.
  • Task inventory: CALVIN’s appendix lists all 34 tasks together with their respective success criteria.The task list provides the benchmark’s complete criterion inventory.
  • Language annotation: Language annotations are automatically extracted by sampling 64-frame windows whose task is solved between the first and last frames but not during the first half.This procedure includes locomotion behavior before the task itself.
Loading 2112.03227v4…