PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, Hongsheng Li

arXiv:2305.11176v3cs.ROcs.AI

TL;DR

Robotic systems need to execute complex, multimodal instructions while integrating perception, planning, and control. Instruct2Act uses an LLM to generate executable programs that invoke visual foundation models and controllers, achieving competitive zero-shot performance across tabletop manipulation tasks. The framework is flexible across instruction modalities, but its computational cost, primitive-action scope, and simulation-only evaluation limit current deployment.

  • Problem

    General-purpose robots must integrate perception, planning, and control to execute complex tasks, while existing policy-code generation can struggle with complex commands and depend on available perception APIs.

  • Method

    Instruct2Act uses an LLM to generate executable perception-to-action code that invokes foundation-model APIs, including SAM and CLIP, and drives robotic controllers.

  • Results

    Instruct2Act shows strong competitive zero-shot performance across six VIMABench meta-tasks and outperforms prior SOTA on several multistep tasks.

  • Takeaways & Limitations

    The framework provides a flexible general-purpose approach for mapping language and visual instructions to sequential robotic actions in tabletop manipulation.

  • Takeaways & Limitations

    The method has high computational cost, depends on basic action primitives, and has so far been tested only in simulation.

Abstract

from arXiv · show

Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing. This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks. Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks. In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them. In this way, the framework leverages the expertise of foundation models and robotic abilities to convert complex high-level instructions into precise policy codes. Our approach is adjustable and flexible in accommodating various instruction modalities and input types and catering to specific task demands. We validated the practicality and efficiency of our approach by assessing it on robotic tasks in different scenarios within tabletop manipulation domains. Furthermore, our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks. The code for our proposed approach is available at https://github.com/OpenGVLab/Instruct2Act, serving as a robust benchmark for high-level robotic instruction tasks with assorted modality inputs.

1 Introduction

Instruct2Act addresses the challenge of building general-purpose robots that integrate perception, planning, and control for complex tasks. It combines multimodal foundation models with LLMs to generate decision-making actions and reports competitive zero-shot performance across six VIMABench meta-tasks.

  • General-purpose robotic systems must perceive surroundings, select relevant skills, and sequence them to accomplish long-term goals in dynamic environments.
  • CaP generates robot-centric policy code from language examples but is limited by available perception APIs and struggles with longer, more complex commands.
  • Instruct2Act uses multimodality foundation models and LLMs to implement perception, planning, and low-level control, with SAM and CLIP locating and classifying objects.
  • Instruct2Act achieves strong competitive performance on six representative VIMABench meta-tasks across object manipulation, visual goal-reaching, and visual reasoning.
  • The framework generates middle-level decision-making actions from natural-language and visual instructions while supporting flexible modality inputs and minimal code overhead.

2 Related Works

Related work has explored language-conditioned planning, multimodal manipulation, visual representations, and foundation models for robotic decision-making. These approaches motivate combining language reasoning with visual and robotic capabilities.

  • Language-based robotic policies aim to provide user-friendly interfaces while supporting cross-task skill generalization and long-horizon reasoning.
  • Prior systems combine language models with visual grounding, admissible action mapping, or closed-loop feedback for robotic task execution.
  • Visual foundation models such as GLIP and SAM improve zero-shot transfer and diverse-task visual analysis through large-scale pretraining and prompt engineering.
  • Other methods use pretrained language or visual representations to initialize policies or enable data-efficient downstream robotic manipulation learning.

3 Methods

Instruct2Act uses an LLM to generate perception-to-action programs that combine visual foundation models, task instructions, and robotic controllers. Its architecture supports multimodal inputs through a unified retrieval system and API-guided program generation.

  • 3 Methods: Instruct2Act maps user instructions and top-view observations to action sequences that change object states toward the described configuration.
  • 3 Methods: Foundation-model APIs provide visual semantic information that the LLM combines with robotic skills to generate decision-making actions.
  • 3 Methods: The LLM generates executable Python functions that invoke perception APIs, reason over the environment, and drive low-level robot controllers.
  • 3.1 How to Drive Robotic by LLM: Prompts contain third-party libraries, API definitions, usage examples, and task instructions to guide robotic program generation.
  • 3.3 Perception with off-the-shelf Foundation Models: SAM produces candidate-object masks and CLIP encodes corresponding crops for classification, with image-processing modules mitigating shadows, holes, redundant masks, and zero-shot errors.
  • 3 Methods: The framework handles pure-language, pointing-language, and scene-level instructions through unified retrieval and multimodal task representations.

4 Experiments

Instruct2Act was evaluated across selected VIMABench tabletop tasks, generalization levels, prompt configurations, foundation-model backbones, and robustness scenarios. It showed competitive or improved performance, while ablations and limitations identified the roles of processing modules, prompt content, computational resources, and simulation scope.

  • Evaluation setup: The evaluation covered six representative VIMABench meta-tasks spanning object manipulation, visual goal-reaching, and visual reasoning, with L1–L4 generalization protocols.The benchmark included placement, combinatorial, novel-object, and novel-task generalization levels.
  • Evaluation setup: 150 instances per meta-task and three random seeds were used to compute average success rates, with simulator success determined by matching final configurations.Experiments used SAM ViT-H, CLIP ViT-H-14, ChatGPT text-davinci-003, and an NVIDIA 3090Ti GPU.
  • Main results: Instruct2Act achieved comparable average performance to VIMA and significantly outperformed it on multi-step Tasks 05 and 17 across single-modal and multimodal instructions.The authors attribute the gains to visual-model generalization and LLM reasoning, while reporting generally better performance for multimodal instructions.
  • Generalization: Across generalization evaluations, the method achieved positive results across levels and tasks, with foundation models producing minimal performance fluctuations under distribution shifts.Figure 5 reports this pattern for the evaluated L2 and L3 generalization tasks.
  • Prompt analysis: Providing both API definitions and in-context examples produced accurate, human-readable Python code, while examples enabled structured outputs and API information supported flexibility and reasoning.Without usage examples, the LLM could generate unnecessary actions or omit executed information; without functional information, it could produce incorrect comments and limited new logic.
  • Ablation studies: Processing ablations increased success rate from 51% without processing to 83.0% with mask post-processing and 84.1% with both mask post-processing and image pre-processing.Image pre-processing alone did not improve performance and could degrade some tasks because its predefined parameters were not tuned for generalization.
  • LLM comparison: Instruct2Act remained plausible with LLaMA-Adapter and reached 77.5% after increasing generation trials when the Python interpreter raised an exception.Comparable results were also obtained with basic filtering that regenerated code when environment-cache usage was missing.
  • Robustness: The method handled human intervention, missing characteristics or misspellings, and synonym replacement in separate robustness scenarios.Examples included stopping after an intervention instruction, grounding despite missing or misspelled characteristics, and interpreting “Spin” as “Rotate.”

5 Conclusion

Instruct2Act maps multimodal instructions to sequential robotic actions through LLM-generated policy code and visual foundation-model APIs. Processing modules help address zero-shot gaps, and experiments verify effectiveness and flexibility in robotic manipulation.

  • Instruct2Act uses an LLM to map multimodality instructions to sequential actions for robotic manipulation.
  • LLM-generated policy code invokes visual foundation models through APIs to obtain visual understanding of task sets.
  • Processing modules are added to mitigate gaps in the zero-shot setting.
  • Extensive experiments verify Instruct2Act’s effectiveness and flexibility in robotic manipulation tasks.

A.1 Full Prompts in Instruct2Act

The full prompt organizes Instruct2Act around file I/O, perception, action, hardware execution, examples, and an instruction-driven Python entry point. Its examples connect multimodal inputs to object perception, action planning, and robot execution.

  • First Level: File IO: The prompt stores multimodal instruction variables such as scene and object descriptions in templates and defines action-space bounds.
  • Second Level: Core Modules: Perception APIs obtain observations, segment objects with SAM, crop them, and retrieve target objects with CLIP.
  • Second Level: Core Modules: Action APIs map masks to locations and provide pick-and-place, distractor-removal, rearrangement, and robot-execution operations.
  • Examples: The prompt supplies examples for placing, rotating, rearranging, and restoring objects, including corresponding perception and action sequences.
  • Examples: A rearrangement example segments observed and goal scenes, matches objects, removes conflicts, composes actions, and executes them.
  • Task Execution: The task prompt instructs the LLM to solve the instruction step-by-step by implementing and outputting a Python-style main function.

A.2 Processing Module in Instruct2Act

Instruct2Act inserts preprocessing and post-processing modules between zero-shot foundation-model outputs and robotic execution. These modules target shadows, misleading masks, irrelevant outputs, holes, gaps, and redundant segments.

  • The execution loop comprises perception, planning, and execution, with processing modules inserted to address zero-shot model incompleteness or incorrectness.
  • Image Pre-Processing: Image preprocessing uses a gray threshold filter to mitigate shadows that can be mistaken for graspable objects.
  • Mask Post-Processing: Mask post-processing addresses discontinuous segments, missing object parts, and holes that can confuse subsequent robotic modules.
  • Mask Post-Processing: Size filtering removes outputs that are clearly not part of the target object, including immovable tables or patterns.
  • Mask Post-Processing: Opening morphology combines dilation and erosion to eliminate small holes or unconnected gaps while limiting mask-size changes.
  • Mask Post-Processing: Non-Maximum Suppression reduces redundant mask outputs when one object receives multiple segmentation results.

A.3 Pointing-Language Enhanced Instruction

Pointing-language mode lets users select target objects with cursor clicks that guide SAM segmentation. The enhanced mode performs better than pure-language instruction mode, while audio can be converted to text for the same process.

  • Pointing-language mode displays the instruction and observation image, then uses user-selected click points as SAM point prompts.
  • The mode was evaluated on Task 01 and Task 03 over 150 instances with one seed.
  • Pointing-language enhanced instruction achieves better results than pure-language instruction mode.
  • The reported improvement could result from stronger prior information supplied by user click operations.
  • For audio instructions, Whisper converts audio input into text before the standard Instruct2Act process generates policy code.

A.4 Evaluation Task Description

The evaluation covers tabletop manipulation tasks spanning simple object manipulation, visual manipulation, scene understanding, rotation, visual goal-reaching, and visual reasoning. Experiments use the first three VIMABench generalization levels, increasing from placement variation to novel object generalization.

  • Evaluation Tasks: The tasks include simple object manipulation, visual manipulation, scene understanding, rotation, visual goal-reaching, and visual reasoning.These tasks respectively test basic actions, object recognition and placement, scene-image grounding, z-axis rotation, goal-state matching, and decisions requiring reasoning or memory.
  • Visual Goal-Reaching: Rearrangement tasks require moving target objects to a goal configuration while avoiding position conflicts from distractors.The rearrange-then-restore variant additionally requires restoring object placements after rearrangement.
  • Visual Reasoning: Pick-in-order-then-restore tasks require sequentially placing a target object into different containers before returning it to its initial container.
  • Evaluation Protocol: VIMABench evaluation uses the first three of four progressively harder levels to test agent generalization.L1 varies target-object placement, L2 creates new combinations of materials and descriptions, and L3 tests novel objects.

A.5 Ablation Studies on Prompt Element

The ablation studies compare prompt configurations for generating robotic programs and examine how generated code performs perception, localization, action planning, execution, and failure handling. The examples show that generated programs can implement the requested pick-and-place pipeline, while prompt elements determine its structure and supporting logic.

  • Experimental Setup: The ablation compares repeated LLM outputs for the same pick-and-place instruction because generation is somewhat random.The shared instruction asks the robot to put a polka dot block into a green container.
  • In-Context Examples: In-context-example prompts can produce the same perception-to-action sequence, including image acquisition, object retrieval, location mapping, pick-and-place, and robot execution.The examples include generated functions that return execution information after applying the action.
  • API Definitions: API-definition prompts generate programs that obtain observations, segment the image, crop objects, retrieve target objects with CLIP, map them to locations, and execute PickPlace.The generated code also includes action bounds, execution results, and, in some trials, failure handling or result output.
  • Execution: Generated programs execute actions through RobotExecution and may save or print results, with some trials checking whether execution succeeded.The code listings show both direct execution and conditional handling for unsuccessful actions.
  • Combined Prompt: Combined prompt trials add explicit action-space choices such as pick and place locations, action boundaries, and distractor-action preparation.The generated program defines object lists, removes the goal object from observed masks, and prepares actions before execution.
Loading 2305.11176v3…