PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Mengao Zhao, Ziang Li, Chaodong Huang, Mengchen Ma, Haoyi Jiang, Yiwei Jin, Xinjie Wang, Yun Du, Xuewu Lin, Taojun Ding, Hongyu Xie, Jackson Jiang, Chunlei Yu, Kaihua Zhang, Lichao Huang, Liu Liu, Tianwei Lin, Zhizhong Su

arXiv:2608.22990v1cs.RO

TL;DR

Many manipulation benchmarks allow policies to succeed through visual shortcuts, leaving whether they follow language unclear. InstructMove makes language indispensable by creating ambiguous pick-and-place scenes, and its experiments show diagnostic value plus improved real-world performance from simulation data, within stated simulation and protocol limitations.

  • Problem

    Many manipulation benchmarks leave instruction following underdetermined because visually salient or uniquely feasible targets can be selected without grounding the language instruction.

  • Method

    InstructMove constructs pick-and-place scenes with semantic distractors, four instruction-grounding task families, held-out evaluation tasks, and language-dependence diagnostics.

  • Results

    InstructMove provides a controlled testbed for diagnosing visual shortcuts, and simulation-data scaling improves real-world instruction-following manipulation performance.

  • Takeaways & Limitations

    The benchmark supports diagnosing whether policies ground action selection in language and studying transfer from scalable simulation demonstrations to real-world manipulation.

  • Takeaways & Limitations

    Simulation scores are not established to correlate with real-world performance, and same-family fine-tuning may encourage overfitting or obscure general instruction-following ability.

Abstract

from arXiv · show

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

1 Introduction

InstructMove addresses a gap in manipulation benchmarks by making language necessary for selecting among multiple plausible actions. It introduces a four-part task taxonomy and infrastructure for controlled training, evaluation, and diagnosis.

  • Motivation: Many manipulation benchmarks permit visual shortcuts because only one object or destination is semantically plausible or physically feasible.Such settings can leave instruction following underdetermined.
  • Motivation: Text-indispensable episodes contain multiple visually plausible and physically feasible actions, while language uniquely identifies the correct one.This design exposes language-ignoring behavior and supports interpretable stage-wise failure attribution.
  • Benchmark contribution: InstructMove evaluates pick-and-place instruction following through category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place.Semantic distractors make alternative objects or destinations plausible but instruction-inconsistent.
  • Contributions: The paper contributes a scalable benchmark, a systematic task taxonomy, and data infrastructure based on curated assets, attribute annotation, and constraint-based layout synthesis.These components support reproducible instruction-following manipulation evaluation.
  • Evaluation: The benchmark provides a train-eval protocol with stage-wise metrics and language-dependence diagnostics for VLA policies.Its infrastructure supports controlled data generation and held-out evaluation.

2 Related Work

Existing manipulation and spatial vision-language benchmarks support diverse capabilities, but they do not primarily test whether language is indispensable for choosing among multiple plausible executable actions. InstructMove targets this missing evaluation axis.

  • Existing benchmarks: Prior benchmarks cover task diversity, long-horizon control, knowledge transfer, spatial understanding, and sim-to-real manipulation.Examples include RLBench, CALVIN, VIMA, LIBERO, SimplEREnv-Instruct, RoboTwin, and spatial vision-language resources.
  • InstructMove: InstructMove differs by evaluating whether language is indispensable for selecting the correct executable action among visually plausible alternatives.This distinguishes its diagnostic focus from benchmark dimensions such as zero-shot evaluation or general task diversity.
  • Missing axis: These benchmarks generally do not construct episodes where language uniquely determines the choice among multiple visually and physically plausible alternatives.Their primary emphases lie elsewhere, even when scenes contain distractors.
  • VLA grounding: Recent VLA methods treat instruction following as grounding through preserved instruction content, object-level supervision, or explicit boxes, points, masks, and spatial priors.These approaches motivate diagnostic evaluation but are distinct from a closed-loop benchmark design.

3 InstructMove Benchmark Design

InstructMove formalizes language-grounded manipulation as selecting one instruction-consistent goal among multiple physically actionable candidates. Its pipeline generates semantically ambiguous scenes and demonstrations across four task families, then evaluates grounding and control separately.

  • 3.2 Benchmark Instantiation Pipeline: The benchmark pipeline aligns an asset library, task-and-layout generation, and expert trajectory synthesis.These components connect scene construction, constraint-consistent layouts, and executable training demonstrations.
  • 3.1 Problem Formulation: Each episode includes an instruction, a unique instruction-consistent goal, and a set of plausible candidate goals.Text-indispensability requires multiple physically actionable candidates while visual feasibility alone cannot identify the target.
  • 3.2 Benchmark Instantiation Pipeline: The asset and layout infrastructure uses diverse objects and constrained scenes so multiple candidates remain plausible while only one matches the instruction.The curated collection contains 3,253 objects, with 1,757 selected for benchmark instantiation.
  • 3.3 Capability Axes and Task Suites: Four task families test category grounding, attribute grounding, spatial grounding, and compositional pick-and-place under controlled ambiguity.The families vary the instruction cue and construct distractors around the same cue.
  • 3.4 Evaluation Protocol and Stage Metrics: The train-eval protocol tests generalization across held-out scene configurations, object instances, attributes, and categories.Stage metrics condition evaluation on instruction-consistent targets to separate semantic grounding from low-level control.
  • 3.4 Evaluation Protocol and Stage Metrics: For compositional pick-and-place, the score averages ReachPick, LiftPick, ReachPlace, and Place indicators equally.These staged measures localize whether failures arise during target grounding or subsequent manipulation control.

4 Experiments

Experiments evaluate four VLA policies on InstructMove’s robot-specific train-eval protocol, counterfactual language conditions, and real-world transfer. Results expose differences between target localization and manipulation completion, reveal visual shortcuts, and show benefits from simulation data.

  • Experimental setup: Four representative policies are evaluated with robot-specific multi-task training across pick category, pick attribute, pick spatial, and compositional place tasks.PiperX policies train on pick category, pick attribute, and place a2b, while Franka policies train on pick spatial and place a2b.
  • Experimental setup: The protocol uses compact balanced training sets, 5,000-step fine-tuning, and held-out evaluation episodes with splits for layouts, distractors, instances, and categories.Each task has 100 training episodes and each reported metric is evaluated over 100 held-out episodes.
  • Main evaluation: π0.5 achieves the strongest target localization on semantic pick tasks and is most robust on compositional place a2b, while GR00T N1.7 generally follows in Reach performance.π0.5 has the highest Reach on pick category and pick attribute, whereas π0 and Motus show moderate Reach performance.
  • Main evaluation: Reach–Lift gaps distinguish localization from execution: π0.5 converts reaching into lifting more reliably, whereas GR00T N1.7 often reaches the correct target but rarely completes the lift.In Franka pick spatial, the policies have comparable Reach rates, highlighting spatial-relation grounding as a bottleneck.
  • Generalization: On held-out instances and categories, π0.5 remains close to seen-split performance, while GR00T N1.7 retains Reach but shows substantially lower Lift after target localization.The results indicate generalization beyond memorized instances and categories, but continued difficulty converting localization into successful manipulation.
  • Language dependence: Counterfactual instructions redirect π0.5’s target selection but do not fully gate grasp initiation: alternative-target instructions yield 0.72 Target Reach and 0.53 Target Lift, while generic, empty, or absent targets produce 0.99–1.00 Any-object Reach.Any-object Lift is 0.64–0.66, although these episodes run for the full horizon and are not directly comparable to target-conditioned lift rates.
  • Real-world transfer: Simulation data improves real-world transfer: 200-sim training raises Reach from 19/50 to 26/50 over 200-real training while matching Lift at 11/50, and 2,000-sim plus 200-real reaches 34/50 Reach and 18/50 Lift.The gains are attributed to broader simulated instance and appearance diversity, with real demonstrations adding complementary visual and physical variation.

5 Limitations

InstructMove’s scope and evaluation protocol impose important boundaries on what its results establish. The benchmark is focused on pick-and-place, uses synthetic simulation assets, and trains and evaluates within the same task families.

  • Scope boundary: The benchmark evaluates language-grounded pick-and-place rather than longer-horizon tasks, articulated-object interaction, or richer environments.Its results are therefore intended as a diagnostic of target selection and compositional pick-and-place, not comprehensive general-purpose manipulation.
  • Simulation-to-real boundary: Synthetic assets and simulated layouts may create a domain gap relative to real deployment.Although simulation training data improved real-world manipulation performance, the experiments do not establish simulation scores as reliable proxies for real-world performance.
  • Evaluation protocol: The train-eval protocol fine-tunes and evaluates policies on the same task families, despite holding out layouts, object instances, and categories.Task-specific fine-tuning may encourage overfitting or degrade pretrained capabilities, complicating separation of general instruction following from adaptation to task templates.

6 Conclusion

InstructMove evaluates language-grounded pick-and-place in semantically dense scenes where vision alone cannot identify the correct target. It decomposes the problem into complementary task families and reports stage-level metrics to diagnose visual shortcuts and support more reliable language-conditioned behavior.

  • Conclusion: InstructMove uses semantic distractors so the correct pick-and-place target cannot be inferred from vision alone.This creates semantically dense scenes for evaluating language grounding.
  • Conclusion: The benchmark decomposes instruction following into four complementary task families and separates semantic grounding from control execution with stage metrics.These diagnostics support analysis of where policies succeed or fail.
  • Conclusion: Future extensions can cover longer-horizon tasks, articulated objects, and richer scenes while preserving text indispensability.The core principle remains that behavior should be grounded in the instruction.

A.1 Benchmark Infrastructure Details

The benchmark infrastructure uses a shared task specification to connect scene construction, motion generation, validation, and reproducible multimodal data output. It combines curated assets, constraint-consistent layouts, reusable motion primitives, and automated quality checks.

  • Infrastructure Overview: Figure 4 summarizes a pipeline linking task specification, scene generation, motion generation, validator-based post-processing, and recorded data output.This high-level pipeline keeps task construction, data generation, and evaluation tied to the same episode specification.
  • Task Specification: Each episode begins with a task specification that serves as the shared source of truth for scene construction, motion generation, and evaluation.It contains scene configuration, a task-library entry, and task components such as objects, instructions, target conditions, and validators.
  • Task Specification: The task library uses Pick and Pick-and-Place primitives to instantiate four benchmark families: Pick Category, Pick Attribute, Pick Spatial, and Compositional Pick-Place.The first two emphasize semantic and attribute grounding, while the latter two emphasize spatial and compositional reasoning.
  • Scene Generation: Scene generation instantiates task specifications with sampled objects, poses, layouts, physical properties, and object combinations.These variations increase visual and semantic diversity while maintaining consistency with task specifications and validators.
  • Motion Generation: Motion generation decomposes tasks into reusable Pick, Move, and Place actions before producing trajectories under kinematic and collision constraints.Candidate grasp poses are filtered by inverse-kinematics feasibility and joint-space cost, then converted into smooth executable trajectories.
  • Validation and Data Output: Validator-based post-processing retains only rollouts satisfying task-specific success criteria such as grasping, lifting, placement, and completion.Validated trajectories are exported with synchronized RGB, depth, robot-state, transformation, and camera-extrinsic data in MCAP format.
  • Asset Infrastructure: The asset pipeline automates generation, VLM annotation, quality curation, and reproducible sampling, with human review for flagged assets.The resulting corpus contains 1,757 objects spanning 7 domains, 18 super-categories, and 103 fine-grained categories; curation addresses scale and other quality errors.

A.5 Real-World Experimental Setup

The real-world experiments use a dual-arm AgileX PiperX tabletop setup with one overhead and two wrist-mounted RGB-D cameras. Demonstrations are collected through ALOHA-style homologous teleoperation, while simulation varies visual appearance with fixed scene geometry and embodiment.

  • Real-World Setup: The real-world platform is a dual-arm AgileX PiperX robot with one overhead Intel RealSense D455 and two wrist-mounted Intel RealSense D405 cameras.All tasks involve fixed-base tabletop manipulation.
  • Domain Randomization: Domain-randomization panels vary illumination, object textures, and table material while holding scene layout and embodiment fixed.Each panel therefore changes a single visual factor relative to the baseline.
  • Real-World Setup: Figure 8 contrasts the dual-arm PiperX experimental setup with an ALOHA teleoperation setup for data collection.The figure shows the robot-side camera arrangement and the teleoperation configuration.
  • Data Collection: Data collection follows the ALOHA methodology, with an operator controlling two 6-DoF slave arms to demonstrate each task.This provides the teleoperation procedure used for real-world demonstrations.

B.1 Failure Case Analysis

π0.5’s Reach failures mostly reflect instruction-inconsistent object selection rather than low-level manipulation errors, with distractors sharing salient cues with the target.

  • Most of π0.5’s 18 Reach failures still completed a lift on an incorrect distractor object.Reach is instruction-conditioned, so physically successful motion on the wrong object remains a failure.
  • Across those failures, 15 lifted distractors shared salient cues with the target, including color, shape, geometry, or affordance.Representative cases include approaching a yellow lotion bottle for “yellow lemon” and a cube-shaped distractor for “stubby sharpener.”
  • The failures were not missing-action or low-level grasp failures, but physically valid picks on instruction-inconsistent objects.
  • The observed pattern suggests partial grounding without reliable binding of the full referring expression to the intended object category.The policy sometimes responds to one salient instruction attribute, such as color or shape, without reliably identifying the complete referent.
  • Because multiple feasible candidates can share salient cues, InstructMove requires grounding the full instruction rather than selecting a plausible partial match.

C Training Recipes

The evaluated policies are trained and evaluated under a shared InstructMove protocol, while retaining model-specific interfaces, preprocessing, and optimization configurations.

  • The training recipes cover four evaluated policies: π0, π0.5, GR00T N1.7, and Motus.Unless otherwise stated, all models are trained for 5,000 steps and evaluated at the final checkpoint without checkpoint selection.
  • Training tasks differ by robot embodiment: PiperX covers pick category, pick attribute, and place a2b, while Franka covers pick spatial and place a2b.Each policy uses its native training pipeline, observation preprocessing, and action interface.
  • π0 and π0.5: π0 and π0.5 share a LoRA-based fine-tuning recipe but use different initialization checkpoints, OpenPI pi0 and pi05 respectively.
  • GR00T N1.7: GR00T N1.7 uses robot-specific AnyMove runs with custom new-embodiment configurations, three RGB cameras, proprioceptive state, and language instructions.
  • Motus: Motus is fine-tuned from the Motus checkpoint with a three-camera stitched RGB input, robot state, language embeddings, and Qwen3-VL image-text inputs.
Loading 2608.22990v1…