PaperlayerView exact revision on arXiv

Source-linked AI summary

One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

Yiwei Liu, Luwei Yang, Shunbo Lei

arXiv:2609.05260v1cs.RO

TL;DR

Robots must distinguish instruction changes that preserve a task from those that require a new action, but unified real-robot evaluation of this ability remains limited. The benchmark fixes physical states and executable actions to test Decision Invariance and Decision Sensitivity, finding near-saturated single-constraint performance but model-dependent degradation under joint constraints.

  • Problem

    A unified real-robot framework is needed to test whether robots preserve or update actions appropriately when language changes preserve or alter task requirements.

  • Method

    The benchmark fixes each physical decision state and executable action space while pairing task-preserving and task-changing instructions to measure Decision Invariance and Decision Sensitivity.

  • Results

    Modern models are near saturation on single-constraint changes, but several degrade when multiple task constraints must be jointly integrated into one executable decision.

  • Takeaways & Limitations

    Reliably integrating multiple language-conditioned task constraints remains a more demanding embodied-reasoning challenge than responding to isolated instruction changes.

Abstract

from arXiv · show

Natural-language instruction changes can directly alter robot behavior. A reliable embodied system should preserve its action when the task is unchanged and update it correctly when the task itself changes. We introduce One Word, Different Action, a real-robot benchmark built on physical decision states and executable actions, using task-preserving and task-changing instruction pairs to jointly evaluate Decision Invariance and Decision Sensitivity, with further evaluation under multi-constraint reasoning and real-RGB grounding. Experiments show that modern models are near saturation on single-constraint instruction changes, yet several models degrade noticeably when multiple task constraints must be integrated into one executable decision. These results suggest that the more salient remaining challenge is no longer recognizing an isolated instruction change, but reliably composing multiple task requirements into a correct robot action decision.

I. Introduction

The paper introduces a controlled real-robot benchmark that tests whether robots preserve actions under equivalent instructions and update them when task requirements change. It further evaluates multi-constraint reasoning and real-RGB grounding, finding near-saturated single-constraint performance but degradation for some models when constraints must be jointly integrated.

  • Problem: Natural-language instruction changes require robots to preserve actions when tasks remain unchanged and update actions when task requirements genuinely change.This distinction motivates evaluating selective stability rather than instruction following alone.
  • Benchmark: One Word, Different Action fixes each physical decision state and executable action space while pairing task-preserving and task-changing instructions with deterministic correct actions.The benchmark uses controlled task families rather than isolated language instructions.
  • Evaluation: The benchmark jointly evaluates Decision Invariance and Decision Sensitivity, extending the same framework to multi-constraint embodied reasoning.Correct decisions are grounded in physical states, verified executable actions, and deterministic task constraints.
  • Results: Current strong models are close to saturation on single-constraint instruction changes, usually producing correct actions for both task-preserving and single task-changing conditions.The result covers both preservation and updating behavior under single-constraint changes.
  • Results: Some models degrade when multiple task constraints must be integrated into one executable decision, while the selected real-RGB condition does not cause substantial overall degradation.The results concentrate the remaining challenge in jointly integrating task requirements.

II. Related Work … III. Language-Conditioned Embodied Reasoning

Prior work increasingly diagnoses embodied reasoning beyond task success, including instruction understanding, corrections, controlled language variation, and multi-constraint following. This work distinguishes task-preserving from task-changing instruction pairs while holding the physical decision state and executable action space fixed.

  • A. Embodied Reasoning and Robot Instruction-Following Benchmarks: Embodied-intelligence benchmarks decompose robot capability into instruction understanding, perception reasoning, planning, affordance judgment, and failure analysis.RoboBench evaluates high-level robot cognition, while VLABench emphasizes complex language-conditioned manipulation and long-horizon reasoning.
  • A. Embodied Reasoning and Robot Instruction-Following Benchmarks: Existing benchmarks diagnose failures across tasks, environments, or capability dimensions, whereas this work fixes the physical decision state and executable action space.The controlled structure shifts evaluation toward finer-grained decision analysis rather than only task completion.
  • B. Instruction Changes, Corrections, and Task Switching: Instruction corrections, refinements, interruptions, and task switching show that changing language can directly alter robot behavior during execution.Prior studies emphasize correction recovery, execution-context modeling, or task-switching policies.
  • B. Instruction Changes, Corrections, and Task Switching: The proposed diagnostic asks whether a changed instruction produces the appropriate decision when the physical decision state remains fixed.This compresses dynamic instruction changes into a controlled decision question distinct from broader policy-level task switching.
  • C. Controlled Instruction Variation and Constraint Composition: Controlled language variation uses minimal pairs and paraphrases to test selective semantic responsiveness and stable execution for the same underlying task.CxMP controls structural differences, while LIBERO-Para evaluates paraphrase robustness in robot manipulation.
  • C. Controlled Instruction Variation and Constraint Composition: This work extends task-preserving paraphrase evaluation by jointly testing task-preserving and task-changing instruction pairs.The distinction isolates whether models should maintain or update their decisions under language variation.
  • C. Controlled Instruction Variation and Constraint Composition: Multi-constraint following requires jointly satisfying spatial and temporal requirements rather than recognizing conditions independently.LIMP maps natural-language tasks to verifiable robot constraints, motivating separate single-constraint and multi-constraint reasoning evaluation.

A. Embodied State and Executable Action Space · B. Task-Preserving and Task-Changing Decisions

The benchmark formulates embodied reasoning as selecting an exactly correct, executable action decision from a physical state under language-specified constraints. It separately evaluates whether models preserve decisions when wording changes without changing the task and update decisions when task requirements change.

  • A. Embodied State and Executable Action Space: A robot decision state s exposes the physical information needed for decision making and defines a candidate set of explicitly identifiable, executable behaviors.The executable action space is denoted A(s).
  • A. Embodied State and Executable Action Space: Each candidate action a_i has a well-defined physical meaning and may represent an object selection, behavior choice, or ordered multi-behavior sequence.The action representation depends on task structure.
  • A. Embodied State and Executable Action Space: For the same physical state, A(s) remains fixed while the instruction determines which actions to select or exclude, how many to choose, and how to order them.This treats language as conditioning an executable robot decision space rather than an isolated classification problem.
  • A. Embodied State and Executable Action Space: Given an instruction x and state s, the model produces an action decision π(x, s), which may be a single action, action set, or ordered sequence.Each task instance has a uniquely determined correct decision a*(x, s), and correctness requires an exact match.
  • B. Task-Preserving and Task-Changing Decisions: Decision Invariance measures whether the model preserves the same correct behavior when linguistic realization changes but the underlying task remains unchanged.A task-preserving transformation x+ should leave the action decision unchanged.
  • B. Task-Preserving and Task-Changing Decisions: Decision Sensitivity measures whether the model changes its action correctly when an instruction change modifies a task constraint affecting execution.A task-changing transformation x− requires a corresponding change in the action decision.
  • B. Task-Preserving and Task-Changing Decisions: Reliable embodied reasoning requires changing behavior precisely when the task changes, avoiding both indiscriminate variation and mechanical preservation.Over-Invariance preserves a decision when it should change, whereas Over-Sensitivity changes it when the task has not changed.

C. Deterministic Executable Decision Representation · IV. Embodied Reasoning Tasks · A. Single-Constraint Reasoning

The benchmark converts embodied reasoning into deterministic, executable decisions checked against physical states and task constraints. Its single-constraint tasks vary one instruction condition across fixed candidate spaces, covering inhibition, set, cardinality, and temporal reasoning.

  • C. Deterministic Executable Decision Representation: Robot outputs are mapped to finite executable representations: target objects or object sets for object-oriented tasks, and actions, action sets, or ordered sequences for action-oriented tasks.This representation enables strict verification instead of free-form text evaluation.
  • C. Deterministic Executable Decision Representation: Set decisions require exact predicted-target agreement, sequential decisions require matching both actions and order, and multi-constraint decisions must satisfy all constraints simultaneously.Partial satisfaction does not receive independent credit in multi-constraint tasks.
  • C. Deterministic Executable Decision Representation: Ground-truth decisions are computed deterministically from physical states and task constraints, while language only expresses those constraints for reproducible machine-checkable evaluation.Natural language does not determine the ground truth.
  • A. Single-Constraint Reasoning: Single-constraint reasoning uses four categories—action inhibition, set, cardinality, and temporal constraints—while keeping the physical state and executable candidate space fixed.Each instruction changes one task condition that directly determines the correct action decision.
  • A. Single-Constraint Reasoning: Action inhibition reasoning determines whether an otherwise feasible candidate action or target is prohibited by the current instruction.The task concerns instruction-based prohibition, not physical feasibility.
  • A. Single-Constraint Reasoning: Set and cardinality reasoning jointly applies exclusion or retention constraints, candidate membership, and required set size to determine the executable action set.Incorrect membership or quantity interpretation changes the final decision through over-selection or under-selection.
  • A. Single-Constraint Reasoning: Sequential reasoning identifies unchanged candidate actions and orders them according to temporal constraints imposed by the instruction.The correct decision is an ordered action sequence.
  • A. Single-Constraint Reasoning: Across categories, the physical state remains fixed while task constraints alter the correct executable behavior, linking instruction understanding directly to robot action.The categories manipulate different structural properties of the decision space under one shared evaluation principle.

B. Controlled Task Families · C. Multi-Constraint Embodied Reasoning

The benchmark organizes controlled task families around shared physical decision states, varying only linguistic task constraints to jointly test Decision Invariance and Decision Sensitivity. Its multi-constraint subset evaluates whether models integrate several requirements into one executable robot decision rather than recognizing them independently.

  • B. Controlled Task Families: Each task family shares the robot state, candidate objects or actions, and executable decision space while varying only the linguistic expression of task constraints.This design treats related instructions as controlled variants of the same physical decision state.
  • B. Controlled Task Families: Task-preserving transformations keep the correct decision unchanged, while task-changing transformations deterministically alter the correct action decision.Together, these transformations support joint evaluation of Decision Invariance and Decision Sensitivity.
  • B. Controlled Task Families: A model succeeds on a complete family only by remaining stable across equivalent formulations and responding correctly when the task changes.The family-level design distinguishes insufficient sensitivity to genuine changes from excessive sensitivity to reformulations that preserve the task.
  • C. Multi-Constraint Embodied Reasoning: Multi-constraint tasks require the final robot action to jointly satisfy several task conditions rather than one primary action condition.This extends the benchmark beyond single-constraint instruction following.
  • C. Multi-Constraint Embodied Reasoning: For an instruction containing K task constraints, the benchmark represents those requirements as a constraint set that the correct robot decision must satisfy.The supplied formulation introduces K as the number of task constraints and defines correctness through joint satisfaction.
  • C. Multi-Constraint Embodied Reasoning: Recognizing local constraints independently is insufficient when the resulting action decision fails to satisfy them jointly.Correctness is determined at the executable decision level, not by isolated recognition of individual requirements.
  • C. Multi-Constraint Embodied Reasoning: The compositional subset combines cardinality and action-inhibition constraints, requiring an action set with the specified size that excludes prohibited candidates.This evaluates integration of multiple constraints into a single executable robot decision.

V. Real-Robot Benchmark Construction … C. Paired Counterfactual Task Construction

The benchmark grounds language-conditioned reasoning in fixed real-world decision states, executable candidates, and uniquely determined actions. Paired task-preserving and task-changing instructions isolate whether models maintain or appropriately update behavior when only task requirements change.

  • A. Physical Decision Anchors: Physical Decision Anchors fix real robot states and associated object or executable-action sets for constructing language-conditioned tasks.These anchors ground every instance in an actual robot decision state.
  • A. Physical Decision Anchors: The benchmark contains 85 physical decision anchors: 21 object-centric anchors and 64 action-sequence anchors.Object-centric anchors support object-selection and constrained reasoning, while action-sequence anchors support sequential action reasoning.
  • A. Physical Decision Anchors: Fixed candidate identities let each anchor support multiple tasks, so behavior changes can be evaluated against instruction changes rather than environmental changes.Different constraints are applied to the same underlying candidate space.
  • B. Executable Action Candidates and Ground Truth: Object-centric tasks select constrained target objects or subsets, whereas action-sequence tasks select and order verified robot actions.The candidate decision space is explicitly defined for each anchor type.
  • B. Executable Action Candidates and Ground Truth: Each task is deterministically specified by action constraints that uniquely determine the target action, action set, or ordered action sequence before language realization.Ground truth is established before the task constraints are expressed as natural-language instructions.
  • C. Paired Counterfactual Task Construction: Counterfactual task pairs keep the physical anchor and executable action space fixed while changing the instruction, testing whether behavior should remain stable or update.The paired construction makes the unchanged physical substrate explicit.
  • C. Paired Counterfactual Task Construction: Task-preserving pairs vary linguistic realization while retaining constraints and decisions, whereas task-changing pairs alter a constraint that changes the correct action.These pair types test decision stability and appropriate behavioral updating, respectively.

D. Benchmark Scale and Physical-State Splitting … A. Reasoning Conditions and Model Configuration

The benchmark combines large-scale physical-state splitting with multi-constraint and real-RGB evaluations of executable action reasoning. Models are tested under structured and visual inputs using fixed action-decision outputs, with visual evaluation limited to models supporting image inputs.

  • D. Benchmark Scale and Physical-State Splitting: The atomic benchmark contains 988 complete task families and 3,952 task instances, plus 84 multi-constraint instances requiring simultaneous satisfaction of multiple task conditions.These instances separately evaluate embodied action reasoning under multiple constraints.
  • D. Benchmark Scale and Physical-State Splitting: Splitting occurs at physical decision anchors, assigning all related task families, language realizations, and action decisions to the same partition.This prevents the same physical state from appearing across different partitions.
  • D. Benchmark Scale and Physical-State Splitting: Held-out decisions must therefore derive from the provided physical state and current instruction rather than from linguistic variants of an anchor seen elsewhere.The partitioning avoids repeated exposure to the same physical substrate.
  • E. Real-RGB Grounded Decision Subset: The real-RGB subset contains 748 task instances in 187 complete task families and covers all 85 physical decision anchors.It evaluates reasoning when physical states are presented through actual robot visual observations.
  • E. Real-RGB Grounded Decision Subset: Structured-state and real-RGB conditions preserve task constraints, family structure, and executable actions while varying primarily how the physical state is presented.This enables direct comparison of instruction-conditioned action reasoning across structured descriptions and real visual grounding.
  • VI. Evaluation of Embodied Reasoning: The benchmark evaluates embodied reasoning through text-grounded and vision-grounded conditions, with the latter preserving the same task instruction and executable decision space.The text condition evaluates Qwen3.5-27B, GLM-5-FP8, and Gemma4-31B, while the vision condition evaluates Qwen3.5-27B and Gemma4-31B because they support visual inputs.
  • A. Reasoning Conditions and Model Configuration: All models return only machine-readable final action decisions under fixed structured-output and decoding protocols across corresponding benchmark instances.Reasoning is disabled for Qwen3.5-27B, while no reasoning-specific control is applied to the other models.

B. Exact Action-Decision Evaluation · C. Paired Decision Metrics · D. Validity Controls

The evaluation uses exact executable action matching, paired metrics for task-preserving and task-changing relations, and controls testing whether performance depends on task-critical instruction information. These measures distinguish decision errors from parsing failures and assess both behavioral relations and benchmark validity.

  • B. Exact Action-Decision Evaluation: Exact Action-Decision Accuracy compares each parsed executable action representation directly with the deterministic ground truth.The metric is instance-level and requires exact agreement with the target representation.
  • B. Exact Action-Decision Evaluation: Single-action, set-valued, and sequential tasks require exact action, member-set, and membership-order matches, respectively.Any additional, missing, or misordered action is incorrect.
  • B. Exact Action-Decision Evaluation: Parse Success Rate separates machine-readable output validity from interface or formatting failures without removing failed parses from accuracy.System-interface failures and embodied decision errors are analyzed separately.
  • C. Paired Decision Metrics: Pair Success requires every member of a paired relation to produce a correct action decision, making it stricter than isolated instance accuracy.It identifies cases where only part of a controlled task relation is solved.
  • C. Paired Decision Metrics: Decision Invariance succeeds only when both task-preserving pair predictions are correct and identical.The metric tests whether unchanged tasks preserve the action decision.
  • C. Paired Decision Metrics: Decision Sensitivity succeeds when both task-changing predictions are correct and behavior changes appropriately with the altered constraint.It jointly captures decision correctness and an appropriate response to task change.
  • C. Paired Decision Metrics: Over-Invariance and Over-Sensitivity measure incorrect unchanged and changed decision relations, respectively.Over-Invariance retains an action when it should change; Over-Sensitivity changes it when the task remains unchanged.
  • D. Validity Controls: Validity controls include a symbolic oracle, majority-type baseline, random-valid baseline, and task-critical information masking.Masking preserves physical state and executable candidates while removing action-determining language; substantial degradation indicates reliance on task-critical instruction information rather than priors or fixed preferences.

E. Statistical Evaluation … B. Reasoning across Constraint Types

The benchmark uses family-level bootstrap methods to evaluate embodied decision reliability while showing that single-constraint reasoning is generally strong. Remaining errors are concentrated in constraints requiring structured reasoning over action sets, motivating multi-constraint evaluation.

  • E. Statistical Evaluation: Family-level bootstrap resampling preserves dependence between task-preserving and task-changing instances within each complete task family.This avoids treating correlated linguistic variants from the same task family as independent samples.
  • E. Statistical Evaluation: The evaluation reports 95% bootstrap confidence intervals and paired comparisons across matched physical decision states and task constraints.Metrics include Exact Action-Decision Accuracy, Pair Success, Decision Invariance, and Decision Sensitivity.
  • A. Single-Constraint Embodied Reasoning: Current models map single task constraints to correct robot action decisions with high reliability.These main atomic results are summarized in Table II.
  • A. Single-Constraint Embodied Reasoning: Pair-level performance remains high alongside instance-level exact accuracy, indicating stable behavior across task-preserving and task-changing relations.Under a controlled single-constraint change, models usually determine whether the action should remain unchanged or be updated.
  • A. Single-Constraint Embodied Reasoning: Single-constraint embodied reasoning is close to saturation, with residual errors concentrated in demanding constraints and limited system-interface events.The errors do not appear as broad failures across task categories.
  • B. Reasoning across Constraint Types: Action-inhibition and sequential constraints exhibit the most stable behavior, supporting reliable suppression of candidate behaviors and reordering of actions.These findings are summarized in Table III.
  • B. Reasoning across Constraint Types: Set-exclusion and cardinality constraints retain more residual errors because they require structured operations over candidate sets rather than local action decisions.Their larger decision space motivates the multi-constraint evaluation, and failures are generally more likely when constraints act on action-set structure.

C. Multi-Constraint Embodied Reasoning · D. Reasoning from Real Visual Observations

Multi-constraint reasoning exposes compositional weaknesses that are less visible in single-constraint evaluation, while real-RGB grounding largely preserves instruction-conditioned action reasoning under the benchmark’s controlled visual conditions.

  • C. Multi-Constraint Embodied Reasoning: Qwen3.5-27B and GLM-5-FP8 degrade measurably when multiple task constraints must be satisfied jointly, whereas Gemma4-31B remains stable.The comparison is between atomic and multi-constraint results in the current compositional setting.
  • C. Multi-Constraint Embodied Reasoning: Multi-constraint reasoning requires integrating each local condition into one final action set that satisfies all conditions simultaneously.Integration errors directly change the resulting executable behavior.
  • C. Multi-Constraint Embodied Reasoning: Pair Success and Decision Sensitivity more directly assess complete action updates under changing task requirements than isolated instance accuracy.These metrics can expose compositional weaknesses that aggregate accuracy may obscure.
  • C. Multi-Constraint Embodied Reasoning: Single-constraint embodied reasoning is highly reliable, but jointly satisfying multiple task constraints remains a model-dependent source of reduced robustness.The results distinguish reliable isolated reasoning from less consistent compositional reasoning.
  • D. Reasoning from Real Visual Observations: On the selected real-robot visual subset, replacing structured descriptions with real RGB observations causes no substantial overall degradation for the evaluated multimodal models.Both multimodal models retain performance close to their structured-state results.
  • D. Reasoning from Real Visual Observations: Under real visual observations, models remain stable on task-preserving transformations and update decisions appropriately on task-changing transformations.Instruction-conditioned action reasoning remains largely preserved within the evaluated physical decision anchors.
  • D. Reasoning from Real Visual Observations: These visual-grounding results apply to the benchmark’s controlled real observations and do not establish that visual grounding is negligible for more complex robot tasks.The evaluated decision relations remain reliable under the represented conditions.

E. Validity of Task-Critical Reasoning · F. Failure Modes in Embodied Decisions · VIII. Conclusion

The benchmark’s validity controls show that models rely on task-critical instruction information rather than output priors, while remaining errors primarily involve composing multiple constraints into executable decisions. The conclusion finds high reliability on single-constraint changes but model-dependent weaknesses under joint constraints.

  • E. Validity of Task-Critical Reasoning: Validity controls show that high performance cannot be explained by deterministic output-frequency priors or random valid-action selection.The symbolic oracle achieves expected deterministic consistency, while majority-type and random-valid baselines remain substantially below evaluated models.
  • E. Validity of Task-Critical Reasoning: Masking task-critical instruction information markedly reduces paired decision performance, especially Decision Sensitivity.This indicates that models rely on instruction information determining correct robot behavior rather than fixed candidate-action patterns.
  • E. Validity of Task-Critical Reasoning: The benchmark meaningfully tests whether models distinguish linguistic reformulations from changes that genuinely modify the robot task.Strong performance requires preserving decisions for task-preserving changes while updating them for task-changing instructions.
  • F. Failure Modes in Embodied Decisions: With atomic performance approaching saturation, residual failures concentrate in cardinality, set-exclusion, and temporal action-decision structures.These errors respectively violate required set size, retain candidates that should be removed, or preserve action members while violating execution order.
  • F. Failure Modes in Embodied Decisions: Pair-level failures reflect either insufficient updating after task-changing instructions or unnecessary instability under task-preserving reformulations.The first reflects inadequate response to altered constraints; the second reflects sensitivity to linguistic variation despite an unchanged robot task.
  • F. Failure Modes in Embodied Decisions: Interface failures are analyzed separately because parsing, connection, and related events indicate invalid executable output rather than direct evidence of incorrect task-constraint understanding.This distinction separates system-interface problems from embodied decision errors.
  • F. Failure Modes in Embodied Decisions: Remaining model differences arise primarily from integrating multiple constraints, which must preserve action-set structure, exclusion relations, and cardinality requirements simultaneously.Joint constraints make otherwise small residual weaknesses more visible than isolated task conditions.
  • VIII. Conclusion: Modern models are highly reliable on single-constraint changes but retain model-dependent residual weaknesses under joint constraints.The study fixes the physical decision state and executable action space while evaluating task-preserving, task-changing, and multi-constraint conditions.
Loading 2609.05260v1…