PaperlayerView exact revision on arXiv

Source-linked AI summary

Tell Robot What Not to Do: A Negation Understanding Perspective

Fazeng Li, Gan Sun, Hao Cheng, Weihong Ren, Yang Cong

arXiv:2610.11952v1cs.RO

TL;DR

Existing VLAs struggle to follow negated instructions, despite the need for robots to respect explicit exclusions, and negation-specific demonstrations are costly. NegaAlign adapts pretrained VLAs using image-language supervision, and experiments report improved negated instruction following while retaining affirmative performance.

  • Problem

    Existing VLAs struggle with negated instructions, and adapting them without costly negation-specific embodied data remains an open problem.

    Source: §1 · pp. 1–2

  • Method

    NegaAlign revises instruction representations and aligns instruction-relevant visual tokens using supervision constructed from existing affirmative demonstrations and hand-crafted negated instructions.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Experiments report improved negated instruction following across simulation and real-world tasks while preserving affirmative performance; π0.5 reaches 88.45% negated success on NegaBench.

    Source: §1 · p. 2, §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    NegaAlign supports negated instruction following without additional negated action trajectories, using image-language supervision while pretrained parameters remain frozen.

    Source: §1 · p. 2, §5 · p. 9

    The approach relies on pretrained VLAs that may not cover all negated scenarios, and performance may degrade for highly complex or ambiguous negated instructions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Instruction following enables robots to perform diverse tasks specified in natural language, making it a fundamental capability for human-robot interaction. Beyond communicating desired outcomes, users also need to specify constraints on what not to do. We investigate how to enable vision-language-action models (VLAs) to follow negated instructions, where robots must accomplish task goals while respecting explicit exclusions. To this end, we propose NegaAlign, a parameter-efficient, plug-and-play framework that extends pretrained VLAs to follow negated instructions through image-language supervision alone. Specifically, we introduce Negation Transformation Layers into selected layers of the vision-language backbone to reshape intermediate instruction representations. Meanwhile, a teacher-guided alignment mechanism is designed to align instruction-relevant visual tokens, transferring action-relevant grounding from instructions that satisfy the negated constraint. The training phase uses supervision constructed from existing demonstrations and updates only the inserted layers, keeping all pretrained parameters frozen, including the action generator. We further introduce NegaBench, a simulation benchmark spanning 10 scenarios across five domains for systematically evaluating manipulation under negated constraints. Experiments across GR00T, $\pi_0$, and $\pi_{0.5}$ demonstrate consistent improvements in negated instruction following. With 11.6M trainable parameters, NegaAlign increases the negated-instruction success rate of $\pi_{0.5}$ from 2.60% to 88.45% on NegaBench and from 12.4% to 88.8% on real-world tasks, while retaining performance on affirmative instructions.

1 INTRODUCTION

Existing VLAs struggle to respect negated constraints, risking unsafe actions, and negation-specific robot demonstrations are costly to collect. NegaAlign adapts pretrained VLAs using image-language supervision, while NegaBench evaluates manipulation under negated constraints.

1 INTRODUCTION

  • Existing VLAs, mostly trained on affirmative instructions, can ignore negated constraints and perform unsafe or undesired actions; collecting negation-specific demonstrations is costly.The paper frames enabling negated instruction following without such additional embodied data as an unexplored problem.Source: §1 · pp. 1–2
  • NegaAlign revises intermediate instruction representations with Negation Transformation Layers and uses teacher-guided alignment to propagate negation semantics to instruction-relevant visual tokens.It constructs teacher-student pairs from existing affirmative-task data and hand-crafted negated instructions, without additional negated action data.Source: §1 · p. 2
  • Training updates only inserted layers with image-language supervision, keeping the pretrained backbone and action expert frozen.Source: §1 · p. 2
  • NegaBench spans 10 scenarios across five domains, and experiments report improved negated instruction following across VLAs while preserving affirmative-instruction performance.Source: §1 · p. 2

2 RELATED WORK

Prior work documents challenges in negation understanding across language and vision, while robot-focused approaches use hierarchical reasoning, inference-time steering, or online correction. This paper situates NegaAlign in the need to preserve language grounding through visual representations to action generation.

2.1 NEGATION IN LANGUAGE, VISION, AND ACTION

  • Prior work finds negation challenges in language models and visual systems, where visual negation requires understanding semantics and grounding exclusion relations.Studies report degradation on negated expressions and difficulty removing or excluding specified content in image editing and retrieval.Source: §2.1 · pp. 2–3
  • Robot methods address negative or corrective language through hierarchical reasoning, inference-time steering, or online correction, while end-to-end propagation of negated semantics to actions remains underexplored.The cited approaches depend on hierarchical reasoning, external grounding, or online interaction.Source: §2.1 · p. 3

2.2 LANGUAGE CONDITIONING AND INSTRUCTION FOLLOWING IN VLAS

  • Reliable VLA instruction following requires sensitivity to linguistic semantics rather than visual shortcuts or language-agnostic imitation.Source: §2.2 · p. 3
  • NegaAlign is motivated by the need for linguistic semantics to propagate through visual representations to action generation, without costly negation-specific robot demonstrations.Source: §2.2 · p. 3

3 METHOD

NegaAlign adapts pretrained VLAs to negated instructions using supervision derived from affirmative demonstrations, revising instruction representations and aligning instruction-relevant visual tokens.

3.1 PROBLEM FORMULATION

  • Valid execution requires both task completion and constraint satisfaction, while adaptation optimizes only lightweight parameters to preserve affirmative instruction following.Source: §3.1 · pp. 3–4

3.2 NEGATION SUPERVISION CONSTRUCTION

  • Negation supervision pairs hand-crafted negated instructions with satisfying affirmative interpretations and an explicitly excluded behavior, using existing affirmative demonstrations.Source: §3.2 · p. 4

3.3 NEGATION TRANSFORMATION LAYER

  • Negation Transformation Layers revise intermediate representations of negated instructions while leaving image, state, and non-instruction prompt tokens unchanged.The frozen VLA remains responsible for executing the acquired skill.Source: §3.3 · p. 5

3.4 TEACHER-GUIDED VISUAL ALIGNMENT

  • Teacher-Guided Visual Alignment transfers revised negation semantics to instruction-relevant visual tokens by aligning the negated student with a satisfying teacher.Alignment uses coordinate-consistent top-camera tokens and stops gradients through teacher-derived weights.Source: §3.4 · p. 5

3.5 TRAINING RECIPE

  • Stage 2 freezes the affirmative-fine-tuned VLA and trains inserted layers with Teacher-Guided Visual Alignment using vision-language data from existing affirmative demonstrations.It requires neither negated action trajectories nor additional robot demonstrations.Source: §3.5 · p. 6

3.6 NEGATION INSTRUCTION FOLLOWING BENCHMARK CONSTRUCTION

  • NegaBench evaluates manipulation under negated constraints in 10 scenarios spanning five categories, with paired instructions and constraint-aware success criteria.Paired instructions share scenes, manipulation skills, and success conditions.Source: §3.6 · p. 6

4 EXPERIMENTS

The experiments evaluate negated-instruction following across three VLA models, NegaBench, and real-world robot tasks. NegaAlign improves negated success while retaining affirmative performance, and ablations examine the roles of visual alignment and transformation-layer placement.

4.1 EXPERIMENTAL SETUP AND EVALUATION METRICS

  • NegaAlign is evaluated on GR00T, π0, and π0.5 under affirmative-only, joint affirmative-and-negated, and NegaAlign settings; success requires completing intended manipulation while satisfying all negated constraints.Figure 3 depicts successful negated-instruction executions in green and failures to meet negated constraints in red.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 COMPARISON RESULTS ON NEGABENCH

  • 88.45% Neg. SR for π0.5 with NegaAlign, versus 2.60% for Pos. and 88.60% for Pos.+Neg., despite using no negated action trajectories.Across GR00T, π0, and π0.5, NegaAlign reaches 57.75%, 63.30%, and 88.45% Neg. SR, respectively; its three-model average is 69.83%.Source: Table 1 · p. 7, §4.2 · pp. 7–8
  • Qwen-π0.5 reaches 55.70% negated success through instruction rewriting, below NegaAlign’s 88.45%, while adding a VLM inference step at each action-chunk request.APT reaches 8.20% negated success despite 75.46% affirmative success with affirmative-only training.Source: §4.2 · p. 7

4.3 ABLATION STUDIES AND MECHANISM ANALYSIS

  • Removing TGVA reduces negated success from 88.45% to 21.35%, while affirmative success remains 91.27%, indicating that alignment helps propagate revised semantics into action generation.Visual-token differences are negligible in shallow layers, concentrate on instruction-relevant objects in middle layers, and spread toward irrelevant regions in deeper layers.Source: §4.3 · p. 8, Table 2 · p. 8, Figure 5 · p. 8
  • Middle-layer NTL placement performs best; moving it to shallow or deep layers reduces negated success to 85.10% or 62.20%, respectively.The reported analysis places middle layers after semantic grounding emerges but before it becomes diffuse.Source: §4.3 · pp. 8–9, Figure 5 · p. 8
  • Removing NTL yields 86.65% negated success but drops affirmative performance from 92.51% to 40.52%, whereas the complete model retains both capabilities.NTL separates semantic content from polarity and gates revision only when needed; TGVA propagates revised semantics through action-relevant visual tokens.Source: Table 2 · p. 8, §4.3 · p. 9

4.4 REAL-WORLD ROBOT EXPERIMENTS

  • 88.8% average negated success on real-world tasks with NegaAlign, up from 12.4%, while affirmative success remains essentially unchanged at 90.8%.The affirmative-only policy's affirmative success rate was 91.6%; each negated instruction was evaluated over 50 trials.Source: Table 3 · p. 9, §4.4 · p. 9

5 CONCLUSION

NegaAlign combines Negation Transformation Layers and Teacher-Guided Visual Alignment to adapt pretrained VLAs to negated instructions without additional negated action trajectories. Simulation and real-world experiments show improved negated-instruction following while affirmative performance is preserved, with limitations in pretrained coverage and complex or ambiguous instructions.

5 CONCLUSION

  • NegaAlign combines Negation Transformation Layers and Teacher-Guided Visual Alignment to enable negated instruction following without additional negated action trajectories.The framework is parameter-efficient and is evaluated with NegaBench for robotic manipulation under negated constraints.Source: §5 · p. 9
  • Simulation and real-world experiments demonstrate substantial improvements in negated instruction following while preserving affirmative performance.Source: §5 · p. 9
  • The approach relies on pretrained VLAs that may not cover every negated scenario, and highly complex or ambiguous negated instructions may degrade performance.Source: §5 · p. 9

AI USE STATEMENT

The authors used generative AI for language editing, figure presentation, mathematical-notation checks, and consistency checks against code. They state that AI did not generate research data or perform core research design, analysis, or interpretation.

AI USE STATEMENT

  • Generative AI assisted with language editing, figure presentation, mathematical-notation checking, and manuscript–code consistency checking.Source: AI USE STATEMENT · p. 10
  • The authors state that generative AI was not used to generate research data, formulate or prove mathematical claims, design experiments, or analyze or interpret results.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted text and suggestions against code, experimental records, and relevant literature, and take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION AND TRAINING DETAILS

Implementation freezes pretrained VLA components and trains inserted Negation Transformation Layers using language-conditioned student and teacher branches. The appendix details the NTL architecture, teacher-guided visual alignment, and optimization settings.

A.1 BACKBONES, TRAINABLE MODULES, AND BRANCH EXECUTION

  • Only inserted NTLs are trainable; pretrained components remain frozen, with 11.6M trainable parameters for π0 and π0.5 and 4.35M for GR00T.At inference, the NTLs revise the instruction and the original action-generation procedure runs unchanged.Source: §A.1 · pp. 14–15
  • Each training example expands into affirmative and negated student branches plus satisfying and excluded no-gradient teacher branches, all sharing one VLA weight set.For GR00T, the frozen Qwen backbone jointly processes three language variants, reusing satisfying hidden states for the teacher and student branches.Source: §A.1 · pp. 14–15

A.2 NTL ARCHITECTURE

  • NTLs mask instruction tokens, compress and pool them into 224-dimensional content and 32-dimensional polarity codes, then use a gate to activate semantic revision for negated inputs.Token-specific modulation lets object, attribute, relation, and action words respond differently; other tokens remain unchanged at the NTL output.Source: §A.2 · p. 15

A.3 TEACHER-GUIDED VISUAL ALIGNMENT

  • TGVA uses frozen satisfying and excluded teachers on the same image and robot state; the satisfying teacher supplies the alignment target while their visual differences identify instruction-sensitive locations.The selected visual Key and Value tokens guide the negated student without object masks, action targets, or trainable teacher networks.Source: §A.3 · pp. 15–16
  • TGVA selects the top 10% of visual tokens by teacher-induced Key and Value contrast, with Figure 6 showing task-focused middle-layer selections and more diffuse deeper-layer selections.Source: §A.3 · p. 16, Figure 6 · p. 17

A.4 OPTIMIZATION

  • Optimization updates only NTL parameters with AdamW, uses a global gradient-norm clip of 1.0, and applies a warmup schedule with endpoint learning rates of 10^-4 and 10^-5.Simulation uses 20,000 steps with 2,000 warmup steps; real-world π0.5 adaptation uses 5,000 steps with 500 warmup steps.Source: §A.4 · p. 16
  • The reported batch size of 4 counts independent scene–instruction examples, while forward passes use bfloat16 and reduction-sensitive calculations use FP32.Source: §A.4 · p. 17

B.1 SIMULATION DATA COLLECTION AND COMPOSITION

The evaluation data include a 10-scenario simulation benchmark and affirmative-only real-world demonstrations, organized into satisfying and excluded instruction relations. Post-training uses these relations to construct teacher and student branches without action targets.

B.1 SIMULATION DATA COLLECTION AND COMPOSITION

  • NegaBench comprises 10 simulation scenarios across five negation categories, with 38 language-conditioned tasks spanning affirmative and negated variants.The scripted expert retains trajectories only when motion planning succeeds and the task success predicate is met.Source: §B.1 · p. 17

B.2 REAL-WORLD DATA COMPOSITION

  • The real-world xArm6 dataset contains 500 affirmative demonstrations across 10 tasks and five semantic contrast pairs, with no negated robot demonstrations.Source: §B.2 · p. 18

B.3 CONSTRUCTION OF INSTRUCTION RELATIONS

  • Each negated instruction preserves the common task while excluding one member of an affirmative pair, so the relation changes skill selection without requiring a new low-level manipulation primitive.Source: §B.3 · p. 18

B.4 END-TO-END POST-TRAINING PIPELINE

  • Stage 2 samples relations, satisfying instructions, and frames hierarchically, then forms student and teacher branches from the same observation without supplying action targets to the loss.At inference, relation lookup and teachers are removed; the trained NTLs precede the unchanged action generator. Qualitative figures show the affirmative-only baseline violating constraints and adapted π0.5 satisfying them in simulation and real-world scenarios.Source: §B.4 · pp. 18, 21, Figure 7 · p. 19, Figure 8 · p. 20

C.1 NEGABENCH EVALUATION PROTOCOL

NegaBench uses a shared client–server evaluation pipeline, with consistent task seeds and fixed comparison settings across policies. Success is checked during execution, and reported category and overall scores macro-average task-level success rates.

C.1 NEGABENCH EVALUATION PROTOCOL

  • The RoboTwin client handles environment setup, instruction selection, rollout, success checks, and logging; model-specific servers convert observations into action chunks.This separation lets each VLA use its own software environment without changing benchmark logic.Source: §C.1 · p. 22
  • Each task variant uses 100 episodes, with expert-validated seeds cached per task/configuration and comparison settings fixed across policies.An expert seed is admitted only when motion planning succeeds and the expert satisfies the same task-specific predicate used for policy evaluation.Source: §C.1 · p. 22
  • Policies receive the instruction, three camera views, and model-specific proprioception; episodes end on success or after 300 high-level actions.π0, π0.5, and GR00T execute 14-dimensional joint-position actions, while APT executes 16-dimensional end-effector pose and gripper actions.Source: §C.1 · p. 22
  • Negated-task success requires completing the manipulation while respecting exclusions, and category and overall scores macro-average task-level success rates.Each episode has a binary outcome; task-level rates use 100 episodes, and longer trajectories receive no extra weight in aggregate scores.Source: §C.1 · p. 22

C.2 REAL-WORLD EVALUATION PROTOCOL

The real-world evaluation tests π0.5 and its NegaAlign-adapted counterpart on five xArm6 scenarios. It uses matched trial counts for affirmative and negated conditions and requires both task completion and constraint compliance for negated success.

C.2 REAL-WORLD EVALUATION PROTOCOL

  • The five xArm6 scenarios exclude a knife, mixed food placement, socks, moving a diamond, or a stainless-steel bowl.The evaluation compares π0.5 with its NegaAlign-adapted counterpart.Source: §C.2 · p. 22
  • A negated trial succeeds only when the shared manipulation goal is completed and the excluded object or outcome is avoided.Source: §C.2 · p. 23
  • Each negated scenario uses 50 trials, matched by pooling two 25-trial affirmative counterparts; results include scenario counts, rates, and an unweighted five-scenario mean.This pairing separates failure to perform the manipulation from failure to obey the negated constraint.Source: §C.2 · p. 23

C.3 DETAILED SIMULATION RESULTS

NegaAlign improves negated-instruction success across all evaluated backbone–task combinations, while affirmative performance remains close to affirmative-only fine-tuning. The task breakdown also shows that residual weaknesses vary by backbone.

C.3 DETAILED SIMULATION RESULTS

  • NegaAlign improves negated success on all 30 backbone–task combinations without negated trajectories, with π0.5 reaching 88.45% overall versus 88.60% with explicit negated demonstrations.Affirmative-only fine-tuning reaches at most 12.00% on π0.5 tasks and 8.00% on GR00T tasks; π0 remains at or below 52.00% per task.Source: §C.3 · p. 23
  • Relative to Pos. FT, NegaAlign changes overall affirmative scores by +1.80, +0.21, and −0.26 percentage points for GR00T, π0, and π0.5, respectively.The task-level rates are aggregated by averaging variants within a task, then macro-averaging across tasks for category and overall scores.Source: §C.3 · pp. 23–24
  • Residual negated-instruction failure modes differ by backbone: GR00T is weakest on object exclusion and the first complex task, while π0 retains larger object and complex-constraint gaps.Source: §C.3 · p. 24

C.4 DETAILED REAL-WORLD RESULTS

In real-world trials, NegaAlign raises negated success in every scenario while retaining nearly the base policy’s affirmative performance. Dining remains the most challenging negated scenario.

C.4 DETAILED REAL-WORLD RESULTS

  • NegaAlign raises negated success in every scenario: Sorting 9/50 to 40/50, Laundry 4/50 to 47/50, Storage 1/50 to 50/50, Cooking 0/50 to 48/50, and Dining 17/50 to 37/50.The gain spans constraints on objects, attributes, and outcomes rather than one task family.Source: §C.4 · pp. 24–25
  • The affirmative-only policy succeeds in 229 of 250 affirmative trials but only 31 of 250 negated trials.Source: §C.4 · p. 24
  • NegaAlign succeeds in 227 of 250 affirmative trials, two fewer than the base policy, and still performs explicitly requested excluded-object actions.Dining is the hardest negated scenario at 37/50; the authors attribute its residual difficulty to ambiguity in open-ended object selection relative to structured placement tasks.Source: §C.4 · p. 25

Related papers

Loading 2610.11952v1…