PaperlayerView exact revision on arXiv

Source-linked AI summary

Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models

Sangwu Park, Wonjoong Kim, Yeonjun In, Chanyoung Park

arXiv:2610.21251v1cs.CVcs.RO

TL;DR

This paper examines whether VLAs use How-level instructions, a capability less studied than identifying what task language specifies. It diagnoses instruction following, trains with segment-level enriched instructions, and introduces SIMPLER-IN; enriched training improves responsiveness, but demanding and out-of-distribution guidance remains difficult.

  • Problem

    Prior VLA work largely studies What-level task instructions, leaving whether models follow free-form How-level guidance about execution largely unexplored.

    Source: §1 · pp. 1–2

  • Method

    The paper diagnoses offline corrections and online interventions, trains with segment-level enriched instructions, and evaluates instruction-necessary tasks in SIMPLER-IN.

    Source: §1 · p. 2

  • Results

    Enriched training raises correction-direction following to about 84% of evaluated episodes versus about 53% for original models and controls, but does not consistently improve task success; SIMPLER-IN exposes failures on demanding and out-of-distribution guidance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    How-level instruction following remains an open challenge for current VLAs.

    Source: §1 · p. 3, §7 · p. 9

    Experiments cover one embodiment, simulator, and training dataset—WidowX in SIMPLER with BridgeData V2—on short-horizon pick-and-place tasks.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Vision-language-action (VLA) models are expected to follow instructions like their language backbones, yet prior work has focused on What-level instructions that specify the task rather than How-level instructions that specify how to carry it out. Through our diagnoses, we find that How-level instructions do not help current VLAs and lower task success. We hypothesize that pairing each training episode with a single instruction limits How-level instruction following, and show that training with segment-level enriched instructions makes actions respond to such instructions without consistently raising task success. To measure How-level instruction following beyond standard benchmarks, we introduce SIMPLER-IN, where success requires conditions given only in the instruction. It shows that the gained responsiveness does not extend to demanding or out-of-distribution How-level instructions. How-level instruction following thus remains an open challenge.

1 Introduction

The paper asks whether VLAs follow How-level instructions, which specify how to perform a task rather than what task to perform. Its diagnoses find unreliable following; enriched segment-level instructions improve action responsiveness, but SIMPLER-IN exposes remaining limits.

1 Introduction

  • The paper distinguishes What-level task instructions from How-level guidance about execution, an ability that could let VLAs incorporate corrections and refine behavior through language.Prior VLA work largely focused on What-level instructions, while free-form How-level following remained largely unexplored.Source: §1 · pp. 1–2
  • The diagnosis uses offline corrections and online VLM interventions to test instruction following under controlled and deployment-like conditions.Offline correction compares a retry, a language correction, and an inverted correction; online intervention uses a VLM monitor to provide guidance during execution.Source: §1 · p. 2
  • Across offline correction and online intervention, current VLAs do not reliably use How-level instructions, even when designed for robust task-instruction following.Corrections succeed less often than retries and do not clearly outperform inverted corrections; frequent online interventions substantially degrade performance.Source: §1 · p. 2
  • The authors hypothesize that pairing each episode with one shared instruction limits How-level following, and identify this training structure as a potential cause.Enriched instructions describe robot movements at different episode segments while leaving demonstrations unchanged.Source: §1 · pp. 2–3
  • About 84% of evaluated episodes follow correction directions after enriched training, versus about 53% for original models and controls, near chance.Enriched instructions maintain comparable task success but do not consistently improve it.Source: §1 · p. 2
  • SIMPLER-IN requires success conditions to appear only in How-level instructions, and evaluated models do not reliably follow demanding or out-of-distribution guidance.Its tasks require a specified path around a transparent barrier, a particular grasp point, or an instructed placement location.Source: §1 · pp. 2–3

2 Related Work

Prior VLA instruction-following work has mainly studied identifying the task or object from language, while free-form guidance about how to execute a task remains less examined. Related hierarchical and recovery systems often route such guidance through predefined commands or another model rather than the VLA itself.

2 Related Work

  • Studies of instruction following find limited behavioral changes when task instructions are modified or removed, and reliance on visual priors, but primarily test what task VLAs identify.Other work reports that paraphrasing task instructions can degrade performance in LIBERO-Goal, where multiple tasks share a scene.Source: §2 · p. 3
  • Robust instruction-following methods target task or object identification, leaving their ability to support How-level following unknown.The paper therefore includes InstructVLA and LangForce in its diagnosis.Source: §2 · p. 3
  • Hierarchical-control and recovery methods use predefined commands, visual symbols, or separate VLM-generated actions, so free-form How-level guidance bypasses the VLA.Whether a VLA can directly follow such guidance, and why it fails, had not been diagnosed.Source: §2 · p. 3

3 Problem Formulation

The paper models a VLA policy as mapping visual observations and language instructions to actions. It separates task-defining What-level content from How-level guidance and formalizes the expectation that accurate How-level information can help task execution.

3 Problem Formulation

  • A VLA policy maps an observation and language instruction to an action, which may be an action chunk, with a random seed governing policy stochasticity.The policy is trained on demonstration episodes.Source: §3 · p. 3
  • What-level content specifies the object and task, while How-level content adds execution guidance such as a path, grasp or placement location, or scene context.The split is conceptual rather than computational, and How-level guidance is added to the task instruction.Source: §3 · pp. 3–4
  • The formulation assumes accurate How-level guidance contains information about an action that accomplishes the task, so using it should raise success or at least not lower it.The text identifies I as conditional mutual information and A⋆_t as an action that accomplishes the task.Source: §3 · p. 4

4 Diagnosis: Do VLAs Follow How-Level Instructions?

Across offline correction and online intervention, How-level instructions fail to improve current VLAs and often lower task success. The diagnosis tests both controlled recovery and deployment-like guidance across five VLA families.

4 Diagnosis: Do VLAs Follow How-Level Instructions?

  • The diagnosis covers three general VLAs and two models designed to strengthen instruction following on SIMPLER WidowX tasks.The two settings test How-level guidance under controlled offline conditions and deployment-like online conditions.Source: §4 · p. 4

4.1 Offline Correction

  • Corrections are tailored to observed failures, and manual review found 44 of 50 sampled corrections grounded in the failures.A VLM receives episode video, the task instruction, and a timeline of physical phases before generating each correction.Source: §4.1 · p. 4
  • Offline correction reruns each failed episode from the same initial state with a retry, an accurate correction, or an inverted correction.The expected success ordering is correction > retry > inverted correction, but the text notes this is not necessary for instruction following.Source: §4.1 · p. 4

4.2 Online Intervention

  • Online intervention uses a VLM as System 2 to guide the VLA as System 1 during execution.The corrector sends guidance only after detecting failure; unified mode sends a sub-step instruction at every monitoring step.Source: §4.2 · pp. 4–5
  • The online evaluation compares no intervention, a failure-triggered corrector, and unified guidance, averaging success over three runs on the same 96 episodes.Episodes run for up to 240 steps, four times the original 60-step evaluation.Source: §4.2 · p. 5

4.3 Results

  • How-level instructions fail to help in either setting and often lower task success, including when interventions are given during execution.The always-intervening unified mode substantially lowers success for every model; the corrector lowers it for all but π0.Source: §4.3 · p. 5
  • Offline retries recover the most failed episodes for every model, and no model clearly benefits from correction content.Corrections do not clearly differ from inverted corrections; inverted corrections recover more episodes for InternVLA-M1.Source: §4.3 · p. 5

5 Hypothesis: Data Structure Limits How-Level Instruction Following

The section tests whether episode-level instruction supervision limits How-level following by changing only the training instructions. Segment-level enriched instructions make actions respond to How-level directions, but this response does not consistently translate into higher task success.

5.1 Hypothesis

  • Episode-level supervision pairs every timestep with one task instruction, providing little supervision for how different How-level instructions should change actions within that task.The authors hypothesize that this training structure contributes to VLAs’ limited How-level instruction following.Source: §5 · p. 5, §5.1 · p. 6
  • Correction attention ratios ranged from 0.41 to 1.03, yet corrections did not clearly outperform inverted corrections, suggesting insufficient attention alone does not explain the failure.The authors caution that attention alone cannot establish how language influences actions or rule out architectural factors.Source: §5.1 · p. 6, Table 2 · p. 6

5.2 Enriched Instructions

  • The intervention adds segment-level imperative instructions describing robot motion, built from public ECoT annotations for subtasks and motion primitives.The resulting data contain 272,662 enriched instructions for 43,771 episodes.Source: §5.2 · p. 6
  • Each of the five diagnosed VLAs was trained with LoRA and enriched segment instructions appended to task instructions, against a control trained on original task instructions alone.Both enriched models and controls received appended How-level instructions during evaluation.Source: §5.2 · p. 6

5.3 Results

  • Without intervention, enriched models matched controls in average task success; online, every enriched model gained success or lost less under unified instructions, while corrector gains were smaller for most.Table 3 reports enriched-model and control results under the same diagnosis.Source: §5.3 · p. 6, Table 3 · p. 7
  • Offline corrections recovered more failed episodes for enriched models than controls, while retries recovered about as many; corrections beat retries only for enriched InternVLA-M1.Correction advantages over inverted corrections or a neutral sentence were not consistent across models.Source: §5.3 · pp. 6–7
  • Enriched models followed specified correction directions in most episodes, but this responsiveness did not consistently yield higher task success.The action-level analysis compares trajectories under corrections and inverted corrections; chance level is 50%.Source: §5.3 · p. 7, Figure 2 · p. 7
  • Because task success can reflect instruction following only indirectly, the authors next evaluate tasks solvable only by following the instruction.Many failures are also recovered by retries alone.Source: §5.3 · p. 7

6 SIMPLER-IN: An Instruction-Necessity Benchmark

SIMPLER-IN tests How-level instruction following by making success depend on conditions stated only in the instruction. Models struggle with demanding and out-of-distribution conditions, showing that responsiveness gained from enriched instructions does not reliably transfer to task success.

6 SIMPLER-IN: An Instruction-Necessity Benchmark

  • SIMPLER-IN makes How-level conditions necessary for success by placing them only in the instruction, unlike standard benchmarks where success reflects following them indirectly.The tasks cover the path to take, where to grasp, and where to place an object.Source: §6 · p. 7, §6.1 · p. 8

6.1 Benchmark Design

  • Path requires approaching a carrot from the instructed side around a transparent, unseen barrier; Grasp specifies a carrot end, and Place specifies a side of the towel midline.In Path, an episode follows the instruction only if the gripper reaches the carrot without passing over the barrier.Source: §6.1 · p. 8
  • The evaluation reports task success and instruction following among successful episodes across 24 episodes per task, with condition values evenly split so instruction-ignoring policies follow by chance 50% of the time.The study evaluates public checkpoints, controls, and enriched models without additional training.Source: §6.1 · p. 8

6.2 Results and Analysis

  • No model follows the Path instruction: every successful episode passes over the barrier, and enriched models also struggle with out-of-distribution conditions in Grasp and Place.The authors conclude that responsiveness remains limited to How-level content seen during training and falls short when strict adherence is required.Source: §6.2 · pp. 8–9
  • Enriched models follow instructions in Grasp and Place near the controls’ and public checkpoints’ chance level, while their training sources rarely include the specified approach side, object part, or object region.The enriched instructions and ECoT annotations more often specify direction of motion, suggesting the evaluated conditions are out of distribution.Source: §6.2 · pp. 8–9
  • SIMPLER-IN provides a testbed for measuring progress, while broader How-level training content may be needed to address the remaining challenge.Enriched instructions make actions respond to How-level guidance but do not consistently raise task success.Source: §6.2 · p. 9

7 Conclusion

The paper finds that current VLAs struggle to follow How-level instructions. Enriched instructions increase action responsiveness without consistently improving task success, and SIMPLER-IN exposes failures on demanding or out-of-distribution instructions.

7 Conclusion

  • Current VLAs struggle to follow How-level instructions, and enriched instructions do not consistently raise task success despite making actions respond to them.On SIMPLER-IN, all models fail on demanding How-level instructions and do not follow out-of-distribution ones.Source: §7 · p. 9
  • SIMPLER-IN tests How-level following where success requires the instruction, revealing limits beyond the responsiveness observed with enriched training.The authors hope the diagnoses and benchmark will help future work toward VLAs that follow How-level instructions.Source: §7 · p. 9
  • How-level instruction following remains an open challenge for current VLAs.The conclusion reports that models fail on demanding and out-of-distribution How-level instructions.Source: §7 · p. 9

Limitations

The study’s conclusions are bounded by its specific robot, simulator, training data, and short-horizon task setting; results may differ in other settings.

Limitations

  • Experiments cover only WidowX in SIMPLER with BridgeData V2 on short-horizon pick-and-place tasks, so other embodiments and longer tasks may behave differently.The authors leave those settings for future work.Source: Limitations · p. 10
  • Path uses a transparent barrier invisible in the observation, so its results evaluate instruction following rather than realistic manipulation.The authors note that transparent objects are often partly visible in the real world.Source: Limitations · p. 10
  • The experiments do not separate the effects of temporal alignment changes from changes in How-level content in the enriched instructions.The authors leave separating those contributions to future work.Source: Limitations · p. 10

A.1 Generation of Corrections

The appendix details how corrections were generated and checked, then examines whether timing or seed choice explains their performance. It also specifies the online intervention setup.

A.1 Generation of Corrections

  • Gemini 3.1 Pro uses a failed-episode video, task instruction, and timeline to identify critical segments and write a correction; another call inverts its actionable content.The prompt and raw timeline segments were selected after visually inspecting 108 outputs; corrections are one or two sentences.Source: §A.1 · p. 13
  • Manual checks found that corrections identified the failure cause in 44 of 50 sampled episodes (88%), and all were judged executable.The sample covered ten episodes from each model family and excluded episodes used for prompt design.Source: §A.1 · p. 13, Table 5 · p. 13
  • Corrections are appended before execution because they are framed as advance How-level advice, often with a temporal cue; giving them before the critical segment does not explain their shortfall versus retries.For 40–47% of failures in each group, critical segments start at the first step; corrections still recover fewer failures than retries for public checkpoints and controls, and about as many for enriched models.Source: §A.1 · p. 13, Table 6 · p. 13

A.2 Corrections with the Retry Seed

  • Rerunning corrections and inverted corrections with the retry seed tests whether retry gains come from the new seed rather than the instruction change; the resulting pattern resembles earlier results.The same seed is used for both correction variants so the comparison with a retry isolates the instruction difference.Source: §A.2 · pp. 13–14, Table 7 · p. 14

A.3 Online Intervention Settings

  • Online monitoring checks execution every eight steps; a corrector appends advice only after a clearly observed failure, while unified mode replaces the instruction at each check.The three runs share 24 scenes per task and differ in policy sampling noise.Source: §A.3 · p. 14

B.1 Attention-Level Analysis

The appendix describes the attention analysis and enriched-instruction training, then tests whether appended sentence format or sentence content explains recovery. The enriched models respond to sentence content, but this response does not consistently improve task success.

B.1 Attention-Level Analysis

  • Attention is measured at each model’s language-to-action component, comparing task-instruction and correction token attention over frames from failed episodes.The ratio excludes prompt-template, separator, special, and image tokens; each part’s attention is normalized by token count before averaging over frames.Source: §B.1 · pp. 14–15, Table 9 · p. 15

B.2 Enriched Instructions and Training

  • Enriched instructions give each episode segment an imperative, motion-grounded sentence, and checkpoints are selected by task success without intervention on four SIMPLER WidowX tasks.The instructions use task, plan, subtask, motion primitives, gripper state, and segment position; 9,421 of 53,192 training episodes retain only the original instruction.Source: §B.2 · p. 15
  • ECoT is excluded from the experiments because its 96 episodes yield too few successes and recoveries for comparison.It succeeds in 3 episodes; among 93 rerun failures, retry recovers 4, corrections 3, and inverted corrections none.Source: §B.2 · p. 15

B.3 Appending a Neutral Sentence

  • For public checkpoints, a neutral appended sentence recovers more failed episodes than corrections and inverted corrections in every model; enriched models respond to sentence content, without consistent success gains.For QwenGR00T and InstructVLA, neutral sentences tend to recover more than corrections; for π0, InternVLA-M1, and LangForce, they recover fewer.Source: §B.3 · p. 16

B.4 Action-Level Analysis by Model

The action-level analysis measures whether trajectories follow directions specified by corrections. Enriched models follow those directions more often and move farther in the specified direction, while the analysis does not cover several other kinds of How-level content.

B.4 Action-Level Analysis by Model

  • Across five model families, lateral directions are correct in 60.6%, 62.3%, and 80.1% of cases, and vertical directions in 52.6%, 50.8%, and 87.1% for public, control, and enriched models, respectively.Most retained pairs specify a single axis, giving an estimated chance level of about 48% under balanced, independent directions.Source: §B.4 · p. 16
  • The analysis retains 2,317 of 2,693 failed episodes with clearly reversed directional corrections and counts an episode as following instructions only when every specified axis is correct.It evaluates lateral and vertical movement, but not forward/backward directions, object parts, or timing.Source: §B.4 · p. 16
  • Enriched models move 1.74 cm further in the specified direction at the median, versus 0.21 cm for public checkpoints and 0.15 cm for controls; the difference remains for displacements over 1 cm.Table 12 reports median displacement and the fraction of episodes moving more than 1 cm in every specified direction.Source: §B.4 · pp. 16–17, Table 12 · p. 17

C.1 Task Design and Judgment

SIMPLER-IN task designs operationalize instruction-only conditions through tasks whose success can be judged from the robot’s path, grasp, or placement. The accompanying content analysis measures how often How-level instruction types appear in ECoT annotations and enriched instructions.

C.1 Task Design and Judgment

  • Path, Grasp, and Place use 24 episodes each, with 12 assigned to each condition; success is judged by avoiding hidden barriers, grasping the specified carrot end, or landing the spoon on the specified towel side.Episodes run for up to 240 steps, and a scripted oracle confirms each task can be solved.Source: §C.1 · p. 17

C.2 Analysis of How-Level Content

  • How-level content frequency is the share of sentences matching whole-word patterns for each content type, measured across 280,526 distinct ECoT subtask annotations and 272,662 enriched instructions.A sentence may match several content types; other ECoT annotations, such as plans, show similar frequencies.Source: §C.2 · pp. 17–18, Table 15 · p. 17

Related papers

Loading 2610.21251v1…