Source-linked AI summary
Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models
Mikey Watts, Yuchen Cui
TL;DR
VLA success can be highly sensitive to instruction phrasing, while existing deployment-side approaches either incur per-step inference cost or require task-specific interaction. The paper distills rollout evidence into explicit rules and uses them to rewrite each instruction once; the rules improve held-out π0 performance and also improve in-finetune π0.5 success on LIBERO.
Problem
VLA policies can respond very differently to equivalent instruction phrasings, and existing deployment-side methods either multiply inference cost or require online interaction with each task.
Source: §I · p. 1Method
The method distills rollout scores for many phrasings of evidence tasks into explicit rules, then uses a rule-conditioned rephraser to rewrite each instruction once for the frozen policy.
Source: §I · pp. 1–2, A. Problem Statement · p. 3Results
Rules improved frozen π0 success by 16 to 27% relative on twelve held-out tasks, with gains concentrated on out-of-distribution tasks; on π0.5, in-finetune success rose from 93.6% to 97.8%.
Source: §I · p. 1, F. Replication on π0.5 and LIBERO · p. 7Takeaways & Limitations
A small set of explicit phrasing rules can transfer rollout evidence to tasks that contributed no evidence while leaving the policy unchanged.
Source: §I · pp. 1–2
Abstract
from arXiv · showhide
Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: $\pi_{0.5}$ turns on a LIBERO stove 100% of the time for "switch on the stove" and 2% for "switch on the hot plate", and a $\pi_0$ checkpoint finetuned with rephrase augmentation still shows swings of up to 61 points. We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks. We then reduce it without modifying the policy. Because the sensitivity is systematic, it can be expressed as explicit rules: we score many phrasings of a few training tasks, have a large language model distill the evidence into ten to twenty rephrasing rules, and at deployment rewrite each incoming instruction once under these rules. The rules improve the frozen $\pi_0$ by 16 to 27% relative on twelve held-out tasks across adversarial, VLM-generated, and human-generated phrasings, with gains concentrated on out-of-distribution tasks. The pipeline replicates on $\pi_{0.5}$ and LIBERO, lifting in-finetune success from 93.6% to 97.8%. The method requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions. Project website: https://sttawm.github.io/rephrase-before-you-act
I. INTRODUCTION
VLA success can change sharply with instruction wording, even after rephrase augmentation. The paper measures this sensitivity and introduces an offline rulebook that rewrites instructions for a frozen policy, improving performance on held-out tasks.
I. INTRODUCTION
- Instruction phrasing sharply changes success: π0.5 succeeds 100% when told “switch on the stove” but 2% for “switch on the hot plate”; on SIMPLER, π0’s purple eggplant task falls from 69% to 8% without color.The studied π0 checkpoint had been finetuned with rephrase augmentation but still showed these swings. Existing deployment-side methods either multiply inference cost or require task-specific online interaction.Source: §I · p. 1
- The method distills rollout-scored phrasings of eight training tasks into 10–20 rules, then rewrites each incoming instruction once before the unchanged VLA acts.The evidence is distilled offline, and the rules transfer to tasks that contributed no evidence.Source: §I · p. 1, Figure 1 · p. 2
- Oracle phrasing nearly closes the 21-point in- versus out-of-distribution gap, while adversarial phrasing costs π0.5 50 points in-finetune and nothing measurable out-of-finetune.The authors suggest the latter pattern indicates that the policy attends to language where training has taught it to.Source: §I · p. 1
- Rules distilled from eight tasks improved frozen π0 success by 16 to 27% relative on twelve held-out tasks, with gains concentrated on out-of-distribution tasks.The evaluation included adversarial, VLM-generated, human-generated, and canonical phrasings; the effect was stable across three applier models and three independent distillations.Source: §I · p. 1
II. RELATED WORK
Related work studies language sensitivity, changes to model training or decoding, and deployment-time optimization or verification. This paper distinguishes its approach as a one-time rewrite of phrasing into forms the executor parses reliably.
II. RELATED WORK
- Prior benchmarks report that instruction perturbations can leave accuracy intact when policies ignore the instruction, while broader VLA evaluations find perturbations cost 14 to 50 points.The paper presents single-edit swings as a finer-grained complement and oracle search as a measure of how much apparent generalization gap is attributable to phrasing.Source: §II · p. 2
- Model-side remedies expand instruction forms through augmentation, generated commands, abstraction-level training, or paraphrase decoding, but they change policy weights or decoding.The studied π0 checkpoint already used rephrase augmentation; this method targets remaining sensitivity without further policy training.Source: §II · p. 2
- Deployment-time verification selects among phrase-action candidates at every inference step, whereas instruction optimization interacts with each task and does not transfer to unseen tasks.The paper instead distills explicit rules offline and applies them once per episode to tasks that supplied no evidence.Source: §II · pp. 2–3
- Language-planning systems choose which instruction to issue, while this rulebook targets surface forms that the executor parses reliably.The paper contrasts its automatically rollout-distilled rulebook with reusable guidance artifacts and human-correction knowledge bases.Source: §II · p. 3
III. MODELS AND BENCHMARKS
The experiments use inherited π0 and π0.5 checkpoints evaluated in simulation on Bridge and LIBERO tasks. The π0 checkpoint had already received rephrase augmentation.
III. MODELS AND BENCHMARKS
- The study uses π0 finetuned on Bridge and π0.5 finetuned on LIBERO, with both checkpoints inherited from prior work.The π0 checkpoint was finetuned with rephrase augmentation to reduce language sensitivity.Source: §III · p. 3
- Success is measured in simulation, at the 60th frame for SIMPLER Bridge tasks and under the described LIBERO procedure.The 60th frame is SIMPLER’s default horizon.Source: §III · p. 3
- Each benchmark provides simulatable tasks with a canonical instruction for each task.The passage specifies this setup for the study’s benchmarks.Source: §III · p. 3
A. Tasks and Canonical Instructions
The task sets comprise 20 SIMPLER Bridge tasks and 130 LIBERO tasks, with sealed or suite-defined splits distinguishing in- and out-of-distribution evaluation.
1) SIMPLER / Bridge:
- SIMPLER Bridge includes four benchmark-shipped tasks and 16 constructed from benchmark-provided primitives, for 20 simulatable tasks.The tasks are WidowX tabletop tasks such as putting a carrot on a plate.Source: 1) SIMPLER / Bridge · p. 3
- Bridge tasks share a tabletop setup and differ in source and destination objects; eight are in-vocabulary because every canonical-instruction noun appears in Bridge training language.Success is determined procedurally by the benchmark.Source: 1) SIMPLER / Bridge · p. 3
- The sealed Bridge evaluation fixes 12 of the 20 tasks ex ante and excludes them from all evidence gathering.The set contains five in-vocabulary and seven out-of-vocabulary tasks, reported as the in-distribution and out-of-distribution strata.Source: 1) SIMPLER / Bridge · p. 3
2) LIBERO:
- LIBERO contains 130 simulatable tabletop tasks grouped into four suites: libero_spatial, libero_object, libero_goal, and libero_100.The first three suites contain 10 tasks each; libero_100 splits into libero_90 with 90 and libero_10 with 10.Source: 2) LIBERO · p. 3
- The inherited π0.5 checkpoint was finetuned on every LIBERO suite except libero_90, making libero_90 out-of-finetune and the other tasks in-finetune.Task success is determined procedurally by the benchmark.Source: 2) LIBERO · p. 3
- LIBERO counts a task successful when any step succeeds before its suite-specific step limit.Representative tasks include putting a moka pot on a stove.Source: 2) LIBERO · p. 3
A. Problem Statement
The paper formalizes phrasing sensitivity in frozen policies and tests for significant success changes between goal-preserving instructions that differ by one edit. The observed swings reveal both recurring lexical patterns and effects that vary by task or model.
A. Problem Statement
- The analysis defines oracle headroom as the largest success gain available by changing wording alone while the frozen policy and task remain fixed.A rulebook is distilled from evidence-task phrase scores and rewrites instructions once before the first action, with the goal of improving tasks that supplied no evidence.Source: A. Problem Statement · p. 3
B. Finding Single-Edit Swings
- Significant single-edit sets retain phrase pairs whose success-rate gap passes p < 0.05; π0.5 additionally requires at least an 18-point gap.The π0.5 search had 426 candidate pairs with no multiple-comparison correction, and eight significant sets were removed when scene ambiguity could explain the gap.Source: B. Finding Single-Edit Swings · p. 4
C. Patterns in the Swings
- Replacing “plate” with “dish” costs π0 19 points and π0.5 56; added color can help Bridge nouns but lowers success on LIBERO.Generic category words underperform specific everyday names; adding “purple” to “eggplant” gains 61 points on one task but costs 30 on another.Source: C. Patterns in the Swings · p. 4
D. Oracle Phrase Search
Oracle phrase search compares canonical instructions with generated alternatives to estimate the success available from wording alone. The paper uses this characterization to motivate a deployment-time rulebook that rewrites each instruction once.
D. Oracle Phrase Search
- On SIMPLER Bridge, π0 had 26 single-edit sets over 16 tasks; on LIBERO, π0.5 had 15 over 13 tasks, with eight scene-ambiguous sets removed.π0.5 swings reached 100 pp: fire up the stove succeeded 100% of the time, whereas turn on the stove succeeded 6%.Source: Figure 4 · p. 4, Figure 5 · p. 4
- Oracle phrasing shrinks the 21-point in- versus out-of-distribution gap to 3 points on the 12 sealed Bridge tasks.Success is measured by phrasing condition over 24 fixed layouts, with the in-distribution stratum covering five tasks and the out-of-distribution stratum seven.Source: Figure 6 · p. 5
- Oracle phrases significantly outperform canonical, adversarial, and natural phrasings for both π0 and π0.5 on separately measured layouts.For π0, selection uses layouts 0–17 and the selected phrase is remeasured across all 24; held-out layouts 18–23 show the same headroom.Source: D. Oracle Phrase Search · p. 5
V. REDUCING LANGUAGE SENSITIVITY WITH LLM-DISTILLED REPHRASING RULES
- The mitigation distills evidence from scored rephrasings into a fixed rulebook, then rewrites each incoming instruction once at deployment.The rules are intended to hold across tasks rather than being optimized separately for each one.Source: §V · p. 5
A. Evidence Gathering
Evidence gathering assembles alternative phrasings for tasks and scores them so a distiller can compare their performance. The evidence includes both simulation rollouts and a proxy derived from expert trajectories.
A. Evidence Gathering
- The evidence set combines seven natural and two adversarial VLM-generated rephrasings per task with phrases measured during oracle search.It includes 5,453 scored task–phrase pairs from 228 evidence tasks, including 208 non-simulatable tasks drawn from BridgeData V2 expert demonstrations.Source: A. Evidence Gathering · p. 5
- On π0.5 and LIBERO, adversarial phrasing costs 50 percentage points in-finetune, while oracle search finds headroom in both in- and out-of-finetune tasks.The caption reports in-finetune success of 97 to 47 and oracle success of 99 / 66 for the in- / out-of-finetune strata.Source: Figure 7 · p. 5
1) Scoring Phrases:
- Simulatable task–phrase pairs are scored by success rate over 36 rollouts on 18 layouts, while non-simulatable pairs use gripper error against expert actions.The proxy averages gripper-dimension absolute error over four frames per trajectory and 16 expert trajectories; across 434 pairs and 15 task variants, it correlates with rollout success at Pearson r = 0.54 within task.Source: 1) Scoring Phrases · p. 5
B. Rule Distillation
The method distills phrase evidence into compact rulebooks and applies them before the policy acts. The evaluation uses a sealed task split, while deployment appliers receive the instruction and rulebook alongside a scene-derived reasoning trace.
B. Rule Distillation
- Distilled rulebooks contain 10 to 20 rules, from lexical substitutions to constraints against replacing a common noun with a sibling noun.Bridge rulebooks are distilled from in-finetune, out-of-finetune, or combined evidence, with three stochastic distillations per type whose results are averaged.Source: B. Rule Distillation · p. 5
C. Rule Application
- At deployment, Claude, Gemini, or Qwen applies the rules once per episode, and the rewritten instruction goes directly to the unchanged VLA.Two model calls occur before movement: Gemini produces a reasoning trace from the instruction and scene image, while the applier receives the trace but not the image.Source: C. Rule Application · p. 6
VI. EVALUATING THE RULEBOOKS
- Rulebooks are evaluated on 12 sealed Bridge tasks across 24 shared layouts; no sealed task entered the evidence bank used for distillation.The study also replicates on π0.5 and LIBERO.Source: §VI · p. 6
A. Phrase Sets
The evaluation uses adversarial, VLM-generated, human-generated, and canonical phrase sets, comparing unchanged instructions, a generic rephraser, and a rephraser with distilled rules. Effects are paired by base instruction and tested with sign-flip permutation tests.
A. Phrase Sets
- 392 human phrases remained after cleaning, 363 unique; canonical results use one phrase per task and are underpowered.Per task, the evaluation uses 6 adversarial, roughly 16 VLM-generated natural, and roughly 30 human-generated phrases.Source: A. Phrase Sets · p. 6
B. Baselines
- CoVer selects the highest-scoring phrase/action pair from 40 candidates at every step, unlike the unchanged-instruction and generic-rephraser baselines.The no-rules baseline isolates distilled-rule effects from rephrasing itself.Source: B. Baselines · p. 6
C. Statistical Tests
- Paired differences use base instructions as the sample units and a two-sided sign-flip permutation test, with 20,000 resamples or exact enumeration when n ≤ 12.The two arms face identical scene layouts, so scene difficulty cancels within each paired difference.Source: C. Statistical Tests · p. 6
D. Results
Out-of-finetune rulebooks improve results over both the no-rephraser and no-rules baselines on adversarial and natural phrase sets. Their gains vary little across rulebook draws, and CoVer trails both baselines on every set.
D. Results
- 20 to 26% relative improvement over no-rephraser across adversarial and natural sets; against no-rules, gains were 26%, 11%, and 4.7%, respectively.The no-rules comparisons are for adversarial, human-generated natural, and VLM-generated natural sets.Source: D. Results · p. 6, Table I · p. 7, Table II · p. 7
- CoVer trails both baselines on every set.Table III reports pooled success (%) at step 60 on sealed tasks, using identical Gemini reasoning traces.Source: Table III · p. 7
- 0.7 to 5.1 points was the pooled-success variation across three rulebook draws, with a median range of 3.0 points.The paper describes these ranges as small relative to the effects.Source: D. Results · p. 7
E. Comparison with Test-Time Verification
The replication on π0.5 improved in-finetune success while leaving out-of-finetune success unchanged, with rulebook gains tied to evidence-supported rewrites. The paper notes that evaluation covered only two VLAs and simulation.
E. Comparison with Test-Time Verification
- CoVer’s native metric is first success within an episode, whereas the paper reports final-state success; auditing first success changed no conclusion.The audit improved all arms by similar amounts.Source: E. Comparison with Test-Time… · p. 7
F. Replication on π0.5 and LIBERO
- 93.6% to 97.8% was the in-finetune success change for π0.5 rulebooks; out-of-finetune success was unchanged.The in-finetune result had p ≤ 0.005, while the out-of-finetune result had p > 0.4.Source: F. Replication on π0.5 and LIBERO · p. 7
- 45 points on roughly 9 phrases came from rewriting hot plate, hotplate, or burner to stove; dish-to-plate rewrites gained 24 points.Onto-to-on and into-to-in rewrites gained 2.5 points over roughly 50 phrases.Source: F. Replication on π0.5 and LIBERO · p. 7
- A prompt requiring rules to help across all tasks produced a rulebook that changed little and gained nothing; asking for average-help rules produced the reported rulebooks.Two of the three rulebooks also rewrote grab or lift to pick up, gaining 2.4 points over roughly 15 phrases.Source: F. Replication on π0.5 and LIBERO · p. 7
VII. CONCLUSION
- Only two VLAs and simulation were evaluated, leaving real-robot performance as a next step.The paper also identifies systematic measurement across models and explaining sensitivity as open directions.Source: §VII · p. 7