PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Language Guidance for Vision-Language-Action Models

Zhihao Zhan, Yuhao Chen, Jiaying Zhou, Qinhan Lyu, Hao Liu, Keze Wang, Liang Lin, Guangrun Wang

arXiv:2601.04052v2cs.ROcs.CL

TL;DR

VLA models often fail to preserve semantic intent when linguistic phrasing changes because visual affordance priors overwhelm sparse language signals. RSS expands linguistic supervision with Monte Carlo Syntactic Integration and subtracts visual priors with Residual Affordance Steering, achieving robust language–action alignment under diverse perturbations.

  • Problem

    VLA models can map different linguistic realizations of the same intent to different actions because visual signals dominate sparse language inputs.

  • Method

    RSS combines dense Oracle-Teacher syntactic expansion with residual steering that subtracts the visual affordance prior from conditional action scores.

  • Results

    RSS achieves state-of-the-art robustness and improves instruction robustness and semantic grounding under diverse linguistic perturbations.

  • Takeaways & Limitations

    RSS restores more consistent language–action alignment by disentangling semantic intent from visual affordance priors.

  • Takeaways & Limitations

    Residual Affordance Steering may hesitate or remain inactive when instructions are extremely vague or underspecified.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have demonstrated impressive capabilities in generalized robotic control; however, they remain notoriously brittle to linguistic perturbations. We identify a critical ``modality collapse'' phenomenon where strong visual priors overwhelm sparse linguistic signals, causing agents to overfit to specific instruction phrasings while ignoring the underlying semantic intent. To address this, we propose Residual Semantic Steering (RSS), a probabilistic framework that disentangles physical affordance from semantic execution. RSS introduces two theoretical innovations: (1) Monte Carlo Syntactic Integration, which approximates the true semantic posterior via dense, LLM-driven distributional expansion, and (2) Residual Affordance Steering, a dual-stream decoding mechanism that explicitly isolates the causal influence of language by subtracting the visual affordance prior. Theoretical analysis suggests that RSS effectively maximizes the mutual information between action and intent while suppressing visual distractors. Empirical results across diverse manipulation benchmarks demonstrate that RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations. We release our code at https://github.com/Doo-mon/RSS.

1 Introduction

VLA models can ignore linguistic intent because sparse language signals are overwhelmed by visual affordances and narrow training coverage. RSS addresses these failures by expanding syntactic supervision and subtracting visual priors to improve language-conditioned robustness.

  • VLA models may map different phrasings of the same intent to different actions, revealing brittle language grounding.
  • Instruction following fails through destructive overwriting, obfuscated reinterpretation, and out-of-distribution semantic transfer.
  • Sparse syntactic coverage encourages surface-form overfitting, while dense visual information drives policies toward affordance-based actions regardless of text.
  • Monte Carlo Syntactic Integration uses an Oracle Teacher to generate a dense linguistic neighborhood and optimize an Expected Semantic Loss toward the semantic posterior.
  • Residual Affordance Steering subtracts the visual Base Affordance Distribution from conditional predictions to isolate the semantic signal.
  • RSS achieves state-of-the-art robustness by mitigating instruction blindness and decoupling semantic intent from visual affordances.

2 Related Work

Prior VLA research has advanced architecture, spatial reasoning, action tuning, and planning, but language signals remain vulnerable to visual dominance. Existing audits report instruction blindness and rote execution, motivating explicit preservation of linguistic influence.

  • Modern VLA work spans large-scale imitation, open-source architectures, spatial cues, continuous action tuning, and chain-of-thought planning.
  • Audits of Libero-Plus and Libero-Pro report instruction blindness and rote execution when dense visual signals overwhelm sparse linguistic tokens.
  • RDT-1B addresses modality imbalance by avoiding simultaneous token injection, while related strategies aim to preserve linguistic gradient magnitude during fusion.

3 Methodology: The RSS Framework

RSS combines syntactic distribution expansion with residual action scoring to make VLA policies more invariant to linguistic variation and less dominated by visual affordances. Its theoretical formulation treats conditional scores as visual priors plus instruction-induced modulation, then amplifies the residual.

  • 3.2 Monte Carlo Syntactic Integration: RSS uses Monte Carlo Syntactic Integration to expand a seed instruction with Oracle Teacher samples and train toward invariance to syntactic noise.
  • 3.3 Residual Affordance Steering: The framework models the visual-only score as an affordance prior and the instruction-dependent shift as semantic modulation.
  • 3.3 Residual Affordance Steering: When language is weak, conditional and visual-only scores converge, motivating subtraction of the visual score to remove visual bias from action selection.
  • 3.3 Residual Affordance Steering: RSS applies residual steering as a bias suppressor rather than a conventional classifier-free quality booster, penalizing actions driven only by visual instinct.
  • 3.4 Theoretical Analysis: The first-order score decomposition separates visual and linguistic embeddings, and γ > 1 increases the relative rank of language features.

4 Experiment

Experiments evaluate RSS under destructive overwriting, obfuscated reinterpretation, out-of-distribution transfer, and component ablations. RSS improves robustness across linguistic perturbations, while MCSI provides especially strong semantic-transfer gains and moderate steering balances language conditioning with stability.

  • Robustness Evaluation: RSS and MCSI achieve higher average success rates across destructive instruction-overwriting variants, improving resilience when linguistic cues are removed or corrupted.The evaluation reports task success rates under Blank, Simple, and high-ratio Mask conditions, among others.
  • Obfuscated Instruction Reinterpretation: Across obfuscated reinterpretations, RAS- and MCSI-augmented models achieve the highest average success rate, with R3 and R4 providing the strongest tests of semantic grounding.R3 stresses implicit reasoning and final-state constraints, while R4 introduces distractor objects associated with other tasks.
  • Out-of-Distribution Semantic Transfer: RSS succeeds across both held-out out-of-distribution tasks, whereas the baseline’s higher long-adaptation average is dominated by overfitting to one task.The vanilla model generalizes poorly under low-step adaptation, especially 10-step adaptation.
  • Out-of-Distribution Semantic Transfer: MCSI alone delivers the strongest few-step improvement, while combining MCSI with RAS provides additional benefits for semantic transfer.The comparison attributes the dominant improvement to MCSI and links it to reduced reliance on task-specific memorization.
  • Ablation Study: Moderate RAS coefficients improve robustness under semantic-preserving perturbations, but excessive coefficients amplify sensitivity to corrupted instructions.The ablation interprets over-strong steering as over-conditioning on unreliable language signals.
  • Ablation Study: Increasing denoising steps does not consistently improve overall performance and can slightly reduce average success, indicating limited additional semantic benefit.The reported interpretation is that extra denoising mainly refines low-level action details.
  • Teacher and Paraphrase Robustness: Paraphrases from DeepSeek-R1 reduce baseline π0.5 performance by 10.54%, compared with a 1.28% decline for the MCSI method.With Qwen-3.5 paraphrases, the baseline drops 5.10% versus 1.71% for the method.
  • Teacher and Paraphrase Robustness: Different VLM teachers produce minimal performance variance, with differences of 2.43% and 0.54% reported for the tested alternatives.The result supports similar effectiveness across the evaluated instruction-generation teachers.

5 Conclusion

The paper identifies instruction blindness and visual-prior overreliance as core VLA limitations, then proposes RSS to separate semantic intent from visual affordances. Its reported experiments show improved robustness and semantic grounding under diverse linguistic perturbations.

  • Conclusion: Current VLA models often fail to robustly ground linguistic intent, becoming instruction-blind and overreliant on visual affordance priors.The conclusion frames this as the central limitation addressed by the paper.
  • Conclusion: RSS combines MCSI for instruction-manifold sparsity with RAS for suppressing visually driven biases and restoring consistent language–action alignment.The conclusion presents these components as the framework’s two-part response to the identified failure modes.
  • Conclusion: Empirical results report improved instruction robustness and semantic grounding, supporting RSS as a principled approach to language-conditioned robotic control.The conclusion states this outcome within the evaluated setting.

Limitations

RSS has a supported limitation under extremely vague instructions, while its formal analysis assumes approximately linear modality contributions, negligible higher-order terms, and a null linguistic feature when text is dropped.

  • Limitations: Extremely vague instructions can make Residual Affordance Steering conservative, producing hesitation or inaction rather than initiating motion.The limitation is illustrated by generic commands such as “do something.”
  • Formal assumptions: The formal analysis assumes higher-order interaction terms and bias are negligible for its first-order approximation.
  • Formal assumptions: The derivation assumes visual projection weights dominate linguistic weights, so standard inference is driven primarily by visual features.
  • Formal assumptions: The null-text baseline treats a dropped instruction as a near-zero linguistic feature, making the unconditional pass represent pure visual affordance.
  • Residual steering derivation: RSS defines a steered logit using the standard residual-steering formula with steering coefficient γ, then reconstructs the full steered logit after cancellation of visual features.

A.3 Analysis of Signal-to-Noise Ratio (SNR)

The SNR analysis defines semantic signal relative to visual contribution and argues that RSS amplifies linguistic influence when visual weights dominate.

  • A.3 Analysis of Signal-to-Noise Ratio (SNR): SNRstd approaches zero under visual-weight dominance, leaving text with minimal impact on action ranking.
  • A.3 Analysis of Signal-to-Noise Ratio (SNR): Choosing γ much greater than one linearly amplifies linguistic contribution without altering the visual affordance landscape, restoring modality balance.

B More results on LIBERO-Plus

On LIBERO-Plus, RSS is reported to outperform baselines on language-perturbed subsets and achieve the best overall benchmark performance without additional retraining.

  • B More results on LIBERO-Plus: RSS achieves the best overall LIBERO-Plus performance while outperforming baselines on language-perturbed subsets.The evaluation uses a trained checkpoint without additional retraining.

C Detailed Results

The detailed-results section breaks LIBERO performance down by individual subtasks, covering canonical instructions and multiple instruction-perturbation settings.

  • C Detailed Results: The detailed tables report success rates for each LIBERO subtask, enabling fine-grained examination under different instruction conditions.
  • C Detailed Results: Table 8 provides the reference results for original, unmodified instructions, while Tables 9–13 cover distinct instruction-perturbation settings.

D Detailed Ablation Results

The ablations show that robustness depends on balancing steering strength with inference depth. Moderate steering and sufficient denoising produce the most stable performance, while excessive steering can degrade results and π0.5 is less sensitive to denoising configuration.

  • D Detailed Ablation Results: Excessively large steering coefficients degrade performance, indicating that over-steering can reduce adaptability to corrupted language inputs.The ablation attributes this degradation to residual affordance steering dominating the policy update.
  • D Detailed Ablation Results: π0.5 shows stronger stability across denoising-step configurations than π0, indicating reduced sensitivity to inference-time hyperparameters.Tables 15 and 16 compare denoising-step effects for π0 and π0.5, respectively.
  • D Detailed Ablation Results: Moderate steering coefficients combined with sufficient denoising steps yield the most stable performance across language perturbations.The results identify a balance between steering strength and inference depth as important for robustness under destructive instruction overwriting.

E Qualitative Analysis under Destructive Instruction Overwriting

The qualitative and tabular analyses evaluate robustness when instructions are paraphrased, distracted, reordered, masked, or removed. MCSI helps integrate syntactic variation, RAS restores affordance guidance under missing language, and their combination supports reliable placement behavior, while the rewriting examples preserve task goals across increasingly indirect forms.

  • E Qualitative Analysis under Destructive Instruction Overwriting: Vanilla policies exhibit unstable grasps or premature terminations when key linguistic tokens are removed despite unchanged visual scenes.The failures include inconsistent localization of the wine bottle and misalignment with the cabinet-top affordance.
  • E Qualitative Analysis under Destructive Instruction Overwriting: MCSI improves robustness to syntactic variation, whereas RAS supplies residual visual-affordance signals when critical semantic content is missing.MCSI alone is insufficient to recover correct action intent under severe language degradation.
  • E Qualitative Analysis under Destructive Instruction Overwriting: RAS+MCSI achieves the most reliable qualitative performance, consistently executing the correct placement behavior across evaluated rollouts.The combination jointly mitigates linguistic uncertainty and reinforces visual affordance alignment.
  • E Qualitative Analysis under Destructive Instruction Overwriting: The evaluation includes original, blank, simple-word, multi-word, random-language, and random-mask instruction settings, with tables reporting success rates across LIBERO tasks.These settings respectively preserve canonical instructions, remove semantic guidance, simplify or paraphrase wording, disrupt word order, or introduce controlled masking.
  • E Qualitative Analysis under Destructive Instruction Overwriting: Steering-coefficient and denoising-step ablations report average success rates across destructive instruction-overwriting benchmarks.Table 14 varies steering coefficient with 10 denoising steps, while Tables 15 and 16 vary denoising steps for π0 and π0.5.
  • F Instruction Rewriting Examples on Obfuscated Instruction Reinterpretation: The rewriting procedure generates semantically equivalent variants that preserve the underlying task goal while increasing surface variation, contextual distraction, and semantic indirection.R1 adds irrelevant context, R2 uses commonsense descriptions, R3 introduces reasoning cues, and R4 adds distractors through negation or contrast.
  • F Instruction Rewriting Examples on Obfuscated Instruction Reinterpretation: Additional prompts preserve drawer-opening, bowl-placement, and cabinet-placement goals while adding conversational context, commonsense descriptions, or sequencing and state constraints.These transformations challenge information extraction without introducing new physical tasks.
Loading 2601.04052v2…