PaperlayerView exact revision on arXiv

Source-linked AI summary

When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding

arXiv:2602.17659v2cs.CVcs.RO

TL;DR

VLAs often ignore weakly supervised instructions and rely on visual shortcuts, exposing a gap in reliable language following. The paper introduces LIBERO-CF to measure this behavior and CAG to regularize language conditioning, reporting consistent gains in simulation and real-world evaluations.

  • Problem

    VLAs can default to well-learned scene behaviors instead of following instructions when scene-specific supervision is limited, raising reliability concerns for robotic deployment.

  • Method

    The paper introduces LIBERO-CF, a counterfactual benchmark using alternative instructions under LIBERO layouts, and CAG, a dual-branch inference scheme combining VLA and language-unconditioned VA actions.

  • Results

    CAG consistently improves language grounding and task success on under-observed tasks while preserving performance on original training tasks.

  • Takeaways & Limitations

    Counterfactual failures are prevalent across VLAs, while CAG provides a plug-and-play way to strengthen language conditioning without changing architectures or pretrained weights.

Abstract

from arXiv · show

Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to faithfully follow language. When presented with instructions that lack strong scene-specific supervision, VLAs suffer from counterfactual failures: they act based on vision shortcuts induced by dataset biases, repeatedly executing well-learned behaviors and selecting objects frequently seen during training regardless of language intent. To systematically study it, we introduce LIBERO-CF, the first counterfactual benchmark for VLAs that evaluates language following capability by assigning alternative instructions under visually plausible LIBERO layouts. Our evaluation reveals that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs. We propose Counterfactual Action Guidance (CAG), a simple yet effective dual-branch inference scheme that explicitly regularizes language conditioning in VLAs. CAG combines a standard VLA policy with a language-unconditioned Vision-Action (VA) module, enabling counterfactual comparison during action selection. This design reduces reliance on visual shortcuts, improves robustness on under-observed tasks, and requires neither additional demonstrations nor modifications to existing architectures or pretrained models. Extensive experiments demonstrate its plug-and-play integration across diverse VLAs and consistent improvements. For example, on LIBERO-CF, CAG improves $π_{0.5}$ by 9.7% in language following accuracy and 3.6% in task success on under-observed tasks using a training-free strategy, with further gains of 15.5% and 8.5%, respectively, when paired with a VA model. In real-world evaluations, CAG reduces counterfactual failures of 9.4% and improves task success by 17.2% on average.

I. INTRODUCTION

VLAs can disregard language under weak scene-specific supervision, defaulting to visually learned behaviors. The paper introduces LIBERO-CF to expose this failure and CAG to improve language conditioning without changing existing models.

  • Motivation: Counterfactual failures occur when VLAs ignore instructions and execute well-learned scene tasks or select frequently trained objects.The authors link this behavior to task-specific, visually dominant datasets and modality imbalance.
  • Motivation: Existing data-centric approaches do not provide a systematic benchmark or general solution for counterfactual language following in robotic VLAs.Prior work includes linguistic rephrasing and counterfactual data augmentation in other domains.
  • Contributions: LIBERO-CF evaluates counterfactual language following by assigning alternative feasible instructions under LIBERO scene layouts.The benchmark covers alternative objects, background objects, long-horizon tasks, and out-of-distribution objects.
  • Contributions: CAG mixes actions from a standard VLA and a language-unconditional vision-action policy to strengthen language conditioning without modifying architectures or pretrained weights.The scheme is designed as a plug-and-play inference method across diverse VLAs.
  • Contributions: The paper reports consistent improvements over state-of-the-art VLAs in LIBERO-CF and real-world experiments.The contribution claims cover both language grounding and task success.

A. Preliminaries

The paper frames VLA behavior as language-conditioned action prediction but argues that visual cues can dominate this conditioning. Dataset and architectural modality imbalance encourage shortcuts that make models default to well-observed tasks.

  • VLA Formulation: VLAs learn policies that map visual observations and language instructions to action chunks for closed-loop robotic manipulation.The policy is trained by imitating expert actions conditioned on both modalities.
  • Sources of Vision Bias: Robotic datasets provide less textual diversity than visual and action data, encouraging VLAs to prioritize visual cues over language.Static instructions often accompany continuous trajectories, while task types remain limited.
  • Sources of Vision Bias: Language tokens are vastly outnumbered by visual tokens in typical VLAs, risking marginal textual influence during action prediction.The paper identifies this as an architectural source of modality imbalance.
  • Problem Formulation: The Bayesian formulation separates a vision-only action prior from a language–action compatibility likelihood.The prior is P(a | o), while the likelihood is P(l | a, o).
  • Problem Formulation: When the posterior collapses toward the vision-only prior, language has limited influence on action prediction.This vision-dominated bias explains why VLAs can default to well-learned tasks under counterfactual instructions.
  • Problem Formulation: Under fixed visual observations, only a subset of feasible tasks receives sufficient scene-specific demonstrations, leaving other tasks under-observed.The paper studies whether models follow these under-observed instructions rather than defaulting to well-learned tasks.

IV. THE LIBERO-CF BENCHMARK

LIBERO-CF tests whether VLAs follow alternative feasible instructions within familiar layouts rather than reverting to original training behaviors. It combines four counterfactual suites with grounding, success, and faithfulness measurements.

  • Benchmark Design: LIBERO-CF assigns under-observed or unseen alternative tasks to the same LIBERO scene layouts used for VLA finetuning.This directly evaluates language following beyond the original in-domain manipulation tasks.
  • Suites: The benchmark contains CF-Spatial, CF-Object, CF-Long, and CF-OOD suites.These suites target spatial grounding, alternative objects, long-horizon instructions, and unseen objects, respectively.
  • Metrics: Grounding measures contact with the instructed target regardless of completion, while success measures whether the robot completes the task.Grounding captures language faithfulness; success is the stricter execution metric.
  • Evaluation Dimensions: The evaluation labels behavior Faithful when the instructed task is followed and Biased when the original training task is executed instead.This explicitly quantifies defaults to scene-specific behaviors.

B. Validating Vision Shortcuts of VLAs

VLAs often rely on visual shortcuts rather than language, defaulting to well-learned behaviors and training-task objects under counterfactual instructions. Ablations and object-removal experiments provide evidence that these visual attractors weaken language-conditioned action selection.

  • OpenVLA-OFT, π0, and π0.5 are evaluated as representative VLA baselines using officially released LIBERO weights.
  • Vision-only inputs preserve high performance, whereas language-only inputs cause near-zero performance, indicating visual cues dominate action prediction.On LIBERO-Goal, vision-only success is 10.0% for OpenVLA-OFT and 10.2% for π0.5.
  • Under counterfactual instructions, π0.5 grasp distributions spread toward both instructed objects and training-task objects, revealing default execution of learned tasks.
  • 25.4% grounding and 20.0% success improvements occur on average when training-task objects are removed from CF-Spatial scenes.The comparison uses identical counterfactual instructions across CF-Spatial and CF-Focused.
  • Training-task objects act as strong visual attractors, motivating counterfactual evaluation and guidance to strengthen language conditioning.
  • Classifier-free guidance offers an inference-time, model-agnostic way to amplify conditioning without architectural changes, motivating the action-guidance formulation.

C. Dual-branch Inference with CAG

CAG combines a language-conditioned VLA with an unconditioned vision-only policy during inference, either by dropping language from the same model or by using a separately trained VA model. Across LIBERO-CF evaluations, both variants improve language grounding and task success, with VA generally stronger.

  • CAG assembles conditioned and unconditioned policies at inference time, preserving existing VLA architectures and pretrained weights.
  • The training-free variant approximates the unconditioned policy by dropping language input from the standard trained VLA.
  • A separately trained Vision-Action model provides a cleaner vision-only prior and generally yields stronger, more stable language grounding.
  • π0.5 baseline grounding rises from 30.8% to 40.5% with TF and 46.3% with VA on LIBERO-CF.
  • π0.5 success increases from 13.2% to 21.7% with CAG, while VA reduces biased grounding by 13.4% and biased success by 24.7%.
  • CAG benefits all evaluated VLAs, but reduces counterfactual failures more on π0.5 than on OpenVLA-OFT.The reported variation suggests CAG reweights the action posterior more effectively when vision shortcuts are weaker.

C. Ablation Studies

The ablations show that CAG improves language grounding and task success without training, while a separately trained VA branch strengthens performance. Guidance scale must be tuned because excessive guidance harms manipulation accuracy.

  • Training Strategies: Training-free CAG consistently improves grounding and success rates over the vanilla π0.5 baseline.Separately trained VA branches perform better than the training-free strategy, with LoRA and full finetuning achieving comparable overall performance.
  • Training Strategies: Language-dropout training performs poorly because weakened instruction conditioning increases vision-dominated behavior.
  • Training Strategies: A fully finetuned VA branch further improves success by 4.3% on CF-Spatial, indicating better manipulation accuracy.
  • Guidance Scales: Increasing guidance scale progressively mitigates vision shortcuts, but over-guidance degrades manipulation accuracy.The best settings are ω = 1.5 for π0 and π0.5, and ω = 3.0 for OpenVLA-OFT.
  • Real-World Evaluation: The real-world evaluation covers object recognition, spatial reasoning, goal execution, and out-of-distribution generalization using under-observed tasks.Experiments use random object positions and 10 trials per instruction.

C. Results

Across real-world evaluations, π0.5 exhibits counterfactual failures on under-observed tasks, while CAG improves grounding and success across object, spatial, goal, and OOD challenges without reducing in-domain performance.

  • Evaluation Scope: The real-world evaluation spans object recognition, spatial reasoning, goal execution, and OOD generalization across scenes.
  • Object Recognition: CAG achieves 100% average grounding and improves success rates by 13.3% in both object-recognition scenes.
  • Spatial Reasoning: CAG improves spatial grounding by 16.6% and task success by 13.3% when identical objects must be selected using spatial language.Baseline grounding on under-observed targets is as low as 20% and 30% in one spatial arrangement, and 60% and 40% in another.
  • Goal Targeting: CAG raises goal-targeting grounding to 96.7% and increases task success by 36.7% when the same object has different intended goals.
  • OOD Generalization: On zero-shot OOD objects, counterfactual failures compromise π0.5’s pretrained generalization, while CAG mitigates this issue.Four of five Cube failures are counterfactual failures.
  • Overall Results: CAG improves grounding and task success on under-observed tasks while preserving performance on the well-learned in-domain task.

APPENDIX

The appendix details CAG’s derivation, additional simulation experiments, and extended real-world evaluations. Its formulation treats CAG as inference-time reweighting that strengthens language likelihood while preserving the execution prior.

  • Appendix B: Appendix B adds simulation experiments on X-VLA and discusses dynamic guidance scale.
  • Appendix C: Appendix C extends real-world evaluation to long-horizon reasoning and provides detailed scene descriptions and complete result breakdowns.
  • Appendix A: Appendix A derives CAG from a Bayesian decomposition of the conditional action distribution.
  • Appendix A: CAG combines conditional and unconditional policies, with guidance scale ω controlling language-conditioning strength.
  • Appendix A: The derivation shows that CAG reweights actions using language likelihood while retaining the visual execution prior.

APPENDIX B EXTENDED SIMULATION EXPERIMENTS

Additional simulation results show that counterfactual failures extend across VLA architectures, while both training-free and Vision–Action variants of CAG improve language grounding and task success without architectural or checkpoint modifications.

  • Additional VLA Baseline: X-VLA exhibits counterfactual failures similar to other VLAs, indicating that the failure mode is not specific to particular architectures.The results attribute this pattern to reliance on scene-specific visual priors and limited robustness to alternative feasible instructions.
  • Quantitative Results: Training-free CAG increases X-VLA’s average grounding from 37.3% to 39.7% and success from 13.8% to 19.5%.The training-free variant also reduces biased success on training tasks.
  • Quantitative Results: The Vision–Action variant raises X-VLA’s average grounding to 41.0% and success to 17.1%.Both CAG variants improve language grounding for X-VLA.
  • General Applicability: CAG is compatible with diverse VLA designs without modifying model architectures or pretrained checkpoints.The appendix reports this property alongside consistent mitigation of counterfactual failures across VLA designs.

APPENDIX C EXTENDED REAL-WORLD EXPERIMENTS

The extended real-world evaluation uses a Franka Research 3 setup and focuses on under-observed tasks, where only one demonstration is collected to create counterfactual evaluation conditions.

  • Real-World Setup: Real-world experiments use a Franka Research 3 robot arm with a Robotiq 2F85 gripper, ZED 2i exterior camera, and ZED Mini wrist camera.The setup follows the DROID platform.
  • Task Supervision: In-domain tasks receive 20 teleoperated expert demonstrations, whereas under-observed tasks receive a single demonstration for minimal warm-up.Under-observed tasks are treated as counterfactual during evaluation.
  • Evaluation Scope: Experiments focus on under-observed tasks to investigate counterfactual failures and validate CAG, while also checking that in-domain performance is preserved.This design evaluates both counterfactual robustness and in-domain behavior.

C. Additional Experiments for Long-Horizon Reasoning

The long-horizon extension tests whether VLAs preserve language grounding across multi-step instructions involving object identity, order, and cardinality. Results show persistent counterfactual failures, while CAG improves grounding and success across long-horizon variants.

  • Additional Dimension: Long-horizon reasoning evaluates whether VLAs follow multi-step instructions and maintain language grounding over extended action horizons.It adds a fifth evaluation dimension beyond object recognition, spatial reasoning, goal targeting, and OOD generalization.
  • Task Design: The benchmark uses two scenes, Move and Pour and Apple and Banana, each containing one well-learned task and two under-observed counterfactual tasks.The scenes test alternative objects, reversed order, and partial instructions.
  • Move and Pour: π0.5 frequently pours the training-task object, coke, instead of the instructed Sprite or Fanta during long-horizon execution.These failures occur more often than in the corresponding short-horizon object-recognition task.
  • Apple and Banana: π0.5 often follows the original apple-then-banana order even when instructions reverse the order or require only the first step.This demonstrates failures in both order and cardinality following.
  • CAG Results: CAG improves grounding and success across all long-horizon variants, including success on the apple-only task.The results associate this with more effective stepwise reweighting of the action distribution and suppression of false continuation.

D. Detailed Results

Detailed real-world results compare CAG and non-CAG rollouts across object recognition, spatial reasoning, goal targeting, OOD generalization, and long-horizon reasoning. The evaluation reports per-instruction outcomes over repeated trials and provides qualitative rollout comparisons.

  • Evaluation Dimensions: The detailed evaluation covers object recognition, spatial reasoning, goal targeting, OOD generalization, and long-horizon reasoning.Qualitative rollout figures are provided for each dimension.
  • Qualitative Results: Real-world comparisons include rollouts with and without CAG, alongside examples for object recognition, spatial reasoning, goal targeting, OOD generalization, and long-horizon reasoning.The corresponding figures are Fig. 8 through Fig. 13.
  • Quantitative Results: Table VII reports each task instruction as successful executions divided by total attempts, with rows denoting instructed targets and columns denoting executed outcomes.This layout supports direct comparison between instructed and executed targets.
  • Task Examples: The real-world task examples include spatial target selection, object recognition, long-horizon pouring, and reversed-order Apple-and-Banana instructions.These examples span the evaluation dimensions and counterfactual task settings.
  • Counterfactual Outcomes: The detailed table entries include alternative drink targets such as Sprite and Fanta and alternative execution outcomes for Apple-and-Banana tasks.These entries expose mismatches between instructed targets and executed behaviors.
Loading 2602.17659v2…