PaperlayerView exact revision on arXiv

Source-linked AI summary

LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries

Shijie Lian, Bin Yu, Xiaopeng Lin, Laurence T. Yang, Zhaolong Shen, Changti Wu, Yuzhuo Miao, Cong Huang, Kai Chen

arXiv:2601.15197v7cs.AIcs.CLcs.CVcs.RO

TL;DR

VLA models can rely on visual shortcuts and struggle to follow language in novel or ambiguous settings. LangForce contrasts vision-only and language-conditioned policies using Bayesian decomposition, improving language grounding and OOD performance across benchmarks and real-world tasks.

  • Problem

    Current VLA models struggle to generalize to novel instructions and complex multi-task scenarios because biased robotic datasets encourage vision-based shortcuts.

  • Method

    LangForce uses Latent Action Queries and dual-branch Bayesian training to maximize conditional PMI between actions and instructions, penalizing information inferable from vision alone.

  • Results

    LangForce improves over VLA baselines across SimplerEnv, RoboCasa, LIBERO, and real-world pick-and-place settings, including an 8.8% OOD improvement on SimplerEnv.

  • Takeaways & Limitations

    The results support using shortcut-preventing objectives to improve language grounding in ambiguous and OOD VLA scenarios without inference overhead.

  • Takeaways & Limitations

    The dual-branch architecture theoretically increases training computation, although reusing shared visual representations makes the actual increase marginal.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown promise in robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a dataset bias. In such datasets, language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into vision-only policies that ignore language constraints and fail in out-of-distribution (OOD) settings. To address this, we propose LangForce, a novel framework that enforces instruction following via Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a vision-only prior $p(a \mid v)$ and a language-conditioned posterior $π(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the vision shortcut and rewards actions that explicitly explain the language command. Without requiring new data, LangForce significantly improves generalization. Extensive experiments across on SimplerEnv and RoboCasa demonstrate substantial gains, including an 11.3% improvement on the challenging OOD SimplerEnv benchmark, validating the ability of our approach to robustly ground language in action.

1. Introduction

Current VLA models can rely on dataset-specific visual correlations instead of language, limiting generalization to novel instructions and OOD settings. LangForce addresses this vision shortcut with Bayesian decomposition, Latent Action Queries, and a PMI-based objective that strengthens language-conditioned action policies.

  • Problem: Current VLA models perform strongly in-distribution but struggle with novel instructions, complex multi-task scenarios, and OOD environments.Narrow post-training datasets can also cause catastrophic forgetting of pretrained VLM capabilities.
  • Vision Shortcut: When instructions are predictable from visual observations, the likelihood collapses to the vision-only term and the posterior degenerates to the prior.This Bayesian failure makes language unnecessary for predicting actions.
  • Vision Shortcut: The resulting vision shortcut ignores language and fails when tasks are ambiguous or environments change.RoboCasa training data illustrates how limited task diversity encourages execution from visual cues rather than instructions.
  • LangForce: LangForce maximizes conditional PMI, or log p(ℓ| a, v) − log p(ℓ| v), requiring actions to provide information about instructions beyond vision alone.The objective explicitly penalizes the vision shortcut.
  • LangForce: LangForce uses Latent Action Queries in dual Priori and Posteriori branches to learn p(a | v) and π(a | v, ℓ), respectively.The queries bottleneck action-relevant features for a downstream Diffusion Transformer policy while enabling Bayesian training.
  • Results: 8.8% improvement in OOD generalization on SimplerEnv accompanies state-of-the-art performance on SimplerEnv and RoboCasa.The reported gains support LangForce’s effectiveness in breaking the vision shortcut.

2. Motivation: Vision Shortcut

Standard VLA training can produce vision-only policies because goal-driven datasets make language predictable from visual observations, causing instruction-action information to collapse. Pilot evaluations show comparable in-distribution performance without language but near-total failure under ambiguous or out-of-distribution conditions.

  • RoboCasa: 44.6% vision-only success on RoboCasa is close to the 47.8% language-conditioned baseline across 24 tasks.The small gap indicates the model can succeed without relying on language instructions.
  • LIBERO: Vision-only performance approaches the full VLA model on LIBERO Spatial, Object, and Long, reaching 90.2%, 99.6%, and 86.0% versus baseline values of 97.8%, 98.8%, and 92.0%.These subsets associate visual scenes with relatively unambiguous tasks.
  • LIBERO Goal: LIBERO Goal exposes the shortcut because identical object configurations can correspond to multiple valid tasks, which vision-only models cannot disambiguate.The model is consequently dominated by the prior p(a | v) learned from dataset statistics.
  • OOD Generalization: 0.13 vision-only action loss on BridgeDataV2 and Fractal is comparable to the full model’s 0.08, revealing visual shortcuts even in diverse in-the-wild data.However, the vision-only baseline achieves near 0% success on visually distinct OOD SimplerEnv environments.
  • Information Collapse: In goal-driven datasets, the deterministic mapping v → ℓ makes H(ℓ|v) ≈ 0, forcing I(ℓ; a | v) to vanish and preventing language-action dependence beyond vision.This formalizes the vision shortcut as information collapse and motivates explicitly maximizing action-provided information.

3. Method: LangForce

LangForce counters information collapse by Bayesianly decomposing VLA action prediction into a vision-only prior and language-conditioned posterior. Learnable Latent Action Queries and an LLR objective enforce language-relevant action representations while preserving standard inference cost.

  • Bayesian Decomposition: LangForce maximizes conditional PMI through a Log-Likelihood Ratio between posterior π(a | v, ℓ) and prior p(a | v) to counteract information collapse.The objective regularizes the policy so actions carry information that explains the instruction.
  • Dual-Branch Training: The Priori Branch places queries before language, so causal masking makes their hidden states vision-only and estimates p(a | v).Prior query states are detached during prior optimization, confining those gradient updates to the DiT action head.
  • Dual-Branch Training: The Posteriori Branch places queries after language, allowing them to attend to vision and language while optimizing the expert action with a main flow-matching loss.Both branches share VLM weights but use different input arrangements.
  • Latent Action Queries: Latent Action Queries use K = 64 learnable tokens as a bottleneck between the shared VLM and continuous action head.Only query hidden states are forwarded to the action head, rather than hidden states from all input tokens.
  • Training Objective and Inference: During inference, LangForce exclusively executes the Posteriori Branch and generates actions through the DiT without additional computational overhead.The final loss combines both branch action losses with LLR regularization, using λ = 0.3 and β = 0.1.

4. Experiment

LangForce achieves strong simulation and real-world results, with especially large gains in language-sensitive and out-of-distribution settings. Ablations attribute these improvements primarily to Bayesian decomposition and PMI-based language-action modeling, while Latent Action Queries provide an additional architectural benefit.

  • SimplerEnv: 66.5% average success rate makes LangForce state of the art on SimplerEnv, improving 11.3% over the same-framework QwenGR00T baseline at 55.2%.LangForce also surpasses π0.5 and Isaac-GR00T-N1.6, both at 57.1%.
  • RoboCasa: 52.6% average success rate makes LangForce state of the art on RoboCasa, exceeding QwenOFT at 48.8%, Isaac-GR00T N1.5 at 48.2%, and QwenGR00T at 47.8%.LangForce also outperforms the VisionOnly baseline at 44.7%.
  • Multimodal capability: LangForce preserves coherent language interaction during action generation, unlike standard QwenGR00T, because its LLR objective enforces dependence on language.This prevents linguistic representations from collapsing or becoming redundant during VLA fine-tuning.
  • Ablation study: 63.5% for full LangForce versus 57.5% for + Action Query yields a +6.0% gain, attributing the core improvement to dual-branch Bayesian learning.The objective explicitly maximizes PMI between instructions and actions to overcome the vision shortcut.
  • Ablation study: Adding Latent Action Queries improves QwenGR00T from 55.2% to 57.5%, indicating an architectural inductive bias beyond the Bayesian objective.The query mechanism compresses task-relevant information into a compact representation for action decoding.

5. Related Work

Related work advances generalist robot policies through large-scale datasets, benchmarks, and increasingly capable VLA architectures. However, existing approaches lack a systematic solution to the vision shortcut that causes models to ignore language instructions in goal-driven datasets.

  • Vision-Language-Action Dataset and Benchmark: LIBERO, BridgeData V2, OXE, Droid 2.0, and AgiBot-World progressively expand robot-learning datasets and benchmarks.OXE aggregates data across 22 robot embodiments, while AgiBot-World scales training data to over 1 million trajectories with human-in-the-loop verification.
  • Vision-Language-Action Models: VLA models bridge semantic understanding and physical control through language-model fine-tuning, diffusion transformers, and dual-system designs.OpenVLA and OpenVLA-OFT fine-tune large language models, while other systems introduce diffusion-transformer or dual-system architectures.
  • Vision-Language-Action Models: Recent VLA approaches improve embodiment transfer, spatial reasoning, visual imagination, and catastrophic-forgetting resistance through specialized prompts, 3D encodings, video prediction, and coordinated brains.X-VLA uses embodiment-specific soft prompts; SpatialVLA injects 3D information; VideoVLA predicts actions and future visuals; TwinBrainVLA coordinates frozen and trainable systems.
  • Vision-Language-Action Models: Existing VLA approaches lack systematic solutions to the vision shortcut problem, allowing models to ignore language instructions in goal-driven datasets.This limitation motivates methods that explicitly address language grounding in action selection.

6. Discussion

The discussion advocates prioritizing ambiguous scenarios so language is needed for disambiguation, while positioning LangForce as a practical way to exploit existing VLA data despite persistent shortcut risks.

  • Data collection should prioritize ambiguous scenarios that compel models to use language for visual disambiguation.
  • LangForce improves the use of existing goal-deterministic VLA datasets without requiring expensive large-scale recollection or re-annotation.The framework addresses the practical limitation that much existing data was collected under approximately goal-deterministic settings.
  • Shortcut behavior can still emerge from local scene-to-task correlations even when datasets become broader, motivating an explicit optimization objective.

7. Limitation

LangForce’s dual-branch design theoretically increases training cost because it computes prior and posterior branches, but shared visual-prefix prefill makes the practical increase marginal. Real-world evaluation remains limited to simple pick-and-place tasks focused on instruction following rather than dexterous manipulation.

  • Computational overhead: The dual-branch architecture theoretically increases computational cost per training iteration by computing both prior and posterior branches.The limitation arises from evaluating both branches during training.
  • Computational overhead: Shared visual inputs and prefix prefill allow visual representations to be computed once and reused across both branches, making the actual training-time increase marginal.Visual tokens vastly outnumber language and latent action query tokens, so reusing vision encoder outputs reduces redundant computation.
  • Real-world evaluation scope: Real-world experiments cover relatively simple pick-and-place settings that primarily assess instruction following rather than dexterous manipulation.This task choice aligns with LangForce’s language-grounding goal and makes failures easier to attribute to instruction following rather than execution errors.

8. Conclusion

The paper identifies a vision shortcut in VLA training under approximately goal-deterministic data and introduces LangForce to encourage instruction-specific action information. LangForce uses a Bayesian contrast between language-conditioned and vision-only action distributions, implemented with Latent Action Queries and a shared dual-branch architecture.

  • LangForce addresses a vision shortcut in which VLA policies rely on visual correlations rather than language under approximately goal-deterministic data.
  • The framework analyzes this failure mode through Bayesian and informationtheoretic perspectives.
  • LangForce contrasts a language-conditioned posterior with a vision-only prior to encourage instruction-specific action information.Its architecture incorporates Latent Action Queries and a shared dual-branch design.

A. Derivation of the LLR Objective · B. Additional Experiments

The LLR objective is derived by expressing conditional PMI between actions and instructions as a ratio between language-conditioned and vision-only policies. Maximizing this ratio penalizes vision-only collapse and favors actions that make instructions more probable, especially for ambiguous or OOD tasks.

  • A. Derivation of the LLR Objective: Conditional PMI between action a and instruction ℓ given observation v is the learning signal optimized by LangForce.The section introduces PMI as the formal motivation for the LLR objective.
  • A. Derivation of the LLR Objective: Applying the probability chain rule rewrites PMI as a log-ratio involving the posterior policy π(a | v, ℓ) and vision-only prior p(a | v).This connects the information-theoretic objective directly to the two policy distributions.
  • A. Derivation of the LLR Objective: The objective maximizes divergence between the language-conditioned policy and vision-only prior, directly penalizing collapse when π(a | v, ℓ) ≈ p(a | v).The collapse corresponds to language failing to alter the action distribution.
  • A. Derivation of the LLR Objective: A second chain-rule form yields the practical LLR objective used by LangForce.This alternative decomposition uses p(ℓ | a, v) and p(a | v).
  • A. Derivation of the LLR Objective: The LLR objective compares the log-likelihood of the instruction given action and vision with its log-likelihood given vision alone.It measures how much the selected action contributes beyond the visual context.
  • A. Derivation of the LLR Objective: Maximizing LLR selects actions that make the instruction significantly more probable than visual context alone would make it.The stated purpose is to extract information needed for ambiguous or out-of-distribution tasks.

B.1. Additional Experiments on LIBERO

LangForce performs comparably on LIBERO’s Spatial, Object, and Long suites while achieving a 99.4% success rate on Goal, surpassing Qwen3-VL-GR00T by 2.0%. Its higher instruction NLL and PPL indicate preserved visual uncertainty on the LIBERO Goal dataset despite repetitive language structure.

  • LIBERO Benchmark: 99.4% success on LIBERO Goal surpasses Qwen3-VL-GR00T’s 97.4% by +2.0%.LangForce performs comparably on the Spatial, Object, and Long suites, where LIBERO baselines exceed 95%.
  • LIBERO Benchmark: LangForce and baselines achieve comparable performance on LIBERO’s Spatial, Object, and Long suites.The evaluation trains one policy across all four suites and reports Avg@500 success rates.
  • Conditional Entropy Analysis: 9.47 nats/token NLL for LangForce exceeds QwenGR00T’s 8.51 nats/token on instructions given visual observations.NLL is computed over 40,000 samples as a proxy for conditional entropy H(ℓ|v).
  • Conditional Entropy Analysis: 12964.9 PPL for LangForce exceeds the baseline’s 4964.1, indicating greater preserved uncertainty from visual observations alone.Higher NLL/PPL is described as preventing the information collapse observed in baselines.
  • Conditional Entropy Analysis: Despite highly repetitive LIBERO Goal language, LangForce achieves higher instruction uncertainty metrics than the baseline.The reported language pattern includes templates such as “put the [object] in/on the [receptacle]”.

B.2. Additional Experiments on RoboCasa · B.3. Real-World Experiments Setup · B.4. Preservation of General Capabilities

Additional RoboCasa results expose a strong vision shortcut, while the real-world setup tests instruction following and OOD robustness. LangForce preserves text-only language behavior and action-relevant language grounding, though broader vision-language conversation may degrade after VLA training.

  • B.2. Additional Experiments on RoboCasa: 44.7% success rate for VisionOnly nearly matches the 47.8% of QwenGR00T on RoboCasa, revealing a substantial vision shortcut.The passage states that LangForce breaks this pattern, but the provided excerpt does not include its quantitative result.
  • B.3. Real-World Experiments Setup: LangForce is evaluated on a Franka Research 3 arm with an Intel RealSense D435 camera in a real-world colored-block pick-and-place task.The task requires picking specified colored blocks and placing them into a box.
  • B.2. Additional Experiments on RoboCasa: Table 9 compares VLA models in the RoboCasa Tabletop simulation environment using results from Isaac-GR00T and starVLA experiment sources.The table highlights the best results in bold and the second-best results with underlining.
  • B.3. Real-World Experiments Setup: 100 expert demonstrations are collected for each Green, Blue, and Yellow block, while Red is held out for OOD generalization.Multiple colored blocks appear simultaneously during collection, and testing adds cabinets and wrenches as distractors.
  • B.4. Preservation of General Capabilities: LangForce preserves normal text-only conversational abilities after VLA training, but general vision-language conversation may experience some degradation.The passage attributes this primarily to adaptation of the vision tower and multimodal projection layers for control.
  • B.4. Preservation of General Capabilities: LangForce prevents collapse of the backbone’s text-only language behavior by using its LLR objective to enforce explicit language grounding.Standard baselines can render instruction tokens redundant through vision shortcuts, weakening language-processing signals and causing failures on pure text queries.
  • B.4. Preservation of General Capabilities: An auxiliary p(ℓ| v) objective yields only limited policy benefit, remaining far below LangForce and showing that generic language preservation is insufficient.The ablation uses the same QwenGR00T backbone and reconstructs training instructions from visual scenes during VLA fine-tuning.
  • B.4. Preservation of General Capabilities: Auxiliary language prediction can preserve surface conversational fluency but may overfit to robot-training instructions or scene-specific associations.This qualitative finding distinguishes generic language retention from action-relevant language grounding.

B.5. Ablation on λ and β in Loss Function · B.6. Ablation on Number of Latent Action Queries · C. Discussion

Ablations show that LangForce benefits from both Bayesian-branch design and LLR optimization, while increasing latent action queries improves performance up to saturation. The discussion highlights ambiguous data, human video, and world models as directions for mitigating vision shortcuts and improving VLA robustness.

  • B.5. Ablation on λ and β in Loss Function: 63.3% average success at λ = 0 exceeds the QwenGR00T baseline of 55.2%, showing that the LLR objective alone regularizes the policy.At λ = 0, the prior branch is used solely for LLR calculation without explicit action supervision.
  • B.5. Ablation on λ and β in Loss Function: 66.5% success with β = 0.1 exceeds 61.3% without LLR optimization and the 55.2% baseline, supporting the value of mutual-information maximization.The β = 0 configuration still benefits from explicitly separating vision-only and language-conditioned pathways.
  • B.6. Ablation on Number of Latent Action Queries: Increasing latent action queries from 16 to 64 substantially improves success rate, while doubling them to 128 leads to performance saturation.This ablation isolates the query mechanism without incorporating Bayesian decomposition.
  • C. Discussion: Goal-driven datasets can induce a vision shortcut because visual scenes deterministically predict language instructions, with H(ℓ| v) ≈0.The discussion hypothesizes that collecting ambiguous scenarios, where the task cannot be inferred from the initial observation alone, could mitigate this bias.
  • C. Discussion: Human activities may provide less sharply determined p(ℓ| v) than curated robot data because they are multimodal, context-dependent, and behaviorally diverse.The paper conjectures that injecting action knowledge from large-scale human video could improve robustness.
  • C. Discussion: World-model VLA formulations can infer actions by imagining a language-consistent future state and combining forward dynamics with an action prior.The formulation treats future state vt+1 as a latent variable conditioned on language and marginalizes future prediction over actions.
  • C. Discussion: LangForce is illustrated in SimplerEnv rollouts trained on BridgeDataV2 and Fractal Dataset frames, indicating a domain gap between training and inference environments.The paper also presents real-world vegetable pick-and-place rollouts following specified language instructions.
Loading 2601.15197v7…