PaperlayerView exact revision on arXiv

Source-linked AI summary

CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models

Catherine Glossop, William Chen, Arjun Bhorkar, Dhruv Shah, Sergey Levine

arXiv:2508.13446v2cs.RO

TL;DR

VLA models struggle with fine-grained instructions, in part because robot datasets provide limited language diversity for similar observations. CAST augments existing datasets with VLM-generated counterfactual labels and behaviors; the paper reports roughly doubled success rates versus standard VLAs in navigation and manipulation, while noting that annotations remain imperfect.

  • Problem

    Existing robot datasets can lack fine-grained language diversity for similar observations, limiting VLA instruction following and risking policies that ignore language.

    Source: §I · p. 1, §III · p. 2

  • Method

    CAST uses VLMs to generate counterfactual language labels and robot actions that augment existing robot datasets.

    Source: §I · p. 1, §VII · p. 7

  • Results

    CAST roughly doubles success rates over equivalent standard VLAs trained without counterfactual labels in both navigation and manipulation.

    Source: §VII · p. 8

  • Takeaways & Limitations

    CAST augmentation without additional human data enables VLAs that attend more closely to instructions by assigning multiple language-action pairs to each observation.

    Source: §VII · p. 7

    CAST annotations remain imperfect, and counterfactual observations could help but would bring challenges in visual consistency and compounding video-generation error.

    Source: §VII · p. 8

Abstract

from arXiv · show

Generalist robots should be able to understand and follow user instructions. Despite providing a powerful architecture for mapping open-vocabulary language instructions to robot actions, current vision-language-action (VLA) models struggle to follow fine-grained commands. One cause for this is a lack of semantic diversity and language grounding in existing robot datasets and, specifically, a lack of fine-grained task diversity for similar observations. To address this, we present a novel method to augment existing robot datasets by leveraging vision-language models to create counterfactual labels. By augmenting existing datasets with these labels, we increase the diversity and granularity of language grounding for robot datasets, ultimately improving the language-following capabilities of VLAs. We evaluate the resulting model's ability to follow language instructions, ranging from simple object-centric commands to complex referential tasks, by conducting vision-language navigation experiments in 3 different indoor and outdoor environments. Our experiments show that counterfactual relabeling (without additional data collection) significantly improves instruction-following in VLA policies, outperforming state-of-the-art methods and doubling the success rate compared to VLAs trained on unaugmented data. We also evaluate our method for manipulation VLAs and find a similar gain in performance on tasks with distractors.

I. INTRODUCTION

CAST targets language-conditioned robot policies that may ignore instructions when observations predict the task, using synthetic counterfactual labels and behaviors to expand training diversity. The paper positions this approach among task-conditioned policies and robot-data augmentation methods, then evaluates it for navigation and manipulation.

I. INTRODUCTION

  • VLA instruction following is limited by training-label diversity; when observations predict tasks more strongly than language, policies can ignore instructions through posterior collapse.Some navigation datasets lack language labels, and adding generic labels can leave policies especially susceptible to posterior collapse.Source: §I · p. 1
  • CAST augments existing robot datasets with alternative instructions and synthesized action behaviors, using VLM-proposed atomic commands and an atomic policy to generate action chunks.The resulting counterfactual language and action data train VLAs to respond to a broader range of instructions.Source: §I · pp. 1–2
  • CounterfactualVLA outperforms VLAs trained without CAST by 27% and improves manipulation performance by 27% over a VLA trained with standard data.The navigation evaluation covers 3 environments and 27 prompts.Source: §I · p. 2

II. RELATED WORK

  • Task-conditioned policies use image goals or language to specify tasks, while CAST seeks to improve VLA language following through more diverse synthetic labels.The paper describes this goal as complementary to architectural and training-strategy changes.Source: §II · p. 2
  • CAST differs from NaVILA by labeling behaviors possible in observed scenes but not actually executed, and presents a scalable pipeline for general language following.Related work also includes dense hindsight labeling and dataset augmentation with low-level policies.Source: §II · p. 2

IV. COUNTERFACTUAL LABEL AUGMENTATION

CAST seeks to make policies attend to instructions by pairing similar observations with distinct language-action behaviors. It generates feasible counterfactual instructions and action chunks that branch from existing trajectories, without requiring counterfactual observations.

IV. COUNTERFACTUAL LABEL AUGMENTATION

  • CAST attaches multiple action-language pairs to an observation so distinct instructions correspond to distinct actions, countering the risk that policies ignore language.The motivation is that instruction labels must correlate with actions even after conditioning on observations.Source: §IV · pp. 2–3
  • A VLM proposes feasible counterfactual instructions and atomic commands at states in existing trajectories, and action chunks branch from those trajectories as counterfactual endings.The overview describes segmenting trajectories into atomic chunks, querying the VLM at decision points, and appending counterfactual endings.Source: Figure 2 · p. 3, §IV · p. 3
  • Action chunking perturbs policies from states covered by existing data toward counterfactual tasks, avoiding the need to generate counterfactual observations.The paper reports that these perturbations can expand language following to novel fine-grained tasks without explicit supervision in the perturbed states.Source: §IV · p. 3

A. How counterfactuals enable language following

The paper frames language following as requiring language to add information about actions beyond what observations reveal. CAST’s counterfactual labels are motivated as a way to increase that information through diverse, predictable atomic commands.

A. How counterfactuals enable language following

  • The paper assumes language following can improve by maximizing observation-conditioned mutual information between language labels and actions.This is presented as a conceptual model, not an established precise relationship for CAST.Source: A. How counterfactuals enable language… · p. 3
  • A language label, action, and atomic command form the Markov chain language → action → atomic command, yielding a lower bound on language-action mutual information.The lemma gives the bound as H(ℓa|o) − H(ℓa|ℓ, o).Source: A. How counterfactuals enable language… · p. 3
  • CAST aims to raise the bound by assigning diverse atomic commands to each observation while keeping each command predictable from its corresponding language label.The paper characterizes the relationship between CAST and these quantities as heuristic.Source: A. How counterfactuals enable language… · pp. 3–4

B. Generating the CAST dataset

CAST constructs a labeled dataset from existing trajectories by establishing atomic behavior and language references, generating counterfactual instructions at selected states, and adding action chunks that branch from the original trajectory.

B. Generating the CAST dataset

  • CAST first constructs an atomic dataset from observations, atomic labels, and actions, and obtains synthetic language labels for the executed trajectories as references.The atomic decomposition and post-hoc language labeling are domain- and dataset-specific.Source: B. Generating the CAST dataset · p. 4
  • At decision points marking atomic-segment boundaries, a VLM receives images, atomic labels, and existing instruction labels to propose plausible alternative instructions and aligned atomic commands.The alternatives describe behaviors the robot could have executed, followed by a new ending to the existing trajectory.Source: B. Generating the CAST dataset · p. 4
  • An atomic policy trained on the atomic dataset generates counterfactual action chunks for proposed commands, which branch from the existing trajectory and are added with their labels as training data.Multiple counterfactuals for an observation provide diverse atomic commands; the resulting chunks are included in the labeled training set.Source: B. Generating the CAST dataset · p. 4

V. IMPLEMENTATION

CAST is instantiated for navigation and manipulation; the navigation implementation builds atomic and hindsight labels, filters them for behavioral consistency, and uses them to train a VLA.

A. Navigation

  • Navigation trajectories are segmented by yaw changes and labeled with six action-derived commands, giving the atomic policy labels that correlate strongly with actions.The policy also inherits collision avoidance from the real-world navigation data.Source: A. Navigation · p. 4
  • A VLM describes objects, structures, and relative positions from subsampled observations, then formats trajectory-level hindsight instructions without using raw actions.A second labeling phase filters instructions against the trajectory’s atomic action sequence.Source: A. Navigation · pp. 4–5
  • The pipeline processes GNM data into 65k hindsight labels for 16k trajectories and generates 86k counterfactual action chunks.Counterfactual actions use an eight-step atomic-policy horizon and are trained only where the starting observation is available.Source: A. Navigation · p. 5
  • The resulting CAST dataset is used to fine-tune a 3B PaliGemma VLM for 40k steps, with added tokenizer tokens for binned actions.Training uses a batch size of 384 and a learning rate of 1e-4.Source: A. Navigation · p. 5

B. Manipulation

For manipulation, CAST retains its core design but replaces the counterfactual command proposer and atomic policy to fit the Bridge dataset and manipulation subtasks.

B. Manipulation

  • Manipulation uses Bridge task labels and open-source subtask decompositions as atomic commands, then proposes counterfactual tasks such as placing an object on another available surface.The counterfactual task-generation pipeline uses a modified prompt.Source: B. Manipulation · p. 5
  • Two Steerable Policies models propose counterfactual atomic commands and predict actions from those commands.Both models are initialized from Prismatic VLMs.Source: B. Manipulation · p. 5
  • Applying the pipeline to 60k Bridge trajectories generates 500k counterfactual instructions and actions.The atomic policy produces non-chunked Cartesian delta actions.Source: B. Manipulation · p. 5
  • The manipulation VLA is trained end-to-end for 60k steps, and these models are called CounterfactualVLAs.Training uses eight H200 GPUs, a batch size of 512, and a learning rate of 1e-4.Source: B. Manipulation · p. 5

VI. EXPERIMENTAL EVALUATIONS

The evaluations assess CAST label quality and test CounterfactualVLA on real-world navigation and manipulation tasks, including robustness to changing conditions.

VI. EXPERIMENTAL EVALUATIONS

  • The evaluation asks whether CAST produces accurate language labels, enables more effective language-conditioned policies for navigation and manipulation, how CounterfactualVLA trained with CAST compares with state-of-the-art methods, and which policy architecture best leverages CAST.Source: §VI · p. 5
  • Navigation evaluation covers 27 instruction-following tasks across three real-world environments, testing object, referential, and continuous navigation.Object navigation requires reaching within 0.5 m; continuous navigation requires staying within 1 m of a referenced object or structure for at least 2 m.Source: §VI · p. 5
  • CAST label accuracy is 60% for hindsight labels and 70% for counterfactual labels.Eleven judges evaluated 440 labels across object or structure accuracy, motion, and grounding of motion in the scene.Source: §VI · p. 5, Figure 3 · p. 6
  • Six hallway navigation tasks test robustness to morning versus night lighting and to a person walking in front of the robot.There are two tasks per instruction category.Source: §VI · p. 6
  • Manipulation evaluation uses six Bridge WidowX tasks with distractor objects, across 120 trials of the standard and counterfactual VLAs.Tasks include both simple picking and more complex placement instructions.Source: §VI · p. 6

A. Evaluating CAST label quality

CAST improves instruction following across navigation and manipulation, while its generated labels are imperfect and the policy remains vulnerable to dynamic obstacles.

A. Evaluating CAST label quality

  • CAST generates hindsight and counterfactual labels at approximately 60% and 70% accuracy, respectively, but motion grounding is the main error source.Despite noisy labels, the resulting policy retains strong collision avoidance and the labels provide sufficient language-action diversity for downstream learning.Source: A. Evaluating CAST label quality · p. 6

B. Evaluating language instruction following

  • CounterfactualVLA achieves a 27% improvement over the standard VLA, which tends toward generic navigation and struggles on object-navigation tasks.Counterfactual labels break the bias toward generic behaviors.Source: B. Evaluating language instruction… · p. 6
  • With distractors, CounterfactualVLA achieves 2× higher success rate on manipulation tasks than the standard VLA.Without distractors the policies perform similarly; distractors make language necessary to distinguish plausible actions.Source: B. Evaluating language instruction… · p. 6

C. Comparing CounterfactualVLA to prior methods

  • CounterfactualVLA outperforms prior methods by 19% on aggregate, with an advantage over CoNVOI on continuous navigation.CoNVOI performs similarly on object and referential navigation, while embodied robot training helps CounterfactualVLA follow nuanced continuous prompts.Source: C. Comparing CounterfactualVLA to… · pp. 6–7, Figure 4 · p. 7
  • The πa + planning baseline performs poorly across tasks, while CAST aligns language annotations with grounded robot experience across trajectories.The baseline’s test-time command decomposition compounds action-chunk errors with VLM scene-reasoning failures.Source: C. Comparing CounterfactualVLA to… · p. 7
  • Dynamic obstacles can make CounterfactualVLA fail to resume its task after avoiding them, while lighting changes do not seem to affect performance.The authors suggest slow inference at 4 Hz or a lack of counterfactual observations may contribute.Source: C. Comparing CounterfactualVLA to… · p. 7

D. Ablating policy architecture for CAST

CAST improves instruction-following without additional human data by assigning multiple language-action pairs to observations. Its architecture comparison finds that VLM-backed policies handle complex instructions better, while CAST annotations remain imperfect.

D. Ablating policy architecture for CAST

VII. DISCUSSION

  • CAST roughly doubles success rates over standard VLAs trained without counterfactual labels in both navigation and manipulation.Assigning multiple language-action pairs to each observation encourages policies to learn the link between language and actions.Source: §VII · pp. 7–8
  • CAST annotations remain imperfect, and improved VLM physical grounding could make the method more effective.The authors also note that counterfactual observations could aid the method, but generating robot images raises visual-consistency and compounding-video-error challenges.Source: §VII · p. 8

APPENDIX

The appendix describes how navigation trajectories are discretized into atomic labels, how label correctness is judged, and how comparison baselines are constructed.

A. Atomic Discretization of Navigation Dataset

  • Navigation trajectories are split at yaw changes into turn segments; otherwise, after 10 steps, segments are labeled Go forward or Stop based on distance traveled.The resulting atomic labels and segment start times are generated independently of observations and are strongly correlated with actions.Source: A. Atomic Discretization of Navigation… · p. 9
  • Bridge manipulation data uses its existing subtasks as atomic units, with labels originally produced by Embodied Chain-of-Thought.The dataset is already decomposed into subtasks.Source: A. Atomic Discretization of Navigation… · p. 9

B. Description of Criteria for Label Quality

  • Label quality requires correct referenced objects or structures, accurate motion descriptions, and correct grounding of motion relative to the scene.For counterfactual labels, grounding also considers whether the generated trajectory would lead to a collision; 11 judges assessed 440 labels against all criteria.Source: B. Description of Criteria for Label… · p. 9

C. Description of Baselines

  • The πa + planning baseline uses an off-the-shelf VLM as a high-level controller and a learned atomic policy for low-level actions.It is most similar to NaVILA, while CoNVOI combines VLM guidance with classical planning and LiDAR-derived occupancy grids.Source: C. Description of Baselines · p. 9

D. Description of navigation tasks

The evaluation covers 27 navigation tasks across three environments and includes manipulation tasks with clutter. The CounterfactualVLA uses a PaliGemma backbone adapted to produce robot actions.

D. Description of navigation tasks

  • Navigation evaluation spans 27 tasks across three environments, covering object, referential, and continuous navigation.Object navigation targets an object; referential tasks require spatial relations, while continuous tasks require sustained behavior relative to a structure.Source: D. Description of navigation tasks · p. 9, Figure 7 · p. 10

E. Description of manipulation tasks

F. Model Architecture and Training

  • CounterfactualVLA uses a PaliGemma VLM backbone and additional action tokens to ground the model in robot actions.The model is fine-tuned with CAST data to follow real-world language instructions; PaliGemma is a 3-billion-parameter model.Source: Figure 9 · p. 11, F. Model Architecture and Training · p. 11

G. Detailed Experimental Results

The appendix reports the scale of real-world evaluations and points to detailed navigation and manipulation results. It also describes VLM-based relabeling, including a counterfactual-generation prompt for manipulation data.

G. Detailed Experimental Results

H. VLM Relabeling

  • Gemini 2.5 Pro is used for counterfactual relabeling and filtering, and the method is described as compatible with similar VLMs.For Bridge manipulation data, counterfactual generation is applied without regenerating language labels because the dataset already has task instructions.Source: H. VLM Relabeling · p. 12
  • The manipulation prompt asks the VLM to propose concise alternative instructions based on the current robot and scene observation.Proposed instructions can involve another object, a different placement, or movement relative to another object.Source: H. VLM Relabeling · p. 12

Related papers

Loading 2508.13446v2…