PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach

Siyuan Yang, Yang Zhang, Haoran He, Ling Pan, Xiu Li, Chenjia Bai, Xuelong Li

arXiv:2512.02834v1cs.ROcs.AI

TL;DR

Fine-tuned VLAs remain fragile because redundant action modes and distribution shift make success depend strongly on sampled inference noise. TACO uses a lightweight pseudo-count verifier to select supported action chunks at test time, and experiments report improved stability and success across simulation and real-world tasks.

  • Problem

    After supervised fine-tuning, VLAs can retain redundant action modes, causing success rates to vary drastically with inference noise.

  • Method

    TACO applies test-time anti-exploration by using a lightweight pseudo-count estimator to verify and select action chunks without policy optimization.

  • Results

    TACO significantly improves inference stability and success rates across diverse VLA models, simulation benchmarks, and real-world tasks.

  • Takeaways & Limitations

    Lightweight test-time scaling can constrain VLA inference toward successful downstream modes while preserving pretrained generalization.

  • Takeaways & Limitations

    TACO assumes demonstrations reflect success density and cannot evaluate newly synthesized actions produced purely by perturbation.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models, trained via flow-matching or diffusion objectives, excel at learning complex behaviors from large-scale, multi-modal datasets (e.g., human teleoperation, scripted policies). However, since VLAs incorporate diverse data modes in the pre-training stage, and the finetuning dataset often contains demonstration data collected in a kinematically suboptimal or undesirable way, it exists redundant action modes that are irrelevant to the success action modes of the downstream task. Specifically, we observe a critical inference-time fragility among various sampled noises after supervised finetuning of pre-trained VLAs. In this paper, we attribute this instability to the distribution shift between the VLA policy and the policy induced by stable success modes of the downstream task dataset. Thus, we propose \textbf{TACO}, a test-time-scaling (TTS) framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks. The VLA models integrated with TACO can execute the actions with maximum pseudo-count from all sampled action chunks, thereby preventing distribution shifts while preserving the generalization ability of VLAs since the constraint is applied only during inference. Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits compared to RL update, especially for flow or diffusion-based VLAs which are difficult to perform RL update due to denoising process. Extensive experiments across four simulation benchmarks (RoboTwin2.0, Robotwin, LIBERO, SimplerEnv) and a dual-arm platform demonstrate that our method significantly improves the inference stability and success rates in downstream-task adaptations.

1 Introduction

VLAs can generalize strongly yet remain highly sensitive to inference noise because redundant action modes persist after fine-tuning. TACO addresses this instability by selecting action chunks with high pseudo-counts at test time, improving performance without changing model weights.

  • Inference fragility: 0% to 80% success rates result solely from changing the fixed inference noise for the same SFT-adapted VLA and scenarios.Figure 1 compares fixed-noise evaluations for π0 and RDT using floating bars.
  • Source of instability: Redundant action modes from pretraining and multimodal SFT demonstrations can remain unrelated to downstream task success.These modes arise from diverse data sources and varying or suboptimal execution styles.
  • TACO: TACO applies test-time anti-exploration by constraining generated actions toward successful modes represented in the downstream SFT data.The approach uses test-time scaling rather than policy optimization.
  • TACO: A lightweight pseudo-count estimator serves as a verifier, allowing TACO to retain pretrained VLA generalization while constraining outputs during inference.The estimator measures action-mode support with minimal computational overhead.
  • Results: TACO significantly improves success rates across diverse VLA models in simulation and real-world tasks without prolonged training and with low latency.The contribution statement reports broad simulation and real-world gains rather than a single benchmark value.

2 Related Work

Related work frames VLA inference instability as analogous to offline-RL distribution shift and situates TACO within test-time scaling methods. Unlike prior verifier approaches, TACO uses lightweight internal VLA representations without modifying the backbone.

  • Vision-Language-Action Models: Generative VLA research has increasingly adopted diffusion and related generative policies to represent multimodal, precise action distributions.The related-work discussion names RDT-1B, DexVLA, π0, GR00T N1, and SmolVLA among these developments.
  • Anti-exploration & Offline RL: Offline RL faces distributional shift between behavior data and learned policies, while VLAs similarly retain broad action modes beyond narrower successful downstream behaviors.Such mismatch can produce suboptimal sampled actions and motivates restricting policies toward successful data support.
  • Test Time Scaling in VLA: Test-time scaling selects better generated candidates with an additional scoring module without modifying network weights.This contrasts with approaches requiring annotated reasoning traces or more complex training.
  • Test Time Scaling in VLA: Prior verifier-based approaches often rely on complex RL training or large verifiers, whereas the paper targets a lightweight verifier using internal VLA features.The stated novelty is exploiting the backbone’s representations without modifying it.

3 Preliminaries

The paper formulates downstream VLA adaptation from scarce demonstrations containing language, multimodal visual observations, proprioception, and action sequences. It introduces Coin Flipping Networks as a pseudo-count estimator for measuring state visitation support.

  • Problem Statement: The SFT dataset contains demonstrations represented by instructions, sequences of multimodal observations, proprioceptive states, and actions.Each observation includes multiple RGB images and a robot proprioceptive state.
  • Problem Statement: Scarce expert demonstrations cover only a limited fraction of observation and action spaces, potentially worsening distribution shift.The limitation follows from the high cost of obtaining expert demonstrations.
  • Coin Flipping Network: A Coin Flipping Network estimates state visitation counts from random binary vectors paired with each encountered state.The estimator learns from a dataset of state–coin-flip pairs.
  • Coin Flipping Network: The learned representation norm approximates inverse state visitation count, enabling CFN-based anti-exploration.The supplied formulation gives ∥fϕ(s)∥2/d ≈ 1/N(s).

4 Method

TACO frames VLA inference instability as an anti-exploration problem and uses a lightweight pseudo-count verifier to select in-support action chunks at test time. It couples VLA internal representations with a CFN, searches for high-fidelity features for denoising-based models, and selects the candidate with the highest estimated count.

  • Inference Instability as an OOD Problem: SFT-adapted VLAs remain multimodal because pretraining and demonstration datasets retain redundant or suboptimal action modes, making sampled actions potentially unsuccessful.The paper models the dataset as a mixture of a desired success mode and suboptimal modes, while the imitation policy cannot distinguish them.
  • Test-Time Scaling as Anti-Exploration: TACO generates multiple action chunks and internal representations, scores them with the CFN, and executes the candidate with the maximum pseudo-count.The two-stage generate-then-verify procedure applies the anti-exploration objective directly during inference.
  • Inference Instability as an OOD Problem: In a contextual-bandit formulation, anti-exploration selects the action chunk maximizing success probability minus an out-of-support penalty, which the method simplifies to maximizing dataset visitation density.The demonstrative-dataset assumption links high-density modes with ideal behavior and motivates choosing the highest-count action chunk.
  • Coupled Pseudo-Count Estimation for VLAs: TACO estimates pseudo-counts with a lightweight CFN head over VLA internal representations, leveraging pretrained computation instead of a separate encoder.The coupled estimator is designed to measure distributional shift with minimal computational overhead.
  • Coupled Pseudo-Count Estimation for VLAs: For flow- and diffusion-based VLAs, high-fidelity feature search identifies an in-distribution representation by querying multiple noise levels and choosing the prediction closest to the clean action.This addresses the mismatch caused by models being trained on noised actions rather than clean dataset actions.
  • Test-Time Scaling as Anti-Exploration: TACO constrains test-time VLA outputs to downstream-task success modes by selecting the most in-support action rather than sampling an unstable multimodal policy.This realizes the anti-exploration objective of avoiding redundant or out-of-support action modes without changing the VLA weights during inference.

5 Experiments

Experiments across simulation and real-world settings test TACO’s performance, efficiency, generality, and design choices. TACO achieves the highest reported average success rates, improves real-world performance, and remains efficient through KV-cache optimization.

  • 5.1.2 Results: TACO achieves the highest average success rate across the evaluated simulation benchmarks, improving performance by 9.1%, 7.5%, and 4.7% on three benchmarks.On Libero-long, TACO further improves π0.5 from 94.8% by 1.8%; on Moka Pots on Stove, success rises from 68% to 86%.
  • 5.1.2 Results: TACO generalizes across flow-matching and autoregressive VLAs, improving OpenVLA’s average success rate by 6.0% despite adding only a lightweight MLP pseudo-counter.Robomonkey achieves a 6.7% improvement for comparison, while Table 4 reports consistent TACO improvements for π0.5 and OpenVLA.
  • 5.2.1 Setup: Selecting the action with the largest pseudo-count almost always chooses the action closest to ground truth while filtering overly aggressive actions.The analysis uses 100 random noise inputs per observation and relates CFN output norms to action L2 distance.
  • 5.2.2 Results: In real-world experiments, TACO improves average success by 16%, including gains of 25% on Paper and Pen and 15% on Laptop.The evaluation uses five tasks on a RealMan75 dual-arm robot, covering human-robot interaction, dual-arm coordination, and long-horizon execution.
  • 5.3 Ablation Studies and Analyses: Ablations show that directly fitting action errors, omitting feature scaling, or replacing internal features with separate encoders can reduce performance and efficiency.Direct error fitting increases learning difficulty and can overfit, while jointly trained separate encoders produce similar features that hinder pseudo-counting.

6 Conclusion

TACO is a test-time-scaling framework that uses a lightweight pseudo-count estimator to verify action chunks and select among sampled candidates. It improves downstream-task stability by constraining inference toward successful action modes without modifying the VLA during inference.

  • 6 Conclusion: TACO uses a lightweight pseudo-count estimator as a high-fidelity verifier for action chunks during inference.The framework is designed to be compatible with various VLA models.
  • 6 Conclusion: The inference procedure samples candidate actions, computes pseudo-counts with the CFN verifier, and selects the action chunk with the highest score.Algorithm 1 also describes reuse of the KV cache during candidate generation.

B Algorithmic Description of TACO during Training

The training procedure couples VLA representations with a lightweight CFN pseudo-count estimator. It searches noise-conditioned representations for those best aligned with clean actions, then trains the estimator to reflect dataset support for anti-exploratory inference.

  • B Algorithmic Description of TACO during Training: The coupled estimator is trained from high-fidelity VLA features selected by repeatedly querying the policy under different noise conditions.For each representation, the procedure selects the candidate that best aligns with the clean action.
  • B Algorithmic Description of TACO during Training: The CFN pseudo-count estimator measures how closely a candidate action lies to the support of the SFT dataset.This score enables anti-exploration during inference.
  • B Algorithmic Description of TACO during Training: Flow matching generates actions by integrating a time-dependent velocity field from a simple prior to the target distribution.Unlike diffusion, the transformation is governed by an ordinary differential equation.
  • B Algorithmic Description of TACO during Training: Algorithm 2 takes an SFT dataset, a VLA policy, a CFN head, a noise schedule, and optimization settings, and outputs a trained CFN with high-fidelity features.The listed inputs include the number of queries, training steps, and batch size.

D Training Implementation Details

The implementation extracts internal VLA representations and constructs task-specific base policies from scripted or benchmark datasets. Training configurations vary by benchmark, while CFN training uses dedicated optimization settings.

  • D Training Implementation Details: The pipeline modifies the VLA action-output function to return both actions and internal representations.The extracted representation comes from the first action token in the final hidden layer.
  • D Training Implementation Details: Robotwin1.0 uses 50 scripted-policy trajectories per task, then fine-tunes the π0 base model for 30k steps with batch size 48.Representations are obtained after injecting 10% action noise and applying one denoising step.
  • D Training Implementation Details: Robotwin2.0 fine-tunes π0 for 10k steps with batch size 32 and uses 100% relative noise when extracting representations.Other representation-extraction settings remain unchanged from the described pipeline.
  • D Training Implementation Details: Libero uses two official or publicly available fine-tuned base policies, while Simpler fine-tunes π0 on Bridge V2 using eight NVIDIA H100 GPUs and batch size 256.These choices provide benchmark-specific base policies for the evaluation pipeline.
  • D Training Implementation Details: CFN training uses Adam and a OneCycleLR scheduler, with the detailed settings summarized in Table 6.The table is identified as the source of CFN training parameters.

E.1 Network Architecture of CFN

CFN is a lightweight MLP that transforms frozen VLA features into pseudo-count estimates. Its training objective and randomized prior make feature norms reflect dataset visitation frequency, with unseen regions initialized conservatively.

  • E.1 Network Architecture of CFN: CFN is a lightweight MLP comprising feature scaling, input projection, MLP blocks, and output projection.The architecture is designed for efficient feature processing and target prediction.
  • E.1 Network Architecture of CFN: Input features are multiplied by 10 before projection to enlarge inter-feature distances and improve feature discrimination.The scaling operation is intended to reduce interference between features.
  • E.1 Network Architecture of CFN: Each MLP block uses two fully connected layers, GELU, dropout, residual connections, and LayerNorm.The first fully connected layer expands to four times the hidden dimension before projecting back.
  • E.1 Network Architecture of CFN: The CFN is trained on high-fidelity features extracted from the frozen VLA backbone rather than raw states.The objective pairs each feature with a randomly sampled Rademacher vector and minimizes reconstruction distance.
  • E.1 Network Architecture of CFN: The learned output norm encodes inverse visitation frequency, so frequently observed features receive larger pseudo-counts than novel or out-of-support features.A frozen random prior initializes unseen representation regions with pseudo-count 1 and diminishes as learning proceeds.

F Additional Simulation Experiment Details

Additional simulation experiments compare pseudo-counter representations and alternatives, showing that High-Fidelity Feature Search best preserves action information during denoising. CFN also achieves consistently better average performance than the RND replacement on evaluated tasks.

  • Further Experimental Exploration on Representations: The representation study evaluates CNN, MLP, internal, and combined internal-plus-search methods across 50 initial noise configurations in Block Handover.Figure 8 tracks 1/Count throughout denoising for these representation settings.
  • Further Experimental Exploration on Representations: CNN and MLP encoders progressively lose discrimination among noise samples as denoising steps increase, eventually becoming overly tolerant to actions.Their values converge toward a single point, and the counter can reach its maximum before denoising finishes.
  • Further Experimental Exploration on Representations: Internal representations retain discrimination after complete denoising but provide limited separation between better and worse noise samples.The authors attribute this limitation to noise injection partially corrupting ground-truth action information during representation extraction.
  • Further Experimental Exploration on Representations: High-Fidelity Feature Search enables the counter to distinguish better from worse noise samples after denoising completes.Combined with internal representations, it preserves ground-truth action information by exploring the predicted action space.
  • More experiments on the choice of pseudo-counter: CFN yields consistently better average performance than RND when RND replaces the pseudo-counter on a subset of tasks.RND uses predictor error as novelty, with higher novelty mapped to a smaller pseudo-count.

G Additional Real-World Experiment Details

The real-world experiments use a dual-arm robotic platform with multiple cameras and evaluate three tasks through a defined execution pipeline. Objects are randomly placed within a small workspace, with task prompts listed in Table 8.

  • Real-World Platform and Task Setup: The real-world platform combines a Realman RM75-6F dual-arm robot, two Robotiq grippers, and three Intel RealSense cameras.One L515 provides the main view, while two D405 cameras are wrist-mounted; deployment uses an RTX 4090 workstation.
  • Real-World Platform and Task Setup: Testing randomly places each object within an approximately 3 cm × 4 cm area.The overall task execution process is illustrated in Figure 9.
  • Real-World Platform and Task Setup: Table 8 lists the three real-world tasks and their corresponding prompts.These prompts define the task instructions used during evaluation.
Loading 2512.02834v1…