PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models

Jacky Kwok, Christopher Agia, Rohan Sinha, Matt Foutter, Shulu Li, Ion Stoica, Azalia Mirhoseini, Marco Pavone

arXiv:2506.17811v2cs.ROcs.AIcs.SYeess.SY

TL;DR

VLA robustness and generalization remain difficult in unstructured environments, while deployment-time compute has received less attention than training and post-training. RoboMonkey scales action sampling and verification with a synthetic-data-trained verifier; the authors report improved performance across in-distribution and out-of-distribution evaluations.

  • Problem

    VLA deployment can involve failures such as imprecise grasping, task progression failure, and collisions, while scaling deployment-time compute has received less attention than pre-training and post-training.

    Source: §1 · p. 1

  • Method

    RoboMonkey samples VLA actions, applies Gaussian perturbation and majority voting to form proposals, and uses a VLM-based verifier trained with synthetic action preferences to select an action.

    Source: §1 · p. 2

  • Results

    Pairing existing VLAs with RoboMonkey improves performance across in-distribution and out-of-distribution tasks, including 25% absolute improvement on real-world out-of-distribution tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that generate-and-verify test-time scaling offers a practical path toward building general-purpose robotics foundation models.

    Source: §6 · p. 9

    Multiple VLA samples and a separate VLM verifier add deployment compute, which may make the framework less suitable for high-frequency control.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in visuomotor control, yet ensuring their robustness in unstructured real-world environments remains a persistent challenge. In this paper, we investigate test-time scaling through the lens of sampling and verification as means to enhance the robustness and generalization of VLAs. We first demonstrate that the relationship between action error and the number of generated samples follows an exponentiated power law across a range of VLAs, indicating the existence of inference-time scaling laws. Building on these insights, we introduce RoboMonkey, a test-time scaling framework for VLAs. At deployment, RoboMonkey samples a small set of actions from a VLA, applies Gaussian perturbation and majority voting to construct an action proposal distribution, and then uses a Vision Language Model (VLM)-based verifier to select the optimal action. We propose a synthetic data generation pipeline for training such VLM-based action verifiers, and demonstrate that scaling the synthetic dataset consistently improves verification and downstream accuracy. Through extensive simulated and hardware experiments, we show that pairing existing VLAs with RoboMonkey yields significant performance gains, achieving a 25% absolute improvement on out-of-distribution tasks and 9% on in-distribution tasks. Additionally, when adapting to new robot setups, we show that fine-tuning both VLAs and action verifiers yields a 7% performance increase compared to fine-tuning VLAs alone.

1 Introduction

RoboMonkey addresses the deployment-time robustness and generalization challenges of VLAs by scaling action sampling and verification. The paper reports gains across task settings and proposes a synthetic-data-trained verifier to select actions.

1 Introduction

  • The work responds to VLA deployment failures such as imprecise grasping, task progression failure, and collisions, which challenge robustness in unstructured environments.It asks whether repeatedly sampling and verifying actions can improve precision and robustness.Source: §1 · pp. 1–2
  • Across a range of VLAs, action error follows an exponentiated power law as generated samples increase, and Gaussian perturbation is the most cost-effective sampling approach.This motivates the paper’s investigation of inference-time scaling for robotics, where deployment compute had received less attention than pre-training and post-training.Source: §1 · pp. 1–2
  • RoboMonkey samples actions from a VLA, applies Gaussian perturbation and majority voting to form an action proposal distribution, then uses a fine-tuned VLM verifier to select an action.The verifier is trained using a pipeline that automatically curates synthetic action comparisons; increasing the synthetic preference dataset size consistently improves performance.Source: §1 · p. 2
  • The paper presents efficient action-sampling methods and a scalable pipeline for generating synthetic action preferences to train VLM-based action verifiers.Source: §1 · p. 2
  • 25% absolute improvement on real-world out-of-distribution tasks and 9% on in-distribution SIMPLER environments are reported for the test-time scaling framework.These results are presented as gains from pairing existing VLAs with RoboMonkey.Source: §1 · p. 2
  • 7% performance increase on LIBERO-Long is reported when fine-tuning both VLAs and action verifiers rather than VLAs alone.Source: §1 · p. 2

2 Preliminaries

The preliminaries model robot control as an MDP with seven-dimensional state and action vectors. They assume a language-conditioned policy and expert demonstrations for imitation learning.

2 Preliminaries

  • The MDP state and action spaces are seven-dimensional vectors representing end-effector pose and gripper state.The six pose dimensions cover translation and rotation; the final dimension indicates whether the gripper is open.Source: §2 · p. 3
  • The framework assumes a language-conditioned robot policy that can sample multiple actions for a state and task instruction.Source: §2 · p. 3
  • The framework assumes expert demonstrations, with an auxiliary dataset of state-action-instruction tuples used alongside imitation learning.The policy is fine-tuned to imitate expert actions using the standard imitation-learning objective.Source: §2 · p. 3

3 Inference-Time Scaling Law

The section evaluates how action error changes with sample count across sampling methods and generalist robot policies. Under oracle selection, errors decrease with more samples, and Gaussian perturbation exhibits power-law scaling across models.

3 Inference-Time Scaling Law

  • The evaluation draws 1,000 state-action-instruction tuples and generates 10,000 actions per tuple, measuring each sample’s Normalized RMSE against the ground-truth action.Source: §3 · p. 3
  • The three tested approaches are uniform random action-token sampling, repeated sampling from a robot policy, and Gaussian perturbation fitted from four policy samples.Source: §3 · p. 3
  • With an oracle verifier selecting the lowest-RMSE action, error consistently decreases as sample count rises across random, policy, and Gaussian-perturbation sampling.Policy sampling yields the lowest error, while Gaussian perturbation performs nearly identically at lower computational cost; more than 100 uniformly random samples also outperform greedy OpenVLA decoding.Source: §3 · p. 3
  • 59.3% RMSE decrease is reported for OpenVLA when sampling 10,000 actions; Gaussian perturbation shows an exponentiated power-law relationship across CogACT, Octo, OpenVLA, and SpatialVLA.Source: §3 · p. 3

4 Proposed Approach: RoboMonkey

RoboMonkey uses a learned action verifier to scale test-time compute through a generate-then-verify pipeline. Its synthetic preference data trains the verifier, while deployment samples and refines candidate actions before selecting one for execution.

4.2 Synthetic Data Generation Pipeline

  • Synthetic comparisons are built by sampling and clustering policy actions, then labeling pairwise preferences by each action’s RMSE against the ground truth.The comparisons are curated from demonstration data without human annotation.Source: §4.2 · p. 4

4.3 Reward Modeling

  • The reward model is trained with a modified Bradley–Terry loss that incorporates preference levels through a margin component.The verifier uses a LLaVA-7B backbone with a reward head.Source: §4.3 · p. 5

4.4 Action Sampling and Verification

  • At deployment, RoboMonkey samples VLA actions, majority-votes the gripper state, fits a Gaussian proposal to motion components, and samples refined actions for verifier ranking.The highest-scoring refined action is selected for execution.Source: §4.4 · p. 5

5 Experiments

RoboMonkey improves success rates on in-distribution SIMPLER tasks and real-world out-of-distribution tasks, and the experiments also examine deployment cost, synthetic-data scaling, and adaptation to LIBERO-Long.

5.1 Implementation Details

  • Experiments evaluate RoboMonkey across simulated and real-world environments, with two embodiments, four real-robot tasks, and 14 simulation tasks.The Bridge V2 training set contains over 40,000 WidowX trajectories from 24 environments.Source: §5 · p. 5, §5.1 · p. 6

5 Experiments

  • 9% increase in average success rate on in-distribution SIMPLER environments and 25% improvement in real-world out-of-distribution experiments using the WidowX robot.Figure 3 summarizes the gains from scaling test-time compute.Source: Figure 3 · p. 6

5.2 Can RoboMonkey improve the precision of VLAs on in-distribution tasks?

  • 47.5% average success rate for RoboMonkey, outperforming OpenVLA by 9% on average in SIMPLER in-distribution tasks.RoboMonkey’s advantage over OpenVLA reaches 19% on eggplant placement and 10% on block stacking.Source: §5.2 · p. 6

5.3 Can RoboMonkey improve the robustness of VLAs on out-of-distribution tasks?

  • 60% average success rate for RoboMonkey, versus 35% for OpenVLA and 30% for V-GPS across real-world OOD tasks.Evaluations cover four task suites with 10 trials each; RoboMonkey also outperforms both baselines across the diverse tasks.Source: §5.3 · p. 7

5.4 How does RoboMonkey enable practical deployment for test-time scaling?

  • RoboMonkey samples and verifies 16 candidate actions in approximately 650 ms, with 41.3% lower latency than naive policy sampling.The real-world system runs at approximately 1.5 Hz on one NVIDIA H100 GPU, using 28 GB of GPU memory.Source: §5.1 · p. 6, §5.4 · p. 7, Figure 5 · p. 7

5.5 How does scaling the synthetic training dataset impact downstream success rate?

  • SIMPLER average success rate rises from 37.5% to 46.3% as the synthetic preference dataset scales, with “Stacking Cube” rates increasing from 27% to 37% to 42%.Task performance grows nearly log-linearly with dataset size.Source: §5.5 · p. 8, Figure 6 · p. 8

5.6 Can we effectively fine-tune the action verifier on new robot setup and task?

  • 6.7% improvement in average success rate on LIBERO-Long when fine-tuning both OpenVLA and the action verifier rather than OpenVLA alone.All methods were evaluated across 500 trials.Source: §5.6 · p. 8, Table 1 · p. 8

6 Discussion and Limitations

RoboMonkey improves VLA precision and robustness across task settings, but deployment compute, synthetic-data scale, and embodiment coverage constrain the current framework.

6 Discussion and Limitations

  • Sampling multiple VLA actions and using a separate VLM verifier adds deployment overhead and may make RoboMonkey less suitable for high-frequency control.The authors mitigate costs with a VLA serving engine and Gaussian perturbation, while identifying lower-latency verification and system optimization as future work.Source: §6 · p. 9
  • Experiments used at most 20 million synthetic action comparisons from Bridge V2 because of compute constraints and fine-tuning cost.The authors identify larger robotics datasets spanning embodiments, tasks, and environments as a future direction.Source: §6 · p. 9
  • Evaluation covered two robot arms, WidowX 250S and Franka, and the authors call for evaluation across more embodiments.Source: §6 · p. 9

7 Related Work

Related work addresses generalist-policy robustness through training, adaptation, planning, and test-time selection; RoboMonkey combines repeated sampling with a learned action verifier.

7 Related Work

  • Generalist robot policies such as π0, OpenVLA, PaLM-E, and RT-X [1] [2] [3] [4] [5] can still fail under distribution shift and compounding prediction errors.Source: §7 · p. 9
  • Prior VLM-based approaches generate high-level action plans [50] [51] [52], whereas RoboMonkey and concurrent works [53] [54] use foundation models to verify low-level policy actions.Source: §7 · p. 9
  • Unlike V-GPS [42], which uses an offline-RL value function to rerank actions, RoboMonkey uses Gaussian-perturbed candidates, majority voting, and a scalable verifier-training pipeline.The paper reports improved task performance and generalization over prior verifier-based approaches.Source: §7 · p. 9

A Evaluation Tasks

The evaluation appendix describes OOD task settings, verifier training, deployment sampling, and baseline configurations across real-world and simulated environments.

A Evaluation Tasks

  • Four WidowX-250 S OOD tasks probe unseen instructions, objects, shapes, distractors, and visual-language grounding amid real-world setup shifts.The tasks include cup stacking, hammer placement, pepper placement among distractors, and distinguishing a banana from a yellow basket; Figure 7 shows representative executions across real-world, SIMPLER, and LIBERO environments.Source: A Evaluation Tasks · p. 14, Figure 7 · p. 14

B.1 Training

  • The verifier uses a LLaVA-7B reward head, ViT-Large vision encoder, and 256-bin discrete action tokens, trained on 20 million Bridge V2 comparisons.Its prompt first requests an action for the instruction and then asks the model to evaluate that action.Source: §B.1 · p. 15

B.2 Deployment

  • For real-world deployment, five OpenVLA actions define a Gaussian proposal for translation and rotation, while majority voting sets the gripper state before 16 candidates are verified.Simulation varies initial and augmented sample counts and reports the best result per task.Source: §B.2 · p. 15

B.3 Baselines

  • The OpenVLA baseline uses sample counts of 10 or 50 and temperatures of 0, 0.1, or 1.0 in simulation, while real-world V-GPS uses 10 samples at temperature 1.0.The evaluation follows the V-GPS implementation and uses a VLA serving engine for batch inference.Source: §B.3 · p. 15

C Ablation Over Action Selection Methods and Number of Samples

RoboMonkey selects lower-error actions than the compared methods and remains scalable as test-time sampling increases, unlike the reported V-GPS behavior.

C Ablation Over Action Selection Methods and Number of Samples

D Ablation Over Generalist Robot Policies

RoboMonkey improves action error across three generalist policies. The ablations favor a small preference margin and preference-based learning for OOD generalization, while optimized batching improves serving throughput.

D Ablation Over Generalist Robot Policies

  • Across CogACT, Octo, and SpatialVLA, RoboMonkey achieves RMSEs of 0.133 (8% below single-attempt 0.145), 0.166 (15.3% below greedy 0.196), and 0.1298 (5.3% below baseline 0.137), respectively.Source: D Ablation Over Generalist Robot… · pp. 16–17

E Ablation on Margin for Reward Modeling

F Ablation Over Preference-Based Learning

  • At 64 candidate actions, preference learning achieves 6% lower action error than RMSE regression in OOD settings, while the approaches perform similarly in-distribution.The evaluation samples 1,000 held-out tuples with unseen instructions and environments.Source: Figure 10 · p. 17, F Ablation Over Preference-Based… · p. 18

G Latency and Throughput Analysis for Sampling and Verification

H Trade-off between Action Error and Computational Overhead

At equivalent computational budgets, Gaussian perturbation yields lower oracle action error than policy sampling, with the performance gap widening as budgets increase.

H Trade-off between Action Error and Computational Overhead

  • At equivalent computational budgets, Gaussian perturbation achieves lower oracle action error than policy sampling, and the gap widens as compute budgets increase.The comparison plots action error against latency.Source: H Trade-off between Action Error and… · p. 18, Figure 11 · p. 19
  • Gaussian perturbation is more practical for deployment because it scales with the verifier alone, whereas policy sampling incurs overhead from both the policy and verifier.The passage identifies this overhead difference as the reason for the growing gap at higher compute budgets.Source: H Trade-off between Action Error and… · pp. 18–19
Loading 2506.17811v2…