PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models

Xinyi Xie, Zican Hu, Zhanyu Liu, Yicheng Dong, Wenhao Wu, Zhenhong Sun, Haoran Li, Chunlin Chen, Zhi Wang, Pichao Wang

arXiv:2607.03751v1cs.RO

TL;DR

VLA generalization is fragile, and improving policies through backbone updates can compromise their generalist capacity. SVA instead distills simulation-based search into a consequence evaluator for frozen VLAs, yielding consistent benchmark gains and enabling a 9B VLA to outperform a 27B VLA by 7 points at 27% lower inference latency.

  • Problem

    VLA generalization remains fragile, while existing post-training updates expensive backbones rather than addressing the missing evaluation of candidate actions’ long-term consequences.

  • Method

    SVA uses Monte-Carlo tree search to collect return-labeled trajectories, distills them into a lightweight Q-value model, and reranks frozen VLA action candidates at test time.

  • Results

    7 points: a 9B VLA with SVA outperforms a 27B VLA at 27% lower inference latency, while gains remain consistent across embodied benchmarks and backbones.

  • Takeaways & Limitations

    Scaling test-time action evaluation can be a more cost-effective alternative to scaling VLA model size within the evaluated embodied benchmarks.

  • Takeaways & Limitations

    The method relies on resettable simulators and evaluates only in simulation, leaving physical-robot testing as a key next step.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models acquire broad embodied capabilities through large-scale pretraining, yet their generalization remains far more fragile than that of LLMs and VLMs. The prevailing remedy, post-training via supervised fine-tuning or reinforcement learning, improves task-specific performance but narrows the generalist capability that makes pretraining valuable. We identify a key bottleneck: VLA failures stem not only from action generation but also from action evaluation. A diagnostic pass@k study confirms that frozen VLAs already contain competent behaviors in their output distribution, with overall success rates rising from 33% at pass@1 to 92% at pass@32. Inspired by this, we propose SVA (Search, Value, and Act), a simple framework that equips frozen VLA policies with long-term consequence awareness. SVA first uses Monte-Carlo tree search in simulation to fully explore the VLA's output distribution and collect diverse trajectories annotated with empirical returns; this knowledge is then distilled into a lightweight Q-value model that predicts the expected consequence of candidate actions; at deployment, the frozen VLA proposes multiple candidates and the evaluator selects the one with the highest uncertainty-regularized Q-value, requiring no simulator access. By decoupling action proposal from consequence evaluation, SVA preserves the generalization capacity of the VLA backbone while substantially improving task success rates. Experiments across embodied benchmarks show that SVA consistently improves generalization on unseen tasks and exhibits strong test-time scaling behavior. Strikingly, SVA enables a 9B VLA to outperform a 27B VLA by 7 points at 27% lower inference latency, suggesting that scaling test-time evaluation is more cost-effective than scaling model size.

1 Introduction

VLAs retain broad embodied capabilities but remain fragile because single-shot action generation lacks consequence evaluation, while post-training can narrow generalization. SVA addresses this bottleneck by distilling simulation-based tree search into a lightweight evaluator for frozen VLAs, improving generalization and test-time efficiency.

  • Motivation: VLAs acquire broad embodied capabilities through large-scale pretraining but remain more fragile than LLMs/VLMs on modest distribution shifts.This fragility makes preserving pretrained generalist capacity especially important.
  • Motivation: Post-training improves task performance but can narrow the generalist capacity acquired during pretraining.The paper frames strengthening VLA performance without sacrificing generalization as a central challenge.
  • Problem: VLA failures reflect an evaluation bottleneck: high-quality actions may be present in the output distribution, but single-shot generation cannot reliably select them or anticipate their consequences.VLAs imitate locally plausible actions without signals about whether they lead to success, collision, or irrecoverable states.
  • Method: SVA distills Monte-Carlo tree search in simulation into a value model that evaluates candidate actions without simulator access at deployment.Search explores the VLA output distribution and annotates diverse trajectories with empirical returns, while the frozen backbone remains unchanged.
  • Results: 7 points and 27% lower inference latency are achieved by a 9B VLA with SVA versus a 27B VLA.The paper presents SVA as a practical alternative to costly policy fine-tuning, with consistent gains across manipulation benchmarks and VLA backbones.

2 Related Work

Related work spans generalist VLA models and action-evaluation methods that guide frozen policies without modifying their backbones. VLAs remain fragile outside training distributions, motivating verifier- and value-based approaches adapted from LLM reasoning.

  • VLA Models: Generalist VLA families include RT-1/RT-2, OpenVLA, π-series, GR00T, and Octo, extending LLM/VLM advances to robotic control.These models represent a growing body of work adapting language and vision-language modeling to embodied domains.
  • VLA Models: VLAs remain markedly more fragile than LLMs and VLMs, often failing on tasks only modestly outside their training distributions.This fragility motivates substantial post-training research.
  • Action Evaluation for Robot Policies: Action-evaluation methods preserve the policy backbone while learning verifiers or value functions to guide candidate generation.This direction draws on outcome- and process-reward models combined with search in LLM reasoning and includes SayCan and V-GPS for robot policies.

3 Diagnosing the VLA Bottleneck: Generation or Evaluation?

The diagnosis shows that frozen VLAs often contain successful behaviors but struggle to identify them before execution, making action evaluation a central bottleneck. Evaluation is most valuable for intermediate-difficulty tasks, while naive pass@k sampling suffers diminishing returns.

  • Diagnostic protocol: The study probes latent policy capability by measuring how often at least one of k independent rollouts succeeds, separating action generation from evaluation.OpenVLA is evaluated on SIMPLER and LIBERO, while π0.5 is evaluated on ROBOTWIN, using 50 independent rollouts per task.
  • Observation 1: 33% at pass@1 rises to 92% at pass@32, a 58-point absolute gain, showing that successful behaviors already exist in frozen VLA output distributions.The same increasing trend holds across every task, despite single-shot failures from stochastic sampling or compounding execution errors.
  • Observation 1: The deployment bottleneck is identifying competent actions before execution, reframing VLA failure as deficient action evaluation rather than solely deficient action generation.Frozen VLAs assign non-trivial probability mass to successful behaviors within their output distributions.
  • Observation 2: Intermediate-difficulty tasks offer the greatest evaluation opportunity: Pick up Book/Soup and Sauce in Basket reach 0.99/0.96 by pass@4, whereas Stack Cubes reaches only 0.50 at pass@32.The cited easy-task values start from 0.84/0.64, while Stack Cubes starts from 0.02.
  • Observation 3: ∆1→2 =0.13, ∆2→4 =0.13, ∆4→8 =0.12, ∆8→16 =0.10, and ∆16→32 =0.08, revealing diminishing returns from naive independent sampling.Although pass@k improves monotonically, the informational gain per additional sample shrinks rapidly as the sampling budget grows.

4 Look Before You Leap: Addressing the Evaluation Bottleneck with SVA

SVA addresses the evaluation bottleneck by using MCTS to explore a frozen VLA’s policy distribution, distilling search outcomes into a lightweight consequence evaluator, and selecting among sampled actions at deployment. This separates action proposal from consequence evaluation while supporting uncertainty-aware and test-time-scalable inference without simulator access at deployment.

  • Inference strategy: SVA treats the frozen VLA as a fixed proposal distribution and selects the sampled action with the highest learned expected-consequence value.At each step, N candidate actions are drawn from πθ(·|st; l) and ranked by ˆQϕ(st, a(i); l).
  • MCTS search: MCTS explores the VLA policy distribution through selection, expansion, simulation rollout, and backup, using returns to bias search toward higher-return branches.At expanded leaves, πθ samples N actions, simulator rollouts run for up to D steps or until termination, and rollout returns update edge values as running means.
  • MCTS search: The search collects diverse successful and failed trajectories that provide contrastive signals for learning relative action quality.These trajectories are gathered from multiple search episodes under a finite budget.
  • Q-value distillation: SVA distills MCTS knowledge into a lightweight Q-model built on a VLM backbone with LoRA adapters and an ensemble of value heads.The ensemble mean predicts Q-values, its standard deviation estimates uncertainty, and only the value heads and LoRA adapters are fine-tuned.
  • Deployment and scaling: At test time, the frozen VLA proposes candidates and the evaluator selects the highest uncertainty-regularized Q-value, with N controlling test-time compute.A lightweight Q-model such as 0.8B adds minimal scoring overhead relative to a VLA backbone such as 7B, while increasing N improves the probability of selecting a high-quality action.

5 Experiments

Across embodied reasoning and manipulation benchmarks, SVA improves frozen VLA policies across backbones and task types. Ablations, test-time scaling, latency measurements, and a qualitative case study attribute these gains to action evaluation over diverse candidates.

  • Experimental setup: Experiments span EmbodiedBench reasoning tasks, SimplerEnv WidowX manipulation, and RoboTwin 2.0, with success rate reported on 3:2 training/evaluation splits.Baselines include GPT-4o, Qwen3.5-4B/9B/27B, Gemma-4-E4B-it, OpenVLA, π0/π0.5, and π0/π0.5+RoboMonkey.
  • EmbodiedBench: +15.4 average success-rate points on EB-Habitat and +13.2 points on EB-Navigation show consistent SVA gains across five backbones.The largest improvements occur in categories requiring long-horizon planning or visual grounding.
  • Manipulation: 50.7% average success rate for π0+SVA on SimplerEnv surpasses π0+RoboMonkey, including gains of +26.4 on Stack Cubes and +13.9 on Eggplant in Basket.The improvements concentrate on contact-rich, multi-step tasks and require no curated preference annotations.
  • Ablations: 56.11 versus 50.83 on EB-Navigation after removing MCTS, and 56.11→43.33 after removing the Q-model, show that both search and value evaluation contribute.The Q-model ablation causes the largest decline, while removing multi-candidate evaluation replaces selection with pass-through behavior.
  • Test-time scaling: ≤1.33s evaluation latency versus up to 11.2s generation latency at N=32 makes SVA test-time scaling practical despite candidate-set growth.Doubling the candidate set increases generation latency by 10.63%–36.58%, averaging 26%.
  • Test-time scaling: 53.6% success at 7.43s for Qwen3.5-9B+SVA with best-of-8 exceeds Qwen3.5-27B’s 46.7% at 10.2s by 7 points at 27% lower latency.The SVA result uses a 3× smaller backbone, illustrating the cost advantage of scaling action evaluation rather than model size.
  • Qualitative analysis: In a distractor instruction, SVA reaches the target table in 4 steps with no invalid actions by selecting a lower-prior plate-centric plan over a sofa-first alternative.The Q-network overrides the base policy’s distractor-driven preference through long-horizon consequence evaluation.

6 Discussion and Limitations · Appendix

The discussion reframes VLA failure as an action-evaluation bottleneck and argues that scaling test-time evaluation can complement policy and data scaling. The action-evaluation analysis shows that a candidate with a lower prior but higher Q-value can be selected correctly.

  • 6 Discussion and Limitations: The action evaluator can select the correct action when its Q-value is higher despite a lower prior.Table 2 summarizes the central action-evaluation relationship: lower prior but higher Q-value leads to correct selection.
  • 6 Discussion and Limitations: The paper reframes VLA failure as an evaluation bottleneck rather than purely a generation problem.This perspective treats evaluation as a first-class lever for improving frozen VLA policies.
  • 6 Discussion and Limitations: Across embodied benchmarks, the method delivers consistent gains over diverse VLA backbones.The discussion attributes these gains to scaling test-time evaluation rather than relying only on larger models.
  • 6 Discussion and Limitations: The authors position evaluation as orthogonal to data scaling and policy fine-tuning.They hope this framing encourages the community to treat evaluation as a first-class lever for VLA improvement.

A Limitations and Future Work

SVA’s staged design, simulator dependence, and simulation-only evaluation leave clear limitations despite consistent benchmark gains. Future work centers on integrating search with learning, replacing simulator rewards when needed, and validating SVA on physical robots.

  • Future Work: Together, these limitations define an agenda of tightening the search–learning loop, relaxing simulator assumptions, and bridging the sim-to-real gap.The authors position SVA as an initial foundation for scalable, deployable, and self-improving embodied systems.
  • Staged Search and Learning: SVA’s decoupled tree search and value learning prevents search from using the learned evaluator and limits policy improvement to test-time re-ranking.An online loop could let an updated Q-model guide MCTS while on-policy rollouts continually refine it.
  • Simulator Dependence: The Search stage requires a resettable simulator and task-success signal, limiting direct use where high-fidelity simulation or reward functions are unavailable.Learned world models or sparse human/autolabeled outcomes could provide alternative evaluation signals.
  • Sim-to-Real Validation: SVA has been evaluated entirely in simulation, leaving the learned Q-model’s calibration on physical robots untested.Sim-to-real co-training or lightweight online residual calibration is identified as the most pressing next step.

B Benchmarks … D.1 Q-Model Architecture

SVA is evaluated on embodied navigation and manipulation benchmarks, with simulation choices and real-robot-trained frozen backbones intended to support physical-deployment relevance. Its Q-model converts observations and candidate actions into action values, using an uncertainty-aware ensemble trained with lightweight adaptation.

  • B Benchmarks: SVA evaluates EmbodiedBench as the main benchmark and uses SimplerEnv and RoboTwin 2.0 to test value-guided reranking for continuous-control manipulation.EmbodiedBench covers household rearrangement and navigation, while the manipulation benchmarks include WidowX and bimanual tasks.
  • C On the Real-Robot Relevance of Our Simulation Study: SimplerEnv was validated as a real-world proxy because policy rankings matched corresponding physical Google Robot and WidowX setups across RT-1, RT-1-X, Octo, and OpenVLA.RoboTwin 2.0 is also described as a benchmark designed for real-robot predictivity.
  • C On the Real-Robot Relevance of Our Simulation Study: SVA preserves real-data-grounded action distributions by freezing π0 and π0.5, which are generalist VLAs trained on large-scale real-robot demonstrations.The reranked distribution is therefore the one produced by the deployed real-robot backbones.
  • C On the Real-Robot Relevance of Our Simulation Study: At deployment, SVA uses only the frozen VLA and lightweight Q-model, while simulation supplies MCTS task-success labels solely during Q-model training.Both deployed components consume RGB, proprioception, and language inputs and emit actions in the robot’s native action space.
  • C On the Real-Robot Relevance of Our Simulation Study: Transferring SVA to a real robot requires no algorithmic change, with either zero-shot use of the simulator-trained Q-model or light fine-tuning on real-robot rollouts.The proposed fine-tuning targets use the same MCTS-style return targets.
  • D.1 Q-Model Architecture: The Q-model starts from Qwen/Qwen3.5-0.8B and adds a special <|VALUE|> token whose hidden state represents each candidate action sequence.Inputs contain the current observation, language instruction, recent interaction history, and candidate action sequence.
  • D.1 Q-Model Architecture: For EmbodiedBench, discrete skills are serialized directly into prompts, whereas SimplerEnv and RoboTwin continuous action vectors are tokenized with FAST.This provides a common language-model input representation for both discrete skills and continuous robot-control chunks.
  • D Experimental Details of SVA: Five bootstrapped Q-heads produce the final ensemble mean and uncertainty estimate, with 2,626,565 trainable head parameters.The backbone hidden dimension is 1024, and the backbone is fine-tuned jointly with the heads using LoRA.

D.2 Search-Stage Data Collection … E Baselines

The paper collects simulator-backed MCTS supervision during training, regresses normalized returns into domain-specific Q-models, and uses them to rerank frozen VLA action candidates at deployment. Baselines provide verifier- and preference-based candidate reranking comparisons for RoboTwin 2.0 and SimplerEnv.

  • D.2 Search-Stage Data Collection: MCTS collects simulator-backed supervision from visited tree edges during training, while act-time evaluation uses no simulator.Each sample stores the instruction, observation, interaction history, candidate action sequence, MCTS-backed return, visit count, and metadata.
  • D.2 Search-Stage Data Collection: For manipulation benchmarks, open-source policy checkpoints serve as action proposers, including π0 initialized from petkopetkov/INTACT in SimplerEnv.
  • D.3 Value-Stage Training: The Q-model is trained by supervised regression to MCTS-backed returns, with targets normalized by each training set’s mean and standard deviation.
  • D.3 Value-Stage Training: Separate Q-models are trained for EB-Habitat, EB-Navigation, SimplerEnv, and RoboTwin 2.0 using 5 epochs, AdamW, bfloat16, batch size 8, and learning rate 1 × 10−4.Training also uses gradient clipping of 1.0 and NVIDIA RTX PRO 6000 GPUs for all training and evaluation experiments.
  • D.4 Act-Stage Evaluation: At each decision step, the base policy samples multiple action sequences or continuous chunks, the Q-model scores them in a batch, and the highest-reranking candidate is executed.The agent replans until task success, invalid execution, or the episode budget is reached.
  • D.4 Act-Stage Evaluation: EmbodiedBench uses sampling temperature 0.7 during act-stage evaluation.
  • E Baselines: RoboMonkey-style reranking is evaluated as a baseline using the same candidate-selection formulation with benchmark-specific verifiers.
  • E Baselines: For RoboTwin 2.0, a Qwen3.5-0.8B scorer learns pairwise preferences from MCTS over candidate chunks, ranks 16 sampled chunks, and executes the top-ranked chunk.For SimplerEnv, the released robomonkey-vla/monkey-verifier-7b remains frozen and scores 7-DoF WidowX chunks by mean verifier reward; 5 π0 proposals are combined with Gaussian samples and grasp state majority voting.

F Detailed Pass@k Results · G Detailed VLA Manipulation Results · H Detailed Results on Ablation Study

Detailed results show that frozen VLAs often contain successful candidates beyond their top-ranked action, while SVA improves manipulation performance and its three-stage design is supported by ablations on EB-Navigation.

  • F Detailed Pass@k Results: Pass@k results show a consistent gap between Pass@1 and larger k across manipulation benchmarks.The findings indicate that base VLAs frequently sample successful or near-successful candidates.
  • F Detailed Pass@k Results: Base VLA likelihood ranking does not always select the successful or near-successful candidates available in its output distribution.This explains the observed difference between top-1 selection and sampling multiple candidates.
  • F Detailed Pass@k Results: Task-level Pass@k results are reported across manipulation benchmarks using popular VLA models.Table 6 provides the benchmark-level comparison underlying the Pass@k analysis.
  • G Detailed VLA Manipulation Results: On SimplerEnv, SVA improves over the π0 baseline on temporally extended manipulation tasks, including stacking cubes and placing the eggplant into the basket.Table 7 reports success rates (%) for the WidowX platform.
  • G Detailed VLA Manipulation Results: On RoboTwin 2.0, SVA improves the average success rate and outperforms the base π0.5 policy on most tasks.Table 8 reports success rates (%) for RoboTwin 2.0.
  • G Detailed VLA Manipulation Results: Tables 7 and 8 provide task-level results for SVA on SimplerEnv and RoboTwin 2.0.The results cover manipulation benchmarks with the WidowX platform on SimplerEnv and success rates on RoboTwin 2.0.
  • H Detailed Results on Ablation Study: Complete ablation results on EB-Navigation further support the necessity of the three-stage SVA design.Table 9 contains the detailed EB-Navigation ablation results.

I More Results on Case Study

Qualitative case studies show that SVA improves instruction following by selecting action sequences according to predicted long-horizon consequences rather than proposal likelihood alone. It avoids distractor-driven and spatially incorrect plans, completing tasks the base policy fails.

  • Case 1: distractor-aware instruction following: SVA ignores an irrelevant apple-on-sofa clause, follows a plate-centric plan, and completes the distractor-aware instruction.The base policy is distracted and fails after repeated invalid actions.
  • Case 2: spatial-relation grounding: SVA grounds the wrench’s spatial relation by navigating to the sink, picking up the wrench, moving right, and placing it there.The base policy repeatedly navigates to the sofa and attempts invalid pickups without task progress.
  • Cross-case analysis: Across both cases, SVA ranks candidates by predicted long-horizon return, rejecting distractor-driven or spatially incorrect plans.This ranking replaces reliance on proposal likelihood alone and better satisfies task constraints.
  • Case 1: distractor-aware instruction following: SVA completes the plate-moving task in 4 steps, whereas the base policy fails after repeated invalid recovery actions.The base policy follows the salient but irrelevant sofa clause.
Loading 2607.03751v1…