PaperlayerView exact revision on arXiv

Source-linked AI summary

RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo, Yuanliang Ju, Qiao Gu, Guillaume Sartoretti

arXiv:2607.26991v2cs.RO

TL;DR

Pretrained VLAs need adaptive ways to generate diverse actions for challenging and out-of-domain tasks without perturbing actions likely to succeed. RL2 uses VLA-latent offline RL for conditional compositional steering, achieving task-wise gains of up to +19.4% over the strongest baseline across simulation tasks and improving real-world deployment.

  • Problem

    The paper asks how and when to steer pretrained VLAs toward diverse action candidates for robustness in challenging and out-of-domain scenarios.

  • Method

    RL2 trains a lightweight offline RL policy on VLA latents and composes its steering velocity with the frozen VLA, activating steering when failure is predicted.

  • Results

    Up to +19.4% task-wise performance gains over the strongest baseline were reported across in-domain and out-of-domain simulation tasks, with real-world gains also demonstrated.

  • Takeaways & Limitations

    Adaptive test-time steering is presented as a practical and modular framework for in-domain and out-of-domain VLA deployment.

  • Takeaways & Limitations

    RL2 assumes a robust verifier can effectively select among its action candidates.

Abstract

from arXiv · show

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce $RL^2$, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight, $RL^2$ activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks, $RL^2$ improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing $RL^2$ as a practical and modular steering framework for VLA deployment.

I. INTRODUCTION · II. RELATED WORKS

VLAs generalize well on familiar manipulation tasks but degrade on unseen instructions and environments, motivating inference-time steering for challenging and out-of-domain settings. RL2 introduces adaptive compositional steering that combines VLA priors with offline-RL-induced action diversity, activating it primarily when failure is predicted.

  • I. INTRODUCTION: 38.2% →14.2% and 70.2% →36.0% show VLA success-rate deterioration on unseen instructions and environments, respectively.Inference-time steering targets OOD generalization without extensive data collection or expensive retraining.
  • II. RELATED WORKS: Discrete steering selects among sampled actions with external verifiers but is limited by sample diversity, whereas differentiable steering guides action generation.These are the two broad directions identified for improving pretrained VLAs at deployment.
  • I. INTRODUCTION: Diverse alternatives are most useful when default behavior is likely to fail, while unnecessary steering can harm actions that are already accurate.The paper motivates adaptive intervention through contrasting human behavior under uncertainty and confidence.
  • I. INTRODUCTION: RL2 addresses adaptive compositional steering by applying Reinforcement Learning on VLA Latents during inference.The framework is introduced to determine how and when pretrained VLAs should be steered toward diverse candidates.
  • I. INTRODUCTION: A lightweight offline-RL flow-matching policy uses expressive VLA action-expert latents, shares the VLA’s offline finetuning dataset, and adds minimal computational overhead.Its flow velocity is composed with the frozen VLA action expert’s flow velocity to steer generation.
  • I. INTRODUCTION: Under an oracle verifier, normalized action error versus generated-sample count follows an exponential power law, with distinct scaling behavior in success and failure states.The analysis finds compositional steering beneficial during failure but performance-degrading during successful states.
  • I. INTRODUCTION: +17.3% is the maximum success-rate improvement over the strongest baselines for OOD tasks across SIMPLER and PolaRiS.Evaluations cover diverse manipulation tasks under in-domain and OOD settings with different verifiers; ablations assess VLA latents and RL training.
  • I. INTRODUCTION: +17.5% average success-rate improvement over the strongest baseline is achieved on hardware with compositional steering on a PiperX manipulator.The paper also reports favorable scaling with sample quantity and action diversity, plus validation across diverse VLAs, verifiers, and manipulation benchmarks.

A. Imitation vs Reinforcement Learning for VLAs … III. PRELIMINARIES

The section contrasts imitation-trained VLAs with reinforcement learning and surveys inference-time steering, test-time scaling, and adaptive inference. It motivates latent RL composition as a lightweight way to preserve pretrained behavioral priors while expanding action diversity and adapting interventions to failure states.

  • A. Imitation vs Reinforcement Learning for VLAs: Behavior-cloned VLAs generalize impressively but remain constrained by demonstration behaviors, whereas RL can discover high-value behaviors beyond demonstrations.End-to-end RL methods such as RECAP and SimpleVLA-RL are effective, but updating foundation models is computationally expensive and optimization-sensitive.
  • B. Inference-Time Steering: Inference-time steering improves pretrained VLAs at deployment without updating the base model.The survey categorizes steering into discrete action selection, differentiable steering, and direct action refinement.
  • B. Inference-Time Steering: Discrete action selection preserves demonstration-derived behavioral priors but remains limited in adapting to desired behaviors outside that distribution.Methods use learned value functions, larger VLM verifiers, preference or contrastive learning, and world-model rollouts to select candidates.
  • B. Inference-Time Steering: The proposed approach combines latent RL composition with verifier-based selection, joining discrete selection’s robustness with differentiable steering’s exploration beyond the policy distribution.This positions latent RL composition as a distinct inference-time steering mechanism rather than full VLA retraining.
  • C. Test-Time Scaling: Test-time scaling allocates additional inference computation, but repeatedly sampling from the same VLA can produce diverse variants of the same failure mode.Prior methods apply repeated action sampling, verifier-based selection, or embodied chain-of-thought reasoning for planning.
  • C. Test-Time Scaling: The proposed test-time scaling characterizes distinct steering behaviors under success and failure states and diversifies candidate actions during failure, when it helps most.This makes latent RL steering conditional on the state’s predicted outcome rather than uniformly applied.
  • D. Adaptive Inference for VLAs: Adaptive inference methods dynamically adjust processing according to state difficulty, using signals such as routing tokens, action consistency, and uncertainty.Examples include deciding whether to reason or act directly, changing latent refinement depth, and invoking deliberative action selection under high uncertainty.

A. Imitation and Reinforcement Learning · B. Classifier Guidance for Flow-Matching Policies · IV. INFERENCE-TIME SCALING LAW

The paper frames VLA action prediction through imitation learning and RL, then describes classifier guidance as velocity-field steering for flow-matching policies. Its scaling analysis finds that sample diversity improves action selection mainly in failure states, while potentially harming already-successful predictions.

  • A. Imitation and Reinforcement Learning: VLA policies are trained on large-scale robot demonstrations to imitate action chunks conditioned on observations and language instructions.The imitation objective maximizes the log-likelihood of demonstrated action chunks.
  • A. Imitation and Reinforcement Learning: RL formulates decision-making as a POMDP in which the policy selects action chunks to maximize expected cumulative return.
  • B. Classifier Guidance for Flow-Matching Policies: Classifier guidance steers pretrained flow-matching VLA policies by modifying their velocity field during the ODE-based action-generation process.The flow field progressively transports initial noise toward a valid action trajectory.
  • B. Classifier Guidance for Flow-Matching Policies: The guided velocity adds λ times a differentiable objective function to the original velocity, with λ controlling intervention strength.This biases generated trajectories toward the desired objective while preserving the underlying policy prior.
  • IV. INFERENCE-TIME SCALING LAW: The scaling study measures normalized root-mean-square action error as generated sample count increases for success and failure tuples using π0 on BridgeV2.Average NRMSE is computed across tuple sets, with ground-truth actions available for the evaluation.
  • IV. INFERENCE-TIME SCALING LAW: The evaluation compares repeated sampling, language rephrasing, RBF repulsion, RL policies, concatenation, residual steering, and RLT-style offline RL steering.The approaches vary in whether they induce diversity through rephrased instructions, flow repulsion, or offline RL policies conditioned on VLA latents and actions.
  • IV. INFERENCE-TIME SCALING LAW: Failure-tuple error follows an exponential power law, with RL2 compositional steering showing the strongest scaling behavior and significantly lower action error.RBF also performs competitively, indicating that diversity is valuable when the base VLA fails.
  • IV. INFERENCE-TIME SCALING LAW: Success-tuple performance reverses: most methods underperform Rephrase, while RBF and RL2 are among the worst because diversity can perturb accurate base-VLA actions.The results support activating compositional steering primarily when base-policy failure is likely.

V. METHOD · A. RL Latent Policy Training

RL2 trains a latent-conditioned RL flow-matching policy from augmented demonstrations and composes its guidance with a frozen VLA to diversify action candidates. The framework applies this steering selectively when an external failure detector predicts that the VLA is failing, while using QAM for stable RL optimization.

  • V. METHOD: RL2 applies RL compositional steering only when an external failure detection module deems the pretrained VLA to be failing.The framework and its pseudocode are presented in Fig. 5 and Algorithm 1.
  • A. RL Latent Policy Training: The steering policy πRL(at:t+H | et) is a flow-matching policy conditioned on latents extracted from the VLA action expert.These latents are aggregated into a single embedding vector e.
  • A. RL Latent Policy Training: BridgeV2 and DROID datasets are augmented with policy latents to construct training tuples D = {oi, ai, ℓi, ei}N_i=1.The augmented data conditions RL policy training on the embedding e.
  • A. RL Latent Policy Training: Each dataset observation is passed through the VLA to extract the embeddings used in the augmented training data.The procedure iterates through every dataset tuple before adding the resulting VLA embeddings.
  • A. RL Latent Policy Training: QAM replaces numerically unstable critic backpropagation through multi-step flow matching with a stable, step-wise matching objective.It uses a lean adjoint state ˜gt as a time-dependent guidance signal, computed backward from terminal action a1 through a reverse ODE.
  • A. RL Latent Policy Training: The resulting policy targets the behavior-regularized distribution π(a|s) ∝ πβ(a|s) exp(τQ(s, a)) while retaining full expressivity through first-order optimization.Adjusting inverse temperature τ is sufficient to outperform behavior cloning baselines.
  • A. RL Latent Policy Training: 500k-1M steps of RL policy training run on two NVIDIA L40s GPUs, with training time largely determined by VLA latent dimensionality.RL2 also supports non-flow-based methods such as V-GPS policies trained with Conservative Q-Learning for autoregressive VLAs such as OpenVLA.
  • V. METHOD: For flow-matching VLAs, RL2 composes RL guidance velocities VRL with frozen-VLA flow velocities VVLA at each flow-matching step to generate diverse, high-quality action candidates.The framework extracts action-expert latents et from the frozen VLA and feeds them into the RL steering policy.

B. Compositional Steering · C. Failure Detection for Adaptive Steering

RL^2 composes frozen VLA and offline-RL flow velocities to diversify candidate actions, then adaptively applies steering only when a failure detector predicts the base policy may fail. The detector uses VLA action-expert features, conformal thresholds, and task-specific calibration to decide between steered and original VLA actions.

  • B. Compositional Steering: RL^2 composes VLA and RL velocity fields as a weighted average during flow matching, steering the frozen VLA toward more diverse, higher-quality candidates.The RL policy is conditioned on expressive VLA-derived latents, while the VLA remains frozen.
  • B. Compositional Steering: For autoregressive VLAs, RL^2 fits a Gaussian to equally mixed VLA and RL action samples before drawing a larger candidate batch.This replaces direct flow-velocity composition when both action heads cannot be treated as flow-matching policies.
  • C. Failure Detection for Adaptive Steering: The failure detector activates steering only when the base VLA is predicted to fail, otherwise retaining the original VLA action distribution.This avoids unnecessary or detrimental perturbations when the base policy already produces feasible actions.
  • C. Failure Detection for Adaptive Steering: The detector maps VLA action-expert features through an LSTM to produce a timestep-specific failure score s_t = f_ψ(e_0:t) in [0, 1].It is trained with binary cross-entropy across timesteps, and feature aggregation strategies are selected using validation performance.
  • C. Failure Detection for Adaptive Steering: Steering triggers when the predicted failure score exceeds a time-varying threshold δ_t calibrated with one-sided conformal prediction.Calibration on successful validation rollouts guarantees s_t < μ_t + h_t for all timesteps with probability at least 1 − α.
  • C. Failure Detection for Adaptive Steering: The CP significance level α controls steering frequency, so RL^2 selects it by validation balanced accuracy and calibrates the top three candidates before final evaluation.Balanced accuracy averages the true-positive and true-negative rates and correlates with downstream task performance.

D. Action Verification

RL^2 uses an action verifier as a reward model to score candidate VLA or steered actions and execute the highest-scoring sample at each timestep. The framework supports verifiers trained through either preference learning or contrastive learning, and adaptive steering improves OOD prompt robustness by up to +14.7% in task-wise success rate over the Rephrase baseline.

  • Action Verification: At each timestep, an action verifier scores candidate VLA or steered action samples and selects the highest-scoring sample for execution.The verifier functions as a reward model for action selection.
  • Action Verification: RL^2 accommodates action verifiers trained with preference learning, such as RoboMonkey, or contrastive learning, such as CoVer.This design makes the verification component adaptable to different training paradigms.
  • Action Verification: +14.7% in task-wise success rate is achieved by adaptive RL^2 over the Rephrase baseline under OOD language instructions.Using π0 on SIMPLER, the average improvement is +10.1%.

VI. EXPERIMENTS · A. Experimental Setup · 1) Compositional steering is robust to standard and OOD language prompts:

The experiments evaluate RL2’s compositional steering for performance, latent/RL effectiveness, scaling, and robustness across standard and out-of-distribution prompts and environments. Results span multiple VLAs, simulators, verifiers, and task settings, with reported gains in success and progress rates.

  • VI. EXPERIMENTS: The experiments test whether compositional steering improves task performance, how latents and RL enable steering, and how RL2 scales with action samples.These questions define the evaluation objectives.
  • A. Experimental Setup: RL2 is evaluated across diverse simulation benchmarks, VLAs, and verifiers, including standard benchmarks and OOD language instructions and task environments.The evaluation follows prior test-time scaling frameworks.
  • A. Experimental Setup: SIMPLER Original evaluates OpenVLA on four BridgeV2 in-domain manipulation tasks using original prompts, with V-GPS-trained CQL as the RL policy and RoboMonkey as verifier.The tasks mimic demonstration setups from large-scale datasets.
  • A. Experimental Setup: SIMPLER OOD evaluates π0 on four tasks with red-teaming instructions and four additional Interleave-VLA environments varying backgrounds, objects, and distractors.The tasks assess improvements from VLM-generated rephrases.
  • A. Experimental Setup: PolaRiS OOD evaluates π0.5 on three DROID-adapted tasks in high-fidelity Gaussian-splat simulation with distractors and OOD red-teaming prompts.QAM provides RL flow-based steering and CoVer serves as verifier.
  • A. Experimental Setup: +17.3% in task-wise success rate (S) and +12.4% in task-wise progress rate (P) are achieved by adaptive RL2 over the Rephrase baseline on PolaRiS OOD prompts.The table reports these as robustness gains under OOD language instructions using π0.5.
  • A. Experimental Setup: +7.5% average performance gain is achieved by adaptive RL2 for OpenVLA, including +19.4% task-wise gain, across the reported benchmarks.The experiments report general success-rate improvements for OpenVLA, π0, and π0.5.
  • 1) Compositional steering is robust to standard and OOD language prompts:: +14.6% in task-wise success rate, with average +8.5%, is reported for adaptive RL2 over the strongest Rephrase baseline in SIMPLER OOD environments.This evaluation uses π0 and targets robustness to OOD environments.

B. Simulation Results … 1) Adaptiveness is crucial for effective steering:

RL^2 improves simulation performance across in-domain and out-of-distribution tasks, with adaptive compositional steering producing diverse actions during predicted failure states. Ablations indicate that adaptiveness, RL training, VLA latents, and efficient failure detection are important components of the method.

  • B. Simulation Results: +10.1% average and +14.7% highest-task gains are achieved by adaptive RL2 over Rephrase for π0 on SIMPLER tasks with OOD language prompts.The highest gain occurs on the Spoon on Towel task.
  • B. Simulation Results: +10.9% success-rate and +7.8% progress-rate gains are achieved over Rephrase across PolaRiS OOD-language tasks, including +17.3% success on Move Latte Cup.These results further support performance gains beyond SIMPLER.
  • B. Simulation Results: RL compositional steering improves robustness by introducing diverse failure-state actions, including higher-quality samples directed toward the target when rephrased actions are uniformly misdirected.The Spoon on Towel visualization contrasts uniformly directed rephrase samples with more diverse steered samples.
  • 2) Compositional steering generalizes well to OOD task environments:: +8.5% average and up to +14.6% task-wise gains over Rephrase show compositional steering generalizes to unseen SIMPLER environments with varied backgrounds, objects, and distractors.The evaluation uses π0 with OOD language prompts.
  • C. Ablations: Ablation studies evaluate whether RL2's components are necessary for downstream performance across multiple benchmarks.The reported ablations cover RL training, VLA latents, sample and rephrase scaling, failure detection, timing, and alpha selection.
  • 1) Adaptiveness is crucial for effective steering:: Adaptive steering yields up to +8.9% average and +16.7% task-wise gains, with stronger benefits across OpenVLA and π0.5 than π0.Removing failure detection makes steering non-adaptive; π0 remains nearly on-par, while broader results favor adaptiveness.

2) Using latents and RL results in effective steering: … 3) Our steering policy and failure detection module introduce minimal additional latency:

The ablations show that RL-trained latent steering improves diversity over behavior cloning and raw-observation conditioning, while SAFE-based failure detection supports adaptive compositional steering. Additional studies report gains from scaling sample quantity and diversity, effective per-task α selection, and minimal added latency.

  • 2) Using latents and RL results in effective steering:: RL outperforms behavior cloning by up to +4.5% for π0/0.5 composition at all times.The result supports RL as a training recipe that introduces diversity for verifier action selection during VLA failure.
  • 2) Using latents and RL results in effective steering:: Latent-conditioned steering is compared against raw-observation conditioning, but raw-observation QAM could not be successfully trained in BridgeV2.The raw-observation comparison therefore uses OpenVLA with a CQL policy trained in V-GPS for vision-language inputs.
  • 3) Failure detection methods outperform other triggers for adaptive compositional steering:: CoVer-based failure detection is evaluated as an external-VLM trigger using verifier scores and conformal-prediction bands.The passage reports that CoVer generally leads to improved results as a failure detector, though the supplied text is truncated before the quantitative comparison.
  • D. Additional Studies: The additional studies validate the scalability and practicality of the RL2 framework.These experiments cover sample scaling, α selection, and inference latency.
  • 1) RL2 scales well with increasing quantity and diversity of samples:: Increasing samples from 1 → 40 per language prompt and increasing rephrases support scaling quantity and diversity during failure states.The study evaluates four SIMPLER tasks with out-of-distribution prompts using π0, with and without compositional steering.
  • 1) RL2 scales well with increasing quantity and diversity of samples:: +18.7% is the maximum performance gain of RL2 over Repeated and Rephrase when both scaling types are combined.This result supports the scaling laws established earlier in the paper.
  • 2) Our alpha selection heuristic is effective in proposing the top per-task alphas during test-time:: The balanced-accuracy heuristic selects task-specific CP significance levels α by ranking tested values from α = 0.05 to α = 0.50 for top-k choices.The evaluation tests k = {1, 3, 5, 10}.
  • 3) Our steering policy and failure detection module introduce minimal additional latency:: RL2’s QAM steering policy and SAFE failure detector remain significantly faster than the VLA forward pass as action-sample counts increase.The modules are lightweight and do not incur substantial latency relative to the verifier-based baselines.

4) Qualitative analysis highlights the efficacy of compositional steering: · E. Real-World Evaluation · VII. CONCLUSION

RL^2 improves VLA behavior by steering action distributions toward task-relevant, diverse actions during failures, while adaptive activation avoids perturbing accurate behavior. Real-robot results support its robustness and modularity, alongside limitations involving steering comparisons, failure detection, and verifier assumptions.

  • 4) Qualitative analysis highlights the efficacy of compositional steering:: RL^2 steers more action samples toward the task object when the VLA fails, diversifying verifier candidates beyond oscillatory or distractor-directed behaviors.The base VLA often moves away from the task object or toward distractors, whereas RL^2 redirects more samples toward the task object.
  • 4) Qualitative analysis highlights the efficacy of compositional steering:: On BridgeV2 failure tuples, compositional steering shifts the π0 action distribution closer to the ground-truth action than the concentrated base-VLA distribution.The distributions are visualized along their two principal axes using PCA.
  • E. Real-World Evaluation: Real-world evaluation uses a PiperX arm and Realsense D405 camera with π0, QAM, and CoVer, but SAFE failure detection does not generalize similarly to real-world settings.The experiments reuse the π0 simulation model weights.
  • E. Real-World Evaluation: The real-world study evaluates four OOD tasks spanning red-teaming base prompts and unseen task environments, including Carrot on Plate, Cube in Toolbox, Tape in Toolbox, and Screwdriver in Toolbox.The latter two environments contain objects not seen in the BridgeV2 dataset.
  • E. Real-World Evaluation: Vanilla π0 performs poorest because it often moves away from or fails to grasp the task object.This qualitative failure motivates comparison with steering baselines on hardware.
  • E. Real-World Evaluation: +17.5% average performance gain over Rephrase and +14.2% over non-adaptive RL^2 are achieved by adaptive RL^2 in real-robot experiments.The gains are attributed to RL-induced diversity during failure states and steering only when failures are detected.
  • VII. CONCLUSION: RL^2 combines lightweight RL on VLA latents with adaptive compositional steering, producing diverse actions beyond demonstration modes and improving modular VLA deployment.The conclusion reports validation across VLAs, verifiers, and manipulation benchmarks, including in-domain and OOD settings.
  • VII. CONCLUSION: The paper identifies unresolved limitations: no comparison with differentiable steering from large models, test-time and online-data costs in failure detection, and an assumption of a robust verifier.Future work targets offline failure-detection training, verifier co-training, and broader steering comparisons.

APPENDIX · A. Additional Implementation Details

The appendix details training setups for RL steering and SAFE failure detection, along with VLA feature aggregation procedures. It reports dataset-specific training budgets, latent dimensionality effects, and implementation choices for extracting fixed-dimensional embeddings.

  • A. Additional Implementation Details: RL steering policies were trained on two NVIDIA L40 GPUs, with training time largely determined by VLA-backbone embedding size.The appendix lists QAM and V-GPS hyperparameters in Tables V and VI.
  • A. Additional Implementation Details: QAM training used π0 latents with BridgeV2 for 500k steps in 4.5 hours and π0.5 latents with DROID for 1M steps in 9 hours.The DROID setup used 90% DROID data and 10% simulation data, following PolaRiS.
  • A. Additional Implementation Details: V-GPS training used OpenVLA latents with BridgeV2 for 500k steps in 48 hours, reflecting OpenVLA’s four-times-larger latent dimension, 4096 versus 1024.The training used Conservative Q-Learning instead of Calibrated Q-Learning because it performed better on BridgeV2.
  • A. Additional Implementation Details: VLA feature extraction aggregates an internal feature vector E ∈ R^n×d′ into a fixed-dimensional embedding e ∈ R^d.The aggregation dimension n may represent token position or diffusion steps, while d denotes feature dimension.
  • A. Additional Implementation Details: SAFE provides four aggregation methods: First, Last, Mean, and Concat, respectively selecting the first, last, average, or first-and-last features along dimension n.Concat concatenates the first and last dimensions along n.
  • A. Additional Implementation Details: For autoregressive VLAs, features are extracted before output-token decoding from the last transformer block; for flow-matching VLAs, extraction occurs before velocity-field generation.The stated examples are OpenVLA for autoregressive VLAs and π0/0.5 for flow-matching VLAs.
  • A. Additional Implementation Details: SAFE failure detection uses the LSTM variant, with its best training hyperparameter selected by grid search.The appendix provides SAFE training hyperparameters in Table VII.

B. Additional Task Description

The evaluation spans in-domain, out-of-distribution, simulation, and real-world manipulation tasks across SIMPLER, PolaRiS, and BridgeV2 settings. OOD scenarios vary objects, prompts, distractors, environments, and deployment conditions, with task-specific action verifiers used for in-domain and OOD tasks.

  • Evaluation Scope: The study evaluates 4 in-domain SIMPLER tasks, 8 OOD SIMPLER tasks, 3 OOD PolaRiS tasks, and 4 real-world tasks.SIMPLER and real-world tasks use the WidowX setup, while PolaRiS tasks use the Franka Panda manipulator.
  • OOD Simulation Tasks: OOD tasks introduce distribution shifts through red-teamed prompts, unseen objects or distractors, altered backgrounds, and longer-horizon scenes.PolaRiS tasks include many distractor objects, while the SIMPLER examples swap task objects or backgrounds relative to standard tasks.
  • PolaRiS OOD Simulation Tasks: The three PolaRiS OOD tasks are Pan Cleaning, Tape into Container, and Move Latte Cup, each evaluated with OOD base prompts.These scenes contain distractors such as condiment bottles, cups, cans, tools, and hand sanitizer.
  • BridgeV2 Real-world Tasks: The four OOD real-world tasks introduce shifts in camera placement, lighting, background, or task objects and require sufficiently high lifts for toolbox placement.They include Carrot on Plate, Cube in Toolbox, Tape in Toolbox, and Screwdriver in Toolbox.
  • Choice of Verifier: In-domain evaluation uses the RoboMonkey action verifier, whereas OOD evaluation uses the CoVer action verifier to select among action candidates.RoboMonkey uses preference learning; CoVer uses contrastive learning with action trajectory history and text-vision embeddings.

C. Additional Experimental Details

Additional experiments validate SAFE as an effective failure-triggering signal, describe task-specific conformal α selection, and show that RL^2 adds limited latency relative to the VLA. Qualitative examples further illustrate adaptive RL^2 recovering from challenging OOD prompts and environments.

  • CP Alpha Selection Heuristics: The conformal prediction heuristic selects the top-3 α values per task, using balanced accuracy and policy performance to break ties.When standalone RL outperforms the VLA, the larger α is selected to permit more frequent steering; the reverse applies when the VLA performs better.
  • Additional Time Analysis: RL^2 maintains a similar scaling trend across RTX5090 and H100 hardware, with QAM and SAFE latency significantly below the VLA’s latency.The comparison supports RL^2’s applicability to real-world deployment.
  • Intervention Trigger Comparisons: SAFE separates success and failure rollouts, supporting its use as an intervention trigger for compositional steering.This separation is shown for both π0 and π0.5, while CoVer is less effective in the corresponding comparison.
  • OOD and Real-World Examples: Qualitative OOD examples show adaptive RL^2 succeeding more often after preemptive failure detection via conformal prediction, unlike repeated rephrasing failures.The examples cover SIMPLER, PolaRiS, and real-robot tasks involving spoons, sponges, carrots, and cubes; the toy-dinosaur example also contrasts adaptive and non-adaptive RL^2.
Loading 2607.26991v2…