PaperlayerView exact revision on arXiv

Source-linked AI summary

DistAL: Distance-based Advantage Learning for VLA Fine-Tuning

Reece O'Mahoney, Ioannis Havoutis

arXiv:2609.18392v1cs.RO

TL;DR

Intervention-based and online-RL fine-tuning bring operational, sample-efficiency, stability, or compute burdens. DistAL uses kNN distance in SigLIP feature space as a dense reward for advantage-conditioned fine-tuning. It improves over the base policy on simulation benchmarks and two real-robot tasks, with a 96.2% LIBERO-10 score.

  • Problem

    Intervention-based fine-tuning adds hardware, operational, and expert-teleoperator burdens, while online-RL approaches face poor sample efficiency, instability, and high computational cost.

    Source: §I · p. 1

  • Method

    DistAL replaces the sparse success/failure reward with a dense per-step kNN distance to the base VLA’s training distribution in SigLIP feature space, leaving the rest of the advantage-conditioning pipeline fixed.

    Source: §I · p. 1, A. kNN feature-distance reward · p. 3, §VI · p. 7

  • Results

    96.2% LIBERO-10 score; DistAL also achieves a 78.4% overall mean across LIBERO-10 and LIBERO-plus (+3.3 over base), and improves over the base policy on two real-robot tasks.

    Source: B. Simulation experiments · p. 5, §VI · p. 7

  • Takeaways & Limitations

    kNN distance in the base VLA’s SigLIP feature space was the strongest tested OOD failure predictor at both trajectory and per-timestep levels.

    Source: §I · p. 2

    The image-based reward ignores force, contact stability, robot state, and language, limiting performance when failure involves an unseen slip such as on the remove pen lid task.

    Source: §VI · p. 7

Abstract

from arXiv · show

Vision-language-action models (VLAs) have trans- formed the field of robotic manipulation in recent years by combining the semantic understanding of LLMs with the precise control of flow-matching policies. Advantage conditioning is a recent technique that iteratively improves VLAs by training a value function on deployment data and using this to train an advantage-conditioned policy. Previous works have only applied simple, low-information success/failure rewards, which leave the value function unable to distinguish states of differing quality beyond how far along the task they appear. Motivated by an exploration of out-of-distribution (OOD) detection methods, we introduce Distance-based Advantage Learning (DistAL), which, by using an embedding space distance as a reward, produces a more informative value function and subsequently a higher downstream task success rate. We validate our method on a series of simulation benchmarks and dexterous bi-manual manipulation tasks on real hardware.

I. INTRODUCTION

Deployment-data methods for improving VLAs face trade-offs: intervention-based training requires expert oversight, while online reinforcement learning is costly and can be unstable. DistAL replaces sparse success/failure rewards in advantage conditioning with a dense feature-distance reward, and is evaluated in simulation and on real-robot tasks.

I. INTRODUCTION

  • Intervention-based improvement requires an expert teleoperator and adds hardware and operational complexity, while online VLA reinforcement learning faces sample-efficiency, stability, and computational-cost challenges.Recent work improving sample efficiency still depends on human-provided rewards and interventions during training.Source: §I · p. 1
  • Sparse success/failure rewards rank states largely by estimated steps remaining, so an unstable grasp and a secure grasp can receive the same value despite different prospects of success.Task-specific rewards are an alternative, but require laborious tuning or frame-by-frame labelling.Source: §I · p. 1
  • DistAL replaces sparse success/failure rewards with a dense kNN feature-distance reward in advantage conditioning, leaving the policy, value function, and training procedure unchanged.The distance is measured in the base VLA’s SigLIP feature space.Source: §I · p. 1
  • The study compares OOD-detection methods as predictors of VLA task failure and identifies kNN distance in the base VLA’s SigLIP feature space as the strongest signal.It evaluates prediction at both trajectory and per-timestep levels.Source: §I · pp. 1–2
  • DistAL is validated on the LIBERO and LIBERO-plus simulation suites and two dexterous bi-manual manipulation tasks on real hardware.Source: §I · p. 2

II. RELATED WORK

Prior VLA improvement approaches use interventions, online reinforcement learning, or offline value learning on logged deployment data. DistAL belongs to the offline family and targets its sparse-reward limitation with a dense embedding-distance reward.

II. RELATED WORK

  • Flow-matching policies generate continuous action chunks, whereas early VLA approaches such as RT-1 [6] discretised actions and RT-2 [1] and OpenVLA [2] predicted tokens autoregressively.OpenVLA-OFT [20] mitigates the speed issue with parallel decoding.Source: §II · p. 2
  • VLA improvement from deployment data spans intervention-based methods, online reinforcement learning or distillation, and offline value learning on logged trajectories.Intervention approaches require an expert teleoperator, while online reinforcement learning faces sample inefficiency, instability, and substantial compute costs.Source: §II · p. 2
  • Within offline value learning, Steering Your Generalists [30] re-ranks candidate actions by value, while π∗0.6/RECAP [8] conditions a flow-matching VLA on a binary advantage label.Source: §II · p. 2
  • DistAL replaces the sparse success/failure rewards used by prior advantage-conditioned methods with a dense embedding-distance reward.The related methods include π∗0.6/RECAP [8].Source: §II · p. 2
  • DistAL repurposes distance-to-training-set signals from OOD detection as per-timestep rewards for offline value learning, rather than using them for runtime rejection.The passage describes feature-space nearest-neighbour and class-conditional Mahalanobis distances, alongside deep ensembles as an uncertainty backbone.Source: §II · p. 2

III. PRELIMINARIES

The preliminaries describe flow-matching VLAs as generating continuous action chunks from observation-language context. Advantage conditioning fits a value function on deployment trajectories, thresholds estimated advantages into labels, and uses classifier-free guidance to steer the policy.

III. PRELIMINARIES

  • Flow-matching VLAs generate continuous action chunks from observation-language context using a learned time-dependent velocity field.Inference integrates that field from Gaussian noise over flow time using a small number of Euler steps, typically 5 to 10.Source: §III · pp. 2–3
  • In the DistAL architecture, the value function is trained on deployment data with the kNN feature-distance reward, and its resulting advantage label is injected into the VLA via classifier-free guidance.Source: Figure 2 · p. 3
  • Advantage conditioning fits a state-value function to deployment-data returns, then thresholds Monte Carlo advantages into positive or null conditioning labels.The label is dropped with probability p during classifier-free guidance training.Source: §III · p. 3
  • At inference, classifier-free guidance steers the velocity field toward actions conditioned on positive advantage, with guidance weight w greater than 1.Source: §III · p. 3

IV. METHOD

DistAL changes the reward used by advantage conditioning: it replaces the constant per-step penalty with a dense reward based on k-nearest-neighbour distance in vision-embedding space.

IV. METHOD

  • DistAL replaces advantage conditioning’s constant per-step penalty with a dense reward based on k-nearest-neighbour distance in vision-embedding space.The section describes the score calculation and encoder choice, then how the score plugs into the rest of the pipeline.Source: §IV · p. 3
  • The distance reward is designed as a replacement within the existing advantage-conditioning pipeline, rather than a change to the pipeline itself.Source: §IV · p. 3
  • The method section covers both how the distance score is calculated and how it is integrated into advantage conditioning.Source: §IV · p. 3

A. kNN feature-distance reward

DistAL scores deployment observations by their k-nearest-neighbours distance from the base VLA’s training embeddings, using the negative distance as a dense reward. This treats familiarity with the training distribution as a signal of state quality, while retaining a terminal failure penalty to protect task completion.

A. kNN feature-distance reward

  • For a deployment observation, kNN distance is computed in a frozen image encoder’s embedding space against embeddings indexed from the base VLA’s training data.The encoder choice matters: a frozen SigLIP embedding from the base VLA performed best among the alternatives described.Source: A. kNN feature-distance reward · p. 3
  • The per-timestep reward is the negative kNN distance, normalized to [−1, 0].Observations near the training distribution receive lower distance scores, while those farther away receive higher scores before negation.Source: A. kNN feature-distance reward · p. 3
  • The reward assumes observations closer to the base VLA’s training distribution are better, but that assumption is not universally true.A policy could remain in familiar early states, and a successful recovery maneuver could leave the training manifold.Source: A. kNN feature-distance reward · p. 3
  • A large terminal penalty on failed episodes is retained so that staying in distribution without completing the task still yields a low return.The dense distance signal measures per-state quality, while the failure penalty preserves a signal of eventual task success.Source: A. kNN feature-distance reward · p. 3

B. Architecture

The architecture largely follows the advantage-conditioning pipeline in [8], with a value function that predicts discretized returns and a VLA policy conditioned on binary advantage labels. The best reported policy fine-tunes only its action expert while freezing the backbone.

B. Architecture

  • The value function combines a pre-trained 270M-parameter Gemma3 language model with a 400M-parameter SigLIP vision encoder, then predicts discretized return bins through a two-layer MLP head.It receives the same image, language, and state conditioning as the policy and is trained with cross-entropy.Source: B. Architecture · p. 3
  • The authors also tried the HL-Gauss loss but found no benefit over cross-entropy.Source: B. Architecture · p. 4
  • The VLA combines a fine-tuned π0.5 model with a PaliGemma 3B-parameter VLM and a 300M-parameter flow-matching action expert.The model follows π0’s architecture.Source: B. Architecture · p. 4
  • The policy receives its binary advantage label by appending “Advantage: positive” or “Advantage: negative” to the tokenized language input.For the unconditional case, nothing is appended.Source: B. Architecture · p. 4
  • The best results came from fine-tuning only the action expert and freezing the backbone; inference conditions on the positive label, optionally with guidance.The positive-label threshold is set so that 30% of advantage labels are positive.Source: B. Architecture · p. 4

C. Training

DistAL training scores deployment observations with negative kNN distance, fits a value function to empirical returns, and uses estimated advantages to condition policy fine-tuning. Classifier-free guidance steers inference toward high-advantage actions.

C. Training

  • Each deployment observation is scored against the training-set embedding index, and its negative kNN distance becomes the per-step reward.Source: C. Training · p. 4
  • The value function is fit to empirical returns, and each timestep receives a positive label when its estimated advantage exceeds a threshold.The estimated advantage is the empirical return minus the value estimate.Source: C. Training · p. 4
  • The VLA is fine-tuned with the advantage label, which is dropped with probability p during training; inference uses classifier-free guidance to steer toward high-advantage actions.Source: C. Training · p. 4

A. OOD detection as a failure predictor

The study compares OOD scores for predicting rollout success or failure, reporting both trajectory-level and timestep-level AUROC. Base-VLA pre-LM SigLIP kNN distance is strongest on both measures, though visual failure signals are weak for pen-lid slips.

A. OOD detection as a failure predictor

  • 0.67 mean per-timestep AUROC and 0.82 per-trajectory AUROC: base-VLA pre-LM SigLIP kNN distance is the strongest detector on both measures.The per-timestep measure is more relevant because the value function is fitted to per-step rewards and must assign credit locally.Source: Table II · p. 4, Table I · p. 4, A. OOD detection as a failure predictor · p. 4
  • Post-LM image tokens are less reliable than pre-LM SigLIP features, with a 0.57 mean AUROC and an inversion to 0.31 on LIBERO.The authors report that the per-timestep AUROCs retain the per-trajectory ordering at uniformly lower values.Source: A. OOD detection as a failure predictor · p. 4
  • Pen-lid is the weak point: its per-timestep AUROC is 0.56, close to chance; early frames in rollouts that eventually fail are often indistinguishable from frames in successful rollouts.On pen-lid, kNN trails the VAE by 0.04 (0.66 vs. 0.70).Source: A. OOD detection as a failure predictor · pp. 4–5
  • The kNN score separates successful and failed rollout rewards more cleanly than the other embedding-based detectors, with the gap opening progressively through the episode.Fig. 3 averages 10 random episodes of each outcome and min-max normalizes rewards per method.Source: A. OOD detection as a failure predictor · p. 5, Figure 3 · p. 5

B. Simulation experiments

DistAL achieves the strongest aggregate simulation results, though its advantage over binary-reward conditioning is small on aggregates and varies by perturbation. Its gains are clearest on visually shifted perturbations, while some other perturbations regress.

B. Simulation experiments

  • Evaluation uses LIBERO-10 and LIBERO-plus, with a shared deployment dataset of 50 labeled rollouts per task and mean success over three seeds of 50 episodes per task.LIBERO-Spatial, -Object, and -Goal are omitted because π0.5 already saturates them.Source: B. Simulation experiments · p. 5
  • DistAL reaches 96.2% on LIBERO-10 and a 78.4% overall mean across LIBERO-10 and LIBERO-plus, +3.3 over base.The dense distance reward adds +1.2 on both aggregates over binary-reward conditioning, a margin small relative to seed-to-seed variation.Source: B. Simulation experiments · p. 5
  • Success-filtered SFT is indistinguishable from base at −0.3 on the overall mean, while advantage-conditioned methods gain through the value function rather than the successful data subset.The comparisons include the base π0.5 policy and a RECAP-style binary-reward pipeline.Source: B. Simulation experiments · pp. 5–6
  • Against base, both advantage-conditioned methods gain most on Camera (+25.2 DistAL; +20.4 binary), Noise (+14.1; +16.3), and Texture (+10.7; +3.0), but regress on Layout and Language.DistAL reaches 100.0% on Camera and Texture; its Light and Noise scores are 2.3 and 2.2 points below binary reward, respectively.Source: B. Simulation experiments · p. 6
  • DistAL wins 5 of 8 perturbation columns and matches on one, with its largest gains over binary reward on Camera (+4.8) and Texture (+7.7).The reported pattern is gains where observations shift from the training distribution and losses where they do not.Source: B. Simulation experiments · p. 6

C. Hardware experiments

On hardware, DistAL’s clearest advantage over binary-reward conditioning is on ethernet-cable removal; pen-lid performance is treated as tied. The results support applying DistAL where failures are visually distinguishable from successes.

C. Hardware experiments

  • On remove ethernet cable, DistAL reaches 87% success versus 46% for base and 72% for binary reward; on remove pen lid it reaches 52%, treated as tied with binary reward.The pen-lid margin over binary reward is four points and is not statistically significant at 50 rollouts.Source: Table IV · p. 6, C. Hardware experiments · p. 6
  • The real-robot evaluation uses two Piper arms and three cameras on two contact-rich tasks, with 50 tele-operated demonstrations and 50 labeled deployment rollouts per task.Success is averaged over 50 evaluation rollouts per task.Source: C. Hardware experiments · p. 6
  • Ethernet failures are visually obvious (per-trajectory AUROC 0.94), while pen-lid slips under load are not visible in images (per-timestep AUROC 0.56).When the reward cannot see failure, the downstream policy degrades to binary reward plus noise.Source: C. Hardware experiments · p. 6
  • DistAL is expected to help most when task failures are visually distinguishable from successes, and to offer little benefit when failures are primarily haptic.The paper presents visual separability as a condition to check before applying the method.Source: C. Hardware experiments · p. 7

VI. CONCLUSION AND LIMITATIONS

DistAL replaces sparse success/failure rewards with a dense reward based on k-nearest-neighbour distance in SigLIP feature space, improving over the base policy across the evaluated simulation suites and two real-world tasks. Its advantage over binary rewards depends on visual separability, and the evaluation leaves assumptions and generality unresolved.

VI. CONCLUSION AND LIMITATIONS

  • DistAL improves over the base π0.5 policy on LIBERO-10, LIBERO-plus, and two real-world bi-manual tasks, and over binary rewards where failure is visually separable.The clearest binary-reward gains occur on Camera and Texture perturbations and the ethernet task.Source: §VI · p. 7
  • On the pen-lid task and perturbations that leave the scene visually unchanged, DistAL and binary rewards are indistinguishable at the evaluated sample sizes.Source: §VI · p. 7
  • DistAL relies on its own deployment data and cannot improve a policy until it reaches an unknown baseline success level sufficient to produce a meaningful signal.Source: §VI · p. 7
  • The evaluation covers one base VLA, two real-world tasks, LIBERO suites, and three training seeds per configuration, leaving generality across embodiments and task families unestablished.Smaller reported differences are not statistically resolved.Source: §VI · p. 7

Related papers

Loading 2609.18392v1…