PaperlayerView exact revision on arXiv

Source-linked AI summary

Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization

Haoru Li, Jinmei Liu, Zhiyong Wang, Xiaoming Li, Zhenhong Sun, Daoyi Dong, Chunlin Chen, Zhi Wang

arXiv:2610.09943v1cs.ROcs.LG

TL;DR

VLA policies remain sensitive to distribution shifts, while standard task rewards do not distinguish among different successful solutions. DRIVE groups rollouts under matched task conditions and uses temporally aligned trajectory comparisons to derive a success-conditioned diversity reward. Across three simulation benchmarks, DRIVE improves OOD macro-average over Vanilla RFT by 5.3 points on π0 and 2.0 points on π0.5, plus a 9.2-point gain on a dual-arm AgileX PiPER-X platform.

  • Problem

    VLA policies remain sensitive to distribution shifts, while standard task rewards do not distinguish among different successful solutions.

    Source: §1 · p. 1, §4.1 · p. 5

  • Method

    DRIVE groups rollouts under matched task conditions and uses temporally aligned trajectory comparisons to derive a success-conditioned diversity reward.

    Source: §1 · p. 2, §4.2 · p. 5, §4.3 · p. 6

  • Results

    Across three simulation benchmarks, DRIVE improves OOD macro-average over Vanilla RFT by 5.3 points on π0 and 2.0 points on π0.5, plus a 9.2-point gain on a dual-arm AgileX PiPER-X platform.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Successful-mode coverage is a useful lens for understanding RFT and a practical objective for improving VLA generalization.

    Source: §6 · p. 10

    Pairwise GAK comparisons add computation and rely on VLM representations; scalability and generality beyond current settings remain to be assessed on longer-horizon tasks, broader embodiments, and online real-world fine-tuning.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited. Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning. Broader successful-mode coverage may provide alternative strategies under distribution shifts. Inspired by this, we introduce DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which turns successful-behavior diversity into an explicit RL objective. DRIVE groups rollouts under matched task conditions, compares their trajectories with temporal alignment, and derives a success-conditioned intrinsic reward from relative behavioral diversity. This design encourages broader coverage of feasible solutions without rewarding diverse failures or superficial timing differences. Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE improves the average out-of-domain (OOD) performance over vanilla RL fine-tuning by 5.3 points on $\pi_0$ and 2.0 points on $\pi_{0.5}$. On a dual-arm AgileX PiPER-X platform, DRIVE further increases average OOD success from 64.1% to 73.3% (+9.2 points), demonstrating gains that persist under physical deployment.

1 INTRODUCTION

VLA generalization remains challenging beyond the fine-tuning distribution, and standard task rewards do not distinguish among successful solutions. The paper analyzes how RFT reshapes successful behaviors and introduces DRIVE to explicitly reward their diversity.

1 INTRODUCTION

  • RFT increases similarity across all rollouts while decreasing it among successful trajectories, and successful behaviors become more accessible and cover more task-valid solutions than under SFT.This suggests broader successful-mode coverage may provide alternative strategies under distribution shifts.Source: §1 · p. 2
  • Standard binary task rewards distinguish success from failure but provide no explicit incentive to maintain broad coverage among different successful solutions.Applying diversity objectives directly can reward timing differences, irrelevant motion, or unsuccessful behaviors.Source: §1 · p. 2
  • Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE raises OOD macro-average over Vanilla RFT from 48.1% to 53.4% for π0 and from 71.1% to 73.1% for π0.5.On a dual-arm AgileX PiPER-X platform, average OOD success rises from 64.1% to 73.3% (+9.2 points).Source: §1 · p. 2
  • DRIVE groups rollouts under matched task conditions, compares trajectories with temporal alignment, and assigns a relative novelty bonus only to successful trajectories.The method turns successful-mode diversity into an explicit optimization target for promoting generalization.Source: §1 · p. 2

2 RELATED WORK

Prior work studies VLA generalization through data, model capabilities, task-reward optimization, and behavioral diversity. DRIVE instead targets broader task-valid solution coverage using success-conditioned diversity across matched-task rollouts.

2 RELATED WORK

  • Prior VLA generalization work emphasizes data augmentation, domain randomization, large-scale pretraining, heterogeneous co-training, and model designs for spatial representation and embodied reasoning.These efforts primarily focus on training data and model capabilities, with less attention to how RFT-shaped behavior distributions affect generalization.Source: §2 · pp. 2–3
  • RFT research uses task-completion or progress rewards and reports success and some OOD gains, but pays less attention to successful-behavior distributions and explicitly promoting their breadth.Behavioral-diversity work includes unsupervised skill discovery, intrinsic-motivation exploration, and task-aware diversity among trajectories or near-optimal behaviors.Source: §2 · p. 3
  • DRIVE applies success-conditioned diversity across rollouts under matched task conditions, targeting task-valid solution coverage rather than global behavioral variation.Figure 2 summarizes RFT's global and successful-trajectory similarity trends, success elicitation, and reference-repertoire coverage.Source: §2 · p. 3, Figure 2 · p. 3

3 EXPLORATION DYNAMICS IN VLA REINFORCEMENT FINE-TUNING

RFT contracts behavior globally while diversifying successful trajectories, making success more accessible and broadening the task-valid solutions sampled under limited rollout budgets. This broader coverage may provide alternatives under distribution shifts.

3 EXPLORATION DYNAMICS IN VLA REINFORCEMENT FINE-TUNING

  • The study measures trajectory similarity across all and successful rollouts under matched task conditions, and samples up to 256 rollouts per policy and configuration for accessibility analysis.The sampling-budget analysis compares SFT and final-RFT π0 on 32 task-initial-state configurations.Source: §3 · p. 3
  • Success rates rise from 41% to 75% for π0 and from 52% to 77% for π0.5 as RFT increases similarity across all rollouts but decreases similarity among successful ones.Global contraction therefore masks successful-mode diversification rather than implying collapse of task-valid exploration.Source: §3 · p. 3
  • RFT achieves substantially higher Pass@K than SFT at small rollout budgets, while the gap narrows as SFT saturates at larger budgets.This suggests successful behavior can be latent in SFT but difficult to elicit, whereas RFT makes it accessible with fewer rollouts.Source: §3 · p. 4
  • Against SFT successes at K = 256, RFT achieves substantially higher Coverage@K at small K and matches SFT coverage with far fewer rollouts.SFT queries use leave-one-out matching to avoid trivial self-matches and ensure a fair comparison.Source: §3 · p. 4
  • Broader successful-mode coverage may provide alternative strategies under distribution shifts, motivating explicit incentives for successful-trajectory diversity.Standard rewards do not distinguish among different successful solutions.Source: §3 · p. 4

4 MORE WAYS TO SUCCEED WITH DRIVE

DRIVE formulates RFT around trajectory-level similarity and converts aligned, within-condition behavior comparisons into success-conditioned diversity shaping. Its reward distinguishes successful solutions while preserving the original optimal policies.

4.1 PROBLEM FORMULATION

  • The method frames language-conditioned control as an MDP and uses PPO, whose clipped importance-ratio objective updates the policy from reward-derived advantages.Because advantages depend on rewards, the reward design determines which behaviors RFT reinforces; standard task rewards do not distinguish behaviorally different successes.Source: §4.1 · pp. 4–5

4.2 DERIVING TRAJECTORY-LEVEL BEHAVIORAL SIMILARITY

  • GAK compares ordered trajectory-feature sequences while preserving temporal order and accommodating local timing differences through monotonic alignments.The local similarity uses shifted cosine similarity mapped to [0, 1], and the resulting kernel is normalized using trajectory self-similarities.Source: §4.2 · p. 5

4.3 SUCCESS-CONDITIONED DIVERSITY SHAPING

  • Within each matched rollout group, relative diversity reflects how behaviorally distinct a trajectory is from the others, and is normalized among successful trajectories while failed trajectories receive zero.The similarity comparison includes both successful and failed rollouts, but only successful trajectories receive a diversity reward.Source: §4.3 · pp. 5–6
  • DRIVE assigns successful trajectories a diversity potential and uses its change between adjacent states as an intrinsic reward; diverse failures receive no bonus.The initial-state potential is zero, avoiding diversity calculations for intermediate prefixes.Source: §4.3 · p. 6
  • The paper states that success-conditioned potential-based shaping changes the learning signal without changing the set of optimal policies.The shaping reward is balanced against environment reward by α.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments assess DRIVE’s generalization across simulation benchmarks, behavioral dynamics during fine-tuning, and physical deployment. The results compare DRIVE with Vanilla RFT and other baselines across multiple benchmarks and two VLA backbones.

5 EXPERIMENTS

  • Experiments compare DRIVE with Vanilla RFT and three exploration or regularization baselines across three simulation benchmarks and two VLA backbones, using a shared SFT initialization and PPO-based Flow-SDE pipeline.The evaluation also includes transfer to a physical dual-arm platform.Source: §5 · pp. 6–7, §5.3 · p. 9

5.1 MAIN RESULTS

  • For π0, DRIVE raises OOD macro-average from 48.1% to 53.4%; for π0.5, from 71.1% to 73.1%, improving 11/12 splits on both versus Vanilla RFT.On π0.5, DRIVE also achieves the highest benchmark-family macro-average among the compared exploration and regularization strategies.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 GENERALIZATION AND BEHAVIORAL DYNAMICS

  • During RFT, both DRIVE variants outperform Vanilla RFT at multiple OOD checkpoints, although OOD performance fluctuates rather than improving monotonically.The tested shaping strengths, α = 0.1 and α = 0.2, show broadly similar performance trends.Source: Figure 4 · p. 8, §5.2 · p. 8
  • DRIVE’s behavioral diversity is clearest among successful trajectories: their GAK similarity falls relative to Vanilla RFT, while all-rollout trends remain broadly similar.The behavioral comparison uses DRIVE with α = 0.1.Source: Figure 4 · p. 8, §5.2 · pp. 8–9
  • Full DRIVE achieves higher IND success than diversity shaping without success conditioning as RFT progresses, with the separation clearer in later stages.The variants have comparable early-stage IND performance, with no consistent early advantage.Source: Figure 5 · p. 8, §5.2 · p. 9

5.3 REAL-WORLD EXPERIMENTS

  • In physical deployment, DRIVE raises average OOD success from 64.1% to 73.3% across two tasks, a 9.2-point gain over Vanilla RFT.DRIVE performs better in seven of eight task-condition combinations and matches Vanilla RFT in the remaining one; each pair is evaluated over 20 trials.Source: Table 2 · p. 9, §5.3 · p. 9

6 CONCLUSIONS, LIMITATIONS, AND FUTURE WORK

DRIVE improves OOD performance while using successful-behavior diversity as a training objective, and its potential-based shaping preserves optimal policies. Its trajectory comparisons retain ordered features and use temporal alignment to accommodate differences in execution speed and length.

6 CONCLUSIONS, LIMITATIONS, AND FUTURE WORK

  • 5.3 points on π0 and 2.0 points on π0.5 OOD macro-average gains over Vanilla RFT extend to a 9.2-point gain on dual-arm AgileX PiPER-X.The reported simulation gains are across three benchmarks.Source: §6 · p. 10
  • Pairwise GAK comparisons add computation and rely on VLM representations to capture meaningful behavioral differences; longer-horizon tasks, broader embodiments, and online real-world fine-tuning remain future assessments.The authors identify these as directions for assessing scalability and generality.Source: §6 · p. 10

A OPTIMAL POLICY INVARIANCE OF DRIVE

  • Following Ng et al. (1999), the proof shows that success-conditioned potential-based shaping preserves the set of optimal policies.The optimal action sets are identical at every state in the original and transformed MDPs.Source: A OPTIMAL POLICY INVARIANCE OF DRIVE · p. 14

B TRAJECTORY REPRESENTATION AND GLOBAL ALIGNMENT SIMILARITY

  • Each trajectory is an ordered variable-length sequence of per-decision VLM-prefix features plus a terminal-observation feature, rather than a single collapsed embedding.The features are mean-pooled over valid prefix tokens.Source: B TRAJECTORY REPRESENTATION AND GLOBAL… · p. 14
  • GAK accommodates different execution speeds and trajectory lengths by aggregating monotonic alignment paths that preserve temporal order.Its recurrence lets either trajectory, or both, advance at each transition.Source: B TRAJECTORY REPRESENTATION AND GLOBAL… · p. 15
  • The implementation evaluates the dynamic program in log space for numerical stability and uses normalized GAK similarity as trajectory-level behavioral similarity.The normalized similarity is used throughout the paper.Source: B TRAJECTORY REPRESENTATION AND GLOBAL… · p. 15

C.1 LIBERO-PLUS

The LIBERO-Plus evaluation keeps the original base tasks while varying task conditions between training and held-out evaluation. It specifies the task pools, perturbation splits, and a fixed one-episode-per-definition evaluation protocol.

C.1 LIBERO-PLUS

  • The split retains all ten original base tasks in each of four LIBERO suites and tests shifts in variation values rather than held-out task identities.The underlying manipulation semantics remain fixed while observation and execution conditions change.Source: §C.1 · p. 15
  • Each suite's IND pool contains 250 task definitions: ten clean LIBERO tasks and 240 selected variants from four perturbation families.The variants cover Confounding Objects, Target Object Pose, Scene Theme, and Surface Appearance; LIBERO Long uses base-task-uniform hierarchical sampling.Source: §C.1 · p. 15, Table 3 · p. 16
  • OOD conditions use perturbation values excluded from the IND pool, with additional held-out shifts in camera viewpoints, robot initial states, language, lighting, and sensor noise.Object-layout, scene-theme, and surface-appearance identities or configurations are separated between IND and OOD.Source: §C.1 · p. 16
  • Each task definition is evaluated at its first available initialization, yielding one episode per definition; reported success is the mean binary success rate over the IND or OOD set.The same task–initialization pairs are used for all compared methods and backbones.Source: §C.1 · p. 16

C.2 MANISKILL3

The ManiSkill3 evaluation uses one pick-and-place objective and tests generalization through visual-language, semantic-grounding, and execution shifts. Its protocol runs a fixed number of episodes per setting and aggregates success rates by OOD category.

C.2 MANISKILL3

  • ManiSkill3 uses the PutOnPlateInScene25Main-v3 pick-and-place task, with 16 training objects and 16 training scenes in the IND environment.Object and plate initial states are randomized.Source: §C.2 · p. 16
  • The OOD evaluation groups settings into visual-language, semantic, and execution shifts while keeping the basic manipulation objective.The shifts cover altered appearance or language, competing target objects or receptacles, and changed initial or dynamic execution states.Source: §C.2 · p. 17
  • Each setting runs 320 episodes; success is averaged within settings and then aggregated by OOD category across 4,160 OOD episodes.The protocol also includes 320 IND episodes and uses the same fixed evaluation configuration for compared methods and backbones.Source: §C.2 · p. 17, Table 5 · p. 17

C.3 ROBOTWIN 2.0

RoboTwin 2.0 evaluates two dual-arm tasks on controlled IND and OOD configuration sets. The split combines validated physical-state configurations with deterministic observation-level perturbations and uses shared evaluation subsets across methods.

C.3 ROBOTWIN 2.0

  • The evaluation covers the dual-arm Click Bell and Press Stapler tasks, with task objectives unchanged across IND and OOD splits.Distribution shifts vary objects, scene appearance, camera and lighting, robot state, language, and visual observations.Source: §C.3 · p. 17, Table 6 · p. 18
  • Each task has 1,000 IND and 500 OOD configurations, with OOD adding held-out values and shifts in lighting, camera viewpoint, language, and sensor observations.The IND set includes clean configurations and variations in object configuration, background appearance, and robot initial state.Source: §C.3 · pp. 17–18
  • Of 1,500 configurations per task, 300 are grounded in expert-validated scene states and 1,200 are deterministic observation-level perturbations derived from validated states.The 300 validated configurations comprise 250 IND and 50 OOD anchors; derived appearance shifts reuse physically valid underlying scenes.Source: §C.3 · p. 18
  • Evaluation uses a shared fixed subset of 350 IND configurations per task and all 500 OOD configurations, scoring each rollout by binary success and macro-averaging the two task rates.The same perturbation assignments are used for all compared methods and backbones.Source: §C.3 · p. 18

D TRAINING AND IMPLEMENTATION DETAILS

The experiments hold core training settings constant within each benchmark–backbone comparison, then specify rollout collection, trajectory comparison, reward shaping, and baseline modifications for DRIVE.

Common RFT setup

  • All methods within each benchmark–backbone setting share an SFT checkpoint, training horizon, rollout distribution, and PPO configuration.PPO uses Flow-SDE sampling, γ = 0.99, GAE λ = 0.95, and clip ratio 0.2; compared methods also share the runner horizon within each LIBERO-Plus table column.Source: D TRAINING AND IMPLEMENTATION DETAILS · p. 18, Table 7 · p. 19

Rollout collection

  • 64 groups of 8 stochastic rollouts yield 512 trajectories per collection, with each group sharing a task and environment condition.DRIVE computes trajectory similarity only within each group.Source: D TRAINING AND IMPLEMENTATION DETAILS · p. 19

DRIVE implementation

  • DRIVE represents each rollout as a temporal sequence of mean-pooled VLM-prefix features from executed action chunks, plus the terminal observation, then computes pairwise normalized GAK similarity within groups.H denotes the action prediction horizon, and H′ the number of actions executed before replanning.Source: D TRAINING AND IMPLEMENTATION DETAILS · p. 19
  • Only successful trajectories receive the relative diversity bonus, which is applied at the terminal action chunk with a fixed α throughout training.This avoids explicitly rewarding behaviorally unusual failures; benchmark-specific α values are reported in Tables 7 and 8.Source: D TRAINING AND IMPLEMENTATION DETAILS · p. 19

Baseline implementations

  • Auxiliary baselines modify one exploration or regularization component while retaining the corresponding Vanilla RFT pipeline settings.The variants increase sampling noise, add a KL penalty toward the SFT reference policy, or raise PPO’s upper clipping bound.Source: D TRAINING AND IMPLEMENTATION DETAILS · p. 19

E REAL-WORLD SETUP AND EVALUATION DETAILS

The physical evaluation tests Click Bell and Press Stapler on a calibrated dual-arm platform, with aligned simulation training and clean and shifted visual conditions. The section also documents representative successful Click Bell executions across those conditions.

Physical platform and simulation alignment

  • The platform uses two AgileX PiPER-X arms, and its corresponding simulation is aligned to the physical robot placement, workspace geometry, and observation viewpoints.The tasks are Click Bell and Press Stapler.Source: E REAL-WORLD SETUP AND EVALUATION… · pp. 20–21

Geometric and camera calibration

  • Each arm is calibrated independently across 48 poses using robot-state transforms and ChArUco-based camera estimates, with both arms referenced to a common workspace frame.OpenCV’s Shah solver recovers each robot base’s workspace pose and the end-effector-to-camera transform; the shared frame also provides relative arm placement.Source: E REAL-WORLD SETUP AND EVALUATION… · pp. 20–21

Simulation alignment

  • Simulation alignment transfers real-camera intrinsics, preserves wrist-camera extrinsics through the fixed PiPER-X asset, and matches the static viewpoint, illumination, and appearance.The wrist-camera extrinsics are not applied again from hand-eye calibration, avoiding a double application of the camera-to-robot transform.Source: E REAL-WORLD SETUP AND EVALUATION… · p. 21

Task initialization and training

  • Expert trajectories collected in aligned simulation provide a common SFT initialization for independently trained Vanilla RFT and DRIVE policies before physical evaluation.Both policies use the same observation and action interface on the physical platform.Source: E REAL-WORLD SETUP AND EVALUATION… · p. 21

Real-world evaluation protocol

  • Each task is tested in one clean condition and three shifts—Distractor, Background, and Lighting—that alter objects, tabletop appearance, or illumination.Each task–condition pair is evaluated over 20 trials using the same conditions for Vanilla RFT and DRIVE.Source: E REAL-WORLD SETUP AND EVALUATION… · p. 21

Representative physical rollouts

  • Figure 7 shows five temporally ordered frames from one successful Click Bell execution per condition, with the task objective unchanged as visual context varies.The four rows correspond to Clean, Distractor, Background, and Lighting conditions.Source: E REAL-WORLD SETUP AND EVALUATION… · p. 21, Figure 7 · p. 22
Loading 2610.09943v1…