PaperlayerView exact revision on arXiv

Source-linked AI summary

DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies

Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Jong Chul Ye

arXiv:2610.00317v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Chunk-level VLA training optimizes local action likelihood without directly accounting for long-horizon task success, while sequence-level reinforcement learning often depends on costly rollouts. DriftOPD decomposes sequence-level reverse KL into local and future terms, optimizing them with one-step drifting and a demonstration-trained critic. Across diverse simulation and real-world settings, it generally outperforms one-step distillation baselines and achieves task success comparable to multi-step policies.

  • Problem

    Chunk-level training does not directly account for final task success, while sequence-level reinforcement learning commonly requires costly closed-loop policy rollouts.

    Source: §1 · pp. 1–2

  • Method

    DriftOPD decomposes sequence-level reverse KL into local action-distribution and future-sequence terms, using one-step drifting and a demonstration-trained critic for rollout-free optimization.

    Source: §1 · p. 3, §4.4 · p. 7, §6 · p. 10

  • Results

    Across diverse VLA configurations in simulation and real-world manipulation, DriftOPD generally outperforms one-step distillation baselines while achieving performance comparable to multi-step policies.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    DriftOPD distills long-horizon behavior into one-step VLA action experts using offline demonstrations, without online interaction or a separate teacher.

    Source: §1 · p. 2, §6 · p. 10

    The offline critic’s action-gradient quality is limited by demonstration coverage, and it adds little beyond the teacher/student baseline when a one-step teacher already matches its multi-step counterpart in several LIBERO settings.

    Source: E LIMITATIONS AND FUTURE WORK · p. 45

Abstract

from arXiv · show

Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.

1 INTRODUCTION

Chunk-level VLA training is convenient across embodiments but favors locally likely actions and faces test-time distribution mismatch, while sequence-level reinforcement learning depends on costly closed-loop rollouts. DriftOPD uses a one-step drifting objective and a demonstration-trained critic to distill sequence-level behavior, outperforming one-step baselines and matching multi-step policies on task success.

1 INTRODUCTION

  • Chunk-level training favors locally likely action chunks without considering final task success and can fail to generalize from demonstration states to model-generated states.The resulting test-time distribution mismatch is one of the stated limitations of this training paradigm.Source: §1 · p. 1
  • Sequence-level reinforcement learning can provide sequence feedback, but requires policy-generated sequences and costly closed-loop interaction on real robots.These methods execute the current policy, evaluate sequences, and often recollect rollouts as the policy changes.Source: §1 · p. 2
  • DriftOPD decomposes sequence-level reverse-KL optimization into a local chunk term and a future-potential term, using one-step drifting and a demonstration-trained critic without a separate teacher or online rollouts.It learns from human demonstrations and generates actions in one step.Source: §1 · pp. 2–3
  • DriftOPD generally outperforms existing one-step distillation methods while achieving task success rates comparable to multi-step teachers across simulation and real-world manipulation.It also provides more stable training in data-scarce settings and faster execution with negligible loss of action diversity.Source: §1 · pp. 2–3

2 RELATED WORK

Related work establishes action chunking as common in VLAs, while identifying gaps in both one-step distillation and sequence-level on-policy distillation. DriftOPD targets these gaps by bringing task-success signals into action-chunk sequences without requiring closed-loop interaction.

2.1 VISION-LANGUAGE-ACTION (VLA) MODELS

  • Action-chunk imitation provides no direct supervision for final task success, and reinforcement-learning feedback from physical rollouts requires costly real-robot interaction.Behavior-cloned policies are also vulnerable to test-time distribution shift toward model-generated states.Source: §2.1 · p. 3

2.2 ONE-STEP DISTILLATION OF DIFFUSION/FLOW MODELS

  • Existing one-step diffusion and flow distillation methods mainly match action-chunk distributions or ODE trajectories without directly distilling task-success supervision.This leaves a gap between chunk-level training objectives and closed-loop test-time behavior.Source: §2.2 · p. 3

2.3 ON-POLICY DISTILLATION

  • Continuous-model OPD still distills within each chunk, while VLA-OPD requires closed-loop interaction and repeated teacher queries; DriftOPD aims to incorporate task-success signals across chunk sequences.Flow-based experts also lack tractable clean-action likelihoods, complicating direct sequence-level OPD.Source: §2.3 · p. 4

3 PRELIMINARY

The preliminaries frame on-policy distillation as sequence-level reverse-KL optimization whose gradient separates immediate discrepancy from future effects. They also introduce drifting models, which use a sample-space vector field to update generated samples toward a target distribution.

3.1 ON-POLICY DISTILLATION

  • Sequence-level reverse-KL optimization separates current-prediction discrepancy from the effect of that prediction on subsequent sequence-level discrepancy.The later-step term accumulates discrepancy over subsequent steps; unlike discrete autoregressive models, continuous generative policies generally lack tractable clean-sample densities.Source: §3.1 · p. 4

3.2 DRIFTING MODELS

  • Drifting models update generated samples using a sample-space vector field induced by the discrepancy between target and model distributions.The local descent direction is the difference between their score functions, and when densities are unavailable the field can be approximated from samples using kernel density estimation.Source: §3.2 · pp. 4–5
  • Figure 2 contrasts demonstration-supervised forward KLD, chunk-level reverse KLD, rollout-based reinforcement learning, and DriftOPD’s sequence-level approximation without online rollouts.DriftOPD combines a drifting objective with critic guidance.Source: Figure 2 · p. 5

4 METHOD

The method decomposes sequence-level reverse-KL into current-chunk discrepancy and the effect of current actions on future behavior, then approximates both for VLA action experts.

4.1 SEQUENCE-LEVEL REVERSE KL DIVERGENCE IN VLA

  • The Bellman decomposition separates current action-chunk reverse-KL from how that chunk changes later discrepancy through future contexts.The future term captures the effect of the current action on subsequent action chunks.Source: §4.1 · pp. 5–6

4.2 SINGLE-POSITIVE LOCAL TERM: DRIFTING FIELD

  • Because action experts lack explicit action-chunk densities, the local reverse-KL is kernel-smoothed and optimized with a target/model score-difference drifting field.The field is formed from the difference between target and model scores.Source: §4.2 · p. 6
  • For each offline demonstration context, its recorded action chunk serves as the single positive sample, alongside samples from the current policy.This lets the local reverse-KL term be optimized from a single positive demonstration chunk.Source: §4.2 · p. 6

4.3 ACTION-VALUE FUTURE TERM: CRITIC SURROGATE

  • A frozen action-value critic approximates the future gradient without current-policy rollouts to evaluate remaining divergence; larger Q values indicate better actions.The critic remains fixed during policy optimization.Source: §4.3 · p. 7

4.4 OBJECTIVE FUNCTION

  • The unified objective pairs drifting local matching with a frozen-critic future surrogate, using offline demonstrations and λ to weight the future term.These two components correspond to the local and future terms in the sequence-level decomposition.Source: §4.4 · p. 7

5 EXPERIMENT

The experiments evaluate DriftOPD across VLA architectures and manipulation settings, comparing one-step distillation with other one-step methods and multi-step policies. Simulation and real-world results favor DriftOPD among one-step methods, with gains in some comparisons over multi-step policies and lower GPU time to success than a 10-step teacher.

5 EXPERIMENT

  • The evaluation covers π0.5, GR00T N1.5/N1.6, and ABot-M0 across fine-tuning strategies, with simulation benchmarks including RoboCasa365 and RoboTwin 2.0.Each embodiment-specific model is trained with DriftOPD for 10k steps; the critic is adapted from a demonstration-trained Q-function, and evaluations follow official protocols.Source: §5 · p. 8, §5.1 · p. 8

5.1 SIMULATION EXPERIMENTS

  • On RoboCasa365 and RoboTwin 2.0, DriftOPD is mostly superior or comparable to sCD, MFD, and Drift, and surpasses the multi-step teacher in several suites.It generally improves over chunk-level drifting distillation by adding critic guidance for future task progress.Source: §5.1 · pp. 8–9, Table 2 · p. 9
  • Critic weights from 0.001 to 0.3 consistently improve success over drifting-only training, while a weight of 1.0 causes performance collapse.The authors attribute the collapse as likely related to RMS normalization fixing the multi-bandwidth drifting loss scale.Source: §5.1 · p. 9
  • 2.60× faster than the 10-step teacher in GPU time to success on 10 RoboCasa365 atomic seen tasks with π0.5.GPU time to success counts policy calls multiplied by per-call latency; the figure distinguishes totals including failed episodes from means over successful episodes.Source: §5.1 · p. 9, Figure 4 · p. 9

5.2 REAL-WORLD EXPERIMENTS

  • DriftOPD achieves the highest one-step success rates on both unimanual pick-and-place and bimanual handover tasks in the real-world evaluation.The evaluation fine-tunes GR00T N1.6 on 638 teleoperated episodes comprising 366k frames, then distills for 10k steps.Source: §5.2 · p. 9, Table 3 · p. 9

6 CONCLUSION

DriftOPD decomposes sequence-level reverse-KL distillation into local and future terms, then approximates them with a drifting objective and a demonstration-trained critic. The paper reports competitive task success across simulation and real-world manipulation, with rollout-free, one-step distillation and no separate teacher.

6 CONCLUSION

  • Across diverse VLA configurations, DriftOPD generally outperforms one-step distillation baselines while matching multi-step policies in task success.The conclusion also reports stable training in data-scarce settings and faster execution than multi-step generation.Source: §6 · p. 10

A.1 DRIFTING FIELD AND REVERSE KL

  • For any η > 0, minimizing the drifting objective follows the same parameter-space descent direction as KDE-smoothed reverse-KL minimization, up to scaling by 2η.The derivation assumes positive, continuously differentiable densities with sufficient boundary decay and valid exchanges of differentiation and expectation.Source: §A.1 · pp. 16–17

A.2 SEQUENCE-LEVEL REVERSE KL DIVERGENCE IN VLA

  • The sequence-level reverse-KL gradient separates the current chunk discrepancy from future discrepancy, but offline demonstration contexts substitute for model-induced contexts.The occupancy error vanishes as the model trajectory distribution approaches the target, though its bound can be loose when policies differ substantially.Source: §A.2 · pp. 18–19

A.3 LOCAL TERM AND DRIFTING FIELD

  • Averaging the single-positive drifting objective over demonstrations upper-bounds reverse KL to the full smoothed target, with zero gap for deterministic target conditionals.Finite-sample KDE error diminishes with more policy samples, while the evaluation-point approximation diminishes as kernel bandwidth tends to zero for a continuous field; the derivation does not require a Gaussian kernel.Source: §A.3 · pp. 19–20

A.4 FUTURE TERM AND CRITIC SURROGATE

  • An ideal frozen critic exactly represents the future-gradient contribution, while the practical critic's approximation depends on its action-gradient mismatch.The future contribution matters when the current action changes the successor distribution and future policy discrepancy remains; it vanishes as that discrepancy vanishes.Source: §A.4 · pp. 20–21

B.1 MODEL CONFIGURATIONS

  • Each one-step student starts from its corresponding teacher checkpoint, and only the parameters specified for its configuration are updated.Depending on the benchmark, experiments use released task-specific checkpoints or fine-tune a base checkpoint first.Source: §B.1 · p. 22

B.2 DRIFTOPD TRAINING DETAILS

The training details specify shared optimization settings alongside DriftOPD-specific drifting, critic, and batch configurations. These choices define the student sample budget and how the two objectives are combined.

B.2 DRIFTOPD TRAINING DETAILS

  • Unless otherwise specified, students train for 10k optimization steps and are evaluated using EMA parameters.The common settings apply to DriftOPD and the one-step baselines.Source: §B.2 · p. 23, Table 6 · p. 23
  • The default drifting setup uses a Laplacian affinity kernel with three bandwidths and a 65-sample pool comprising one demonstration positive and 64 policy samples.For RoboCasa365, π0.5 uses 32 policy samples and a 33-sample pool; generated samples are perturbed, and per-bandwidth normalized fields are averaged.Source: §B.2 · pp. 23–24
  • The critic weight starts at zero, warms up over 500 steps, and uses a gradient-norm ratio clipped to [0.1, 10] for rescaling.Simulation uses λ = 0.01 without critic-score clipping; real-world runs additionally hard-clip the critic term when generated-action Qϕ exceeds demonstration-action Qϕ.Source: Table 7 · p. 23, §B.2 · p. 24
  • Each optimization step generates eight gradient-carrying chunks per task context, with Nc varying from 4 for π0.5 to 64 for ABot-M0.Within each model and benchmark, all four students share the same context count and sample budget, so objective differences are isolated.Source: §B.2 · p. 24, Table 8 · p. 24

B.3 CRITIC TRAINING

The critic is trained once from offline demonstrations to provide action-dependent values, then frozen during policy distillation. The section also describes the comparison baselines and their implementation controls.

B.3 CRITIC TRAINING

  • The critic combines a frozen VLA prefix, robot state, and action chunk, then reads an action-conditioned value through twin Q-heads.It is trained per policy backbone and embodiment from offline demonstrations and frozen throughout distillation.Source: §B.3 · p. 25, Figure 5 · p. 25
  • Monte-Carlo returns from successful demonstration episodes supervise value regression because sparse episode caches make temporal-difference bootstrapping impractical.The critic objective also includes Cal-QL and EGR terms to induce local action dependence, which is needed because DriftOPD uses the action gradient rather than absolute value.Source: §B.3 · pp. 25–27
  • Eight OOD action chunks per training sample, split evenly between local and marginal candidates, provide action-variation supervision beyond demonstrated state–action pairs.The candidate-based objectives suppress OOD values and encourage value ordering to track distance from the demonstrated chunk.Source: §B.3 · pp. 26–27

B.4 BASELINE IMPLEMENTATIONS

  • DriftOPD and Drift share settings and differ by exactly the frozen critic's action-gradient term, while all one-step baselines share checkpoints, cached demonstrations, conditioning, and sample budgets.The baseline suite also includes a one-step teacher, sCD, and MFD.Source: §B.4 · pp. 27–28

B.5 SIMULATION EVALUATION PROTOCOL

The simulation evaluation follows benchmark execution protocols and uses task-specific episode and initialization procedures. The protocols include paired initial conditions where specified and an accepted-seed manifest for RoboTwin.

B.5 SIMULATION EVALUATION PROTOCOL

  • RoboCasa365 evaluates 50 tasks over 30 episodes each, with π0.5 methods sharing seed-7 initial conditions while official GR00T evaluations are unpaired.The task groups comprise 18 atomic seen, 16 composite seen, and 16 composite unseen tasks.Source: §B.5 · p. 29
  • RoboTwin evaluates 50 ALOHA-AgileX tasks under the demo-clean configuration, executing each chunk for 25 environment actions before replanning.The configuration disables several environment randomizations, and episodes end on success or the task-specific action limit.Source: §B.5 · p. 29
  • RoboTwin uses a shared per-task accepted-seed manifest containing expert-solvable initial conditions, enabling episode-by-episode alignment across compared methods.Thirty accepted episodes require a median of about 36 candidate seeds per task; short, medium, and long groups correspond to average demonstration lengths below 150, 150–279, and at least 280 steps.Source: §B.5 · p. 29
  • LIBERO evaluates 40 tasks over 50 episodes each using seed 7 and benchmark-provided initial states, while LIBERO-Plus evaluates 10,030 perturbed instances once each without additional training.LIBERO-Plus reuses the LIBERO students and execution protocol.Source: §B.5 · p. 29

B.6 REAL-WORLD DATA AND EVALUATION

Real-world experiments use a demonstration-trained GR00T N1.6 policy on YAM robots for unimanual and bimanual manipulation, with paired-scene evaluation. The section also describes the hardware/software setup and an oscillation metric for assessing command smoothness alongside task success.

B.6 REAL-WORLD DATA AND EVALUATION

  • 638 teleoperated episodes yielded 366,373 aligned observation-action frames for training and evaluation across 12 tasks on the YAM platform.The policy uses RGB observations and predicts seven-dimensional actions per arm; unimanual tasks use the right arm, while bimanual tasks use both arms.Source: §B.6 · p. 30
  • Four unimanual pick-and-place tasks span Easy to Challenging, alongside two bimanual handover-and-place tasks with swapped object and target locations.Evaluation replays the same 15 fixed scenes across methods, with visual similarity thresholds used to match initial conditions.Source: §B.6 · p. 31
  • Robot-controller speed and timeout limits apply equally to all methods, and collisions during an episode count as task failures.Episodes affected by operator error are repeated before evaluation rather than discarded afterward.Source: §B.6 · p. 31

B.7 HARDWARE AND SOFTWARE

  • Each benchmark uses a separate Python environment because simulator dependencies conflict; the installed software versions are reported where recoverable.All student-distillation and critic-training runs use one GPU per process.Source: §B.7 · p. 32

B.8 OSCILLATION METRIC

  • Real-robot evaluation adds command oscillation because binary success alone cannot reveal step-to-step direction reversals that stress hardware.Oscillation is the residual RMS after smoothing joint-position commands; it excludes grippers and is averaged over joints and accepted episodes, including failures.Source: §B.8 · pp. 32–33

C FULL RESULTS

The expanded results show that performance patterns vary across benchmarks and backbones: unseen composite RoboCasa tasks are difficult, while RoboTwin horizon gaps differ by backbone and LIBERO-Plus variation tracks perturbations more than method. Real-robot oscillation and LoRA-rank analyses further distinguish the one-step objectives.

C FULL RESULTS

  • Most composite unseen RoboCasa365 tasks are at 0% for every method, so the suite averages rest on a few tasks.The per-task tables cover π0.5 full fine-tuning, π0.5-LoRA, GR00T N1.5, and GR00T N1.6.Source: C FULL RESULTS · p. 33, Table 17 · p. 34, Table 18 · p. 35
  • At long RoboTwin 2.0 horizons, π0.5 loses 14.5 points from multi-step to one-step teacher performance and students partly recover the gap; ABot-M0 teachers both score 36.97%.Tasks are ordered by average demonstration trajectory length and grouped into short, medium, and long horizons.Source: C FULL RESULTS · p. 33, Table 21 · p. 38, Table 22 · p. 39
  • On LIBERO, methods separate mainly on long-horizon libero 10 because both backbones are near saturation on spatial, object, and goal suites, except MFD on ABot-M0, which scores 0% everywhere.The table reports success rates for each of the four standard suites and the 40-task average.Source: C FULL RESULTS · p. 33, Table 23 · p. 40
  • On LIBERO-Plus, π0.5-LoRA methods stay within about one point of the teacher across perturbation factors, while rates range from 71% for camera viewpoints to 98% for lighting.The 10,030 perturbed task instances are broken down by perturbation factor.Source: C FULL RESULTS · p. 33, Table 24 · p. 40
  • The RMS-normalized drifting loss stays near 1 by construction, so the critic term −Q is the meaningful moving component that distinguishes DriftOPD from Drift.Figure 7 plots the total DriftOPD loss with its drift term and critic term.Source: C FULL RESULTS · p. 33, Figure 7 · p. 41
  • Before normalization, bandwidth-wise RMS shows how drifting-term magnitude changes during training and is distributed across kernel bandwidths.Figure 8 reports this quantity for Drift and DriftOPD on RoboCasa365.Source: C FULL RESULTS · p. 33, Figure 8 · p. 42

D.1 OSCILLATION ANALYSIS FOR REAL-WORLD ROBOT

  • On the real robot, sCD and MFD oscillate more than the 4-step teacher on every task and embodiment, whereas Drift and DriftOPD remain at teacher-level oscillation.The full comparison covers right-arm 7-D and bimanual 14-D actions; Table 25 reports task success and oscillation.Source: §D.1 · p. 42, Table 25 · p. 42

D.2 STABILITY ACROSS LORA RANKS

  • DriftOPD outperforms the other objectives at every π0.5 LoRA rank and under full fine-tuning on RoboCasa365 atomic seen.The rank sweep compares student capacity settings against a 1-step teacher at 31.30%.Source: §D.2 · p. 43, Figure 9 · p. 43

D.3 ANALYSIS FOR SINGLE-POSITIVE DRIFTING FIELD

The single-positive analysis finds that MSE-only distillation collapses action diversity, while drifting distillation retains noise-dependent variation under the tested setup. A separate success-field analysis finds that a global chunk-level signal adds little compared with the observation-conditioned signal.

D.3 ANALYSIS FOR SINGLE-POSITIVE DRIFTING FIELD

  • MSE-only spread is 0.016, versus 0.120 for Drift and 0.117 for DriftOPD, indicating near-deterministic outputs that discard the initial noise.The comparison measures pairwise RMS among 32 one-step chunks at each of 288 fixed observations.Source: §D.3 · pp. 43–44, Figure 10 · p. 43
  • Additional independent noise samples per observation did not change measured diversity, and the authors attribute the gap to self-masking, pretrained initialization, and a multi-bandwidth kernel.The proposed ingredients preserve repulsion at scales where a single bandwidth would have saturated.Source: §D.3 · p. 44

D.4 SUCCESS FIELD DRIFTING

  • Across βSuccess values from 0.1 to 3.0, success stays flat, drops at 0.03, and collapses at 10; the broad stable range remains close to Drift’s 47.22 baseline.DriftOPD’s dashed baseline is 49.44; the authors characterize the chunk-level signal as a weak substitute for the conditioned signal.Source: Figure 11 · p. 44, §D.4 · p. 44
  • Marginalizing the success signal over observations appears to discard useful information because whether a chunk succeeds remains a function of its generating observation.The authors identify DriftOPD’s per-context, episode-level signal as the informative form.Source: §D.4 · pp. 44–45

E LIMITATIONS AND FUTURE WORK

The current demonstration-trained critic approximates the future-gradient contribution, but its action-gradient quality is limited by demonstration coverage. The formulation is not restricted to continuous action-chunk VLAs, though extending it to autoregressive models is future work.

E LIMITATIONS AND FUTURE WORK

  • The demonstration-trained critic’s action-gradient quality is limited by demonstration coverage, particularly where the one-step teacher already matches its multi-step counterpart.In several LIBERO and LIBERO-Plus settings, the critic adds little improvement beyond the teacher/student baseline.Source: E LIMITATIONS AND FUTURE WORK · p. 45
  • The study evaluates continuous action-chunk VLAs; applying sequence-level OPD to autoregressive discrete-action or action-token models remains future work.The paper notes that step-wise conditional probabilities are directly available for such models.Source: E LIMITATIONS AND FUTURE WORK · p. 45

Related papers

Loading 2610.00317v1…