PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Teacher On-Policy Distillation for Flow-Based Vision-Language-Action Models

Lichao Ni, Yuhan Liu, Pengfei Li, Guanghuan Fang

arXiv:2610.21522v1cs.RO

TL;DR

Flow-based VLA consolidation must address mismatches in both student-induced environment observations and intermediate action-generation states. The paper trains one shared student with frozen specialists’ velocity guidance on student-generated states and renewed student rollouts. It reports stronger aggregate performance than the SFT initialization, Action Loss OPD, and Teacher-rollout SFT across three benchmarks, while noting that the consolidated student remains below specialists in aggregate performance.

  • Problem

    For flow-based VLAs, teacher guidance can be mismatched to both student-induced environment observations and intermediate states along the student’s action-generation trajectory.

    Source: §1 · p. 1

  • Method

    Frozen domain specialists supervise one shared student using local velocity targets on student-generated action states and renewed student environment rollouts.

    Source: §1 · p. 1, §4.1 · p. 4

  • Results

    The shared student outperforms the SFT initialization, Action Loss OPD, and Teacher-rollout SFT on all aggregate metrics across LIBERO, Meta-World, and CALVIN.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    Two-level on-policy supervision integrates domain specialists into a single student deployed across domains.

    Source: §7 · p. 7

    The consolidated student still falls short of the corresponding specialists in aggregate performance, and domains retain different gripper-coordinate conventions.

    Source: §6 · p. 7

Abstract

from arXiv · show

Specialized vision-language-action (VLA) policies acquire strong domain-specific skills, yet consolidating these skills into one shared policy remains challenging. On-policy distillation (OPD) transfers teacher knowledge on inputs induced by the student, addressing the distribution mismatch encountered during deployment. For flow-based VLAs, this mismatch arises at two levels: environment observations and intermediate action-generation states. We propose a two-level on-policy distillation framework that couples student-driven environment interaction with teacher velocity supervision along student-generated action flow trajectories. Frozen domain specialists jointly supervise one student. After each update, the student collects new environment experience and generates fresh action-flow trajectories. Starting from a joint supervised fine-tuning (SFT) initialization, Velocity Loss OPD yields absolute success-rate gains of 22.55% on LIBERO-40, 33.4% on CALVIN five-instruction sequences, and 22.6% on Meta-World MT50. Compared with Action Loss OPD and Teacher-rollout SFT, it achieves absolute success-rate gains of 6.4-9.5% on LIBERO-40, 9.7-18.3% on CALVIN, and 3.2-4.2% on Meta-World MT50. Analysis by sequence length further shows larger success-rate gains on longer CALVIN sequences. These results demonstrate effective multi-domain capability consolidation through two-level on-policy supervision, with a single student deployed across domains.

1. Introduction

The paper frames flow-based VLA consolidation as a two-level distribution-matching problem: student behavior shapes both future environment observations and intermediate action-generation states. It proposes a shared student trained with specialist guidance on both, and reports gains over several baselines across three benchmarks.

1. Introduction

  • VLA policies map visual observations and language instructions to actions, while cross-domain consolidation must accommodate distinct observation and native action interfaces.The shared parameters integrate specialist supervision across LIBERO, Meta-World, and CALVIN.Source: §1 · p. 1
  • Teacher guidance on demonstrations can mismatch deployment observations, while flow-based policies also require choosing between final-action and intermediate-velocity supervision.A missed grasp can change both the next observation and the needed recovery action.Source: §1 · p. 1
  • The paper identifies two evolving input distributions: environment observations induced by executed actions and intermediate states induced by the student’s velocity field.It asks how to couple teacher guidance to the shared student at both levels as its capabilities evolve.Source: §1 · p. 1
  • The proposed framework combines student-driven environment interaction with frozen specialists’ local velocity targets along student-generated action trajectories.Weighted domain-loss gradients update the shared student, which then collects fresh experience and trajectories.Source: §1 · p. 1
  • The shared student outperforms the SFT initialization, Action Loss OPD, and Teacher-rollout SFT on all aggregate metrics across LIBERO, Meta-World, and CALVIN.The framework consolidates specialist capabilities in a shared π0.5 student.Source: §1 · p. 2

2. Related Work

The related work connects on-policy distillation to student-induced sampling in imitation learning, language models, and embodied policies, and situates the paper among multi-teacher consolidation and continuous-generation methods. The paper distinguishes its approach by coupling renewed student environment rollouts with local teacher velocity guidance on student action-generation paths.

Related work

  • Multi-teacher consolidation has been studied in language models and robotics through student-generated trajectories, domain scheduling, generated data, parameter merging, and student execution.This paper focuses on consolidating known domain experts into one action policy across manipulation benchmarks.Source: §2 · p. 2
  • Continuous-generation distillation includes student-trajectory supervision, transition matching, flow-map methods, and teacher velocity targets for robot policies.The paper retains the existing action-generation interface and focuses on cross-domain capability consolidation.Source: §2 · p. 3
  • Prior embodied-policy work includes student-rollout teacher queries, vulnerable-state supervision, and world-action-model approaches using fixed histories or fresh denoising paths.Concurrent work also studies sampling over observation histories and denoising states, including online OPD with renewed rollouts.Source: §2 · p. 3
  • The framework couples renewed student environment rollouts with local teacher velocity guidance on student action-generation paths across LIBERO, Meta-World, and CALVIN.The comparisons examine both the supervision target and the joint source of supervision inputs.Source: §2 · p. 3

3. Problem Formulation

The formulation defines a shared student that generates action chunks from observation-and-instruction contexts across domains with native interfaces. It distinguishes environment-context sampling from intermediate flow-state sampling and specifies shared parameters, domain-specific supervision, and loss weighting.

3. Problem Formulation

  • At each decision time, the shared student receives an observation-and-instruction context and generates an action chunk, executing part of its horizon before replanning.Horizon and native action dimension may differ by domain; preprocessing and action decoding follow each environment interface.Source: §3 · p. 3
  • The student starts from common demonstration-SFT initialization, is evaluated per domain against that initialization and its specialist teacher, and is deployed without teachers.The known domain selects the teacher used for training supervision.Source: §3 · p. 3
  • The formulation distinguishes environment decision contexts from intermediate action-generation states, whose distribution is induced by the initial noise and student velocity field.Flow time τ is distinct from environment time t; numerical solvers produce student and teacher action chunks.Source: §3 · p. 4
  • Each domain loss averages squared error over native action-chunk coordinates, excluding padding, before fixed weights combine domain contributions.Averaging each loss before weighting prevents differing action dimensions from implicitly determining relative domain weights.Source: §3 · p. 4

4. Multi-Teacher On-Policy Flow Distillation

The method alternates student collection, domain-teacher labeling, and student updates so supervision follows the evolving policy at both environment contexts and action-flow states. Its velocity objective uses local teacher targets on student-generated states, while the updated student refreshes both sampling distributions each round.

4. Multi-Teacher On-Policy Flow Distillation

  • Each round collects student-generated contexts and intermediate flow states in every domain, queries the corresponding frozen specialist, and updates one shared student.The procedure alternates collection, labeling, and updates to keep supervision aligned with the evolving policy at both levels.Source: §4 · p. 4, §4.3 · p. 4

4.1. Student-Driven Environment Collection

  • Student actions determine the environment contexts where each domain’s specialist supplies supervision, and contexts across domains train shared student parameters.Both successful and failed episodes are retained.Source: §4.1 · p. 4

4.2. Velocity Loss on Student Flow States

  • At each collected context, the teacher supplies a local velocity target at intermediate states sampled along the student’s action-generation trajectory.The implementation averages supervision over selected nonterminal solver times; contexts, states, and teacher targets remain fixed during the update.Source: §4.2 · p. 4

4.3. Alternating Collection and Student Updates

  • After collecting the same number of episodes per domain, the method accumulates weighted domain-loss gradients for one student update, then uses the updated student for the next round.Training uses frozen teachers without an additional reward objective; only the shared student is required at deployment.Source: §4.3 · p. 4

5. Experimental Evaluation

The evaluation compares shared-student distillation methods across LIBERO, Meta-World, and CALVIN, using a common joint-SFT initialization and matched training protocol. Velocity Loss OPD leads the shared-student comparisons, with additional results examining supervision targets, input sources, suite variation, and domain weighting.

5.1. Domains and Training Setup

  • All methods start from the same joint-SFT checkpoint and are evaluated through round 80 with fixed domain weights, learning rates, trainable parameters, and normalization statistics.Evaluation uses round-80 endpoint comparisons; student collection and evaluation use the same initial-state pools for LIBERO and Meta-World, while CALVIN collection uses scenes A/B/C and evaluation uses scene D.Source: §5.1 · pp. 4–5

5.2. Comparison Objectives

  • Velocity Loss OPD matches teacher velocities at student-generated flow states and environment contexts; Action Loss OPD instead matches final action chunks, while Teacher-rollout SFT uses teacher-generated inputs.The latter changes environment and internal-state sources jointly while retaining the local velocity objective and stop-gradient treatment.Source: §5.2 · pp. 5–6

5.3. Capability Transfer and Supervision Design

  • Velocity Loss OPD achieves the best shared-student performance across all three benchmarks, improving LIBERO by 22.55 percentage points, Meta-World by 22.60 points, and CALVIN average length by 1.127 instructions over πθ0.CALVIN Len-5 success rises from 39.5% to 72.9%.Source: Table 1 · p. 5, §5.3 · p. 6
  • Under student-driven collection, Velocity Loss OPD exceeds Action Loss OPD by 9.50 points on LIBERO, 4.20 points on Meta-World, and 0.618 instructions in CALVIN average length.Its aggregate metrics remain higher at every post-update evaluation; the CALVIN success-rate margin grows from 3.4 points for the first instruction to 18.3 points for all five.Source: Figure 2 · p. 5, §5.3 · p. 6
  • With the local velocity objective held fixed, Velocity Loss OPD exceeds Teacher-rollout SFT by 6.40 points on LIBERO, 3.20 points on Meta-World, and 0.288 instructions in CALVIN average length.Its CALVIN Len-5 margin is 9.70 points versus 0.90 points for the first instruction; Meta-World favors Teacher-rollout SFT at rounds 50–70 before reversing at round 80.Source: Figure 2 · p. 5, §5.3 · p. 6
  • Velocity Loss OPD exceeds Teacher-rollout SFT on all four LIBERO suites and leads Action Loss OPD on Object, Goal, and LIBERO-10, with a 21.2-point Object margin.The main loss weights outperform equal weighting on every aggregate domain metric, including a 29.0-point LIBERO-10 gain despite assigning LIBERO a smaller relative weight.Source: Figure 2 · p. 5, §5.3 · p. 6

6. Discussion and Limitations

Two-level on-policy supervision shows larger gains on longer tasks, while cross-domain action conventions and evaluation design bound the findings. The comparison of complete procedures does not isolate the contribution of each sampling level.

Benefits on longer tasks

  • The gain over Teacher-rollout SFT grows with task length: 13.8 points on LIBERO-10 versus 1.6–5.8 on the other LIBERO suites, and 9.7 points on CALVIN Len-5 versus 0.9 for the first instruction.The authors say this pattern is consistent with compounding distribution shift during longer rollouts.Source: §6 · p. 6
  • Student-driven environment states and intermediate flow states both receive guidance, which the authors offer as a plausible explanation for stronger gains on longer tasks.Early deviations can lead to observations and recovery situations increasingly unlike teacher trajectories, while changing observations also alter action-generation conditions.Source: §6 · pp. 6–7

Cross-domain action alignment

  • The shared student still trails the corresponding specialists in aggregate, and native action conventions assign gripper control to different coordinates across domains.Meta-World uses coordinate four, while LIBERO and CALVIN use coordinate seven; the authors propose harmonized action semantics as a direction for narrowing the remaining gap.Source: §6 · p. 7

Cost of decoupling the two sampling levels

  • Mixed-source variants were not evaluated because they require an additional complete integration path, so the comparison measures joint changes to both sampling distributions rather than each level separately.The additional path introduces sequential solver dependencies, unlike queries at already available path states, which can be batched.Source: §6 · p. 7

Evaluation scope

  • Each method has one training run; LIBERO and Meta-World evaluation uses collection initial-state pools, while CALVIN uses A/B/C collection and scene-D evaluation with prior teacher exposure to D unspecified.Meta-World rankings vary during training.Source: §6 · p. 7

7. Conclusion

The paper consolidates domain specialists into one shared student using two-level on-policy distillation. The method outperforms the compared alternatives on aggregate metrics, with larger gains over Teacher-rollout SFT on longer CALVIN sequences.

7. Conclusion

  • Velocity Loss OPD outperforms the SFT initialization, Action Loss OPD, and Teacher-rollout SFT on all aggregate metrics across LIBERO, Meta-World, and CALVIN.It also outperforms Action Loss OPD at every post-update evaluation.Source: §7 · p. 7
  • The framework consolidates domain-specialized flow-based VLAs into one shared student by combining teacher velocity guidance on student-generated action trajectories with renewed student-driven environment interaction.Domain specialists supervise student-induced environment and action-generation states.Source: §7 · p. 7
  • On CALVIN, Velocity Loss OPD’s success-rate gains over Teacher-rollout SFT are larger as instruction-sequence length increases.Source: §7 · p. 7

A. Implementation Details

A joint SFT checkpoint initializes the shared student, which is then trained with domain-specific teachers and a common student policy. The implementation uses domain-specific action interfaces and reports matched collection budgets across distillation methods.

A. Implementation Details

  • Each domain’s frozen teacher supervises its corresponding domain, and all domain losses update one shared student initialized from the joint SFT checkpoint.Teacher configurations are specified for LIBERO, Meta-World, and CALVIN; all methods report updates 1–80.Source: A. Implementation Details · p. 10

Shared SFT initialization

  • The shared π0.5 student starts from a joint SFT checkpoint at step 14,000, trained on LIBERO-40, Meta-World MT50, and CALVIN A/B/C demonstrations.The training partitions contain 1,880, 2,350, and 15,178 trajectories, respectively; each update samples 64 action windows per domain.Source: A. Implementation Details · p. 10

A. Implementation Details

  • Table 2 presents training settings shared by Action Loss OPD and Velocity Loss OPD.Source: Table 2 · p. 10

Action interface

  • Action interfaces use (H, m, N) = (5, 5, 5) for Meta-World and CALVIN, with domain- and suite-specific settings for LIBERO.Native action dimensions are 4 for Meta-World and 7 for LIBERO and CALVIN.Source: A. Implementation Details · p. 10

Teacher-rollout implementation and collection budgets

  • Teacher-rollout SFT uses frozen teachers for environment trajectories and flow states, while student-rollout methods use larger microbatches; each method collects 15,360 episodes over 80 updates.Total control steps are 4,575,796 for Action Loss OPD, 4,295,699 for Velocity Loss OPD, and 3,545,208 for Teacher-rollout SFT.Source: A. Implementation Details · p. 10

B. Expanded Evaluation Results

The expanded evaluation describes round-80 endpoint comparisons and checkpoint-level results, alongside training-loss and SFT-initialization figures. In LIBERO, Velocity Loss OPD leads on three suites at round 80, while Action Loss OPD leads on Spatial.

B. Expanded Evaluation Results

  • Table 3 reports checkpoint evaluations through round 80, with endpoint comparisons at round 80; CALVIN entries give average completed length and Len-5 success rate.LIBERO and Meta-World MT50 entries are success percentages; round 0 repeats the common initialization.Source: Table 3 · p. 11, B. Expanded Evaluation Results · p. 11
  • At round 80, Velocity Loss OPD leads on LIBERO Object, Goal, and LIBERO-10, while Action Loss OPD scores higher on Spatial.Figure 3 plots success rates over 80 rounds, with 500 episodes per suite at each checkpoint and the same measured initialization for all methods.Source: Figure 3 · p. 11
  • Figure 4 plots domain-average and weighted total distillation losses for Action Loss OPD, Velocity Loss OPD, and Teacher-rollout SFT, with separate vertical scales for each objective and supervision distribution.The velocity objectives use student- and teacher-generated inputs, respectively, while Action Loss OPD matches final actions.Source: Figure 4 · p. 12
  • Figure 5 tracks conditional flow-matching losses over 14,000 SFT optimizer updates through the checkpoint used as the shared initialization, with equal domain weights.Colors identify LIBERO, Meta-World, and CALVIN; prominent curves show trailing 100-update means.Source: Figure 5 · p. 12
  • Evaluation figures and Table 3 are generated from checkpoint-level evaluation records, while training-loss figures use logged metrics and SFT TensorBoard events.Evaluation curves include every reported checkpoint and share the measured round-0 initialization.Source: B. Expanded Evaluation Results · p. 12

C. Sensitivity to Domain-Loss Weights

The comparison tests two domain-loss weight settings and finds that the main weights yield higher aggregate scores, while equal weighting improves two LIBERO suites. The trade-off varies across domains and task families.

C. Sensitivity to Domain-Loss Weights

  • The comparison uses weights of (1/4, 1/4, 1/2) and (1/3, 1/3, 1/3) for LIBERO, Meta-World, and CALVIN, respectively, across the first 80 updates with seed 42.The runs share initialization, teachers, normalization, objective, optimizer, trainable parameters, and collection protocol; evaluation occurs every 10 rounds.Source: C. Sensitivity to Domain-Loss Weights · p. 13
  • Equal weighting is lower by 7.65 percentage points on LIBERO, 3.60 points on Meta-World, and 0.225 instructions on CALVIN, with a 7.70-point reduction in Len-5 success.Table 4 compares the two Velocity Loss OPD weight settings at round 80.Source: C. Sensitivity to Domain-Loss Weights · p. 13, Table 4 · p. 13
  • Equal weighting improves Spatial and Goal to 96.2% and 69.4%, while Object falls from 98.4% to 89.2% and LIBERO-10 from 46.4% to 17.4%.Source: C. Sensitivity to Domain-Loss Weights · p. 13
  • At intermediate checkpoints, the main weights yield higher aggregate scores, while equal weighting improves two LIBERO suites.Figure 6 shows changes in the ordering across checkpoints and illustrates how domain weighting changes transferred capability across task families.Source: C. Sensitivity to Domain-Loss Weights · p. 13, Figure 6 · p. 13
  • Each setting uses one training seed, and Figure 6 evaluates 2,000 LIBERO episodes, 500 Meta-World episodes, and 1,000 CALVIN sequences at each checkpoint.Equal weighting collects 15,360 episodes and 4,391,623 environment control steps within this horizon; counts vary with policy behavior.Source: Figure 6 · p. 13, C. Sensitivity to Domain-Loss Weights · p. 13

Related papers

Loading 2610.21522v1…