PaperlayerView exact revision on arXiv

Source-linked AI summary

CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models

Pengyang Ling, Jiazi Bu, Yujie Zhou, Yibin Wang, Zeqiang Lai, Xiaoxiao Ma, Yi Jin, Huaian Chen, Yuhang Zang

arXiv:2609.34658v2cs.CV

TL;DR

Existing multi-teacher OPD routes prompts by semantic category, binding capability control to wording and limiting inference-time adjustment. CapField-OPD uses explicit coordinates to combine teacher anchors into a continuous field. The student supports specialized and joint behaviors, prompt rewriting robustness, and coordinate-based scaling.

  • Problem

    Existing multi-teacher OPD uses semantic routing that binds capability intent to prompt wording and prevents users from adjusting capability strength at inference time.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    CapField-OPD conditions a student on explicit capability coordinates and uses base, specialized, and joint teachers as anchors for a continuous capability field.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    CapField-OPD supports direct switching between specialized and joint behaviors, matches or exceeds teacher performance on target tasks, and keeps requested capabilities active under prompt rewriting.

    Source: §1 · p. 2, §4.1 · pp. 6–7, §5 · p. 9

  • Takeaways & Limitations

    Calibration identifies a highest-mean-reward default coordinate and frequently optimal coordinates as candidates for prompt-specific test-time search.

    Source: §1 · p. 2

    The student may inherit reward-hacking behaviors from teachers trained during the RL stage.

    Source: §4.6 · p. 9

Abstract

from arXiv · show

Reward-specialized post-training produces strong experts for flow-based generative models, while multi-teacher on-policy distillation (OPD) consolidates their capabilities into a single student. Existing methods, however, route each prompt to a single teacher according to its semantic category, implicitly binding the desired capability to prompt content. This coupling makes capability invocation vulnerable to prompt perturbations and prevents users from explicitly adjusting the strength of the desired capability at inference time. In this work, we introduce CapField-OPD, an OPD framework that integrates multiple teachers into a continuous capability field through explicit capability coordinates. We use teacher models as anchors to construct this field, with the coordinates determining how their outputs are combined. Each capability configuration thus receives a unique supervision target, and capability control no longer depends on prompt semantics. Since the training anchors may not be optimal at inference time, we further profile the learned field on a small calibration set. The coordinate with the highest mean reward serves as the recommended default, while coordinates that are frequently optimal offer a promising candidate set for test-time scaling. Extensive experiments on compositional generation, text rendering, and visual aesthetics demonstrate that CapField-OPD consolidates multiple specialized teachers into a single student while preserving or surpassing their performance, reliably invokes the desired capabilities under semantics-preserving prompt variations, and supports continuous capability control and coordinate-based test-time scaling.

1 INTRODUCTION

Existing multi-teacher OPD binds capability intent to prompt semantics, limiting explicit control and creating ambiguity when prompts require multiple capabilities. CapField-OPD uses explicit coordinates and joint-anchored teacher supervision to form a continuous capability field.

1 INTRODUCTION

  • Semantic hard routing binds capability intent to prompt content, making capability activation sensitive to wording and leaving users unable to control capabilities at inference time.When prompts require multiple capabilities, activating multiple teachers can also produce conflicting velocity targets under the same student condition.Source: §1 · pp. 1–2
  • CapField-OPD conditions one student on explicit capability coordinates and combines base, single-capability, and joint-capability teachers as anchors of a continuous field.Joint-teacher outputs supply shared capability components, while single-capability teachers cover remaining activation.Source: §1 · p. 2
  • Profiling in-range and extrapolative coordinates on a small calibration set identifies a highest-mean-reward default and candidates for prompt-specific test-time search.Teacher anchors serve as reference states rather than hard performance limits.Source: §1 · p. 2
  • Experiments show the framework consolidates reward-specialized experts, improves robustness to prompt perturbations, and supports continuous control and coordinate-based test-time scaling.The stated contributions include direct invocation of desired capabilities despite semantic changes to prompts.Source: §1 · p. 2

2 RELATED WORK

Preference alignment methods adapt flow models to downstream objectives through offline preference learning or online reward optimization. On-policy distillation consolidates teacher capabilities, while existing multi-teacher methods still rely on semantic routing.

2.1 PREFERENCE ALIGNMENT IN FLOW MODELS

  • Offline DPO-style methods turn preference comparisons into flow-matching objectives, while online methods optimize sampled trajectories using rewards and policy updates.DiffusionNFT and Advantage Weighted Matching instead re-noise scored generated images for reward-derived weighted optimization.Source: §2.1 · p. 3

2.2 ON-POLICY DISTILLATION IN FLOW MODELS

  • On-policy distillation samples student trajectories and queries teachers at student-visited states, providing dense velocity supervision while reducing exposure bias.Flow-OPD and DiffusionOPD use task routing and trajectory-level matching; existing multi-teacher OPD still entangles capability intent with prompt semantics.Source: §2.2 · p. 3

3 METHOD

The method moves from conventional flow matching and on-policy distillation to explicit capability coordinates that define continuous, teacher-anchored supervision. It profiles the learned field to recommend coordinates and uses reward-weighted coordinate search to select outputs for a fixed initial noise.

3.1 PRELIMINARIES

  • Flow matching trains a time-dependent velocity field to transport Gaussian noise toward clean data, while OPD matches teacher behavior on student-visited states.The flow model may also be conditioned on text or image; when student and teacher share per-step covariance, velocity matching is equivalent to minimizing per-step KL divergence up to a time-dependent coefficient.Source: §3.1 · pp. 3–4

3.2 OBSERVATION

  • Task-based routing hides the selected teacher from the student, which must infer the intended capability from prompt semantics.This entangles capability activation with prompt content.Source: §3.2 · p. 4

3.3 JOINT-ANCHORED EXPLICIT CAPABILITY FIELD

  • CapField-OPD defines a continuous field from base, single-capability, and joint teachers, with coordinate-dependent weights assigning each capability configuration its own target.For supported pairs, overlap between two capability coordinates is assigned to the joint teacher; pairs without a joint teacher use axis-aligned coordinates only.Source: §3.3 · pp. 4–5

3.4 CAPABILITY LANDSCAPE PROFILING

  • Capability landscape profiling evaluates coordinates on shared prompt–noise pairs, tracking both mean reward and how often each coordinate is optimal.These statistics differ: a frequently optimal coordinate may perform moderately on other pairs, while a high-mean-reward coordinate may rarely be best.Source: §3.4 · p. 5

3.5 TEST-TIME SCALING OVER CAPABILITY COORDINATES

  • Test-time scaling samples coordinates from the profiling-derived candidate set and selects the highest-reward output using the same initial noise for every candidate.This explores capability configurations without resampling random seeds.Source: §3.5 · p. 6

4 EXPERIMENTS

Experiments evaluate compositional generation, text rendering, and visual aesthetics using single- and joint-capability teachers. They compare CapField-OPD with other unified methods and examine prompt robustness, continuous control, test-time scaling, ablations, and training limitations.

4.1 MAIN RESULTS

  • Single-capability coordinates achieve the best GenEval and OCR scores overall and slightly exceed the aesthetic teacher on all three aesthetic metrics; joint coordinates offer balanced operating points with strong task accuracy.Both modes use the same student and differ only in inference coordinates; some single-capability coordinates lie beyond the training anchors.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 ROBUSTNESS TO PROMPT REWRITING

  • Under semantics-preserving prompt rewriting, CapField-OPD achieves the best result on all three metrics, while compared OPD methods degrade significantly, especially on GenEval and OCR.The evaluation rewrites prompts without retraining; CapField-OPD invokes capabilities through coordinates that remain unchanged with wording.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 CONTINUOUS CAPABILITY CONTROL

  • With prompts and initial noise fixed, sweeping capability coordinates produces smooth transitions in task emphasis while keeping subjects and overall content recognizable.Moving toward aesthetics enriches scene details or changes lighting and style, while text fidelity is strongest near the OCR endpoint.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 TEST-TIME SCALING OVER CAPABILITY COORDINATES

  • At matched budgets N ∈ {3, 6, 9}, coordinate-based scaling matches seed-based scaling across three reward objectives, with a difference of about 0.5%.It fixes noise and searches coordinates, correcting capability defects while largely preserving the scene; the authors favor it when the initial generation is satisfactory and only minor defects remain.Source: §4.4 · p. 8, Figure 5 · p. 9

4.5 ABLATION STUDIES

  • Explicit coordinates let one student support specialized and joint behaviors, addressing the ambiguity of multiple plausible teacher targets for one prompt.Without a capability signal, adding joint teachers to DiffusionOPD and DanceOPD improves aesthetic metrics but reduces task accuracy.Source: §4.5 · p. 8
  • Under joint activation, adding joint teachers raises GenEval from 0.7723 to 0.9322 and OCR from 0.8716 to 0.9256, while improving all three aesthetic metrics for both capability pairs.The results indicate that single-capability anchors alone do not fully determine behavior at joint coordinates.Source: §4.5 · p. 8, Table 3 · p. 9
  • Expanded profiling improves all metrics over in-range profiling, selecting coordinates beyond the training range.The authors attribute this to continuity in the learned field, making training anchors reference states rather than necessarily optimal inference settings.Source: §4.5 · p. 9

4.6 LIMITATION

  • Compared with DiffusionOPD, CapField-OPD increases per-step training time from 11 s to 12 s, about 9%; students may also inherit reward-hacking behavior from teachers.The additional time comes from frozen-teacher forward passes for the joint-anchored target; teacher outputs require no gradients and do not participate in rollout.Source: §4.6 · p. 9

5 CONCLUSION

CapField-OPD learns a continuous capability field using explicit coordinates and reward-specialized teachers as anchors. The conclusion reports that capabilities remain active under prompt rewriting and may extend beyond the training anchors.

5 CONCLUSION

  • CapField-OPD conditions the student on explicit capability coordinates and uses reward-specialized teachers as anchors for a continuous capability field.The coordinates decouple capability control from prompt semantics.Source: §5 · p. 9
  • The student matches or exceeds every anchor on its target task, and requested capabilities stay active under prompt rewriting.The coordinates can extend beyond the training anchors, which serve as reference states rather than hard performance limits.Source: §5 · p. 9
  • Explicit capability coordinates offer a way to specify generative-model behavior through user-selected coordinates rather than prompt inference.The conclusion frames this as a broader direction for user-specified generative-model behavior.Source: §5 · p. 9

A APPENDIX

The appendix collects training pseudocode, hyperparameters and capability curves, and further visualizations of coordinate-based strength control.

A APPENDIX

  • The appendix provides CapField-OPD training pseudocode.It is presented in Sec. A.1.Source: A APPENDIX · p. 13
  • The appendix includes the full hyperparameter configuration and capability curves during distillation.These materials are listed in Sec. A.2.Source: A APPENDIX · p. 13
  • Additional visualizations show coordinate-based strength control.These visualizations are listed in Sec. A.3.Source: A APPENDIX · p. 13

A.1 PSEUDOCODE

CapField-OPD freezes reward-post-trained single and joint teachers and uses them to supervise the student at student-visited states. Prompt-matched capability coordinates govern rollout and loss supervision.

A.1 PSEUDOCODE

  • Reward-post-trained single-capability and joint teachers are frozen and supervise the student at student-visited states.For prompts without a joint teacher, rollouts and supervised coordinates are restricted to axis-aligned ones.Source: §A.1 · p. 13
  • Each prompt is paired with matching capability coordinates; rollout and OPD-loss computation can use different coordinates.A single rollout can supervise many coordinates, so rollout cost does not scale with the number of supervised coordinates.Source: §A.1 · p. 13
  • Training initializes the student from the base teacher with a trainable LoRA and coordinate conditioner, then updates them using mean OPD loss.The algorithm samples a balanced prompt batch and rolls out the student using a coordinate drawn from that prompt's matching set.Source: §A.1 · p. 13

A.2 HYPERPARAMETER CONFIGURATION AND CAPABILITY CURVES

The appendix lists training hyperparameters and reports validation-reward curves showing steady improvement across capabilities during distillation.

A.2 HYPERPARAMETER CONFIGURATION AND CAPABILITY CURVES

  • All capabilities improve steadily during distillation, according to validation-reward curves evaluated every 200 steps on prompts from each capability set.The curves are shown in Fig. 6.Source: §A.2 · p. 13, Figure 6 · p. 14
  • Table 5 lists the hyperparameter settings used for CapField-OPD training.The supplied table passage identifies its scope as CapField-OPD hyperparameters.Source: §A.2 · p. 13, Table 5 · p. 13
  • Fig. 6 presents capability curves during CapField-OPD training.The curves track validation reward over distillation steps.Source: §A.2 · p. 13, Figure 6 · p. 14

A.3 ADDITIONAL VISUALIZATIONS OF CAPABILITY CONTROL

The additional coordinate sweeps visualize inference-time capability control: outputs shift smoothly along a capability axis while composition stays close to the shared starting point.

A.3 ADDITIONAL VISUALIZATIONS OF CAPABILITY CONTROL

  • Sweeping coordinates moves outputs smoothly along the capability axis while keeping composition close to the shared starting point.The reported behavior holds across the supported capability pairs and seeds.Source: §A.3 · p. 14
  • Within each row, fixing the prompt and initial noise while varying coordinates makes output differences attributable to the capability field rather than the seed.Source: §A.3 · p. 14
  • The coordinate sweeps cover the GenEval–aesthetics and OCR–aesthetics capability pairs.Source: Figure 7 · p. 15, Figure 8 · p. 16

Related papers

Loading 2609.34658v2…