Source-linked AI summary
Flux-OPD: On-Policy Distillation with Evolving Contexts
Yuran Wang, Zekun Wang, Bohan Zeng, Ruixu Zhang, Wenxuan Liu, Liu Yang, Yifan Dai, Yang Shi, Bozhou Li, Chengzhuo Tong, Daili Hua, Yuanxing Zhang, Wentao Zhang
TL;DR
Open-ended LLM training lacks effective supervision for task preferences, while evolving contexts can destabilize distillation through shifting and conflicting targets. Flux-OPD decomposes the reverse-KL objective and stabilizes context-based on-policy distillation with contextual corrections weighted by teacher conflict. On open-ended tasks, it achieves the best total scores across evaluated student-teacher settings and maintains steadier training than OEL.
Problem
Open-ended domains lack effective supervision for complex task preferences, and evolving contexts can create unstable and conflicting distillation targets.
Method
Flux-OPD decomposes reverse KL into distillation and conflict terms, then anchors contextual corrections to a context-free teacher and weights them by teacher conflict.
Results
Flux-OPD achieves the best total scores across student-teacher settings on two open-ended tasks, with steadier loss than OEL and higher out-of-distribution prompt-level strict accuracy than OPD.
Takeaways & Limitations
The results highlight the potential of combining teacher supervision with evolving contexts for open-ended task-preference learning.
Abstract
from arXiv · showhide
Large language model training in open-ended domains lacks verifiable rewards, making task preferences difficult to formalize as effective supervision. Contexts can convey such preferences, yet provide little additional supervision once distilled into the student, motivating contexts that evolve with student performance. However, directly using evolving contexts as in-training supervision results in an unstable distillation target and conflicting distributions, requiring mechanisms to stabilize target and downweight conflicts. In this paper, we analyze the effect of contexts through a decomposition of the reverse KL objective, revealing two findings: the student is distilled toward the geometric mean of context-conditioned teachers, and the objective contains a conflict term that measures conflicts among these teachers. Based on this decomposition, we propose Flux-OPD, an OPD paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains. Flux-OPD treats the differences between context-conditioned and context-free teachers as contextual difference signals, injects them as contextual corrections into the context-free teacher anchor, and weights their correction strength using the conflict term as an indicator. Experiments on open-ended tasks show that Flux-OPD outperforms existing OPD paradigms, highlighting the potential to combine teacher supervision with evolving contexts.
1. Introduction
Open-ended domains lack verifiable rewards and dense supervision for complex task preferences, motivating evolving contexts as in-training signals. Flux-OPD stabilizes this approach through reverse-KL analysis, contextual corrections, and conflict-based weighting, and outperforms existing OPD paradigms on two open-ended tasks.
- Motivation: Open-ended domains often lack verifiable rewards, making it difficult to formalize supervision that captures complex preference priors.Rubrics-as-Rewards provides supervision but remains limited by sparse feedback.
- Motivation: Context distillation uses context-conditioned teachers to provide dense supervision while directly teaching behaviors expressed after observing contexts.This allows explicit contexts to convey task preferences that are difficult to formalize in open-ended domains.
- Challenge: Evolving contexts can abruptly shift target distributions, causing loss surges, training instability, and conflicts among contexts that produce conflicting targets.These issues arise when evolving contexts are used for in-training supervision throughout a single training run.
- Analysis: Reverse-KL analysis decomposes the context-distillation objective into distillation toward the geometric mean of context-conditioned teachers and a term measuring conflicts among them.The decomposition identifies both the effective target distribution and the conflicts that complicate contextual supervision.
- Flux-OPD: Flux-OPD iteratively extracts contexts from student trajectories and distills them during one training run to capture task preferences as in-training supervision.Each iteration comprises context extraction followed by context distillation.
- Flux-OPD: Contextual correction anchors targets to a stable context-free teacher and injects contextual difference signals instead of directly using context-conditioned teachers as targets.The signals are defined as differences between context-conditioned and context-free teachers; contextual weighting uses conflict information to adjust correction strength.
- Results: Flux-OPD outperforms existing OPD paradigms on prompt optimization for video generation and medical question answering across different student-teacher settings.Both tasks involve complex preferences and rely on rich contextual information.
2. Related Works
Related work addresses weak supervision in open-ended domains through structured rewards, dense on-policy teacher supervision, and context distillation. However, existing approaches face sparse feedback or limitations from fixed contexts and distribution mismatch.
- Reward and distillation supervision: RLVR compresses task preferences into coarse outcomes, while RaR provides structured rubric supervision but still yields sparse feedback.These approaches target the difficulty of formalizing preferences when open-ended tasks lack verifiable rewards.
- Reward and distillation supervision: OPD trains on trajectories sampled from the current student policy and provides dense teacher supervision.
- Context distillation: Context Distillation transfers behaviors from a context-conditioned teacher, while OPCD samples from the student to reduce offline-data and generation-distribution mismatch.
- Context distillation: Although contexts convey difficult-to-formalize task preferences, contexts fixed before training cannot adapt to the evolving student.
3. Preliminaries
This section defines vanilla OPD and OPCD, then shows that reverse-KL distillation targets the normalized geometric mean of context-conditioned teachers and separates teacher disagreement into a conflict term. The conflict term affects the objective but has no direct student-gradient contribution, clarifying instability under evolving contexts.
- On-Policy Distillation: Vanilla OPD trains the student on-policy to match a context-free teacher distribution at each decoding history.The student autoregressively generates responses, while the teacher distribution is defined over vocabulary tokens conditioned on the decoding history.
- On-Policy Context Distillation: OPCD gives the teacher a privileged context while the student observes only the decoding history, and minimizes reverse KL on on-policy histories.This creates context-conditioned teacher distributions that provide supervision unavailable directly to the student.
- Forward KL: Under forward KL, the student is distilled toward the arithmetic mean of context-conditioned teachers.Prior work decomposes the forward-KL objective into a distillation term toward a marginal teacher and a mutual-information term.
- Reverse KL: Under reverse KL, the objective decomposes into distillation toward the normalized geometric mean of context-conditioned teachers plus a nonnegative conflict term −log Z.The geometric mean favors tokens assigned consistently high probability across contexts, while lower teacher overlap increases −log Z.
- Reverse KL: For fixed histories, context distribution, and context-conditioned teachers, −log Z is independent of the student distribution and contributes no direct gradient.Only the distillation term is directly optimized, with the geometric mean serving as the distillation target.
- Evolving Contexts: With evolving contexts, context-conditioned teachers shift substantially while the context-free teacher remains stable, and conflict variations reflect sampled histories and context pools rather than direct conflict optimization.These observations motivate treating evolving contexts as a source of target instability and distinguishing their effects from direct student optimization.
4. Method
Flux-OPD uses evolving contexts as in-training supervision in an iterative process that alternates context extraction and context distillation. It stabilizes the evolving target through contextual correction anchored to the context-free teacher and contextual weighting based on teacher-distribution conflict.
- Iterative Context Evolution: Flux-OPD divides training into K iterations that extract contexts from student trajectories and distill the teacher conditioned on those contexts.Contexts evolve with student performance, using a small set of trajectories at each update to provide timely feedback.
- Iterative Context Evolution: Context extraction accumulates new experience items from student interaction trajectories with previous items to form a context pool.Trajectories include student responses and optional task-specific supplements, and the process is repeated for M turns with different random seeds.
- Contextual Correction: Contextual correction anchors the target to the context-free teacher q0 and injects contextual difference signal ∆k through log-space interpolation toward qgeo,k.λk ∈[0, 1] controls correction strength: λk = 0 yields q0, while λk = 1 recovers qgeo,k.
- Contextual Weighting: Contextual weighting adjusts correction strength using the conflict term, which measures distributional disagreement among context-conditioned teachers.Corrections strengthen when teacher distributions are consistent and weaken when they conflict through a function that decreases monotonically with δk.
- Contextual Weighting: The weighting mechanism calibrates correction strengths through scaling or clipping, with α, τ, λmin, and λmax controlling the resulting bounds.Without strength calibration, α = 1, λmin = 0 and λmax = 1; the same weight applies to all vocabulary logits at each decoding position.
5. Experiments
Experiments evaluate Flux-OPD across multiple student–teacher settings on prompt optimization and medical question answering, using downstream video and clinical benchmarks. Flux-OPD achieves the strongest overall results and shows advantages in stability, generalization, and its evolving-context, correction, and weighting components.
- Experimental Setup: Experiments use three student–teacher configurations spanning Qwen3-VL-Instruct, Qwen2.5-VL-Instruct, and Qwen3 models.Students are initialized from offline teacher-distilled SFT models, while teachers remain base models.
- Tasks and Evaluation: The evaluation covers prompt optimization for video generation and medical question answering, assessed with VBench, Video-Bench, and HealthBench.The tasks target complex preferences and contextual information; HealthBench contains 5K clinical conversations.
- Main Results: Flux-OPD achieves the best total scores across student–teacher settings and both open-ended tasks, consistently outperforming OPD across three independent runs.OPD improves over the initial policy in medical question answering but degrades it in prompt optimization, where video-level feedback reflects task preferences.
- Ablation Studies: Evolving contexts, contextual correction, and conflict-based contextual weighting each improve performance over their corresponding ablations.Full Flux-OPD outperforms the no-evolving-context variant, OEL, and V2 with each of three static correction strengths.
- Stability and Generalization: Flux-OPD decreases training loss steadily, maintains stable gradient norms, and achieves higher prompt-level strict accuracy than OPD on out-of-distribution IF-Eval.OEL instead exhibits loss surges and larger gradient fluctuations near context updates.
6. Conclusion
Flux-OPD uses evolving contexts as in-training supervision to capture task preferences in open-ended domains. It stabilizes distillation by anchoring to a context-free teacher, injecting contextual difference signals, and weighting them by teacher conflicts.
- Flux-OPD uses evolving contexts as in-training supervision to capture task preferences in open-ended domains.
- The method anchors distillation to the context-free teacher, injects contextual difference signals, and weights them according to conflicts among teachers.These mechanisms address unstable distillation targets and conflicting distributions.
Appendix · A. Prompt Template · B. Context Example
The appendix documents concise prompt templates, context-extraction and context-distillation prompts, system-prompt usage, and example contexts for OPCD, OEL, and Flux-OPD across two tasks.
- A. Prompt Template: The prompts are intentionally concise and require minimal human engineering, without human-crafted domain knowledge.This design lets models rely on their own capabilities.
- Appendix: Together, the appendix materials cover prompt design, context extraction, context-conditioned distillation, system-prompt configuration, and concrete context examples.The materials span OPCD, OEL, and Flux-OPD across prompt optimization and medical question answering.
- A.1. Context Extraction Prompt: Context-extraction prompts for OPCD, OEL, and Flux-OPD extract new experience items beyond existing ones.They are used for prompt optimization and medical question answering.
- A.2. Context Distillation Prompt: Context-conditioned teachers in OPCD, OEL, and Flux-OPD use in-context learning prompts for prompt optimization and medical question answering.The appendix presents these prompts in Figs. 9 and 10.
- A.3. System Prompt: A system prompt is used for offline SFT data construction, student training, and evaluation in the prompt optimization task.No explicit system prompt is used for medical question answering.
C. Additional Training Details · C.1. Hyperparameters
The paper specifies shared training settings across paradigms and implementation choices for efficient distillation. Flux-OPD’s dedicated hyperparameters are listed in Tab. 7, with evaluation scores reported in Tabs. 1–3.
- C.1. Hyperparameters: All training paradigms use a batch size of 64.
- C.1. Hyperparameters: All training paradigms use a learning rate of 2 × 10^-6.
- C.1. Hyperparameters: SFT runs for 5 epochs, whereas the other paradigms run for 10 epochs.
- C.1. Hyperparameters: The implemented distillation loss retains the top-64 target logits and renormalizes them over the selected tokens.
- C.1. Hyperparameters: All experiments are conducted on 8 NVIDIA A800 GPUs.
- C.1. Hyperparameters: Flux-OPD’s hyperparameters are listed in Tab. 7, while corresponding evaluation scores appear in Tabs. 1, 2, and 3 of the main paper.
C.2. Training Pseudocode of Flux-OPD
Flux-OPD uses a single continuous training run in which contexts are extracted from current-student trajectories and distilled as contextual difference signals into a context-free teacher anchor. Correction strength is adjusted according to conflict among context-conditioned teacher distributions.
- Flux-OPD performs a single continuous training run over multiple iterations.
- Each iteration extracts contexts from trajectories generated by the current student.
- Contextual difference signals are distilled into a context-free teacher anchor, with correction strength adjusted according to teacher-distribution conflict.
C.3. Conflict Dynamics
Conflict measures disagreement among context-conditioned teachers, with higher values indicating lower distributional overlap. Conflict is higher for Qwen3-VL-Instruct 8B prompt optimization than for Qwen2.5-VL-Instruct 32B settings, while rubric supplements can improve consistency.
- Conflict measurement: Conflict monitors distributional disagreement by combining context-conditioned token distributions through a geometric mean in log-probability space.A larger conflict value indicates lower distributional overlap and stronger disagreement among sampled context-conditioned teachers.
- Teacher effects: Qwen3-VL-Instruct 8B prompt optimization settings show substantially higher conflict than Qwen2.5-VL-Instruct 32B settings, indicating lower context consistency.The 32B teacher maintains markedly lower conflict values, consistent with more stable token preferences across sampled contexts.
- Context extraction: The medical question answering setting has low conflict comparable to the 32B prompt optimization settings despite using an 8B teacher.Rubrics used as optional supplements during context extraction may provide a standard reference and improve context consistency.
- Context consistency: Overall context consistency depends on both teacher capability and the optional supplements available during context extraction.This conclusion follows from the contrast between teacher sizes and the rubric-supported medical question answering setting.
D. Additional Evaluation Details
Additional evaluation details cover prompt optimization across VBench dimensions and specify Flux-OPD’s iterative context extraction and distillation procedure. The appendix also reports task-dependent conflict dynamics across teacher sizes and domains.
- Training configuration: Flux-OPD requires training prompts, a student, frozen teacher, environment, context prompts, iterations, rollouts, extraction seeds, distillation contexts, and conflict-control hyperparameters.The specified controls include scaling factor α, conflict threshold τ, clipping bounds [λmin, λmax], and top-B size.
- Context extraction: Each training iteration extracts contexts sequentially from rollouts with multiple seeds, forming a context pool before context distillation.The algorithm constructs Ck = {ck,1, . . . , ck,M} after extracting contexts from the rollout set.
- Context distillation: For each mini-batch, Flux-OPD computes a context-free teacher distribution and multiple context-conditioned teacher distributions before constructing and filtering the distillation target.The procedure retains and renormalizes the top-B target probabilities, then updates the student parameters.
- Conflict dynamics: Prompt optimization with the 8B teacher exhibits higher conflict, while the 32B teacher shows higher overall context consistency; medical question answering maintains low conflict.The passage attributes the medical result potentially to task-specific rubrics serving as optional supplements during context extraction.