PaperlayerView exact revision on arXiv

Source-linked AI summary

DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

Zihan Li, Feifei Li, Wenhui Que

arXiv:2608.14644v1cs.LGcs.CL

TL;DR

Runtime-injected prohibitions require models to follow configurable, token-localized rules without sacrificing legitimate-query helpfulness, a setting conventional post-training handles poorly. DUET uses same-weight teachers with differing prohibition visibility to learn from token-level disagreement, and across Qwen2.5 and Qwen3 at 1.5B–8B it improves violation refusal and helpfulness without catastrophic forgetting on general-capability benchmarks.

  • Problem

    Runtime-injected prohibitions create a need for token-localized compliance that preserves helpfulness when rules change or are absent.

  • Method

    DUET uses same-weight positive and negative teachers whose per-token disagreement selects informative positions and defines a token-level push/pull learning direction.

  • Results

    Across Qwen2.5 and Qwen3 at 1.5B–8B, DUET consistently improves violation refusal and legitimate-query helpfulness with no catastrophic forgetting on general-capability benchmarks.

  • Takeaways & Limitations

    The findings support same-weight teacher disagreement as a unified mechanism for improving prohibition compliance while retaining legitimate-query helpfulness.

Abstract

from arXiv · show

Real-world LLM deployments increasingly rely on runtime-injected prohibitions--enterprise policies, PII redlines, tool boundaries--that vary per request and per tenant. Conventional post-training is structurally ill-suited: SFT hides the violation signal in compliant labels, and DPO's sequence-level preferences mismatch token-localized violations. We propose DUET, a token-selective on-policy distillation method for prohibition compliance. DUET pairs a teacher that sees the prohibition (positive) with an identical-weight teacher that does not (negative). Because the two teachers differ only in prohibition visibility, their per-token disagreement isolates the prohibition's causal effect--yielding a clean supervision signal uncontaminated by model capacity or mismatch. This disagreement drives two complementary mechanisms: signal cleaning, which discards agreement tokens as redundant or prefix-corrupted, and preference-directed learning, which pushes the student away from the negative teacher and toward the positive one at token granularity, embedding DPO-style optimization directly into OPD without offline preference data. We construct an industrial Prohibition-Compliance benchmark spanning five task families covering explicit-refusal, paraphrase robustness, and over-refusal. Across 1.5B-8B Qwen variants, DUET achieves 72.3-85.2% violation compliance while preserving 88-93% normal utility, dramatically outperforming teacher model and other distillation baselines. External evaluation on SysBench confirms improved safety alignment with minimal degradation on GSM8K and MATH-500.

1 Introduction

Runtime-injected prohibitions require models to suppress specific violating tokens while preserving helpfulness, exposing structural limitations in conventional post-training and vanilla OPD. DUET addresses these gaps with same-weight teachers whose prohibition-conditioned disagreement selects prohibition-relevant supervision and improves refusal compliance without catastrophic forgetting.

  • Motivation: Runtime-injected prohibitions vary across requests and tenants, creating costly risks from data leaks, out-of-policy tool use, and brand-safety incidents.The model must still answer the underlying query faithfully, making over-refusal a joint compliance-utility failure.
  • Limitations of Existing Methods: SFT and RLHF lack an explicit loss channel for which violating token must be suppressed at which position.SFT cannot treat the violating token itself as a negative supervisory signal.
  • Limitations of Existing Methods: Vanilla OPD follows student failure trajectories but can become unreliable when teacher conditioning follows the student toward a violation.This creates a structural gap in using direct teacher supervision for prohibition compliance.
  • DUET: DUET pairs same-weight teachers differing only in prohibition visibility and uses their per-token disagreement for violation-token selection.Agreement positions are discarded because they contain no prohibition-relevant information and may provide redundant supervision.
  • Results: DUET consistently outperforms existing distillation baselines on explicit-violation refusal and legitimate-query helpfulness across Qwen2.5 and Qwen3 scales from 1.5B to 8B.The framework is reported to avoid catastrophic forgetting on general-capability benchmarks.

2 Related Work

Prior work advances on-policy distillation through sampling-policy and divergence choices but does not dynamically screen teacher feedback or isolate task-relevant error positions. Existing constraint-following benchmarks also separate soft instruction following from safety refusal, whereas this work jointly evaluates industrial prohibition compliance and utility preservation.

  • On-policy distillation: On-policy distillation research explores student rollouts with teacher feedback and alternative divergences, including reverse KL in MiniLLM to reduce emphasis on the teacher’s low-probability tail.GKD trains on student-generated sequences with divergences beyond forward KL.
  • On-policy distillation: Existing on-policy distillation does not dynamically monitor potentially erroneous teacher feedback or isolate task-relevant error-prone positions from teacher–student distribution mismatch.DUET uses disagreement between weight-shared positive and negative teachers that differ only in task-relevant context as its screening criterion.
  • Constraint-following benchmarks: Constraint-following benchmarks largely target either verifiable soft instructions or refusal behavior under harmful or under-specified queries, with few outright prohibitions or embedded benign counterparts.IFEval, FollowBench, and FoFo emphasize structural or stylistic requests, while SORRY-Bench and CoCoNot focus on safety-oriented refusal behavior.
  • Constraint-following benchmarks: The proposed benchmark covers industrial prohibition-compliance scenarios while jointly measuring explicit-violation refusal, paraphrase robustness, and over-refusal risk on legitimate queries in the same business context.It addresses the gap in jointly evaluating prohibition compliance and utility preservation under industrial hard constraints.

3 Preliminaries

DUET builds on on-policy distillation, where the student generates rollouts and teachers provide token-wise supervision. Its setup uses identical-weight positive and negative teachers that differ only in prohibition visibility, with aligned response-segment logits and a union-based vocabulary reduction.

  • On-Policy Distillation: On-policy distillation has shifted from teacher-sampled trajectories to student-generated rollouts supervised token-wise by the teacher.The generic OPD objective supports forward KL, reverse KL, or interpolations such as GKD and DistiLLM.
  • Teacher Construction: The positive and negative teachers share identical base weights and differ only in whether the prohibition appears in their system prompts.This isolates prohibition visibility as the defining difference between T+ and T−.
  • Prompt Construction: T+ receives the prohibition and a thinking prefix, T− omits both, and the student uses the T+ prompt without that prefix.The prefix conditions T+ for compliance and balances refusal on violating samples with normal answers on legitimate ones.
  • Token Alignment: For a shared rollout, S, T+, and T− produce token-wise aligned logits, while losses apply only to response tokens in the effective-token set M.Prompt positions and padding are excluded from M.
  • Vocabulary Reduction: At each position, the method unions the separately selected top-KV indices from T+ and T− and assigns −104 to unselected logits.Unless otherwise noted, V denotes the resulting shrunk vocabulary.

4 Method

DUET uses same-weight positive and negative teachers to isolate prohibition-caused per-token disagreement, then applies that signal to clean supervision and direct preference learning. Its filtering retains high-disagreement tokens through KL ranking, top-K selection, and neighborhood dilation.

  • Disagreement Signal: Same-weight teachers differ only in prohibition visibility, so their per-token disagreement isolates the prohibition’s effect from capability and distributional confounders.Agreement positions are excluded because they provide redundant or potentially misleading supervision.
  • Violation-Token Selection: DUET discards teacher-agreement tokens because they reinforce prohibition-irrelevant next-token statistics or may reflect erroneous student-prefix pollution.These two failure modes motivate excluding agreement positions from the loss.
  • Violation-Token Selection: KL disagreement identifies positions worth supervising, providing the token-level basis for DUET’s violation-token selection.The method defines disagreement for each position before applying sequence-level retention and dilation.
  • Violation-Token Selection: Per sequence, DUET retains K_b = ⌈0.1 · |M_b|⌉ highest-disagreement positions, preventing long-sequence dominance and short-sequence starvation.This top-K mask is followed by max-pool dilation with kernel 5, covering neighboring sub-word tokens that may belong to violations.
  • Preference-direction Learning: At selected high-disagreement positions, the student is pulled toward T+ and pushed away from T−, embedding online token-level DPO-style optimization into OPD.The approach requires no offline preference pairs; Lpos uses p+ as the target distribution.

Pushing away from T −.

DUET purifies the positive teacher’s target by extrapolating away from the negative teacher along the disagreement direction, then combines preference-directed losses with top-K utility preservation. The resulting objective implements token-level DPO-style optimization on high-disagreement tokens while limiting over-refusal on legitimate queries.

  • Synthesized target: DUET extrapolates zT + away from zT − along the disagreement direction to synthesize a purer target.This removes residual T −-direction contamination from the positive teacher’s soft target.
  • Synthesized target: A standard forward KD term aligns the student with the synthesized target, refining it along the shared preference axis.The independent positive and negative teacher losses establish the student’s coarse direction before the synthesized dual-teacher loss refines it.
  • Utility preservation: Top-K KD against T + on legitimate queries preserves utility and prevents over-refusal within the same business context.The target retains only T +’s top-Ktk candidates after truncated renormalization, while excluding T − and the V∗ selection.
  • Overall objective: L = λpos Lpos + λneg Lneg + λsyn Lsyn + λutil Lutil combines the three preference-directed terms with utility preservation.The first three terms implement token-granular DPO-style preference optimization on V∗, while Lutil safeguards utility.

5 Benchmark

The paper introduces an industrial Prohibition-Compliance benchmark that jointly evaluates explicit-violation refusal, paraphrase robustness, and over-refusal within one business context. It covers five task families and uses bucket-specific evaluation to measure prohibition satisfaction and substantive responses to legitimate queries.

  • Benchmark motivation: The benchmark addresses a gap by jointly measuring explicit-violation refusal, paraphrase-probe robustness, and over-refusal risk in one business context.Existing Constraint-Following benchmarks did not jointly measure these three dimensions.
  • Benchmark design: The benchmark spans five task families covering PII and memory, safety redlines, custom business content, and additional prohibition-compliance scenarios.The visible task-family description specifies T1–T3 and states that the benchmark contains five families overall.
  • Benchmark design: Each sample records the system role, scenario context, prohibition, user query, and, for some samples, tools and tool-call history.These fields support scenarios involving business text, business-flow SOPs, and user profiles.
  • Benchmark design: Samples are partitioned into induction, boundary, and normal buckets that share system role and context within each family but differ in user query.Induction queries naturally elicit violations without prohibitions, boundary queries are surface-sanitized while retaining leakage potential, and normal queries require no prohibition access.
  • Data scale and validation: 2,500 training samples and 700 evaluation samples are constructed, with 98.4% of manually inspected samples passing human validation.The evaluation set includes 600 induction-plus-boundary samples and 100 normal samples; 200 of the former use prohibitions from a new pool outside the training constraint pool.
  • Evaluation: Evaluation uses bucket-dependent LLM judges: prohibition satisfaction on induction and boundary, and substantive non-refusal on normal queries.The violation judge evaluates the user query, assistant response, and prohibition; the normal judge evaluates the user query and assistant response.

6 Experiments · 6.1 Experimental Setup · 6.2 Prohibition-Compliance Evaluation

The experiments evaluate DUET across Qwen model scales using controlled distillation baselines and an industrial Prohibition-Compliance benchmark. Results show that DUET addresses teacher imbalance, token-localized violation learning, and teacher–student mismatch while improving compliance and utility jointly.

  • Backbones and Scales.: DUET is evaluated across Qwen2.5-7B →Qwen2.5-1.5B, Qwen3-8B →Qwen3-1.7B, and Qwen3-8B →Qwen3-8B self-distillation settings.All LLM-as-a-judge evaluations use DeepSeek-V4-Flash.
  • Baselines.: Comparisons include raw students and teachers, vanilla OPD-F and OPD-R, AlignDistil, and TIP.OPD-F and OPD-R use single-teacher forward and reverse KL, respectively; AlignDistil uses two preference models, while TIP selects tokens using uncertainty and teacher–student signals.
  • Training and Rollout.: Training uses DDP with rank-r = 16 LoRA applied uniformly to attention and MLP projection matrices on a single node with 8×NVIDIA H20 GPUs.Two GPUs handle rollout and six handle policy training.
  • 6.2 Prohibition-Compliance Evaluation: The benchmark compares DUET with representative distillation baselines on the industrial Prohibition-Compliance benchmark.Table 1 covers five task families, with Viol., Util., Over., and Hum. reporting violation refusal, legitimate utility, overall accuracy, and human-rated score.
  • 6.2 Prohibition-Compliance Evaluation: 43.8% vs. 95.0%: the raw Qwen2.5 teacher is much weaker on Viol. than Util., while adding a thinking prefix raises Qwen3 Viol. from 86% to 46% Util.These results indicate that prompt-side constraints do not jointly achieve compliance and utility, motivating weight-level prohibition alignment.
  • 6.2 Prohibition-Compliance Evaluation: 72.3: DUET exceeds Qwen2.5 OPD-F/OPD-R violation-refusal rates of 51.0/54.8, while reaching 83.8 versus 81.3/81.0 for Qwen3-1.7B.The gap reflects DUET’s ability to detect polluted-prefix contamination and focus supervision on decisive compliance tokens.
  • 6.2 Prohibition-Compliance Evaluation: 49.6 vs. 56.1: AlignDistil’s Qwen2.5 Over. trails OPD-F, and 57.7 vs. 81.7 shows the same pattern for Qwen3-1.7B.Sequence-level chosen/rejected labels contaminate neutral positions and dilute gradients at token-localized violations.
  • 6.2 Prohibition-Compliance Evaluation: 59.3%: Qwen2.5’s weaker teacher is easily dragged toward the negative teacher by erroneous prefixes, whereas DUET removes misleading positions through per-token teacher disagreement.The passage contrasts this with Qwen3-8B’s teacher, which already reaches 80.2% Viol.; DUET’s denoising gain is larger with dirtier teachers and larger teacher–student mismatch.

6.3 Constraint-Following Evaluation: SysBench

SysBench evaluates fine-grained system-prompt constraint following using criterion- and turn-level pass rates, without contributing training data. DUET outperforms all distillation baselines on both metrics, indicating task-agnostic prohibition alignment rather than template memorization.

  • Benchmark and metrics: SysBench comprises 500 system prompts spanning six constraint categories and evaluates system-prompt constraint and prohibition following.It is designed specifically for fine-grained evaluation of LLM constraint following.
  • Benchmark and metrics: CSR measures per-criterion pass rate, while ISR measures all-criteria-pass rate within a single turn.The evaluated model is trained on Qwen2.5-7B →Qwen2.5-1.5B, with no SysBench training data used.
  • Analysis: DUET significantly outperforms all distillation baselines on both CSR and ISR (p < 0.05 ∼p < 0.001).Because training contains no SysBench-style system prompts, the result supports task-agnostic general prohibition alignment rather than memorization of specific constraint templates.

6.4 General Capability Preservation

DUET preserves general reasoning and mathematical capability, achieving 68.3 on GSM8K and 52.2 on MATH-500 without training on either benchmark, with subject-level changes remaining small and uneven.

  • Benchmark evaluation: GSM8K and MATH-500 evaluate multi-step logical reasoning and competition-level mathematical ability using multiple-choice accuracy.The evaluation extracts each response’s selected option and reports accuracy.
  • Capability preservation: 68.3 on GSM8K and 52.2 on MATH-500 show no significant degradation after DUET training.These results come from the Qwen2.5-7B → Qwen2.5-1.5B distillation model, trained without GSM8K or MATH-500 data.
  • Capability preservation: Algebra and Prealgebra remain unchanged, while Geometry and Precalculus improve slightly on MATH-500.The small aggregate difference is not uniform across subjects.

6.5 Ablation Study

Ablations show that DUET’s on-policy sampling, disagreement-based target synthesis, and four complementary losses are each necessary for prohibition compliance and utility preservation. Removing individual components causes substantial drops in the target metric they support.

  • Framework choices: Teacher sampling (off-policy) drops Over. from 75.3 to 62.9, showing that on-policy trajectories are necessary to expose the student’s true failure trajectories.Keeping only T + yields the worst Viol. at 58.8, while all-token supervision reaches 66.0.
  • Loss ablations: Removing Lsyn drops Viol. from 72.3 to 63.7, the largest violation-refusal decline across the ablations.The synthesized target is obtained by extrapolating along the T +/T − disagreement direction after student stabilization.
  • Loss ablations: Removing Lpos drops Over. by 5.7 points because the student over-relies on Lneg, which pulls violation-refusal rates downward.Lpos anchors the student toward T + while Lneg teaches from T − in reverse.
  • Loss ablations: Removing Lneg drops Viol. from 72.3 to 67.0, showing that suppressing violation-candidate mass independently complements the pull from Lpos.The suppressed mass is represented as (p−−p+)+.
  • Loss ablations: Removing Lutil collapses Util. from 93.0 to 62.0, the largest utility drop, confirming its role in preventing over-refusal.Lutil applies top-K KD on the legitimate query within the same business context.
  • Overall ablation finding: Removing any one of the four losses lowers its corresponding target, verifying the losses’ orthogonality and design necessity.Table 3 evaluates framework-level choices and each of the four total-loss terms, with DUET as the complete method.

7 Conclusion

The paper formalizes Prohibition Compliance as runtime-configurable, token-localized rule following under a joint compliance–utility requirement. DUET addresses teacher drift and undifferentiated token-mean supervision through same-weight teacher disagreement.

  • Prohibition Compliance requires runtime-configurable, token-localized rule following together with joint compliance and utility.
  • Vanilla on-policy distillation has two structural gaps: teacher drift under the student’s prefix and undifferentiated token-mean supervision.
  • DUET uses a same-weight negative teacher whose per-token disagreement with the positive teacher addresses both gaps.
Loading 2608.14644v1…