PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruct, Not Answer: Using Instruction Privileges in On-Policy Context Distillation

Hantao Yu, Xiaoxue Han, Udaya Ghai, Ferhat Erata, Joseph Lilien, Aman Goel, Ali Torkamani

arXiv:2609.32201v2cs.AIcs.LG

TL;DR

Instance-specific gold privileges can hurt OPCD performance, particularly out of distribution, while evidence for reusable instructions remains limited. The paper targets common student mistakes with general formatting instructions and finds that they outperform gold in most OOD settings while remaining similar in-domain.

  • Problem

    Prior work reports that privileged contexts can degrade OPCD, especially OOD, and it is unclear whether domain-specific instructions can outperform instance-specific gold.

    Source: §1 · p. 1

  • Method

    The authors design short, generic formatting instructions around training-set student mistakes and use them as OPCD teacher privileges.

    Source: §1 · pp. 2–3

  • Results

    Matched instructions outperform gold in 7 of 8 OOD settings and perform similarly to gold in-domain in 5 of 8.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    In autoformalization, generic instructions targeting student vulnerabilities can improve performance while avoiding OOD degradation associated with gold privileges.

    Source: §7 · p. 9

    The study focuses on autoformalization from natural language to first-order logic; privilege types may differ in other domains.

    Source: §7 · p. 9

Abstract

from arXiv · show

On-Policy Context Distillation (OPCD) has recently emerged as a powerful technique for transferring context to student models and for self-improvement. In OPCD, the teacher is conditioned on privileged information, and the goal is to minimize the Kullback-Leibler (KL) divergence between the privileged teacher and the student, evaluated on student-generated tokens. Many existing studies show that using instance-specific gold answers or gold demonstrations as the default privilege can hurt training performance, especially out-of-distribution (OOD). In this work, we instead design general instructions that target common student mistakes observed on the training samples, and show that such simple instructions can outperform gold as the OPCD privilege. In autoformalization tasks, using a matched formatting instruction as the privilege could outperform gold in OOD accuracy by a large margin. In 7 out of 8 experiments using ProverQA, ProofWriter, and ProntoQA as datasets, and Qwen3-Thinking and Olmo3-Thinking families as models, matched instruction privileges outperform gold in OOD by 4 to 17 points, while remaining on par with gold in-domain. Each instruction is only a few sentences (and thus contains much less information compared to all instance-specific gold) and is applied uniformly to every training sample. These results indicate that a general instruction, which applies equally to source and target domain examples, can be substantially more transferable than instance-specific gold in OPCD while maintaining in-domain performance.

1 INTRODUCTION

The paper asks whether reusable instructions can outperform instance-specific gold privileges in OPCD, especially on out-of-distribution tasks. It designs instructions around student formatting mistakes and reports stronger OOD results than gold in most settings.

Research Question

  • The study tests whether domain-specific instructions can improve in-domain performance and generalize better than instance-specific gold privileges.Prior studies report that privileged contexts can degrade OPCD performance, especially OOD.Source: §1 · pp. 1–2

Testbed

  • Autoformalization translates natural-language statements into first-order logic formulas that a solver uses to determine a verdict.The paper motivates it as a testbed because student mistakes are relatively easy to identify and target.Source: §1 · p. 2

Main Results

  • The experiments use Qwen3-Thinking and Olmo3-Thinking models with ProverQA, ProofWriter, and ProntoQA.The paper evaluates OPCD with teacher-student model pairs and cross-dataset OOD transfers.Source: §1 · p. 2, Figure 1 · p. 2
  • Matched instructions outperform gold in 7 of 8 OOD settings and perform similarly in-domain in 5 of 8.The comparison is against instance-specific gold privileges.Source: §1 · pp. 2–3, Figure 1 · p. 2

Approach

  • The authors identify training-set formatting errors and write generic instructions that target those mistakes as teacher privileges.The instructions are a few sentences, contain less information than per-problem gold formulas, and are applied uniformly.Source: §1 · pp. 2–3

2 PRELIMINARIES

The preliminaries define distillation losses and the paper’s autoformalization task. They distinguish teacher-generated from student-generated traces and explain how privileged context conditions the teacher.

2 PRELIMINARIES

  • The preliminaries use θ_T and θ_S for teacher and student distributions, x for the prompt, y for the output, and D_KL for KL divergence.Source: §2 · p. 3

2.1 KNOWLEDGE DISTILLATION FOR LLMS

  • Knowledge distillation trains the student toward the teacher’s soft probability distribution, while off-policy traces are generated by the teacher.On-policy distillation instead samples student-generated traces so the teacher can correct mistakes on those traces.Source: §2.1 · p. 3

2.2 CONTEXT DISTILLATION FOR LLMS

  • Context distillation gives the teacher privileged context and minimizes KL divergence against the student; OPCD samples output sequences from the student.When teacher and student are the same model, OPCD is called On-Policy Self-Distillation.Source: §2.2 · p. 3

2.3 AUTOFORMALIZATION

  • Autoformalization translates natural-language premises and questions into first-order logic formulas, which a solver uses to return a verdict.Translation errors such as dropped quantifiers or inconsistent predicate names can lead to incorrect solver verdicts; formatting instructions are studied as privileges.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 WHY AUTOFORMALIZATION IS IDEAL FOR OUR GOALS

Autoformalization offers an objective way to evaluate privileges and diagnose student errors. Shared formatting rules across datasets also enable direct tests of transfer.

Objective Verification

  • A symbolic solver supplies discrete TRUE, FALSE, or UNKNOWN verdicts, making correctness objectively verifiable.Source: §3 · p. 4

Diagnosable Failure Modes

  • Discrete errors such as unparseable formulas, dropped quantifiers, and mistranslated XORs can be attributed to specific causes and targeted by instructions.The passage contrasts these diagnosable failures with entangled errors in free-form math and science generation.Source: §3 · p. 4

Transfer Evaluation

  • The same formatting rules govern the three datasets, enabling source-to-target tests of whether privileges transfer.Gold formulas and dataset-level rules can be compared under identical training and evaluation.Source: §3 · p. 4

4 EXPERIMENTAL SETUP

The experiments compare teacher privileges in OPCD with off-policy context distillation and direct question-answering across logical autoformalization benchmarks. They evaluate two model families using verdict accuracy metrics and a shared training and evaluation setup.

4 EXPERIMENTAL SETUP

  • Experiments compare OPCD privileges with off-policy context distillation and direct question-answering on the same datasets.For direct question-answering, the privilege is the gold TRUE/FALSE/UNKNOWN verdict.Source: §4 · p. 4
  • Qwen3-1.7B students use Qwen3-4B or Qwen3-8B teachers, while Olmo3-7B students use Olmo3-32B teachers.The tested families are Qwen3-Thinking (Yang et al., 2025) and Olmo3-Thinking (Ettinger et al., 2026).Source: §4 · p. 4
  • Table 1 covers autoformalization on ProverQA, ProofWriter, and ProntoQA, with parenthetical values marking accuracy changes after adding gold FOL privilege.The caption defines the table’s scope and the meaning of its parenthetical changes.Source: Table 1 · p. 5
  • Training and evaluation use ProverQA (Qi et al., 2025), ProofWriter (Tafjord et al., 2021), and ProntoQA (Saparov & He, 2023).Autoformalization targets use ProverQA gold FOL formulas or deterministic regex converters for ProofWriter and ProntoQA; direct question-answering uses gold verdicts.Source: §4 · p. 5
  • Average@8 and pass4 verdict accuracy are the main metrics for both the autoformalization-solver pipeline and direct question-answering.The model is trained once and evaluated eight times with different random seeds; pass4 rewards consistent correct rollouts more strongly.Source: §4 · p. 5
  • Each dataset uses 1,200 training samples and 300 evaluation samples, with full-vocabulary forward-KL logit distillation without clipping.Rollouts and evaluation use temperature 0.7; experiments use H100 GPUs with LoRA (Hu et al., 2022).Source: §4 · p. 5

5 EXPERIMENTAL RESULTS

Across these experiments, OPCD with formatting-instruction privileges often transfers better out of domain than gold FOL, while preserving in-domain performance. The experiments diagnose dataset-specific formatting errors and compare targeted instructions with gold and direct prompting.

5.1 GOLD FOL PRIVILEGE IMPROVES VERDICT ACCURACY

  • With a Qwen3-4B teacher, gold FOL raises OPCD pass4 accuracy by +8.8, +5.0, and +6.6 across ProverQA, ProofWriter, and ProntoQA.OPCD also exceeds CD in autoformalization; for ProofWriter with a Qwen3-4B teacher, pass4 accuracy is 58.2 versus 51.4.Source: §5.1 · pp. 5–6
  • Gold privilege can improve direct question-answering through distillation, but adding the gold verdict to the teacher prompt has almost no effect on accuracy across the datasets.Table 2 reports direct question-answering results and marks accuracy changes from adding the gold verdict in parentheses.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 FORMATTING ERRORS ARE THE PROBLEM

  • Formatting errors vary by dataset: XOR translation is ProverQA’s most common mistake, ∀-dropping is most common in ProofWriter, and ProntoQA has major ∀-drop and coref-split errors.ProntoQA is the cleanest of the three for formatting but has the most semantic problems; Table 3 summarizes training-set error percentages across datasets.Source: §5.2 · p. 6, Table 3 · p. 7

5.3 FORMATTING INSTRUCTIONS AS PRIVILEGE

  • Targeted formatting instructions can beat gold in-domain on ProverQA and ProofWriter, whereas gold dominates the instructions on ProntoQA.The ProverQA coref-split instruction improves pass4 by +13.0 versus gold’s +8.8; on ProofWriter, ∀-drop gives +17.1 versus gold’s +5.0. Directly adding formatting instructions to student prompts gives almost no improvement and often hurts performance.Source: §5.3 · p. 7, Table 5 · p. 8

5.4 OUT-OF-DOMAIN FORMATTING INSTRUCTION TRANSFERRING

  • Matched formatting instructions outperform gold in OOD evaluations, while gold can degrade performance and instructions avoid that degradation.For ProverQA and ProofWriter, the best instructions beat gold in-domain; on ProntoQA, gold is strongest in-domain but degrades OOD. In 5 of 6 cases, gold degrades performance by a large margin, while even average instructions avoid such degradation.Source: §5.4 · pp. 7–8, Table 5 · p. 8

6 RELATED WORK

Prior work studies context and privilege design in distillation, on-policy self-distillation, and autoformalization. This work uses autoformalization to compare formatting instructions with instance-specific gold privileges.

Privilege Design in Context Distillation

On-Policy (Self) Distillation

Autoformalization

  • Prior autoformalization work includes solver pipelines and fine-tuning, with most existing work using off-policy methods.This study designs formatting instructions to examine privilege design in off-policy and on-policy distillation.Source: §6 · p. 9

7 DISCUSSIONS

The discussion reports an empirical study of privilege design in OPCD and argues that generic instructions targeting student mistakes merit further study beyond autoformalization.

7 DISCUSSIONS

  • Formatting instructions as OPCD privileges improved student performance while avoiding the OOD degradation associated with gold privileges.The instructions are generic and contain much less information than instance-specific gold privileges.Source: §7 · p. 9
  • The study is limited to autoformalization, where models translate natural language into first-order logic; other domains may require different privileges.The authors identify studying privilege types and their in-domain/OOD tradeoffs as a future direction.Source: §7 · p. 9

A.1 SYSTEM PROMPTS

The appendix specifies the prompt setup for autoformalization: the model formalizes a bare statement, while the solver determines its truth.

A.1 SYSTEM PROMPTS

  • The ProverQA-style truth-question preamble is removed so the model formalizes a bare statement rather than deciding its truth.The solver decides truth.Source: §A.1 · p. 13
  • The system prompt asks for FOL formulas for each premise and the statement in a fenced code block, ending with the statement’s conclusion formula.It instructs the model to reason about predicates, entities, quantifiers, and connectives, and not to decide whether the statement is true.Source: §A.1 · p. 13
  • The example user prompt supplies premises and a statement to formalize, with a privilege placeholder for examples such as gold FOL.The example asks for premise formulas followed by the conclusion formalization.Source: §A.1 · p. 13

A.2 FULL EXPERIMENTAL DETAILS

The appendix identifies a hyperparameter table.

A.2 FULL EXPERIMENTAL DETAILS

  • Table 9 presents the experiment hyperparameters.The supplied passage gives the table caption only.Source: Table 9 · p. 14

A.3 COMPLETE ERROR TYPE TABLE

Table 10 presents how OPCD changes the error decomposition relative to a no-privilege Qwen3-1.7B student on evaluation rollouts.

A.3 COMPLETE ERROR TYPE TABLE

  • Table 10 reports the change in error decomposition after OPCD.The comparison is relative to the no-privilege student on evaluation rollouts.Source: Table 10 · p. 14
  • The reference model is the no-privilege Qwen3-1.7B student.Source: Table 10 · p. 14
  • The reported decomposition is measured on evaluation rollouts.Source: Table 10 · p. 14
Loading 2609.32201v2…