Source-linked AI summary
GATES: Self-Distillation under Privileged Context with Consensus Gating
Alex Stein, Furong Huang, Tom Goldstein
TL;DR
The paper studies self-distillation when verified labels, rewards, and graders are unavailable, using document access to create an asymmetric tutor–student setting. GATES samples multiple tutor trajectories, gates learning by answer consensus, and distills agreed reasoning into the document-free student. It improves both held-out asymmetric evaluation and public document-free math performance, while relying on assumptions about consensus reliability and answer extraction.
Problem
Self-distillation lacks reliable supervision when no ground-truth labels, verifiable rewards, or external graders exist, and naive self-distillation can reinforce errors.
Method
GATES samples multiple document-grounded tutor rollouts, uses answer consensus to gate training, and distills eligible full reasoning trajectories into a document-free student.
Results
GATES improves held-out in-domain student accuracy from 46.0% to 62.0% and average maj@8 accuracy on public document-free math benchmarks from 20.2% to 35.4%.
Takeaways & Limitations
Consensus gating alone provides sufficient reliability modeling for effective self-distillation in this document-grounded, document-free transfer setting.
Takeaways & Limitations
Consensus may reflect shared tutor errors, and skipping low-agreement questions reduces training updates and may limit sample efficiency.
Abstract
from arXiv · showhide
We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.
1 Introduction
GATES addresses unreliable self-distillation by using document access to create an asymmetric tutor–student setup and consensus to decide when tutor reasoning is trustworthy. It then distills agreed tutor trajectories into a document-free student, improving transfer over baselines.
- Standard self-distillation can reinforce systematic errors or degenerate shortcuts because the model itself supplies supervision.
- GATES uses the same model as tutor and student, with the tutor receiving a document and the student receiving only the question.The asymmetric context provides the tutor with privileged evidence while preserving shared model weights.
- Multiple tutor rollouts are filtered by answer consensus, and questions without sufficient agreement are skipped.Strong consensus requires at least τ matching answers among k sampled tutor responses.
- Consensus-gated training distills full tutor reasoning trajectories, providing dense token-level supervision rather than learning only from final answers.
- Held-out in-domain student accuracy improves from 46.0% to 62.0%, while average maj@8 accuracy on public document-free math benchmarks improves from 20.2% to 35.4%.
- GATES outperforms unfiltered trajectory distillation, answer-only fine-tuning, and outcome-based reinforcement learning, with ablations identifying consensus gating as critical.
2 Related Work
GATES combines privileged-context distillation with self-agreement-based reliability filtering. Its distinguishing feature is using tutor consensus to gate dense trajectory learning without verified correctness or preference supervision.
- Unlike classical distillation, GATES uses the same model as tutor and student while giving only the tutor privileged document context.
- The method addresses self-training error amplification by filtering supervision through agreement among multiple tutor rollouts.
- GATES uses consensus to gate dense trajectory-level distillation, rather than selecting an answer at inference time or filtering correct-answer chains.
- Preference-based self-improvement derives supervision from internally generated comparisons, whereas GATES derives it from agreement.
- Concurrent approaches differ because they rely on verified correctness or preference signals, while GATES uses self-agreement under asymmetric context alone.
3 GATES: Gated Asymmetric Trajectory Self-Distillation
GATES uses tutor consensus to decide when a shared model can safely distill document-grounded reasoning into its document-free student role. It supports off-policy imitation of eligible tutor trajectories and on-policy weighting of student trajectories, while skipping unreliable questions.
- 3.1 Setting and Notation: GATES uses one model as a document-conditioned tutor and a question-only student, with supervision derived online from tutor consensus.The roles share parameters, and training uses fixed document–question pairs without verified answers.
- 3.2 Consensus-Gated Training: For each question, GATES samples k tutor rollouts and activates learning only when at least τ agree on a final answer.Questions failing the consensus threshold contribute zero loss.
- 3.2 Consensus-Gated Training: Eligible off-policy tutor trajectories are those matching the consensus answer and passing document-leakage guardrails.These trajectories include the full reasoning completion and final answer, providing token-level supervision rather than answer-only targets.
- 3.3 Training Objectives: Off-policy distillation trains the student to imitate eligible tutor-generated trajectories, while on-policy distillation scores the student’s own trajectories under document context.Both modes use the same question-level consensus gate, but on-policy training has no trajectory-level eligibility filter.
- 3.3 Training Objectives: On-policy advantages upweight tokens that the document-aware tutor assigns higher likelihood than the student, with clipping and no gradient flow through the advantage.This encourages document-grounded reasoning while retaining the student’s on-policy trajectories.
- 3.3 Training Objectives: The total objective combines off-policy and on-policy distillation losses and can be extended with consensus-reward and KL-regularization auxiliary terms.The consensus-reward term is ablated, while KL regularization targets a frozen reference policy.
4 Empirical Evaluation
GATES is evaluated on a fixed-challenger dataset using asymmetric student transfer, document-free math benchmarks, and controlled ablations. It achieves the strongest reported transfer, with consensus gating and off-policy trajectory distillation driving the gains.
- 4.1 Experimental Setup: The fixed-challenger dataset contains 551 training questions and 50 held-out evaluation questions, with tutor rollouts filtered by answer consensus and document-leakage guardrails.Questions require at least 5/8 agreeing tutor answers during construction; training uses 8 tutor rollouts and a ≥4/8 consensus gate.
- 4.1 Experimental Setup: Evaluation measures document-free student accuracy on 50 held-out questions and public math benchmarks using symbolic final-answer equivalence.The held-out evaluation uses greedy decoding, while benchmark results use maj@8 decoding.
- 4.2 Main Results: GATES raises average maj@8 benchmark accuracy from 20.2% for the pretrained model to 35.4%, outperforming Tutor-Trajectory SFT at 32.3%.Outcome RL reaches 21.3%, comparable to the pretrained baseline.
- 4.2 Main Results: GATES improves held-out student accuracy from 46.0% to 62.0%, whereas answer-only SFT reduces it to 12.0% and 10.0%.The tutor reaches 70.1% versus 35.8% for the student on the filtered training set, indicating a substantial document-access gap.
- 4.3 Ablations: Removing consensus gating lowers student accuracy to 54.0% and benchmark average to 31.1%, while reversing or removing off-policy distillation causes further degradation.The canonical configuration reaches 62.0% student accuracy and a 35.4% benchmark average.
- 4.3 Ablations: Adding oracle correctness loss provides no meaningful benchmark gain, with 35.7% versus 35.4%, and lowers student accuracy from 62.0% to 54.0%.Oracle-only training reaches 52.0% student accuracy and a 33.2% benchmark average, supporting the value of dense trajectory imitation.
5 Discussion
GATES separates supervision reliability from learning density by using tutor consensus to filter trajectories before distilling them. Its scope is constrained by consensus quality, answer extraction, privileged-context asymmetry, and the choice of challenger.
- Consensus may reflect shared errors when tutor rollouts are biased, insufficiently capable, or lack reasoning diversity.
- Reliable answer extraction and normalization are prerequisites for consensus estimation and downstream learning.
- Multiple tutor rollouts increase training-time computation relative to single-pass supervision.
- The method is evaluated exclusively in document-grounded question answering, where privileged documents create a strong tutor–student asymmetry.
6 Conclusion
The paper studies self-distillation without verified supervision and uses tutor consensus to gate trajectory transfer into a document-free student. GATES improves held-out student accuracy while showing that consensus gating, rather than sparse correctness reward, is the active ingredient.
- GATES improves held-out student accuracy from 46.0% to 62.0% in document-grounded question answering without verified labels, rewards, or external graders.
- Consensus gating selects trustworthy self-supervision before trajectory-level distillation transfers document-grounded reasoning to the document-free student.
- Ablations identify the gate as the active ingredient, while adding a sparse correctness reward does not improve student transfer.
A.1 Prompts
The appendix specifies separate prompts for the document-free student and document-accessing tutor. Both prompts request direct answers with displayed reasoning, while only the tutor receives the document.
- A.1.1 Student Prompt: The student prompt requests step-by-step work and a final answer in \boxed{} at the end.
- A.1.1 Student Prompt: The student has no access to the document during training and evaluation.
- A.1.2 Tutor Prompt: The tutor prompt is used during training.
- A.1.2 Tutor Prompt: The tutor has access to the document while generating training responses.
- A.1.2 Tutor Prompt: The tutor is explicitly instructed not to mention the document.
A.2 Auxiliary Loss Terms
The auxiliary objective combines off-policy and on-policy trajectory losses with optional consensus reward and KL regularization. Consensus gates all terms, and experiments find trajectory distillation sufficient without the sparse reward.
- Section 3.3 adds two optional loss terms to the two primary distillation losses.
- The consensus-correctness reward uses a binary pseudocorrectness label based on whether each rollout matches the question’s consensus answer.
- KL regularization optionally regularizes the student toward a frozen reference policy.
- The total objective weights off-policy distillation, on-policy distillation, consensus reward, and KL regularization.
- The consensus reward is disabled by default, and all objective terms remain gated by tutor consensus.
- Consensus-gated trajectory distillation alone provides sufficient supervision, while the consensus reward can reduce student accuracy.
A.3 Hyperparameters
Tables 6 and 7 provide the training and evaluation hyperparameters used in the submission.
- Table 6 lists the training hyperparameters used unless otherwise specified.
- Table 7 lists the evaluation hyperparameters used in this submission.
A.4 Training Guardrails
The training setup uses masking, prompt alignment, validity checks, leakage prevention, and deterministic evaluation to stabilize asymmetric self-distillation. Additional evaluations show that GATES improves answer consistency and student transfer more than single-sample or coverage metrics alone reveal.
- Training guardrails: Losses apply only to completion tokens, while prompts are masked and must end exactly with Solution:.
- Training guardrails: Tutor and student prompts share the same structure and delimiter, differing only in whether the document is present.
- Training guardrails: Rollouts without a parsable final boxed answer, including truncated outputs, are invalid and receive zero loss weight.
- Training guardrails: A two-stage document-mention filter removes document-referencing questions and tutor trajectories from distillation targets.
- Training guardrails: Evaluation fixes IDs, prompts, and grading while separately tracking validity rates alongside accuracy.
- Additional evaluation: 40.0% versus 40.3% average greedy accuracy shows near-identical single-sample performance for GATES and Tutor-Trajectory SFT.
- Additional evaluation: 61.3% average pass@8 accuracy for GATES exceeds 60.6% for Tutor-Trajectory SFT, 55.2% for Outcome RL, and 54.3% for the base model.
- Additional evaluation: Tutor-Trajectory SFT reaches 74% tutor accuracy but only 54% student accuracy, producing the widest tutor–student gap.