Source-linked AI summary
RAPID: Reliability-Aware Pair Importance Distillation
Ali Mahdavi, Azadeh Zamanifar, Amirfarhad Farhadi, Omid Kashefi
TL;DR
Inter-example relational distillation faces quadratic pair costs and limited-budget sampling choices that can conflate which relations to trust with which to evaluate. RAPID separates a reliability-gated target from a full-support adaptive proposal and applies exact inverse-proposal correction. Across AG News and SST-2, gated relational distillation has the highest observed mean accuracy, while RAPID ranks second within seed-level variation under fixed budgets.
Problem
Relational distillation evaluates O(B^2) example pairs, while uniform subsampling may use a limited relation budget inefficiently and can overlook heterogeneous pair value.
Method
RAPID separates a reliability-gated relational target from a full-support proposal informed by calibrated teacher information and detached student-teacher residuals, with exact inverse-proposal correction.
Results
Across AG News and SST-2, reliability-gated relational distillation has the highest observed three-seed student mean, while RAPID ranks second within seed-level variation under fixed relation budgets.
Takeaways & Limitations
The results support separating target reliability, proposal quality, and systems cost when approximating relational distillation under exact fixed-budget accounting.
Takeaways & Limitations
The evaluation covers two datasets, two architecture pairs, two relation budgets, three paired seeds, and a single batch size and GPU, limiting claims about variability and scalability.
Abstract
from arXiv · showhide
Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently. We introduce Reliability Aware Pair Importance Distillation, or RAPID, which separates a reliability gated relational target from a full support adaptive pair proposal. Reliability determines which teacher relations are emphasized, while calibrated teacher entropy and detached student-teacher residuals determine which relations are evaluated. Exact inverse proposal correction makes the loss and gradient estimators conditionally unbiased with respect to the gated mini batch target. We evaluate RAPID in two text classification settings: AG News with BERT-to-DistilBERT distillation using three paired seeds and a relation budget of 256, and SST-2 with DistilBERT to DistilBERT distillation using three paired seeds and a relation budget of 64. Reliability gated relational distillation achieves the highest observed mean student accuracy on both datasets: 94.285 plus or minus 0.054 percent on AG News and 88.800 plus or minus 0.532 percent on SST-2. RAPID ranks second, achieving 94.241 plus or minus 0.025 percent and 88.685 plus or minus 0.462 percent, respectively, compared with 94.154 plus or minus 0.124 percent and 87.271 plus or minus 0.162 percent for the cross entropy baseline. Pilot evaluations are counted toward the same total budget as the main relation evaluations. Across both settings, the gated target yields the highest mean accuracy, while the adaptive proposal remains within seed-level variation. These results support the modular view that target reliability and evaluation priority are separable design dimensions.
1 Introduction
RAPID addresses the quadratic cost and heterogeneous value of inter-example relational distillation by separating target reliability from evaluation priority. It uses a reliability-gated objective and a corrected adaptive proposal under a fixed relation budget, with reported engineering benefits conditional on the experimental protocol.
- Motivation: O(B^2) pair growth makes relational transfer costly, while uniform subsampling ignores heterogeneous pair losses and uncorrected hard-pair mining can change the intended objective.A batch of B examples contains M = B(B −1)/2 unordered relations.
- Motivation: Reliability determines how strongly a teacher relation contributes, whereas evaluation priority determines how often it is sampled under a limited budget.Large student-teacher residuals can motivate sampling even when reliability keeps the target contribution downweighted.
- Method: RAPID combines a normalized label-aware reliability gate with a distinct proposal driven by calibrated teacher information and detached student-teacher residuals.Endpoint- and pair-level defensive mixtures provide full support, while inverse-proposal correction targets the gated all-pairs objective in expectation.
- Method: Exact inverse-proposal correction yields conditional unbiasedness for the relational loss and gradient, while bounded weights prevent any realization from dominating the gradient.Pilot evaluations are detached from the proposal’s optimization graph and correction is applied to sampled main relations.
- Results: 62.4 minutes and 2127 MB are RAPID’s lowest AG News student-training time and peak memory among the five distillation methods.The comparison uses matched budgets; other distillation methods require 86–131 minutes and 2574–2783 MB.
- Results: RAPID remains within seed-level variation of other relational methods on held-out accuracy and is within 0.115 points of the top SST-2 held-out mean at K = 64.These advantages are conditional on paired seeds, selection-separated reporting, fixed budget, and a single-GPU protocol.
2 Related Work
Prior work transfers intermediate states, attention relations, pairwise geometry, or contrastive representations, while importance sampling and calibration address heterogeneous contributions and teacher confidence. RAPID positions itself as a label-aware reliability-weighted relational target with a separate uncertainty-informed proposal.
- Language-model distillation: Patient KD, TinyBERT, and MiniLMv2 transfer intermediate states, multiple Transformer objectives, or self-attention relations rather than only output predictions.These approaches motivate structural transfer for language-model distillation.
- Relational distillation: RKD transfers pairwise distances and angles, while CRD aligns teacher and student representations contrastively.Recent work also combines adaptive multi-teacher weighting with inter-sample similarity-matrix transfer.
- Importance sampling: Importance sampling addresses heterogeneous training contributions, and dynamic importance sampling has been applied to output classes in knowledge distillation.Influence-based methods have also been used to score text-classification samples for dataset distillation.
- Teacher calibration: Teacher confidence can vary and be miscalibrated, motivating calibration-related work based on margin, curvature, or feature-modulated label smoothing.RAPID uses calibrated teacher information but retains a distinct role for uncertainty in proposal construction.
- Positioning: RAPID uses a label-aware reliability gate based on observed-label probability and predictive margin, while entropy guides proposal priority rather than target trust.The paper explicitly does not equate the gate with a calibrated probability that the teacher is correct.
3 Method
RAPID defines a reliability-gated relational target and estimates it with a separate, full-support adaptive pair proposal under a fixed relation budget. Inverse-proposal correction provides conditional unbiasedness for the loss and stopped-proposal gradient, while factorization and defensive mixtures trade expressivity and variance for bounded, supported sampling.
- Relational objective: The relational objective uses mask-aware mean-pooled representations, cosine relations, and squared discrepancies over unordered example pairs.For B < 2, the relational term is zero.
- Adaptive proposal: The proposal combines calibrated teacher informativeness, reliability, and detached pilot residuals, then interpolates static and adaptive endpoint distributions.Residual adaptation begins after warm-up and reaches at most τmax = 0.5; pilot estimates affect proposal efficiency, not the target.
- Reliability-gated target: Reliability is a normalized, label-aware target weight, distinct from the proposal used to prioritize relation evaluation.The mixture interpolates uniform relational KD and the fully gated target, with λ = 0.5 in the reported configuration.
- Pair sampling: RAPID samples unordered pairs from a factorized endpoint proposal with endpoint- and pair-level defensive mixtures, ensuring support over every batch relation.After pilot evaluations, it draws the remaining main relations with replacement; undersized batches use all available relations directly.
- Unbiased correction: Exact inverse-proposal correction makes the sampled relational loss and stopped-proposal gradient conditionally unbiased for the reliability-gated objective.The gradient result treats reliability, pilot residuals, proposal probabilities, and sampled indices as constants for automatic differentiation.
- Budget and complexity: The total relation budget is K = min{M, R, Ccap}, including pilots; when all pairs fit the budget, RAPID enumerates them exactly.Given representations, all-pairs similarity costs O(B^2d), while pilot and main similarities cost O(Kd), with additional O(B^2) pair-proposal construction.
4 Experiments
The experiments compare relational distillation methods on AG News and SST-2 under matched relation budgets, measuring held-out accuracy, allocation, runtime, and memory. Reliability-gated relational distillation has the highest observed mean accuracy on both datasets, while RAPID remains close within seed-level variation.
- Experimental settings: The evaluation uses AG News with BERT-to-DistilBERT and SST-2 with DistilBERT-to-DistilBERT across seeds 42–44.The nominal relation budgets are K = 256 for AG News and K = 64 for SST-2.
- Held-out accuracy: 94.285 ± 0.054% is the highest observed AG News mean for reliability-gated relational distillation, followed by RAPID at 94.241 ± 0.025%.The student baseline is 94.154 ± 0.124%.
- Held-out accuracy: 88.800 ± 0.532% is the highest observed SST-2 mean for reliability-gated relational distillation, followed by RAPID at 88.685 ± 0.462%.The student baseline is 87.271 ± 0.162%.
- Interpretation: The five distilled students differ by only 0.096 points on AG News and 0.230 points on SST-2, with both spreads within seed noise.The experiments therefore do not claim RAPID dominates the gated-only control.
- Relation budget: RAPID allocates 235.2 main and 20.8 pilot evaluations at K = 256 on AG News, and 59.177 main plus 4.797 pilot evaluations at K = 64 on SST-2.Pilot evaluations are charged to the same total budget as main evaluations.
5 Discussion
RAPID is presented as a fixed-budget relational distillation method whose adaptive proposal and reliability-gated target address different efficiency and supervision roles. On the reported workloads, it offers favorable systems measurements while its accuracy remains close to other relational methods, with conclusions limited to the evaluated settings.
- RAPID recovers the gated mini-batch relational loss and gradient in expectation through inverse-proposal correction, without hidden bias from adaptive sampling.The proposal is detached from the autograd graph.
- On AG News, RAPID records 62.4 minutes and 2127 MB, the lowest student-training time and peak memory among the distillation methods.The comparison is at matched relation budget and reports 86–131 minutes and 2574–2783 MB for the other distillation methods.
- On SST-2 at K = 64, RAPID is within 0.115 points of the gated-only control at near-identical memory.The reported result places RAPID among the tightly grouped held-out means.
- The experiments do not establish scalability: both workloads use a single batch size, and runtime can reflect implementation choices as well as algorithmic budget.A matched budget sweep over batch sizes is left for future work.
6 Limitations
The evaluation is limited to two datasets, two architecture pairs, fixed relation budgets, three paired seeds, and a single batch size. Additional budget sweeps, seeds, diagnostics, and teacher conditions are needed to assess variability, scalability, and broader validity.
- The evaluation covers two datasets, two architecture pairs, and relation budgets K = 256 and K = 64.AG News uses K = 256, while SST-2 uses K = 64 with a compressed DistilBERT-to-DistilBERT pair.
- Three paired seeds are insufficient for fine-grained statistical testing of method rankings.The paper suggests at least five seeds and paired tests such as a sign test or paired bootstrap.
- Scalability conclusions are unavailable because all experiments use B = 32 on a single GPU.The paper leaves matched budget sweeps over increasing batch sizes for future work.
- Reliability depends on observed labels and sufficiently accurate annotations, while the factorized proposal cannot represent arbitrary pair-specific importance.Pilot adaptation also reduces the number of corrected main draws.
7 Ethics and Broader Impact
The study uses public AG News and SST-2 benchmarks and publicly available pretrained models, but inherits their biases and annotation artifacts. Distillation may also transfer teacher errors and social biases, while label-aware reliability may reinforce artifacts when labels are systematically noisy.
- The study introduces no new human-subject data and uses public sentiment and news-classification benchmarks.The pretrained BERT and DistilBERT checkpoints are also publicly available.
- The evaluation inherits biases and annotation artifacts present in the public resources.This is a scope boundary of using those benchmarks and checkpoints.
- Distillation can transfer teacher errors and social biases to the student.The paper identifies this as a broader-impact risk of the compression setting.
- Label-aware reliability may reinforce annotation artifacts when observed labels are systematically noisy.This risk follows from using labels in the reliability mechanism.
8 Conclusion
RAPID separates reliability in the relational target from priority in the sampling proposal, using defensive sampling and inverse-probability correction under a fixed relation budget. Across both evaluated settings, reliability-gated distillation leads observed accuracy means, while RAPID ranks second within seed-level variation.
- RAPID separates reliability in the relational target from priority in the sampling proposal under a fixed total relation budget.Normalized gating preserves relational scale, while the proposal adapts evaluation priority.
- Reliability-gated relational distillation has the highest observed three-seed student mean on both AG News and SST-2, while RAPID is second-highest.Both methods improve on the cross-entropy student, standard KD, and uniform relational sampling.
- RAPID remains within seed-level variation of the gated-only control rather than demonstrating strict dominance.The conclusion treats the method ranking as compatible with the fixed-budget modular design, not as definitive superiority.
A Additional Derivations
The derivations establish conditional variance and gradient-covariance properties for corrected RAPID, while identifying assumptions, estimand boundaries, and the trade-off between pilot guidance and main evaluations.
- Conditional variance: Pilot randomness changes the proposal but not the conditional target, so corrected draws retain the same conditional mean for every pilot realization.This follows because the conditional mean is Lλ for every pilot realization.
- Conditional variance: Pilots can reduce variance by improving the proposal, but they also reduce the number of main relation evaluations.The total-budget accounting makes this a direct variance-versus-sample-size trade-off.
- Gradient covariance: Gradient-oriented proposals prioritize pair gradients, which generally differs from the proposal that optimizes scalar-loss estimation.The derivation distinguishes gradient covariance objectives from scalar-loss importance sampling.
- Guarantee conditions: The guarantees require normalized gating, full-support probabilities, unclipped inverse corrections, and detached proposal construction.Correction clipping introduces bias by design, while the probability floor is inactive at the reported batch sizes.
- Scope and limitations: The theorems concern finite-population current-mini-batch targets, not fixed global relation graphs, and unbiasedness alone does not ensure variance reduction.Variance improvement requires proposal alignment with the gated loss or gradient integrand sufficient to offset pilot cost.
B Seed-Level Results
Seed-level results report per-seed held-out accuracies for AG News and SST-2 under matched paired-seed comparisons, with SST-2 values showing close variation among relational methods.
- AG News: Table 4 reports AG News held-out accuracy by paired seed at K = 256.The supplied table reference identifies the dataset, metric, pairing, and relation budget.
- SST-2: 88.991/89.335/88.073 is the SST-2 reliability-gated accuracy for seeds 42/43/44.The same seeds yield RAPID accuracies of 88.303/89.335/88.417 and uniform relational accuracies of 88.761/88.876/88.188.
- SST-2: SST-2 seed-wise accuracies are within the table’s standard deviations across the reported methods.The passage states this comparison before referring readers to mean ± standard deviation summaries.
C Reproducibility and Accounting
The artifacts record detailed software, hardware, seed, and source metadata, while accounting separately tracks relation-evaluation components and explains the slight SST-2 total shortfall.
- Reproducibility: The experiments used Python 3.12.9, PyTorch 2.6.0 with CUDA 12.6, batch size 32, split seed 1729, and student seeds 42–44.Both datasets used the same student-seed range and a single NVIDIA GeForce RTX 3080 with no concurrent GPU jobs.
- Reproducibility: The artifacts also record cuDNN 90501, NumPy 2.0.2, source hash 76311e67e5b4eea66d8bbf2cd7b657 5cffbf5f37e85b42ca42127021c870d785, and experiment commands.These metadata support reconstruction of the reported runs.
- Accounting: SST-2 totals are slightly below K = 64 because pilot allocation rounding and the final incomplete batch affect implementation accounting.Unique main and pilot counts are measured within each component, and their union was not stored.
- Accounting: Table 5 reports mean relation evaluations and component-wise unique counts per batch for AG News at K = 256 and SST-2 at K = 64.The table distinguishes dataset-specific budgets and summarizes evaluation accounting by component.