PaperlayerView exact revision on arXiv

Source-linked AI summary

Teacher-Anchored Selection of Post-Training Quantized Models under Domain Shift

Alejandro Rodriguez Dominguez, Muhammad Shahzad, Xia Hong

arXiv:2609.31155v1cs.LGcs.AI

TL;DR

Under domain shift, choosing among fixed candidates from a shared teacher is difficult when target labels are absent or scarce. The paper develops teacher-anchored selection and finds it reduces regret versus direct validation at the smallest label budget, with its advantage fading as the supervised estimate becomes reliable.

  • Problem

    Given a fixed family of candidates sharing a teacher, the task is to select the candidate with lowest target cross-entropy under domain shift, with target labels absent or scarce.

    Source: §1 · p. 1

  • Method

    The paper uses teacher anchoring with a supervised term, derives quadratic identities for a canonical analogue, and characterizes attenuation under symmetric label corruption.

    Source: What the Family Implies · p. 3, §5 · p. 7

  • Results

    With ten target labels, anchoring reduces mean regret below direct validation in all four settings and under corruption in every budget–rate cell.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Anchoring is most advantageous at the smallest label budgets, with the advantage fading as the supervised estimate becomes reliable alone.

    Source: §1 · p. 2, §5 · p. 7

    The selector is not label-free because its coefficient is cross-validated on the labeled sample.

    Source: What the Family Implies · p. 3

Abstract

from arXiv · show

Compressing a trained model yields a family of deployment candidates, and under domain shift the most compressed one need not be the one to deploy. We study selection over such a family, with candidates and teacher fixed and target labels absent or scarce. Two findings organize the label-free case. Minimum teacher distortion behaves almost as a constant rule, selecting the same eight-bit, per-channel, unclipped configuration in every run, which does not minimize empirical target cross-entropy. Established estimators divide sharply: in the overconfident-collapse regime of the CNN families, confidence-based estimators order the family close to backwards, and the diagnostics that identify it need the labels the setting denies, while output-distribution estimators match the teacher-relative anchor and on one architecture beat it. Distortion is nonetheless stable, so a supervised term can move selection away from it. Combining the two, we give exact quadratic identities for a canonical quadratic analogue of the family. We also show that under symmetric corruption the label-dependent part of a criterion linear in the label indicator is multiplied by one common factor whenever its coefficient sums are candidate-invariant, a class holding teacher contrasts and accuracy but not cross-entropy. These characterize the score's components without bounding selection regret. Across one hundred and thirty-four candidate families, one per independently trained convolutional or Vision Transformer teacher, anchoring reduces mean regret at the smallest label budget in every setting, an advantage that fades beyond twenty-five labels.

1 Introduction

The paper studies choosing among fixed, teacher-related compression candidates for a shifted target domain, where the best candidate may differ from the one favored by construction. It proposes anchoring scarce-label selection in teacher distortion and analyzes the resulting selectors and their behavior under label corruption.

1 Introduction

  • Compression yields multiple plausible deployment candidates, and source or calibration behavior may not reveal which has the lowest target risk under domain shift.Selection is distinct from constructing the candidate family: candidates remain fixed during selection, and target cross-entropy is the objective.Source: §1 · p. 1
  • Teacher distance is stable but can favor the least-compressed candidate, while scarce labeled validation is noisy and becomes less reliable with corrupted labels.The paper identifies these as complementary failures: a stable but biased signal and an unbiased but noisy one.Source: §1 · p. 1
  • Anchored selection uses teacher distortion as a label-free base and cross-validates a supervised correction, while quadratic identities and a corruption result characterize selector components.Under symmetric corruption, a common attenuation factor applies to linear-in-label criteria when their coefficient sums are candidate-invariant; this includes teacher contrasts and accuracy, but not cross-entropy.Source: §1 · p. 1
  • The paper evaluates label-free baselines, develops anchored selection and its identities, and reports that anchoring improves on direct validation at the smallest label budgets.The advantage fades as the supervised estimate becomes reliable on its own.Source: §1 · pp. 1–2

2 Related Work

Related work addresses shifted-domain performance estimation, teacher-guided compression, and selection with scarce or unreliable labels. This paper differs by selecting within a fixed family that already shares a teacher, using that teacher as a reference alongside a supervised term.

Performance estimation under shift

  • Performance estimators target absolute performance, whereas selection requires the correct ordering of candidates whose risks may differ only slightly.Agreement-based estimators require an external reference; this setting already has a shared teacher, unlike methods that model joint classifier scores.Source: §2 · p. 2

Shared teachers and compressed-model construction

  • Compression methods use teachers to guide candidate construction, but this paper begins after the family is fixed and carries teacher distance into deployment-time selection.The teacher guides construction in distillation and quantization methods; here its distance is a selection signal, but is insufficient alone.Source: §2 · p. 2

Selection with scarce or unreliable labels

  • With scarce or unreliable labels, direct validation is a natural baseline, and different losses can induce different rankings when calibration and confidence shift.This work uses the same labeled examples and budget for each selector, rather than selecting which examples to annotate.Source: §2 · p. 2

3 Anchored Selection and Its Identities

The paper formulates selection among fixed candidates sharing a teacher and builds selectors by adding a labeled-sample function to teacher distortion. It then derives identities and conditions under which label information attenuates uniformly across candidates.

3 Anchored Selection and Its Identities

  • The framework selects among a fixed family sharing a teacher, using an unlabeled target pool and, when available, a labeled sample; regret is test cross-entropy above the best candidate’s.The held-out test set is used only for evaluation.Source: §3 · p. 2

One Family of Selectors

  • Each selector adds a labeled-sample function to teacher distortion, with α = 0 recovering distortion alone and cross-validation choosing the coefficient, including the option to revert to the anchor.The choices include empirical cross-entropy, alignment toward the observed class, and a permutation control that preserves scale while breaking label-input correspondence.Source: One Family of Selectors · p. 2

What the Family Implies

  • The alignment statistic separates into a label-dependent part and a teacher-only part, but the implemented teacher-component ablation remains cross-validated on labeled data.Its score component is label-independent; the selector itself is not.Source: What the Family Implies · p. 3
  • Under symmetric K-class corruption, candidate comparisons attenuate by one common factor when the label-coefficient sums are candidate-invariant.This covers teacher contrasts and accuracy indicators, but not cross-entropy; the result does not extend to criteria outside that class.Source: What the Family Implies · p. 3

4 Empirical Evaluation

Across 134 fixed candidate families, the evaluation compares selection with no labels, scarce clean labels, and corrupted labels. Anchoring helps most when supervision is limited or degraded, while its benefit generally fades as clean-label estimates become reliable.

Selection without Target Labels

  • Minimum KL distortion selects the same 8-bit, per-channel, unclipped configuration in all 134 families, while confidence estimators nearly reverse candidate ordering on ResNet50 under overconfident collapse.On ResNet50, the worst candidate's confidence is 1.000 against 0.408 for the best; in the two ViT settings, the degradation pattern is inverted and the same estimators select well.Source: Selection without Target Labels · pp. 3–4
  • Output-distribution estimators are indistinguishable from the anchor on ResNet50, while three have lower mean regret on MobileNetV2; COT reaches 0.007 there, but supervised selectors do not improve on it until n = 300.A post-hoc ResNet50 audit finds validation cross-entropy has lower ECE and cross-entropy but lower accuracy than distortion; calibration is the dominant source of disagreement.Source: Selection without Target Labels · p. 4

Selection with Scarce Clean Labels

  • With ten clean labels, CE-combo lowers mean regret in all four settings; on MobileNetV2 it falls from 0.227 to 0.103.Across 23 budget comparisons, it wins in 11 and loses in 12; from 50 labels onward, it loses in 11 of 15 comparisons.Source: Selection with Scarce Clean Labels · pp. 4–5

What the Directional Term Contributes

  • On ResNet50, alignment beats the permuted-label control at every budget, from 0.1870 versus 0.2170 at ten labels to 0.0270 versus 0.2320 at 150; its value is not uniform across settings.The teacher-only component is best at ten labels on ResNet50 (0.145) and worst on the other three settings; budget is a clearer moderator than anchor regret.Source: What the Directional Term Contributes · pp. 5–6

Selection under Symmetric Label Corruption

  • At η = 0.4, anchoring improves on direct validation on both architectures; the directional variant lowers median regret from 0.165 to 0.102 on ResNet50 and from 0.779 to 0.393 on MobileNetV2.Source: Selection under Symmetric Label… · p. 6

Scarce and Corrupted Labels Together

  • With ten labels and corruption, CE-combo beats direct validation in all 20 cells with η > 0; at η = 0.4, mean regret drops from 0.393 to 0.278 on ResNet50 and 0.864 to 0.485 on MobileNetV2.On MobileNetV2, validation-accuracy regret is above 1.1 at ten labels, yet is lowest in all six cells with n ≥ 50 and η > 0, reaching 0.028 at n = 300 and η = 0.4, where CE-combo reaches 0.803.Source: Scarce and Corrupted Labels Together · p. 6

Does Selection Survive a Cost Constraint?

  • Under a cost constraint, CE-combo improves on direct validation in seven of eight budget–architecture cells at ten labels, while the within-budget cross-entropy spread remains above 10.At a budget of ≤22% of full precision, the frontier cost is 0.007 on ResNet50 and 0.868 on MobileNetV2.Source: Does Selection Survive a Cost… · p. 6

Does the Theory Predict the Measurement?

  • The symmetric-corruption identity's predicted and measured slopes differ by at most 0.016, and validation accuracy attenuates by the same factor as Brier alignment; this is not a performance guarantee.The label-dependent component is multiplied by a common factor when coefficient sums are candidate-invariant; the teacher component is unchanged to numerical precision.Source: Does the Theory Predict the Measurement? · p. 7, Table 5 · p. 7

5 Conclusion

For a fixed shared-teacher family, existing label-free estimators can fail, while anchoring combines stable distortion with supervised evidence. Anchoring reduces regret at the smallest label budget, but its advantage is bounded by anchor strength and label regime.

5 Conclusion

  • Minimum teacher distortion selects the same eight-bit configuration in all 134 families, while confidence-based estimators can rank overconfident collapsed predictors from the wrong end.No evaluated estimator detects which regime holds without target labels.Source: §5 · p. 7
  • With ten target labels, anchoring reduces mean regret below direct validation in all four settings and under corruption in every budget–rate cell.The identities characterize the score components but do not bound selection regret.Source: §5 · p. 7
  • Anchoring is a general construction rather than a teacher-specific property; a stronger anchor improves the anchored selector.With plentiful but unreliable labels, a bounded accuracy criterion can outperform anchoring on MobileNetV2.Source: §5 · p. 7

Supplementary Material

The supplement collects notation, proofs, protocol details, and supporting analyses. Its formal results give a local smoothness interpretation and exact identities for a canonical quadratic analogue of the candidate family.

Supplementary Material

  • The supplement organizes notation and supporting material into proofs, protocol, calibration and sensitivity checks, additional analyses, degradation regimes, and excluded-run documentation.Table S1 collects notation used throughout.Source: Supplementary Material · p. 10

Smoothness interpretation

  • The smoothness bound is only a local structural interpretation: it applies when distortion and perturbation terms share inputs, not to the implemented score, and is numerically weak at the chosen probability floors.The cross-entropy gradient's Lipschitz constant grows with the inverse probability floor.Source: Smoothness interpretation · p. 10

Exact quadratic identity

  • The quadratic identity holds to 1.3×10−15 across 90 CNN runs; with λ = 2 and shared inputs, its criterion selects the same candidate as empirical Brier validation.The teacher term is candidate-independent, so the identity is presented as a consistency check, not a new scoring rule.Source: Exact quadratic identity · p. 10

Observed-label and teacher components

  • The canonical fully quadratic teacher-only criterion favors the candidate with the smallest average squared predictive norm, but this does not establish equivalence to the empirical KL-based ablation.The ablation retains KL distortion and a selected coefficient.Source: Observed-label and teacher components · p. 10

Attenuation under symmetric corruption

Under symmetric label corruption, a criterion linear in the label indicator has a common attenuation factor when its coefficient sums are candidate-invariant. This condition holds for teacher contrasts and accuracy, but not cross-entropy.

Attenuation condition

  • When coefficient sums are candidate-invariant, symmetric corruption multiplies candidate comparisons for a label-linear criterion by one common factor; this holds for teacher contrasts and accuracy, but not cross-entropy.Teacher contrasts sum to zero and accuracy indicators to one, whereas cross-entropy's coefficient sum varies across candidates.Source: Attenuation under symmetric corruption · p. 10

Attenuation under symmetric corruption

  • For K = 30, the attenuation factor stays positive when η < 29/30 ≈0.967, so the label signal shrinks without inverting throughout the evaluated corruption range.Both slopes are verified empirically in Table 5 to within 0.016.Source: Attenuation under symmetric corruption · p. 10
  • Any criterion linear in the label indicator with candidate–teacher differences orthogonal to the all-ones vector, including empirical accuracy, has the same attenuation; noisy validation accuracy preserves clean accuracy ordering in expectation.Table 5 verifies both slopes on the same runs.Source: Attenuation under symmetric corruption · pp. 10–11

B Experimental Protocol

The protocol fixes a 72-candidate quantization family for each teacher and evaluates selectors on target pools while reserving disjoint test sets for evaluation. The supplement specifies run accounting, teacher training, candidate construction, and estimator calibration.

Runs, executions, and resampling units

  • Each run pairs one teacher with its fixed 72-candidate family; the two CNN settings pool three independent executions, yielding 45 runs per architecture, while each ViT setting has one execution.Source: Table S2 · p. 11
  • The candidate family is B × Q × G × E, yielding 6 × 3 × 2 × 2 = 72 configurations per teacher; quantization is weight-only, with endpoint layers optionally left in floating point.Per-channel and per-tensor granularity, bit widths, and clipping percentiles define the grid.Source: Runs, executions, and resampling units · pp. 11–12
  • The selection pool combines calibration and selection splits, but weight-based clipping means calibration data do not construct candidates; test sets are disjoint and used only after selection.On a 150-example half-pool, distortion ordering between the two 8-bit per-channel unclipped candidates can reverse.Source: Runs, executions, and resampling units · p. 12
  • CNN teachers use ImageNet transfer learning, while ViT teachers use continual distillation; seeds 0, 1, and 2 index repetitions.The passage specifies training schedules, optimizers, batch sizes, and distillation temperatures for the two teacher families.Source: Runs, executions, and resampling units · p. 12
  • ATC and DOC are calibrated on the target calibration split using the teacher's arg-max as reference; Li et al. (2026) is excluded because it requires an external foundation model.SoftmaxCorr and COT use a uniform class prior, permitted by balanced target splits.Source: Runs, executions, and resampling units · p. 12

Selector definitions and coefficient selection

Anchored selection reuses a label-free teacher-distortion score and cross-validates the weight of a supervised correction. The protocol specifies its coefficient search, resampling units, and the scope of its empirical comparisons.

Selector definitions and coefficient selection

  • The anchor is computed once on the complete pool and reused across budgets and folds; only the supervised term uses labeled examples, and no candidate is retrained or adapted.The held-out test set is never touched during selection.Source: Selector definitions and coefficient… · p. 12
  • Cross-validation selects the coefficient by held-out supervised loss, using five folds for n ≥25 and leave-one-out below that budget.The selected coefficient is then used to choose a candidate on all of S.Source: Selector definitions and coefficient… · p. 12
  • The coefficient grid includes zero, allowing reversion to pure distortion, while large α approaches direct validation; every mechanism variant uses the same coefficient-selection procedure.At αmax = 64, the selector returns the direct-validation candidate in 98.0%, 95.3%, and 100% of outcomes at ten, fifty, and all labels, respectively.Source: Selector definitions and coefficient… · p. 13
  • Formal inference averages the three CNN executions within each shift–seed configuration, leaving 15 paired units per architecture; subsets and corruption realizations are not independent units.Descriptive summaries average repeated outcomes within each family-run.Source: Selector definitions and coefficient… · p. 13
  • Between-shift variation accounts for 90% of total variance on ResNet50 and 74% on MobileNetV2 for CE-combo at ten labels, indicating substantial benchmark heterogeneity.These estimates group the 45 runs by target shift.Source: Selector definitions and coefficient… · p. 13
  • CNN bootstrap intervals reflect approximately five effective shift clusters per architecture, and the 15-unit sign-flip test has limited power; per-setting non-significance is therefore inconclusive.The paper treats per-setting values as primary evidence.Source: Selector definitions and coefficient… · p. 13

C Calibration and Candidate-Family Sensitivity

Validation cross-entropy selects different candidates from validation accuracy on ResNet50, while distortion remains poor even in restricted candidate families. Changing the anchor input set has little effect, but optimal-transport anchoring helps MobileNetV2 and worsens ResNet50.

C Calibration and Candidate-Family Sensitivity

  • On ResNet50, validation cross-entropy selects candidates with lower ECE and cross-entropy but lower top-1 accuracy; on MobileNetV2 it lowers both with a small positive accuracy difference.This explains why validation accuracy fails as a corruption-robust criterion on ResNet50: the relevant quantity is invisible to a 0–1 loss.Source: C Calibration and Candidate-Family… · p. 13
  • Distortion remains poor on ResNet50 among high-precision candidates and the top 20 by distortion, with median regret 0.471 in the low-precision family.The difficulty is not an artifact of one catastrophic low-bit candidate.Source: C Calibration and Candidate-Family… · p. 13
  • The protocol uses a fixed teacher and its 72-candidate family, and all supervised selectors receive the same labeled sample, including labels used for coefficient selection.The largest label budget is listed separately from the selection-pool size.Source: Table S4 · p. 14, Table S3 · p. 14

Anchor placement and anchor choice

Clipping and metric sensitivity

The checks show that distortion regret is insensitive to the probability floor, while confidence-based estimators also fail on accuracy regret, the metric they were designed to rank.

Clipping and metric sensitivity

  • Changing the probability floor across three values spanning six orders of magnitude moves distortion regret by at most 0.0003.The reported check attributes the large regret magnitudes to the candidate families rather than clipping.Source: Clipping and metric sensitivity · p. 14
  • Paired run-level differences are tested with two-sided sign-flip tests and Holm adjustment across comparisons within each row.The procedure uses complete enumeration for at most 22 paired units and 200,000 Monte Carlo draws otherwise.Source: Table S5 · p. 14
  • On accuracy regret, the anchor incurs 0.003 and 0.008 while average confidence incurs 0.318 and 0.310, respectively across the two settings.The confidence estimator's failure therefore is not limited to evaluating it outside its intended objective.Source: Clipping and metric sensitivity · p. 14

D Additional Experimental Results

Additional experiments examine label budgets, corruption, anchor choices, coefficient behavior, directional controls, and constrained candidate families. Anchoring's advantage is strongest at the smallest clean-label budgets, while other checks show settings where label-free rules or alternative anchors perform better.

Anchor placement and anchor choice

  • Table S7 reports mean regret over the first complete execution for each CNN setting, using 15 runs per architecture; columns vary the teacher-distortion input set or replace the anchor with a rescaled optimal-transport estimator.Source: Table S7 · p. 15

Metric and clipping checks

  • Across three probability floors, distortion’s cross-entropy regret magnitudes do not depend on clipping; on accuracy regret, the confidence estimator fails on its own metric.Source: Table S8 · p. 15

Budget sweep and upper tail

Paired inference and coefficient stability

  • Under 40% corruption on MobileNetV2, the coefficient reaches λmax in 53% of runs; on ViT–CIFAR-20 at η = 0.4, the supervised selectors choose the same candidate in every run.Mean regret decreases with λ on both DomainNet CNN settings, steeply on ResNet50 between roughly λ = 4 and λ = 8, then more gradually.Source: D Additional Experimental Results · p. 15, Table S11 · p. 16, Figure 3 · p. 17

Corruption at the complete pool

The directional term and its control

  • The permutation control tracks distortion within 0.025 everywhere; the teacher component is best on ResNet50 at ten labels and worst elsewhere at that budget.The implemented control permutes whole teacher-residual directions across labeled examples, rather than labels alone.Source: D Additional Experimental Results · p. 16, Table S13 · p. 17

Deployment cost

E Degradation Regimes and the Failure of Confidence-Based Estimators

Confidence-based estimators fail in the CNN degradation regime, where accuracy and confidence are uncorrelated or inversely related and collapsed candidates can be more confident than the best candidate. The evidence supports the regime distinction but not a causal mechanism, and the reported cohort excludes incomplete or structurally unusable runs.

E Degradation Regimes and the Failure of Confidence-Based Estimators

  • Across CNN settings, accuracy and mean confidence are uncorrelated or inversely related, and the worst candidate is more confident than the best; on ViTs the relationships reverse.Confidence-based estimators select chance-accuracy candidates in most runs of each CNN architecture and in none on the ViTs.Source: E Degradation Regimes and the Failure… · p. 17, Table S16 · p. 19
  • On ResNet50, collapsed candidates have only 34% distinct confidence values and reach 1.000; on MobileNetV2, calibration roughly halves median regret from 10.81 to 4.89.On MobileNetV2, ATC-MC and ATC-NE still select chance-accuracy candidates in 39 and 35 of 45 runs.Source: E Degradation Regimes and the Failure… · p. 18
  • Collapse concentrates at two and three bits across all four settings, while granularity effects at intermediate widths vary by architecture.MobileNetV2 also collapses at four and five bits; reported regrets use the best candidate actually attained.Source: E Degradation Regimes and the Failure… · p. 18, Table S17 · p. 19
  • The proposed normalization explanation remains untested because architecture and teacher provenance are confounded, and the BatchNorm account does not distinguish the two CNNs.The paper therefore reports the phenomenon and its consequence while leaving the mechanism open.Source: E Degradation Regimes and the Failure… · p. 18
  • Teacher distortion is unaffected by the confidence-collapse regime because collapsed candidates remain far from the teacher's output distribution; identifying the regime requires target-label diagnostics.The diagnostics include worst-candidate accuracy and the accuracy–confidence correlation.Source: E Degradation Regimes and the Failure… · p. 18

F Cohort Inclusion Criteria

  • The final cohort contains 45 runs per CNN, 17 on ViT–DomainNet, and 27 on ViT–CIFAR-20; one incomplete 47-candidate ViT–DomainNet run was excluded.Runs were also excluded for incomplete families, uniform outputs across candidates, or all candidates at chance accuracy.Source: F Cohort Inclusion Criteria · pp. 18–19
Loading 2609.31155v1…