Source-linked AI summary
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
Mike Ranzinger, Jon Barker, Greg Heinrich, Pavlo Molchanov, Bryan Catanzaro, Andrew Tao
TL;DR
Feature losses can weight teachers implicitly because their activation scales differ, motivating closer study of loss choice and normalization in label-free multi-teacher distillation. The paper evaluates statistical normalization and Hadamard-based approaches, including PHI-S. PHI-S standardizes dimensions with a uniform scale and produces the best student models across the evaluation suite.
Problem
Teacher activation variances differ, and scale-sensitive MSE and Smooth-L1 losses can implicitly weight teachers, motivating study of feature-distillation loss choice.
Source: §1 · p. 2Method
The authors study standardization, whitening, rotation matrices, and Hadamard-based normalization, including PHI-S, which uses a uniform scale across dimensions.
Source: §1 · p. 2Results
PHI-S produces the best student models across the evaluation suite.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that students can remain robust to potentially high-distortion transforms, with roughly uniform per-channel estimation errors for HCA and PHI-S.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Various visual foundation models have distinct strengths and weaknesses, both of which can be improved through heterogeneous multi-teacher knowledge distillation without labels, termed "agglomerative models." We build upon this body of work by studying the effect of the teachers' activation statistics, particularly the impact of the loss function on the resulting student model quality. We explore a standard toolkit of statistical normalization techniques to better align the different distributions and assess their effects. Further, we examine the impact on downstream teacher-matching metrics, which motivates the use of Hadamard matrices. With these matrices, we demonstrate useful properties, showing how they can be used for isotropic standardization, where each dimension of a multivariate distribution is standardized using the same scale. We call this technique "PHI Standardization" (PHI-S) and empirically demonstrate that it produces the best student model across the suite of methods studied.
1 INTRODUCTION
Agglomerative models distill heterogeneous visual foundation models without labels, but teacher activation differences can make feature losses weight teachers implicitly. The section motivates studying normalization and Hadamard matrices, culminating in PHI Standardization (PHI-S).
1 INTRODUCTION
- Agglomerative models fuse heterogeneous visual foundation models through label-free multi-teacher distillation; Theia and UNIC use feature standardization.AM-RADIO uses equal teacher weights and combines Cosine and Smooth-L1 objectives for feature matching.Source: §1 · pp. 1–2
- SAM’s activation standard deviation is 191× DFN CLIP’s, so scale-sensitive MSE and Smooth-L1 can implicitly weight teachers differently.The section asks whether AM-RADIO’s feature-distillation loss is optimal.Source: §1 · p. 2
- The study examines teacher distributions from Ranzinger et al. (2024) and adds SigLIP to the set of teachers studied.Theia and UNIC apply feature standardization and demonstrate its importance.Source: §1 · pp. 1–2
- The authors use standardization and whitening to assess effects on downstream metrics, and study how rotation matrices affect student errors.They identify distribution orientation as a potential influence on those errors.Source: §1 · p. 2
- The authors study Hadamard matrices for whitening and standardization, introducing PHI-S as a uniform-scale standardization method.Their contribution is presented as an application of Hadamard matrices to both normalization approaches.Source: §1 · p. 2
- PHI-S standardizes dimensions using a uniform scale and produces the best student models across the evaluation suite.For ViT-B/16, standardization techniques work best, with PHI-S the strongest normalization method studied.Source: §1 · p. 2, Table 1 · p. 3
2 METHOD
The method studies loss choices and invertible normalization for matching heterogeneous teachers, then uses Hadamard matrices to create balanced transforms. PHI-S standardizes every dimension with a uniform scale and is reported to produce the best student models across the evaluation suite.
2.1 BASELINE
- The baseline uses MSE and compares cosine distance and AM-RADIO’s cosine-plus-Smooth-L1 hybrid, with a cosine-plus-MSE variant also considered.Cosine loss is expected to match distributions poorly because it is magnitude-invariant, though its downstream effects are uncertain.Source: §2.1 · p. 3
- The study examines AM-RADIO’s β = 0.9 setting and the interpolation endpoints, where β = 1.0 is cosine loss and β = 0.0 is MSE loss.The authors do not search for the optimal β.Source: §2.1 · pp. 3–4
2.2 NORMALIZATION
- Rather than weighting teacher losses, the method applies invertible linear mappings to teacher targets, preserving the ability to estimate their true distributions.Global standardization uses one mean and standard deviation across the feature map; regular standardization uses channelwise statistics.Source: §2.2 · p. 4, §2.2.1 · p. 4
2.2.2 WHITENING
- Whitening removes feature covariance as well as variance differences; its valid transforms include ZCA and PCA-W, while the chosen rotation affects student-error profiles.The whitening losses use MSE on transformed teacher features.Source: §2.2.2 · pp. 4–5, §2.2.3 · p. 5
2.2.4 HADAMARD WHITENING (HCA)
- Hadamard whitening distributes variance evenly across dimensions, so equal-sized errors on any single dimension have identical magnitude in the original space.The Hadamard rotation is used as the whitening matrix Q.Source: §2.2.4 · pp. 5–6
2.2.5 PCA-HADAMARD ISOTROPIC STANDARDIZATION (PHI-S)
- PHI-S rotates a mean-centered distribution so every resulting dimension has equal variance, then applies a uniform scale to standardize it.The theorem states this is possible for mean-centered normal data when the Hadamard-matrix dimension is satisfiable.Source: §2.2.5 · pp. 6–7
2.2.6 VISUALIZING DISTRIBUTIONS
- PHI-S is rotation-invariant and isotropic, while regular standardization depends on orientation; standardization error magnitudes lie between PCA-W and PHI-S, with whitening errors differing by rotation.For PHI-S, the scale change is uniform; the error-bound comparison has equality at the specified covariance conditions.Source: Figure 4 · p. 7, §2.2.6 · p. 8
- PHI-S achieves the best job simultaneously minimizing all teachers’ mean squared errors for a ViT-B/16 student.Table 3 compares algorithms for matching the teachers.Source: Table 3 · p. 8
3 IMPLEMENTATION DETAILS
The implementation adapts AM-RADIO’s training procedure to reduce computational cost and enable the ablations, while adding SigLIP as a teacher.
3 IMPLEMENTATION DETAILS
- The authors modify AM-RADIO’s procedure to reduce training cost, which they say was necessary to cover the studied ablations.The implementation also adds SigLIP as a teacher.Source: §3 · p. 8
- The implementation adds SigLIP as a teacher and trains students at 256px resolution for 300k rather than 600k steps.Teacher features are downsampled to match the student resolution, except SAM features, which are padded and cropped because downsampling degrades their quality.Source: §3 · p. 8
- Each teacher receives a separate partition and a batch of 256 images, giving 1024 images per iteration.The method initializes from pretrained TIMM ViT-B/16 or ViT-L/16 patch-16 224 models.Source: §3 · p. 8
4 RESULTS
PHI-S has the lowest average benchmark rank across the evaluation suite for both student architectures. It also performs best overall in teacher matching, with comparatively uniform errors across channels and teachers.
4 RESULTS
- PHI-S best matches teacher distributions overall, beating other methods on DFN CLIP and SigLIP, ranking second on DINOv2, and remaining competitive on SAM.The baseline performs much worse because SAM’s larger activation variance draws most of its matching effort.Source: §4 · p. 8
- The normalization visualizations compare how transforms redistribute errors, including PCA-W’s emphasis on the highest-variance axis and PHI-S’s isotropic error scaling.For PHI-S, the figure shows errors maintained along a circle in normalized and denormalized spaces.Source: Figure 5 · p. 9, Figure 7 · p. 10
- PHI-S achieves the lowest average benchmark rank across the suite for both ViT-B/16 and ViT-L/16 students.Table 4 reports average benchmark ranks for ViT-L/16; the authors state the same result for both architectures.Source: Table 4 · p. 10, §4 · p. 10
4.1 EMPIRICAL ERRORS
- Hadamard-based methods have low channelwise error-variance ranges, while PHI-S also has the lowest or nearly lowest range after denormalization.The authors report that HCA’s uniform error property does not translate into useful benchmark performance; PHI-S retains nearly uniform normalized error variance.Source: §4.1 · p. 11
- PHI-S and HCA have the tightest error bounds between channels, and PHI-S has the most similar loss distributions across teachers.Global Standardization remains susceptible to outlier errors, unlike PHI-S’s variance-equalizing behavior.Source: §4.1 · p. 11
5 RELATED WORK
The work builds on label-free multi-teacher distillation and prior feature-normalization research, examining how normalization choices affect matching heterogeneous teacher distributions. It presents PHI-S as an alternative to regular standardization used in related models.
5 RELATED WORK
- Agglomerative models distill features from heterogeneous visual foundation models without ground-truth labels, unlike many multi-teacher methods with unified targets.AM-RADIO is identified as a model in this setting; Theia, UNIC, and UNIT also build on it.Source: §5 · pp. 12–13
- Theia and UNIC use feature standardization, which this work reports as an improvement over AM-RADIO but not as effective as PHI-S.The authors describe their study as complementary to those works.Source: §5 · p. 13
- Prior work highlights feature normalization in distillation, while the authors focus on invertible techniques so the student can estimate the teacher’s true distribution.They extend normalization’s role to controlling relative magnitudes across teachers.Source: §5 · p. 13
- The authors study Hadamard matrices for whitening rotations and introduce PHI-S as a scalar normalization for multivariate distributions with a known Hadamard matrix.PHI-S is described as standardizing with a uniform scale across dimensions.Source: §5 · pp. 13–14
6 CONCLUSION
PHI-S is the best normalization method studied, with isotropic methods strongest at ViT-B/16 and PHI-S best at ViT-L/16. The conclusion also identifies standardization’s advantage over whitening and possible future uses.
6 CONCLUSION
- Plain MSE without balancing substantially hurt student quality, while standardization outperformed whitening, which the authors hypothesize is unstable for non-full-rank teacher distributions.Standardization avoids degenerate normalization factors because distribution principal components are spread across dimensions.Source: §6 · p. 14
- At ViT-B/16, Global Standardize and PHI-S worked best; at ViT-L/16, PHI-S remained best and produced models competitive with AM-RADIO’s ViT-H.Overall, PHI-S appears to be the best normalization method studied.Source: §6 · p. 14
- Whitening methods showed no significant downstream-metric result for reconstruction errors, and per-channel estimation errors were generally nonuniform unless uniform estimation was chosen.The authors report student robustness to the potentially high-distortion transforms.Source: §6 · p. 14
- The authors identify PHI-S as a general standardization technique under assumptions such as distribution normality and dimensionality, and suggest post-hoc use or combination with quantization.The proposed quantization combination is framed as an opportunity that could potentially improve fidelity.Source: §6 · p. 14
A.1.1 CONSTRUCTING HADAMARD MATRICES
The appendix describes ways to construct Hadamard matrices for feature dimensions used in the studied models, including dimensions that are not powers of two.
A.1.1 CONSTRUCTING HADAMARD MATRICES
- Sylvester’s construction directly covers sizes that are powers of two, while the Hadamard Conjecture proposes valid matrices for every size divisible by four.The appendix notes that many relevant feature dimensions are not powers of two.Source: §A.1.1 · p. 19
- The appendix uses Kronecker products and Paley constructions alongside Sylvester matrices to construct additional Hadamard sizes.It gives Paley rules for sizes q + 1 and 2(q + 1) under the stated prime-power congruence conditions.Source: §A.1.1 · p. 19
- The appendix lists constructions for ViT-B (768), SigLIP-L (1152), ViT-H (1280), and ViT-g (1408) feature dimensions.Sylvester and Paley constructions were sufficient to cover the models studied.Source: §A.1.1 · p. 19
A.1.2 USING HADAMARD MATRICES FOR NOISE SUPPRESSION / QUANTIZATION
The appendix distinguishes QuaRot’s use of structured matrices for quantization from this paper’s use of Hadamard matrices for statistical normalization.
A.1.2 USING HADAMARD MATRICES FOR NOISE SUPPRESSION / QUANTIZATION
- QuaRot applies structured matrices to eliminate activation outliers.The passage describes this as a different application from the paper’s statistical normalization use.Source: §A.1.2 · p. 19
- Eliminating activation outliers makes low-bit quantization more effective in QuaRot.Source: §A.1.2 · p. 19
- QuaRot’s quantization application is described as unrelated to this paper’s use of Hadamard matrices for statistical normalization.Source: §A.1.2 · p. 19
A.2.1 PROOF OF HCA UNIFORM ERROR PROFILE
The appendix analyzes Hadamard-based error profiles, teacher-distribution statistics, and normalization behavior, then compares PHI-S across student sizes and balancing variants.
A.2.1 PROOF OF HCA UNIFORM ERROR PROFILE
- For a unit step along any single whitened-space axis, HCA and PHI-S have identical error magnitude after projection to the original space.The authors hypothesize that equal per-dimension effects may improve learning dynamics compared with PCA-W, which emphasizes the first dimension.Source: §A.2.1 · p. 20
A.4 TEACHER DISTRIBUTION STATISTICS
- Teacher activations are not channel-centered, have widely varying global and per-channel scales, and have effective ranks much smaller than their channel counts.The reported statistics include SAM channel means from −62.0312 to 19.1719 and standard deviations from 2.6953 to 31.6094.Source: §A.3 · p. 20, Table 7 · p. 21, §A.4 · p. 21
A.5.1 ROLE OF PCA
- PCA is unnecessary to find PHI-S’s scale, but the orthogonal transform HU⊺ is needed to produce a dimensionally balanced distribution.The normalization scale is invariant to orthogonal transforms; the orientation U can prevent H from evenly distributing the variance in Λ.Source: §A.5.1 · p. 21
A.5.2 COMPARISON BETWEEN GLOBAL STANDARDIZATION AND PHI-S
- PHI-S uses a larger, distribution-specific scale than global standardization for the studied teachers; equal channel means would make their large-sample scales nearly equal.The appendix measures alignment with the ideal PHI-S basis using abs(HU⊺) and the Hungarian algorithm.Source: §A.5.2 · p. 22, §A.5.3 · p. 22
A.5.4 DEGENERATE RANK DISTRIBUTIONS
- PHI-S remains well-defined when the effective rank is at least one, while whitening can distort scales sharply along zero- or negligible-variance dimensions.The authors note that vanilla standardization can also suffer, with sensitivity depending on the original distribution’s orientation.Source: §A.5.4 · p. 23
A.6 NORMALIZATION WITHOUT RUNTIME PENALTY
- Normalization’s mean subtraction and scaling can be folded into the model’s final linear layer, avoiding a runtime penalty while restoring outputs to the original teacher space.The reported PHI-S-RADIO models add two training stages, increasing resolution and then adding high-resolution partitions.Source: §A.6 · pp. 23–24, §A.7 · p. 24
A.8.1 ADAPTIVE BALANCING
- AdaLoss shifts weight toward summary losses, yielding classification gains at the expense of dense tasks; comparisons with other agglomerative models are confounded by differing settings.The cross-paper comparison reports original-paper numbers without replication, and notes differences in training data, resolution, teachers, and objectives.Source: §A.8.1 · p. 24, §A.9 · p. 25
A.8.3 VIT-L/16
- PHI-S is the strongest normalization method in benchmark ranks, and it best simultaneously minimizes teacher errors for the ViT-L/16 student.The AdaLoss comparison also finds AdaLoss+PHI-S better than AdaLoss alone.Source: §A.8.1 · p. 24, §A.8.3 · p. 25, Table 12 · p. 26