What Makes High-Magnification Knowledge Transferable? A Study of Cross-Resolution Distillation in Whole-Slide Imaging

1 INTRODUCTION

• Cross-resolution transfer seeks high-magnification supervision during training while keeping low-magnification inputs at deployment, where 20× processing requires sixteen times as many patches as 5×.A regional mean averages high-magnification features within a low-magnification field, while spatial targets retain within-field variation.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The study asks whether teacher targets help the task, low-magnification inputs can predict them, and slide models benefit from those predictions.It uses formal analysis and matched experiments across five classification and grading cohorts and five survival cohorts.Source: §1 · p. 2
• The paper decomposes task risk and uses controlled experiments to examine teacher content, prediction, and delivery separately.This design addresses the limited understanding of when high-magnification supervision is transferable.Source: §1 · p. 2
• Observed regional teacher means improve average primary scores when added to native features, but student predictions do not inherit that value automatically.Predictions underrepresent teacher-target diversity, better reconstruction does not consistently improve task scores, and retaining native features changes performance.Source: §1 · p. 2

2 RELATED WORK

• Knowledge distillation transfers teacher predictions, features, or attention; related approaches adapt guidance to student capacity, preserve student features, or predict teacher–student feature differences.Cross-resolution transfer additionally changes what information is available at deployment, connecting it to learning with privileged information.Source: §2 · pp. 2–3
• Earlier cross-resolution systems transfer high-resolution features, outputs, or class-specific attention, while recent methods align magnification-specific representations or combine regional, spatial, patch, and slide guidance.The cited component ablations assess particular systems rather than independently isolating all transfer stages.Source: §2 · p. 3
• Closer teacher reconstruction need not improve generalization, motivating separate tests of what representations contain, what restricted models can use, and whether native features remain useful.Prior work also evaluates both reconstruction and slide-task performance in spatial token compression.Source: §2 · p. 3

3 PROBLEM FORMULATION AND ANALYTICAL RATIONALE

• The pipeline maps high-magnification features to teacher targets, predicts them from low-magnification input, delivers native features and predictions to a task representation, and maps that representation to an outcome.Delivery may combine or replace native features; a model receiving only teacher targets is not equivalent to one receiving both native input and targets.Source: §3 · pp. 3–4
• The risk decomposition compares a trained model using its representation with the best rule using that representation, separating teacher access, representation loss, and model excess.If the representation retains the complete native input, representation loss is zero; comparisons of two such augmentations attribute their population-loss difference to model excess.Source: §3 · p. 3
• Teacher access measures the value of observing teacher targets alongside native input; representation loss measures task information discarded by the student representation, and model excess is the trained task model’s gap from the best rule using that representation.A deterministic student prediction can only transform information already present in the native input, though that transformation may ease learning for a finite model.Source: §3 · pp. 3–4
• The analysis evaluates reconstruction error separately from task performance because they describe different quantities.Its risk decomposition applies to classification log-loss and discrete survival NLL with common observation weights.Source: §3 · p. 4
• For a shared regional output trained with squared error, supervision against individual teacher features gives the same gradient as supervision against their regional mean.With 16 teacher vectors, sampling one uniformly preserves the expected gradient, though variability and training trajectory may change.Source: §3 · p. 4
• Under squared error, direct prediction and residual prediction have the same unrestricted optimum: the conditional mean of the teacher target given the available input.Differences in finite-network reconstruction can still arise from capacity, initialization, and optimization; the conditional mean is distinct from the observed regional mean.Source: §3 · p. 4

4 STUDY DESIGN

• Teacher-summary prediction is trained on paired 5× and 20× features without task labels, then frozen and evaluated on an unseen dataset with supervised slide-level learning.The pipeline optionally concatenates native 5× features with predicted teacher features.Source: Figure 2 · p. 5, §4 · p. 5
• The experiments span classification, grading, and survival prediction across ten datasets, using 5× and 20× patches and five random seeds.The cohorts include CAMELYON16/17, BRACS, PANDA, Private-CRC, and five TCGA cancer cohorts.Source: §4 · p. 5, Figure 3 · p. 5
• To test teacher usefulness, separate task models receive native features with a teacher mean, a mean plus spatial components, or zero padding.The comparison holds input width fixed and evaluates joint native and teacher access.Source: §4 · pp. 5–6
• Direct and residual predictors estimate regional teacher means from a native feature and its slide mean, with zero or random final-layer initialization compared.Direct prediction estimates the target; residual prediction estimates a correction added to the native feature.Source: §4 · p. 6
• A learned predictor of regional means and spatial components is compared with a constant training-target average to test prediction beyond that average.Mean and spatial reconstruction errors are reported separately on datasets excluded from predictor training.Source: §4 · p. 6
• Native-feature retention is tested by comparing [xi; qi] with [0; qi] while keeping each prediction fixed, and adapted encoders are separately evaluated under alternative teacher targets.The encoder-adaptation study uses conditional flow matching and discards both training heads before Stage 2 task evaluation.Source: §4 · p. 6

5.1 IS A HIGH MAGNIFICATION TARGET USEFUL?

• Adding the regional 20× mean to native 5× features increases mean primary task scores, supporting the mean as a useful transfer target.The comparison uses teacher summaries added to native features.Source: Table 1 · p. 7, §5.1 · p. 7
• Adding spatial components lowers mean primary scores in seven of ten datasets, while changing token count or teacher content also changes task performance.Individual teacher features generally improve classification over native input, but individual features do not consistently outperform repeated means at matched token count.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 CAN LOW-MAG PREDICT THE TARGET?

• Direct prediction has lower MSE than residual prediction on every dataset, although native or training-mean baselines remain more accurate for some datasets.The training-target mean remains more accurate on Private-CRC, PANDA, and UCEC, with nearly equal average error on LUAD.Source: Table 3 · p. 8, §5.2 · p. 8
• Predicted features do not cover the full diversity of teacher targets, and their spread differs from their accuracy against matched regional targets.In original coordinates, direct outputs have lower centered variance than teacher means, while residual outputs have higher variance in every dataset.Source: §5.2 · p. 8, Figure 4 · p. 8
• Reconstruction gains usually concentrate in the regional mean; spatial-component predictions remain less accurate than their constant baselines.The pattern also appears on independent evaluation groups from the nine distillation datasets.Source: §5.2 · pp. 8–9

5.3 DOES DOWNSTREAM MODEL BENEFIT FROM PREDICTION?

• With teacher predictions held fixed, retaining native features gives more consistent primary-score gains for direct prediction, while NLL often improves for both forms.Native retention allows recovery of the complete native bag, but the task difference cannot be attributed solely to representation loss.Source: §5.3 · p. 9, Figure 6 · p. 9
• Changing the teacher target from individual features to regional means leads to distinct encoder outcomes under the same adaptation procedure.The target choice also determines the network path input under the flow objective, changing the training signal.Source: §5.3 · p. 9

6 DISCUSSION AND CONCLUSION

• Across ten cohorts, observed regional teacher means improve average primary scores with native features, but student predictions do not automatically inherit that value.Better reconstruction does not consistently improve task scores, and retaining native features changes performance at fixed predictions.Source: §6 · pp. 9–10
• Direct prediction has lower average reconstruction error than residual prediction, while its outputs vary less than teacher targets.The authors recommend separately testing teacher usefulness, prediction beyond a training-target constant, and downstream benefit.Source: §6 · p. 10
• The study covers one magnification pair and selected model families, with incomplete patient linkage and sparse survival events further constraining inference.Target coordinates, sampled regions, and input widths also differ across experiments, limiting comparisons between them.Source: §6 · p. 10
• Table 4 compares encoder performance under different auxiliary teacher targets; both flow models retain mean KD and supply only encoder features to TransMIL.The passage states the comparison scope but does not report which target performs better.Source: Table 4 · p. 10
• The supplementary materials include code, numerical exports, and reproduction scripts, while Appendix B.7 specifies remaining requirements for reproducing all model fits and intervals.Source: REPRODUCIBILITY STATEMENT · p. 11

A DEFINITIONS AND PROOFS

• The appendix provides proofs for the three study questions and squared-error identities clarifying shared regional outputs and prediction recovery.Source: A DEFINITIONS AND PROOFS · p. 18

A.1 INFORMATION ACCESS AND TASK RISK

• Teacher access, representation loss, and model excess distinguish the value of joint teacher information, task information lost in delivery, and a trained model’s gap to unrestricted risk.Source: §A.1 · p. 19
• Concatenating native input B with a prediction makes representation loss zero, while residual sums or replacements may retain task-relevant information without recovering all of B.Source: §A.1 · p. 19
• Equal information can still yield different finite-model performance because architecture, data, estimation, optimization, and model selection contribute to model excess.Source: §A.1 · p. 19
• The decomposition applies to observation-wise losses; losses coupling observations through risk sets need a joint formulation, and AUC, C-index, macro F1, and quadratic κ remain separate evaluation criteria.Source: §A.1 · p. 19

A.2 MEAN SUPERVISION FOR A SHARED REGIONAL OUTPUT

• For a shared output compared with teacher vectors in one region, the variation term is independent of the prediction, so both squared-error objectives yield gradient 2(q −m).Source: §A.2 · p. 20
• The gradient equivalence assumes fixed teacher features, uniform within-region weights, common coordinates, and shared outputs; separate positional outputs, other losses, and jointly learned teachers fall outside it.Source: §A.2 · p. 20

A.3 DIRECT AND RESIDUAL PREDICTION UNDER SQUARED ERROR

• Under squared error, the unique optimal direct prediction is the conditional mean of the target given B, and reconstruction error separates conditional variation from error in that prediction.Source: §A.3 · pp. 20–21
• Unrestricted direct and residual predictors can produce exactly the same outputs, with the residual optimum equal to the conditional mean minus the native input.Source: §A.3 · p. 21
• The conditional mean’s centered variation cannot exceed the teacher target’s, so reduced feature variation is compatible with optimal squared-error prediction.Source: §A.3 · p. 21
• Observed reconstruction error combines conditional target variation and predictor error, so it alone cannot separate those contributions or establish representation loss or model excess.Source: §A.3 · p. 21
• Population shifts can change the conditional mean and variation, and poor performance by a trained predictor alone does not establish an information limit in B.Source: §A.3 · p. 21

B DATA AND COMPARISON DESIGN

• Prediction and transfer studies use Stage 1 distillation followed by Stage 2 task training, whereas teacher comparisons fit task models directly on observed features.Source: B DATA AND COMPARISON DESIGN · p. 22

B.1 DATASETS AND TASK PARTITIONS

• Sparse test classes and survival events limit the precision of task comparisons across the ten datasets.The study uses five classification and grading datasets and five cancer-specific TCGA overall-survival cohorts; all evaluations use internal holdouts.Source: §B.1 · pp. 22–23

B.2 PHYSICAL FIELDS AND FROZEN FEATURES

• The 5× and 20× patches share pixel dimensions and stride, but physical patch extent depends on slide metadata.The 5× patch aligns to sixteen 20× positions, and complete fields outside the slide are excluded; earlier cached and newly extracted features can differ for the same region.Source: §B.2 · pp. 23–24

B.3 TOKEN ORDER AND RECIPIENT PROCESSING

• TransMIL processes serialized token order rather than slide coordinates, and comparing repeated means with a single mean changes sequence length and processing together.Individual teacher features and repeated means share token count, order, padding, and positional processing.Source: §B.3 · p. 24

B.4 PATCH COUNTS AND REPRESENTATION SIZE

• Augmented inputs reduce token count sixteen-fold versus individual teacher features, but feature-entry reductions are only 8-fold or about 2.91-fold.The projected and full-channel augmentations use one token of width 3,072 and 8,448, respectively; these dimensions do not measure runtime or memory use.Source: §B.4 · p. 24, Figure 7 · p. 25

B.5 CROSS-RESOLUTION DISTILLATION AND DOWNSTREAM FITTING

• Leave-one-dataset-out distillation excludes the evaluated dataset from Stage 1 predictor training, transformation estimation, and selection, then fits its task model in Stage 2.Task test outcomes select neither stage.Source: §B.5 · pp. 24–25

B.6 COMPARISON DESIGNS

• The comparison designs vary teacher content, token count, augmentation, prediction, and native-feature retention while matching support and training conditions within contrasts.The native-retention experiment holds predictions fixed and compares inputs with or without the native block.Source: §B.6 · pp. 26–27

B.7 EXPERIMENTAL VARIATION AND REPRODUCIBILITY

• Five realizations vary different factors by study, and overlapping test sets mean their sample standard deviations do not represent five independent samples.Reported means and SDs describe observed comparisons but do not establish population improvement or equivalence.Source: §B.7 · p. 27

C.1 THE MEAN SUMMARY IN THE TRAINING DIAGRAM

• The paper’s 768-coordinate mean-summary example appends a random-feature map of the same regional mean, adding no independent teacher observation.The main teacher comparison instead uses DCT components, while raw mean prediction uses all 1,536 original coordinates.Source: §C.1 · p. 29

C.2 THE FOUR SPATIAL COMPONENTS

• LF4 retains the regional mean and three broad spatial changes in embeddings, which are predicted from 5× inputs after projection and normalization.The components are spatial frequencies of embeddings, not image pixels.Source: §C.2 · p. 29

C.3 SPATIAL FREQUENCY OF THE TEACHER FEATURES

• In original teacher features, DC holds 62.56% of total energy, while three nonconstant LF4 modes retain 35.84% of post-DC energy versus 20.06% after permutation.The summaries describe sampled feature banks, not a uniform patient sample or the exact downstream test population.Source: §C.3 · p. 30, Figure 8 · p. 30

C.4 SEPARATING REGIONAL CONTENT FROM TOKEN COUNT

• Teacher means do not consistently outperform native features as replacements, and individual teacher features sometimes outperform repeated means without making the mean a better replacement.Repeating a mean changes task scores without adding feature information, so finite-model processing may contribute to those differences.Source: §C.4 · p. 31, Table 7 · p. 31

C.5 TEACHER INFORMATION ALONGSIDE THE SAME NATIVE INPUT

• Appending the teacher regional mean raises average primary scores in every dataset, while its NLL effect is less consistent and spatial-component NLL effects are mixed.The augmentation measures complementary utility for this training procedure but does not separate teacher-information contribution from model-excess changes.Source: §C.5 · pp. 31–32, Table 8 · p. 32

C.6 SPATIAL UTILITY WHEN ALL FEATURE CHANNELS ARE RETAINED

• With all channels retained, neural spatial predictions have mixed task effects, and selection-group spatial-head errors remain above the constant reference.Those errors describe checkpoint selection, not independent generalization; independent component errors are unavailable for these heads.Source: §C.6 · p. 33

C.7 AN ALTERNATIVE WAY TO DELIVER SPATIAL SUMMARIES

• Full-channel spatial summaries improve average task performance in some datasets and reduce it in others, while permuting teacher positions can change the ordering.The results do not establish a common benefit from preserving spatial order.Source: §C.7 · p. 33

C.8 REGIONAL MEANS ACROSS TASK MODELS

• All four task models favor regional means on average for CAM16 and PANDA, while results vary elsewhere.The comparison uses each model’s native input as its reference; five initializations share one data split.Source: §C.8 · p. 34, Table 13 · p. 36

D NEURAL PREDICTION OF TEACHER TARGETS

• Regional-mean prediction is evaluated on independent fields using selected direct and residual heads, with native features and a training-target constant as references.The excluded-dataset test samples contain 231 to 256 fields, and no task labels enter these measurements.Source: D NEURAL PREDICTION OF TEACHER TARGETS · p. 37, §D.1 · p. 37

D.1 REGIONAL MEAN PREDICTION ON INDEPENDENT TEST FIELDS

• All four predictors improve on the constant across independent distillation-dataset samples, but generalization to excluded datasets varies; direct prediction with zero initialization beats the constant in seven dataset averages.Residual prediction can also approximate targets more accurately than native features while remaining less accurate than the constant.Source: §D.1 · p. 37, Table 14 · p. 37
• Direct prediction has lower MSE than residual prediction in every dataset average under both initializations.The unrestricted direct and residual functions share the same optimum, so the observed difference reflects the finite networks and their optimization.Source: §D.1 · p. 38

• After training, both prediction forms have lower MSE and higher pairwise-distance correlation in every dataset average, while their variation shifts in opposite directions.Direct variance increases and residual variance decreases; trained direct predictions remain less variable than teacher means, while residual predictions remain more variable.Source: §D.2 · p. 39, Table 16 · p. 40
• With zero-output initialization, direct MSE falls in eight dataset means but rises for Private-CRC and PANDA, while residual MSE falls in every dataset mean.The direct predictor starts at zero output; the residual prediction initially equals the native feature.Source: §D.2 · p. 39, Table 17 · p. 41
• Trained direct prediction improves MSE and pairwise-distance correlation relative to native input in every dataset average; residual MSE improves throughout with zero initialization and except for Private-CRC under random initialization.These comparisons show improvement over native input, not superiority to every predictor.Source: §D.2 · p. 39
• Lower reconstruction error and positive distance correlations do not identify representation loss or teacher access, and unmatched feature variation may be irrelevant to the task.The measurements use saved predictors on sampled fields; patch encoders remain frozen.Source: §D.2 · p. 39

D.3 TEACHER CORRESPONDENCE IN A COMMON FEATURE MAP

• The displayed maps use trained seed-42 predictions of regional means from native features plus slide context; residual outputs add a learned teacher-minus-native correction.The target is the arithmetic mean of sixteen aligned 20× features, and the patch encoders remain frozen.Source: §D.3 · p. 41
• In the KIRC slide, direct outputs have 0.5284 times the teacher’s centered variance versus 1.5041 for residual outputs, while distance correlations are 0.8816 and 0.9215, respectively.Direct prediction has slightly lower average reconstruction error across the corresponding targets, showing that spread and distance agreement describe different aspects.Source: §D.3 · p. 42, Table 18 · p. 43
• On all ten displayed slides, direct prediction has smaller variation and lower MSE than residual prediction, though residual variation is not always larger than teacher variation on individual slides.The visual patterns do not establish a common number of recovered clusters or a classification-versus-survival distinction.Source: §D.3 · p. 42, Table 18 · p. 43
• Map concentration is not calibrated variance or density in the original space, and native-versus-teacher separation cannot be attributed solely to magnification because their extraction histories differ.Original-coordinate measurements determine reconstruction accuracy and variation.Source: §D.3 · pp. 42–43

D.4 PREDICTING PROJECTED MEAN AND SPATIAL COMPONENTS

• The next diagnostic predicts four 192-coordinate target blocks jointly from a 2,304-coordinate concatenation of UNI2-h and DINOv3 features from one 5× region.It uses a different distillation partition and head from the raw analysis.Source: §D.4 · p. 48

• The mean head beats the constant on every independent distillation-dataset evaluation and in eight excluded-dataset averages, while every nonconstant spatial component has higher error than the constant in every realization on both populations.Observed errors combine conditional target variation and estimation error, so the spatial deficit does not identify an irreducible limit.Source: §D.5 · p. 48
• Because the four equal-width components are averaged in total MSE, prediction can improve through the mean while spatial error remains high relative to its own constant reference.The constant makes much more error per coordinate on the mean than on spatial components; this does not establish that loss weighting caused the spatial deficit.Source: §D.5 · p. 48

D.6 INDEPENDENT NEURAL ERROR CORRECTION

• 95.7% of pooled error reduction comes from the regional mean; correction lowers total error on independent groups, but spatial error rises in every excluded dataset.Correction improves the mean on average, with adverse mean effects on CAM16 and Private-CRC; the spatial deficit remains unresolved.Source: §D.6 · pp. 48, 51

D.7 SPATIAL ACCESS TO THE LOW MAGNIFICATION IMAGE

• Local spatial input lowers average feature error on five cohorts, but only CAM17 and KIRC gain on average task score, and KIRC gains despite worse reconstruction.These descriptive orderings do not establish a causal relationship between feature error and task performance.Source: §D.7 · p. 51, Figure 15 · p. 52

E DOWNSTREAM USE AND ATTRIBUTION TO DISTILLATION

• The appendix separates whether teacher targets help, students can predict them, and slide models benefit from those predictions.It also tests transfer by adapting the image encoder and deploying its features.Source: E DOWNSTREAM USE AND ATTRIBUTION TO… · p. 53

E.1 LEARNING THE NONCONSTANT SPATIAL COMPONENT

• Learning the spatial head lowers average NLL on PANDA, BRACS, KIRC, KIRP, LUAD, and UCEC, but raises it on CAM16, CAM17, Private-CRC, and STAD.Results vary across realizations; PANDA, KIRC, and LUAD improve in all five, while BRACS improves in three.Source: §E.1 · p. 53

E.2 GENERAL AUGMENTATION CONTROLS

• With native input retained, the initial and trained predictions are both functions of that input, so their population loss difference measures a difference in model excess.Zero padding keeps input width constant, while the observed teacher provides a separate reference with access to (B, T).Source: §E.2 · p. 53

E.3 REMOVING SPATIAL PREDICTIONS WHILE RETAINING THE SAME MEAN

• Omitting spatial prediction blocks has mixed effects: PANDA improves over the full prediction in all five realizations, while CAM16 and LUAD remain below the native reference.Removing a harmful component can improve an augmented model without establishing a general benefit from distillation.Source: §E.3 · pp. 53–54, Table 22 · p. 54

E.4 DIRECT AND RESIDUAL PREDICTION ACROSS TEN DATASETS

• Direct prediction has lower mean reconstruction error in every dataset under both initializations, yet residual prediction often performs better downstream; the preferences disagree in 35 of 50 zero-initialization and 28 of 50 random-initialization realizations.The task comparison uses standalone predictions; reconstruction error alone cannot rank the resulting task representations.Source: Figure 16 · p. 55, §E.4 · p. 55

E.5 NATIVE RETENTION WITH AN UNCHANGED TEACHER PREDICTION

• With predictions and reconstruction errors unchanged, native retention improves average primary scores more often for direct predictions, while NLL favors retention across most conditions of both forms.Neither retention nor teacher prediction consistently improves on the matched native reference; effects vary by task criterion and realization.Source: §E.5 · p. 56, Figure 17 · p. 57, Table 24 · p. 58

E.6 INDIVIDUAL FEATURES OR THEIR MEAN AS AUXILIARY SUPERVISION

• The final study adapts UNI2-h by distillation, freezes it for downstream tasks, and compares mean-KD and mean-KD-plus-flow encoders using the same downstream readout.For flow-trained encoders, deployment uses only adapted image features; the mean head and velocity field are discarded.Source: §E.6 · p. 59
• Individual-feature flow exceeds the frozen encoder on LUAD and STAD, but both flow encoders remain below it on KIRP; the prediction head can change the ordering relative to its encoder.The comparison changes the auxiliary target while retaining mean supervision, but differs from frozen-feature MSE prediction and cannot isolate the loss function.Source: §E.6 · pp. 59–60, Table 26 · p. 61
• Flow-target preference varies by dataset: individual-feature flow has higher test macro F1 on CAM17, Private-CRC, and BRACS, while regional-mean flow is higher on CAM16 and PANDA.The preferences reverse on average validation scores, and do not establish a rule for choosing targets by task category or recovery of the teacher-feature conditional distribution.Source: §E.6 · p. 60