Source-linked AI summary
Localizing Transfer Between Memorization Tasks
Yimiao Yu, Florentin Guth
TL;DR
The paper asks why pre-training can accelerate learning on another task and what learned features underlie transfer. It studies memorization tasks with controlled weight ablations, finding equivalent and non-equivalent transfer patterns and separating magnitude-driven from structure-driven effects.
Problem
Method
Results
Matched tasks show one-for-one epoch savings after warm-up; mismatched random-pixel pre-training can initially beat direct training, with transfer split into magnitude-driven FC and structure-driven convolutional effects.
Source: §4 · p. 6Takeaways & Limitations
Weight scale is an important confounder in transfer experiments and should be accounted for in future investigations.
Source: §3 · p. 4The study uses one 7-layer CNN and CIFAR-10, and its SVD experiments only partially localize structure-driven transfer.
Source: §4 · p. 6
Abstract
from arXiv · showhide
A central puzzle in transfer learning is why pre-training on one task can accelerate training or improve performance on another task, and what mechanisms underlie this transfer. In this work, we examine the transfer between memorization tasks of random input-output mappings. We find two surprising transfer patterns: equivalent transfer, where each additional pre-training epoch saves approximately one downstream fine-tuning epoch; and non-equivalent transfer, where pre-training on a mismatched task can be even more efficient than directly training on the downstream task itself. Through ablation experiments, we decompose and localize the transfer into two separate effects: a "trivial" magnitude-driven transfer in the last layer, and a "non-trivial" structure-driven transfer, partially attributable to the covariance of the other layers. These results advance our understanding of the underlying mechanisms of transfer learning and have the potential to lead to principled pre-training strategies.
1 Introduction
The paper asks when transfer occurs between tasks and how learned features are encoded in network weights, studying memorization to separate training dynamics from generalization. It identifies equivalent and non-equivalent transfer, tests which upstream task properties matter, and decomposes transfer into magnitude- and structure-driven effects.
1 Introduction
- The study isolates optimization by examining memorization tasks with randomized labels or image pixels, varying target functions and input distributions independently.The authors distinguish training dynamics from generalization measured by test loss or another performance metric.Source: §1 · p. 1
- Matched image distributions yield near one-to-one epoch savings after warm-up, while random-pixel pre-training across mismatched distributions can initially be more efficient than direct downstream training.The authors call these patterns equivalent and non-equivalent transfer, respectively.Source: §1 · p. 2
- Fixed upstream labels are essential for transfer, but fixed pre-training images are not, suggesting simpler pre-training strategies.Changing labels after each epoch prevents transfer; changing the pre-training images does not make data consistency necessary.Source: §1 · p. 2
- Ablations separate magnitude-driven transfer, attributed to growing weight scale, from structure-driven transfer, partially attributed to weight covariance.The authors localize these effects within network weights but do not fully account for the structure-driven component.Source: §1 · p. 2
2 Equivalent and non-equivalent transfer in memorization tasks
The experiments measure downstream fine-tuning time after pre-training on controlled memorization tasks with random labels. Transfer can persist without shared data, but its form and efficiency depend on the task distribution.
2 Equivalent and non-equivalent transfer in memorization tasks
- Transfer is measured by downstream fine-tuning time t0 to convergence as a function of pretraining time, with transfer defined by reduced t0 versus training from scratch.The study focuses on memorization rather than generalization.Source: §2 · p. 2
- Random-pixel pretraining beats the −1 reference over a range, while identical-image pretraining yields near-unit epoch-for-epoch savings after ≈5 warm-up epochs.The authors characterize these as non-equivalent and equivalent transfer, respectively; equivalent transfer also roughly persists across disjoint image splits.Source: Figure 2 · p. 3, §2 · pp. 3–4
- With disjoint image splits from a shared distribution, t0 decreases by slightly more than 1 per pretraining epoch, and equivalent transfer roughly persists across four splits.The split uses randomized labels on two halves of the training set.Source: Figure 2 · p. 3, §2 · p. 3
- Resampling labels each epoch prevents transfer, whereas resampling random-pixel inputs weakens but preserves non-equivalent transfer.Under label resampling, t0 is consistently larger than the from-scratch baseline.Source: §2 · p. 3, Figure 3 · p. 4
- Testing equivalent transfer without repeated images would require independent, identically distributed CIFAR-10 copies, such as samples from a class-conditioned generative model.Source: §2 · p. 4
3 Locating the source of transfer: ablation experiments
Weight ablations separate transfer carried by fully connected and convolutional weights from the little transfer carried by batch-normalization affine parameters. Rescaling and SVD experiments distinguish magnitude-driven FC transfer from convolutional transfer that is partly associated with weight covariance.
3 Locating the source of transfer: ablation experiments
- Keeping fully connected or convolutional weights preserves substantial transfer, while batch-normalization affine parameters alone carry little in both task settings.The weight ablations compare random-label and random-pixel transfer.Source: §3 · p. 4, Figure 4 · p. 5
- After norm rescaling, FC transfer disappears, whereas convolutional transfer remains, showing that FC transfer is fully explained by norm growth but convolutional transfer is not.The rescaled convolutional result is also reported for random-pixel pre-training.Source: §3 · pp. 4–5
- FC weights grow by more than an order of magnitude, while convolutional and batch-normalization weights remain near initialization scale in both pre-training settings.The authors test whether this norm growth explains downstream transfer by rescaling weights to their initialization norm.Source: §3 · p. 4, Table 1 · p. 5
- In the SVD experiment, transfer disappears when S and V are reset, but some persists when U is reset and S and V—and thus covariance C—are retained.The authors apply alignment to address hidden-activation mismatch caused by SVD replacement.Source: §3 · pp. 5–6, Figure 5 · p. 6
- Covariance-only transfer initially lowers t0 more than full convolutional transfer but plateaus, while full convolutional checkpoints continue lowering t0 with pre-training.The pattern is reported for random-label transfer and also observed for non-equivalent transfer with random pixels.Source: §3 · p. 6, Figure 5 · p. 6
4 Discussion
The discussion summarizes equivalent and non-equivalent transfer and localizes transfer to weight magnitude and structure, with covariance accounting for only part of convolutional transfer. It also identifies limits in the tested architecture and dataset and in the SVD-based localization.
4 Discussion
- The authors report equivalent transfer, mismatched-task pre-training that can initially outperform direct downstream training, and magnitude-driven FC versus structure-driven convolutional transfer partly encoded in covariance.These effects also arise when the pre-training task is devoid of structure; the covariance-alignment explanation does not account for all transfer settings.Source: §4 · p. 6
- Experiments use one architecture, a 7-layer CNN, and CIFAR-10; SVD only partially localizes convolutional transfer because covariance-only transfer saturates with pre-training time.The authors suggest exploring other architectures and datasets and more refined weight partitions.Source: §4 · p. 6
A Supplementary Results
Supplementary experiments report that alignment does not change the weight-ablation conclusions and extend the ablations to random-pixel upstream tasks. Additional figures vary the weight-reset procedure and report repeated-run summaries.
A Supplementary Results
- Adding the alignment procedure to the Figure 4 experiments does not change the conclusions across panels.Figure 6 repeats the same experiment as Figure 4 with alignment.Source: Figure 6 · p. 8
- With random-pixel upstream tasks, rescaling makes FC-only transfer nearly flat, while convolution-only transfer persists but weakens; alignment does not change these conclusions.Keeping only batch-normalization affine parameters preserves negligible transfer.Source: Figure 7 · p. 8
- Figure 8 repeats the Figure 4 experiment using freshly drawn weights for resets, so keep-only curves no longer start at the original curve’s point.Figure 9 applies fresh-weight resets to the random-pixel-upstream version of Figure 4.Source: Figure 8 · p. 8, Figure 9 · p. 9
- Figure 10 repeats the Figure 5 experiment with freshly drawn weights and reports the mean and min–max range across three runs with different seeds.The curves summarize three runs.Source: Figure 10 · p. 9
B Experimental Details
The experiments use a six-convolutional-layer CNN with batch normalization, ReLU, max pooling, global average pooling, and a linear classifier. Training uses fixed-rate SGD on CIFAR-10 with either random labels or independently sampled random pixels.
B Experimental Details
- The architecture has six convolutional layers in three blocks of widths 128, 256, and 512, followed by global average pooling and a 512-to-10 linear classifier.Each convolution is 3 × 3 with stride 1 and padding 1, followed by batch normalization and ReLU; each block ends with 2 × 2 max pooling.Source: B Experimental Details · p. 10
- Training uses SGD with learning rate 0.01, momentum 0.9, no weight decay, batch size 128, and cross-entropy loss, stopping at 99.9%.Source: B Experimental Details · p. 10
- Experiments use CIFAR-10 training images without augmentation; random labels are uniform over 10 classes, while random pixels are independently sampled uniformly from [0, 255] before normalization.Both data settings use the same per-channel mean and standard deviation for normalization.Source: B Experimental Details · p. 10
C Alignment Procedure
The section describes an empirical alignment procedure based on layer activations and cross-covariance, using an SVD-derived orthogonal alignment applied to the next layer’s weights.
C Alignment Procedure
- The section empirically demonstrates the alignment procedure of Guth et al. [2024].It compares post-BN-ReLU activations of each layer in the original and modified models.Source: C Alignment Procedure · p. 10
- The procedure forms a layer-wise cross-covariance matrix and derives an orthogonal alignment matrix from its SVD.The resulting matrix minimizes the activation discrepancy over orthogonal matrices.Source: C Alignment Procedure · p. 10
- The alignment is applied to the next layer’s weight as W′_{j+1} = W_{j+1} Â_j.W_{j+1} is the initial weight of the next layer in the modified checkpoint.Source: C Alignment Procedure · p. 10
NeurIPS Paper Checklist
The checklist reports the paper’s stated contributions, experimental documentation, and limitations. It also records author responses on reproducibility, compute, ethics, and study scope.
Claims and scope
- The authors say the abstract and introduction state the paper’s claims about equivalent and non-equivalent transfer, transfer localization, and random-pixel experiments.The checklist identifies the transfer patterns in Section 2 and localization and decomposition in Section 3.Source: NeurIPS Paper Checklist · p. 11
Limitations and theory
- The authors identify the single architecture and dataset as limitations and describe the work as empirical, without theoretical proofs.The checklist also notes open questions about whether higher-order statistics fully encode transfer.Source: NeurIPS Paper Checklist · pp. 11–12
Reproducibility
- The checklist says experimental details are in the appendix and code will be released with the camera-ready version.The authors also state that CIFAR-10 is publicly available and properly cited.Source: NeurIPS Paper Checklist · pp. 13, 15
Statistical significance
- The equivalent-transfer relationship was validated across four seeds, the SVD replacement experiment across three, and norm-rescaling demonstrations used representative seeds.The checklist reports these as the approaches used for key and supporting experiments, respectively.Source: NeurIPS Paper Checklist · p. 13
Compute resources
- Exact compute times were not recorded; experiments used a 7-layer CNN on CIFAR-10 with NVIDIA RTX 4090 and RTX 5090 GPUs.The checklist reports hardware but not execution times.Source: NeurIPS Paper Checklist · p. 14
Ethics and societal impact
- The authors report ethics compliance, no anticipated direct negative societal impacts, no released misuse-prone assets, and no human-subjects research.The checklist also states that the paper does not involve crowdsourcing.Source: NeurIPS Paper Checklist · pp. 14–16