PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase Transition Frequency as a Training Time Predictor of Test Accuracy in ResNets

Arunan J

arXiv:2609.05194v1cs.LGcs.AI

TL;DR

The paper asks whether discrete class-separability reorganisations during ResNet fine-tuning predict final test accuracy, and evaluates transition counts across architectures and benchmarks. It finds a strong in-distribution association that weakens under harder tasks and distribution shift, supporting a scoped training-quality probe rather than a universal predictor.

  • Problem

    The paper asks whether training-time representation dynamics provide evidence about the generalisation gap between fitted training loss and eventual test loss.

  • Method

    The study counts epoch-to-epoch class-separability reorganisations during fine-tuning across 75 experiments, four benchmarks, and three pretrained ResNet architectures.

  • Results

    Strong negative associations occur on CIFAR-10 and CIFAR-100, while the transition–accuracy relationship attenuates on TinyImageNet and CIFAR-10-C.

  • Takeaways & Limitations

    Transition count is best interpreted as an inexpensive in-distribution training-quality probe among candidate training-curve signals, not a universal accuracy predictor.

  • Takeaways & Limitations

    The evidence is limited by observational correlations, small and imbalanced seed counts, a fixed detection threshold, restricted architectures, and omitted activation- or gradient-level competitors.

Abstract

from arXiv · show

The number of discrete class-separability jumps observed during ResNet finetuning is examined empirically as a predictor of final test accuracy. Across 75 experiments spanning four benchmarks (CIFAR-10, CIFAR-100, TinyImageNet, and CIFAR-10-C) and three architectures (ResNet-18, ResNet-50, and ResNet-101), with five to ten seeds per configuration, a strong within-dataset negative correlation is obtained on standard i.i.d. classification benchmarks: \(r = -0.84\) on CIFAR-10 (\(p < 10^{-8}\), \(n = 30\)) and \(r = -0.87\) on CIFAR-100 (\(p < 10^{-5}\), \(n = 15\)). Under distributional stress, the relationship attenuates: TinyImageNet yields \(r = -0.45\), and the CIFAR-10-C corruption benchmark yields \(r = -0.19\). Two additional analyses discipline the empirical claim. A partial correlation controlling for architecture depth, treated as a linear covariate, shows that on CIFAR-100 the transition count retains statistically significant predictive power (\(r_{\mathrm{partial}} = -0.69\), \(p = 0.007\)); the corresponding result under the stricter categorical conditioning is not established at \(n = 15\). A comparison against six alternative training-curve signals shows that transition count achieved the strongest correlation among the evaluated signals on CIFAR-100 and one of the strongest on CIFAR-10, but is dominated by other signals on the two stressed benchmarks. The comparison is restricted to training-curve-level signals; comparisons against effective rank, Hessian sharpness, Fisher information, margin, and neural-collapse measures, which are the strongest competitors in the current literature, are not part of the present study and remain open. The observation is presented as an in-distribution training-quality probe among a family of candidate probes, and an inexpensive detection procedure suitable for logging alongside a standard training loop is provided.

1 Introduction

The paper studies whether discrete reorganisations of internal class geometry during training help explain variation in final test accuracy. Across four benchmarks, transition count is useful on standard in-distribution tasks but weakens under harder tasks and distribution shift.

  • Motivation: Phase transitions are epoch-to-epoch moments when internal representation class geometry reorganises, detected using a class-separability threshold.The count is designed as an inexpensive training-dynamics metric that can be computed alongside a standard training loop.
  • Main empirical pattern: r ≤ −0.83, p < 10^-5: transition count strongly negatively predicts within-dataset test accuracy on CIFAR-10 and CIFAR-100.The relationship attenuates substantially on TinyImageNet and CIFAR-10-C.
  • Main empirical pattern: CIFAR-10-C shows an attenuated association and a mild inversion in architectural ordering despite large differences in transition count.This contrasts with the stronger in-distribution pattern on CIFAR-10 and CIFAR-100.
  • Scope and controls: Controlling architecture depth linearly preserves transition-count predictive power on CIFAR-100, while the stricter categorical control is not claimed as established.The categorical result weakens because transition counts vary sharply between architectures and the effective sample is small.
  • Scope and controls: The study covers three ResNet variants and four image-classification benchmarks, excluding Vision Transformers, MLP-Mixers, and non-vision modalities.The resulting claim is a benchmark-scoped empirical characterization rather than a universal statement across architectures or modalities.

2 Related Work

Related work connects training trajectories, representation geometry, and robustness to final performance, while distinguishing this paper’s event-counting metric from endpoint geometry and delayed generalisation phenomena. Figure 1 reports strong negative associations on standard benchmarks that attenuate with task difficulty and distribution shift.

  • Representation and training dynamics: The Information Bottleneck literature concerns compression and task-relevant information, whereas this paper measures reorganisation of internal class geometry without estimating mutual information.The paper positions its measurement as distinct from the contested information-bottleneck interpretation of deep-network training.
  • Phase-transition literature: Prior studies report sudden loss jumps, jamming, critical periods, and delayed generalisation, but the present transitions are epoch-to-epoch class-separability reorganisations counted during training.They are therefore distinct from grokking’s delayed generalisation jump on a fixed dataset.
  • Robustness and generalisation: r ≤ −0.84, p < 10^-5: the negative transition-count association is strong on CIFAR-10 and CIFAR-100 but attenuates as task difficulty and distribution shift increase.This pattern is summarised by within-dataset regression fits in Figure 1.
  • Trajectory versus endpoint measures: The proposed metric counts discrete reorganisation events rather than characterising endpoint geometry such as neural collapse.The paper describes it as complementary to trajectory and endpoint analyses of SGD-trained networks.
  • Robustness and generalisation: CIFAR-10-C places the metric in a robustness setting where the transition–accuracy correlation attenuates under corruption.The result connects the paper’s training-time signal to, but does not resolve, broader questions about robustness under distribution shift.

3 Methodology

The method monitors class separability in pretrained ResNets fine-tuned across four benchmarks and counts thresholded epoch-to-epoch changes as phase transitions. Efficient trace computation and a fixed detection threshold make the procedure suitable for grid-scale experiments and routine training logs.

  • Detection signal: Class separability is computed from the Fisher criterion and used as the primary signal for detecting representation reorganisation.Representation magnitude and effective rank are tracked as auxiliary diagnostics, while separability was selected as most sensitive to reorganisation events.
  • Detection signal: The Fisher criterion uses within-class and between-class scatter traces, but naive scatter-matrix construction can exceed 100 GB of host memory.This issue is especially severe in wide early ResNet layers and benchmarks with many classes.
  • Efficient computation: Trace identities replace full scatter matrices, reducing computation to O(N · d_ℓ) time and memory while preserving numerically identical trace values.The optimisation made the 200-class TinyImageNet experiments feasible.
  • Phase-transition detection: Algorithm 1 takes separability values across epochs and layers plus threshold τ, and outputs the set of transition epochs.A transition is detected when relative separability change exceeds the threshold.
  • Phase-transition detection: τ = 0.05: the detector flags a relative change exceeding 5% from the epoch-zero baseline.A full threshold sweep was not conducted; the study reports a rank-based robustness proxy instead.
  • Experimental setup: Three pretrained ResNet architectures are fine-tuned on four benchmarks, with CIFAR-10 using ten seeds per configuration and the other benchmarks using five.The total experimental grid contains 75 cells, and all four ResNet blocks are monitored with separability computed every five epochs.

4 Results

Across the 75-experiment grid, transition count is most predictive within the two in-distribution benchmarks, while its association weakens under distributional stress and depends partly on how architecture is controlled.

  • Aggregate: r = −0.33 (p = 3.6 × 10−3) pooled across all 75 experiments, but dataset baselines dilute the within-dataset signal.The dataset-conditional analysis is identified as the appropriate estimand.
  • Within-Dataset Correlation: 95% confidence intervals exclude zero on CIFAR-10 [−0.91, −0.72] and CIFAR-100 [−0.97, −0.68], but not on TinyImageNet [−0.81, +0.06] or CIFAR-10-C [−0.62, +0.32].The stressed-benchmark intervals are therefore compatible with no within-dataset association.
  • Partial Correlation Controlling for Architecture: rpartial = −0.687 (p = 0.007) on CIFAR-100 after linear-depth conditioning, with combined R2 = 0.87.The transition coefficient’s bootstrap interval is [−1.34, −0.03], excluding zero narrowly at n = 15.
  • Partial Correlation Controlling for Architecture: rpartial = −0.259 (p = 0.35) under categorical architecture conditioning on CIFAR-100, so the stricter result is not claimed at n = 15.The categorical analysis removes between-architecture variation and leaves limited residual within-cell variation in transition count.
  • Comparison Against Alternative Training-Dynamics Signals: |r| = 0.87 makes transition count the strongest of seven evaluated predictors on CIFAR-100, while it ranks second on CIFAR-10 and sixth on TinyImageNet after excluding a trivial gap predictor.On CIFAR-10-C, no evaluated predictor exceeds |r| = 0.44; literature measures such as effective rank and Hessian sharpness were not compared.

5 Discussion

The evidence supports a two-regime interpretation: transition count is informative on in-distribution benchmarks but loses predictive power under harder tasks and corruption shift. The authors therefore position it as an in-distribution training-quality probe, not a universal accuracy predictor.

  • Interpretation: Many transitions may reflect optimisation inefficiency or representational compression, but the observational data does not distinguish these explanations.The paper explicitly states that no intervention reduces transitions and measures the resulting accuracy change.
  • Two regimes: On TinyImageNet, transition counts effectively saturate across architectures, reducing their ability to discriminate final accuracy.The task difficulty causes all three architectures to accumulate many transitions.
  • Two regimes: On CIFAR-10-C, corruption robustness depends on factors not visible in clean-training dynamics, and the transition–accuracy correlation attenuates.The benchmark corrupts only test data, making it a direct distribution-shift setting.
  • Conclusion: Transition counting is presented as a useful in-distribution training-quality probe rather than a universal predictor under distribution shift.Its scope is among candidate probes rather than a claim of universal accuracy prediction.

6 Limitations

The study's conclusions are constrained by its ResNet-only, small-scale benchmark design, imbalanced seed counts, fixed detection threshold, measurement choice, incomplete competitor comparison, and correlational evidence.

  • Scope: Every experiment uses a ResNet variant, leaving extension to other architectures an open question.Vision Transformers, MLP-Mixers, ConvNeXt, EfficientNet, and other non-vision architectures were not evaluated.
  • Datasets: The datasets are limited to at most 200 classes and 64 × 64 resolution, while Full ImageNet-1K was not evaluated because of compute constraints.This limits evidence at larger scale and resolution.
  • Statistical power: The key CIFAR-100 partial-correlation claim is estimated from n = 15, and wider uncertainty remains for TinyImageNet and CIFAR-10-C.The seed density is ten per configuration on CIFAR-10 but five on the other benchmarks.
  • Measurement: The transition detector uses a fixed threshold τ = 0.05 without a full sensitivity sweep.Spearman agreement checks ranking robustness but not numerical robustness of r under alternative thresholds.
  • Measurement: The experiments measure class separability rather than the STFT-based spectral entropy motivating the appendix framework.The main empirical results do not depend on that framework.
  • Comparators: The alternative-signal comparison excludes effective rank, Fisher information, Hessian sharpness, margin, and neural-collapse metrics.Testing those competitors would require collecting per-epoch activation and gradient tensors.
  • Causality: The results are correlational, with no intervention showing that reducing transition count improves test accuracy.The paper describes transition count as a diagnostic rather than a prescription.

7 Future Work

The paper identifies six extensions to test generality, scale, comparative value, metric robustness, and causality of transition count.

  • Six extensions are proposed to address remaining objections to the study.The extensions target architecture, dataset scale, competing signals, threshold robustness, and intervention.
  • The experimental grid could be expanded to Vision Transformers, WideResNets, and EfficientNets at matched seed density.
  • Repeating the experiments at ImageNet-1K scale would test whether within-dataset correlations survive at larger scale.
  • Recording effective rank, Hessian sharpness, Fisher information, margin, and neural-collapse metrics would compare transition count with activation- and gradient-level alternatives.
  • Sweeping transition thresholds τ ∈ {0.02, 0.03, 0.05, 0.07, 0.10} would provide a full metric-robustness analysis.
  • An intervention penalizing epoch-to-epoch feature drift is proposed to test whether transition frequency has a causal role.

8 Conclusion

Across four benchmarks and three ResNet variants, transition frequency predicts test accuracy strongly under standard i.i.d. training but attenuates under harder tasks and distribution shift. The metric is therefore presented as an in-distribution probe among candidate training-quality signals, not a universal law.

  • 75 experiments across three ResNet architectures and four benchmarks found strong within-dataset prediction under standard i.i.d. training.
  • On CIFAR-100, transition count was strongest among seven training-curve-level predictors and remained significant after linear adjustment for architecture depth.
  • Under categorical conditioning on architecture, the residual CIFAR-100 signal was not established at n = 15.
  • The signal attenuated substantially on TinyImageNet and CIFAR-10-C, yielding a two-regime characterization rather than a universal law.
  • Transition count is proposed as an in-distribution training-quality probe among candidate signals, with activation- and gradient-level comparisons left open.

Declarations

The supplied declarations and appendix passages report funding, ethics, availability, notation, definitions, conjectures, and proof sketches, including explicit limits on the appendix framework.

  • No external funding or competing interests are reported, and the study did not involve human participants or animal subjects.
  • The datasets and code used in the study are not publicly available.
  • The appendix presents three statements as conjectures rather than theorems, and its empirical framework is not required by the main results.
  • The appendix defines spectral representations through STFT activations and spectral entropy as part of an exploratory frequency-domain framework.
  • The proof sketches acknowledge that key assumptions and the bridge between parameter distance and spectral entropy are not rigorously established.

Appendix B Additional Experimental Detail

The experiments used fixed training and evaluation settings, including a 50-epoch cosine-annealed schedule and a transition threshold of τ = 0.05.

  • All hyperparameters were fixed a priori without test-set tuning.
  • Training used an initial learning rate of 0.01 with cosine annealing over 50 epochs, momentum 0.9, weight decay 5 × 10^-4, and batch size 64.
  • The metric was evaluated every 5 epochs on 1000 training examples, using transition threshold τ = 0.05.

B.2 Compute

Experiments were run as separate cloud jobs with documented hardware, protocols, and structured outputs. Statistical analyses used Pearson and partial correlations, permutation checks, and bootstrap confidence intervals.

  • Each dataset–architecture cell used one NVIDIA T4 GPU, 32 vCPUs, and 128 GB of host RAM.Cell runtimes ranged from 1.5 to 8 hours, totaling approximately 60 wall-clock hours.
  • Pearson correlations used two-sided tests of ρ = 0, with within-dataset sample sizes of 15 or 30.Permutation tests with 10,000 label permutations matched the reported parametric conclusions.
  • Partial-correlation significance used a t-statistic with n − 3 degrees of freedom, encoding architecture depth as layer counts 18, 50, and 101.
  • Bootstrap confidence intervals used 10,000 resamples of transitions–accuracy pairs within each dataset.
  • 75 experiments were logged with final accuracies, per-epoch training and test trajectories, transition epochs, seeds, and full configuration records.
Loading 2609.05194v1…