PaperlayerView exact revision on arXiv

Source-linked AI summary

A Topological Signature of Learning under Noisy Labels

Zhongtian Sun, Fan Mo, Prayag Tiwari, Kelin Xia

arXiv:2610.23491v1cs.LGcs.AI

TL;DR

Training accuracy reveals little about how noisy supervision changes representation geometry. The paper uses persistent homology to measure activation-loop structure and finds a late clean–corrupted persistence contrast, alongside pipeline-dependent crossover and calibration results.

  • Problem

    Training accuracy measures agreement with labels but reveals little about how noisy supervision changes representation geometry.

    Source: §1 · p. 1

  • Method

    The paper applies persistent homology to activation point clouds to track loop structure across distance thresholds.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    At epoch 50, corrupted CIFAR-10 probe subsets have higher persistence despite only 7–11% noisy-label matches; normalized trajectories cross on CIFAR-10 but not CIFAR-100.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Scale controls and label-fitting measurements offer a way to study geometric responses to label corruption beyond training accuracy.

    Source: §10 · p. 10

    Group-level summaries do not resolve individual-example responses; the fixed cutoff needs a threshold sweep, and finer checkpoint sampling would improve timing resolution.

    Source: §8 · p. 10

Abstract

from arXiv · show

How does noisy supervision change the representations learned by a neural network? Training accuracy measures agreement with training labels but provides little information about representation geometry. We use persistent homology to track loop structure across distance thresholds in neural activations. Experiments span image classification, text classification, and modular arithmetic. On a fixed CIFAR-10/ResNet-18 training probe, equally sized clean-label and corrupted-label subsets show no detectable persistence difference at epoch 10, whereas corrupted-label subsets have higher persistence at epoch 50. The contrast survives uniform-scale controls, even though predictions match the assigned noisy labels for only 7-11% of corrupted probe examples at that checkpoint. Normalisation preserves the crossover between clean and noisy training trajectories on CIFAR-10/ResNet-18, whereas CIFAR-100/ResNet-18 shows no crossover under any of the five evaluated constructions. In a separate noise-rate calibration comparison across five pipelines, mean activation norm achieves a higher mean in-sample R2 than raw persistence (0.936 versus 0.908). These findings show that noisy supervision leaves a measurable geometric signature even when agreement with corrupted labels is low. Comparing persistence under scale controls with label-fitting measurements offers a way to study how representations respond to label corruption beyond what training accuracy reveals.

1 Introduction

The paper uses persistent homology to study how noisy supervision changes activation geometry, beyond what training accuracy reveals. Its fixed-probe comparison finds a late persistence difference between corrupted- and clean-label groups, while broader analyses distinguish representation characterization from noise-rate calibration.

1 Introduction

  • Training accuracy alone reveals little about how noisy supervision changes representations, motivating a geometric analysis of conflicting-label examples.The question connects to work on memorization, intrinsic dimension, the information bottleneck, neural collapse, and grokking [Zhang et al., 2016, Arpit et al., 2017, Ansuini et al., 2019, Shwartz-Ziv and Tishby, 2017, Papyan et al., 2020, Power et al., 2022].Source: §1 · p. 1
  • The analysis combines persistence with activation-scale measures, normalised point clouds, and non-topological geometric measurements to distinguish scale effects from structural differences.Uniform enlargement increases raw persistence while preserving shape.Source: §1 · pp. 1–2
  • At epoch 50, corrupted subsets have higher persistence despite only 7–11% of examples matching their assigned noisy labels and 71–79% matching original labels.No difference is detected at epoch 10; the late contrast also appears in summaries invariant to uniform rescaling.Source: §1 · p. 2
  • Across five pipelines, mean activation norm has a higher mean in-sample R2 than raw persistence for noise-rate calibration.The paper distinguishes estimating noise amount from characterising its effects on representations.Source: §1 · p. 2

2 Setup and Observable

The paper measures loop structure in penultimate activations using persistent homology, with explicit controls for scale and sampling. It applies this setup across varied datasets and pipelines, while cautioning against direct numerical comparisons when aggregation protocols differ.

2 Setup and Observable

  • Trajectory analysis tracks evaluation-mode penultimate activations on held-out examples, while the fixed training probe compares clean- and corrupted-label subsets within one model.Global persistence does not require per-sample corruption labels, unlike the subset comparison.Source: §2 · p. 2
  • The observable is total H1 persistence: the sum of retained loop lifetimes, with bars retained when their lifetimes exceed 0.01Lmax(t).With no explicit filtration cap, feature deaths are not truncated at a manually chosen distance.Source: §2 · pp. 2–3
  • Unit normalisation, cosine dissimilarity, per-feature z-scoring, and centered RMS rescaling test sensitivity to different aspects of activation scale.Uniformly scaling activations can increase raw persistence without changing loop structure across corresponding scales.Source: §2 · p. 3
  • The additional summaries ΦH1/ΦH0 and Φ/diam(Z) are invariant to uniform positive rescaling when their denominators are nonzero.They are not invariant to arbitrary anisotropic transformations.Source: §2 · p. 3
  • Per-class persistence sums should not be compared numerically with global sums as though point counts and aggregation were matched.The default global computation samples n = 300 activation vectors; per-class protocols use different counts and aggregation.Source: §2 · p. 3
  • The study spans image, text, and modular-arithmetic tasks, using architectures from MLPs and ResNets to ViTs and Transformers.The original suite reports over 900 training runs and 8,000 PH computations; the normalisation reanalysis covers 1,540 saved activation files.Source: §2 · pp. 3–4

3 Early Response and Late Training-Set Evidence

Noise associations vary over training and across pipelines, while a fixed CIFAR-10 probe finds a late persistence contrast between corrupted- and clean-label subsets. The contrast appears under scale controls, but the available controls do not establish a deformation beyond class geometry.

3 Early Response and Late Training-Set Evidence

  • Mean activation norm achieves a higher mean in-sample R2 than raw persistence for noise-rate calibration: 0.936 versus 0.908.At epoch 10, raw persistence is associated with noise level in the evaluated MLP, ResNet-18, ViT-Tiny, and DistilBERT pipelines.Source: §3 · p. 4
  • Raw-persistence correlations with noise level weaken from epoch 10 to 50 on CIFAR-10/ResNet-18 and MNIST/MLP, while ViT-Tiny/CIFAR-10 remains at −0.962.These correlations order noise conditions across trained models; a weaker correlation alone does not establish a clean–noisy crossover or locate memorization.Source: §3 · p. 4

3.1 A fixed training probe

  • The fixed probe uses 4,000 CIFAR-10 training images, with clean and corrupted groups matched in size but not class composition, and computes PH on 300 points per group.Each condition uses eight seeds and paired exact two-sided Wilcoxon tests.Source: §3.1 · p. 4
  • At epoch 50, corrupted subsets have higher raw persistence (corrupted/clean ratios 1.29 and 1.37); this holds in eight seeds except Φ/diam(Z) at sf = 0.25, where it holds in seven.Source: §3.1 · p. 4
  • No excess-persistence difference is detected against separately fitted class-conditional Gaussian nulls, so the contrast does not uniquely identify deformation beyond second-order class geometry.Across six epoch-wise comparisons, test-set and training-probe statistics show no statistically detectable difference (minimum p = 0.11); these comparisons do not establish equivalence.Source: §3.1 · pp. 4–5

4 Geometric Interpretation and Controlled Comparisons

Controlled comparisons show that persistence responds to where conflicting labels fall and is associated with local geometric changes. These measurements describe sampled activation clouds, but do not uniquely establish folding or new holes in the underlying data manifold.

Geometric interpretation

  • Persistence depends on sampling, class geometry, and distance scale, and the piecewise-affine structure of the studied residual networks guarantees neither an H1 cycle nor a lower bound on its lifetime.The stated piecewise-affine argument does not apply to LayerNorm or softmax attention.Source: §4 · p. 5

Class-pair controls

  • At epoch 50, raw persistence was 1.445 times as large in FAR as in NEAR, with the direction agreeing in all ten seeds (p = 0.002).The five disjoint class pairs per condition were selected by clean-representation centroid distance; their mean distances were 4.94 for FAR and 3.32 for NEAR.Source: §4 · p. 5
  • At epoch 50, clean centroid distance and noisy/clean persistence ratio were associated across class pairs, with Spearman correlations of 0.867, 0.614, and 0.413 for CIFAR-10/ResNet-18, CIFAR-10/ViT-Tiny, and CIFAR-100/ResNet-18, respectively.The correlations vary over training.Source: §4 · p. 5

Synthetic contradictions and local geometry

  • Across 140 individual seed runs with 0–50% near-boundary label flips, flip rate was associated with epoch-10 persistence (ρ = 0.33, p = 6.1 × 10−5).For seed 0 at epoch 10, total persistence was 8.38 with 0% flips and 11.08 with 50% flips.Source: §4 · pp. 5–6, Figure 2 · p. 6

Local geometric measurements

  • At epoch 50, local PCA dimension correlated with noise level at ρ = 0.946 on CIFAR-10/ResNet-18 when the fully randomized condition was excluded.In the exploratory relabelling comparison, local PCA dimension was higher in the random-relabeling arm in all 24 seed–rate pairs, although the arms differed in realised corruption rate and relabelling structure.Source: §4 · p. 6

5 Normalisation, Crossover, and Noise Type

Crossover and relabelling comparisons vary with the pipeline and distance construction, while scale controls distinguish changes beyond uniform rescaling. The section also contrasts raw-persistence trajectories and loop-count summaries across settings.

Crossover across distance constructions

  • CIFAR-10/ResNet-18 first crossed at epoch 20 under cosine, unit-normalised, centered RMS, and z-score constructions, versus epoch 30 under raw Euclidean distance; CIFAR-100/ResNet-18 showed no crossover under any of five constructions.The CIFAR-10 comparison uses sf = 0.5; the CIFAR-100 result was confirmed in three seeds.Source: §5 · p. 7, Table 3 · p. 7

Persistent clean trajectories

  • On ImageNette/ResNet-50, the mean clean raw-persistence trajectory rose overall and stayed above noisy means at displayed checkpoints, despite fluctuations over 100 epochs.Figure 3 shows means across seeds with sample-standard-deviation error bars.Source: §5 · p. 7, Figure 3 · p. 7

A direct scale check

  • On CIFAR-10/ResNet-18 at epoch 50, the estimated log ΦH1-versus-log ΦH0 slope was 1.577 using four noise levels, rejecting pure uniform rescaling in that comparison.Across four vision configurations and 14 checkpoints excluding CIFAR-100, slopes differed from one in 52/56 cells using four levels, versus 19/56 using three.Source: §5 · p. 8

Exploratory comparison of relabelling conditions

  • At epoch 50, raw persistence was higher in the random-relabeling arm in all eight seeds, but the relabelling arms also differed in realised corruption rate and structure.Across CIFAR-10N, unit-normalised and cosine persistence correlated with corruption rate at ρ = 0.936 and 0.929, compared with ρ = 0.302 for raw persistence (p = 0.15).Source: §5 · p. 8, Table 4 · p. 8

Alternative summaries and sampling

  • Loop count was invariant to uniform positive rescaling under the relative lifetime cutoff, but its noise correlations were not significant on the reported CIFAR-100, FashionMNIST, and MNIST comparisons.Epoch-10 CIFAR-10/ResNet-18 sampling correlations were −0.850 at 100 points and −0.945 at each tested size from 200 to 2,000; equal rank correlations do not eliminate finite-sampling effects.Source: §5 · p. 8

Noise-rate calibration

  • Table 5 reports epoch-10 in-sample noise-rate calibration across five pipelines, summarizing R2 ranges and means across pipelines.The ranges are not confidence intervals.Source: Table 5 · p. 9

6 Grokking and Decreasing Persistence

The paper examines persistence across delayed generalization in modular-arithmetic Transformers. Raw persistence decreased across the transition in most runs, but the results do not establish a causal role for unfolding.

Grokking and persistence

  • Raw persistence decreased across the grokking transition in ten of twelve modular-arithmetic runs, with a mean change of −35.2 across all twelve runs.In the other two runs it increased by 3.7 and 4.5; among the ten decreasing runs, the mean change was −43.1.Source: §6 · p. 9

Measure

  • ∆Φgrok compares mean test-set persistence at high-training-accuracy checkpoints with test accuracy ≥0.90 against checkpoints with test accuracy < 0.50.Persistence was computed on 300 test points as the sum of all finite H1 lifetimes, without a relative cutoff.Source: §6 · p. 9

Scope

  • The evidence is limited to modular arithmetic and does not establish a causal role for unfolding or show that the pattern holds under scale normalisation.The twelve runs used two-layer Transformers on modular addition and multiplication with primes 97 and 113.Source: §6 · p. 9

7 Diagnostic Use: Calibration and Representation Comparisons

The section separates noise-rate calibration from describing representation changes, and cautions against treating persistence as an objective for training decisions.

Calibration

  • Mean activation norm achieves mean in-sample R2 0.936, compared with 0.908 for raw persistence across five pipelines.These are in-sample calibration results, not held-out prediction performance.Source: §7 · p. 9

Diagnostic scope

  • Calibration is distinct from detecting individual corrupted examples or improving robust classification accuracy, and the reported regression R2 does not measure either task.Checkpoint selection used validation accuracy and does not establish an advantage over choosing the best validation-accuracy checkpoint; adaptive smoothing’s displayed mean accuracy was 0.07 percentage points lower than fixed smoothing.Source: §7 · p. 9

Representation comparisons

  • Persistence should not be treated as an objective to maximize or minimize: its associations with effective rank and accuracy vary, and interventions change persistence without a monotone test-accuracy ordering.The effective-rank proxy correlates with raw persistence at ρ = 0.831 on CIFAR-10/ResNet-18 but ρ = 0.002 on FashionMNIST; persistence-peak height has an overall correlation of ρ = 0.27 with final accuracy.Source: §7 · p. 9

8 Scope and Limitations

Results differ across datasets, architectures, and measurement protocols. The authors identify unresolved experimental comparisons and measurement choices that constrain interpretation.

Scope

  • Noise associations and crossover patterns vary by pipeline: AG News/DistilBERT has a positive early association, evaluated vision settings have negative associations, and CIFAR-100/ResNet-18 has no crossover.Clean persistence remains elevated on ImageNette and Tiny-ImageNet; calibration is therefore pipeline-specific.Source: §8 · p. 10

Open questions

  • In the exploratory ResNet-18 relabelling comparison, noise structure and realised corruption rate vary between arms, leaving their individual contributions unresolved.The class-conditional Gaussian control detects no group difference in excess persistence, so folding must be assessed alongside class geometry, anisotropy, and sampling.Source: §8 · p. 10
  • The fixed 0.01Lmax cutoff needs a threshold sweep, and finer checkpoint sampling would improve timing resolution beyond the 14-checkpoint study.The piecewise-affine account remains to be extended to LayerNorm and attention, and normalised persistence during grokking remains to be tested beyond modular arithmetic.Source: §8 · p. 10

9 Related Work

Prior work applies persistent homology to neural representations, noisy-label methods, and other representation or training analyses. This study uses persistence to characterize representation changes during noisy-label training.

Related work

10 Discussion and Conclusion

The discussion presents persistence as a way to distinguish geometric responses to noisy supervision from noise-rate calibration, while emphasizing limits on interpretation and comparison.

10 Discussion and Conclusion

  • On the fixed probe, corrupted subsets develop higher persistence late in training, including under uniform-scale controls, while most corrupted examples are still predicted as their original labels.These measurements distinguish estimating noise amount from studying representational effects.Source: §10 · p. 10

A Impact Statement

  • Persistence describes groups of examples, so treating it as evidence of individual annotation errors risks confusing training-pipeline effects with label quality.Using it alone to filter data could remove valid examples, including those from underrepresented groups; combine the diagnostic with annotation review and assessment across groups.Source: A Impact Statement · p. 14

B Training Hyperparameters

  • The appendices specify distinct training, relabelling, normalization, classwise, and ViT protocols, alongside persistence conventions and statistical procedures.The protocols include symmetric relabelling, a synthetic 2D setup, transformation-specific normalization, differing classwise and ViT sampling rules, and seed-level tests with pooled analyses treated descriptively.Source: B Training Hyperparameters · p. 14

C Subsampling Sensitivity

  • Subsampling at n = 300 yields Spearman correlation 0.973 with the 2,000-point reference across 433 configurations, at 0.058 s per call versus 2.944 s for the reference.Mean subsampling coefficient of variation is 0.022, with a maximum of 0.086.Source: C Subsampling Sensitivity · p. 15, Table 7 · p. 15

D Raw Persistence Associations over Training

E Large-Scale Results

  • The original extended CIFAR-100/ResNet-50 experiment reports crossover at epochs 60–70, but the superseded ResNet-18 result does not support a monotone delay with depth.A matched analysis using the same extraction, normalization, and checkpoint protocol is needed for that comparison.Source: E Large-Scale Results · p. 15

F Negative Results

The negative results show that ACRP is not a universal proxy for persistent homology and that peak persistence does not reliably predict final accuracy. The accompanying analyses specify a checkpoint-selection procedure, report compute accounting, and compare persistence across regularisation methods.

F Negative Results

  • ACRP correlates with Φ on CIFAR-10/ResNet-18 (ρ = 0.831) and CIFAR-100 runs (ρ = 0.504, n = 168), but not on FashionMNIST or MNIST, so it is not a universal proxy.The reported correlations are ρ = 0.002 for FashionMNIST and ρ = −0.321 for MNIST.Source: F Negative Results · p. 16
  • Peak Φ is not a reliable predictor of final accuracy: its overall correlation with final accuracy is ρ = 0.27, and stopping at its peak changes performance by +4.2% at sf=0.5 and −4.1% at sf=0.0.The correlation rises to ρ = 0.56 at sf=0.5.Source: F Negative Results · p. 16

G TGCS Algorithm

  • TGCS evaluates the Φ-peak and final checkpoints on validation data and returns the one with higher validation accuracy.The procedure trains through T epochs, selects the peak and final checkpoints, then compares them on V.Source: G TGCS Algorithm · p. 16

H Compute Budget

  • The compute-budget table reports GPU-hours and CPU-hours separately, with training using GPUs and persistent-homology and response-period analyses using CPUs.Source: Table 9 · p. 16

I Regularisation Methods

  • At epoch 50, label smoothing and high weight decay reduce Φ by approximately 50% relative to baseline, while TGT increases it; test accuracy does not follow a monotone persistence ordering.The comparison averages results over noise levels on CIFAR-10/ResNet-18.Source: I Regularisation Methods · p. 16

J Grokking Results

Across modular arithmetic and several vision and text pipelines, persistence responses vary with training stage, dataset, architecture, and measurement construction. Calibration, checkpoint-selection, and regularisation analyses further distinguish descriptive geometric signals from demonstrated predictive or performance gains.

J Grokking Results

K Calibration Targets

  • Mean activation norm and raw persistence achieved mean in-sample R2 values of 0.936 and 0.908, respectively, for noise-rate calibration across five pipelines.Excluding the fully randomised condition, the corresponding means were 0.892 and 0.729.Source: K Calibration Targets · pp. 17–18
  • Adaptive smoothing beat fixed smoothing by +0.31 percentage points at sf=0.25, but underperformed it by −0.07 percentage points on average.The authors describe this as proof-of-concept; the reported differences do not isolate calibration error from the smoothing rule.Source: K Calibration Targets · p. 18

M TGCS Results

  • TGCS defines δ as the positive gain of the Φ-peak checkpoint over the final checkpoint; assessing it as a selection procedure requires same-budget validation-based early stopping.Reported entries include a representative subset of 40 runs.Source: M TGCS Results · p. 18, Table 13 · p. 18

N Depth Ablation

  • The original depth-ablation values use a different activation-extraction and computation pipeline from the canonical reanalysis, so the comparison is withheld pending reconciliation.Those results do not currently support a general claim that depth delays folding.Source: N Depth Ablation · pp. 18–19

O ViT Per-Class PH Investigation

  • On ViT-Tiny/CIFAR-10, classwise PH shows ρ ≤−0.93 with noise, but its larger aggregate cannot establish diagnostic improvement because its grouping and point budget differ.The analysis establishes sensitivity to grouping and sample budget, not a unique explanation for the original weak signal.Source: O ViT Per-Class PH Investigation · p. 19, Table 14 · p. 19

P Tiny-ImageNet Scaling

  • On Tiny-ImageNet/ResNet-18, global PH separates clean and noisy conditions; classwise PH correlates with noise at ρ = −0.969 at epochs 10 and 25, but uses 15,000 points.The clean global trajectory rises from 84.4 to 165.6; this does not establish a universal rule against crossover.Source: P Tiny-ImageNet Scaling · p. 19, Table 15 · p. 19

Q Prospective Crossover Test on ResNet-34

  • On CIFAR-10/ResNet-34, all six noisy runs stayed below the clean reference at every sampled checkpoint, matching a no-crossover prediction recorded before noisy training.The prediction followed a clean trajectory that remained elevated through later checkpoints.Source: Q Prospective Crossover Test on… · p. 20, Table 16 · p. 20
Loading 2610.23491v1…