Stable Dimensionality Reduction via Isometric Centroid-Encoder

1 Introduction

• Repeated training can produce different class layouts despite similar classification performance, limiting interpretability and reproducibility in scientific applications.The variation is associated with random initialization, stochastic optimization, and model architecture choices.Source: §1 · p. 2
• ICE learns latent class centroids while preserving their pairwise geometry, avoiding the large computational burden of sample-level distance preservation.The framework explicitly targets reproducible class-level geometry rather than only separation or neighborhood preservation.Source: §1 · pp. 2–3
• CNS measures consistency of class-level neighborhood structure across independent runs, complementing measures that assess embedding quality within a single run.The paper also shows that an M-class centroid geometry can be represented in at most M −1 dimensions, explaining saturation of distance-preservation measures beyond a certain dimension.Source: §1 · p. 3
• Experiments on image and high-dimensional biological datasets show that ICE produces more stable, geometrically consistent embeddings while maintaining competitive classification performance.The paper frames CNS, centroid-geometry preservation, and latent-dimension guidance as complementary contributions.Source: §1 · p. 3

2 Related Work

• Linear supervised methods emphasize class separation, while neighborhood and deep metric-learning approaches generally model sample-level relationships rather than prescribed class-centroid geometry.Centroid reconstruction methods such as CE encourage intra-class compactness but leave pairwise centroid relationships unconstrained.Source: §2 · p. 4
• ICE preserves prescribed pairwise distances among dynamically learned class centroids, unlike prior approaches that do not explicitly constrain this geometry.The paper states that, to its knowledge, no existing supervised dimensionality reduction method jointly learns latent class centroids while explicitly preserving prescribed class-centroid geometry.Source: §2 · p. 5

2.1 Relationship of ICE to Existing Representation Learning Methods

• ICE combines explicit class-centroid geometry preservation with intra-class compactness, making reproducible class-level geometry part of its objective.Table 1 compares representative supervised representation-learning paradigms by optimization objective and modeled geometric relationships.Source: §2.1 · pp. 5–6, Table 1 · p. 5

3.1 Problem Setup:

• ICE recomputes latent class centroids from encoder outputs during training and uses ambient-space pairwise centroid distances as the reference geometry.The latent centroids are recomputed at each training epoch.Source: §3.1 · p. 6

3.2 Objective Function:

• ICE jointly encourages samples to approach their latent class centroids and latent centroid distances to match the reference geometry, rather than artificially separating classes.The reference geometry may use any user-defined symmetric class-distance matrix.Source: §3.2 · p. 7

3.3 Comparison with vanilla Centroid-Encoder:

• Table 2 presents a comparison between the classical Centroid-Encoder (CE) and the proposed ICE framework.The supplied table caption identifies the comparison but does not specify its dimensions or outcomes.Source: Table 2 · p. 8

3.4 Theoretical Analysis

• The affine span of M class centroids has dimension at most M−1, so dimensions beyond M−1 cannot further improve pairwise centroid-distance preservation.This bound explains the observed saturation of centroid stress and Pearson correlation as latent dimension increases.Source: §3.4 · p. 9

3.5 Evolution and Stabilization of Latent Class Centroids

• On MNIST and FMNIST, latent centroids reorganized toward the prescribed geometry as mean epoch-to-epoch drift fell to a small residual, with minor fluctuations from stochastic optimization.The trajectory and drift analyses indicate progressive stabilization rather than persistent oscillation or collapse.Source: §3.5 · p. 10, Figure 2 · p. 10, Figure 3 · p. 11

3.6 Training loss analysis:

• At p = M, centroid-isometry loss continued decreasing on FMNIST and MNIST, unlike the 2D plateau, while centroid-consistency loss decreased in all experiments.The figure compares direct two-dimensional embeddings with class-dimensional embeddings; the higher-dimensional setting ended with substantially lower isometry loss.Source: §3.6 · pp. 11–12, Figure 4 · p. 12

4 Centroid Neighborhood Stability (CNS)

• CNS measures agreement in nearest-centroid neighborhood membership across independent runs, whereas Procrustes Disparity measures geometric alignment; the metrics capture complementary properties.CNS compares unordered neighbor sets, not neighbor ordering, and is invariant to global translation, rotation, and reflection.Source: §4 · pp. 13–14, §4.1 · p. 16
• For each class, CNS pools its k nearest centroid neighbors over R runs and scores how concentrated the resulting neighbor counts are; the overall score averages across classes.A score of 1 indicates the same k neighbors recur in every run; values approaching 1/R indicate maximally dispersed assignments.Source: §4 · p. 13
• The theoretical lower bound depends on class count, k, and R; the experiments use k = 3 and R = 5, and tested parameter changes preserve the overall comparative conclusions.Increasing k broadens the neighborhood scale, while increasing R improves estimate reliability without changing the neighborhood definition.Source: §4 · p. 14

4.1 CNS versus Procrustes Disparity: A Controlled Experiment

• With identical sample coordinates, Procrustes Disparity was zero across all 45 comparisons, while CNS varied from 0.73 to 0.87 (mean 0.784, SD 0.031).The class labels assigned to clusters were permuted, changing neighborhoods among identified classes without changing the embedding geometry.Source: Figure 5 · p. 15, §4.1 · p. 16

5.1 Datasets

• Image datasets provide established benchmarks, while high-dimensional biological datasets with more classes test scalability, stability, and class-geometry preservation.Table 3 summarizes the datasets used for benchmarking.Source: §5.1 · p. 17, Table 3 · p. 17

5.2 Baselines

• ICE is compared with CE, SE, IVIS, supervised UMAP, and class-constrained t-SNE, spanning centroid-based, similarity-learning, and neighborhood-preserving approaches.The cited baselines are CE, SE, IVIS, supervised UMAP (McInnes et al., 2018), and csTSNE (Meng et al., 2023).Source: §5.2 · p. 17

5.3 Evaluation Metrics

• Evaluation combines stress, Pearson correlation, Procrustes Disparity, CNS, and 5-nearest-neighbor classification accuracy to assess geometry, local neighborhoods, and stability.Stress and Pearson correlation measure distance preservation; Procrustes Disparity assesses centroid-configuration similarity after alignment, and CNS measures run-to-run neighborhood reproducibility.Source: §5.3 · p. 17

5.4 Experimental Protocol

• ICE is trained with Adam at learning rate 10^-3 for 300 epochs, and the reported experiments omit batch normalization, dropout, and weight decay.The network architectures and dataset-specific settings are listed in Table 4.Source: §5.4 · pp. 17–18, Table 4 · p. 18
• Classification uses an 80:20 train-test split and a 5-nearest-neighbor classifier, with results averaged over 15 independent runs; csTSNE uses PCA-reduced features only when original-space optimization is prohibitive.ICE variants are linear, shallow (one 500-unit hidden layer), and deep (1000, 500, and 125 units); λ is selected by five-fold cross-validation on the first split and fixed across runs.Source: §5.4 · p. 18

6.1 Effect of Latent Dimensionality on Centroid Geometry

• As latent dimension increases, stress decreases and Pearson correlation increases, then both measures rapidly saturate near the number of classes across datasets and encoder depths.The deep-ICE results are averaged over five independent runs; linear and shallow ICE show the same saturation pattern.Source: §6.1 · pp. 19–20, Figure 6 · p. 19, Figure 7 · p. 20
• Centroid geometry for M classes requires at most M−1 dimensions, and the observed saturation is largely independent of encoder depth rather than a consequence of insufficient capacity.The analysis varies latent dimension from 2 to 26 for deep, linear, and shallow ICE.Source: §6.1 · pp. 19–20
• Dimensions beyond M−1 provide little additional benefit for preserving class-centroid geometry, although the effective dimensionality may be lower than this upper bound.The section motivates selecting dimension from the centroid eigenspectrum.Source: §6.1 · p. 20

6.2 Practical Selection of the Latent Dimension

• The centroid configuration of M classes can be represented in at most M−1 dimensions, but real-world datasets may have substantially lower effective geometric complexity.The passage particularly notes biological datasets.Source: §6.2 · p. 21
• Choose the smallest latent dimension p* whose cumulative centroid-eigenvalue share reaches a user-defined explained-variance threshold τ, such as 95% or 98%.The criterion aims to preserve dominant centroid geometry without unnecessary dimensions.Source: §6.2 · p. 21
• For Mouse Bladder and Worm Neuron, centroid eigenspectra decay rapidly, indicating that only a few principal directions capture much of the variation and that effective dimension can fall below M−1.Their theoretical bounds are 15 and 9 dimensions, respectively; Mouse Bladder has a dominant first direction, while Worm Neuron approaches explained-variance saturation after a few directions.Source: §6.2 · p. 22, Figure 9 · p. 22
• Estimating dimension from the centroid eigenspectrum gives a deterministic, data-driven alternative to setting it directly from the number of classes.This guideline remains consistent with the M−1 theoretical upper bound.Source: §6.2 · pp. 21–22

6.3 Effect of Regularization Parameter λ

• In two dimensions, excessively large λ can reduce CNS as the geometry is not generally exactly representable, despite little change in global Stress.The resulting compromise makes class neighborhoods more sensitive to small perturbations.Source: §6.3 · pp. 22–23, Table 6 · p. 23
• Smaller λ values produce slightly higher 5NN accuracy by emphasizing centroid consistency and improving local sample neighborhoods.This contrasts with the isometry objective, which improves geometric regularization but can eventually be overemphasized.Source: §6.3 · p. 23
• On USPS, accuracy stays above 95% across λ values; highest CNS occurs at λ = 1, while highest classification accuracy occurs at λ = 10.The differing optima show that stability and classification performance capture complementary properties.Source: §6.3 · pp. 23–24, Table 7 · p. 23
• On 10X PBMC, moderate λ values (1–10) provide consistently high CNS with low Stress, while λ = 100 improves centroid-distance preservation but reduces accuracy.The authors suggest excessive isometry emphasis may slightly compromise local sample organization.Source: §6.3 · p. 24, Table 8 · p. 24
• Moderate λ values provide a favorable balance between centroid consistency and isometry, yielding stable class-level geometry with strong classification performance across image and biological datasets.When latent dimensionality is sufficient (p ≳ M − 1), ICE consistently achieves both low Stress and high CNS.Source: §6.3 · pp. 23–24

6.4.1 Consistency-Only ICE: Removing the Centroid-Isometry Term

• Removing centroid-isometry produces degenerate solutions: Stress reaches 1.00, correlation is negligible or undefined, and accuracy approaches the 10% chance level.The consistency loss can be minimized when samples and dynamically computed class centroids collapse to one location; for FMNIST with p = 10, one run has zero centroid-distance variance, making Pearson correlation undefined.Source: §6.4.1 · p. 25, Table 9 · p. 25
• Consistency alone cannot determine the relative locations of latent classes, so the centroid-isometry term is key to preventing trivial collapse and preserving prescribed geometry.The consistency-only ablation sets λ = 0 and removes the isometry term.Source: §6.4.1 · pp. 24–25, §6.4 · p. 24

6.4.2 Isometry-Only ICE: Removing the Centroid-Consistency Term

• With isometry alone, class-dimensional embeddings retain Stress of 0.01–0.06, Pearson correlation of approximately 1.00, and CNS of 0.94–0.99 across four datasets.These values are close to those of full ICE; removing consistency can nevertheless reduce local classification accuracy, including MNIST p = 2 from 97.26% to 94.26% and FMNIST p = 2 from 88.95% to 84.69%.Source: Table 10 · p. 25, §6.4.2 · p. 26
• Liso adjusts encoder outputs through dynamically computed class centroids toward the prescribed pairwise geometry, while Lcons explicitly keeps same-class observations near their latent centroid.Liso does not directly minimize each sample’s distance to its class centroid.Source: §6.4 · p. 24, §6.4.2 · p. 26

6.4 Ablation Study

• Across ablations, centroid-isometry establishes global class-centroid geometry and stable embeddings, while centroid-consistency improves within-class localization and local sample organization.The complete objective combines these properties rather than relying on either objective alone.Source: §6.4.2 · p. 26

6.5 Sensitivity of CNS to Its Parameters

• For fixed k, absolute CNS generally decreases as R increases; for fixed R, CNS generally increases with larger k.Additional runs allow more opportunities for neighborhood variation, whereas larger neighbor sets are less sensitive to individual neighbor changes.Source: §6.5 · pp. 26–27
• CNS evaluates consistency of unordered neighboring-class sets, and k sets the neighborhood scale from the nearest class at k = 1 to broader neighborhoods.Changing k can alter absolute CNS values and the relative ordering of closely performing methods.Source: §6.5 · pp. 26–27
• In two-dimensional embeddings, method rankings can change as k and R vary because small centroid shifts can change neighborhood membership when the full class geometry is constrained.The effect of increasing R on absolute CNS is systematic rather than method-specific.Source: §6.5 · pp. 27–28, Table 11 · p. 27
• In class-dimensional embeddings, ICE achieves the highest CNS for every tested k ∈ {1, 3, 5} and R ∈ {3, 5, 15}.The relative ordering of the other methods changes only modestly across these configurations.Source: Table 12 · p. 27, §6.5 · p. 28

6.6 Effect of PCA Preprocessing on High-Dimensional Biological Data

• PCA retained the smallest number of training-derived principal components preserving at least 98% of training-set variance before ICE was trained under the original-feature protocol.Results are mean ± standard deviation over 15 independent runs, with CNS computed from three non-overlapping groups of five runs.Source: §6.6 · p. 28, Table 13 · p. 28
• In two dimensions, PCA improves Mouse Bladder Stress from 0.20 to 0.09 and accuracy from 67.03% to 79.81%, and Worm Neuron Stress from 0.27 to 0.16 and accuracy from 69.86% to 75.53%.The 10X PBMC improvement is more modest.Source: Table 13 · p. 28, §6.6 · p. 29
• At p = M, PCA raises Worm Neuron accuracy from 87.14% to 93.79% while Stress changes from 0.07 to 0.04 and CNS remains 0.99 versus 0.95.For 10X PBMC, accuracy rises from 89.27% to 91.04% while CNS remains 0.99 versus 0.98; for Mouse Bladder, Stress falls from 0.07 to 0.03, CNS rises from 0.91 to 0.98, and accuracy rises from 78.12% to 83.86%.Source: Table 13 · p. 28, §6.6 · p. 29
• With p = M, ICE already achieves low Stress and high CNS on original features, so PCA’s geometry and stability changes are generally smaller even as accuracy improves.The paper treats 5NN accuracy and CNS as complementary measures of local sample discrimination and run-to-run class-neighborhood reproducibility.Source: §6.6 · p. 29

7.1 Results in Low Dimension (p = 2)

• In 2D, ICE generally preserves centroid distances best, but the limited space forces trade-offs between local neighborhood structure and the complete global centroid configuration.ICE has the highest CNS on FMNIST and USPS and the highest KNN accuracy on MNIST, FMNIST, and USPS; on PBMC, BLADDER, and NEURON, it ranks second on CNS.Source: §7.1 · pp. 29–31, Table 14 · p. 30

7.2 Results in Higher Dimension (p = M)

• At p = M, ICE preserves centroid geometry across image and biological datasets, with the lowest Procrustes Disparity on five of six datasets and 0.07 versus 0.06 for csTSNE on NEURON.ICE also has low Stress, high Correlation, generally high CNS, and the highest KNN accuracy on MNIST, FMNIST, USPS, and NEURON.Source: §7.2 · p. 31, Table 15 · p. 32
• Across five MNIST runs, ICE keeps class neighborhoods more consistent than CE and SUMAP, while SE’s higher CNS accompanies collapsed class geometry.In PCA projections from M-dimensional embeddings, ICE’s stability improves; the text attributes this to sufficient degrees of freedom to realize centroid geometry, unlike direct 2D embeddings.Source: §7.2 · pp. 34–35

7.3 Visualization of FMNIST

• On FMNIST, ICE maintains consistent class shapes and relative positions across runs; PCA projections likewise show more stable global and local structure than competing methods.CE’s footwear-class positioning varies, while SUMAP, SE, and IVIS show greater variation in PCA projections.Source: §7.3 · p. 37

7.4 Visualization of 10XPBMC Embeddings

• On 10XPBMC, ICE maintains a consistent global arrangement across runs and agreement between direct 2D embeddings and PCA projections, although some classes overlap.The dataset has eight classes in 16,653 dimensions; classes 6 and 8 overlap noticeably, and classes 4, 7, and 5 mix in the PCA projection.Source: §7.4 · pp. 37–38

7.5 Computational Efficiency

• ICE’s encoder-only implementation reduces training time and CPU memory use on large datasets, but its GPU memory rises with dataset size because centroids are computed on the GPU.At 300K samples, training time falls from 97.5 s for CE to 64.6 s for ICE, while peak CPU RSS falls from 6118 MB to 3760 MB; the GPU-memory increase is attributed to a one-time implementation choice.Source: §7.5 · pp. 41–42, Table 16 · p. 41

• ICE combines centroid consistency with explicit preservation of pairwise class-centroid geometry to maintain global class arrangement and local class localization.Source: §8 · p. 42
• The centroid configuration of M classes requires at most M −1 dimensions; direct 2D embeddings can distort geometry and show greater run-to-run variability.The paper reports that faithful complete class geometry generally requires latent dimensionality close to M −1.Source: §8 · p. 42
• CNS measures run-to-run consistency of class neighborhoods, complementing Stress, Correlation, and Procrustes Disparity, which assess geometry.The paper argues these measures should be considered jointly because they capture different aspects of embedding behavior.Source: §8 · p. 42
• Future work will examine relaxed centroid-isometry objectives for very low dimensions, uncertainty-aware formulations, and extensions incorporating sparsity or domain-specific class distances.Source: §8 · p. 42