PaperlayerView exact revision on arXiv

Source-linked AI summary

A Fixed-Radius Distance-Band Benchmark for Dimensionality-Reduction Fidelity

Yoshio Takaeda

arXiv:2608.21779v1cs.LG

TL;DR

The paper addresses limited evidence that standard local and global DR metrics jointly provide fair distance-fidelity evaluation. It introduces fixed-radius distance-band Shepard ρ with complementary value-based and membership-restricted scores, then benchmarks eight methods on known-geometry synthetic data. The benchmark finds systematic disagreement with k-NN reference metrics and shows that strong global rank fidelity can coexist with severe local scale collapse or failed outlier and minority placement.

  • Problem

    Standard DR evaluation uses global rank statistics that can hide near-distance errors and local k-NN metrics whose variable radii and hard thresholds favor neighbor-graph methods.

  • Method

    The paper evaluates eight DR methods on five known-geometry synthetic datasets using fixed-radius distance-band Shepard ρ, value-based metrics, membership-restricted scores, and addplot monitoring.

  • Results

    The benchmark reports systematic disagreement between k-NN references and fixed-radius bands, while strong global ρ can coexist with ≈93× within-cluster scale collapse and failed outlier or minority placement.

  • Takeaways & Limitations

    Distance-fidelity assessment needs separate near and global views plus value- and membership-based diagnostics for failures that all-pair or rank-based scores can miss.

  • Takeaways & Limitations

    The benchmark uses synthetic known-structure data under a redundancy-rich isotropic noise model, and its rankings need not transfer to sparse or irrelevant-feature regimes or real downstream tasks.

Abstract

from arXiv · show

Dimensionality-reduction (DR) methods are routinely judged by how well each point's k nearest neighbors survive the 2-D embedding (recall@k, trustworthiness, continuity). We argue this family is a biased measure of distance fidelity: its per-point variable radius and hard inclusion threshold favor neighbor-graph methods (t-SNE, UMAP) and penalize methods that preserve absolute distances. We instead score DR fidelity with a fixed-radius distance-band Shepard rho: the Spearman correlation between high-D and 2-D pairwise distances, restricted to cumulative distance bands so that near and global structure are reported separately, with every point judged on the same absolute radius. On synthetic datasets with known ground-truth geometry (non-uniform density, dense clusters, a closed-loop transition, off-subspace outliers, imbalanced two-population data) at realistic noise (SNR=1, D=768, N=1000), we benchmark eight methods -- PCA, Isomap, t-SNE, UMAP, PyMDE, PCC, DREAMS, and the closed-source toorPIA -- and show that (i) high global Shepard rho can coexist with a ~93x collapse of within-cluster scale, invisible to rank-based scores but obvious in a value-based over-compression metric; (ii) recall@k and the fixed-radius band disagree systematically, in the direction the bias predicts; (iii) a membership-restricted Shepard rho resolves single-point and minority-population questions that many-pair statistics cannot -- questions on which even DREAMS, a recent local-plus-global hybrid, fails silently. A supplementary out-of-sample (addplot) test asks whether a never-seen anomaly lands outside the normal region and whether its direction identifies its source. All metrics are computed exactly on all pairwise distances, independently of any method's internals, and every number is reproducible offline: the closed-source method's output coordinates (not its algorithm) are committed to the artifact.

1 Introduction

The paper argues that standard DR fidelity metrics split the evaluation problem: global Shepard ρ hides near structure, while k-NN metrics use variable radii and hard thresholds that favor neighbor-graph methods. It proposes fixed-radius bands, complementary value-based and membership-restricted metrics, and a reproducible eight-method benchmark on known-geometry synthetic data.

  • Motivation: Global Shepard ρ can hide near-distance errors because high-dimensional pairwise distances concentrate in a dominant far mode.A method can collapse clusters while retaining a near-perfect global rank correlation.
  • Motivation: Recall@k, trustworthiness, and continuity judge points on variable k-NN radii with hard inclusion thresholds, favoring neighbor-graph methods.The paper contrasts this with methods that preserve absolute distances.
  • Approach: The proposed fixed-radius distance-band Shepard ρ reports near and global fidelity separately using one absolute radius for every point.Value-based metrics and membership-restricted variants complement the band statistic.
  • Benchmark: The benchmark evaluates eight methods on five known-geometry synthetic datasets at SNR=1, D=768, and N=1000.It includes the closed-source toorPIA and releases its output coordinates for offline reproduction.
  • Reproducibility: The paper provides a reproducible, externally citable characterization of toorPIA without relying on the vendor’s judgment.The artifact commits the method’s output coordinates rather than its algorithm.

2 Related Work

Related work spans linear, geodesic, spectral, neighbor-graph, minimum-distortion, and cluster-aware DR methods, alongside global value-based and local rank-based fidelity metrics. The paper focuses on distance concentration as the reason all-pair global statistics can overlook near-distance errors.

  • Methods: PCA provides a linear variance-maximizing baseline, while Isomap uses graph-geodesic distances and spectral methods solve neighborhood-graph problems.The related methods represent distinct DR families.
  • Methods: t-SNE and UMAP optimize neighbor distributions or graphs, preserving local neighborhoods while weakly constraining large-scale distances.PyMDE is configured here with an absolute-distance loss, and PCC combines cluster observability with distance correlation.
  • Evaluation metrics: Shepard diagrams and Kruskal’s stress are global value-based readouts, whereas trustworthiness, continuity, and recall-type scores are local rank-based metrics.Cluster-level metrics extend evaluation to inter-cluster structure.
  • Motivation: Distance concentration makes all-pair statistics dominated by far pairs, leaving global Shepard ρ structurally blind to near-distance errors.The fixed-radius near band is presented as the direct remedy.

3 Metrics: fixed-radius bands vs. variable-radius k-NN

The metric framework replaces variable-radius, thresholded neighborhood scoring with fixed-radius distance bands, then adds value-based and membership-restricted diagnostics. Composite rankings use full and near Shepard ρ, while outlier and minority questions receive dedicated pair restrictions.

  • Distance bands: Shepard ρ is the Spearman correlation between high-D and 2-D distances over all point pairs, but concentration makes its global value insensitive to near errors.On the clusters dataset, within-cluster pairs are ≈14% of roughly 5×10^5 pairs and are out-voted roughly 6:1.
  • Distance bands: Cumulative bands retain pairs within the lowest p% of high-D distances, producing a near-to-far profile whose p=100 value recovers global Shepard ρ.The headline near band uses the first-mode valley of the pairwise-distance profile.
  • Fixed-radius fairness: The near band applies one absolute distance threshold to the whole dataset and scores continuous rank correlation on actual distances.This fixed-radius design avoids the variable-radius and hard-threshold choices of k-NN metrics.
  • Reference metrics: Recall@k, trustworthiness, and continuity remain a labelled, unscored reference block; fixed-radius bands favor distance-preserving methods while t-SNE and UMAP favor recall@k.The two evaluation blocks disagree in the direction predicted by the stated bias.
  • Value-based metrics: Band stress detects value distortion after optimal scaling, complementing scale-invariant rank correlation.It is designed to expose methods that preserve ordering while distorting distance values.
  • Value-based metrics: The tightest-cluster over-compression factor flags only crushing beyond 5×, because crushed internal structure cannot be read back while inflation remains legible.A value near 1 preserves relative scale; values much greater than 1 indicate crushing.
  • Membership diagnostics: Membership-restricted ρ isolates outlier and minority-population questions because single-point failures affect only O(1/N) of all pairs.The composite adds anomaly-pair or minority-pair columns to prevent such failures from being hidden.

4 Datasets and Experimental Setup

The benchmark constructs noisy, high-dimensional observations from known low-dimensional geometries and evaluates eight 2-D dimensionality-reduction methods. It spans density variation, dense clusters, closed-loop transitions, off-subspace outliers, and imbalanced populations, with distance-fidelity rankings reported at SNR=1.

  • Data construction: Each dataset starts in a low-dimensional latent space, is mapped into D ambient dimensions by a random orthonormal projection, and receives isotropic Gaussian noise.The orthonormal map preserves the clean latent distances, making the clean ambient Euclidean geometry the ground truth.
  • Data construction: The redundancy-rich projection spreads latent coordinates across 768 ambient columns, so signal adds coherently while isotropic noise self-averages.The design represents a noise-friendly extreme in which ambient dimensionality is nominal rather than a curse of dimensionality.
  • Methods: The benchmark compares PCA, Isomap, t-SNE, UMAP, PyMDE, PCC, DREAMS, and closed-source toorPIA on the same input vectors.Open-source methods use library defaults, while PCC is run label-free because no method receives caller-supplied cluster assignments.
  • Datasets: The five synthetic settings test density distortion, near-versus-global geometry, continuity through a closed loop, single-point separation, and minority-population structure.The settings include a non-uniform density dataset, seven dense clusters, a cyclic transition, off-subspace outliers, and imbalanced two-population data.

5 Results

Across synthetic benchmarks at SNR=1, distance-fidelity results expose trade-offs that global or local rank metrics alone miss. Fixed-radius near/global scores, value-based compression, and membership-restricted diagnostics distinguish methods across density, clusters, transitions, outliers, and minority populations.

  • 5.1 Non-uniform density: PCC achieves the best global ρ (0.820) on density while over-compressing the tightest cluster by ≈93×.Its near-band ρ is −0.028, whereas toorPIA preserves within-cluster scale at ≈0.4× and tops the composite with Σ=8.
  • 5.1 Non-uniform density: DREAMS leads density near-band ρ at 0.363 but ranks fifth globally with ρ=0.585, illustrating the near/global trade-off.Neighbor-graph methods lead the reference block but remain near the bottom of the fixed-radius global column.
  • 5.2 Distinct dense clusters: On distinct clusters, toorPIA tops the composite (Σ=8), DREAMS leads near-band ρ at 0.469, and PCC again crushes the tightest cluster by ≈9×.Isomap’s near-band ρ is negative (−0.056), while the embeddings show different forms of cluster compression, streaking, or merging.
  • 5.3 Continuous closed-loop transition: On the transition dataset, toorPIA has the best global ρ (0.729) but ranks seventh near, while PCC wins the composite (Σ=7) and DREAMS leads near ρ at 0.769.Only toorPIA renders both the seven dense states and their closed ring; DREAMS preserves cyclic arrangement but tears bridges.
  • 5.3 Continuous closed-loop transition: toorPIA keeps every transition bridge connected with a largest void of 14%, compared with tears in 4/7 t-SNE bridges, 4/7 UMAP bridges, and 3/7 DREAMS bridges.Low gaps for PCA, Isomap, PyMDE, and PCC reflect connectivity-by-blurring rather than faithful separated-cluster connectivity.
  • 5.4 Off-subspace outliers (single-point separation): For off-subspace outliers, toorPIA reaches outlier ρ=0.649, approximately 4× PCA’s 0.156, while DREAMS scores 0.055 and draws anomalies amid bulk clusters.The membership-restricted statistic tests single-point separation directly; neighbor-graph methods score near the bottom and t-SNE fuses same-kind pairs.
  • 5.5 Imbalanced two populations (minority-structure preservation): At 5% minority prevalence, only toorPIA keeps both minority-internal ρ=0.244 and cross-population ρ=0.660 clearly positive.PCA places the minority correctly but destroys its internal structure, while t-SNE, UMAP, and DREAMS preserve internals but place the minority arbitrarily.
  • 5.5 Imbalanced two populations (minority-structure preservation): The minority-pair statistic alone is insufficient: PCA scores 0.706 because 95% of minority-involving pairs are cross-population, while PCC’s global ρ=0.761 misses minority destruction.Internal minority pairs contribute only 0.25% of terms at 5%, so paired diagnostics are needed rather than a single composite.

6 Supplement: out-of-sample monitoring (addplot)

The addplot test evaluates whether unseen anomalies remain outside a normal-data basemap and whether their directions identify their source cluster. Only toorPIA answered both questions among operable methods, while several methods lacked out-of-sample operation.

  • Protocol: The basemap was fitted on normal data, after which cluster-anchored anomalies and normal controls were added one at a time.Each anomaly shared a normal cluster’s measured-feature profile but deviated along new orthogonal dimensions.
  • Operational criterion: Only toorPIA answered both monitoring questions: every anomaly fell outside the normal region and its direction identified the source cluster.Anomalies landed at a median 9.7× the bulk radius, with a minimum of 8.7×, and attribution was 10/10.
  • Operational criterion: t-SNE, PyMDE, and PCC expose no out-of-sample operation, so adding data requires refitting and can rearrange the map.The paper treats this lack of an out-of-sample operation as itself a monitoring finding.
  • Protocol: DREAMS maps each new point by openTSNE partial optimization against the fixed basemap, rather than by applying its fit-time regularization.Its regularization term is a fit-time objective over the full embedding and does not act during transformation.

7 Discussion and Limitations

The benchmark limits its claims to synthetic known-structure fidelity under a specific noise regime, while using independent metrics and preregistered hypotheses to reduce vendor-judgment dependence. Closed-source reproducibility remains asymmetric because toorPIA coordinates cannot be regenerated without API access.

  • 7.1 Scope: The benchmark does not claim superiority on real downstream tasks or general usefulness for real-world outlier detection.Its rankings characterize synthetic anomaly structure and distance preservation; validation on real data is future work.
  • 7.1 Scope: All five-dataset rankings use a redundancy-rich isotropic noise model at SNR=1 and need not transfer to sparse or irrelevant-feature regimes.The paper explicitly cautions against extrapolating these rankings beyond the tested noise setting.
  • 7.2 Conflict of interest and trust measures: Metrics are computed independently on all pairwise distances, while hypotheses are documented before corresponding results and outcomes are reported even when they contradict them.Recall@k-family metrics remain reported alongside the primary metrics as labelled biased references.
  • 7.2 Conflict of interest and trust measures: The reported results include counterexamples to vendor-favoring expectations: DREAMS leads the near band on all five datasets, while PCC and PCA lead selected comparisons.On the transition dataset, PCC wins the composite; on populations, PCA wins the scored minority-pair ρ.
  • 7.3 Reproducibility of a closed-source method: toorPIA is evaluated from committed output coordinates rather than inspected internals, enabling offline metric recomputation without an API key.The open-source methods are also byte-reproducible end-to-end from the deterministic driver.
  • 7.3 Reproducibility of a closed-source method: Third parties cannot regenerate toorPIA coordinates without vendor API access, so independent reproduction is limited to recomputing metrics from the committed cache.Readers must treat the rows as vendor-supplied coordinates if they do not trust the cache.

8 Conclusion

The paper proposes fixed-radius distance-band Shepard ρ and complementary value- and membership-based metrics to expose near-structure and population-specific failures hidden by standard scores. Across five known-geometry datasets, the benchmark shows that global rank fidelity can coexist with severe local scale collapse and that standard local metrics disagree in the predicted direction.

  • Metric contribution: Fixed-radius distance-band Shepard ρ reports a structure-adaptive near band separately from the global number, judging every point on the same absolute radius.The near band is the first mode of the pairwise distance profile.
  • Metric contribution: Value-based complements and membership-restricted variants address scale collapse, single-point outliers, and minority-population structure.The complements include band stress and tightest-cluster over-compression.
  • Interpretation: Standard recall@k, trustworthiness, and continuity use variable-radius k-NN agreement rather than faithful near-distance reproduction, and benchmark results disagree with them in the predicted direction.The paper retains these metrics as labelled references.
  • Benchmark result: A top-tier global ρ can coexist with an approximately 93× collapse of the tightest cluster’s scale, while minority structure can be destroyed despite strong global ρ.These failures are visible only when the metric set includes value-based and membership-restricted views.

Data and Code Availability

The benchmark code, committed results, figures, and toorPIA output-coordinate caches are publicly available in release v1.4.0 and archived on Zenodo.

  • Availability: The repository and Zenodo archive contain the benchmark code, committed results, figures, and toorPIA output-coordinate caches.The release tag is v1.4.0.
Loading 2608.21779v1…