ibUMAP: Coherent and Scalable Field Evaluation for UMAP Optimization

1 INTRODUCTION

• ibUMAP evaluates attraction and degree-weighted repulsion from one embedding snapshot and applies both synchronously, rather than updating coordinates after each sampled negative event.It retains UMAP’s graph construction and initialization while changing the layout-optimization stage.Source: §1 · p. 1, Figure 1 · p. 2
• The paper examines efficient evaluation of the degree-weighted repulsive field and how optimizer mechanisms affect final quality under controlled graph construction and initialization.The controlled comparisons complement end-to-end tests of speed, fidelity, and repeatability.Source: §1 · p. 1
• Three-moment FFT field evaluation supports CPU and GPU optimization; the paper reports median CPU speedups of 3.29× unseeded and 5.79× seeded over umap-learn.It also reports unseeded GPU gains over cuML on million-scale datasets and fidelity trade-offs.Source: §1 · p. 2

BACKGROUND

• Practical UMAP schedules positive graph edges, samples negative vertices, and applies each interaction immediately, making later events observe modified coordinates.Its fuzzy cross-entropy motivates the optimizer but is not the object the implementation directly optimizes.Source: §2 · p. 2

RELATED WORK

• Negative-sampling analyses motivate degree-dependent repulsion, while related work links repulsion balance and attraction shape to embedding structure and pairwise dynamics.The paper uses these analyses to motivate controlled comparisons under a fixed graph and initialization.Source: §2 · p. 2
• Prior field methods include N-body and fast-multipole gradients, GPU texture splatting, FIt-SNE’s mesh–FFT pattern, and adaptive kernel transforms.ibUMAP targets the degree-weighted repulsive operator induced by UMAP-style optimization semantics; cuML instead accelerates sampled UMAP.Source: §2 · pp. 2–3

BACKGROUND

• Embedding fidelity spans local, global, density, and inter-cluster aspects, while fidelity and repeatability measures address different questions.Rankings can depend on preprocessing, parameters, datasets, and metrics.Source: §2 · p. 3

3.1 PRELIMINARIES AND THE DEGREE-WEIGHTED REPULSIVE FIELD

• The optimizer preserves UMAP’s graph construction, initialization, attraction kernel, and parameter conventions, while replacing sampled repulsion with an FFT-evaluated field and synchronous updates.The field is motivated by the conditional expectation of negative sampling for a fixed embedding, but differs from the exact scheduled-update expectation because it changes event weights and omits per-interaction clipping.Source: §3 · p. 3, §3.1 · p. 3

3.2 THREE-MOMENT FIELD EVALUATION

• In two dimensions, degree-weighted repulsion is reconstructed from three scalar moments, with each source depositing charges (1, yj,x, yj,y) through a shared scalar kernel.The point’s own contribution cancels exactly in the reconstruction.Source: §3.2 · p. 4
• ibFFT uses adaptive-mesh deposition, FFT convolution, and interpolation; all reported configurations use p = 1, making deposition O(n) and interpolation a lookup.The resulting piecewise-constant scheme’s large zero-displacement interaction is bounded by the kernel cap; higher interpolation order did not improve fidelity in this setting and costs runtime.Source: §3.2 · p. 4

3.3 SYNCHRONOUS UPDATES AND SAFEGUARDS

• Each epoch evaluates attraction and repulsion from the same snapshot before applying updates, while safeguards clip repulsion and damp attraction at high-degree hubs.These safeguards address hub ejection under summed synchronous forces; production aggregates attraction at scheduled-edge heads, unlike controlled two-endpoint variants.Source: §3.3 · pp. 4–5

3.4 EXECUTION PATHS, COMPLEXITY, AND REPEATABILITY

• For p = 1, a sparse k-NN graph, and nominal M = Θ(n), optimizer cost is O(T[nk + n log n]) versus O(Tn^2) for direct all-pairs fields.This bound covers only optimization and does not establish an asymptotic advantage over negative sampling; fixed-order accumulation supports repeatability on the same device under a fixed backend, precision, and reduction mode.Source: §3.4 · p. 5

4 HOW EACH MECHANISM CHANGES EMBEDDING QUALITY

• The eight-variant study uses 59 datasets, fixed graphs and spectral initializations, 200 epochs, and three optimizer seeds averaged per dataset.Datasets span tabular, image, and single-cell data with 150–94,415 observations; datasets are weighted equally.Source: §4 · p. 5
• Synchrony lowers TW on all 59 datasets and NP on 57; the frozen-state event expectation recovers about 47% of mean NP loss.The raw degree-weighted field lowers both global scores on most datasets; kernel capping reverses this on 54 and 53 datasets but lowers NP on 54.Source: §4 · p. 5
• FFT evaluation leaves all five paired distributions centered at zero, with mean changes no greater than 0.0005 in local scores and 0.0016 in ρD.Its near-field error arises when pairs share a mesh box and interact at the capped zero-displacement kernel value.Source: §4 · p. 5
• Relative to UMAP, production ibUMAP matches continuity but lowers mean TW, NP, RTA, and ρD by 0.0051, 0.0209, 0.0037, and 0.0179, respectively.It improves NP over variant G on 58/59 datasets but lowers both global scores on 52/59.Source: Figure 2 · p. 6, §4 · p. 6
• Repulsion-norm clipping and attraction damping raise mean ρD by 0.0228 with a mean NP change of −0.0002.They improve C, RTA, and ρD on most datasets.Source: §4 · p. 6

5.1 BENCHMARK SETUP

• The benchmark covers 71 datasets with 150 to 4.17 million observations; runs use two-dimensional embeddings, 15 neighbors, min dist=0.1, and a requested 200-epoch budget.Five-run median timings include graph construction, initialization, optimization, and synchronized host materialization.Source: §5.1 · p. 6

5.2 END-TO-END PERFORMANCE

• 3.29× unseeded and 5.79× seeded are ibUMAP’s median CPU speedups over umap-learn; both profiles win on 60/66 datasets.Both profiles lose only below 1,000 observations.Source: §5.2 · p. 6, Figure 3 · p. 7
• 0.875 is ibUMAP’s unseeded GPU suite-median speed ratio versus cuML, despite a 1.44× median speedup across six million-scale datasets; against TorchDR it wins on 62/66 datasets with a 1.53× median speedup.Runtime comparisons use all available pairs; five large datasets were resource-skipped for CPU methods and TorchDR.Source: §5.2 · p. 6, §5.1 · p. 6, Figure 3 · p. 7

5.3 EMBEDDING FIDELITY

• ibUMAP has lower median TW, NP, RTA, and ρD but similar C; marginal medians do not establish per-dataset advantages, so the fidelity trade-off depends on downstream requirements.The main fidelity comparison uses the common 66 datasets under unseeded execution.Source: §5.3 · p. 7, Table 1 · p. 7

5.4 RUN-TO-RUN EMBEDDING STABILITY

• ibUMAP improves 15-NN overlap on 58/66 datasets versus umap-learn, 67/71 versus cuML, and 66/66 versus TorchDR; distance correlation improves on most datasets.These geometric gains imply neither input-space fidelity nor repeatable downstream partitions.Source: §5.4 · pp. 7–8

6.1 DOWNSTREAM CLUSTERING IN A BRAQUE-DERIVED WORKFLOW

• The workflow uses a BRAQUE-derived public L2 sample with 56,962 cells and 62 selected features, frozen preprocessing, and fixed HDBSCAN settings; it is not an exact reproduction of the biological analysis.Each of four CPU configurations has five runs.Source: §6.1 · p. 8
• Figure 4 shows two scheduled unseeded UMAP runs with 19.78% changed assignments, including 3.49% switching noise status, after maximum-overlap cluster matching.Run 2 is rigidly aligned for display; clustering uses the original coordinates.Source: Figure 4 · p. 8, §6.1 · p. 8
• Across ten unseeded UMAP run pairs, median distance correlation is 0.937, yet median ARI is 0.611 and 20.80% of assignments change; ibUMAP’s 0.999 distance correlation coexists with 22.01% changed assignments.The partition differences include splits and merges and measure computational variation, not biological misclassification.Source: §6.1 · p. 8

6.2 COST OF REPEATABLE REUSE

• Seeded ibUMAP is 6.91× faster than seeded UMAP, or 6.42× including HDBSCAN; its fit takes 7.10 s versus 49.06 s for seeded UMAP.Both methods produce bitwise-identical coordinates and identical HDBSCAN labels across their five seeded runs; ibUMAP retains eight jobs and matches its unseeded time.Source: §6.2 · pp. 8–9

7 LIMITATIONS AND DISCUSSION

• The present ibFFT construction is specialized to two-dimensional output; higher dimensions require separate accuracy and scaling study, and sampled UMAP may remain preferable below the empirical crossover or when fidelity dominates throughput.Fixed FFT and grid-management overhead contribute to this boundary.Source: §7 · p. 9
• Same-device bitwise repeatability is narrower than portable reproducibility; agreement can depend on the backend, FFT library, compiler, numeric type, and reduction order.The claim also does not establish local stability under small initialization and negative-sampling perturbations.Source: §7 · p. 9
• The optimizer study fixes one graph and spectral initialization, while the downstream study uses one BRAQUE-derived proxy without expert phenotype labels, leaving other initializations, domains, and human-interpretation effects untested.The dataset collection is also dependent, and the ibFFT approximation is assessed by final-embedding effects and an interpolation-order sweep rather than field-level error against the direct sum.Source: §7 · p. 9
• Because synchrony and kernel capping change cluster contraction, layout area and compactness should not be read as input density without a corresponding diagnostic.Extensions to other sampled interaction kernels require their own estimator audit, field decomposition, numerical reference, and optimizer study.Source: §7 · p. 9

8 CONCLUSION

• On 59 fixed-graph, fixed-initialization datasets, synchrony and kernel capping alter local–global fidelity, while FFT evaluation changes average quality little; end-to-end benchmarks report CPU speedups of 3.29× unseeded and 5.79× seeded over umap-learn.Unseeded GPU execution has a 1.44× median speedup over cuML on million-scale datasets; geometric stability does not ensure repeatable clustering in the BRAQUE-derived case.Source: §8 · p. 9

AI-assisted research tasks

• AI tools helped unify theoretical notation and wording, derive and audit appendix calculations, refine hypotheses and experimental design, and propose two ablation variants.The authors say they derived the main claims without AI assistance.Source: AI USE STATEMENT · p. 10

Implementation and communication

• The authors wrote the UMAP kernel and overall architecture, while AI tools helped migrate and consolidate code, prepare scripts, create and revise figures, and draft or polish text.AI implemented the downstream-case demo and synthesized narration from a script it helped edit.Source: AI USE STATEMENT · p. 10

Research decisions

• The authors report that AI tools did not generate synthetic datasets, formulate the main claims, or select evaluation metrics.They say interpretations remained theirs, although AI drafted some interpretive text that they reviewed and revised.Source: AI USE STATEMENT · p. 10

Review and verification

• The authors reviewed AI-assisted work and checked every cited reference against its original publication.They verified migrated code with unit and protocol tests, bitwise input rebuilding, rescoring checks, and regeneration of tables and figures from frozen run summaries.Source: AI USE STATEMENT · p. 10

Reported results

• Reported runtimes and quality scores come from experiment scripts and recorded run outputs.The authors take responsibility for the final content and artifacts, including AI-assisted material.Source: AI USE STATEMENT · p. 10

• The study involves no human subjects and no new data collection.Source: ETHICS STATEMENT · p. 10
• All datasets are previously published and publicly available, and human-derived single-cell data are used only as released by their providers.The repository provides download scripts rather than redistributing the data.Source: ETHICS STATEMENT · p. 10
• Faster or bitwise-repeatable embeddings are not thereby more faithful, and two-dimensional embeddings can mislead interpretation.The authors note that ibUMAP retains fidelity trade-offs.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The repository provides source code, environment specifications, data and experiment scripts, plotting scripts, result tables, and a downstream-case demo.Appendices document the expected-field derivation, mechanism-study protocols, end-to-end benchmark, and downstream case.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPECTED FIELD: DERIVATION AND IMPLEMENTATION AUDIT

• Across 59 replayed datasets, the scheduled negative-sample total is about 0.17 of its mean at epoch 1, then stays within 0.91–1.10 of that mean from epoch 10 onward.The replay uses the reference schedule without random draws, optimization, or quality evaluation.Source: A EXPECTED FIELD: DERIVATION AND… · p. 15
• The field expectation is conditional on a frozen embedding and scheduled event count, unlike online reference updates that use already-modified coordinates.The asynchronous control matches reference UMAP bitwise in all 177 retained dataset–seed comparisons, but the expectation does not imply that variant trajectories agree.Source: A EXPECTED FIELD: DERIVATION AND… · pp. 15–16
• Production ibUMAP uses head-wise attraction, giving each point half the reference attractive-event count, while its repulsive weight restores the reference expected balance between repulsion and attraction.The audit reports a median balance of 1.014 for production H; these are expected event weights, and no variant isolates the factor experimentally.Source: A EXPECTED FIELD: DERIVATION AND… · pp. 15–16
• Production safeguards combine a kernel cap, repulsion-norm clipping at 4αt, and degree-based attraction damping; the first dense-field update is purely repulsive and uses the full initial learning rate.In variant G, the repulsion-norm clip activates most at that first update, clipping a median 26.6% of points across datasets.Source: A EXPECTED FIELD: DERIVATION AND… · p. 16

B MECHANISM STUDY: PROTOCOL AND ROBUSTNESS

• The mechanism study uses fixed graphs and spectral initializations across eight variants on 59 datasets, but excludes one C. elegans dataset after an unresolved bitwise mismatch with the reference control.The study reports descriptive comparisons without noninferiority or significance claims, and its datasets are not 59 independent collections.Source: B MECHANISM STUDY: PROTOCOL AND… · p. 17
• Production ibUMAP matches UMAP in C (mean +0.0004) but trails it in mean TW, NP, RTA, and ρD by 0.0051, 0.0209, 0.0037, and 0.0179, respectively.The comparison gives the net effect of production H against A (UMAP), since H differs from G in multiple factors.Source: B MECHANISM STUDY: PROTOCOL AND… · p. 18, Table 3 · p. 18
• Without the repulsion-norm clip, points escape during the first purely repulsive update; without attraction damping, accumulated attraction displaces hubs for much of optimization.The final optimizer with both safeguards had a median bounding-box area ratio of 1.01 and no far points across the 59 datasets; the safeguard thresholds were not recalibrated for the final version.Source: B MECHANISM STUDY: PROTOCOL AND… · pp. 18–20, Figure 5 · p. 19
• Across the 30-dataset interpolation sweep, increasing order beyond p = 1 offered no consistent improvement in three local metrics, while CPU optimizer time rose 10.3% at p = 2 and 23.9% at p = 3.The tested configurations do not establish the numerical accuracy of the approximate field.Source: B MECHANISM STUDY: PROTOCOL AND… · p. 20, Table 6 · p. 20
• Under equal family weights, synchrony’s NP change remains −0.0231, FFT’s change is +0.0010, and production ibUMAP remains below UMAP on NP by −0.0139 and on both global scores.The weighting sensitivity check does not eliminate all dataset dependence; the image family contains only four datasets.Source: B MECHANISM STUDY: PROTOCOL AND… · p. 21, Table 7 · p. 21

C.1 PROTOCOL, IMPLEMENTATION PROFILES, AND DATA COVERAGE

• The end-to-end benchmark covers 71 datasets for cuML and ibUMAP CUDA, while CPU and TorchDR comparisons cover 66; resource skips are not controlled evidence of a memory advantage.FastUMAP is excluded because its upstream graph, initialization, preprocessing, timing, and quality evaluation cannot be matched independently.Source: §C.1 · p. 21
• Benchmark timings measure warm-call end-to-end fitting, including construction, graph building, initialization, optimization, synchronization, and host transfer, but not startup, loading, warmup, evaluation, or plotting.The results characterize the recorded execution configuration, not hardware-independent performance; no comparable peak-device-memory measure is available.Source: §C.1 · pp. 22–23

C.2 ADDITIONAL END-TO-END PERFORMANCE RESULTS

• CPU ibUMAP loses on all six datasets below 1,000 observations and wins on every tested larger dataset, but the suite does not establish a controlled crossover.There are no observations between 646 and 1,567 rows, and datasets also differ in dimensionality and structure.Source: §C.2 · p. 23
• Against cuML, the unseeded median speed ratio reaches 1.444 at or above 10^6 observations, rising from 0.359 below 10^4 to 0.905 and 1.062 in the next two size bins.Table 9 reports baseline/ibUMAP ratios; values above one favor ibUMAP.Source: Table 9 · p. 23, §C.2 · p. 23
• CPU seeded/unseeded median ratios are 0.961 for ibUMAP and 1.664 for umap-learn, while million-scale GPU medians are 11.718 for ibUMAP CUDA and 9.191 for cuML.On six million-scale GPU datasets, seeded ibUMAP’s median speedup over cuML is 1.05×; graph construction dominates the seeded cost.Source: §C.2 · pp. 23, 25

C.3.1 EVALUATION MEASURES AND AGGREGATION

• Fidelity evaluation uses shared sampled indices and distinguishes local neighborhood scores from global triplet and distance-correlation scores; paired differences are separate from marginal score summaries.Seeded quality uses the first run, unseeded quality averages five runs, and neither paired-difference summary counts wins or establishes statistical significance.Source: §C.3.1 · p. 27

C.3.2 EXTENDED FIDELITY SUMMARIES

• Tables 11–13 report mean and median fidelity by profile and coverage, then means and medians of paired ibUMAP-minus-baseline differences on each pair’s full available dataset set.Positive paired differences favor ibUMAP; the common-set main table uses 66 datasets.Source: §C.3.2 · p. 27, Table 11 · p. 28, Table 12 · p. 28

C.3.3 DATASET-LEVEL VARIATION AND AGGREGATION SENSITIVITY

• On the common 66 datasets, CPU ibUMAP’s higher continuity marginal median does not demonstrate improvement on a typical paired dataset; collection reweighting retains negative means for all three unseeded comparisons in TW, NP, RTA, and ρD.Reweighting changes magnitudes and some conclusions; seeded GPU differences in NP and ρD versus cuML become slightly positive. Figure 10 retains full-pair dataset-level differences.Source: §C.3.3 · pp. 29, 31, Table 14 · p. 29, Figure 10 · p. 30

C.4 REPEATABILITY AND RUN-TO-RUN STABILITY PROTOCOL

• Unseeded stability compares each group’s first run with runs 2–5; ibUMAP improves neighbor overlap on 58/66 datasets versus umap-learn, 67/71 versus cuML, and 66/66 versus TorchDR.These full-pipeline reruns can differ in graph construction, initialization, and execution order, so the comparison does not causally attribute stability to the repulsive-field evaluator.Source: §C.4 · pp. 31–32

C.5 COMPLETE DATASET MANIFEST

• The 71-dataset benchmark includes 66 common-quality entries plus five evaluated only with cuML and ibUMAP CUDA; the manifest identifies processed shapes and dataset provenance.The suite includes Tabula Sapiens, C. elegans, MERFISH, image, word-vector, and other datasets.Source: Table 17 · p. 32, §C.5 · p. 32

• The reuse case uses a frozen matrix of 62 selected features for 56,962 cells and evaluates computational partitions, not spatial organization or expert annotation quality.The public L2 single-cell protein-expression sample was processed once, and all repetitions used the same matrix and cell order.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · p. 33
• Both embedding methods use the same stated 50-neighbor, 200-epoch, spectral-initialization configuration, while seeded and unseeded profiles differ in random-state and execution settings.Seeded UMAP uses one job; seeded ibUMAP enables deterministic execution, and both methods rebuild graphs and initialization for each fit.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · pp. 33–34
• Embedding time includes graph construction, initialization, and optimization, while the combined time sums the two fit calls and is not the complete BRAQUE runtime.Each profile retained five measured runs after warmup in fresh processes; warmup does not guarantee removal of every size-dependent first-call overhead.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · p. 34
• Run disagreement is based on Hungarian matching that captures cluster splits, merges, and noise transitions, whereas local-neighborhood scores use fixed induced subsets rather than full-data neighborhoods.The ten unordered run pairs share runs and are not ten independent replicates; no noninferiority threshold or significance test is applied.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · p. 34
• 16.29% of cells changed non-noise clusters and 3.49% changed noise status between plotted runs (19.78% total); seeded trustworthiness was 0.970230 for UMAP and 0.962791 for ibUMAP.The Table 18 value of 20.80% is the median across all ten UMAP run pairs, not the plotted pair; the reported benefit is lower-cost within-profile repeatability with fidelity differences retained.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · p. 34, Table 18 · p. 34
• The measured results do not establish matching partitions across methods, improved unseeded clustering stability, cross-device reproducibility, or robustness to other seeds.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · p. 35
• The live demo reruns embedding and HDBSCAN on fixed-matrix subsets, but its single interactive-run timings and change fractions are not reported results.Figure 11 shows a repeated seeded ibUMAP run with 0.0% changed assignments and an identical fingerprint.Source: D DOWNSTREAM REUSE: BRAQUE PROTOCOL… · p. 35, Figure 11 · p. 35