PaperlayerView exact revision on arXiv

Source-linked AI summary

The Data Manifold under the Microscope

Marios Koulakis, Constantin Seibold

arXiv:2606.15760v1cs.LGstat.ML

TL;DR

Existing benchmarks offer either geometric control with limited realism or realistic data with poorly accessible geometry, limiting controlled study of manifold-based theory. The paper introduces a dense-manifold benchmarking framework with geometric estimators and demonstrates its use for testing manifold-fitting bounds and tracking β-VAE geometry across layers.

  • Problem

    Existing benchmarks lack both realistic data structure and precise geometric control, while manifold-based theory depends on quantities rarely observable in real data.

  • Method

    The framework constructs dense synthetic manifolds and uses finite-difference methods to estimate geometric quantities for controlled manifold-fitting studies.

  • Results

    The framework supports case studies assessing existing manifold-fitting bounds and tracking how a β-VAE reshapes manifold geometry across layers.

  • Takeaways & Limitations

    Ground-truth geometric quantities make the framework suitable for probing theoretical assumptions, validating estimators, and stress-testing bounds under controlled perturbations.

  • Takeaways & Limitations

    The framework is limited to low-dimensional manifolds of up to 4–5 dimensions and is intended for benchmarking rather than arbitrary datasets.

Abstract

from arXiv · show

A significant gap exists between theory and practice in deep learning. Generalization and approximation error bounds are often derived for simplified models or are too loose to be informative. Many rely on the manifold hypothesis and on geometric regularity such as intrinsic dimension, curvature, and reach. Progress requires insight into data-manifold geometry and suitable benchmarks, yet existing options are polarized: analytic manifolds with known geometry but limited applicability, or real-world datasets where geometry is only coarsely estimable. We introduce a benchmarking framework for studying data geometry. We repurpose and extend dSprites and COIL-20 with additional transformation dimensions and dense, axis-aligned sampling, and pair them with finite-difference estimators that recover curvature, reach, and volume at near-ground-truth accuracy in a regime where general-purpose estimators are unreliable or difficult to deploy. The framework is intended as a controlled testbed, useful as a calibration environment for geometric estimators and a sandbox for probing theoretical assumptions. To illustrate its use, we present two application studies, namely assessing the scaling behavior of the bounds of Genovese et al. and Fefferman et al., and tracking the layer-wise geometry of a $β$-VAE, highlighting the behavior of current bounds and the value of controlled benchmarks for guiding and validating future theory. A reference implementation is available at https://github.com/koulakis/manifold-microscope.

1. Introduction

The introduction identifies a gap between theoretical manifold-learning bounds and practical understanding of how networks fit data manifolds. It presents a controlled benchmarking framework with densely sampled image manifolds, geometric estimators, and case studies connecting theory to generative-model behavior.

  • Motivation: Existing manifold-learning theory emphasizes curvature, reach, sampling density, and smoothness, but its informativeness for realistic data remains unclear.The introduction cites manifold-estimation and approximation results while noting that realistic data geometry is difficult to characterize.
  • Motivation: The field needs sharper or more data-adaptive bounds and benchmarks that combine realistic data with controlled geometry.Synthetic manifolds offer geometric simplicity, whereas real-world datasets obscure geometry entirely.
  • Framework: The proposed framework densely samples low-dimensional image families along controlled transformation axes and estimates curvature, reach, and volume using finite differences.Example transformation axes include rotation, translation, and scale.
  • Contributions: Its reproducible datasets and experimental pipeline support systematic tests of generative, representation, discriminative, and geometric-measure learning.The framework is designed as a general testbed rather than only a generative-model benchmark.
  • Case studies: Two case studies compare observed errors with existing manifold-fitting bounds and track how a β-VAE reshapes manifold geometry across layers.These studies illustrate how the framework probes theory–practice alignment and layer-wise representation geometry.

2. Related Work

Prior work provides theoretical guarantees for manifold geometry, fitting, reach preservation, and property estimation, but empirical validation is limited by missing exact geometric ground truth. The framework addresses this gap with dense axis-aligned sampling and finite-difference estimators in a controlled calibration testbed.

  • Manifold Analysis: Manifold-analysis research studies topology, geometry, and learned representations under the manifold hypothesis, including homology recovery and limited curvature in VAE latent spaces.Related geometric invariants include curvature, tangent spaces, reach, and volume.
  • Manifold Fitting Bounds: Genovese et al. established the minimax rate O((1/n)2/(2+d)), later shown optimal, although the corresponding estimator is computationally infeasible.Other work gives nonasymptotic rates depending on manifold smoothness and addresses tangent-space, curvature, and unbounded-noise settings.
  • Reach-Based Methods: Reach-based methods preserve a structural geometric primitive and noisy-data extensions reduce complexity from double- to single-exponential, but guarantees depend on unknown reach and volume.The constants in Yao & Xia (2025) scale as τ −2.
  • Geometric Property Estimation: Recent estimators target reach, curvature, the second fundamental form, and Ricci curvature, yet empirical validation remains difficult without ground-truth geometric data.These methods include nonasymptotic sample-complexity and curvature-estimation guarantees.
  • Contribution: The framework combines dense, axis-aligned sampling with finite-difference estimators to validate geometric estimators and probe theoretical predictions at low intrinsic dimension.It is intended as a controlled calibration testbed for empirical geometric validation.

3. Background and Definitions

The paper models controlled datasets as low-dimensional manifolds with known transformation factors, discretizes them on structured grids, and defines geometric quantities and manifold-fitting bounds for analysis. Its framework emphasizes intrinsic dimension, reach, curvature, volume, and Hausdorff error as central descriptors and evaluation targets.

  • Datasets as Manifolds: Datasets are modeled as unions of connected smooth d-dimensional manifolds embedded in R^D, with known and fixed intrinsic factors of variation.Each semantic class typically corresponds to a separate connected component.
  • Datasets as Manifolds: Each manifold is assumed homeomorphic to [0, 1]^r × (S^1)^s, combining non-cyclic interval directions with cyclic directions.The dSprites example has topology [0, 1]^3×S^1 and is embedded in R^4096.
  • Datasets as Manifolds: A structured grid samples every transformation dimension, and each grid point maps through class-specific transformations to a discrete parametrization of the corresponding manifold.Cutting along cyclic directions produces discrete manifold patches.
  • Geometric Measures: The framework studies volume, scalar curvature, and reach globally and locally, while Hausdorff distance measures the largest pointwise discrepancy between a true and learned manifold.Reach is characterized by unique nearest neighbors within its radius, and local counterparts include volume elements, pointwise curvature, and local reach.
  • Manifold Fitting and Bounds: The reviewed fitting results express estimation difficulty through intrinsic dimension and manifold geometry, with Genovese et al. showing exponential dependence on d but not ambient dimension D.The comparison uses Genovese et al.’s dimension-dependent minimax scaling and Fefferman et al.’s linear-fit scaling.

4. Methods and Framework

The framework builds controlled low-dimensional manifolds on dense, axis-aligned grids, enabling finite-difference recovery of geometric quantities and reproducible evaluation of manifold-fitting theory. It is designed as a controlled benchmark rather than a competitor to general point-cloud estimators.

  • Controlled grid construction: Controlled manifolds with intrinsic dimension d = 1–4 are sampled on regularly spaced grids, enabling stable finite-difference computation of geometric quantities.The framework supports accurate computation of induced metrics, volume elements, curvature tensors, and reach.
  • Controlled grid construction: Analytic manifolds use known parametrizations, while image-based datasets use systematic translations, rotations, and scalings to sample all transformation combinations.This creates an analogous structured grid for domain-specific datasets.
  • Geometric computation: Finite differences achieve O(h2) error for metric-derived quantities, volume forms, Christoffel symbols, curvature tensors, and scalar curvature under stated smoothness and non-degeneracy conditions.The method assumes C3 smoothness for reach and volume and C5 smoothness for scalar curvature; the metric must satisfy λmin(g) ≥c > 0.
  • Comparison and validation: General curvature and reach estimators handle broader unstructured-sample settings but require sophisticated interpolation, have complex rates and constants, incur computational overhead, and have few implementations.The authors therefore position the grid-based approach as a straightforward, reproducible benchmark and unit-test environment rather than a competing general estimator.
  • Geometric computation: Finite-difference tangent estimates have error O(h2), contributing O(h) under tangent-perturbation stability when grid separation satisfies δ ≍h.Thus the conservative local-curvature regime retains its dominant theoretical term up to constants and logarithmic factors.
  • Comparison and validation: The finite-difference method is sanity-checked against Sritharan et al. (2021) on known-curvature manifolds by comparing scalar-curvature RMSE across sample sizes.For the comparator, the study additionally grid-searched its sensitive radius hyperparameter.

5. Applications

The applications use controlled toy and image-manifold benchmarks with MMLS and β-VAE fitting to evaluate approximation-error bounds and track learned manifold geometry. Fefferman et al.’s scaling better matches empirical errors, while deeper β-VAE layers become more curved, lower-reach, and more class-separated.

  • Applications: The framework combines analytically tractable toy manifolds with adapted dSprites and COIL-20 datasets for controlled geometric evaluation.The image datasets use improved anti-aliasing, additional transformations, and dense grid sampling with margin oversampling to support finite-difference estimates.
  • Applications: MMLS fits manifolds directly in data space, whereas β-VAE learns a latent manifold and is used only for image datasets.MMLS uses a weighted local d-plane fit; β-VAE reconstruction distance is interpreted as distance to the learned manifold.
  • Error bounds evaluation: The error-bound study compares intrinsic dimensions, fitting models, and datasets using directed Hausdorff distance from the ground-truth manifold to the fitted manifold.Theoretical constants are estimated by aligning bound shapes with observed error curves, then selecting envelope constants using empirical error quantiles.
  • Error bounds evaluation: Fefferman et al.’s scaling is closer to empirical error curves than Genovese et al.’s dimension-only rates across datasets and models.“Closer” refers to the fitted log-log decay exponent, not predictive constants or absolute curve height, because constants are selected post hoc to envelope empirical curves.
  • Layer-wise geometry: In deeper β-VAE layers, curvature increases and reach decreases, while average distances between class manifolds increase toward the latent representation.These trends indicate progressively more intricate intermediate manifolds and stronger class separation as semantic information strengthens.

6. Discussion

The framework enables controlled manifold-fitting studies by combining dense synthetic manifolds with accurate ground-truth geometric estimates. Its limitations include low intrinsic dimension, simple topology, and potential rasterization aliasing, while future work would broaden datasets and estimator evaluation.

  • Contributions: The framework constructs dense manifolds and accurately estimates their geometric properties for controlled studies of manifold fitting.Demonstrated applications assess manifold-fitting bounds through log-log scaling fits and analyze layer-wise geometry in a β-VAE.
  • Contributions: Ground-truth geometric quantities balance realism and control, supporting theoretical-assumption probing, estimator validation, and stress-testing bounds under controlled perturbations.This addresses the unreliability of geometry estimates on real datasets and the limited representational richness of simple analytic manifolds.
  • Limitations: The framework analyzes only low-dimensional manifolds, up to 4-5 dimensions, so it is intended for benchmarking and understanding rather than arbitrary datasets.Its stated scope limits direct analysis of general high-dimensional data.
  • Limitations: The framework currently supports simple topology, [0, 1]r × (S1)s, while more general manifolds can be handled chart-wise.Covering a manifold with charts allows the framework to be applied separately on each chart.
  • Limitations: Fine transformation grids in rasterized image datasets can cause aliasing artifacts at fixed resolution, requiring sufficient resolution or metric-tensor smoothing.Grid refinement should be paired with increased image resolution or regularization such as metric-tensor smoothing.
  • Future work: Future work could add richer transformations, occlusions, and modalities, and evaluate geometric estimators against accurate finite-difference values on test manifolds.The proposed extensions include text and audio modalities and comparisons for curvature or reach estimation.

A. Bounds from Fitting a putative manifold to noisy data (Fefferman et al., 2018)

This section derives the sample complexity of Fefferman et al.’s manifold-fitting bound for noisy data. Under smoothness, reach, sampling, and Gaussian-noise assumptions, the algorithm outputs a manifold with a high-probability Hausdorff-error guarantee controlled by r².

  • Assumptions: The analysis assumes a boundaryless C2 d-dimensional manifold M in R^D with reach at least τ, sampled i.i.d. from a measure on M.The sampling measure is described relative to the d-dimensional Hausdorff measure λ_M.
  • Assumptions: Independent isotropic Gaussian noise with standard deviation σ in D dimensions is added to the sampled points, with V = λ_M(M) denoting manifold volume.The noise variables are D-dimensional and isotropic.
  • Guarantee: With high probability, the algorithm constructs an output manifold M_o having the same properties as M and satisfying H(M_o, M) < C d^7 r^2.The guarantee is based on Theorem 10 of Fefferman et al. (2018).
  • Combined bound: Combining the admissible-range and sample-size constraints yields a Hausdorff-error bound expressed through the maximum of the resulting constraints.The displayed combined expression is presented as H(M_o, M) < C d^7 max.

B. Derivation of sample complexity for scalar curvature … METHODS CONFIGURATION

The appendix derives finite-difference sample-complexity rates for scalar-curvature estimation, extending central-difference accuracy to partial and mixed derivatives under smoothness and metric-conditioning assumptions. It also specifies controlled manifold benchmarks and compares the proposed grid-based estimator with Sritharan et al. (2021).

  • B. Derivation of sample complexity for scalar curvature: The appendix derives asymptotic sample-complexity results for the finite-difference methods used throughout the paper.The derivations establish the theoretical basis for the scalar-curvature estimator.
  • B.1. Useful lemmas for higher-order central differences: Higher-order central differences use a recursively defined operator whose moment cancellations eliminate terms below order k and the order-k+1 term.The stated identities include S_k(k) = 2^k k! and S_{k+1}(k) = 0.
  • B.1. Useful lemmas for higher-order central differences: For f ∈ C^{k+2}, the k-th central difference approximates the corresponding derivative with O(h^2) error.The proof uses Taylor expansion and the moment cancellations to remove lower-order terms.
  • B.2. Extension to partial and mixed derivatives: The same finite-difference construction estimates partial and mixed derivatives for multi-index α when f ∈ C^{|α|+2} and all stencil points remain in the domain.For non-cyclic directions, the implementation uses interior grid points and crops a margin proportional to the stencil radius.
  • B.3. Implication for scalar curvature: Under u ∈ C^5 and a uniformly non-degenerate metric, derivatives through order 3 and the resulting scalar-curvature calculations inherit O(h^2) error.The propagation applies through sums, products, and multiplication by g^{-1}.
  • B.3. Implication for scalar curvature: The pointwise scalar-curvature estimator converges at O(h^2), translating under h ≍ n^{-1/d} to the stated sample-complexity scaling up to derivative- and conditioning-dependent constants.The grid-spacing relation applies to n points in d dimensions with comparable resolution.
  • C. Empirical comparison on scalar curvature estimation; HYPERSPHERES S2, S3, S4; HYPERBOLOID H22: Empirical evaluation compares the proposed estimator with Sritharan et al. (2021) on unit hyperspheres, a two-dimensional hyperboloid, and a two-dimensional torus.The hypersphere benchmark has constant scalar curvature R(ϕ1, . . . , ϕd) = d(d − 1), while the hyperboloid uses semi-axes (1, 2, 2).
  • SAMPLING; METHODS CONFIGURATION: For target sizes n ∈ {2000, 4000, . . . , 20000}, each parameter is sampled on an m-point Cartesian grid with m = ⌊n^{1/d}⌋, so realized sizes equal m^d and vary across dimensions.The proposed method runs directly on the sampled grid, whereas the baseline is ported from MATLAB, evaluated at 50 candidate hyperparameter values, and reported using the best mean squared error, making scores mildly optimistic.

D. Grid-Resolution Ablation for Image-Based Measures

The ablation tests whether image-based volume, curvature, and reach estimates change moderately under finer transformation-parameter grids. It supports density n = 16 as a practical compromise, while showing that fixed image resolution limits strict convergence and can introduce rasterization artifacts.

  • Protocol: The ablation varies transformation-parameter grid density around the main-experiment setting of n = 16 values per dimension for dSprites and COIL-20.dSprites varies horizontal translation, vertical translation, scale, and rotation; COIL-20 varies scale, in-plane rotation, and turntable view.
  • Comparison method: Consecutive grids are interpolated to a common quadrature grid and compared with a negative Sobolev norm rather than pointwise L2.The weak norm reduces sensitivity to high-frequency differences, small spatial shifts, and rasterization noise.
  • Observations: Volume element estimates are most stable, whereas curvature shows larger class-dependent variation and stronger high-frequency changes across densities.At fixed image resolution, finer grids increasingly expose interpolation and subpixel rasterization artifacts, which curvature amplifies through further finite differences of the metric.
  • Conclusion: The results support balanced image datasets at density n = 16 as a practical compromise between computational cost and estimator stability.The estimates are not strictly asymptotically converged, but the main trends are not attributable to one very coarse discretization choice.
  • Limitations and recommendations: Increasing parameter density at fixed image resolution can make neighboring images differ mainly through subpixel rasterization and interpolation effects.The paper recommends weak-metric comparisons, increasing image resolution with parameter density, and considering metric-tensor smoothing.

E. Datasets … E.3. Geometric Measures on Projections

The paper develops dense, transformed dSprites and COIL-20 benchmarks with controlled sampling and preprocessing, then evaluates geometric measures on their projections. These datasets support geometric estimation and reveal smoother large-scale structure in absolute scalar curvature than in signed scalar curvature.

  • E. Datasets: The framework extends dSprites and COIL-20 into controlled dense grids for studying data-manifold geometry.The datasets are paired with geometric measurements and projection-based analyses.
  • E.1. dSprites: The balanced dSprites variant contains 3 × 16^4 = 196,608 images across shape, scale, orientation, and two-dimensional position factors.Optional bidirectional buffers add b grid steps on scale and translation axes while retaining the central 16 values for evaluation.
  • E.2. COIL-20: The COIL-20 grid adds independent in-plane rotation and scale factors while keeping outputs at fixed 64 × 64 resolution.The construction supports optional buffers for finite-difference estimators requiring neighboring points along non-cyclic axes.
  • E.2. COIL-20: 92,160 images comprise the COIL-20 configuration with 20 objects, 18 subsampled views, and 16 scale values, excluding buffer-only samples.The count is given as 20 × 18 × 16^2.
  • E.3. Geometric Measures on Projections: On heart dSprites and duck COIL-20 projections, absolute scalar curvature shows smoother large-scale structure and clearer alignment with reach than signed scalar curvature.Signed scalar curvature exhibits stronger local fluctuations that should not be overinterpreted.

F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold · G. Manifold Fitting

The paper isolates a two-dimensional rasterized ellipse manifold to examine finite-difference geometry estimates, showing that fine grids can amplify rasterization artifacts and that metric smoothing reduces high-frequency variation. It also introduces the experimental setup for the manifold-fitting use case.

  • F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold: The ellipse manifold varies horizontal translation x and in-plane rotation θ, with x non-cyclic and θ ∈ [0, 2π) cyclic.This controlled grid supports direct inspection of finite-difference estimates.
  • F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold: Central finite differences estimate the Riemannian metric, from which the volume element √det g and scalar curvature are derived.Direct estimates are compared with results obtained after smoothing the metric tensor.
  • F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold: At fixed 64 × 64 resolution, increasingly fine parameter grids can make finite-difference quotients amplify interpolation and rasterization artifacts.The resulting estimates may reflect image-formation discretization rather than only the underlying continuous transformation.
  • F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold: Metric-tensor smoothing filters finite-difference components over the parameter grid before recomputing the volume element and scalar curvature.The implementation uses a separable Gaussian kernel with reflection padding for x and wrapped padding for θ.
  • F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold: Smoothing reduces much of the high-frequency variation, especially in scalar curvature, when neighboring metric values should vary gradually on a sufficiently fine grid.The paper nevertheless uses unsmoothed estimates in its main experiments because their moderate density is already in a useful range.
  • F. Rasterization Effects and Metric Smoothing on a Two-Dimensional Ellipse Manifold: UMAP is used only to visualize the ellipse samples; identical coordinates for smoothed and unsmoothed rows isolate differences in estimated measures.The projection shows the same qualitative reduction of local high-frequency variation after smoothing.
  • G. Manifold Fitting: The manifold-fitting use case documents its datasets, fitting models, experimental procedure, and estimation of bound curves.The supplied passage identifies these as the setup details covered in the section.

G.1. Fitting methods … POINTWISE PROXY FOR THE HAUSDORFF DISTANCE

The paper separates fitting from evaluation through dense reference sets, implements MMLS and β-VAE reconstruction procedures, and compares theoretical bound curves with empirically fitted power laws. It also uses a pointwise projection-error proxy for Hausdorff distance, which tracks the full distance closely except when sample sizes are low.

  • THE FITTING PROCESS AND DATA SELECTION: Dense evaluation set Y is kept separate from fitting set X, and maximum projection or reconstruction error on Y approximates the directed Hausdorff distance.For toy manifolds, Y is densely and approximately uniformly sampled; for image manifolds, Y is selected from a grid while X excludes Y.
  • THE FITTING PROCESS AND DATA SELECTION: Evaluation sets are generated by closed-form grids or Sinkhorn-based centroid selection for toy manifolds and maximal-distance subsampling for image manifolds.Fitting points use uniform sampling for toy manifolds and a grid-based distribution normalized by the per-point volume element for image manifolds.
  • MANIFOLD MOVING LEAST SQUARES (MMLS): MMLS estimates local affine structure with distance-weighted neighbors and projects queries using a degree-1 local linear approximation.The experiments use k = 5 for toy manifolds, k = 2d+1 for image manifolds, and Gaussian bandwidth σ = 1.0.
  • MANIFOLD MOVING LEAST SQUARES (MMLS): The β-VAE uses model B for dSprites and model H for COIL-20, with latent dimension zd = 10 and dataset-specific learning rates.The learning rates are 5 × 10−4 for dSprites and 10−4 for COIL-20; the decoder reconstructs evaluation points Y after training on X.
  • TAKEAWAYS ON CONSTANTS IN THE GENOVESE ET AL. BOUNDS: Genovese et al.’s bounds are used mainly for their n-scaling and intrinsic-dimension dependence because explicit constants become impractical in high-dimensional settings.The lower-bound rate is n−2/(d+2), while upper-bound constants become extremely large through combinatorial and ambient-dimension dependence.
  • ESTIMATING THE BOUNDS’ CONSTANTS: Empirical upper and lower constants are selected by fitting logarithmic error curves and placing theoretical curves above or below 0.99 empirical-error quantiles.The regression uses log Rn ∼−g(d) log n + log C = −A log n + log C, with sample fractions [0.01, 0.02, 0.05, 0.1, . . . , 1.0] repeated three times.
  • ANALYSIS OF THE FITTED EMPIRICAL CURVES: Most empirical curves are well approximated by single power laws, but the β-VAE cross-model experiment has substantially lower fit quality, with R2 = 0.62.The lower R2 agrees with fitting issues noted for the β-VAE setting.
  • POINTWISE PROXY FOR THE HAUSDORFF DISTANCE: The pointwise proxy upper-bounds the symmetric Hausdorff distance and generally follows its trend with mild overestimation, while discrepancies are largest at lower sample sizes.The proxy is reasonable when the fitted manifold is not strongly distorted and the nearest projected point remains close to its paired projection.

H. Per-Class Manifold Geometry on COIL-20

Per-class COIL-20 geometry is reported across all 20 object classes for the six measures tracked through the β-VAE. The curves retain the main-text trends while revealing inter-class variation hidden by Figure 6’s aggregated ±1σ band.

  • Layer-wise layout: The layer axis runs from input through encoder and latent µ to decoder and output, with encoder and decoder halves shaded blue and orange.All measures are normalized except volume.
  • Aggregate comparison: Per-class curves follow the main-text qualitative trends while exposing inter-class spread aggregated into Figure 6’s ±1σ band.The main-paper summary averages the COIL-20 results over the 20 object classes.
  • Per-class results: Figure 31 reports six geometric measures separately for each of the 20 COIL-20 object classes.Each curve represents one object class rather than repeated training runs.
Loading 2606.15760v1…