PaperlayerView exact revision on arXiv

Source-linked AI summary

MaDeL: Manifold-Decomposed Feature Losses for Generative Modeling

Beomsu Kim, Jong Chul Ye, Kwanyoung Kim

arXiv:2610.04419v1cs.LGcs.AI

TL;DR

Isotropic objectives do not distinguish valid manifold-parallel variation from potentially invalid manifold-orthogonal deviation, and explicit manifold estimation is challenging. MaDeL learns complementary features from corrupted observations and uses them to define an anisotropic loss. It improves support recovery and angular accuracy on single-step benchmarks and reduces protein-backbone steric clashes across one- and few-step sampling budgets.

  • Problem

    Isotropic regression is largely direction-agnostic, although generative models must represent variation along a data manifold and suppress deviations away from it.

    Source: §1 · p. 2

  • Method

    MaDeL learns complementary features by predicting clean samples and corruptions from corrupted observations, then uses them in a geometry-aware objective.

    Source: §1 · p. 2

  • Results

    MaDeL improves support recovery and average angular W1 on single-step benchmarks and reduces protein-backbone steric clashes across one- and few-step sampling budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest that shaping the training signal's geometry is a practical alternative to parameterizing the data manifold, particularly for sparse and highly constrained generative problems.

    Source: §6 · p. 10

    The analysis is local and leading-order in corruption, and experiments do not establish scalability across models, datasets, or modalities.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative models are often trained with isotropic objectives such as mean-squared error. For data concentrated near a low-dimensional manifold, however, such losses conflate displacement along the manifold, which may represent valid variation, with displacement away from it, which produces invalid samples. This mismatch is especially problematic in sparse, highly constrained domains, where ambient-space regression can encourage off-manifold interpolation. We ask whether a generative objective can distinguish manifold-parallel variation from manifold-orthogonal deviation directly from data, without explicitly estimating the manifold. We introduce a manifold-decomposed feature loss (MaDeL) that learns complementary representations from corrupted observations: one is trained to recover the clean sample, while the other is trained to recover the corruption. We show that, under a feature bottleneck, their Jacobians align with the tangent and normal spaces, exactly for linear manifolds and locally for smooth manifolds. Together, these representations define an anisotropic objective that separately measures intrinsic variation and off-manifold deviation. Across synthetic, Earth and climate science, and torsion-angle benchmarks, MaDeL improves support recovery and average angular $W_1$ under single-step sampling; on protein backbones, it reduces steric clashes across one- and few-step sampling budgets.

1 INTRODUCTION

Isotropic regression does not distinguish valid variation along a data manifold from invalid deviation away from it, even though generative models must handle both. MaDeL learns complementary features to separate these directions and uses them in a geometry-aware objective, with theoretical and benchmark validation.

1 INTRODUCTION

  • Isotropic losses treat prediction errors largely without regard to direction, although equal-sized tangent and normal displacements can represent valid variation or invalid deviation, respectively.Generative models must suppress deviations away from valid data while modeling variation along the manifold.Source: §1 · p. 2
  • Explicitly estimating tangent and normal spaces in complex high-dimensional data is challenging, motivating learned feature spaces as a data-dependent alternative.The introduction builds on learned feature spaces used as perceptual losses for image synthesis.Source: §1 · p. 2
  • MaDeL trains complementary representations from corrupted observations by predicting the clean sample and the corruption, then measures intrinsic variation and off-manifold deviation separately.The objective distinguishes predictions that are equally distant in ambient space but differ in compatibility with the data distribution.Source: §1 · p. 2
  • Under feature bottlenecks, x-prediction and δ-prediction encoder Jacobians lie in the tangent and normal spaces, exactly for linear manifolds and locally for smooth manifolds.Their feature losses yield complementary manifold-parallel and manifold-orthogonal gradients.Source: §1 · p. 2
  • MaDeL improves support recovery and average angular W1 on single-step benchmarks and reduces steric clashes in protein backbones across one- and few-step budgets.The evaluations span a helix, Earth and climate science data, protein and RNA torsion angles, and protein backbone generation.Source: §1 · p. 2

2 BACKGROUND

This background contrasts isotropic ambient-space regression with approaches that incorporate geometry explicitly or through learned features. Prior methods draw on known state spaces or pretrained models, while feature-loss approaches typically offer a single notion of similarity.

2 BACKGROUND

  • Isotropic objectives penalize equal-magnitude residuals identically regardless of direction, despite tangent perturbations representing meaningful variation and normal perturbations potentially producing invalid deviations.The passage identifies diffusion and flow-based models as examples of methods commonly trained with isotropic ambient-space objectives.Source: §2 · p. 3
  • Prior geometry-aware methods use known state spaces or pretrained-model geometry, whereas MaDeL learns complementary tangent- and normal-sensitive features from data for the training objective.The cited categories include Riemannian dynamics, methods for known manifolds, diffusion-score decompositions, and guidance from pretrained autoencoders.Source: §2 · p. 3
  • Feature losses and regularized autoencoders can introduce directional structure, but the described approaches provide a single similarity notion rather than complementary sensitivities to intrinsic variation and off-manifold deviation.Examples include perceptual losses, denoising or reconstruction fields, discriminator features, and feature-based consistency objectives.Source: §2 · p. 3

3 MOTIVATION

Feature-loss gradients depend on encoder Jacobians, so complementary encoders can induce directional geometry. A circle toy experiment isolates the effect of the metric and shows that an oracle decomposition avoids off-manifold interpolation while retaining angular distribution accuracy.

3 MOTIVATION

  • A feature-loss gradient depends on the encoder Jacobian, and complementary encoders are assigned to manifold-parallel variation and manifold-orthogonal deviation.The resulting terms are intended to provide complementary update directions along and away from the data manifold.Source: §3 · p. 3
  • The toy comparison uses a mixture of four von Mises distributions on the unit circle, identical generators, and the same OT-matched objective, varying only the regression metric.N = ∞ denotes training with fresh samples from the underlying distribution.Source: §3 · p. 4
  • The oracle decomposed metric reduces off-manifold samples to nearly zero across dataset sizes while attaining comparable angular W1 to the target distribution.In the sparse-sample setting, Cartesian MSE and Pseudo-Huber interpolate between modes along Euclidean chords, while the oracle preserves support.Source: §3 · p. 4, Figure 2 · p. 4

4 MAIN CONTRIBUTION: MADEL

MaDeL derives complementary feature representations from x-prediction and δ-prediction on corrupted data, then uses them to define manifold-parallel and manifold-orthogonal losses. Its analysis characterizes this separation exactly for linear manifolds and locally for smooth manifolds.

4 MAIN CONTRIBUTION: MADEL

  • MaDeL learns the two representations from corrupted observations and freezes them before training the generator with ambient and feature-space losses.A shared encoder is partitioned into parallel and orthogonal feature subsets, each supplied exclusively to its corresponding decoder.Source: §4 · p. 4, §4.3 · pp. 6–7

4.1 THEORY: LINEAR MANIFOLDS AND AUTOENCODERS

  • For linear manifolds, x-prediction selects high-variance directions while δ-prediction selects low-variance directions, which can span the normal space under a suitable bottleneck.When n⊥=d−r, the δ-prediction encoder spans the complete normal space.Source: §4.1 · pp. 4–5

4.2 THEORY: GENERAL SMOOTH MANIFOLDS AND AUTOENCODERS

  • For smooth manifolds, the encoders’ Jacobians lie in the local tangent and normal spaces to leading order, respectively.The δ-prediction Jacobian spans the normal space when n⊥=d−r; the characterization is local as corruption and sample perturbation approach zero.Source: §4.2 · pp. 5–6

4.3 FROM AUTOENCODERS TO GENERATIVE LEARNING

  • The combined feature loss produces generator updates in complementary manifold-parallel and manifold-orthogonal directions.The x-prediction encoder confines feature gradients to parallel directions, while the δ-prediction encoder confines them to orthogonal directions.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments compare MaDeL-augmented OT matching with established baselines across manifold-valued and structured-data benchmarks, reporting improved single-step errors and reduced backbone clashes.

5 EXPERIMENTS

  • Evaluation spans a helix on S2, Earth and climate events, protein and RNA torsion angles, and SCOPe backbones, with comparisons to RFM, MeanFlow, RMF, and coordinate-space OTM.OTM and OTM + MaDeL share the coupling, generator architecture, and sampling procedure, differing only in regression loss.Source: §5 · pp. 7–8

5.1 MAIN RESULTS

  • Earth average angular distance falls from 5.89° to 5.59°, and torsion mean angular W1 falls from 9.540° to 8.828° with OTM + MaDeL, which wins on 3/4 and 4/5 datasets, respectively.OTM remains strongest on Flood, while OTM + MaDeL is essentially tied on Pre-Pro (6.03° versus 5.97°).Source: Table 2 · p. 8, §5.1 · p. 8
  • On the helix embedded in a sphere, OTM + MaDeL reconstructs the global helical support in one network evaluation, unlike baselines that spread samples across the sphere.Riemannian methods preserve the spherical state-space geometry but do not recover the lower-dimensional helix.Source: §5.1 · p. 8, Figure 4 · p. 8
  • Across ten backbone lengths and three sampling budgets, MaDeL lowers median steric overlap; at L = 128 and NFE = 10, overlap drops from 1.375 to 1.011 Å.Designability does not improve (10/640 versus 12/640), and median self-consistency RMSD remains essentially unchanged (8.63 versus 8.66 Å).Source: §5.1 · p. 9, Figure 5 · p. 9

5.2 ABLATION STUDY AND ANALYSIS

  • Under the same OT pairing, combining the two feature branches achieves the best average angular W1, 5.59°, versus 5.84° for the autoencoder and 6.13°/6.08° for the individual branches.MaDeL attains the lowest error on Earthquake, Fire, and Volcano, though Flood is an exception and all feature-based variants underperform coordinate-space OTM there.Source: §5.2 · p. 9, Table 3 · p. 9
  • On the helix, parallel-feature Jacobian energy ratios are 15.88–45.04, while orthogonal-feature ratios are 0.02–0.04 across noise levels.The ratio measures manifold-parallel versus manifold-orthogonal Jacobian energy at noise-perturbed samples.Source: §5.2 · p. 9

6 CONCLUSION

MaDeL separates variation along the data manifold from deviation away from it, producing complementary tangent- and normal-aligned generative updates without explicit manifold access. The authors report improved support preservation and distributional or local geometric accuracy relative to isotropic regression, while noting limits in analysis and evaluation scope.

6 CONCLUSION

  • MaDeL distinguishes variation along the data manifold from deviation away from it, yielding complementary tangent- and normal-aligned generative updates without explicit manifold access.The characterization holds exactly for linear manifolds and locally for smooth manifolds under a feature bottleneck.Source: §6 · p. 9
  • Across controlled manifolds and structured real-world domains, MaDeL better preserves data support and improves distributional or local geometric accuracy relative to isotropic regression.The conclusion presents geometry-shaped training signals as a practical alternative to parameterizing the manifold, particularly for sparse, highly constrained generative problems.Source: §6 · p. 10
  • The analysis is local and leading-order, and experiments cover limited synthetic and structured scientific benchmarks without establishing scalability across models, datasets, or modalities.Branch dimensions and loss weights must be chosen without knowing the intrinsic dimension.Source: §6 · p. 10

A PROOFS

The proofs establish how bottlenecked prediction encoders select complementary eigendirections and how their Jacobians yield tangent- and normal-aligned loss components. The smooth-manifold argument uses local leading-order reductions to linear problems.

A.1 LINEAR MANIFOLDS

  • For linear manifolds, the x-prediction proof reduces optimization to a rank-n∥ approximation that retains the largest-variance eigendirections, while δ-prediction selects the smallest-variance directions.The δ-prediction composite map has rank n⊥; tied eigenvalues at the truncation boundary permit any optimal subspace within that eigenspace.Source: §A.1 · pp. 12–13

A.2 GENERAL SMOOTH MANIFOLDS

  • For smooth manifolds, the local leading-order x-prediction problem reduces to denoising with tangent covariance, while δ-prediction retains its n⊥ smallest-eigenvalue directions.The derivation bounds squared remainders and cross terms as o(ϵ² + σ²).Source: §A.2 · p. 13

A.3 DIRECTIONAL DECOMPOSITION OF THE GENERATIVE LOSS

  • The locally optimal encoder Jacobians lie in the tangent and normal spaces, so the decomposed loss gradient separates into corresponding tangent- and normal-aligned components.The proof derives the directional decomposition by combining the Jacobian-space inclusions with the gradient identity.Source: §A.3 · p. 14

B SCOPE OF THE ANALYSIS AND LIMITATIONS

The theoretical characterization is local and leading-order, and finite corruption or geometric and optimization effects can mix the two components. The experiments and representation choices also limit the scope of what has been established.

B SCOPE OF THE ANALYSIS AND LIMITATIONS

  • The characterization holds locally and to leading order in corruption, so finite corruption, strong curvature, manifold boundaries, and imperfect optimization can mix the components.These conditions constrain how cleanly the learned geometry separates parallel and orthogonal directions.Source: B SCOPE OF THE ANALYSIS AND LIMITATIONS · p. 14
  • Corruption distribution, branch dimensions, and loss weights must be selected without knowledge of the intrinsic dimension, and feature geometry requires separate representation training for each domain.Source: B SCOPE OF THE ANALYSIS AND LIMITATIONS · p. 14
  • Experiments concern sparse structured distributions and single-step generators; validation across architectures, modalities, and scale remains unestablished.Source: B SCOPE OF THE ANALYSIS AND LIMITATIONS · p. 14

C.1 SHARED SETUP FOR THE HELIX, EARTH, AND TORSION EXPERIMENTS

The helix, Earth, and torsion-angle experiments share a training and evaluation framework, while using domain-specific data, corruption, and evaluation choices. The supplied passages describe experimental setups and metrics, not comparative outcomes.

C.1 SHARED SETUP FOR THE HELIX, EARTH, AND TORSION EXPERIMENTS

  • RFM, MF, and RMF share a four-layer, width-256 architecture, while flow methods are evaluated at NFE 1, 2, 5, and 10 and OTM variants use one evaluation.OTM and OTM + MaDeL use identical Hungarian pairings and differ only in regression loss; the feature encoder is separately trained and frozen.Source: §C.1 · pp. 14–15

C.2 HELIX ON SPHERE

  • The helix benchmark embeds clean samples in a three-dimensional subspace of S512 while ambient tangent-space corruption can leave that subspace.Evaluation includes sliced W2, energy distance, distance to the helix, displacement outside the embedding subspace, and coverage along the helix.Source: §C.2 · p. 15

C.3 EARTH AND CLIMATE SCIENCE

  • The Earth benchmark contains four event-location datasets on S2, uses deterministic 80/10/10 splits, and evaluates geodesic W1 with great-circle distance.Repeated event coordinates are retained during training to encode empirical event density.Source: §C.3 · p. 15

C.4 PROTEIN AND RNA TORSION ANGLES

  • The torsion benchmarks comprise four two-dimensional Ramachandran distributions and one seven-dimensional RNA torsion distribution, evaluated by geodesic W1 under flat-torus distance.Angles are represented by cosine and sine, and the main-table values use NFE = 1.Source: §C.4 · pp. 15–16

C.5 SCOPE PROTEIN BACKBONE GENERATION

The protein-backbone experiments compare MaDeL with a matched RMF baseline using SCOPe backbones, a frozen protein feature encoder, and identical sampling settings. They evaluate designability and steric validity across sampling budgets.

C.5 SCOPE PROTEIN BACKBONE GENERATION

  • The dataset contains 3,673 SCOPe backbones of 60–128 residues, represented by residue-local rigid frames, and the generator follows the FrameFlow architecture.The model uses node and edge dimensions 256 and 128, six invariant-point-attention blocks, eight attention heads, and self-conditioning.Source: §C.5 · p. 16
  • Reported steric evaluations use one-million-step checkpoints, although training runs for up to two million iterations.The shared RMF training setup uses the SplitMeanFlow objective and AdamW optimization.Source: §C.5 · p. 16
  • A frozen split-latent U-Net protein encoder is trained for 250,000 iterations with equal clean- and corruption-prediction weights on structured backbone corruptions.The 13 corruption operations include rigid-segment and loop displacements, chain breaks, and atom-level bond, angle, and torsion perturbations.Source: §C.5 · p. 16
  • MaDeL adds a decomposed feature-semigroup term with weight 0.0029 to the standard RMF objective; the baseline sets that weight to zero.Architecture, optimization, conditioning, and sampling settings are otherwise identical.Source: §C.5 · p. 16
  • Designability is assessed by generating 64 backbones per checkpoint, length, and NFE, then testing eight ProteinMPNN sequences per backbone after ESMFold refolding.A backbone is designable when its minimum RMSD across the eight refolded sequences is below 2.0 Å.Source: §C.5 · p. 16
  • Steric validity uses median batch maximum van der Waals overlap for backbone N, Cα, and C atoms, excluding oxygen, with real SCOPe references at 0.28–0.30 Å.Evaluation covers NFE values of 1, 5, and 10 using matched priors for baseline and MaDeL samples.Source: §C.5 · pp. 16–17
Loading 2610.04419v1…