Align Your Tangent: Training Better Consistency Models via Manifold-Aligned Tangents

1 INTRODUCTION

• Consistency models can generate samples with one or two neural-network evaluations, addressing interest in reducing diffusion and flow-model inference cost.The paper notes that CMs can be unstable and prone to divergence during training.Source: §1 · p. 1
• CM tangents near convergence contain oscillatory components parallel to the data manifold, which the authors hypothesize can hinder convergence.They propose amplifying manifold-orthogonal components that point toward the manifold.Source: §1 · p. 2
• MFD uses feature maps whose Jacobian directions point toward the data manifold, producing manifold-aligned tangents with fewer oscillatory components.The loss is presented as simpler to construct than LPIPS and does not require human supervision.Source: §1 · p. 2
• AYT accelerates CM convergence by orders of magnitude over pseudo-Huber, improves generative performance, beats LPIPS, and yields competitive FID with batch size 16.Experiments cover CIFAR10 and ImageNet 64 × 64.Source: §1 · p. 2

Regularization for DMs and CMs

• Regularization work includes early stopping, data augmentation, and auxiliary learning signals such as contrastive objectives and alignment with pretrained representations.These approaches target overfitting, generalization, robustness, or training stability.Source: §2 · pp. 2–3

Perceptual objectives

• Perceptual metrics can be used directly as consistency losses, while their use in diffusion-model training may require different strategies.Some diffusion methods apply perceptual losses after pretraining or as auxiliary losses.Source: §2 · p. 3

Fast sampling of diffusion and flow models

• Fast-sampling methods include improved ODE solvers, trajectory-straightening approaches, diffusion distillation, and flow-map methods.Consistency models are a flow-map special case that predicts the final image from noise by setting the target timestep to zero.Source: §2 · p. 3

3 BACKGROUND

• A forward corruption process takes data x toward noise, and the probability-flow ODE has marginals matching the time-indexed distributions p_t.The process is parameterized by time and has boundary condition x_0 = x.Source: §3 · p. 3
• Numerically solving the probability-flow ODE requires multiple velocity evaluations, making generation slow and costly.Sampling starts from noise at a sufficiently large time and integrates down to zero.Source: §3 · p. 3
• A consistency model is trained to give identical outputs along probability-flow ODE trajectories, allowing a single evaluation to map noise to data.The model has boundary condition fθ(·, 0) = id.Source: §3 · p. 3
• The discrete consistency objective compares outputs at consecutive timesteps using a loss such as LPIPS, mean squared error, or pseudo-Huber.Consistency distillation estimates the earlier point using flow velocity, while consistency training uses the corruption-process derivative.Source: §3 · p. 4
• A CM tangent is the derivative of its output along a probability-flow trajectory, and CM learning can be viewed as contracting each output path toward its data endpoint.Discrete and continuous CM differ in whether tangents use finite differences or an exact derivative.Source: §3 · p. 4

The oscillatory tangent hypothesis

• The authors assume data lies on a low-dimensional manifold and hypothesize that stochastic training can make tangents oscillatory and hinder CM convergence.They call these claims the oscillatory tangent hypothesis.Source: §4 · pp. 4–5

Observations on CIFAR10

• CIFAR10 tangents after 400k training iterations showed structured patterns suggesting large movements along the manifold rather than toward it.The model was trained until FID no longer changed substantially.Source: §4 · p. 5

Analysis on synthetic data

• The synthetic dataset consists of two-dimensional discs moving vertically or horizontally, forming a curved manifold with tangent directions corresponding to those movements.The CM was trained for 200k iterations and analyzed at several noise levels.Source: §4 · p. 5
• The tangent decomposition defines the manifold-parallel component as its projection onto the manifold tangent space and the orthogonal component as the remainder.The two components are mutually orthogonal.Source: §4 · p. 5
• On the converged synthetic-disc CM, tangents contained non-trivial manifold-parallel components, especially at t ≥ 1, and the analysis found overwhelmingly large manifold-parallel components.The authors note that oscillations at large t conflict with mapping noise to data.Source: §4 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5

5.1 CONSISTENCY MODEL TRAINING WITH FEATURE DISTANCE

• With squared feature distance, the CM tangent is a linear combination of feature-map Jacobian rows, so the map controls its direction.With the identity map, the tangent can point in any direction.Source: §5.1 · p. 6

5.2 ALIGN YOUR TANGENT (AYT) WITH MANIFOLD FEATURES

• AYT constructs features from transformations of the data manifold; their level-set gradients are expected to point toward the manifold and align tangents.The level sets represent increasingly perturbed versions of the manifold as the transformation parameter grows.Source: §5.2 · p. 6
• The image-domain implementation uses 15 degradation, geometric, and color transformations and a VGG16 network for manifold features.Intermediate max-pooling features are also used, in the spirit of LPIPS.Source: §5.2 · p. 7
• Unlike the original CM's fixed identity map, AYT learns its feature map dynamically.The proposed manifold feature distance is self-supervised and requires no human supervision.Source: §5.2 · p. 7
• AYT is designed to align CM tangents toward the data manifold, whereas LPIPS relies on ImageNet labels and human-curated patch similarities.The paper reports that AYT outperforms LPIPS for CM training.Source: §5.2 · p. 7

6.2 COMPARISON WITH OTHER METHODS.

• AYT lowers 1-step FID from 3.60 to 2.61 on CIFAR10 and from 5.51 to 4.41 on ImageNet 64 × 64, with comparable 2-step FIDs.The sample-quality comparison covers unconditional CIFAR10 and class-conditional ImageNet 64 × 64.Source: Table 1 · p. 8, §6.2 · pp. 8–9

6.1 ABLATION STUDIES

• Compounding feature transformations benefits CM learning, with geometric transformations producing the largest FID improvement.The authors interpret this as evidence that vanilla tangents provide a weak training signal toward the manifold in geometric directions.Source: §6.1 · pp. 7–8
• AYT retains strong FID scores at batch size 16 and beats ECT trained with batch size 128.The reported batch-size comparisons include 16, 32, 64, and 128.Source: §6.1 · p. 8
• AYT beats LPIPS in one- and two-step generation, while LPIPS-trained CMs show rapidly diverging denoising FIDs after 50k iterations.The denoising FIDs compare LPIPS, pseudo-Huber, and AYT at t = 0.8.Source: §6.1 · p. 8, Figure 6 · p. 8

6.2 COMPARISON WITH OTHER METHODS.

• AYT reaches 2.83 FID versus iCT on CIFAR10 and competitive ImageNet performance with batch size 128, 8× smaller than iCT's 1024.The authors report that AYT does not require schedule tuning or multi-stage optimization.Source: §6.2 · p. 9
• On CIFAR10, AYT is competitive with CTM (FID 1.87) and SiD (FID 1.92) without pretrained teachers or adversarial training, and it outperforms several distillation approaches on ImageNet 64 × 64.The authors report training AYT from scratch despite distillation baselines often inheriting pretrained diffusion-model priors and score functions.Source: §6.2 · p. 9
• AYT's 2-step performance surpasses DPM-Solver++ and DPM-Solver-v3, which use NFE ≥10.The authors describe this as high-quality generation under extreme computational budgets.Source: §6.2 · p. 9

• The framework could be explored on audio, text, or multimodal data using domain-specific augmentations.For audio, the discussion names time-stretching, pitch-shifting, masking, and background noise as examples.Source: §7 · p. 9
• The study is limited to relatively small-scale settings and evaluation up to 64 × 64 resolution, with additional training and memory use.The authors describe the auxiliary classifier as lightweight and expect these constraints to be less critical at larger scale.Source: §7 · p. 9
• The authors suggest the approach may fit latent-diffusion pipelines, which downsample images by a factor of 8, but higher-resolution experiments remain open.The discussion frames the work as a first step toward scaling and broader applications.Source: §7 · p. 9

8 CONCLUSION

• MFD uses an auxiliary network trained to detect off-manifold perturbations to align CM tangents toward the data manifold.The conclusion reports orders-of-magnitude stabilization over pseudo-Huber while improving sample quality.Source: §8 · p. 9

A EXPERIMENT SETTINGS

• The experiments follow ECT defaults for preprocessing, the forward process, timestep sampling, and evaluation unless noted otherwise.The consistency models are initialized from pretrained diffusion models of the corresponding architectures.Source: A EXPERIMENT SETTINGS · p. 13, §A.2 · p. 13

A.1 DATA PREPROCESSING

• CIFAR10 and ImageNet 64 × 64 preprocessing follows code provided by Karras et al. (2022).Source: §A.1 · p. 13

A.2 MODEL ARCHITECTURES AND INITIALIZATION

• The manifold-feature classifiers use VGG16 networks trained from scratch, while the CM backbones are DDPM++ for CIFAR10 and EDM2-S for ImageNet 64 × 64.The CM backbones are initialized with pretrained diffusion models of their corresponding architectures.Source: §A.2 · p. 13

A.3 CLASSIFIER TRAINING CONFIGURATIONS

• Manifold features use Adam at learning rate 0.0001 and train for 400k iterations to minimize Eq. (15).The color and geometric transformation pipeline largely follows Karras et al. (2020).Source: §A.3 · p. 13
• Classifier augmentations include Gaussian noise x + αϵ and Mixup samples (1 − α)x + αy.The noise magnitude is sampled uniformly from 0 to αmax; Mixup uses α sampled uniformly from 0 to 0.5.Source: §A.3 · p. 13

A.4 CONSISTENCY MODEL TRAINING CONFIGURATIONS

• CM runs use global batch size 128 except in the batch-size ablation, with EMA enabled throughout training.Source: §A.4 · p. 13
• CIFAR10 CMs train for 400K iterations with RAdam at learning rate 0.0001 and EMA decay 0.9999, without a multi-stage schedule.Source: §A.4 · p. 13
• ImageNet 64 × 64 CMs train for 200k iterations with the ECT multi-stage schedule, enabling AYT after 75K iterations to mitigate early overfitting.Training uses Adam at learning rate 0.001 with inverse-square-root decay; EMA uses the EDM2 Power-EMA formulation without post-hoc EMA.Source: §A.4 · p. 13

A.5 SAMPLING AND EVALUATION

• FID compares the training set with 50K generated samples for one- and two-step sampling.Unless stated otherwise, the evaluation follows ECT and reports FID using 50K samples.Source: §A.5 · p. 13
• For two-step sampling, the intermediate timestep is t = 0.821 on CIFAR10 and t = 1.526 on ImageNet 64 × 64, following ECT.Source: §A.5 · p. 13

B CONSISTENCY MODEL SAMPLES

• Uncurated one-step CM samples are shown for CIFAR10 and ImageNet 64 × 64.Source: Figure 7 · p. 14, Figure 8 · p. 15