PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Discriminative Geometry for Drifting Models

Doudou Zhang, Wenwen Hou, Yilin Chen, Qi Chen

arXiv:2610.04703v1cs.LGcs.CV

TL;DR

Drifting Models can generate in one step, but their performance on complex images depends on representation geometry, and the role of that geometry remains unclear. This paper learns discriminative representations across batches, relates KDE ratio and drift-regression gradients under matched conditions, and reports substantially lower FID than pixel-space drifting across multiple datasets. It also identifies velocity clipping and adapting pretrained representations as further sources of gains.

  • Problem

    How representation geometry shapes drifting optimization—and whether effective geometry can be learned with the generator rather than fixed in a pretrained encoder—remains poorly understood.

    Source: §1 · p. 1

  • Method

    Persistent representation learning accumulates discriminative updates across batches as the generator evolves, and the paper establishes a matched-condition current-step gradient equivalence between KDE ratio loss and drift regression loss.

    Source: §1 · p. 2

  • Results

    Approximately 82–95% lower FID than the original pixel-space Drifting Models across multiple datasets without pretrained encoders; adapting pretrained representations and velocity clipping provide further gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Representation geometry determines KDE weights and drift directions, and learning that geometry across batches supports effective drifting directly from pixels.

    Source: §1 · pp. 1–2

    The method may inherit or amplify biases, privacy risks, and other limitations of underlying data and pretrained models, so deployment should include appropriate safeguards and license compliance.

    Source: Ethics statement · p. 10

Abstract

from arXiv · show

Recently proposed Drifting Models shift iterative distribution refinement from inference to training, enabling effective one-step generation. However, their performance on complex image datasets depends strongly on the representation used to construct the drifting field: pixel-space drifting performs poorly, whereas pretrained feature spaces substantially improve sample quality for reasons that remain unclear. We trace this gap to the discriminative geometry of the representation, which determines sample weighting in kernel density estimation (KDE) and, consequently drift. We introduce persistent representation learning, which continuously learns a more discriminative representation geometry as the generator evolves across batches. We further establish a current-step gradient equivalence between the KDE ratio loss and drift regression loss under matched conditions, connecting density-ratio-based generator optimization to empirical drifting and motivating direct control of the drifting velocity. Across multiple datasets, our method learns effective discriminative representations directly from pixels and reduces FID by approximately $82-95\%$ over the original pixel-space Drifting Models, without pretrained encoders. Adapting pretrained representations and applying velocity clipping provide further gains.

1 Introduction

The introduction asks why representation geometry makes drifting more effective and whether that geometry can be learned jointly with the generator. It presents persistent representation learning, a matched-condition gradient equivalence, and velocity clipping as ways to address that question.

1 Introduction

  • Pixel-space drifting often yields poor sample quality, while pretrained feature spaces improve performance; how representation geometry shapes drifting optimization remains poorly understood.The paper asks whether suitable representations can emerge jointly with the generator rather than relying on fixed external features.Source: §1 · p. 1
  • Representation-induced distances and neighborhoods set KDE weights and drift directions; fixed pixel geometry has limited ability to distinguish real from generated samples.Pretrained encoders induce more discriminative geometry and empirically yield more effective drifting fields [7, 43].Source: §1 · pp. 1–2
  • Persistent representation learning accumulates discriminative updates across batches, allowing geometry to evolve with the generator and supporting learning from pixels or adapting pretrained encoders.The paper also establishes current-step gradient equivalence between KDE ratio loss and drift regression loss under matched conditions, and motivates velocity clipping to stabilize training as representation evolves.Source: §1 · p. 2
  • Approximately 82–95% lower FID than the original pixel-space Drifting Models across multiple datasets, without pretrained encoders; adapting pretrained representations and velocity clipping provide further gains.The analysis also examines how representation learning affects drift alignment and KDE discrimination.Source: §1 · p. 2

2 Related Work

Related work spans drifting models, adversarial methods, and Wasserstein gradient flows. The paper connects these lines by studying how adaptive discriminative representations shape empirical drifting and generator updates.

Drifting Models

  • Prior drifting work connects Gaussian-kernel fields to score differences and diffusion interpretations, extends drifting to pretrained features, and develops Wasserstein-flow methods; adapting discriminative representations during training remains open.The cited work reports that training from scratch without pretrained features remains open [43].Source: §2 · p. 2

Generative Adversarial Networks

  • GAN work links optimal discriminators to density ratios and generator directions to Wasserstein flows, while learned or pretrained feature spaces demonstrate the importance of discriminator representations.Persistent discriminator learning and gradient regularity constraints address how discrepancies are represented and how their gradients act on the generator.Source: §2 · pp. 2–3

Wasserstein Gradient Flows

  • Wasserstein-flow analyses describe distribution-level drifting dynamics but leave finite-batch field construction and regularity of generator updates as separate algorithmic questions.This work studies how discriminative representation shapes empirical drifting and generator updates, adapts representations across batches, and motivates clipping drifting velocity.Source: §2 · p. 3

3 Background

Drifting Models construct a sample-based vector field and train the generator toward targets displaced by that field. The background relates Gaussian-kernel drift to smoothed-density scores and a Wasserstein gradient-flow interpretation.

3 Background

  • Drifting Models construct a vector field from real and generated samples, then regress the generator toward field-displaced targets to absorb iterative distribution refinement into one-step generation.The empirical field uses current-batch samples and is held fixed during the generator update.Source: §3 · p. 3
  • The field's real-sample component attracts points toward nearby real samples, while its generated-sample component repels them from nearby generated samples.The generator regresses toward the displaced stop-gradient target through the drift regression loss.Source: §3 · p. 3
  • For a Gaussian kernel, drifting admits a score-difference interpretation through Parzen-smoothed densities and approximates the Wasserstein gradient flow associated with KL(qθ∥p) [14].The same-distribution KDE estimates exclude the evaluation point for simplicity.Source: §3 · p. 3

4 Methodology

The methodology constructs drifting fields in representation space and learns representation geometry alongside the generator. It connects KDE-ratio supervision to drift regression and bounds the feature-space velocity used for generator updates.

4 Methodology

  • The method rebuilds representation-space KDEs from real and generated feature samples and uses the resulting field for the generator update.The encoder is held fixed during the generator update, with gradients propagated through it to the generator.Source: §4.1 · p. 4, §4 · p. 4

4.1 KDE-Ratio and Drifting Updates in Representation Space

  • With matched bandwidth and feature anchors and no velocity clipping, the current generator gradients of KDE-ratio optimization and drift regression coincide.The encoder and feature anchors are held fixed during this generator update, while gradients pass through the encoder to its input.Source: §4.1 · pp. 4–5

4.2 Learning Discriminative Geometry through Persistent Representations

  • The encoder is retained across batches while KDEs and drifting fields are rebuilt from current batches, allowing earlier updates to shape later geometry without storing earlier samples.The generator still outputs images in pixel space; in from-scratch experiments, the encoder is initialized as the identity mapping.Source: §4 · p. 4, §4.2 · p. 5
  • The encoder is trained with a KDE-based real-versus-generated classification objective to expose current discrepancies and learn a more discriminative geometry for drift construction.The score uses equal class priors and no additional trainable scalar head; after K encoder updates, the encoder is frozen for the generator update.Source: §4.2 · pp. 5–6
  • Persistent representation fields align more closely with the reference field over training than raw-space fields, which are disorganized by nuisance coordinates.Figure 1 compares field structure and alignment in a controlled example.Source: Figure 1 · p. 6
  • A direct velocity constraint bounds the feature-space displacement used in the generator target while preserving the direction of nonzero drift.The radial projection guarantees a drift norm no greater than Vmax; the resulting parameter gradient still depends on encoder and generator Jacobians.Source: §4.2 · p. 6

5 Experiments

Experiments examine how representation geometry affects drift estimation and generation, whether persistent representation learning works without pretrained features, and how persistence and velocity control affect sample quality. Across three image datasets, the method improves on standard drifting and SNGAN.

5.2 Main Benchmark Results

  • On CIFAR-10, generated samples are clearer than standard drifting and have more coherent object structure than SNGAN.The gain is also reported at equal training time.Source: Figure 3 · p. 7, §5.2 · p. 8

5.1 Mechanistic Analysis

  • In a toy setting with nuisance dimensions, persistent representations produce more coherent drift fields and better align estimated drift with the semantic-space reference.The controlled example uses eight Gaussian modes in two semantic dimensions and 30 nuisance dimensions.Source: §5.1 · p. 7
  • On CIFAR-10, pixel-space KDE discrimination stays near chance, while frozen MoCo-v2 and persistent adaptation progressively improve discrimination and sample quality.The pixel-space result is not explained by an untuned kernel bandwidth.Source: §5.1 · p. 7, Figure 2 · p. 7

5.2 Main Benchmark Results

  • Across SVHN, CIFAR-10, and CIFAR-100, the method improves FID and IS over standard drifting and outperforms SNGAN; its FID converges lower than standard drifting.Table 1 reports benchmark comparisons, while Figure 4 tracks FID over training steps.Source: Table 1 · p. 8, Figure 4 · p. 8, §5.2 · p. 8

5.3 Ablation Studies

  • Within 10k CIFAR-10 training steps, persistence only in the representation improves performance, while persistence in the scalar critic markedly worsens it.The ablation compares no persistence, persistent representation only, persistent scalar critic only, and persistence in both components.Source: §5.3 · p. 8, Figure 5 · p. 9
  • On CIFAR-10, velocity clipping improves both IS and FID across evaluated budgets, whereas spectral normalization of the encoder degrades performance.The best reported clipping budget is Vmax = 0.005.Source: §5.3 · p. 8, Table 2 · p. 9
  • The factorial ablation attributes most improvement to persistent representation, with velocity clipping providing additional gains.Table 3 uses Vmax = 1.5 in pixel space for velocity clipping.Source: Table 3 · p. 9, §5.3 · p. 9

6 Conclusion

The conclusion frames fixed pixel geometry as a representation bottleneck and presents persistent representation learning and a KDE–drift connection as responses. Experiments improve sample quality without pretrained encoders, with further gains from pretrained adaptation and velocity clipping.

6 Conclusion

  • Persistent representation learning updates geometry alongside the generator, addressing the bottleneck that fixed pixel space provides weak discrimination for constructing drift.The conclusion also connects the KDE density-ratio potential to the empirical drifting field and relates its local regularity to drifting velocity, motivating velocity clipping.Source: §6 · p. 9

Ethics statement

  • Experiments use previously released datasets and do not involve human subjects or release a new dataset.Source: Ethics statement · p. 10
  • The approach could be used to generate misleading, harmful, or inappropriate content and may inherit or amplify biases and privacy risks from underlying data and pretrained models.The paper says deployment should be accompanied by safeguards and compliance with relevant model and dataset licenses and intended-use conditions.Source: Ethics statement · p. 10

Reproducibility statement

  • Appendix D derives the matched KDE-ratio and drift-regression loss gradient identity, discusses the GAN connection, and Appendix E provides generator-update pseudocode.Appendix A gives detailed experimental configurations for reproducibility.Source: Reproducibility statement · p. 10

A Implementation Details

Appendix A documents configurations for the main benchmark and ablations, plus additional settings for toy, critic, and pretrained-representation experiments.

A Implementation Details

A.1 Additional Configuration Settings

  • The toy experiment uses an eight-Gaussian semantic ring with 30-dimensional Gaussian nuisance vectors, while the persistent scalar critic uses an SNGAN discriminator and supplies drift through its input gradient.The toy generator and representation are updated for 1,000 steps; the critic is updated once per generator step with gradients clipped to norm 1.Source: §A.1 · p. 14
  • The MoCo-v2 setting initializes a ResNet-50 from an ImageNet-pretrained checkpoint; the adapted variant unfreezes only the final residual block.The frozen configuration keeps the backbone fixed, while the adapted configuration uses the persistent schedule.Source: §A.1 · p. 15

A.2 Persistent Representation Architecture

The appendix describes a residual encoder and diagnostics of learned geometry, drift magnitudes, and pixel-space bandwidth choices. The bandwidth sweep shows that tuning temperature alone does not bring pixel-space drifting close to persistent representation learning.

A.2 Persistent Representation Architecture

  • The DRN-style encoder preserves input resolution and uses an identity-initialized residual parameterization, Eφ(x) = x + Rφ(x), with no downsampling or normalization layers.Zero-initializing the final projection makes the initial encoder equal to the identity; subsequent training modifies it through the residual branch.Source: §A.2 · p. 15

B.1 Evolution of the Learned Representation Geometry

  • The learned encoder's CKA against its identity initialization fell to 0.46 at 50k steps, while neighboring late checkpoints remained above 0.95 from 30k onward.The representation gradually changes and then stabilizes among late-training checkpoints.Source: §B.1 · p. 16, Figure 7 · p. 16

B.2 Drift-Magnitude Distributions and Velocity Budgets

  • Pre-clipping drift-magnitude distributions had medians near 0.70 at steps 500, 1000, and 2000; budgets of 0.05 or less fell below their main mass.The distributions were measured in the Vmax = 0.005 run; the figure marks the five budgets evaluated in Table 2.Source: §B.2 · p. 16, Figure 8 · p. 16

B.3 Kernel Bandwidth in Pixel Space

  • FID remained 134–376 across pixel-space KDE temperatures from 0.002 to 2, versus 27.41 for persistent representation learning under the same 10k-step budget.The paper concludes that pixel-space drifting's ineffectiveness reflects representation geometry, not merely poorly tuned bandwidth.Source: Figure 9 · p. 17, §B.3 · p. 17

B.4 Training Time Cost Comparison

Under the same training-time budget, persistent representation learning reaches lower CIFAR-10 FID than standard drifting.

B.4 Training Time Cost Comparison

  • FID 21.57 versus 124.94 for standard drifting under the same training-time budget on CIFAR-10 using four GH200 GPUs.Source: §B.4 · p. 17

C Additional Qualitative Results

The appendix adds qualitative SVHN samples generated by the method.

C Additional Qualitative Results

  • The additional SVHN samples come from the checkpoint attaining the method’s best FID of 2.88.Source: Figure 11 · p. 18

D.1 Proof of Proposition 1

The appendix relates drifting, KDE-ratio, and GAN generator updates, establishing a matched current-step gradient identity and describing how representation geometry changes generator supervision.

D.1 Proof of Proposition 1

  • Under matched Gaussian-KDE conditions, KDE-ratio and detached drift-regression losses have identical current-step generator gradients, not identical losses or training trajectories.Source: §D.1 · p. 19

D.2 Pixel-Space Special Case and Comparison

  • An encoder changes which anchors contribute to KDE scores and pulls feature directions through its Jacobian; neither pixel nor encoded geometry is universally superior.Source: §D.2 · p. 20

D.3 Connection to Density-Ratio GAN Objectives

  • At the function-space optimum, a GAN discriminator encodes the data-to-model density ratio, whose potential gradient directs generated mass toward underrepresented data regions.Source: §D.3 · p. 21
  • The gradient identity specializes to h(a) = τa with Gaussian KDE ratios; it does not establish equivalence for arbitrary GAN generator losses.Source: §D.3 · p. 21

Algorithm 2: Drifting Models Training

Algorithm 3: Divergence-Based GAN and Matched KDE Update

E A Detailed Comparison of Generator Updates

F Limitations

The study identifies limits in learned-from-scratch representation quality and in the scale of evaluated datasets.

F Limitations

  • Representations learned from scratch improve standard drifting but still trail pretrained encoders.Source: F Limitations · p. 22
  • The effectiveness of persistent representation learning on larger-scale datasets remains unestablished because of computational constraints.Source: F Limitations · p. 22

Related papers

Loading 2610.04703v1…