PaperlayerView exact revision on arXiv

Source-linked AI summary

Discriminative Score Function: Turning Pretrained Models into Functional Generative Priors

Junhoo Lee, Hyeonjin Kim, Sangbum Han, Nojun Kwak

arXiv:2610.19743v1cs.CVcs.LG

TL;DR

Discriminative models may capture data-manifold structure beyond their task labels, but prior methods often depend on task-specific handles or auxiliary generators. DSF extracts an image-space field from a pretrained model’s loss-gradient geometry, and the paper reports generation across model types alongside effective zero-shot quantization calibration. These results support treating discriminative models as sources of reusable generative structure.

  • Problem

    The paper asks whether discriminative models’ implicit data-manifold structure can be exposed for generation without task-specific guidance, unlike prior approaches that rely on external or task-specific handles.

    Source: §1 · pp. 1–2

  • Method

    DSF uses a pretrained discriminative model’s loss-gradient geometry and a functional-space alignment to produce an image-space update direction.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    DSF generates model-consistent images across ResNet, DETR, DINOv2, and CLIP, supports several generation-related tasks, and outperforms existing data-free alternatives in zero-shot quantization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Discriminative models can expose reusable generative structure tied to learned representations, and DSF-generated samples improve zero-shot quantization calibration.

    Source: §7 · p. 10

    DSF is not presented as a replacement for high-fidelity diffusion generators; its gradient proxy can depend on architecture and training recipe, and parameter-gradient updates make sampling costlier than first-order image optimization.

    Source: §7 · p. 10

Abstract

from arXiv · show

Discriminative models are trained with task-specific objectives such as classification, detection, or representation learning, yet their practical utility depends on generalizing beyond the training task to unseen samples from the same domain. This suggests that a discriminative model may retain an implicit trace of the data manifold beyond its task labels. We ask whether this trace can be exposed as a generative signal without task-specific guidance. We introduce the Discriminative Score Function (DSF), which extracts an image-space generative field from the loss-gradient geometry of a pretrained discriminative model. DSF provides a training-free update rule that moves images toward model-consistent visual regions without retraining or task-specific targets. We apply DSF to diverse pretrained discriminative models, including ResNet classifiers, DETR detectors, and DINOv2 ViT encoders. DSF generates unconditional images from noise and supports conditional guidance, editing, inpainting, and explanation through the same field. DSF-generated samples also provide effective zero-shot quantization calibration when real images or labels are unavailable, preserving feature geometry better than data-free calibration baselines. These results show that discriminative training leaves a generative functional signal for synthesis and calibration. Project page: https://junhoo.me/dsf.

1 Introduction

The introduction asks whether discriminative models retain data-manifold structure that can serve as a generative objective without task-specific guidance. It presents DSF as a label-free field derived from loss-gradient geometry, usable across model types and generation-related tasks.

1 Introduction

  • Discriminative models must generalize beyond their training tasks, suggesting they capture visual data-manifold structure that could support unconditional generation.The introduction asks whether this implicit structure can be exposed as a standalone generative objective.Source: §1 · p. 1
  • Prior approaches rely on task-specific handles or auxiliary models, limiting evidence that discriminative models expose the manifold and generality beyond classification.The limitation is especially relevant to detectors and self-supervised encoders that may lack class probabilities or labels.Source: §1 · pp. 1–2
  • DSF converts a pretrained model’s loss-gradient geometry into an image-space update direction based on candidate alignment in functional space.The introduction describes this as moving images toward visual structure consistent with the pretrained model.Source: §1 · p. 2
  • A label-free augmentation-invariance surrogate in LTK functional space lets DSF apply one formulation to classifiers, detectors, and self-supervised models.The method replaces inaccessible image-space manifold distance with functional distance induced by the Loss Tangent Kernel (LTK), a loss-aware variant of NTK [24].Source: §1 · p. 2
  • Across ResNet, DETR, DINOv2, and CLIP, DSF supports generation, conditional guidance, editing, inpainting, and explanation; its generated images also calibrate zero-shot quantization without real data or labels.The introduction reports that this calibration outperforms existing data-free alternatives.Source: §1 · p. 2

2 Related works

Related work contrasts DSF’s image-space generative field with diffusion generation, prior image synthesis from discriminative models, and studies that probe learned structure through other interfaces. The paper positions DSF as a label-free functional score for fixed discriminative models.

2 Related works

  • Diffusion models iteratively reverse a noise process toward the data manifold, with training procedures that learn scores and handle data at each time step.Noisy classification can also provide class-conditional generation along the trajectory.Source: §2 · p. 2
  • Prior image synthesis from discriminative models often needs specified activations, architecture-specific priors, auxiliary generators, or overfitted classifiers, and is largely limited to classification.The paper positions DSF as targeting arbitrary pretrained discriminative vision models in continuous image space.Source: §2 · p. 2
  • Other learned-structure studies analyze features or observable responses, whereas DSF constructs an image-space generative field; Template Infilling starts from a model already trained for generation.The examples include Deep Edge Filter [30], CSF [33], and Do Not Think About Pink Elephant! [22].Source: §2 · p. 3
  • The paper frames its evidence as generation across architectures and objectives plus zero-shot quantization calibration without real data, labels, or auxiliary generators.Its framework extracts a label-free functional score from loss-gradient geometry for fixed classifiers, detectors, and self-supervised encoders.Source: §2 · p. 3

3 Preliminaries

The preliminaries introduce kernel methods as inner products in feature space and describe NTK through model gradients. They distinguish LTK by its inclusion of loss derivatives.

3 Preliminaries

  • A kernel represents a feature-space inner product, and RKHS links closeness in that feature space to closeness of inputs.Fixing one input yields the functional object k(·, x).Source: §3 · p. 3
  • NTK is defined by gradients of the network and reflects training in input space, while LTK adds loss derivatives to account for loss dynamics.The passage identifies LTK as a loss-aware variant of NTK [24].Source: §3 · p. 3

4 Discriminative Score Function

The section recasts generation as movement toward a data manifold, then uses the geometry of a pretrained model’s training map to construct a functional surrogate. LTK makes that surrogate usable as a training-free, label-free image update.

4.1 Connection between score-based model and Geometry

  • A manifold-distance update is difficult to implement because the manifold metric is hard to estimate and computation scales with the dataset size.The section motivates replacing this inaccessible image-space distance with a functional surrogate informed by the discriminative model’s generalization geometry.Source: §4.1 · p. 4

4.2 Map the manifold to function space

  • With the training algorithm and architecture fixed, the training map sends datasets to trained models, so nearby datasets can correspond to nearby functions.The section uses local continuity of this map to treat the trained model as a local functional representative of the data manifold.Source: §4.2 · pp. 4–5, Figure 2 · p. 4

4.3 Direct Use of Discriminative Model through LTK

  • LTK supplies a training-free proxy for the training map by measuring similarity through loss gradients; its aggregate alignment uses the mean dataset loss-gradient direction.In the data-free setting, the pretrained parameter direction approximates that aggregate under ℓ2-regularized stationarity, or acts as a trajectory-level trace of accumulated descent.Source: §4.3 · p. 5
  • Because label-dependent losses can prevent unconditional generation or be unavailable, DSF uses a label-free surrogate with similar training dynamics, based on augmentation invariance.The augmentation-invariance loss supplies the gradient feature used in the alignment template.Source: §4.3 · pp. 5–6
  • The DSF field updates candidate images using loss-gradient geometry toward the pretrained model’s functional manifold, with a layerwise normalized ℓ1 distance used in implementation.The figure shows unconditional samples from DINOv2/LVD-124M, DETR/COCO, and ResNet/ImageNet.Source: §4.3 · p. 6, Figure 3 · p. 6

5 Experiments

Experiments use public, unmodified ResNet, DINOv2, and DETR checkpoints to test unconditional and conditional generation and zero-shot calibration with DINOv2.

5.1 Unconditional Generation

  • From random noise without labels or external guidance, DSF produces model-dependent structure for ResNet, DINO, and DETR, from object-centric patterns to scene-like samples.The models represent classification, self-supervised learning, and object detection, respectively.Source: §5.1 · p. 7

5.2 Conditional Generation

  • DSF combines with external classifier or vision-language guidance across architectures, including ResNet–ResNet, DINOv2–ResNet, and ResNet–CLIP pairings.The experiment uses a conditional loss term during sampling.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Zero-Shot Quantization

  • Across 8-, 6-, and 4-bit quantization, DSF matches or exceeds the real-image oracle in nearly all settings and consistently outperforms data-free calibration sources.The comparison uses a quantized DINOv2 feature extractor; the real-image calibration is an oracle reference.Source: Table 1 · p. 8, §5.3 · p. 8
  • Figure 5 contrasts original images in the top row with edited images in the bottom row.The figure presents examples of editing through DSF.Source: Figure 5 · p. 8
  • Zero-shot calibration is especially challenging for self-supervised encoders such as DINOv2, which lack classifier logits, label-conditioned objectives, and BatchNorm statistics.Post-training quantization uses representative inputs to estimate activation ranges and feature statistics, while zero-shot calibration lacks real calibration images or labels.Source: §5.3 · pp. 7–8
  • For quantization, DSF-generated samples calibrate the same fixed DINOv2 feature extractor being quantized, alongside real-image and data-free sources.The calibration target is a representation model, and evaluation considers downstream usability and preservation of full-precision feature geometry.Source: §5.3 · p. 8

6 Generative Properties

DSF serves as a reusable image-space field for sampling, editing, restoration, and explanation. The section shows that image-level edits can preserve surrounding structure, masked updates can restore missing regions, and sampling heuristics can improve generated images.

6 Generative Properties

  • Because DSF acts directly on images without labels, masks, or a trained generator, the same field supports editing, restoration, explanation, and sampling heuristics.These uses require minimal task-specific changes.Source: §6 · p. 8

6.1 Image Editing

  • Applying conditional generation directly to real images changes semantics without largely deforming the input, with the score function restraining edits from straying off-manifold.The method adds neither noise to the target image nor masking; a chimpanzee background light becomes an iguana head in the example.Source: §6.1 · p. 9

6.2 Image Inpainting

  • Restricting DSF updates to masked regions and adding a boundary-consistency regularizer successfully restores randomly masked image regions.Figure 6 places the inpainted image, reconstruction, and original image from left to right.Source: §6.2 · p. 9, Figure 6 · p. 9

6.3 Global Explanation

  • DSF visualizations can expose co-occurrences and biases encoded by a model, while comparison with [13] distinguishes manifold-guided visualization from activation-driven artifacts.A ResNet trained on ImageNet repeatedly produces analog clocks showing 10:10 and plates with contextual elements such as wine glasses and dishes.Source: §6.3 · p. 9, Figure 7 · p. 9

6.4 Unlocking the potential of DSF through diffusion model

  • Linear step-size decay with proportional noise injection produces fewer artifacts and more coherent DINOv2 unconditional samples than the fixed-step baseline.The field itself is unchanged; the improvement comes from the sampling trajectory.Source: Figure 8 · p. 9, §6.4 · p. 10

7 Conclusion

The conclusion presents DSF as a training-free way to extract a generative field from a pretrained discriminative model’s loss-gradient geometry. Its samples improve zero-shot quantization calibration, while the authors distinguish its role from high-fidelity diffusion generation.

7 Conclusion

  • DSF extracts a training-free generative field from a pretrained discriminative model’s loss-gradient geometry, without labels, data, or architectural modification.The authors report that DSF-generated samples improve zero-shot quantization calibration, indicating that the field captures model-specific activation structure rather than merely producing visual artifacts.Source: §7 · p. 10
  • DSF is not presented as a replacement for high-fidelity diffusion generators; its gradient proxy may depend on architecture and training recipe, and parameter-gradient updates make sampling costlier than first-order image optimization.Broader domains and stronger samplers remain future work.Source: §7 · p. 10

A Impact Statement

DSF reuses fixed pretrained discriminative vision models as training-free generative priors. The impact statement notes both possible deployment benefits and risks from model-induced correlations.

A Impact Statement

  • Reusing fixed pretrained discriminative vision models as training-free generative priors may reduce the need for separate generative-model training in some inspection, calibration, and diagnostic workflows.This could lower computation and data-access requirements; zero-shot calibration is especially relevant when real calibration data cannot be shared.Source: A Impact Statement · p. 15
  • DSF samples are model-induced visualizations, not faithful reconstructions of training data or complete descriptions of the data manifold, and may amplify encoded biases, spurious correlations, or artifacts.The authors caution that synthesized calibration or debugging data may inherit undesirable correlations from the source model.Source: A Impact Statement · p. 15

B Existing Assets and Licenses

The experiments use existing public models, datasets, and baseline methods under their original terms. The listed assets are described as used or referenced in the experiments.

B Existing Assets and Licenses

  • Public models, datasets, and baseline methods are used for research evaluation under their original sources and applicable terms, without redistributing third-party datasets or pretrained checkpoints as new assets.ImageNet calibration images follow ImageNet access terms; COCO usage follows COCO terms and associated image licenses, and dependencies and baselines follow repository licenses.Source: B Existing Assets and Licenses · p. 15
  • Table 2 lists existing assets used or referenced in the experiments.Source: Table 2 · p. 15

C Implementation Details

The implementation forms a label-free image update by aligning loss-gradient directions with a model-side parameter direction. It applies this procedure across model architectures and specifies its optimization, compute, and zero-shot quantization settings.

Operational DSF update

  • DSF updates images using the input gradient of a layerwise normalized alignment between augmentation-invariance loss gradients and −θ.The loss uses two augmented views; the image update may include regularization, conditional guidance, or Langevin noise.Source: C Implementation Details · pp. 15–16

Parameter direction as a data-free aggregate-gradient proxy

  • The parameter direction −θ is an aggregate-gradient proxy, exact up to scale only under an ideal ℓ2-regularized stationary condition.Outside that condition, it is not a pointwise equality; the practical assumption is that learned parameters summarize training-induced aggregate descent.Source: C Implementation Details · pp. 16–17

Layers and normalization

  • Selected late layers provide high-level representation geometry, while layerwise normalized ℓ1 alignment reduces sensitivity to layer-scale variation and outlier coordinates.The selected subset is also chosen for computational tractability.Source: C Implementation Details · p. 17

Architecture instantiation

  • The same DSF update is instantiated across target models; for DETR, it uses encoder representations without Hungarian matching, box regression loss, or detector-head targets.The architecture table describes the representation and parameter subset used for each instantiation.Source: Table 3 · p. 17, C Implementation Details · p. 17

Optimization and compute

  • 1.4–1.7 seconds per image is the reported cost of DSF in A6000 sweeps with batch size 50.The image update differentiates through parameter gradients, requiring mixed second-order differentiation.Source: C Implementation Details · p. 17

Zero-shot quantization protocol

  • For DINOv2 quantization, each calibration source supplies 100 images; DSF uses generated images, and evaluation covers 8-, 6-, and 4-bit quantization.Diagnostics include k-NN top-1, fpQ@5, CLS and Patch cosine similarity, and frozen-head ImageNet top-1; Table 4 reports selected activation-scale multipliers.Source: C Implementation Details · pp. 17–18, Table 4 · p. 18

D Generalization of Score Model

The section frames generation as movement toward a data manifold and develops a geometric interpretation of the score. It then motivates a direct distance-based update for DSF sampling.

Generation as manifold motion

  • A score-based generation step moves x along the gradient of log p(x), iteratively toward the distribution’s data manifold.The section presents this as a process for approximating the desired data distribution.Source: D Generalization of Score Model · p. 18

Geometric interpretation

  • The score is interpreted geometrically as a pull toward high-density regions, with the density view extended from KDE to a continuous manifold integral.The section introduces the score as the gradient of log-density.Source: D Generalization of Score Model · p. 18

DSF sampling

Related papers

Loading 2610.19743v1…