PaperlayerView exact revision on arXiv

Source-linked AI summary

SCALE: Improving Guidance-Free Visual Generation via Self-Contrastive Alignment for Likelihood Estimation

Jiwan Hur, Dong-Jae Lee, Dongyeun Lee, Gyojin Han, Junmo Kim

arXiv:2610.23263v2cs.CV

TL;DR

CFG improves visual sample quality but doubles sampling cost, and single-inference guidance-free methods remain bounded by CFG and inherit its limitations. SCALE learns self-contrastive guidance from real and self-generated samples in a guidance-free training framework. It consistently outperforms CFG-mimicking baselines across diffusion and autoregressive settings, with further gains from DPO; the formulation adds training overhead.

  • Problem

    CFG improves sample quality at doubled sampling cost, while guidance-free methods that approximate it remain bounded by CFG and inherit its limitations.

    Source: §1 · p. 1

  • Method

    SCALE reparameterizes a signal contrasting model predictions on real and self-generated samples into likelihood training, producing SCG-enhanced predictions in one forward pass.

    Source: §1 · pp. 1–2

  • Results

    SCALE consistently surpasses CFG-mimicking guidance-free methods across diffusion and autoregressive models, and DPO on top brings further gains.

    Source: §1 · p. 2, §5.2 · p. 9

  • Takeaways & Limitations

    The results support self-contrastive guidance as a complementary direction to CFG and preference-based fine-tuning for guidance-free visual generation.

    Source: §6 · p. 10

    SCALE’s nested guidance formulation introduces additional training overhead, and broader visual alignment tasks and larger, more recent architectures remain open directions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Classifier-free guidance (CFG) is essential for improving sample quality in visual generation, but it incurs substantial sampling overhead. Existing guidance-free approaches halve this cost by approximating the CFG with a single model inference, but their performance is bounded by the CFG and inherits its limitations. In this paper, we propose self-contrastive guidance (SCG), which contrasts the model's prediction on real and self-generated samples to obtain a direct, model-based signal that pushes generation toward the true data manifold. Integrating SCG into the standard likelihood objective via reparameterization yields Self-Contrastive Alignment for Likelihood Estimation (SCALE), a guidance-free training framework that produces SCG-enhanced predictions in a single forward pass. Across class-conditional and text-to-image generation on diffusion and autoregressive models, SCALE consistently outperforms CFG-mimicking guidance-free baselines and improves further when combined with DPO, demonstrating that the self-contrastive signal yields gains complementary to both CFG and preference-based fine-tuning.

1 Introduction

CFG improves visual sample fidelity but doubles sampling cost, and guidance-free methods that mimic it remain bounded by its limitations. SCALE instead learns self-contrastive guidance from real and self-generated samples, producing SCG-enhanced predictions in one forward pass and outperforming CFG-mimicking baselines.

1 Introduction

  • CFG improves fidelity by contrasting unconditional and conditional predictions, but doubles sampling cost and can reduce diversity by overemphasizing the condition.Existing single-inference guidance-free approaches approximate CFG, so their performance remains bounded by it.Source: §1 · p. 1, Figure 1 · p. 2
  • SCG contrasts model predictions on real and self-generated samples to estimate a model-based signal that pushes generation toward the true data manifold.The signal relies only on model predictions and applies to diffusion and autoregressive models.Source: §1 · pp. 1–2
  • SCALE reparameterizes the SCG signal within the standard likelihood objective so a single model learns to produce SCG-enhanced predictions.For conditional generation, it integrates CFG to refine rather than merely mimic the CFG distribution.Source: §1 · p. 2
  • SCALE consistently surpasses CFG-mimicking guidance-free methods across diffusion and autoregressive models on class-conditional ImageNet and text-to-image generation.The evaluated models include DiT, LlamaGen, and Stable Diffusion v1.5; DPO applied on top of SCALE brings further gains.Source: §1 · p. 2

2 Background

The background describes likelihood-based diffusion and autoregressive models, then reviews guidance-free methods that approximate CFG with one inference. Those methods halve CFG inference cost but retain its target distribution and limitations.

2.1 Likelihood-based Generative Models in Vision

  • Diffusion models learn to reverse a Gaussian noising process, while autoregressive models model token sequences through conditional next-token probabilities.The paper focuses on these two representative likelihood-based model types.Source: §2.1 · p. 2
  • In a three-class 2D Swiss-roll toy experiment, CFG-mimicking methods drift from class boundaries and the data manifold, while SCALE sharpens samples while preserving both.The effective guidance scale is fixed at 5.0 across the guidance-free methods.Source: Figure 2 · p. 3

2.2 Guidance-Free Visual Generation via Mimicking CFG

  • CFG amplifies conditional predictions to improve sample clarity, but guidance-free methods that mimic CFG remain constrained by its distribution.Guidance-free approaches include distillation, Condition Contrastive Alignment, and Guidance-Free Training, which incorporate CFG targets into single-inference models.Source: §2.1 · p. 3, §2.2 · p. 3

3 Revisiting Information-Based Guidance for Guidance-Free Training

CFG-mimicking guidance-free methods reduce inference cost but do not resolve distortions inherited from CFG. The section motivates a complementary, model-based signal derived from real and self-generated samples.

3 Revisiting Information-Based Guidance for Guidance-Free Training

  • CFG entangles prompt alignment and perceptual quality, so increasing its strength can push samples off the data manifold and narrow them toward per-class templates.CFG-mimicking guidance-free methods halve inference cost but inherit this distortion.Source: §3 · p. 3
  • Information-based guidance can amplify salient information, but requires extra sampling-time evaluations and heuristic, domain-specific choices of information and degraded references.Combining it with CFG raises the cost to three model evaluations per step.Source: §3 · p. 4
  • SCG contrasts predictions on real and self-generated samples during training, teaching a single-inference model to approximate a signal that steers generation toward the data manifold.Because it compares model outputs, the approach applies to diffusion and autoregressive models.Source: §3 · p. 4

4 Self-Contrastive Alignment for Likelihood Estimation

SCALE defines self-contrastive guidance by contrasting real and self-generated samples, then incorporates that signal into likelihood training for guidance-free sampling. Its practical formulation uses PatchMix regularization and can integrate CFG; comparisons describe stronger data alignment than CFG-mimicking methods.

4.1 Self-Contrastive Guidance

  • SCG contrasts model outputs on real and self-generated samples, using their residual discrepancy to steer generation toward high-density data regions.The signal applies to continuous and discrete modalities.Source: §4.1 · p. 4

4.2 Guidance-Free Training with Self-Contrastive Guidance

  • SCALE reparameterizes the SCG target within standard likelihood training, conditioning the model on α so lower α amplifies the learned guidance at sampling.At the ideal regression optimum, the diffusion model produces the conditional mean SCG prediction in one forward pass, with guidance strength controlled by α.Source: §4.2 · pp. 4–5

4.3 Practical Implementation

  • PatchMix interpolates real and self-generated samples by replacing a fraction of real patches, providing a stable guidance anchor while preserving local semantic structure.The approach applies to continuous and discrete representations, unlike linear interpolation for vector-quantized tokens.Source: §4.3 · p. 5

4.4 Comparison with Other Methods

  • SCALE's nested guidance formulation adds training overhead, while its reported computational comparison retains low training overhead and halves inference cost relative to CFG.The comparison also describes improved target-distribution alignment.Source: Table 1 · p. 6

4.3 Practical Implementation

  • For conditional generation, SCALE applies regularized SCG to CFG predictions, combining both signals in one forward pass; α = 1 recovers GFT and β = 1 recovers basic SCALE.The self-generated reference is sampled from the unconditional prediction during training, adding no extra cost and avoiding CFG or SCG amplification.Source: §4.3 · p. 6

4.4 Comparison with Other Methods

  • SCALE amplifies a model-based density ratio rather than training only to reproduce the CFG distribution, providing stronger alignment with the data distribution and superior sample quality.The compared CFG-mimicking methods are bounded by CFG itself.Source: §4.4 · p. 7

5 Results

Across class-conditional and text-to-image generation, SCALE improves guidance-free results over existing baselines. The analysis examines comparisons with guided sampling and DPO, inference-time controls, training from scratch, and interpolation choices.

5.2 Enhancing Guidance-free Generation with SCALE

  • On ImageNet class-conditional generation, SCALE outperforms all baselines on FID and IS across DiT-XL/2 and LlamaGen-L; on SD1.5 text-to-image, it achieves superior performance with competitive FID.The comparisons cover ImageNet 256 × 256 and SD1.5 text-to-image generation.Source: §5.2 · pp. 7–8, Table 2 · p. 8, Table 3
  • For SD1.5, SCALE wins over GFT on HPSv2.1 and PickScore, and its CLIPScore–FID trade-off is superior across effective CFG scales from 4.0 to 12.5 at α = 0.8.The text-to-image comparison sets the effective CFG scale for guidance-free methods to 7.5.Source: §5.2 · p. 8, Figure 4 · p. 8
  • SCALE alone matches GFT+DPO, while adding DPO to SCALE brings further gains across all metrics; qualitative samples also show finer details and higher visual quality than base SD1.5 and GFT.DPO barely improves over base SD1.5 in the guidance-free setting, while applying DPO to SCALE further enhances sample qualities.Source: §5.2 · pp. 8–9, Figure 5 · p. 9
  • Without sampling-time guidance, SCALE matches guided sampling methods and surpasses CFG in FID, despite those methods introducing additional sampling overhead.Comparators include CFG, SAG, and discriminator guidance (G++).Source: Table 4 · p. 9, §5.2 · p. 9

5.3 Analysis on SCALE

  • Decreasing β improves fidelity by strengthening the conditional signal but can shift semantic structure toward canonical templates, whereas decreasing α targets fidelity while preserving semantic structure.Lowering α may fall short when samples lie far from the data manifold.Source: §5.3 · pp. 9–10, Figure 6 · p. 9
  • Patch-wise mix-up improves robustly over linear interpolation by preserving local semantics in dense visual data.The interpolation is used to maintain compatibility with diverse generative architectures.Source: Table 6 · p. 10, §5.3 · p. 10
  • Lower interpolation ratios achieve better FID and IS, and a small but non-zero ratio suffices for stable, effective guidance.Table 7 reports each ratio’s best FID with its corresponding IS.Source: Table 7 · p. 10, §5.3 · p. 10
  • SCALE can be trained from scratch without a pretrained teacher, and experiments on DiT-B/2 show improved guidance-free generation; scaling this setup to larger models remains future work.The from-scratch result is verified on DiT-B/2 because of resource constraints.Source: §5.3 · p. 10, Table 5 · p. 10

6 Conclusion

The conclusion presents SCALE as a guidance-free framework based on self-contrastive guidance and reports improvements over guidance-free baselines across diffusion and autoregressive models. It also identifies additional training overhead and several open extensions.

6 Conclusion

  • SCALE uses self-contrastive guidance derived from the model’s predictions on real and self-generated samples to push generation toward the true data manifold.The conclusion grounds SCG in the information-based guidance theorem.Source: §6 · p. 10
  • Experiments show SCALE outperforming existing guidance-free baselines on diffusion and autoregressive models, with complementary gains when combined with human preference-based fine-tuning.Source: §6 · p. 10
  • SCALE’s nested guidance formulation adds training overhead; broader visual alignment tasks, post-training beyond DPO, and larger, more recent architectures remain open directions.Source: §6 · p. 10

A Derivation of Generalized Information-Based Guidance for Discrete Generative Models

The derivation expresses generalized information-based guidance as joint-likelihood maximization plus amplification of the information in a degraded sample. It explains the objective through the conditional density and relates its joint form to an alternative conditional formulation.

A Derivation of Generalized Information-Based Guidance for Discrete Generative Models

  • Generalized information-based guidance recovers a true sample from a degraded one by maximizing joint likelihood and amplifying the correlation between the degraded sample and missing salient information.The objective adds s times the log conditional probability of the salient information given the degraded sample to the joint log-density.Source: A Derivation of Generalized… · p. 15
  • Using the probability chain rule, the objective amplifies log p(h_t | x̄_t), the pointwise mutual information, by a factor of s on top of the base log-density.The derivation obtains this form by substituting the expanded conditional log-density into the objective.Source: A Derivation of Generalized… · p. 15
  • The joint and conditional formulations have identical optima when h_t is fixed; the joint form avoids defining a separate distribution over h_t.Their objectives differ only by an additive constant inside the arg max.Source: A Derivation of Generalized… · p. 15

B Proof of Theorem 4.1 and Analogous Result for AR Models

The diffusion proof derives the SCG prediction from the score-matching optimum and the model-generated reference. The analogous autoregressive proof shows that SCALE recovers the SCG next-token distribution at convergence.

B Proof of Theorem 4.1 and Analogous Result for AR Models

  • The diffusion reference is formed by estimating clean data with Tweedie’s formula and re-noising it to the same noise level.The reference is generated from the model’s current prediction.Source: B Proof of Theorem 4.1 and Analogous… · p. 15
  • At the diffusion regression optimum, the SCG branch predicts the conditional mean of the adjusted noise target given the noisy input.The reference branch is held fixed by stop-gradient.Source: B Proof of Theorem 4.1 and Analogous… · p. 16
  • For autoregressive models, the optimal cross-entropy predictor equals the true next-token conditional, with a vocabulary-wide additive logit constant absorbed by softmax.The next-token predictor outputs a categorical distribution at each generation step.Source: B Proof of Theorem 4.1 and Analogous… · p. 16
  • At convergence, the SCALE-trained autoregressive model recovers the SCG-induced next-token distribution in the probability domain.The result mirrors the score-domain result for diffusion.Source: B Proof of Theorem 4.1 and Analogous… · p. 16

C Analysis of the SCALE Training

  • The training analysis considers how the self-generated reference makes SCALE’s target depend on current predictions under stop-gradient updates and PatchMix.Its diffusion analysis fixes timestep and conditioning strength and averages over noise and an independent PatchMix mask.Source: C Analysis of the SCALE Training · p. 16

C.1 Training with Stop-Gradient and PatchMix

Stop-gradient updates use a frozen reference within each step, while the reference moves across model snapshots. Under the stated assumptions, PatchMix reduces displacement-dependent error in the finite-capacity bound.

C.1 Training with Stop-Gradient and PatchMix

  • Within each update, stop-gradient freezes the reference and makes the dependency chain acyclic, though shared parameters let guided-slice updates change the base slice across iterations.At α = 1, the objective reduces to standard noise regression toward a fixed, model-independent target.Source: §C.1 · p. 17
  • Across updates, changes in the base slice induce a moving target for α<1 slices, and the target variation vanishes as the base-slice change tends to zero.The α<1 slices converge to the limit if their regression error relative to the moving target also vanishes.Source: §C.1 · p. 17

C.2 Optimum with PatchMix under Finite Capacity Bounds

  • At the ideal optimum, PatchMix with λ=1 recovers the original SCG result, while λ=0 disables guidance.The finite-capacity analysis assumes bounded prediction errors and an L-Lipschitz base predictor between the reference inputs.Source: §C.2 · pp. 17–18
  • Under the stated assumptions, PatchMix suppresses the displacement-dependent contribution to the finite-capacity bound on deviation from the PatchMix-SCG optimum.Mixing degraded self-generated samples with real inputs brings the reference query closer to the real input in RMS.Source: §C.2 · p. 18

D SCALE Training Pipeline for Autoregressive Models

The autoregressive SCALE pipeline adapts the diffusion training procedure to next-token prediction and applies PatchMix at token level.

D SCALE Training Pipeline for Autoregressive Models

E Further Implementation Details

The appendix details SCALE’s fine-tuning and training-from-scratch setups, its blind human-preference study, and background on visual model families and contrastive guidance.

E.1 Fine-tuning

  • Fine-tuning uses less than 5% of the base model’s pre-training budget, and the text-to-image setup evaluates SD 1.5 on COCO prompts and Pick-a-Pic preference prompts.The SD 1.5 model is fine-tuned with LoRA on 18M LAION-Aesthetics 5+ text-image pairs; sampling uses 50-step DPM-Solver++.Source: E Further Implementation Details · p. 19, §E.1 · p. 19

E.2 Training From Scratch

  • For training from scratch, the paper uses DiT-B/2 because larger-scale pre-training was infeasible in its academic research setting.The setup starts with a CFG-like objective to stabilize training before incorporating the self-contrastive signal as generation quality matures.Source: §E.2 · p. 20

E.3 Human Preference Study

  • The blind A/B study compares SCALE with GFT using 300 votes from 20 participants, who choose an image or tie with model identities hidden and left/right order randomized.Trials use prompts from the Pick-a-Pic test set.Source: §E.3 · p. 20

F.1 Visual Generative Models

  • Diffusion, autoregressive, masked, and flow-matching models differ architecturally, yet the paper describes all as relying on additional guidance to address unconditional-fidelity gaps.Source: §F.1 · p. 20

F.2 Contrastive Signals for High-Fidelity Visual Generation

  • Information-based guidance amplifies salient information relative to a degraded reference, while prior methods can require extra forward passes and domain-specific heuristics.The paper frames CFG as a special case in which the degraded reference drops the condition.Source: §F.2 · p. 21
  • SCALE adds a real-versus-self-generated contrast alongside CFG, aiming to improve generation beyond reproducing CFG behavior.DPO instead uses paired preference data, and typically still relies on CFG at inference in visual generation.Source: §F.2 · p. 22

G.1 Computational Overhead Analysis

SCALE achieves the highest IS and competitive FID without sampling-time guidance in the reported 250-step DDPM comparison. Its training overhead is limited, and patch-wise interpolation supports a stable self-contrastive anchor.

G.1 Computational Overhead Analysis

  • SCALE’s ×1.12 training-step overhead matches guidance distillation, compared with ×1.06 for GFT and ×1.69 for CCA.SCALE computes stop-gradient unconditional and SCG reference outputs without backward passes.Source: §G.1 · p. 22
  • SCALE keeps VRAM at ×1.00 and inference cost at ×0.50, matching GFT’s memory use and halving CFG’s inference cost.Its added conditioning embeddings account for roughly 0.0012% of base-model parameters, leaving the parameter ratio at ×1.00.Source: §G.1 · p. 22

G.2 Visualization of PatchMix Interpolation

  • PatchMix transitions smoothly between VQ-token images, while linear interpolation collapses to noise; smaller λ keeps PatchMix samples closer to the real-data manifold.The latter relationship is reflected in monotonically decreasing LPIPS distance as λ shrinks.Source: Figure 8 · p. 23, §G.2 · p. 23

G.3 Comparison with Guided Sampling Methods

  • Without sampling-time guidance, SCALE achieves the highest IS and competitive FID in the 250-step DDPM comparison, surpassing CFG and matching guided methods on these measures.SAG and G++ incur additional inference cost.Source: §G.3 · p. 23, Table 8 · p. 23
  • SAG and G++ add an inference pass and have generalization constraints: SAG uses a tiny guidance scale, while G++’s gains are sensitive to solver configuration.Both methods are designed for score-based diffusion and do not readily apply to discrete AR models.Source: §G.3 · p. 23
  • SCALE applies across diffusion solvers and AR models without model-specific tuning, unlike the guided methods in this comparison.The comparison describes SCALE as guidance-free at sampling time.Source: §G.3 · p. 23

H Additional Visualizations and Detailed Analysis

Varying implicit CFG and SCG signals produces different quality-diversity tradeoffs, while SCG can refine CFG predictions and DPO-post-trained outputs. The figures show these changes without extra sampling-time computation.

Additional Visualizations and Detailed Analysis

  • Across the visualizations, varying α and β at inference time does not incur additional computational cost.The samples are generated with a single model without sampling-time guidance.Source: H Additional Visualizations and… · p. 24

Effect of varying β (implicit CFG)

  • Decreasing β strengthens the conditional signal and can improve quality, but may shift samples toward canonical templates and reduce diversity.The semantic structure can change steeply as CFG becomes stronger.Source: H Additional Visualizations and… · p. 24

Effect of varying α (implicit SCG)

  • Decreasing α amplifies SCG, improving quality while preserving semantic structure and producing more diverse high-quality samples.SCG may not produce high-quality samples when the unguided sample is far from the data manifold.Source: H Additional Visualizations and… · p. 24

Combination of SCG and CFG

  • Combining SCG with CFG lets moderate CFG enhance prompt alignment while SCG improves quality on top of the CFG prediction.The paper describes the combination as compensating for limitations of each guidance signal individually.Source: H Additional Visualizations and… · p. 24

Visualization with DPO post-training

  • SCALE further enhances sample quality over GFT after DPO fine-tuning, and decreasing α gradually improves the DPO-fine-tuned SCALE model’s quality.These samples use a single model without guidance at sampling time; varying α and β adds no computational cost.Source: H Additional Visualizations and… · p. 24

I Limitations and Future Work

This section discusses unresolved questions and limitations around SCALE, including efficiency, alignment, post-training, and scaling, as well as misuse risks and safeguards for responsible deployment. It also summarizes checklist practices for model release, asset handling, voluntary human-study protections, LLM use, and limitations in code release and error-bar reporting.

I Limitations and Future Work

  • SCALE’s hierarchical CFG-plus-SCG formulation adds 6%p training overhead per step over CFG, motivating a more efficient unified, single-step guidance formulation.The current two-step approach combines the benefits of CFG and SCG and provides an interpretable guidance signal.Source: I Limitations and Future Work · p. 24
  • Future work could test self-contrastive signals for safe visual alignment and validate SCALE on larger rectified-flow models such as Stable Diffusion 3 [14] and FLUX.The proposed safety-alignment use would pull models toward safe data and away from unsafe, NSFW distributions without sampling-time guidance or loss of sample quality; current experiments cover DiT, SD 1.5, and LlamaGen at academic scale.Source: I Limitations and Future Work · pp. 24, 27
  • SCALE’s complementarity with DiffusionDPO [60] has only been explored preliminarily, leaving systematic study of RL-based post-training methods—including DSPO, D3PO [65], SPO [35], FlowGRPO [38], and DanceGRPO [64]—as future work.The authors report empirical complementarity from post-training a SCALE-trained model with DPO.Source: I Limitations and Future Work · pp. 26–27

J Ethics Statement

  • SCALE may enable more convincing deceptive or misleading media that harms privacy, reputation, and public trust, and it inherits biases, memorization, and harmful patterns from its pretrained models and data.The authors say SCALE is not a safety mechanism and recommend robust prompt- and output-level moderation, provenance signals where appropriate, and human oversight for sensitive applications.Source: J Ethics Statement · p. 27

K Declaration of LLM Usage

  • LLMs were used only for editorial assistance and figure/table visualization support, not for ideation, research direction, data analysis, or core algorithmic design.The authors state that they authored and verified all technical statements, equations, experimental results, and scientific contributions.Source: K Declaration of LLM Usage · p. 28

NeurIPS Paper Checklist

  • The checklist reports that experiment code is not included, although the dataset and baseline code are described in the publicly available paper.The authors answer [No] to open access to data and code.Source: NeurIPS Paper Checklist · p. 31
  • The checklist reports no error bars, while noting that evaluation uses 50,000 class-conditional and 30,000 text-to-image samples under a protocol described as statistically stable across runs.The authors answer [No] to reporting error bars or other statistical-significance information.Source: NeurIPS Paper Checklist · p. 32
  • Stable Diffusion 1.5’s built-in safety checker filters unsafe outputs, and the authors release no large pretrained model, only fine-tuning code and weights based on existing checkpoints and public datasets.They say the released materials inherit the underlying models’ licenses and safety settings, and characterize DiT-B/2 as low risk because it is small and trained on public, non-sensitive ImageNet data.Source: NeurIPS Paper Checklist · p. 34
  • The authors credit the open-source baseline implementation and report releasing no new assets or models pretrained on sensitive, private datasets.They state that no new assets are released in the paper.Source: NeurIPS Paper Checklist · pp. 34–35
  • The blind A/B study used voluntary, uncompensated adult participants who were informed that the images were AI-generated; participants were anonymized and potentially unsafe or sensitive prompts were filtered out.The authors say a detailed study description is provided in the appendix.Source: NeurIPS Paper Checklist · p. 35

16. Declaration of LLM usage

  • The authors declare LLM use for writing, editing, and formatting in the appendix.Source: §16 · p. 36
Loading 2610.23263v2…