SALD: Self-Referenced Advantage Learning for Diffusion Models

• Standard diffusion objectives average gradients uniformly despite varying difficulty across image-caption pairs, noise levels, and visual details.The introduction notes that this can continue emphasizing learned signals while underweighting difficult examples and visual components.Source: §1 · p. 2
• Prior methods address parts of the problem, but do not combine sample-level, temporal, and spectral difficulty in diffusion training.The cited discussion distinguishes noise weighting, noise selection, curricula, robust optimization, and frequency-aware objectives.Source: §1 · p. 2
• SALD evaluates each image-caption pair at easy and hard noise levels with one model, using the detached easy path as a reference and the hard path for updates.It adapts the hard-path objective from the paths’ relative errors rather than distilling the easy-path prediction.Source: §1 · p. 2
• AGD weights samples by relative error, TAM tracks per-pair difficulty over time, and SAD derives latent-element weights from spectral residual comparisons.Together, the components adapt sample importance, future noise-level gaps, and frequency-derived training emphasis.Source: §1 · p. 2, Figure 2 · p. 3
• SALD reports consistent generation-quality gains, with component-wise ablations assessing AGD, TAM, and SAD.The introduction reports these findings across multiple architectures and datasets.Source: §1 · p. 3

2 PROPOSED APPROACH

• AGD, TAM, and SAD respectively address which examples are difficult, how large their future noise gaps should be, and which frequencies remain difficult, sharing the same model parameters.The framework requires neither an external teacher nor an additional trainable model during training or inference.Source: §2 · p. 3

2.1 FLOW-MATCHING SETUP

• The same model evaluates each pair at easy and hard noise levels, using the no-gradient easy path as a reference and updating parameters through the differentiable hard path.The two levels use independent noise samples; their gap controls how difficult the hard state is, and TAM adapts that gap over training.Source: §2.1 · pp. 3–4

2.2 ADVANTAGE-GUIDED DIFFUSION (AGD)

• AGD measures the hard-path loss increase over the stopped easy reference and converts the relative difficulty into a differentiable, positive mini-batch weight.The normalized weights change examples’ relative importance without changing the average batch weight.Source: §2.2 · p. 4

2.3 TEMPORAL ADVANTAGE MEMORY (TAM)

• TAM stores each sample’s running difficulty and uses it to adapt the next easy-to-hard noise gap, increasing gaps for persistent positive difficulty and decreasing them for negative difficulty.The memory is non-trainable; updated values are used the next time the pair is visited, and clipping keeps gaps within the valid range.Source: §2.3 · p. 4

2.4 SPECTRAL ADVANTAGE DECOMPOSITION (SAD)

• SAD compares easy- and hard-path residual power spectra, maps frequency-wise advantages into normalized differentiable latent weights, but discards phase and cannot localize errors.The weight indicates which frequency patterns remain difficult, not where the corresponding errors occur in the latent representation.Source: §2.4 · pp. 4–5

2.5 UNIFIED SALD OBJECTIVE

• Unified SALD weights hard-path residual elements with SAD and then weights mini-batch examples with AGD; it stops only easy references and does not directly match hard predictions to easy predictions.SALD uses two forward evaluations, with only the hard evaluation storing a backward graph, and leaves the inference procedure unchanged.Source: §2.5 · p. 5

2.6 THEORETICAL INTERPRETATION

• Under the stated lower-boundedness and smoothness assumptions, Theorem 1 guarantees convergence toward a stationary point for the differentiable SALD objective.The spectral gate is larger where hard residual energy exceeds the easy reference; the cited supplementary Table 7 is reported to support the theoretical claim and show faster, better convergence.Source: §2.6 · pp. 5–6

3.1 RESULTS AND DISCUSSION

• SALD consistently outperforms acceleration and distillation baselines across SANA, PixArt, and Flux.2, achieving the best FID, CLIP, and LPIPS scores in most evaluated settings.Reported improvements span model scales and datasets, with particularly substantial FID reductions at both 512 and 1024 resolutions.Source: Table 2 · p. 6, §3.1 · pp. 7–8
• Across SANA 0.6B and PixArt-Σ, SALD preserves finer details, clearer structures, and prompt-specific semantics; comparisons also show improved coherence, textures, and semantic fidelity across other datasets.Figure 3 compares SALD with Base, SpeeD, SRA, and Temporal Diff; Figure 5 compares Base and SALD, including high-resolution examples.Source: Figure 3 · p. 7, Figure 5 · p. 7, §3.1 · p. 8
• On COCO and Oxford 102 Flowers, SALD converges faster and maintains lower FID than the other baselines across training steps.Figure 4 reports FID convergence across training steps.Source: Figure 4 · p. 7, §3.1 · p. 8

3.2 ABLATION STUDY

• AGD provides the strongest individual-component improvement, SAD adds complementary structural-fidelity gains with modest CLIP degradation, and TAM alone is weakest; all three together close the full performance gap.AGD+SAD is the strongest pair, while TAM+SAD is weakest; the full SALD combination surpasses all pairs.Source: §3.2 · p. 8
• AGD temperature τ = 0.5 best balances gradient focus and perceptual stability; both lower and higher temperatures degrade gains.Low τ over-concentrates on extreme samples and slightly harms LPIPS, while high τ weakens the difficulty signal toward a near-uniform loss.Source: §3.2 · p. 8
• TAM’s default EMA decay α = 0.90 balances curriculum responsiveness and stability; aggressive decay makes difficulty estimates noisy, while near-frozen memory misses training changes.The wider performance spread on MAGICK indicates longitudinal difficulty tracking is more critical at higher resolution.Source: §3.2 · p. 9
• The default maximum noise gap ∆max = 250 performs best on both benchmarks; narrower and wider gaps weaken or destabilize the advantage signal.A narrow gap weakens hard/easy contrast, while an overly wide gap pushes the easy path near noiselessness and overestimates difficulty.Source: §3.2 · p. 9
• Core design ablations show that relative difficulty, warm-up, detached weighting, and same-sample noise contrast matter; SALD at 12 steps still beats the 20-step uniform baseline.Using absolute loss collapses gains toward baseline, especially at 1024; disabling detach lets the model exploit the weights, and cross-batch advantage underperforms same-sample contrast.Source: §3.2 · p. 9

4 RELATED WORK

• Noise schedules and SNR weighting assign global timestep-dependent weights, whereas AGD estimates per-sample relative difficulty from the loss gap between low- and high-noise paths.The passage contrasts this with noise-instance analysis, which finds unequal signals but does not provide AGD’s per-sample relative weighting.Source: §4 · p. 9
• Unlike curricula and hard-example methods, TAM stores an EMA of relative advantage to adapt each sample’s timestep gap; SAD uses contrastive spectral residuals for spatial loss weights without changing architecture, sampler, or noising.SAD is linked to spectral-bias and frequency-aware diffusion work.Source: §4 · p. 9
• Language-model self-distillation uses demonstration- or feedback-conditioned teachers, while SALD is described as easy-to-hard distillation without an extra model-parameter copy.The passage also notes that the language-domain approaches require demonstration examples and an EMA-based teacher.Source: §4 · p. 9

5 CONCLUSION

• SALD reallocates gradient budget across samples, noise levels, and frequency components in flow-based diffusion training.It contrasts low- and high-noise losses, retains per-sample difficulty over time, and weights latent elements using contrastive spectral residuals.Source: §5 · pp. 9–10
• AGD estimates relative sample difficulty, TAM adapts future timestep gaps, and SAD weights frequency patterns that remain under-learned.The easy-path reference is detached, preserving the original prediction target while adjusting where and how training effort is applied.Source: §5 · p. 10
• Experiments across diverse datasets and architectures demonstrate SALD's effectiveness, while theoretical analysis supports the empirical findings.The framework is described as architecture-agnostic and requires no model, sampler, or inference-procedure modification.Source: §5 · p. 10

A WHY IS UNIFORM TRAINING SUBOPTIMAL?

• Standard flow-matching loss gives samples and latent elements the same explicit coefficient and does not adapt sample importance to current difficulty or distinguish residual frequencies.The section notes that unequal residual magnitudes can still produce unequal gradients despite uniform explicit coefficients.Source: A WHY IS UNIFORM TRAINING SUBOPTIMAL? · p. 14
• Uniform weighting may spend optimization effort on already learned samples, while mean-squared error collapses residuals without identifying difficult frequency bands.Power spectra discard Fourier phase, so frequency-derived weights should not be interpreted as localized pixel-level error maps.Source: A WHY IS UNIFORM TRAINING SUBOPTIMAL? · p. 14
• Because sample difficulty changes during training, SALD uses two noise states of the same pair: a detached lower-noise reference and a higher-noise path that supplies gradients.AGD reweights samples, TAM remembers relative difficulty and adapts future noise gaps, and SAD constructs frequency-derived latent weights without an external teacher or additional model copy.Source: A WHY IS UNIFORM TRAINING SUBOPTIMAL? · p. 14

B EXTENSION TO DDPM

• SALD can be extended to DDPM-style noise prediction, but the paper does not experimentally validate that extension.The extension replaces the flow-matching velocity residual with a DDPM noise-prediction residual; the two paths retain their easy-reference and hard-gradient roles.Source: B EXTENSION TO DDPM · pp. 14–15
• For DDPM, TAM retains its per-pair memory update, while SAD compares noise-prediction residual spectra; its latent weights are not localized error maps.SALD permits any nonnegative loss-weighting rule w(t) when the same rule is used at the easy and hard noise levels, and applies w(th) to the final hard-path loss.Source: B EXTENSION TO DDPM · p. 15

C PROOF AND SCOPE OF THEOREM 1

• Theorem 2 gives a conditional stationarity bound for the differentiable SALD objective under assumptions including smoothness, unbiased gradients, and bounded variance.The theorem's stated scope does not establish these assumptions for a particular neural network or show faster convergence than uniform flow matching.Source: C PROOF AND SCOPE OF THEOREM 1 · pp. 15, 17
• With constant step size and nonzero stochastic variance, the bound approaches a nonzero error floor, guaranteeing a stationarity neighborhood rather than exact stationarity.A vanishing average gradient norm follows when variance is zero or the step size is decreased appropriately; η = Θ(J^-1/2) yields an O(J^-1/2) rate under the stated step-size constraint.Source: C PROOF AND SCOPE OF THEOREM 1 · p. 16
• The convergence argument fixes TAM memory; extending it to SALD's changing memory requires controlling objective drift or analyzing the joint model-and-memory state.The theorem also does not imply that SALD and baseline objectives have the same stationary points.Source: C PROOF AND SCOPE OF THEOREM 1 · p. 17

D ADDITIONAL RESULTS

• The additional-results section describes supplementary qualitative and quantitative analyses across datasets, architectures, and training settings, including component contributions to quality, stability, and detail preservation.The training step uses easy and hard paths, SAD spectral weighting, AGD sample weighting, and TAM's non-trainable memory update before averaging weighted hard-path losses.Source: D ADDITIONAL RESULTS · p. 17

D.1 TRAINING AND EVALUATION CONFIGURATION

• Experiments cover SANA, PixArt-Σ, and FLUX.2 variants at 512 × 512 and 1024 × 1024 resolutions, with settings kept consistent between baseline and SALD comparisons.The configuration table reports parameter counts, per-forward-pass FLOPs, training resolution, learning rates, guidance scales, and sampling steps.Source: §D.1 · p. 17

D.2 WALL-CLOCK EVALUATION

• SALD incurs approximately 25% additional wall-clock time per training step, measured on the same NVIDIA RTX A6000 setup.The overhead comes from an additional easy-noise forward pass; gradients and backpropagation are computed only for the hard path.Source: §D.2 · p. 18
• To approximately match total training time, SALD runs for 10k steps while baselines run for 12.5k steps.The comparison uses the stated relationship 10k × 1.25 ≈ 12.5k.Source: §D.2 · p. 18
• Under comparable wall-clock budgets, SALD improves FID, CLIP, and LPIPS over corresponding baselines across evaluated backbone configurations on CUB-200 and MultiGen.The authors report better generation quality across both datasets despite training SALD for fewer steps.Source: §D.2 · p. 19

D.3 IMPACT OF INFERENCE STEPS AND MODEL SCALABILITY

• Across SANA 0.6B/1.6B and PixArt-Σ 0.6B, SALD preserves quality more effectively as inference steps fall, with lower FID and LPIPS and comparable or higher CLIP scores.The cross-scale results suggest stronger quality–efficiency trade-offs across the evaluated models.Source: §D.3 · pp. 19–20
• At 15 steps, SANA 1.6B achieves FID 30.60 vs. 33.11 for Base at 20 steps, and SANA 0.6B achieves 25.54 vs. 30.64.Each SALD result uses 25% fewer sampling steps; LPIPS and CLIP remain slightly below the corresponding 20-step Base values.Source: §D.3 · p. 19, Table 6 · p. 21
• At low sampling budgets, SALD’s 10-step output already resembles its 15- and 20-step generations, while the baseline remains unstable at 5 and 10 steps.Figure 8 provides the qualitative comparison for SANA 1.6B trained on DALL·E 3 1M.Source: §D.3 · pp. 20–21, Figure 8 · p. 20

D.4 WHAT HAPPENS IF WE TRAIN 10× MORE?

• The comparison extends both the Baseline and SALD to 100k steps, or 10× the standard 10k-step training budget, across two SANA scales and two datasets.The table reports results for SANA 1.6B and 0.6B on MultiGen and Oxford 102 Flowers.Source: §D.4 · p. 21, Table 7 · p. 22
• At 10k steps, SANA 1.6B SALD reduces FID from 26.44 to 23.86 on MultiGen and from 33.11 to 27.37 on Oxford 102 Flowers.The FID advantage persists through the 100k-step comparison; CLIP and LPIPS remain comparable.Source: §D.4 · pp. 21–22, Table 7 · p. 22
• After approximately 80k steps, both methods improve more slowly, but SALD’s FID advantage remains at 100k steps.For SANA 1.6B, the gap stays 2.58 on MultiGen and narrows from 5.74 to 3.71 on Oxford 102 Flowers; for SANA 0.6B on Oxford, it narrows from 4.81 to 3.57.Source: §D.4 · p. 22

D.5 ADDITIONAL VISUAL SAMPLES

• Across MAGICK and DALL·E 3 1M at 1024 × 1024, SALD produces sharper, more structurally coherent generations with stronger fine-grained semantic detail than baseline finetuning.Reported improvements include cleaner symmetry and facial composition on MAGICK, and more stable anatomy, hair detail, lighting, and object boundaries on DALL·E 3 1M.Source: §D.5 · pp. 22–23
• PixArt-Σ and Flux.2 klein samples on DALL·E 3 1M show semantic consistency, realistic illumination, and fine textures across diverse subjects and scenes.The reported examples include detailed portraits, reflective objects, indoor scenes, and animals.Source: §D.5 · p. 23
• SALD samples span DALL·E 3 1M, CUB-200, and Oxford-102 Flowers, with reported compositional stability, high-frequency detail, and prompt alignment across settings.The examples include SANA 1.6B urban scenes, CUB-200 bird generations, and PixArt-Σ flower generations.Source: §D.5 · p. 23

E LIMITATIONS

• The easy-path evaluation raises per-step training cost by approximately 25%, and caching its predictions risks staleness as model parameters change.The paper says wall-clock-matched experiments show optimization-efficiency gains can offset the overhead, while reducing per-step cost remains future work.Source: E LIMITATIONS · p. 23
• TAM’s one-scalar-per-sample memory grows linearly with dataset size, and its curriculum effect weakens when examples are revisited only a few times.Group-level, clustered, or compressed statistics may reduce cost and increase update frequency, but may sacrifice per-sample precision.Source: E LIMITATIONS · p. 23
• Empirical evaluation covers fine-tuning pretrained text-to-image flow-matching models, not training from scratch; the described DDPM-style extension is unvalidated experimentally.The convergence analysis also holds only for a fixed TAM memory state.Source: E LIMITATIONS · p. 23