PaperlayerView exact revision on arXiv

Source-linked AI summary

AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

Sicong Kang, Fangao Zeng, Yuhan Li, Hao Zhou, Mengfei Xu, Wenxiang Shang, Wei Li, Bingbing Ni, Pipei Huang

arXiv:2610.29380v1cs.CV

TL;DR

Image-generation pipelines often apply RL alignment and few-step distillation sequentially, and joint DMD methods raise questions about preserving RL gains. AdvFidelity adds online adversarial fidelity supervision to RL, jointly training alignment and few-step generation without explicit distillation. Across two backbones and three task settings, it matches full-step LeapAlign with 4 or 8 steps and no CFG.

  • Problem

    Image generation needs preference alignment and efficient few-step inference, but sequential RL and distillation can be cumbersome and distillation may wash out reward gains; preservation of full-step RL gains in joint methods is also insufficiently evaluated.

    Source: §1 · p. 1

  • Method

    AdvFidelity combines task rewards with an online discriminator trained on high-quality references and content-matched reconstructions, jointly learning reward alignment and few-step generation within one RL stage without DMD.

    Source: §1 · p. 3, §5 · p. 12

  • Results

    Across two model backbones and three task settings, 4- or 8-step AdvFidelity without CFG matches full-step LeapAlign in alignment, task performance, and overall generation quality, and remains highly competitive with evaluated full-step RL methods.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    Within the evaluated settings, task and fidelity rewards support joint alignment and few-step generation without an explicit distillation objective.

    Source: §5 · p. 12

Abstract

from arXiv · show

Practical image generation requires both human preference alignment and few-step efficiency, yet combining reinforcement learning (RL) with distillation is often multi-stage and may erode RL gains We observe that few-step RL progressively improves structural coherence, even with specialized rewards, but leaves low-level artifacts that these rewards may fail to penalize. To address this gap, we propose AdvFidelity, which integrates artifact-sensitive fidelity supervision into RL through an online adversarial reward. This enables joint reward alignment and high-fidelity few-step generation within a single RL stage, without distribution matching distillation (DMD) or other explicit distillation objectives. We train a discriminator to rank high-quality references (positives) above content-matched reconstructions (negatives) produced by the current generator. Reconstructing moderately noised references largely preserves style and scene layout while exposing low-level degradations. This encourages the discriminator to assess artifacts rather than content differences. With negatives reconstructed online, alternating updates adapt fidelity supervision to the generator's evolving failure modes. References can come from existing target-domain data or full-step model samples. Built on LeapAlign, AdvFidelity backpropagates gradients from both rewards through the few-step sampling trajectories used at inference. Across two model backbones and three task settings, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality and remains highly competitive with other evaluated full-step RL methods, using only 4 or 8 steps without CFG.

1 INTRODUCTION

AdvFidelity combines reward alignment and few-step generation in one RL stage, adding an online fidelity reward to address low-level artifacts. Across two backbones and three task settings, it matches full-step LeapAlign using 4 or 8 steps without CFG.

1 INTRODUCTION

  • Few-step RL can improve image quality, but subtle noise artifacts may remain because task-specific rewards do not necessarily penalize them.This gap motivates adding artifact-sensitive fidelity supervision alongside task rewards.Source: §1 · p. 2
  • Fidelity references may be full-step base-model outputs, existing target-domain images, or samples from a model covering the desired visual domain.Base-model outputs can suit identity, text-accuracy, or composition rewards; real images can serve when photorealism is prioritized.Source: §1 · p. 2
  • AdvFidelity jointly learns reward alignment and few-step generation in one RL stage, without distillation warm-up or parallel DMD training.It avoids matching a frozen, RL-agnostic base distribution.Source: §1 · p. 3
  • An online discriminator compares high-quality references with content-matched negatives, adapting to evolving artifacts while reducing high-level differences that could provide discrimination shortcuts.Negatives are reconstructed from moderately noised references using the current generator, largely preserving style and scene layout.Source: §1 · pp. 2–3
  • With 4 or 8 steps and no CFG, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality, while remaining highly competitive with evaluated full-step RL methods.The evaluations span two model backbones and three task settings.Source: §1 · p. 3

2 RELATED WORK

Prior work approaches image-generation alignment through policy gradients, preference learning, reward-based objectives, and differentiable reward gradients, while distillation reduces inference steps. AdvFidelity instead learns few-step generation through task and fidelity rewards within RL, without an explicit distillation objective.

Reinforcement Learning

  • Policy-gradient methods optimize terminal rewards, while DPO-based approaches learn from preference pairs; direct-gradient methods propagate reward gradients through sampling, and LeapAlign extends them to arbitrary generation steps.DDPO, Flow-GRPO, DanceGRPO, and MixGRPO adapt group-relative policy optimization to diffusion and flow models [Black et al., 2024; Liu et al., 2026; Xue et al., 2025b; Li et al., 2025].Source: §2 · p. 3

Few-Step Distillation

  • Consistency-based methods and DMD use explicit distillation objectives for few-step generation, whereas AdvFidelity learns few-step generation through task and fidelity rewards within RL.Applying distillation after RL may reduce reward gains.Source: §2 · p. 3

Joint Reward Alignment and Few-Step Generation

  • DMDR and RTDMD combine reward optimization with DMD after a distillation cold start, but their comparisons leave preservation of full-step RL gains insufficiently established.The cited evaluations have gaps in corresponding baselines or comparability across training data and reward configurations.Source: §2 · pp. 3–4

Online Adversarial Rewards

  • Reward-Instruct uses regularized reward maximization without diffusion distillation losses, while AdvFidelity learns an online fidelity reward and compares against its underlying full-step RL method.Adv-GRPO also trains an adversarial reward, but its setup lacks content-matched reference–reconstruction pairs, leaving style or content differences as potential shortcuts.Source: §2 · p. 4

3 METHOD

AdvFidelity jointly trains few-step reward alignment and fidelity in one RL stage. It combines task rewards with an online discriminator trained on content-matched reconstructions.

3.1 PRELIMINARIES: RL ON FEW-STEP ROLLOUTS

  • Few-step rollouts are optimized against task or preference rewards, while an online discriminator supplies a fidelity reward for persistent low-level artifacts.The combined reward is rspec + λDϕ(x), with λ balancing the rewards.Source: §3.2 · p. 4

3.5 ADVERSARIAL TRAINING PROCEDURE

  • Each training iteration updates the generator with both rewards, then repeatedly obtains references, constructs current-policy negatives, and updates the discriminator.References may be existing images or model-generated samples; the procedure jointly learns alignment and few-step generation.Source: Figure 2 · p. 5, §3.5 · pp. 6–7

3.2 OVERVIEW OF ADVFIDELITY

  • AdvFidelity jointly learns preference alignment and high-fidelity few-step generation in one RL stage, using reward gradients through the few-step sampling trajectory.The implementation uses LeapAlign (Liang et al., 2026).Source: §3.1 · p. 4, §3.2 · p. 5

3.3 ADVERSARIAL FIDELITY REWARD

  • The discriminator ranks high-quality references above current-policy reconstructions with a Bradley–Terry loss, using whole images and random crops to detect localized defects.Cropping makes local defects occupy more of the input and can reduce reliance on global cues such as tone or border effects.Source: §3.3 · p. 5

3.4 CONTENT-MATCHED PAIR CONSTRUCTION

  • Moderately noised references are reconstructed by the current generator to create negatives that largely preserve style and layout while exposing low-level degradations.The experiments randomly sample noise levels from {0.3, 0.5}; reconstruction uses one or two denoising steps in practice.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate AdvFidelity across preference alignment, compositional generation, and virtual dressing, using different fidelity-reference sources. Results compare few-step AdvFidelity with full-step RL methods and examine its components.

4 EXPERIMENTS

  • AdvFidelity evaluates HPDv2 preference alignment and GenEval compositional generation from FLUX.1-dev, plus virtual dressing; it uses 4 or 8 FLUX sampling steps and 8 dressing steps, without CFG.Source: §4 · p. 7

4.1 PREFERENCE ALIGNMENT ON HPDV2

  • For HPDv2, 50,000 reference image–prompt pairs come from 50-step HPSv2-aligned LeapAlign samples because the available image collection does not consistently reflect reward-preferred appearance.AdvFidelity starts from FLUX.1-dev and does not use the reference model’s scores or denoising trajectories.Source: §4.1 · pp. 7–9

4.1 PREFERENCE ALIGNMENT ON HPDV2; 4.2 COMPOSITIONAL GENERATION ON GENEVAL; 4.3 VIRTUAL DRESSING ON REAL DATA

  • Across HPDv2, GenEval, and virtual dressing, few-step AdvFidelity matches or exceeds full-step LeapAlign on reported alignment, compositional, and dressing outcomes.On HPDv2, 8-step AdvFidelity scores 0.4125 on HPSv2 versus 0.4096 for 50-step LeapAlign; on GenEval, 4- and 8-step scores are 0.7535 and 0.7759 versus 0.7420.Source: Table 1 · p. 8, Table 2 · p. 8, §4.1 · p. 9

4.2 COMPOSITIONAL GENERATION ON GENEVAL

  • GenEval uses 50,000 training prompts, 553 test prompts, and 5,000 50-step FLUX.1-dev reference pairs, as its visual-style changes are milder than HPDv2’s.The references supervise fidelity without needing to satisfy every compositional constraint; the generator is optimized directly with HPSv2.Source: §4.2 · pp. 9–10

4.3 VIRTUAL DRESSING ON REAL DATA

  • Virtual dressing reuses real try-on images from the 1,000-pair training split as fidelity references and uses DINO feature similarity as its task reward.Evaluation uses 124 test samples, with Gemini scores averaged across three runs.Source: §4.3 · pp. 10–11

4.4 ABLATION STUDIES

  • Online discriminator updates, target-style references, and content-matched negatives progressively improve metrics; sequential DMD lowers LeapAlign’s HPSv2 from 0.4096 to 0.3978.The full AdvFidelity configuration achieves the strongest results among the ablation configurations.Source: Table 4 · p. 11, §4.4 · pp. 11–12

5 CONCLUSION

AdvFidelity combines reward alignment and few-step generation in one RL stage, adding online fidelity supervision to address low-level artifacts. Across two backbones and three task settings, it matches full-step LeapAlign with 4 or 8 steps without CFG.

5 CONCLUSION

  • Few-step AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality across two backbones and three task settings, using 4 or 8 steps without CFG.It remains highly competitive with other evaluated full-step RL methods.Source: §5 · p. 12
  • AdvFidelity trains an online discriminator on content-matched pairs to complement task rewards with fidelity supervision directed at low-level artifacts.Both rewards are optimized through the few-step sampling trajectories used at inference.Source: §5 · p. 12
  • The framework jointly learns reward alignment and few-step generation in a single RL stage, without distillation warm-up or DMD.Source: §5 · p. 12

A ADDITIONAL TRAINING DETAILS

The appendix summarizes training configurations and explains the use of LoRA under GPU memory constraints.

A ADDITIONAL TRAINING DETAILS

  • Training configurations report iterations, effective global batch size, image resolution, and generator LoRA rank for each experimental setting.Each variant is trained at its inference sampling budget; 4-step and 8-step FLUX variants are separately trained models.Source: A ADDITIONAL TRAINING DETAILS · p. 15, Table 5 · p. 15
  • LoRA is used for GPU memory constraints, with adapters on attention and MLP projections and remaining base-model weights frozen.The discriminator backbone is also frozen, and only its MLP head is trained.Source: A ADDITIONAL TRAINING DETAILS · p. 15
  • The effective global batch size equals per-device batch size multiplied by gradient accumulation and eight GPUs; iterations are counted after gradient accumulation.Source: Table 5 · p. 15

B RELIABILITY OF AUTOMATED METRICS ON FEW-STEP ARTIFACTS

The appendix tests whether six automated metrics distinguish references from artifact-laden, content-matched short reconstructions. Their paired rankings vary substantially, so they do not uniformly provide reliable fidelity supervision.

C ADDITIONAL QUALITATIVE RESULTS

The appendix provides additional qualitative results across preference alignment, compositional generation, and virtual dressing.

Related papers

Loading 2610.29380v1…