PaperlayerView exact revision on arXiv

Source-linked AI summary

NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

Tianlin Pan, Lianyu Pang, Cheng Da, Huan Yang, Changqian Yu, Kun Gai, Wenhan Luo

arXiv:2606.27771v5cs.LGcs.CV

TL;DR

RL post-training can improve reward alignment in flow-based generators while degrading perceptual quality that reward proxies miss. This paper introduces NormGuard, a training-time one-sided penalty on excess velocity norm, and finds consistent quality and realism gains across models, methods, and reward proxies while preserving reward.

  • Problem

    RL post-training degrades perceptual quality beyond what reward proxies capture, while aggregate drift controls do not distinguish its structural components.

  • Method

    NormGuard adds a training-time one-sided penalty activated only when the fine-tuned velocity norm exceeds the reference norm.

  • Results

    Across two base models, three post-training methods, and two reward proxies, NormGuard consistently improves image quality and realism while preserving reward, especially under few-step inference.

  • Takeaways & Limitations

    Training-time suppression of excess velocity-norm growth is supported as a practical alternative to inference-time renormalization for improving perceptual quality without sacrificing reward.

  • Takeaways & Limitations

    The analysis is limited to velocity-local objectives, leaving extension to trajectory-level objectives such as Flow-GRPO for future work.

Abstract

from arXiv · show

Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm $\|v_θ\|$ by $5\%$ to $15\%$ relative to the reference. A form of norm inflation has been studied in classifier-free guidance (CFG), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts. However, this inference-time correction does not transfer cleanly to RL: rescaling $v_θ$ to match $\|v_{\text{ref}}\|$ at inference time neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Since inference-time renormalization fails while norm suppression carries no reward cost, training-time intervention is the appropriate strategy. Together, these findings motivate NormGuard, a hinge penalty that activates only when $\|v_θ\|$ exceeds $\|v_{\text{ref}}\|$ and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, NormGuard consistently improves MLLM-judged image quality and forensic realism while preserving reward, with gains that amplify under few-step inference and are not explained by early stopping.

1 Introduction

RL post-training inflates flow-model velocity norms and degrades perceptual quality in ways reward proxies miss, while inference-time renormalization fails to correct the drift. NormGuard therefore suppresses excess velocity norms during training, improving quality and realism while preserving reward gains.

  • Problem: RL post-training degrades perceptual quality through over-sharpening, color shift, unnatural lighting, and lost fine texture despite reward gains.These effects are not captured by the reward proxy.
  • Motivation: 5% to 15%: RL post-training inflates per-step velocity norms relative to the reference across NFT, AWM, and DPO.The inflation is reported uniformly along the denoising trajectory for SD3.5-Medium with PickScore.
  • Inference-time correction: Inference-time renormalization leaves reward essentially unchanged while failing to fix over-sharpening and unnatural lighting in RL-fine-tuned models.The contrast with CFG arises because RL norm inflation is co-adapted into model weights rather than created by an explicit inference-time combination.
  • Reward sensitivity: 3× to 100×: norm-scaling sensitivity has this noise-to-signal ratio, with heterogeneous per-sample signs and a batch mean close to zero across 6,400 samples.Thus, velocity magnitude rescaling reveals no coherent first-order batch-level reward signal.
  • NormGuard: NormGuard applies a training-time hinge penalty only when the candidate velocity norm exceeds the reference norm.This follows from inference-time renormalization failing and norm suppression showing no systematic first-order reward cost.
  • Validation: Across two base flow models, three post-training methods, and two reward models, NormGuard consistently improves MLLM-judged quality and generally improves forensic realism while preserving most reward gains.Image-space measurements also show RL inflates luminance energy by 24% to 46%.

2 Related Work

Related work adapts RLHF-style training to flow-based text-to-image generators using reward models such as PickScore, GenEval, and HPS. This literature also documents reward over-optimization, in which proxy rewards improve while perceptual quality degrades, and proposes several mitigation strategies.

  • RL for Flow-Based Generation: RLHF-inspired methods for flow-based generative models include Diffusion-DPO, DDPO, Flow-GRPO, Dance-GRPO, Diffusion-NFT, and AWM, all trained with reward-model signals.The cited reward models include PickScore, GenEval, and HPS.
  • Reward Hacking: Reward over-optimization can improve proxy reward while degrading perceptual quality through over-sharpening, color bias, and unnatural lighting.These artifacts are reported when optimizing rewards such as PickScore or HPS.
  • Mitigating Reward Hacking: Recent mitigations stabilize optimization, flatten reward landscapes, improve reward alignment with photorealism, or scale up the reward model.The cited approaches include GRPO-Guard, RSA-FT, RealGen, and RewardDance.

3 Norm Inflation: Phenomenon and Diagnostics

RL post-training consistently inflates per-step velocity norms by 5% to 15%, producing characteristic image artifacts that an energy argument links to excess latent variation. Inference-time renormalization fails to improve reward or quality, while norm-scaling sensitivity shows no stable batch-level first-order reward signal, motivating training-time suppression.

  • Norm inflation: 5% to 15%: RL fine-tuning shifts per-step velocity norm distributions upward uniformly across denoising trajectories, with over-sharpened edges, boosted contrast, and unnatural white balance.This occurs across NFT, AWM, and DPO on SD3.5-Medium + PickScore.
  • Energy interpretation: 2δ: a δ-level velocity-norm inflation raises the generated latent’s second moment by roughly 2δ, passing surplus energy to the decoded image as excess variation around its mean.The energy argument assumes uniform trajectory scaling with unchanged direction and negligible noise–latent correlation.
  • Inference-time diagnostics: Inference-time renormalization does not improve reward and produces over-sharpening and unnatural lighting in sampled images, unlike the CFG setting where renormalization removes inference-induced artifacts.In RL, the inflated norm appears to have been absorbed into the model rather than introduced only during deployment.
  • Implication: These findings motivate training-time regularization: inference-time renormalization is insufficient, while suppressing excess velocity-norm growth is unlikely to systematically remove reward-carrying signal.The conclusion follows from the failed reward and quality correction and the near-zero batch-level sensitivity.
  • Sensitivity diagnostics: 3× to 100×: norm-scaling sensitivity has substantial sign-heterogeneous per-sample behavior but a batch mean near zero across timesteps, revealing no stable batch-level first-order reward signal.The measurement uses 6,400 SD3.5-Medium samples with PickScore and HPSv2.

4 NormGuard

NormGuard targets excess velocity-norm growth with a one-sided, reference-budget penalty in the same local velocity space as the post-training losses. This design leaves under-budget updates unconstrained and composes with NFT, AWM, and DPO objectives.

  • Velocity-local structure: Velocity-local losses update parameters through per-timestep residuals in the model’s local velocity output space.The scalar weighting may depend on state, timestep, condition, or reward label.
  • Velocity-local structure: NFT, AWM, and DPO fit the velocity-local template, whereas Flow-GRPO operates through trajectory-level likelihood ratios over reverse transitions.NFT uses reward-weighted residuals, AWM uses an advantage-weighted residual with a KL term, and DPO uses sigmoid-weighted local deviations.
  • NormGuard penalty: NormGuard activates only when the current velocity norm exceeds the reference-model norm, suppressing excess growth while minimally interfering with directional updates.The penalty is one-sided and targets the norm budget defined by the reference velocity.
  • NormGuard penalty: The hinge leaves updates unconstrained when ∥v_θ∥2^2 ≤ ∥v_ref∥2^2 and penalizes only norm beyond that reference budget.The regularization strength is controlled by λ > 0.
  • Compatibility: Because the penalty acts in the same velocity output space as the base loss, Rnorm composes naturally with NFT, AWM, and DPO.When active, its gradient modifies updates through the local velocity at the sampled state rather than a separate parameter-space constraint.

5 Experiments

Across two base models, three post-training methods, and two reward models, NormGuard improves judged image quality and forensic realism while largely preserving reward. Its benefits persist across inference-step counts, are not explained by early stopping, and remain complementary to KL regularization.

  • Experimental Setup: NormGuard is evaluated across two base flow models, three RL post-training methods, and two reward models.The experiments use SD3.5-Medium and FLUX.2-klein-base-4B with NFT, AWM, DPO, PickScore, and HPSv2.
  • Image Quality and Realism: NormGuard is preferred for image quality in all seven settings by both Qwen3.5-35B and GPT-4.1.The two judges agree on preference direction across configurations spanning both base models, all three fine-tuning algorithms, and both reward signals.
  • Image Quality and Realism: MLLM quality improves substantially while PickScore remains largely unchanged under NormGuard.Figure 6 shows nearly vertical transitions from the unregularized baseline to NormGuard across configurations.
  • Few-step robustness: NormGuard’s MLLM win-rate gap grows from 9% at 28 steps to 20% at 4 steps.Over the same reduction, baseline RealScore falls from 0.239 to 0.189 while NormGuard remains comparatively stable.
  • Not explained by early stopping: NormGuard outperforms earlier baseline checkpoints in reward, RealScore, and MLLM score, ruling out early stopping as the explanation.The comparison uses baseline checkpoints at iterations 160, 180, and 200 versus NormGuard at iteration 200.
  • Sensitivity to λ: Nonzero regularization consistently outperforms λ= 0, with λ= 1 achieving the highest PickScore at 200 iterations.Performance is stable across a broad range of nonzero λ values, while λ= 10 appears slightly over-regularized early in training.

6 Conclusion

RL post-training inflates per-step velocity norms by 5 to 15%, and inference-time renormalization fails because the inflation is trained into the weights. Training-time intervention is therefore necessary and safe, while the dynamic origin and broader applicability of radial inflation remain open questions.

  • Conclusion: 5 to 15%: RL post-training inflates per-step velocity norms, producing the same artifact family as CFG overamplification.Unlike CFG, RL inflation is trained into the model weights.
  • Conclusion: Inference-time renormalization fails, while suppressing velocity magnitude carries no coherent first-order reward signal at the batch level.These findings motivate training-time intervention as necessary and safe.
  • Limitations and future work: The dynamic origin of radial inflation remains open, and the reward-neutrality claim is limited to first-order batch-level effects.Per-sample radial perturbations affect reward but average to zero across prompts; the analysis covers velocity-local objectives NFT, AWM, and DPO.

A Theoretical Analysis of Velocity-Local Post-Training Losses · A.1 Notation

The appendix formalizes velocity-local post-training losses under unified notation, defining the noisy latent, velocity predictors, reference field, target, weighting, and Jacobian used in subsequent derivations.

  • A Theoretical Analysis of Velocity-Local Post-Training Losses: The appendix unifies Diffusion-DPO, Diffusion-NFT, AWM, and Flow-GRPO to identify which methods admit a velocity-residual gradient form.This formal analysis underlies the velocity-local property discussed in Section 4.1.
  • A.1 Notation: x_t = (1 − t)x_0 + tε with ε ∼ N(0, I) denotes the noisy latent at time t.
  • A.1 Notation: v_θ(x_t, t, c) denotes the trainable velocity predictor.
  • A.1 Notation: v_target := ε − x_0 denotes the flow-matching target.
  • A.1 Notation: v_ref denotes the frozen reference velocity field supplied by the pretrained model.
  • A.1 Notation: ω(t) denotes a timestep-dependent weighting used in the formal loss notation.
  • A.1 Notation: J_θ(x_t, t, c) := ∇_θv_θ(x_t, t, c) denotes the per-sample velocity Jacobian.
  • A.1 Notation: The flow-matching pretraining objective is introduced as part of the notation for the subsequent theoretical derivations.

A.2 Method Decompositions in Velocity Coordinates

This section expresses each method’s training signal in the established velocity-coordinate system and verifies that it is velocity-local under Definition 1.

  • The methods’ training signals are rewritten in the preceding coordinate system.
  • The decomposition covers each method’s training signal in velocity coordinates.
  • The analysis verifies the velocity-local property specified by Definition 1.

A.2.1 Diffusion-DPO … A.3 Summary

The paper characterizes Diffusion-DPO, Diffusion-NFT, and AWM as velocity-local post-training methods, while Flow-GRPO operates through a trajectory-level likelihood ratio and is out of scope. The appendix summarizes these methods by their velocity-locality under Definition 1.

  • A.2.1 Diffusion-DPO: Diffusion-DPO rewrites the Bradley–Terry preference objective as per-timestep comparisons between winning and losing samples.Its gradient is a sigmoid-weighted sum of velocity-residual gradients targeting v_ref, making it velocity-local.
  • A.2.2 Diffusion-NFT: Diffusion-NFT constructs implicit positive and negative policies from the same trainable parameters.Its supervised forward-process regression uses a binary reward-derived optimality label, and its gradient is a reward-weighted flow-matching residual, hence velocity-local.
  • A.2.3 Advantage-Weighted Matching (AWM): AWM replaces DDPO’s per-step reverse-transition likelihood with a sequence-level policy whose likelihood is approximated through the flow-matching ELBO.Its likelihood ratio is estimated from differences of flow-matching losses, with a velocity-space MSE KL term.
  • A.2.3 Advantage-Weighted Matching (AWM): AWM’s policy update is the same per-timestep flow-matching residual as pretraining, modulated only by the advantage weight A_i.Both the policy and KL terms are velocity-local.
  • A.2.4 Flow-GRPO: Flow-GRPO formulates denoising as an MDP whose policy is the reverse transition and optimizes a clipped policy gradient with an importance ratio.It introduces stochastic exploration by converting the deterministic ODE into an SDE and discretizing it with Euler–Maruyama.
  • A.2.4 Flow-GRPO: Flow-GRPO’s Gaussian reverse-transition mean depends on v_θ through the drift term, while shared variance reduces the importance ratio to a function of the Gaussian means.These passages establish how the velocity enters the trajectory-level transition model.
  • A.2.4 Flow-GRPO: Flow-GRPO’s gradient uses a trajectory-level likelihood ratio rather than a per-timestep velocity residual, so v_θ enters only indirectly through the Gaussian transition mean.It therefore violates Definition 1 and is treated as out of scope.

B Image-Space Energy Statistics

The section defines luminance and chrominance energy statistics from mean-centered YCbCr images, averaged across the evaluation set. These statistics measure non-DC spatial structure and are invariant to global brightness or color offsets.

  • Decoded images are normalized to [0, 1] per RGB channel and converted to YCbCr using the BT.709 convention.
  • Y carries luminance, while Cb and Cr carry chrominance; per-image spatial second central moments are computed for resolution H×W and averaged over the evaluation set.
  • Subtracting each channel mean removes the 2D DFT’s DC coefficient, so Parseval’s identity relates the moments to non-DC spatial-frequency energy.
  • Elum measures total non-DC luminance energy, whereas Echroma measures the corresponding energy across both chroma channels.
  • Both statistics ignore global brightness or color offsets and respond only to structure around each image’s own mean.
  • High Elum indicates excessive luminance contrast, while high Echroma indicates oversaturation and low Echroma indicates desaturation.

C Training Details · D MLLM Evaluation Prompts · E Prompts for Main-Paper Figures

The appendices specify training hyperparameters, the MLLM realism-comparison protocol, and the prompts used for main-paper qualitative examples. Training details are organized by experiment and optimization scope, while evaluation requires structured pairwise realism judgments and figure prompts sampled from HPDv3.

  • C Training Details: Training details provide full hyperparameters for every model, method, and reward combination, with rollout sampling steps and group size explicitly defined.Method-specific and shared optimization settings are provided separately in Tables 8 and 9.
  • C Training Details: Method-specific hyperparameters and shared optimization settings are documented separately to complement the main experiment configurations.These settings are listed in Tables 8 and 9.
  • D MLLM Evaluation Prompts: The Image Realism Judge compares image_A and image_B to determine which appears more like a genuine, unmanipulated photograph.The comparison is framed as a realism judgment between two input images.
  • D MLLM Evaluation Prompts: Evaluation prioritizes physical plausibility, texture and material fidelity, and coherent object boundaries, while penalizing overly smooth, plastic, painted, or oily-looking surfaces.Criteria include consistency of lighting, shadows, reflections, perspective, scale, and natural micro-detail.
  • D MLLM Evaluation Prompts: Each image receives a 1–10 realism score, followed by a winner decision of A, B, or tie and a brief evidence-based justification.The prompt requires concrete realism strengths and weaknesses for both images and strict JSON output with scores, evidence, winner, and justification.
  • E Prompts for Main-Paper Figures: The qualitative-example prompts are grouped by figure and sampled from HPDv3 [31].The appendix organizes the text prompts by the figures in which the generated examples appear.
  • E.1 Teaser: The E.1 Teaser includes prompts for a tropical coastline, a bird over a mountain vista, and a tree silhouette at sunset.The accompanying descriptions specify detailed natural scenes involving coastlines, mountains, and sunset silhouettes.
  • E.1 Teaser: The sunset teaser prompt describes a computer-generated artwork with a dark horizon, blue-gray distance, and a slender silhouetted tree against a vibrant sky.The tree has a thin trunk and sparsely distributed leaves.

E.2 Main Results

The section presents qualitative image examples spanning SD3.5 and FLUX.2 with PickScore under NFT and AWM, including a 10-step AWM setting. The examples cover varied subjects and visual styles, from urban scenes and landscapes to festive and surreal imagery.

  • SD3.5 + PickScore + NFT: SD3.5 with PickScore and NFT includes urban-night, sculpted-book, and pumpkin-patch image examples.The examples depict a red-lit person holding a smartphone, a three-dimensional landscape emerging from an open book, and an autumn pumpkin field.
  • SD3.5 + PickScore + AWM: SD3.5 with PickScore and AWM includes Buddha-statue, seated-woman, and graffiti-wall examples.The scenes range from a twilight setting with a torii gate to an urban portrait and colorful street art featuring a woman and alien character.
  • SD3.5 + PickScore + AWM + 10 Steps: SD3.5 with PickScore and AWM at 10 Steps shows abstract-village, mountain-precipice, and Hanukkah-gift examples.These examples include a stylized painted landscape, a solitary figure overlooking rugged mountains, and a snow globe with a dreidel beside wrapped gifts.
  • FLUX.2 + PickScore + NFT: FLUX.2 with PickScore and NFT includes a sepia-toned couple sitting together on a skateboard at dusk.The image is described as an intimate, nostalgic scene on a deserted road.
Loading 2606.27771v5…