PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models

Xin Xie, Jiaxian Guo, Dong Gong

arXiv:2601.15968v2cs.CV

TL;DR

Diffusion alignment must improve prompt fidelity and visual quality without sacrificing efficiency, generalization, or diversity. HyperAlign trains a hypernetwork to generate conditioned low-rank adaptations that steer denoising toward rewards, and reports superior semantic consistency and aesthetic quality across diffusion and rectified-flow backbones. Preference regularization and granularity variants address reward over-optimization while offering efficiency trade-offs.

  • Problem

    Diffusion alignment seeks to improve semantic consistency and visual quality, while test-time methods add computation and training-based methods risk reward over-optimization and reduced diversity.

  • Method

    HyperAlign trains a hypernetwork that dynamically generates input-and-state-conditioned low-rank adaptation weights to steer frozen models’ denoising trajectories toward target rewards.

  • Results

    HyperAlign delivers superior semantic consistency and aesthetic quality across diffusion and rectified-flow backbones compared with existing fine-tuning and test-time alignment approaches.

  • Takeaways & Limitations

    Granularity variants provide flexible trade-offs between computational efficiency and alignment precision across multiple generative paradigms.

  • Takeaways & Limitations

    Reward optimization is constrained by inaccurate early-stage reward signals and risks reward hacking or degraded visual fidelity; the framework’s compatibility with flow-matching models is noted through experiments such as FLUX.

Abstract

from arXiv · show

Diffusion model alignment aims to bridge the gap between generated outputs and human preferences by enhancing both semantic consistency with textual prompts and overall visual quality. Existing alignment methods face a challenging trade-off: test-time approaches enable input-specific adaptability but introduce significant computational overhead and tend to under-optimize, while fine-tuning approaches risk reward over-optimization and loss of generation diversity. To bridge this gap, we propose HyperAlign, a framework that trains a hypernetwork for efficient and effective test-time alignment. Instead of modifying latent states directly, HyperAlign dynamically generates input-and-state-conditioned low-rank adaptation weights to modulate the denoising trajectory toward target rewards. We introduce multiple HyperAlign variants of varying granularity to balance alignment quality and computational efficiency. The hypernetwork is optimized with a reward objective regularized by preference data to mitigate reward hacking. We evaluate HyperAlign across multiple generative paradigms, including Stable Diffusion and FLUX, where it significantly outperforms existing alignment methods in semantic consistency and visual quality.

1 Introduction

Diffusion alignment seeks better prompt fidelity and visual quality, but existing test-time and training-based approaches trade adaptability, efficiency, generalization, and diversity. HyperAlign addresses this trade-off with a hypernetwork that generates conditioned LoRA adaptations for denoising, with variants balancing quality and efficiency.

  • Diffusion alignment targets semantic consistency with textual prompts and overall visual quality, addressing gaps between generated outputs and human preferences.
  • Training-based alignment can reshape the output distribution but has limited generalization across varying prompts and sampled initial noise.
  • Test-time alignment adapts denoising to each query and intermediate state, but gradient calculations and repeated forward passes impose substantial computational overhead.
  • HyperAlign combines input-specific adaptability with training-based efficiency by generating input-and-state-conditioned adaptation weights on the fly.
  • HyperAlign dynamically generates low-rank adaptation weights for frozen diffusion models to steer intermediate denoising steps toward target rewards.
  • Three weight-generation strategies vary in granularity to balance alignment quality and efficiency, while preference regularization mitigates reward hacking and preserves diversity.
  • HyperAlign improves quality and efficiency across SD V1.5 and FLUX compared with state-of-the-art alignment methods.

2 Related Work

Prior alignment research includes reward-based fine-tuning, test-time scaling, and hypernetwork-based prediction. These approaches respectively optimize preferences, spend additional inference computation, or predict network parameters for task-specific adaptation.

  • Reward-based methods learn preferences through direct optimization, reinforcement learning, or preference optimization, but long-trajectory gradients can be unstable.
  • Test-time scaling increases inference computation through additional denoising steps, sampling-based selection, or other query-specific procedures.
  • Hypernetworks predict primary-network weights and have been applied across language, vision, image editing, personalization, and diffusion noise prediction.

3 Problem Setup: Diffusion Model Alignment

Diffusion models generate conditional samples by reversing a gradual noising process, while alignment modifies this process toward outputs favored by a reward model. The setup frames alignment as a reward-tilted distribution balanced against consistency with the base model.

  • 3.1 Preliminary on Score-based Generative Models: Diffusion models learn score functions that support reverse trajectories transforming Gaussian noise into structured outputs.
  • 3.1 Preliminary on Score-based Generative Models: The reverse process starts from x_T ∼ N(0, I) and progressively denoises latent states into a clean sample reflecting conditioning prompt c.
  • 3.1 Preliminary on Score-based Generative Models: Score-based formulations unify diffusion and flow-matching models through stochastic or ordinary differential equation trajectories.
  • 3.1 Preliminary on Score-based Generative Models: Conditional diffusion models use c, such as an image-generation prompt, to control reverse-process sampling from noisy latent states.
  • 3.2 Aligning Diffusion Model with Reward: Alignment addresses outputs that lack visual quality and prompt semantic consistency because standard denoising optimizes reconstruction rather than human preference.
  • 3.2 Aligning Diffusion Model with Reward: A reward model conditioned on c captures consistency between generated content and prompts alongside visual aesthetic preference.
  • 3.2 Aligning Diffusion Model with Reward: HyperAlign steers denoising by injecting step-wise modulation weights produced by a hypernetwork into the generative model.
  • 3.2 Aligning Diffusion Model with Reward: Reward alignment formulates a tilted distribution that balances reward maximization with consistency with the base model through KL regularization.

4 Alignment as Trajectory Steering and HyperAlign

HyperAlign reframes test-time diffusion alignment as learned trajectory steering: a hypernetwork generates input- and state-conditioned LoRA adaptations that guide denoising toward target rewards. Its variants trade alignment quality against efficiency by changing update granularity, while preference regularization addresses reward over-optimization.

  • HyperAlign: HyperAlign learns a hypernetwork for efficient test-time trajectory steering in diffusion and flow models.It aims to retain input-specific adaptability while amortizing test-time alignment into learned model operations.
  • Test-time alignment: Gradient-based test-time alignment injects reward gradients into intermediate denoising transitions, conditioning trajectory adjustments on prompts and latent states.The approach modifies each transition toward a reward-tilted distribution rather than searching only over initial noise.
  • Hypernetwork design: HyperAlign replaces repeated test-time gradient computation with a hypernetwork that generates input-and-step-conditioned low-rank adaptation weights for a frozen model.The generated LoRA weights modify the score function on the fly, steering intermediate denoising states while preserving the pretrained model parameters.
  • Efficiency variants: HyperAlign-I, HyperAlign-P, and HyperAlign-S vary LoRA update frequency to balance inference efficiency and trajectory adaptivity.HyperAlign-I updates once, HyperAlign-P updates at selected stage boundaries, and HyperAlign-S updates at every generation step.
  • Efficiency variants: The piece-wise variant groups denoising steps with similar dynamics, using stage transitions to reduce the frequency of LoRA regeneration.Early steps make larger changes for coarse structure, whereas later steps refine fine-grained appearance; curvature identifies transition points.
  • Training objective: Reward-based training keeps the pretrained diffusion model frozen and uses preference-data regularization to reduce reward hacking and preserve visual fidelity.The regularizer counters inaccurate early-stage rewards caused by blurry predicted clean data and aggressive reward optimization.

5 Experiments

Experiments evaluate HyperAlign across SD V1.5 and FLUX using AI feedback, qualitative comparisons, diversity tests, efficiency measurements, user studies, and ablations. Results report strong alignment and visual quality while preserving diversity, reducing deployment overhead, and benefiting from reward–preference regularization.

  • Evaluation Setup: HyperAlign is evaluated on Pick-a-Pic, GenEval, HPD, and Partiprompt using six AI feedback models covering preference, prompt alignment, and visual appeal.All methods use 50 denoising steps for fair comparison.
  • Quantitative Analysis: HyperAlign outperforms previous alignment methods on Pick-a-Pic for SD V1.5 and FLUX, while faster variants retain competitive performance.The comparison uses HPSv2 as the sole reward model and attributes the alignment to step-by-step trajectory adjustment.
  • Qualitative Results: Qualitative comparisons show coherent layouts, semantically rich prompt-aligned content, and improved aesthetics across SD V1.5 and FLUX backbones.Additional visual results report strong performance across all three HyperAlign variants despite differing LoRA-generation frequencies.
  • Diversity: HyperAlign preserves output diversity across random seeds, unlike prior methods whose outputs can collapse toward a single style or identity.Diversity is assessed with LPIPS and DINOv2 similarities over 50 images per HPD prompt.
  • Efficiency: HyperAlign amortizes iterative test-time computation into hypernetwork passes, requiring 8 GPU hours for SD V1.5 and 12 GPU hours for FLUX with minimal LoRA overhead.For SD V1.5, training is roughly equivalent to generating 178 DyMO images or 240 FreeDoM images.
  • Ablation Study: Reward-only optimization boosts most preference scores but severely degrades CLIP, whereas preference-only supervision yields marginal gains.The results support combining reward optimization with preference-data regularization to avoid reward over-optimization.

6 Conclusion

The conclusion presents HyperAlign as a model-agnostic framework that generates low-rank modulation weights across denoising steps for trajectory-level alignment. Experiments across diffusion and rectified-flow backbones show stronger semantic consistency and aesthetic quality with tunable efficiency–precision trade-offs.

  • Conclusion: HyperAlign dynamically generates low-rank modulation weights across denoising steps to align generative trajectories using reward signals.Its variants trade alignment precision against computational efficiency.
  • Conclusion: Experiments on diffusion and rectified-flow backbones report superior semantic consistency and aesthetic quality compared with fine-tuning and test-time alignment approaches.The conclusion also identifies lighter hypernetwork designs and improved dynamic adaptation as future directions.
  • Model Scope: HyperAlign applies to both diffusion and flow-matching models by transferring reward-guided trajectory steering to the corresponding latent dynamics.The main formulation uses diffusion models, while FLUX demonstrates the flow-matching application.

B.1 Architecture of the Hypernetwork

The HyperAlign hypernetwork conditions on noisy latents, timesteps, and text embeddings, then generates standardized low-rank factors that are combined with module-specific matrices to adapt all attention projections. Zero initialization and LoRA-modulated trajectories support stable train–inference behavior.

  • Architecture: The hypernetwork generates LoRA weights for all attention projection modules across the pre-trained denoising model.The covered projections include to.q, to.k, to.v, and to.out.0 across N modules.
  • LoRA Weight Generation: Module-specific auxiliary matrices absorb varying input and output dimensions, while the hypernetwork produces unified-dimensional factors on the fly.The resulting LoRA update is expressed as ∆W_i = A_iB_i.
  • Architecture: The perception encoder takes noisy latent x_t, timestep t, and prompt embedding c as inputs and produces features for weight generation.The encoder uses downsampling blocks from the pre-trained SD V1.5 U-Net.
  • Architecture: A transformer decoder processes encoder features and zero-token inputs before producing hypernetwork outputs reshaped into low-rank factors for each module.The decoder uses four transformer blocks and a final linear layer.
  • Training Stability: LoRA-modulated denoising trajectories ensure training–inference consistency, with auxiliary matrices initialized to zero for stable training.The configurations, model size, training setup, and cost are documented in the accompanying tables.

C.1 Additional Qualitative Results

Additional qualitative results compare HyperAlign with baselines on SD V1.5 and FLUX, showing high-quality images aligned with contextual semantics and human preferences. All three variants maintain strong visual performance despite different LoRA-generation frequencies.

  • Additional Qualitative Results: Additional SD V1.5 and FLUX visual comparisons show images that more closely match contextual semantics and human preferences.The results are presented in Figures 13 and 14.
  • Additional Qualitative Results: All three HyperAlign variants achieve strong qualitative performance despite differing frequencies of new LoRA weight generation.This demonstrates flexibility across the variants’ adaptation granularities.
  • Ablation Visualizations: Ablation visualizations are consistent with numerical results across HPSv2-based and PickScore-based configurations.The ablations examine different reward models and preference datasets.

C.2 Additional Quantitative Results

Additional results evaluate HyperAlign across SD V1.5, FLUX, distilled models, and comparisons with HyperNoise. The method shows strong benchmark performance and favorable amortized inference complexity, while cross-setting comparisons require care because the methods target different scopes.

  • HyperAlign performs strongly on GenEval with SD V1.5, including overall performance, attribute binding, and object synthesis.
  • HyperAlign is evaluated on FLUX-based backbones with GenEval and qualitative comparisons, including an objective combining HPSv2 and CLIP.
  • HyperAlign requires 8 GPU hours for SD V1.5 and 12 GPU hours for FLUX, versus 12h for AlignProp, 216h for DanceGRPO, and 192h for Diffusion-DPO.
  • 8.83h total latency on Pick-a-Pic with 1,000 prompts compares with 66h for FreeDoM and 83h for DyMO using SD V1.5.
  • Direct comparison with HyperNoise is not straightforward because HyperNoise targets one-step distilled generators, whereas HyperAlign adjusts trajectories in general multi-step denoising.
  • HyperAlign achieves competitive or superior performance to HyperNoise across one-step, four-step, and sixteen-step inference settings.

C.5 Additional Ablation Study Results

The ablations examine scalability, LoRA update frequency, timestep evolution, and prompt-specific variation. Results support a performance–efficiency trade-off and show that generated LoRA weights change across timesteps and inputs.

  • The default LoRA rank and transformer decoder depth provide sufficient expressiveness while balancing performance and computational cost.
  • HyperAlign-I, HyperAlign-P, and HyperAlign-S differ only in LoRA update frequency, enabling controlled performance–efficiency comparisons.
  • HyperAlign is compared with HyperNoise on SD-Turbo and FLUX.1.Schnell across multiple inference-step settings.
  • Uniform three-segment updates remain competitive with ten-segment updates, supporting the curvature-based segmentation strategy.
  • As denoising progresses, cosine similarity to initial LoRA weights decreases while the ℓ1 relative change rate increases.
  • PCA visualizations over 200 prompts show scattered LoRA parameters, confirming input-specific weight generation across denoising timesteps.

C.7 More Details on Human Evaluation

The human evaluation uses structured surveys presenting multiple methods for the same prompt and asks participants to judge overall preference, visual appeal, and prompt alignment. Additional figures and tables document qualitative comparisons, prompts, and ablations.

  • Each survey section presents multiple images generated by different methods for the same prompt.
  • Participants answer separate questions about overall preference, visual appeal, and how well images fit the text description.
  • The appendix includes prompts for generated-image figures and a scalability table based on HyperAlign-S.
  • Qualitative comparisons cover SD V1.5, FLUX, SD-Turbo, and FLUX-based objective variants.
  • The visual ablation study compares baseline FLUX outputs, HPS-only optimization, and joint HPS and CLIP optimization.
Loading 2601.15968v2…