PaperlayerView exact revision on arXiv

Source-linked AI summary

ViPO: Visual Preference Optimization at Scale

Ming Li, Jie Wu, Justin Cui, Xiaojie Li, Rui Wang, Chen Chen

arXiv:2604.24953v2cs.CVcs.AI

TL;DR

Scaling visual preference optimization is hindered by conflicting signals and data-quality bottlenecks. The paper introduces Poly-DPO and the ViPO dataset, showing improved learning on noisy data, state-of-the-art results on ViPO, and convergence to standard DPO on high-quality data.

  • Problem

    Conflicting preferences and limited resolution, prompt diversity, and balance in existing datasets constrain evidence for scaling visual preference optimization.

  • Method

    The paper introduces Poly-DPO for confidence-based sample weighting and constructs ViPO with 1M image pairs and 300K video pairs.

  • Results

    Poly-DPO improves performance on noisy datasets, ViPO-trained models achieve state-of-the-art results, and optimization converges to standard DPO on ViPO.

  • Takeaways & Limitations

    Effective scaling depends on both adaptive optimization for noisy data and high-quality preference data, with sophisticated optimization unnecessary when data quality is sufficient.

  • Takeaways & Limitations

    ViPO relies on generative models and VLMs for automated preference-data generation, filtering, and labeling rather than collecting human preferences directly.

Abstract

from arXiv · show

While preference optimization is crucial for improving visual generative models, how to effectively scale this paradigm remains largely unexplored. Current open-source preference datasets contain conflicting preference patterns, where winners excel in some dimensions but underperform in others. Naively optimizing on such noisy datasets fails to learn preferences, hindering effective scaling. To enhance robustness against noise, we propose Poly-DPO, which extends the DPO objective with an additional polynomial term that dynamically adjusts model confidence based on dataset characteristics, enabling effective learning across diverse data distributions. Beyond biased patterns, existing datasets suffer from low resolution, limited prompt diversity, and imbalanced distributions. To facilitate large-scale visual preference optimization by tackling data bottlenecks, we construct ViPO, a massive-scale preference dataset with 1M image pairs at 1024px across five categories and 300K video pairs at 720p+ across three categories. State-of-the-art generative models and diverse prompts ensure reliable preference signals with balanced distributions. Remarkably, when applying Poly-DPO to our high-quality dataset, the optimal configuration converges to standard DPO. This convergence validates dataset quality and Poly-DPO's adaptive nature: sophisticated optimization becomes unnecessary with sufficient data quality, yet remains valuable for imperfect datasets. We validate our approach across visual generation models. On noisy datasets like Pick-a-Pic V2, Poly-DPO achieves 6.87 and 2.32 gains over Diffusion-DPO on GenEval for SD1.5 and SDXL, respectively. For ViPO, models achieve performance far exceeding those trained on existing open-source preference datasets. These results confirm that addressing both algorithmic adaptability and data quality is essential for scaling visual preference optimization.

1 INTRODUCTION

The introduction identifies conflicting and biased preference signals, alongside low resolution, limited prompt diversity, and imbalanced distributions, as key obstacles to scaling visual preference optimization. It proposes Poly-DPO and the large-scale ViPO dataset, whose complementary validation shows adaptive optimization helps imperfect data while high-quality data can make sophisticated optimization unnecessary.

  • Problem: Existing preference datasets also suffer from low visual resolution, limited prompt diversity, imbalanced distributions, and outdated generation models.These limitations arise alongside biased or overly simple preference samples.
  • Method: Poly-DPO extends Diffusion-DPO with a polynomial term that dynamically adjusts sample weighting from prediction confidence.The method emphasizes informative samples amid conflicting preferences and avoids over-confidence on trivially distinguishable preferences.
  • Problem: Conflicting preference patterns cause standard Diffusion-DPO to miss genuine signals and reach performance saturation despite increased data scale.Winner images may excel in aesthetics but underperform in text-image alignment, creating contradictory training signals.
  • Dataset: ViPO provides 1M high-resolution image pairs and 300K video pairs with reliable, balanced preference signals constructed using state-of-the-art models and systematic categorization.The dataset is designed as a large-scale benchmark for preference learning.
  • Validation: Poly-DPO excels on biased datasets but converges to standard DPO (α→0) on ViPO-Image-1M, where sophisticated optimization becomes unnecessary.The convergence is robust across neighboring α values and supports mutual validation of the algorithm and dataset quality.

2 RELATED WORKS

Diffusion-based visual generation has advanced substantially, but aligning outputs with complex prompts and multiple quality dimensions remains challenging. Related work applies RLHF paradigms to visual generation, while scalability limitations persist for on-policy approaches.

  • Diffusion-based Visual Generation: Diffusion-based visual generation has achieved remarkable progress through foundational models and subsequent scaling.The passage cites pioneering diffusion models, scaling efforts, and advanced image and video generators.
  • Diffusion-based Visual Generation: Two key challenges are aligning outputs with complex user prompts and optimizing multiple quality dimensions simultaneously.
  • Reinforcement Learning from Human Feedback (RLHF): RLHF approaches divide into on-policy methods requiring iterative sampling and reward-model evaluation, and off-policy methods such as DPO.
  • Reinforcement Learning for Visual Generation: Recent visual-generation research extends RL through ReFL-based reward maximization and PPO-based diffusion-as-MDP methods.
  • Reinforcement Learning for Visual Generation: On-policy visual-generation methods face scalability constraints from computational intensity and reward hacking.

3 DIFFUSION PREFERENCE OPTIMIZATION WITH POLY-DPO

Poly-DPO reformulates Diffusion-DPO as binary preference classification and adds a confidence-aware polynomial term. Its α parameter reweights uncertain or high-confidence samples according to dataset characteristics, while α = 0 recovers standard DPO for balanced, high-quality preferences.

  • Diffusion-DPO as the Binary Classification Task: Poly-DPO reinterprets Diffusion-DPO as binary classification, where the model learns to maximize the probability of correctly ranking preferred generations.The preference probability measures the model’s relative preference for the winner over the loser compared with the reference model.
  • Polynomial Expansion of Preference Learning: Poly-DPO adds a perturbation term to the polynomial expansion, ultimately modifying only the first gradient-dominant term.The general Poly-N loss perturbs the Top-N contributing polynomials, whereas Poly-DPO uses a simpler single-term modification.
  • Confidence-Aware Optimization: α > 0 upweights uncertain samples and downweights extreme cases, helping models learn consistent signals from conflicting preference patterns.This focuses learning on borderline cases where consistent improvement is possible.
  • Confidence-Aware Optimization: α < 0 reduces contributions from high-confidence samples in trivially distinguishable datasets, preventing over-fitting and encouraging continued exploration of winner-loser differences.The paper gives synthetic data with shuffled losers as an example of trivially distinguishable preferences.
  • Confidence-Aware Optimization: α = 0 recovers standard DPO when preference signals are balanced and high quality, with strong robustness to α selection.Poly-DPO is described as adding a single confidence-aware term that dynamically reweights samples across models and preference datasets.

4 LARGE-SCALE VISUAL PREFERENCE DATASET CONSTRUCTION

ViPO addresses key bottlenecks in existing preference datasets by providing balanced, high-dimensional image and video preferences at large scale. An open-source alternative replaces proprietary model outputs with publicly available generators for reproducibility and downstream use.

  • Motivation and Design Principles: Existing datasets impede scaling through low resolution, limited prompt diversity, unreliable early-generation signals, and imbalanced random sampling.These limitations restrict fine-grained learning and underrepresent critical visual aspects.
  • ViPO-Image-1M: ViPO-Image-1M contains five dimensions with 200K image pairs each: aesthetics, text-image alignment, text rendering, portrait quality, and composition.The construction uses publicly available HuggingFace prompts and state-of-the-art generators.
  • ViPO-Video-300K: ViPO-Video-300K contains three dimensions with 100K video pairs each: motion quality, video-text alignment, and visual quality.Diverse I2V, T2V, and T2I2V generation strategies create varied preference patterns.
  • Open-Source Datasets: The open-source ViPO version replaces proprietary outputs with FLUX.2-dev for images and Wan2.2-A14B-I2V for videos.This enables full reproducibility and broader downstream applications such as diffusion distillation.

5 EXPERIMENTS

Experiments evaluate Poly-DPO across image and video models, preference datasets, and multidimensional benchmarks. ViPO training improves composition, alignment, aesthetics, text rendering, motion understanding, and human-centric quality, supported by reliable annotations.

  • Experimental Setup: Experiments cover SD1.5, SDXL, SD3, FLUX, and Wan2.1-T2V-1.3B, training image models on ViPO-Image-1M and the video model on ViPO-Video-300K.SD1.5 is additionally trained on PickaPic-v2 for comparison and preference-noise resilience testing.
  • Evaluation Protocol: Evaluation combines CLIP-based rewards and established test sets for SD1.5 with aesthetics, alignment, and text-rendering benchmarks for high-resolution models.The SD1.5 protocol uses ImageReward, HPSv2.1, and Aesthetic Predictor, while high-resolution evaluation includes DeQA, DPG-Bench, and CVTG-2K.
  • Preference-Dataset Ablations: 20.79% of Pick-a-Pic V2 pairs have consistent rankings across five reward dimensions, motivating Poly-DPO’s adaptive weighting for conflicting preference signals.The experiments identify α=8 as the best configuration in this noisy setting.
  • Preference-Dataset Ablations: On the high-quality, balanced ViPO-Image-1M dataset, the optimal α converges to approximately zero, making Poly-DPO equivalent to standard DPO with robust performance.This contrasts with the positive and negative α settings used for noisy and imbalanced datasets.
  • Image Generation Results: SD1.5 trained on ViPO-Image-1M improves GenEval overall from 0.42 to 0.52 (+23.8%), while SDXL reaches 0.63 and improves attribute binding from 0.16 to 0.42.SD1.5 also gains strongly on two-object generation, increasing from 0.38 to 0.66.
  • Image and Video Quality Results: ViPO training raises SD3.5-Medium human accuracy from 73.25% to 85.25% and word accuracy from 0.4378 to 0.6995, while video metrics improve across motion and human-centric criteria.For video, Dynamic Spatial Relationship increases from 24.64 to 33.82 (+37.4%), and Human Identity rises from 62.18 to 67.99.

6 CONCLUSION · A OVERVIEW OF APPENDIX · B DATASET CONSTRUCTION DETAILS

The paper identifies dataset conflict as a barrier to scaling visual preference optimization and introduces Poly-DPO and ViPO to improve robustness and data quality. The appendix organizes additional material on dataset construction, experiments, implementation, open-source datasets, limitations, and LLM use.

  • 6 CONCLUSION: Poly-DPO dynamically adjusts sample weighting based on confidence levels, enabling learning across diverse data characteristics.The method addresses conflicting preference patterns that limit scaling in existing datasets.
  • 6 CONCLUSION: ViPO provides 1M image pairs and 300K video pairs with reliable preference signals across multiple quality dimensions.The dataset is constructed at large scale to address visual preference optimization data bottlenecks.
  • 6 CONCLUSION: 81.2% VLM accuracy exceeds the 74.7% average individual-human accuracy against consensus labels; image accuracy is 84.0% versus 74.9%.On video tasks, the VLM performs comparably to humans at 71.7%, as reported in the figure passage.
  • A OVERVIEW OF APPENDIX: The appendix includes Section B on dataset construction details.This section is listed as part of the appendix organization.
  • A OVERVIEW OF APPENDIX: The appendix covers more experiments and analysis, implementation details, and results and details of open-source datasets.These topics correspond to Sections C, D, and E.
  • A OVERVIEW OF APPENDIX: The appendix also includes discussion, limitation and future work, plus the use of large language models.These topics correspond to Sections F and G.

B.1 VIPO-IMAGE-1M DATASET · ViPO-Video-300K · B.2 VIPO-VIDEO-300K

ViPO constructs image preference pairs through task-specific generation, perturbation, and automated or model-based judging, then extends the dataset to video preferences across motion, visual, and video-text alignment. The resulting pipelines emphasize controlled comparisons, semantic consistency, and high-resolution preference labeling.

  • B.1 VIPO-IMAGE-1M DATASET: Image-text alignment pairs use one generation model with distinct prompts, treating the original-caption image as the winner and the perturbed-caption image as the loser.Seedream-3.0 generates all 200K pairs for this task.
  • B.1 VIPO-IMAGE-1M DATASET: Text-rendering data combines 208K CoverBook prompts, 100K movie-poster prompts, and LAION-Aesthetics samples, yielding 200K filtered prompts.Pairs are generated exclusively with Qwen-Image, HiDream-I1, Seedream-3.0, and FLUX.1-dev.
  • B.1 VIPO-IMAGE-1M DATASET: Text-rendering preferences use PaddleOCR-3.0 for initial assessment and Seed-VL-1.5 when both images succeed or both fail.The first stage automatically labels the image with correct text and the other with character-level errors as the winner.
  • ViPO-Video-300K: Image-composition pairs draw on 200K unique prompts from two HuggingFace datasets and compare Seedream-3.0 outputs with images generated by alternative models or sourced from MidJourney-V7.The construction uses shared prompts to form paired image comparisons.
  • ViPO-Video-300K: Aesthetics pairs sample 200K non-overlapping prompts and images, generate Seedream-3.0 counterparts, and use Qwen2.5-VL-32B-Instruct, Seed-VL-1.5, and Q-Insight as judges.The three VLMs establish preference labels based on aesthetics.
  • B.2 VIPO-VIDEO-300K: Motion-quality video pairs use an image-to-video pipeline to preserve spatial consistency between paired videos while integrating samples from four HuggingFace datasets.The process includes 6,763 videos and prompts from WenhaoWang/ShareVeo3 and extracts each video's first frame.
  • B.2 VIPO-VIDEO-300K: Video-quality and video-text alignment subsets reuse ViPO-Image-1M pairs with shared motion prompts, targeting visual quality plus spatial and temporal alignment.The video-text subset selects 50K image-text alignment pairs, while the visual-quality subset selects 100K pairs with unanimous VLM labels.

C MORE EXPERIMENTS AND ANALYSIS

The experiments show that ViPO benefits from winner-only SFT followed by Poly-DPO, achieves strong preference-alignment and text-rendering results, and remains stable during training. Ablations further identify broad data coverage and longer SFT as beneficial, while Poly-DPO outperforms baselines on DPG-Bench.

  • Text Rendering: 0.4378 to 0.6995 (+59.8%) is SD3.5-Medium’s average word-accuracy improvement on CVTG-2K, with NED reaching 0.8853.FLUX.1-dev also improves from 0.4878 to 0.6859 despite SFT alone causing degradation.
  • SFT and Poly-DPO Ablation: Winner-only SFT followed by Poly-DPO is the best SD1.5 integration strategy, outperforming SFT on full winner-loser pairs and direct Poly-DPO.The paper uses this winner-only SFT configuration for its main experiments.
  • DPG-Bench: 67.02 and 75.67 are Poly-DPO’s highest Overall scores on DPG-Bench for SD1.5 and SDXL, respectively, surpassing existing baselines.These experiments use Pick-a-Pic V2 as the training dataset.
  • ViPO Sub-datasets: SFT on All Datasets followed by DPO on ViPO sub-datasets consistently outperforms the base model and SFT-only models across evaluation metrics.All Datasets DPO reaches 85.25 for GPT-4o Accuracy and 0.6995 for CVTG-2K.
  • SFT Training Steps: 73.25 to 77.50 is the increase in GPT-4o Accuracy as SFT training steps extend from 1,000 to 4,000, while Alignment saturates or fluctuates after early steps.Longer SFT also continuously improves Text Rendering and Human Quality until the reported peak.
  • Training Stability: Both Diffusion-DPO and Poly-DPO show high training stability on Pick-a-Pic V2, with PickScore, ImageReward, Aesthetic Score, and HPSv2 increasing toward convergence without model collapse.The comparison tracks four evaluation metrics throughout training.

D IMPLEMENTATION DETAILS

The paper uses distinct training protocols for Pick-a-Pic V2, ViPO-Image-1M, and ViPO-Video-300K. These protocols specify stage structure, optimization settings, and resolution handling for each dataset and model.

  • Training on Pick-a-pic V2 Dataset: Pick-a-Pic V2 training follows Diffusion-DPO’s official implementation, using batch size 512, base learning rate 4e-9, 512×512 resolution, and 4,500 steps.The final learning rate is 512×4e−9=2.048e−6, and α is selected by grid search.
  • Training on Pick-a-pic V2 Dataset: The Pick-a-Pic V2 experiments keep the reference model fixed and do not use Exponential Moving Average (EMA).These choices apply throughout the 4,500-step training process.
  • Training on ViPO-Image-1M Dataset: ViPO-Image-1M experiments use two-stage training: winner-only SFT followed by Poly-DPO, with both models initialized from the SFT-tuned checkpoint.This protocol is adopted for all models after initial SD1.5 validation.
  • Training on ViPO-Image-1M Dataset: SD1.5 uses batch size 512, learning rates 4e-9 and 1e-9, 8,000 steps per stage, and β=500 for SFT and Poly-DPO, respectively.The smaller β was selected after observing that it performed better following initial SFT.
  • Training on ViPO-Video-300K Dataset: ViPO-Video-300K applies Poly-DPO directly to Wan2.1-T2V-1.3B for 2,000 steps with batch size 256, base learning rate 1e-8, β=500, and α=0.Training uses dynamic resolution without resizing the dataset videos.

E OPEN-SOURCE DATASET CONSTRUCTION · E.1 MOTIVATION · E.2 ANNOTATION PIPELINE

The released ViPO dataset replaces proprietary-model outputs that cannot be distributed while preserving most data generated from open-source models and public prompts. A unified three-VLM majority-vote pipeline provides preference labels for both image and video pairs using category-specific evaluation criteria.

  • E.1 MOTIVATION: Licensing and intellectual-property constraints prevent release of data generated by Seedream-3.0 and Seedance-1.0.These proprietary models were used for image and video generation, respectively.
  • E.1 MOTIVATION: The open-source release is motivated by the need for openly accessible, high-quality visual preference data.The authors identify open access as essential for advancing visual preference research.
  • E.1 MOTIVATION: Most released ViPO data remains unchanged because it was generated with open-source models and public HuggingFace prompts.Examples include FLUX.1-dev, HiDream-I1, Qwen-Image, and WanVideo.
  • E.1 MOTIVATION: Only subsets originally generated by Seedream-3.0 and Seedance-1.0 are substituted in the released dataset.The passage identifies these as the image and video subsets, respectively.
  • E.2 ANNOTATION PIPELINE: A unified multi-VLM voting pipeline annotates both image and video preference pairs for consistent and reliable labels.The pipeline is applied across all annotations in the released dataset.
  • E.2 ANNOTATION PIPELINE: Each candidate pair is independently judged by Qwen3-VL-30B-A3B-Thinking, Molmo2-8B, and Seed-1.8.The judges independently select a preferred sample from each pair.
  • E.2 ANNOTATION PIPELINE: The final preference label is determined by majority vote among the three VLM judges.This voting strategy is intended to mitigate annotation inconsistency across preference pairs.
  • E.2 ANNOTATION PIPELINE: Category-specific instructions guide VLM evaluation of aesthetic quality, text-image alignment, and motion quality.The criteria are tailored to the category being assessed.

E.3 EXPERIMENTAL VALIDATION

Experiments on the released ViPO-Image-1M-Open reproduce the paper’s two-stage SFT-plus-Poly-DPO training across four visual generation models. Results preserve the original conclusions: winner-image SFT improves performance, Poly-DPO adds gains, and benchmark trends remain consistent.

  • Experimental setup: SD1.5, SDXL, SD3.5-Medium, and FLUX.1-dev are trained on ViPO-Image-1M-Open using the released dataset’s SFT-then-Poly-DPO pipeline.This re-runs the paper’s main experiments to verify the released dataset’s quality and effectiveness.
  • Experimental findings: SFT on winner images consistently improves base-model performance across all architectures and evaluation dimensions.The released-dataset experiments confirm this conclusion across the reported evaluations.
  • Experimental findings: Poly-DPO further yields significant gains on top of SFT, demonstrating effective preference optimization.This finding holds with the released dataset and supports the paper’s core optimization claim.
  • Experimental findings: Overall performance trends across benchmarks remain consistent with those reported using the original dataset.The released dataset therefore preserves the paper’s core experimental conclusions.

F DISCUSSION, LIMITATION AND FUTURE WORK. … REPRODUCIBILITY STATEMENT

The paper’s discussion links Poly-DPO’s value to dataset quality, while its limitations concern AI-generated labels, α selection, and costly dataset construction. Future work targets human validation, adaptive optimization, finer preference control, and broader curation, alongside statements on LLM use, ethics, and reproducibility.

  • F DISCUSSION, LIMITATION AND FUTURE WORK.: Poly-DPO is most valuable on noisy preference data, whereas high-quality ViPO reduces the need for sophisticated optimization.The authors describe algorithmic design and data quality as symbiotic contributions.
  • F DISCUSSION, LIMITATION AND FUTURE WORK.: An optimal α near zero indicates a balanced, reliable dataset where standard optimization is sufficient.Large positive α may indicate noise or conflicting signals, while negative α may indicate trivially simple patterns and model overconfidence.
  • F DISCUSSION, LIMITATION AND FUTURE WORK.: ViPO demonstrates scalable AI-driven curation using state-of-the-art generators and VLMs for automated filtering, generation, and labeling.This pipeline largely avoids the cost and scalability challenges of directly collecting human preferences.
  • F DISCUSSION, LIMITATION AND FUTURE WORK.: ViPO’s preference labels are AI-generated proxies rather than direct measurements of human preferences, without large-scale human correlation testing.Multiple state-of-the-art VLMs were used to improve robustness and consistency, but their inherent limitations remain.
  • F DISCUSSION, LIMITATION AND FUTURE WORK.: Poly-DPO’s optimal α currently requires computationally intensive grid search and may depend on noise, model architecture, or content domain.The authors identify automated or dynamic α selection as a practical improvement.
  • F DISCUSSION, LIMITATION AND FUTURE WORK.: The ViPO dataset required generation from over a dozen state-of-the-art models, making initial construction costly for smaller research groups.The public release helps democratize access to high-quality preference data, but creation remains resource-intensive.
  • F DISCUSSION, LIMITATION AND FUTURE WORK.: Future work includes large-scale human validation, improved pseudo-labeling and reward models, learnable α, and fine-grained modeling of quality trade-offs.The proposed directions also include extending AI-driven curation to new modalities such as 3D.
  • G THE USE OF LARGE LANGUAGE MODELS (LLMS): LLMs such as GPT and Gemini were used only for language polishing, not for core ideas, data, or experimental results.The authors state that the technical content and analyses were produced by the authors.
Loading 2604.24953v2…