PaperlayerView exact revision on arXiv

Source-linked AI summary

MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency

Nicolas Dufour, Lucas Degeorge, Arijit Ghosh, Vicky Kalogeiton, David Picard

arXiv:2510.25897v2cs.CVcs.LG

TL;DR

Post-training text-to-image pipelines can discard informative data and optimize a single reward, with reported costs to diversity, semantic fidelity, and efficiency. MIRO conditions pretraining on multiple rewards, and the paper reports stronger benchmark performance and faster convergence with substantially less compute than FLUX-dev.

  • Problem

    The common pipeline filters training images and aligns generators to a single reward, a setup reported to harm diversity, semantic fidelity, and efficiency.

    Source: §1 · pp. 1–2

  • Method

    MIRO annotates training images with multiple reward scores and conditions the generator on their reward vector during pretraining.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    A 0.36B-parameter MIRO surpasses 12B FLUX-dev on GenEval and PartiPrompts at 370× less training compute, with up to 19× faster convergence and 3× less inference compute under sample-based scaling.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    Reward conditioning preserves training signals across quality levels and lets users control reward trade-offs at inference.

    Source: §1 · p. 2

Abstract

from arXiv · show

The default paradigm of post-training text-to-image generators includes post-hoc selection of generated images, and subsequent training with one reward model to align the generator to the reward, typically user preference. This discards informative data as well as optimizes only for a single reward, hence harming diversity, semantic fidelity and efficiency. Instead, we propose MIRO, a method that conditions the model on multiple rewards during training, thus letting the model learn user preferences directly. MIRO pre-training both improves the visual quality of the generated images and speeds up the training, achieving state of the art on the GenEval compositional benchmark and user-preference scores (PickAScore, ImageReward, HPSv2).

1. Introduction

Text-to-image training separates web-scale pretraining, curated finetuning, and reward-based alignment, with each stage narrowing the data or objectives available to the model. MIRO instead conditions pretraining on multiple rewards, reporting faster convergence and stronger benchmark performance with less compute.

1. Introduction

  • The standard pipeline uses web-data pretraining, curated supervised finetuning, and RLHF to target data distribution, visual quality, and user preference, respectively.The paper describes this as the prevailing three-stage approach.Source: §1 · p. 1
  • Finetuning discards informative low-quality data, while single-reward RLHF can harm diversity and semantic fidelity and fixes the reward trade-off at training time.The paper says these stages successively contract the distribution shaped by the previous stage.Source: §1 · p. 2
  • MIRO annotates training images with seven complementary rewards and conditions generation on their reward vector, without filtering or a separate RLHF stage.The reward dimensions include aesthetics, user preference, text-image alignment, visual reasoning, and scientific correctness.Source: §1 · p. 2
  • Reward vectors preserve signals across the data’s quality levels, let users adjust rewards at inference, and provide dense supervision that accelerates convergence.A multi-reward extension of classifier-free guidance steers generation toward jointly high-reward regions.Source: §1 · p. 2
  • A 0.36B-parameter MIRO surpasses 12B FLUX-dev on GenEval and PartiPrompts at 370× less training compute, with up to 19× faster convergence and 3× less inference compute under sample-based scaling.The paper also reports state-of-the-art results on GenEval and user-preference benchmarks.Source: §1 · p. 2

2. Method

MIRO integrates multiple reward scores into pretraining and uses reward conditioning for both training and inference. Its design retains samples across quality levels while enabling users to select reward trade-offs at generation time.

2. Method

  • MIRO conditions image generation on multiple reward scores during pretraining, preserving the spectrum of quality levels while enabling fine-grained inference-time control.This integrates alignment into training rather than relying on separate alignment stages.Source: §2 · p. 2

2.1. Dataset Augmentation with Reward Scores

  • Reward annotations augment each image-caption pair with scores from multiple reward models, and uniform binning balances representation across quality levels.The resulting training data contains a multidimensional quality assessment for each sample.Source: §2.1 · pp. 2–3

2.2. Multi-Reward Conditioned Flow Matching

  • Flow matching trains the model to predict the difference between noise and the clean image conditioned on both the prompt and desired reward levels.Training exposes the model to low- through high-quality samples and their associated visual features.Source: §2.2 · p. 3

2.3. Inference with Reward-Guided Sampling

  • Users can set custom reward targets or condition on maximum reward values, while multi-reward classifier-free guidance contrasts positive and negative reward targets.The default guidance targets use maximum and minimum rewards; under conditional independence, the log-odds decomposes over rewards and steers toward the Pareto frontier.Source: §2.3 · pp. 3–4

2.4. Advantages of MIRO over traditional alignment approaches

  • Integrating alignment into pretraining removes separate finetuning or reinforcement-learning stages and their associated multi-stage hyperparameter tuning.The paper also states that MIRO reaches reward-aligned behavior with faster convergence than regular pretraining.Source: §2.4 · p. 4
  • Training across low-, medium-, and high-reward regions models the full data distribution and lets users choose a point on the diversity-quality spectrum at inference.The stated theoretical properties include full-spectrum coverage and preservation of marginal entropy.Source: §2.4 · p. 4
  • MIRO’s multidimensional conditioning is described as preventing reward hacking by requiring a balance among objectives, while reward settings expose trade-offs to users.Users can adjust individual reward levels and observe the resulting trade-offs.Source: §2.4 · p. 4

3. Experiments

Across six experiments, MIRO improves reward quality and compositional alignment, accelerates training, and supports inference-time reward control. It also improves sample efficiency and performance with synthetic captions.

3.1. Reward-Conditioned Pretraining Improves Model Quality

  • 19× faster AestheticScore convergence, 6.2× for HPSv2, 3.5× for PickScore, and 3.3× for ImageReward accompany dense reward supervision.Qualitative examples also show MIRO reaching a tiger-in-a-tuxedo quality level by 50k steps versus 200k for the baseline.Source: §3.1 · p. 5

3.2. Improving Text-Image Alignment

  • MIRO scores 57 on GenEval versus the baseline’s 52, with gains in Color Attribution, Two Objects, and Counting; SciScore alone reaches 58 but reduces aesthetic quality.MIRO also outperforms single-reward approaches across aesthetic and preference metrics, while multi-reward conditioning mitigates reward hacking.Source: §3.2 · pp. 5–6, §3.1 · p. 5, Table 1 · p. 6

3.5. Comparison to State-of-the-Art Models

  • 68 GenEval versus FLUX-dev’s 67, with 4.16 TFLOPs versus 1540 TFLOPs, gives MIRO a 370× training-compute advantage.The comparison is against a 12B-parameter FLUX-dev model.Source: Table 1 · p. 6, §3.5 · p. 7

3.4. Synergizing with Test-Time Scaling

  • For ImageReward, 8 MIRO samples match the baseline’s 128; for PickScore, 4 match 128, while MIRO also beats the baseline across reward metrics.For Aesthetic Score and HPSv2, one MIRO sample reaches performance the baseline cannot reach with 128 samples.Source: Figure 6 · p. 7, §3.4 · p. 7

3.3. MIRO and Synthetic Captions

  • 68 GenEval with synthetic captions exceeds the synthetic-caption baseline’s 57 (+19%) while maintaining equivalent aesthetic quality to MIRO without synthetic captions.MIRO without synthetic captions also outperforms the synthetic-caption baseline across reward metrics and is more computationally efficient than recaptioning with large vision-language models.Source: §3.3 · pp. 6–7

3.5. Comparison to State-of-the-Art Models

  • 46 on Position beats the previous state of the art of 34 by 31%, and 52 on Color Attribution exceeds FLUX-dev’s prior best of 47 by 11%.On PartiPrompts, MIRO reaches 6.81 Aesthetic Score versus FLUX-dev’s 6.56 and 1.61 ImageReward versus Sana-1.6B’s 1.23.Source: §3.5 · p. 7
  • When optimizing for HPSv2, MIRO reaches an ImageReward score of 1.35, outperforming models specifically trained for that metric.MIRO maintains a 3× efficiency advantage over FLUX-dev at 128-sample scaling (532 TFLOPs versus 1540 TFLOPs).Source: §3.5 · pp. 7–8

3.6. Flexible Reward Trade-offs at Inference

  • Aesthetic selection can reduce GenEval, while an aesthetic weight of 0.625 gives the highest GenEval; weighted selection matches 128-sample ImageReward selection with one selection.Pairwise reward interpolation varies two objectives while holding the others high, enabling exploration of trade-offs between specific quality dimensions.Source: §3.6 · p. 8

4. Conclusion

The conclusion presents MIRO as reward-vector-conditioned pretraining that integrates alignment into training. On a 16M-image setup, it reports faster convergence, stronger compositional alignment, and higher performance than FLUX-dev on GenEval and PartiPrompts at a fraction of the compute.

4. Conclusion

  • MIRO conditions pretraining on a vector of reward scores, integrating alignment into training rather than adding it as a post-hoc stage.The model learns p(x | c, s), with reward targets as controllable inputs.Source: §4 · p. 8
  • On a 16M-image setup, MIRO outperforms no-conditioning and single-reward baselines, converges faster, mitigates reward hacking, and strengthens compositional alignment.Source: §4 · p. 8
  • Despite its smaller size, MIRO surpasses FLUX-dev on GenEval and PartiPrompts at a fraction of the compute.Source: §4 · p. 8

Supplementary Material Contents

The supplementary material collects related work, implementation and captioning details, additional experiments, theoretical analyses, and reproducibility and ethics statements.

A. Related Work

G. Log-Odds Maximization Theory

  • Appendix G develops MIRO’s guidance theory, including a result that it samples from a reward-tilted distribution and performs implicit gradient ascent on log-odds.Source: G. Log-Odds Maximization Theory · p. 13

A.1. Diffusion and Flow-Based Generative Models

  • The diffusion and flow-model review covers foundational generative methods, transformer architecture and training innovations, and autoregressive alternatives.Source: §A.1 · p. 13

A.2. Conditioning and Controllable Generation

  • The conditioning review covers classifier and classifier-free guidance, text and spatial conditioning, and quality-aware conditioning; MIRO extends the latter to multiple reward dimensions.Source: §A.2 · pp. 13–14

A.3. Data Efficiency and Training Strategies

The related-work discussion situates MIRO among data-efficiency and reward-alignment approaches. It emphasizes conditioning on reward vectors during pretraining rather than filtering data or adding a separate alignment stage.

A.3. Data Efficiency and Training Strategies

  • Data curation can improve metrics while discarding potentially useful signal; MIRO instead conditions on reward scores to learn from the full data spectrum with quality control at inference.The discussion also reviews public datasets, latent-space efficiency, representation alignment, and synthetic captioning.Source: §A.3 · p. 14

A.4. Reward-Based Alignment

  • The reward-alignment review describes complementary and sometimes conflicting reward models, alongside RL, gradient-based fine-tuning, preference learning, multi-reward methods, and test-time scaling.Source: §A.4 · pp. 14–15
  • MIRO differs by learning the full reward-conditioned distribution from scratch during pretraining, whereas prior reward-based alignment methods operate on pretrained models.The passage identifies RiC as conceptually similar but applied through fine-tuning on language models.Source: §A.4 · p. 15
  • Reward-vector conditioning is presented as avoiding a separate alignment stage, learning reward levels across the data spectrum, supporting stable training, and enabling inference-time trade-offs without retraining.The discussion connects MIRO to coherence-aware training and the upside-down alignment paradigm.Source: §A.4 · p. 15

B. Implementation Details

MIRO adds binned reward signals to the flow-matching model and uses reward scores to annotate the training data. The setup combines seven reward models with a 16M-pair dataset.

Architecture and reward conditioning

  • The flow-matching network takes the noisy sample, text condition, and binned reward vector as input, with reward embeddings concatenated to text tokens.Each reward bin index is sinusoidally embedded and projected into the text-token space.Source: B. Implementation Details · p. 15

Reward scoring and binning

  • Reward preprocessing calibrates each model on a representative subset, divides scores into equal-population bins, and maps each training score to a bin index.Binning normalizes reward scales, balances training across quality levels, and makes higher indices correspond to better quality.Source: B. Implementation Details · p. 16

Experimental setup

  • MIRO training uses seven reward models on 16M image-text pairs from CC12M and LAION Aesthetics 6+, with flow matching at 256px resolution.The listed reward models cover aesthetics, preference, text-image correspondence, visual comprehension, long-caption CLIP scoring, and scientific accuracy.Source: B. Implementation Details · pp. 16–17

Baselines

C. Captioning Pipeline

The captioning pipeline prepares long descriptive captions and short captions for reward scoring, while preprocessing costs are dominated by captioning. The reported comparisons examine guidance-rate behavior, single-reward GenEval performance, and aesthetic-weight trade-offs.

C. Captioning Pipeline

  • Long captions are prompted to describe image details and relationships for reconstruction, while short captions are generated separately for reward scoring.The long-caption pipeline uses google/gemma-3-12b-it; the short-caption prompt requests a concise caption enclosed in caption tags.Source: C. Captioning Pipeline · pp. 18–19

D. Preprocessing Computational Costs

E.1. CFG Rate Analysis

  • MIRO scores highest on Image Reward at 1.05 with CFG 5, exceeding the ImageReward-specialized model's 1.04; reported optimal CFG rates are around 5–7.The aesthetic-only model reaches 6.65 on Aesthetic Score at CFG 7 but underperforms on CLIP-based metrics.Source: §E.1 · p. 20, Figure 11 · p. 21

E.2. Single-Reward Comparison on GenEval

  • 67.7 vs. 19.0 on Position and 55.3 on Counting: MIRO's GenEval profile is balanced across categories, while the aesthetic-only model is weak on compositional metrics.MIRO records the highest Two Objects score (67.7) and a Color Attribution score of 37.5; VQA and SciScore lead Single Object, at 97.2 and 94.4.Source: §E.2 · p. 20, Figure 12 · p. 21

E.3. Aesthetic Weight Sensitivity

  • 74.9 GenEval is reached at aesthetic weight 0.625, while weight 1.0 yields 67.7; Aesthetic Score rises from 3.7 to 6.3 as weight increases.Text-alignment metrics peak at intermediate weights, including OpenAI CLIP at 0.375.Source: §E.3 · pp. 21–22, Figure 13 · p. 22, Figure 14 · p. 22

E.4. Training Dynamics

MIRO's training curves show fast early gains and strong performance across reward metrics, while leave-one-out results reveal trade-offs between aesthetic and compositional scores.

E.4. Training Dynamics

  • 1.05 vs. 0.54 on Image Reward: MIRO exceeds the baseline, with near- or above-baseline performance across reward metrics and competitive scores within the first 100k steps.The reported early point is approximately 20% of total training; the Aesthetic specialist reaches 6.65 Aesthetic Score and the HPSv2 specialist reaches 0.295 HPSv2 Score.Source: §E.4 · pp. 22–23, Figure 15 · p. 23

E.5. Leave-one-out Experiments

  • Removing Aesthetic Score lowers its metric from 6.23 to 5.05 but improves GenEval Position by 11.8 points and Two Objects by 11.4 points.Removing ImageReward or CLIP tends to hurt semantic metrics such as Two Object detection and Color accuracy.Source: §E.5 · p. 23

E.6. Binning Ablation

Quantile binning is the reported balance across reward and GenEval metrics; MIRO fine-tuning approaches the full model on some rewards but slightly trails its overall GenEval score.

E.6. Binning Ablation

  • 57 vs. 54 overall GenEval: Quantile Bins outperforms Refined Quantile, which has slightly higher reward scores such as Aesthetic 6.40 vs. 6.23.Uniform Bins reach Position 25.8 vs. 12.2 for Quantile Bins, but have lower reward scores overall; quantile bins also require no refinement-threshold tuning.Source: §E.6 · pp. 23–24, Figure 18 · p. 25

E.7. Post-training Ablation

  • 56 vs. 57 overall GenEval: post-training MIRO slightly trails full MIRO, despite using MIRO for only 50k of 500k total steps.Aesthetic Score is 6.22 vs. 6.23; Position is 23.2 for post-training versus 12.2 for full MIRO.Source: §E.7 · p. 24, Figure 17 · p. 25

F. Additional Training Progression Examples

Qualitative training examples show MIRO developing recognizable objects and complex compositions earlier than the baseline. The theory section explains how reward conditioning implicitly supplies classification information used by MIRO’s guidance.

F. Additional Training Progression Examples

  • By 75k steps, MIRO produces recognizable objects while the baseline remains blurry; at 200k, MIRO has clear details and coherent composition while the baseline retains artifacts.The comparison is described for simple prompts across training progression.Source: F. Additional Training Progression… · p. 25
  • On complex prompts, MIRO progressively assembles multiple attributes and objects, with compositional ability evident by 100k steps for a prompt combining semantic content and artistic style.The examples include a beaver in a library and a raccoon in formal attire and cubist style.Source: F. Additional Training Progression… · pp. 25–26

G. Log-Odds Maximization Theory

  • MIRO’s guidance is presented as sampling from a reward-tilted distribution, with a complementary interpretation in terms of implicit gradient ascent on log-odds.The theory section’s roadmap covers the distributional result, optimization view, and extension to multiple rewards.Source: G. Log-Odds Maximization Theory · pp. 26–27

G.3. Multi-Reward Conditioning as Implicit Classification

  • Conditioning the velocity field on reward levels implicitly learns a classifier, so MIRO does not require the separate noisy-image classifier used by classical classifier guidance.The velocity difference between reward levels encodes the gradient of a log-probability ratio.Source: §G.3 · pp. 27–28

G.4. Main Result: Guidance as Sampling from a Reward-Tilted Distribution

MIRO’s guidance is characterized as sampling from a reward-tilted distribution and as implicit gradient ascent on reward log-odds. The multi-reward analysis treats correlated rewards jointly and shows how conditional independence and binning shape the guidance interpretation.

G.4. Main Result: Guidance as Sampling from a Reward-Tilted Distribution

  • The guided velocity field samples from a reward-tilted distribution that reweights the high-reward conditional by the high-versus-low reward log-odds raised to guidance scale ω.At ω = 0 the high-reward conditional is sampled without additional tilting; increasing ω favors higher log-odds.Source: §G.4 · pp. 28–29

G.5.1. DECOMPOSING THE GUIDED VELOCITY

  • Each guided denoising step combines a high-reward-conditioned base velocity with a correction that steers toward regions of higher reward contrast.The paper interprets the correction as implicit gradient ascent on the log-odds ratio.Source: §G.5.1 · p. 29, §G.5.2 · p. 30

G.5.3. TIME-VARYING STEP SIZE

  • The effective guidance step size grows as generation approaches clean samples, yielding an automatic schedule with minimal guidance near pure noise and maximal guidance late in generation.The effective step size is η_eff(t) = ω(1 − t)|∆t|.Source: §G.5.3 · p. 30

G.5.4. COMPARISON WITH EXPLICIT GRADIENT-BASED METHODS

  • MIRO uses two velocity-network forward passes instead of explicit reward-gradient computation, and the comparison reports no need for differentiable rewards or manual step-size tuning.The paper contrasts this with backpropagation through reward models and describes the additional memory cost as O(1).Source: §G.5.4 · p. 30

G.6.1. THE GENERAL CASE (NO INDEPENDENCE ASSUMPTION)

  • Without an independence assumption, the reward-tilted distribution uses joint probabilities, which naturally account for correlations among reward dimensions.The analysis distinguishes this general case from the conditional-independence case.Source: §G.6.1 · p. 31

G.6.2. FACTORIZATION UNDER CONDITIONAL INDEPENDENCE

  • Under conditional independence, the log-odds factor into a sum across rewards, exponentially favoring samples that score well across dimensions over samples extreme on one reward but poor on others.The independence condition may be approximate when reward models assess distinct aspects and the image provides enough information about each reward.Source: §G.6.2 · p. 31

G.6.4. EFFECT OF REWARD CORRELATIONS

  • For symmetric quantile binning, positive reward correlations do not change the log-odds ratio, making the independence-based analysis exact.The derivation states that symmetric binning gives z+ = −z−, so the quadratic correction vanishes.Source: §G.6.4 · pp. 32–33
  • With asymmetric binning under the Gaussian copula analysis, correlation increases the log-odds when the high-reward quantile is more extreme and decreases it when the low-reward quantile is more extreme.The paper notes that MIRO uses approximately symmetric binning in practice.Source: §G.6.4 · p. 33

G.6.5. ORTHOGONAL DECOMPOSITION ANALYSIS

The analysis characterizes MIRO’s multi-reward guidance through reward-space geometry and contrasts its single-objective training with DDPO’s reward–KL trade-off. It argues that DDPO gradients can conflict when high-reward samples are rare under the reference distribution, while MIRO has a unique optimal function in function space.

G.6.5. ORTHOGONAL DECOMPOSITION ANALYSIS

  • Under Gaussian assumptions, the reward covariance eigenbasis yields uncorrelated components and expresses the log-odds ratio as a sum over those components.The analysis uses a covariance eigendecomposition to define orthogonal coordinates.Source: §G.6.5 · pp. 33–34
  • Low-variance directions contribute more to log-odds discrimination per unit distance, while high-variance directions dominate sampling but discriminate less.When eigenvalues decay rapidly, guidance operates mainly in a lower-dimensional subspace; the effective number of independent rewards ranges from 1 to N.Source: §G.6.5 · p. 34

G.7. Summary and Key Takeaways

  • MIRO’s reward-conditioned velocity difference implicitly encodes a classifier over reward levels, and its guidance samples from a reward-tilted distribution controlled by ω.The analysis also interprets denoising as implicit gradient ascent on log-odds with a time-varying step size.Source: §G.7 · p. 35
  • Under conditional independence, the log-odds decompose across individual rewards, favoring samples that score well across dimensions.The theoretical summary also states that positive reward correlations make the high-reward region more coherent and easier to sample from.Source: §G.7 · p. 35

H. Comparison with RL-Based Alignment (DDPO)

  • DDPO balances reward maximization against reference-distribution preservation, and its gradients oppose when high-reward samples are rare under the reference.This creates a trade-off between maximizing reward and staying close to the reference distribution.Source: §H.3 · pp. 36–37, §H.2 · p. 36

H.4. MIRO’s Single-Objective Formulation

  • MIRO’s reward-conditioned objective has a unique function-space optimum, with all training samples contributing toward the same optimal function.With sufficient data covering caption and reward combinations, the empirical minimizer converges to the population minimizer; neural-network parameter loss can nevertheless remain non-convex.Source: §H.4 · pp. 38–39

H.5. Variance Analysis

This section analyzes MIRO’s variance and distributional coverage relative to DDPO, including how reward-conditioned learning may preserve coverage while DDPO may concentrate on high-reward regions. It also outlines MIRO’s stability rationale and reproducibility provisions, noting that its conditional diversity control assumes higher-reward images are more concentrated in feature space.

H.5. Variance Analysis

  • DDPO’s REINFORCE reward-gradient estimator is unbiased but can have arbitrarily large variance, requiring large batches, variance-reduction methods, or careful learning-rate tuning.High reward variance or large score-function magnitudes can drive the variance upward.Source: §H.5 · p. 39
  • MIRO’s gradient estimator has bounded variance, enabling stable training with moderate batches and no specialized variance-reduction techniques.Unlike REINFORCE, the model gradient lacks the 1/pθ factor that can cause variance explosion.Source: §H.5 · pp. 39–40

H.6. Hyperparameter Sensitivity

  • DDPO’s KL coefficient β is critical: β → 0 risks mode collapse and reward hacking, β → ∞ freezes the model, and intermediate values require reward- and dataset-specific tuning.MIRO eliminates β and instead lets users adjust reward guidance at inference through scale ω without retraining.Source: §H.6 · pp. 40–41

H.7. Mode Collapse and Distribution Coverage

  • The section establishes a fundamental contrast: MIRO preserves the flow-matching objective’s full data diversity, whereas DDPO shifts the distribution toward high-reward modes.DDPO’s reward-driven shift is analyzed as distribution collapse; MIRO conditions on reward vectors while retaining coverage.Source: §H.7 · p. 41

H.7.1. DDPO INDUCES MODE COLLAPSE

  • DDPO’s optimal distribution shrinks support, concentrates probability near reward-maximizing samples as β decreases, and reduces expected pairwise sample distance.The entropy reduction is strict when the reward is non-constant.Source: §H.7.1 · pp. 41–42

H.7.2. MIRO PRESERVES FULL DISTRIBUTION COVERAGE

  • MIRO preserves coverage across reward levels and the marginal data distribution, while allowing diversity to decrease as reward increases under Assumption H.8.Users can select a point on the diversity-quality spectrum at inference time; the monotonic diversity claim relies on conditional variance decreasing with reward level.Source: §H.7.2 · p. 43
  • MIRO’s learned entropy structure lets users trade diversity for quality by choosing reward levels, with the entropy gap depending on how strongly the reward concentrates high-scoring samples.The entropy ordering assumes conditional distributions have similar functional forms; the gap is larger for strongly concentrated rewards and smaller, but typically still present, for weaker concentration effects.Source: §H.7.2 · pp. 44–45

H.7.3. QUANTITATIVE COMPARISON OF COVERAGE

  • DDPO’s reward tilting excludes a positive-measure portion of the effective support, whereas MIRO’s marginal preserves the training support.MIRO preserves support because each training example belongs to a reward bin with positive probability.Source: §H.7.3 · p. 45

H.7.4. ILLUSTRATIVE EXAMPLE: GAUSSIAN MIXTURE

  • In the Gaussian-mixture example, DDPO concentrates on the preferred mode as β → 0, while MIRO preserves all K modes and its marginal recovers the full mixture.Conditioning on a reward level selects its corresponding mode, including lower-quality but more diverse modes.Source: §H.7.4 · p. 45

H.8. Summary of Theoretical Advantages

  • MIRO’s conditional density-estimation formulation is presented as explaining its empirical stability: it avoids fundamental instabilities of RL-based approaches while achieving comparable or superior reward performance.Source: §H.8 · p. 46

I.1. Reproducibility Statement

  • The reproducibility statement describes publicly available training datasets and documents the model, objective, captioning pipeline, reward-model configurations, normalization, and binning; code is to be made available upon publication.Source: §I.1 · p. 49

I.2. Ethics Statement

The ethics statement flags concerns about the datasets used, reward-model bias, misuse, and environmental impact. It reports efficiency comparisons while noting that explicit malicious prompts can still produce harmful content.

Data and Privacy

  • MIRO uses publicly available CC12M and LAION Aesthetics 6+ datasets amid community concerns about copyrighted material and private individuals in web-crawled data.Source: §I.2 · p. 49

Bias and Fairness

  • Reward models reflect their annotators’ preferences and biases, so optimizing for high reward may amplify beauty, race, or gender biases.The statement cautions that reward-defined quality is subjective and not culturally universal.Source: §I.2 · p. 49

Environmental Impact

  • MIRO converges up to 19× faster than baseline pretraining and uses 370× less inference compute than FLUX-dev to achieve comparable quality; the authors believe this can reduce carbon footprint.Source: §I.2 · p. 49

Misuse

  • MIRO does not inherently prevent harmful, offensive, or misleading content when users explicitly prompt for malicious content.The statement notes that multi-reward conditioning improves alignment with safe prompts, but does not prevent such generation under explicit malicious prompts.Source: §I.2 · p. 49

Related papers

Loading 2510.25897v2…