PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling

1. Introduction

• Consistent image generation remains difficult because large-scale consistency datasets are scarce and human perceptual preferences are complex to model.The paper studies Image Editing and Text-to-ImageSet: aligning outputs with references or generating coherent image sets from one prompt.Source: §1 · p. 2
• PaCo-Reward uses sub-figure pairing to build a human-annotated consistency-ranking dataset, then models pairwise preferences through autoregressive token prediction.Task-aware instructions and CoT-style reasons are included to support generalization and interpretability.Source: §1 · p. 2
• PaCo-GRPO trains on low-resolution images while retaining full-resolution inference, and uses log-tamed multi-reward aggregation to reduce cost and stabilize optimization.The methods can be combined with prior optimizations to improve efficiency without compromising performance.Source: §1 · p. 2
• PaCo-Reward improves correlation with human preferences by 8.2%-15.0%, while PaCo-GRPO achieves state-of-the-art results with 10.3%-11.7% consistency-metric gains across both tasks.The reported improvements preserve prompt fidelity, with nearly doubled training efficiency and greater stability.Source: §1 · p. 2

2.1. Consistent Image Generation

• Text-to-ImageSet and Image Editing share the goal of preserving identities, styles, or visual conditions despite different task formulations.The paper unifies them under a consistent-image-generation framework to address curated-data dependence, narrow formulations, and limited generalization under complex constraints.Source: §2.1 · p. 2

2.2.1. Reward Models for Image Generation

• Existing reward models largely assess image-text alignment or aesthetics, while consistency preferences require pairwise comparisons that capture multiple shared attributes.Similarity measures such as CLIP and DreamSim are not designed to capture the multifaceted nature of human consistency perception.Source: §2.2.1 · p. 3

2.2.2. RL Algorithms for Image Generation

• RL sampling remains a computational bottleneck, especially when consistency tasks require multiple images; PaCo-GRPO's low-resolution training complements prior efficiency methods.The paper identifies MixGRPO and FlowGRPO-Fast as methods with which its strategy can be combined.Source: §2.2.2 · p. 3

3.1. Paco-Dataset: Pairwise Consistency Dataset

• PaCo-Dataset covers 6 categories and 32 subcategories, with rankings for one reference and four comparison images; 3,136 instances form the ConsistencyRank benchmark.The scenarios include character, design-style, and process generation, spanning identity, style, and logical consistency.Source: §3.1 · p. 3
• Automated sub-figure pairing yields 33,984 ranking instances from 708 prompts and 2,832 images using 2 × 2 grids.Four grids with different seeds are generated per prompt, then paired across grids; six annotators label the resulting instances.Source: §3.1 · p. 3
• Rankings are converted into pairwise comparisons and combined with manually verified pairs, producing 54,624 human-labeled pairs augmented with GPT-5-generated CoT reasoning.The resulting set contains 27,599 consistent and 27,025 inconsistent pairs.Source: §3.1 · pp. 3–4

3.2. PaCo-Reward: Pairwise Reward Modeling

• PaCo-Reward predicts the likelihood of “Yes” or “No” for an image pair as an autoregressive VLM task; the “Yes” probability serves as the consistency score.Scores relative to a reference image can also be used to infer rankings among candidate images.Source: §3.2 · p. 4
• A weighted likelihood objective trains both the binary decision and CoT reasoning, with α balancing their supervision; α = 0.1 gave the best generalization in this setting.The paper motivates the weighting as a balance between binary-only overfitting and weakening the main signal by optimizing the full CoT sequence.Source: §3.2 · p. 4

4.1. Preliminaries

• FlowGRPO and DanceGRPO adapt GRPO to image generation by converting deterministic ODEs into SDEs, introducing sampling stochasticity.Stochastic sampling promotes diversity and exploration during RL training.Source: §4.1 · pp. 4–5
• PaCo-GRPO addresses costly sampling for high-resolution or multi-image tasks and reward domination in multi-reward optimization.Reward domination occurs when one signal overwhelms others, potentially producing suboptimal or degenerate results.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.2. PaCo-GRPO Strategies

• The framework combines resolution-decoupled training with log-tamed multi-reward aggregation to improve efficiency and mitigate reward domination during RL.The first strategy reduces sampling and training costs; the second is intended to stabilize optimization.Source: §4.2 · pp. 5–6
• PaCo-GRPO trains with low-resolution images for reward computation and optimization while retaining high-resolution generation at inference and evaluation.This reduces sampling and training overhead; low-resolution training is described as compatible with existing efficiency strategies.Source: §4.2 · p. 5
• Log-tamed aggregation adaptively compresses rewards with large fluctuations to prevent them from overwhelming optimization.It uses each reward’s coefficient of variation to identify fluctuations, then logarithmically transforms rewards above a threshold while preserving sample ordering.Source: §4.2 · p. 6

5. Experiments

• The experiments ask whether PaCo-Reward captures human consistency preferences, whether it improves RL generation, and whether PaCo-GRPO improves efficiency and stability.They comprise reward-modeling evaluation and RL evaluation, with separate setups for each.Source: §5 · p. 6, §5.1 · p. 6, §5.1.1 · p. 6

5.2. Experimental Results

• On EditReward-Bench, PaCo-Reward-7B outperforms all open-source baselines and approaches GPT-5.The paper describes this as generalization to diverse visual-consistency preferences.Source: Table 1 · p. 6, §5.2 · p. 7
• PaCo-Reward-7B gains 10.5% in Accuracy and 0.150 in Spearman’s ρ over Qwen2.5-VL-7B on ConsistencyRank.The paper reports that both PaCo-Reward variants surpass all baselines on ConsistencyRank.Source: Table 2 · p. 6, §5.2 · p. 7
• On T2IS-Bench, FLUX.1-dev with PaCo-Reward-7B gains 0.117 under Qwen2.5-VL-7B and 0.103 under Gemma-3-4B over the strongest open-source baseline.The two evaluators provide cross-model evaluation of visual consistency.Source: Table 3 · p. 7, §5.2 · p. 8
• PaCo-Reward-7B improves Semantic Consistency (SC) and Prompt Quality (PQ) across base models on GEdit-Bench, preserving image quality where EditReward degrades it for Step1X-Edit.The reported results also include gains for Qwen-Image-Edit.Source: Table 4 · p. 7, §5.2 · p. 8

5.3. Case Studies

• Fixed-seed case studies depict improving identity, style, and logic consistency, including a sketch-to-drawing sequence in which each panel extends the previous one.The examples also report higher prompt fidelity and improved aesthetics in the style case.Source: Figure 7 · p. 8, §5.3 · p. 8

5.2. Experimental Results

• At 512 × 512, training reaches 1024 × 1024 performance after about 50 epochs, while 256 × 256 fails to provide reliable reward evaluation.The paper reports higher reward variance at low resolution and states that moderate resolution reduction can accelerate training without degrading final performance.Source: Figure 4 · p. 7, Figure 5 · p. 7, §5.2 · p. 8
• Log-tamed aggregation keeps the Consistency-to-Prompt Alignment reward ratio below 1.8, while standard weighted-sum aggregation exceeds 2.5 after 50 epochs.The comparison uses PaCo-Reward-7B for Consistency and CLIP-T for Prompt Alignment.Source: Figure 6 · p. 7, §5.2 · p. 8

6. Conclusion

• PaCo-Reward and PaCo-GRPO together provide more human-aligned and computationally efficient training for consistent image-generation tasks.Source: §6 · p. 8

• Figures 8–11 are captioned as training-progression visualizations for image editing with different prompts and Text-to-ImageSet generation.Source: Figure 8 · p. 12, Figure 9 · p. 13, Figure 10 · p. 14
• The prompts include a chronological leaf-decomposition scene with realistic style, consistent lighting, and forest-floor details.Source: Ref. Image · p. 13
• Other prompts describe honey-themed beverage labels, a staged chef emoji, a colorful parrot from four perspectives, and a Chinese ink painting of pumpkins and vines.Source: Ref. Image · pp. 14–16
• Figures 12–14 are captioned as comparisons of different methods for multi-image generation and Text-to-ImageSet generation.Source: Figure 12 · p. 16, Figure 13 · p. 17, Figure 14 · p. 18

PaCo-Reward Implementation Details

• PaCo-Reward variants are fine-tuned from Qwen2.5-VL-7B-Instruct for two epochs on PaCo-Dataset, requiring about 18 GPU hours.Both use LoRA rank 32, α = 64, learning rate 2 × 10^-4, and batch size 8; only PaCo-Reward-7B applies first-token weighting.Source: §7 · p. 19

PaCo-RL Implementation Details

• PaCo-RL reserves one GPU for vLLM reward computation and uses seven for fine-tuning, with group size 16 and batch size 1 by default.The setup uses 42 unique samples per epoch, learning rate 3 × 10^-4, clipping ε = 1×10^-4, and zero KL-loss weight.Source: §7 · p. 19

Text-to-ImageSet

• Text-to-ImageSet training uses FLUX.1-dev at 512 × 512 resolution and evaluates at 1024×1024, aggregating consistency and text-image alignment rewards.It uses LoRA rank 64, CFG scale 3.5, 10 denoising steps, SDE sampling at timestep 1, and δ = 0.2 for reward aggregation.Source: §7 · p. 19

Image Editing

• Image Editing fine-tunes FLUX.1-Kontext-dev and Qwen-Image-Edit at 384 × 384 and evaluates at 1024 × 1024, using a single reward signal.Both use LoRA rank 64 and α = 128; CFG scales are 2.5 and 4.0, respectively, and multi-reward aggregation is not used.Source: §7 · p. 19

Text-to-ImageSet

• Text-to-ImageSet uses one template with detailed dataset-derived consistency criteria and another with only the input prompt for generalization evaluation.The prompt-only version asks whether two images maintain style, logic, and identity consistency.Source: §8 · p. 19

Image Editing

• Image Editing evaluates both prompt following and semantic consistency, asking whether edits obey instructions and preserve unchanged aspects of the original.The templates request a Yes-or-No answer first, followed by detailed reasons.Source: §8 · p. 19

Resolution-Decoupled Training Analysis

• 0.725–0.848 Pearson correlations between 1x and 0.5x resolutions indicate largely preserved image-set quality rankings and reliable reward signals.Correlations weaken substantially between 1x and 0.25x, especially for Aesthetics and Consistency.Source: §9 · p. 20, Figure 15 · p. 20
• Validation across additional reward models and generators found that 0.5x reduction preserved reward reliability while 0.25x did not, and no systematic artifacts appeared at full-resolution inference.The authors report that reward improvements transfer across resolutions.Source: §9 · p. 20

10. Ablation on PaCo-GRPO Components

• Removing resolution-decoupled training produced suboptimal performance despite doubling training time from 6h to 12h.The authors attribute this to incomplete convergence at higher resolution and report that 0.5x training halves computational cost while enabling more effective exploration in the same time budget.Source: §10 · p. 20
• Without log-tamed aggregation, visual consistency reward dominated, raising V.C. to 0.557 while aesthetics and prompt following degraded and image sets collapsed toward near-identical, low-quality outputs.The authors identify the log-tamed strategy as necessary for balancing multiple reward objectives and preventing reward hacking.Source: §10 · p. 20

11. PaCo-Dataset Details

• PaCo-Dataset annotations use independent evaluations by multiple annotators, majority voting, and discard ties to retain clear binary preferences.Annotators follow detailed guidelines and illustrative examples covering style, logic, and identity.Source: §11 · p. 20
• The dataset details organize categories and subcategories by their corresponding consistency dimensions.These mappings are summarized in Table 6.Source: §11 · p. 20, Table 6 · p. 21