Source-linked AI summary
Refinement via Regeneration: Enlarging Modification Space Boosts Image Refinement in Unified Multimodal Models
Jiayi Guo, Linqing Wang, Jiangshan Wang, Yang Yue, Zeyu Liu, Zhiyuan Zhao, Qinglin Lu, Gao Huang, Chunyu Wang
TL;DR
Existing UMM refinement methods rely on editing instructions and pixel-level preservation, which can incompletely specify mismatches and restrict modification space. RvR instead regenerates images from the target prompt and initial-image semantic tokens, improving performance across three T2I benchmarks. The paper supports regeneration as a more effective foundation for image refinement in UMMs.
Problem
RvE uses incomplete editing instructions and strict pixel-level consistency, unnecessarily restricting refinement's modification space and potentially limiting prompt–image alignment.
Method
RvR reformulates refinement as conditional image regeneration conditioned on the target prompt and semantic tokens of the initial image, using independently generated samples for supervision.
Results
RvR consistently improves performance across Geneval, DPGBench, and UniGenBench++, including Geneval from 0.78 to 0.91.
Takeaways & Limitations
Regeneration provides a more flexible foundation for semantic correction by discarding editing instructions and unnecessary consistency constraints.
Abstract
from arXiv · showhide
Unified multimodal models (UMMs) integrate visual understanding and generation within a single framework. For text-to-image (T2I) tasks, this unified capability allows UMMs to refine outputs after their initial generation, potentially extending the performance upper bound. Current UMM-based refinement methods primarily follow a refinement-via-editing (RvE) paradigm, where UMMs produce editing instructions to modify misaligned regions while preserving aligned content. However, editing instructions often describe prompt-image misalignment only coarsely, leading to incomplete refinement. Moreover, pixel-level preservation, though necessary for editing, unnecessarily restricts the effective modification space for refinement. To address these limitations, we propose Refinement via Regeneration (RvR), a novel framework that reformulates refinement as conditional image regeneration rather than editing. Instead of relying on editing instructions and enforcing strict content preservation, RvR regenerates images conditioned on the target prompt and the semantic tokens of the initial image, enabling more complete semantic alignment with a larger modification space. Extensive experiments demonstrate the effectiveness of RvR, improving Geneval from 0.78 to 0.91, DPGBench from 84.02 to 87.21, and UniGenBench++ from 61.53 to 77.41.
1 Introduction
The paper identifies limitations in editing-based refinement and proposes regeneration-based refinement to enlarge the modification space and improve prompt–image alignment. Experiments report substantial gains across multiple T2I benchmarks.
- RvE can incompletely address prompt–image mismatches because editing instructions may describe only part of the required semantic corrections.The paper gives examples involving object count, layout, and appearance changes.
- RvE also restricts refinement through strict content consistency for unedited regions, limiting broader structural changes.
- RvR reformulates refinement as conditional image regeneration using the target prompt and semantic tokens of the initial image.It removes intermediate editing instructions and pixel-level consistency requirements.
- RvR enlarges the effective modification space, allowing revisions to regions that hinder prompt satisfaction while producing semantically correct and spatially coherent images.
- RvR is trained with supervision from independently generated T2I samples with varying prompt-alignment levels rather than editing pairs enforcing strict content consistency.
- Geneval improves from 0.78 to 0.91, DPGBench from 84.02 to 87.21, and UniGenBench++ from 61.53 to 77.41.
2 Related Work
Related work covers unified multimodal models and refinement methods for text-to-image generation. Existing approaches combine multimodal reasoning with image updates through editing, reflection, or coordinated text–vision processes.
- Unified multimodal models integrate language or vision-language models with image understanding and generation capabilities.
- T2I refinement aims to reduce prompt–image mismatches in preliminary samples, especially for compositional prompts.
- Prior refinement methods alternate textual reflection with image synthesis or organize refinement as a unified chain-of-thought across text and vision.
3 Refinement via Regeneration
RvR reframes UMM-based image refinement as conditional regeneration, avoiding intermediate editing instructions and unnecessary pixel-level preservation constraints. Its training data and pipeline enlarge the modification space while retaining semantic conditioning on the initial image.
- RvE formulation: RvE decomposes refinement into instruction generation followed by image editing, using visual and VAE representations of the original image.The instruction stage predicts an editing instruction from the prompt and image, while the editing stage generates the refined image under that instruction.
- Limitations of RvE: Incomplete or ambiguous editing instructions can limit subsequent edits and accumulate errors across the two-stage refinement process.RvE also expects edited outputs to preserve non-target content, imposing constraints suited to editing but not necessarily to refinement.
- RvR formulation: RvR conditions directly on the target prompt and semantic tokens of the initial image, reformulating refinement as conditional image regeneration.The approach removes reliance on an intermediate editing instruction and does not enforce pixel-level consistency with the input image.
- RvR data construction: RvR data uses independently generated images with varying prompt alignment rather than editing pairs requiring strict content consistency.Prompts span randomly selected semantic dimensions; multiple T2I generators produce candidates, and a VLM labels alignment to form misaligned-image, aligned-image, prompt triplets.
- RvR pipeline: RvR removes VAE tokens from the conditioning context so regeneration can make larger modifications toward prompt alignment.During inference, the model uses the prompt and ViT semantic tokens from the misaligned image to regenerate an improved image.
4 Experiment
Experiments compare RvR with T2I and refinement baselines, assess iterative refinement and robustness to initial semantics, and ablate training choices. RvR achieves leading benchmark performance while preserving compatible content, discarding conflicting content, and benefiting mainly from refinement training.
- Experimental Setup: RvR is compared with generation-only models, UMMs, and RvE methods using BAGEL and representative baselines including UiG, Uni-CoT, and IRG.The evaluation uses Geneval, DPGBench, and UniGenBench++ for quantitative comparison, alongside qualitative refinement cases.
- Qualitative Comparison: RvR corrects diverse semantic misalignments involving object quantity, relative position, world knowledge, negation grammar, and object composition.Qualitative comparisons use identical initial images across methods and highlight misaligned regions.
- Main Results: 0.91 vs. 0.85 on Geneval, 87.21 vs. 85.11 on DPGBench, and 77.41 vs. 69.86 on UniGenBench++ show RvR outperforming editing-based refinement.Against BAGEL, RvR improves Geneval from 0.78 to 0.91, DPGBench from 84.02 to 87.21, and UniGenBench++ from 61.53 to 77.41.
- Multi-round Generation: Additional RvR rounds further correct unresolved semantics while preserving semantics already aligned after the first round.The second round can also improve minor visual details, such as removing an unnecessary bench armrest.
- Robustness to Initial Image Semantics: RvR reuses prompt-compatible initial semantics and discards conflicting semantics before generating prompt-aligned content.Compatible background elements can be preserved, whereas incompatible scenes are replaced with newly aligned images.
- Ablation Studies: 84.08 vs. 84.02 shows that RvR training preserves basic T2I capability, while 84.62 vs. 84.02 indicates only a minor gain from same-scale SFT.Adding editing data reduces DPGBench from 87.21 to 85.70, and adding VAE features reduces it to 86.41.
5 Conclusion
The paper reframes unified multimodal image refinement as regeneration rather than editing. RvR removes editing-specific constraints, trains on independently generated images with varied alignment, and improves text-to-image performance across benchmarks.
- Conclusion: RvR conditions another generation round on the target prompt and input-image semantic tokens, discarding editing instructions and unnecessary consistency constraints.Its training paradigm uses independently generated images with different prompt-alignment levels to encourage semantic correction rather than appearance preservation.
A Attention Mask
RvR uses causal attention for prompt text and full attention for image tokens during unified multimodal training.
- Attention Mask: Causal attention applies to text tokens, while full attention applies to ViT tokens of the misaligned image and noisy VAE tokens of the aligned image.This follows the standard omni-attention mechanism adopted for RvR training.
B Refinement Data Examples
RvR training uses independently generated misaligned and aligned images rather than editing pairs, removing pixel-level constraints and supporting bilingual refinement data.
- Refinement Data Examples: RvR refinement samples pair independently generated images with different prompt-alignment levels, encouraging semantic correction without pixel-level correspondence.The training prompts use a 1:1 ratio of English and Chinese, allowing bilingual refinement.
C Additional Qualitative Results
Additional qualitative results further demonstrate RvR’s refinement performance, including examples with Chinese prompts that showcase bilingual refinement capability.
- RvR’s additional qualitative results include Chinese-prompt examples demonstrating bilingual refinement capability.