Source-linked AI summary
Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models
Sen Ye, Mengde Xu, Shuyang Gu, Di He, Liwei Wang, Han Hu
TL;DR
Multimodal models face a trade-off because generation and understanding can compete during optimization. The paper introduces R3, which integrates understanding into an iterative generation process, and reports stronger generation with preserved or improved understanding related to generation.
Problem
Unified multimodal models face a trade-off because generation and understanding are optimized as competing objectives, with one capability improving at the expense of the other.
Method
R3 decomposes generation into a structured generate–understand–regenerate process that reasons about a prompt, drafts an image, reflects on alignment, and refines the output.
Results
R3 achieves stronger generation while avoiding degradation of understanding, with training evidence linking later understanding gains to accelerated generation accuracy.
Takeaways & Limitations
Integrating understanding into the generative flow offers insights for designing unified multimodal models and balancing generation with understanding.
Takeaways & Limitations
Developing methods that foster more generalized understanding remains an important direction for future research.
Abstract
from arXiv · showhide
Current research in multimodal models faces a key challenge where enhancing generative capabilities often comes at the expense of understanding, and vice versa. We analyzed this trade-off and identify the primary cause might be the potential conflict between generation and understanding, which creates a competitive dynamic within the model. To address this, we propose the Reason-Reflect-Refine (R3) framework. This innovative algorithm re-frames the single-step generation task into a multi-step process of "generate-understand-regenerate". By explicitly leveraging the model's understanding capability during generation, we successfully mitigate the optimization dilemma, achieved stronger generation results and improved understanding ability which are related to the generation process. This offers valuable insights for designing next-generation unified multimodal models. Code is available at https://github.com/sen-ye/R3.
1 INTRODUCTION
Unified multimodal models face a trade-off: improving generation can degrade understanding, while separate optimization leaves the two capabilities in conflict. R3 reframes generation as a generate–understand–regenerate process that integrates understanding into generation and improves both capabilities.
- Multimodal models struggle to improve visual understanding and generation simultaneously, with gains in one capability often weakening the other.Examples include weaker counting and spatial reasoning after image-synthesis fine-tuning, and weaker creative generation after VQA or dense-captioning optimization.
- Existing approaches use unified tokenization or separate architectural capacity, but only achieve partial success in resolving the conflict.
- R3 converts single-shot generation into a structured generate–understand–regenerate process.The framework reasons over the request, produces an initial draft, reflects on whether it meets the request, and refines the output.
- R3 explicitly incorporates understanding into the generative pipeline, mitigating the conflict caused by separate optimization.
- Extensive experiments report stronger generative performance while avoiding degradation of understanding.
2 METHODOLOGY
R3 addresses the generation–understanding conflict by turning image generation into a staged Reason–Reflect–Refine process, with understanding actively guiding iterative correction. Its training separates Reason from Reflect–Refine policies and uses stage-wise rewards to support progressive improvement.
- Unifying Generation and Understanding: Fine-tuning generation or understanding alone degrades the complementary capability, motivating an approach that couples their optimization.Naive mixed-data training provides only negligible gains because the tasks follow different optimization dynamics.
- Reason: R3 begins by expanding the user prompt into a detailed plan and synthesizing an initial image.The Reason policy models joint plan-and-image generation as πθ(I1, t1|c) = πθ(I1|t1, c)πθ(t1|c).
- Training Procedure: Training alternates Reason and Reflect–Refine optimization, passing stage outputs and rewards forward through a replay buffer and importance sampling.Tree-based rollouts sample diverse rewards and support effective training on long generation chains with multiple refinement rounds.
- Reflect–Refine: The model then reflects on image–prompt alignment and either emits a termination signal or an editing instruction for refinement.Reflection produces “No further edit needed.” when the image is satisfactory; otherwise it identifies discrepancies and generates an editing instruction.
- Reflect–Refine: The inference trajectory repeats Reflect–Refine after Reason, enabling progressive correction rather than relying on a single generation attempt.The framework treats the full sequence as Reason → Reflect → Refine → Reflect → Refine… and uses reflection to assess the current output against the prompt.
- Stage-Wise Reward: Stage-wise rewards use image–prompt alignment for initial generation and correctness-based reflection rewards to encourage improvement or correct termination.The initial diffusion reward is Vj, while text generation receives Vj + rj,format; reflection and refinement rewards are based on the correctness metric.
- Unifying Generation and Understanding: Reflection-based training develops visual comprehension even though the reinforcement-learning objective does not directly optimize understanding tasks.The model learns to evaluate image–prompt alignment through the reflection reward, linking comprehension to generation.
3 EXPERIMENTS
Experiments evaluate R3 on generation, understanding, trajectory length, capability evolution, generalization, and cross-topic transfer. R3 improves generation and understanding, with reflection-refinement providing the largest gains while learned understanding remains domain-specific.
- Evaluation Setup: The experiments evaluate generation on GenEval++ and understanding through ITA and VQA protocols, using GPT-4.1 and annotated ground truth for assessment.ITA measures image-text alignment, while VQA probes compositional visual understanding.
- Generation and Understanding: 1% point overall improvement over Echo-4o on GenEval++ demonstrates stronger instruction-following generation, with larger gains in complex Multi-Count cases.Table 1 evaluates generation quality with GPT-4.1; the table caption identifies BAGEL as the baseline reference.
- Generation and Understanding: 12.77 on ITA and 3.15 on VQA exceed reasoning-only gains of 1.16 and 0.24, respectively, showing the importance of reflection-refinement.The full framework also adds nearly 10% points over the reasoning-only baseline on GenEval++.
- Trajectory Length: Performance gains are largest after the first reflection-refinement turn and generally saturate at four or five inference-time turns across GenEval, GenEval++, and TIIF.Training experiments identify a trajectory length of two as the best balance between computational cost and performance.
- Capability Evolution: After 150 training steps, understanding rises noticeably and is associated with accelerated generation accuracy, producing performance above the reflection-free baseline.Earlier training primarily follows the baseline in generation while VQA changes minimally.
- Generalization and Transfer: TIIF results suggest benefits transfer to general-domain generation, but cross-topic evaluation finds understanding improvements localized to trained categories.The authors identify more generalized understanding as an important direction for future research.
4 RELATED WORK
Related work develops unified multimodal models by aligning representations, separating capacities, or combining modality-specific objectives. Reinforcement learning is presented as a route to discovering and optimizing complex multi-step generative strategies.
- Unified Large Multi-modal Models: Unified multimodal models have evolved from shared discrete tokens toward improved tokenizers, decoupled encoders, continuous tokens, latent-space representations, and mixed understanding-generation objectives.These approaches use next-token prediction for understanding and diffusion objectives for generation in some models.
- Unified Large Multi-modal Models: BAGEL is identified as a particularly powerful unified model with significantly improved generation and understanding performance.The passage positions BAGEL among recent unified multimodal approaches.
- Reinforcement Learning for Multi-modal Models: Reinforcement learning enables models to move beyond imitation by autonomously discovering and optimizing complex, multi-step generative strategies.The related-work discussion presents RL as a paradigm for enhancing reasoning capabilities in large language models.
5 CONCLUSION
The paper identifies a trade-off between multimodal generation and understanding and proposes Reason-Reflect-Refine (R3) to address it. R3 uses understanding and iterative refinement during generation, yielding stronger generation and improved understanding related to generation.
- The paper identifies competitive optimization objectives as a possible source of the generation–understanding conflict.Shared model parameters may be optimized for one objective at the expense of the other.
- R3 reframes generation as a multi-step process that explicitly uses understanding to iteratively refine outputs.The framework integrates reflection into the generative flow rather than treating generation as a single-step task.
- R3 helps ease the optimization dilemma, producing stronger generation results and improved understanding ability related to the generative task.
- The framework offers insights for designing next-generation unified multimodal models.
A.1 CLARIFICATION ON THE USE OF LLM
The authors used an LLM to improve manuscript readability and correct grammatical errors. They retained responsibility for reviewing and editing the suggestions and for the paper’s intellectual content.
- An LLM was used to enhance the manuscript’s readability.
- The LLM was also used to correct grammatical errors.
- The authors carefully reviewed and edited all AI-generated suggestions.
- The authors retain responsibility for the paper’s intellectual content, analyses, arguments, and final publication text.
A.2 RL TRAINING WITH GRPO
The methodology uses Group-Relative Policy Optimization to refine both textual reasoning and the diffusion-based image-generation process. Standard GRPO is used for text CoT policies, while FlowGRPO adapts the framework for diffusion models.
- The methodology uses Group-Relative Policy Optimization to refine two components of training.
- GRPO refines Chain-of-Thought generation in the Reasoning and Reflect stages.
- The methodology also applies policy optimization to denoising for image generation and editing.
- The text-based CoT policy uses the standard GRPO algorithm.
- FlowGRPO is adopted for the diffusion model.
A.2.1 GROUP-RELATIVE POLICY OPTIMIZATION (GRPO)
GRPO samples multiple responses, evaluates their rewards, and standardizes each response’s advantage relative to the group. It then updates the policy with a clipped objective and a KL-divergence penalty.
- GRPO stabilizes policy-gradient training by normalizing advantages across a group of sampled responses.This reduces advantage-estimate variance and supports more consistent policy updates.
- The optimization process samples a group of G responses for a prompt using the current policy.
- Each sampled response receives a reward, which is standardized against the statistics of the entire group to calculate its relative advantage.
- A small constant δ is added for numerical stability when calculating the group-relative advantage.
- The policy is updated by maximizing a clipped surrogate objective regularized with a KL-divergence penalty.The penalty prevents large deviations from a reference policy, typically the initial supervised fine-tuned model.
- The per-token importance-sampling ratio compares the new policy with the old policy.
A.2.2 FLOWGRPO FOR DIFFUSION MODEL OPTIMIZATION
FlowGRPO frames iterative diffusion denoising as an MDP and introduces stochastic sampling so policy optimization can explore generation paths. MixGRPO combines SDE and ODE sampling to reduce training cost while maintaining model performance.
- FlowGRPO adapts GRPO to continuous state-space diffusion models trained with flow matching.
- The diffusion denoising process is formulated as an MDP whose state contains the prompt, timestep, and noisy sample, while the action generates the next less-noisy sample.
- The model generates groups of images for each prompt, and trajectory advantages are computed from group-level final rewards and remain constant across timesteps.
- SDE-based sampling adds stochasticity to otherwise deterministic ODE generation, enabling exploration of different generation paths for reinforcement learning.
- MixGRPO combines SDE and ODE sampling during denoising to substantially reduce policy-training computation while maintaining strong model performance.
A.3.1 GENERATION BENCHMARKS
The evaluation spans text-to-image generation, multimodal understanding, computational efficiency, and maze navigation. Results include stronger performance with reinforcement-trained R3, faster convergence, adaptive refinement, and broader application beyond text-to-image generation.
- Generation Benchmarks: R3 is evaluated on three text-to-image benchmarks: GenEval, GenEval++, and TIIF.
- Generation Benchmarks: GenEval tests compositional generation, GenEval++ adds more complex instructions and stronger VLM evaluation, while TIIF tests fine-grained instruction following.
- Understanding Benchmarks: The understanding evaluation measures compositional VQA accuracy and Image-Text-Alignment agreement against Gemini 2.5 Flash-based ground truth.
- Ablation and Efficiency: 0.689 vs. 0.439 at convergence: the RL-trained model substantially outperforms Bagel across iteration counts in GenEval++ inference.
- Ablation and Efficiency: Within 2 Reflection-Refine rounds, the RL-trained model reaches 0.689, whereas Bagel requires 3 rounds to plateau at 0.439.
- Ablation and Efficiency: On GenEval++, 45% of prompts require no refinement, 26% require 1 turn, 14% require 2 turns, and 15% require 3+ turns.
- Beyond Text-to-Image: R3 is also applied to maze navigation, where iterative processing decomposes complex path generation beyond text-to-image generation.