UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning

1 Introduction

• The paper identifies costly, large-scale-data pretraining and post-training approaches that may require external data, focus on one task, or customize for specific tasks as key challenges.The introduction says effective post-training for unified multimodal models remains a significant challenge.Source: §1 · p. 1
• UniRL generates images from constructed prompts, then feeds those images and corresponding questions back to the model for answer prediction and joint supervision.The model’s predicted answers and ground-truth answers are both used to improve generation and understanding.Source: §1 · pp. 1–2
• The introduction frames unified-model performance across generation, understanding, and the balance between them, focusing on basic visual features such as count, color, and position.The same feature scope is used in training and evaluation to align the two tasks.Source: §1 · p. 2
• GenEval reaches 0.77 for Show-o and 0.65 for Janus after UniRL post-training.The method is also reported to improve individual task performance while reducing the generation–understanding imbalance.Source: §1 · p. 2
• The authors present UniRL as a self-improving post-training method using SFT and GRPO, and compare the strategies for unified multimodal models.They describe this as the first such method to their knowledge.Source: §1 · p. 2
• UniRL introduces a metric for quantifying generation–understanding imbalance and reports that its method substantially improves both task qualities while reducing that imbalance.The authors say this yields better consistency between text-to-image generation and multimodal understanding.Source: §1 · p. 3

2.1 Unified multimodal understanding and generation

• Unified-model approaches use varied architectures, including autoregression for both tasks, diffusion alongside an LLM, or different generation methods, but primarily focus on pretraining.The cited examples include Chameleon [38], Emu3 [41], Janus [42], JanusPro [6], SEED-X [13], Show-o [46], and JanusFlow [27].Source: §2.1 · p. 3

2.2 GRPO in LLM and MLLMs

• SFT and GRPO are explored for unified-model post-training alongside alternatives such as DPO and PPO, which the paper describes as adding complexity through predefined preference examples or a separate value network.The paper notes prior GRPO optimization of LLMs through groups of chain-of-thought outputs and task-specific rewards [33].Source: §2.2 · p. 3

3.1 Preliminary

• Show-o combines next-text-token prediction with masked-image-token reconstruction, whereas Janus predicts each next image or text token autoregressively.The paper focuses on these two example unified multimodal models.Source: §3.1 · pp. 3–4

3.2 Prompt and QA pair construction

• Prompts and matching question–answer pairs are constructed across six GenEval-inspired categories, with training prompts kept separate from GenEval test prompts.The 160 object classes combine 80 COCO classes with 80 GPT-4o-generated everyday objects.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Self-improving optimization

• UniRL uses the model’s generated images as training data, with understanding feedback guiding image quality and generation supplying data for understanding.Generated image tokens and the prompt are fed back into the model, then the image sequence and corresponding question are used to predict an answer.Source: §3.3 · p. 5

3.4 SFT and GRPO optimization

• UniRL uses generated images and corresponding questions to optimize answer prediction with supervised fine-tuning or reinforcement learning.SFT uses predicted and ground-truth answers; GRPO samples multiple images and answers and computes rewards against the ground-truth answer.Source: §3.4 · pp. 5–6
• GRPO weights sampled-answer likelihoods by relative rewards and adds a KL penalty to keep the updated policy close to a reference model.Rewards are centered on the group mean and scaled by α to emphasize better answers; the KL penalty is scaled by β.Source: §3.4 · p. 6

3.5 End-to-end training vs. non-end-to-end training

• For models with mismatched image representations, separate GRPO objectives train generation and understanding, while reward signals retain cross-task interaction.Janus and JanusPro use separate representations that cannot meaningfully guide discrete tokens through gradient flow; end-to-end models instead use a unified GRPO loss.Source: §3.5 · p. 6

3.6 The balance between image understanding and generation

• The proposed metric evaluates task consistency in both directions and finds imbalance across Show-o, Janus, Janus-Pro, and Vila-u.T2I→MMU measures answer accuracy given a correct generated image; MMU→T2I measures image correctness given a correct answer.Source: §3.6 · p. 7, Table 3 · p. 7

4.1 Implementation details

• Show-o uses end-to-end training, with 5,000 SFT iterations and 3,000 GRPO steps; Janus uses non-end-to-end training for 1,000 iterations.Show-o GRPO uses 8 H100 GPUs; Janus SFT is omitted from the baseline comparison because it causes unstable training and failure.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Quantitative results

• On GenEval, SFT rises from 0.60 to 0.76 and GRPO scores 0.71; on MMU, GRPO scores 0.79.Understanding generalization is tested on external images generated by models such as JanusPro.Source: §4.2 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 8

4.3 Visualization results

• Qualitative comparisons show more visually faithful prompt-conditioned generations and more accurate answers to corresponding questions.The authors describe this as improved consistency between the two tasks.Source: §4.3 · p. 8, Figure 3 · p. 9

4.4 Analysis of SFT vs. GRPO

• On fully disjoint single-object data, SFT underperforms both the original model and GRPO, while shared abstract features partly mitigate overfitting and GRPO generalizes better.The single-object training and test sets are disjoint in both examples and object types.Source: §4.4 · p. 9
• Under end-to-end training, SFT outperforms GRPO on generation; the authors hypothesize that indirect, distributed T2I supervision may favor its ground-truth-answer loss.They also hypothesize that MMU is more prone to overfitting because it learns mainly from text-token patterns.Source: §4.4 · p. 9
• For non-end-to-end models with different task representations, direct image supervision can destabilize training, while GRPO improves performance.The authors speculate that repeatedly reinforcing early generation errors causes input-distribution drift and eventual collapse.Source: §4.4 · p. 9

• UniRL combines SFT and GRPO to improve generation and understanding performance while reducing task imbalance, with experiments on Show-o and Janus.Source: §5 · p. 9

• The appendix covers implementation and non-end-to-end pipeline details, additional results and visualizations, SFT–GRPO analysis, limitations, and broader impact.Source: A Technical appendices · p. 14

• Training uses constructed prompts as the training set and GenEval prompts as the test set, with a learning rate of 1 × 10−5 for both models.Source: §A.1 · p. 14
• Understanding evaluation uses correctly generated external-model images paired with questions, scoring the percentage of answers matching ground truth.Source: §A.1 · p. 14
• The imbalance metric evaluates answer correctness with Qwen or GPT and image correctness with MMDetection and Mask2Former.Source: §A.1 · p. 14

A.2 Straight-Through gumbel softmax

• ST-GS preserves hard discrete choices in the forward pass while using a soft relaxation for backward gradients.This differentiable sampling supports end-to-end optimization.Source: §A.2 · p. 14
• The Gumbel-Softmax distribution forms a continuous approximation of one-hot samples using category probabilities, Gumbel noise, temperature, and the number of categories.The temperature controls smoothness.Source: §A.2 · p. 14
• ST-GS lets downstream-task gradients propagate through discrete image tokens so the generation module participates in joint optimization with understanding.Source: §A.2 · p. 14

A.3 Non end-to-end training pipeline

• Non-end-to-end GRPO training decouples optimization into separate objectives for the T2I and MMU modules.Source: §A.3 · p. 15
• T2I optimization updates image-token logits using rewards derived from MMU performance.The objective takes a prompt and sampled image-token sequences, with reward-based importance weights derived from MMU outputs.Source: §A.3 · p. 15
• MMU optimization directly updates answer prediction using rewards based on answer correctness.Its inputs include image tokens and a question; importance weights are computed from the answer reward.Source: §A.3 · p. 15
• Although the modules are optimized independently, their reward signals allow cross-task influence and mutual supervision.Source: §A.3 · p. 15

A.4 More generation results

• The comparison uses guidance scale 5 and 50 generation steps, extending the main Show-o setup of guidance scale 2 and 16 steps.Source: §A.4 · p. 15
• 0.79 for SFT-based UniRL and 0.74 for GRPO-based UniRL are the final scores under guidance scale 5 and 50 generation steps.Source: Table 7 · p. 15, §A.4 · p. 15
• Both UniRL optimization strategies significantly outperform all baselines in the higher-guidance, longer-generation comparison.Source: Table 7 · p. 15, §A.4 · p. 15

A.5 More understanding results

• The broader POPE and MMMU evaluations report slightly lower scores, which the authors attribute to basic-feature training and MMMU's emphasis on text reasoning and numerical calculation.They suggest broader training categories or a larger model may help mitigate this limitation.Source: Table 8 · p. 15, §A.5 · p. 15

A.6 More visualization results

• Figure 4 includes out-of-distribution examples, and the authors report that GRPO generalizes beyond the categories seen during training.Source: §A.6 · p. 15, Figure 4 · p. 16

A.7 More analysis of SFT and GRPO

• In end-to-end training, SFT may outperform GRPO because ground-truth answers provide direct supervision when T2I feedback is indirect.The authors say this ground-truth signal helps SFT converge more efficiently and does not heavily depend on base-model capacity.Source: §A.7 · p. 16
• For MMU, SFT can overfit text patterns, while GRPO's reward training and KL regularization encourage generalization; in non-end-to-end T2I training, SFT can be unstable and GRPO is more robust.Source: §A.7 · p. 16

A.8 Limitations

• The work focuses on basic visual understanding, leaving mathematical problem-solving and abstract inference outside its scope; GRPO iterations are relatively slow because they require multiple inference passes.Source: §A.8 · p. 16

A.9 Broader impacts

• The authors state that they believe the method will not bring negative societal impacts.Source: §A.9 · p. 16