Unified Multimodal Models as Auto-Encoders

1. Introduction and Motivation

• Joint training can be brittle because generative objectives may degrade understanding, while separately training the modules misses potential cross-task benefits.The paper argues that without demonstrable mutual gains, unification risks amounting to two components trained side by side.Source: §1 · p. 2
• The auto-encoder perspective uses text as an intermediate representation: images are encoded into semantics and those semantics are decoded back into images.Reconstruction quality serves as a proxy for whether the encoder captured essential visual structure and the decoder recovered it.Source: §1 · p. 2
• Unified-GRPO jointly optimizes the encoder and decoder with reconstructive rewards, encouraging richer image semantics and more faithful generated images.The paper describes the two modules as co-evolving through cross-modal feedback.Source: §1 · p. 2
• Generation scores rose from 0.73 to 0.86 on GenEval and from 0.296 to 0.475 on GenEval++, while small-object detection rose from 0.05 to 0.45 and Person ReID from 0.15 to 0.75.The paper also reports maintained overall visual-understanding performance and improved unification on Unified-Bench.Source: §1 · p. 2
• Unified-GRPO applies to varied encoder–decoder multimodal systems, with reported gains in text-to-image generation and fine-grained understanding, alongside trade-offs in OCR-heavy scenarios.Examples of fine-grained understanding include grounding and small-object recognition.Source: §1 · p. 2

2. UAE Methodology

• The methodology encodes an input image as a semantic description and reconstructs an image from that description, using semantic similarity between source and reconstruction as the learning objective.Text is the shared intermediate representation connecting image-to-text understanding with text-to-image generation.Source: §2 · p. 3

2.1. Unified-GRPO

• Unified-GRPO applies reconstructive reinforcement learning to UMMs, optimizing their language-model components toward higher-fidelity reconstruction.For UMM-1, the LLM is trained while the diffusion transformer remains frozen; for UMM-2, the autoregressive model performs both understanding and generation, with a CLIP-embedding similarity reward.Source: §2.1 · p. 3

2.2. Unified-Bench: A Benchmark tailored for Evaluating the Unified Models

• Unified-Bench measures reconstruction similarity across vision backbones and assesses caption quality through pairwise comparisons with commercial LLM judges.Its protocols use 100 source images for reconstruction scoring and evaluate whether captions support image generation; the tables cover unified scores and caption friendliness.Source: §2.2 · pp. 3–4, Table 2 · p. 4, Table 3 · p. 4

3.1. Ablation on Unified-GRPO

• The gains are significant on generation and unification metrics: Unified-GRPO improves generation by 4∼5% and unified reconstruction quality by over 6%, where reconstruction is directly optimized.Understanding gains are modest; the authors attribute this to limited capacity of current generation models, since imperfect reconstructions can introduce negative feedback to the encoder. The UniWorld-based model performs stronger than Janus in both generation and understanding and is used as the primary backbone for subsequent experiments.Source: §3.1 · p. 4

3.2. Unification Evaluation

• UAE achieves the best Overall unified score (86.09), surpassing GPT-4o-Image (85.95), and leads on CLIP (90.50), DINO-v2 (81.98), and DINO-v3 (77.54).It reaches statistical parity on LongCLIP (94.35 vs. 94.37).Source: §3.2 · p. 5

3.3. Text-to-Image Generation Evaluation

• GenEval Overall score is 0.86 without LLM rewriting; with rewriting, UAE averages 0.89 and is reported as state of the art.Without rewriting, it leads on Counting (0.84) and Color attribution (0.79), co-leads on Colors (0.90), and ranks second on Position (0.71).Source: Table 4 · p. 4, §3.3 · p. 5
• GenEval++ Overall score reaches 0.475, with UAE leading on Color/Count (0.550) and Pos/Count (0.450) for challenging instruction-following generation.The authors also report runner-up performance on Color/Pos (0.550) and Multi-Count (0.400).Source: Table 5 · p. 4, §3.3 · p. 7

3.4. Image-to-Text Understanding Evaluation

• Caption evaluation reports pairwise win rates of 94.7 vs. Show-o, 71.4 vs. OmniGen2, 64.3 vs. Bagel, and 76.3/71.5 vs. Qwen-2.5-VL (3B/7B), while remaining competitive with GPT-4o (47.2).The trained model's captions also yield the highest reconstruction similarity across four backbones: 90.50 (CLIP), 94.35 (LongCLIP), 81.98 (DINO-v2), and 77.54 (DINO-v3), with 86.09 Overall.Source: §3.4 · p. 7

3.5. Evaluation on the Understanding Benchmark.

• Fine-grained MMT-Bench recognition gains include +40.0% on Small Object Detection and +60.0% on Person Re-ID, while the overall score changes by only +0.2%.The fine-grained overall score increases by +24.4%; the supplied Table 8 caption identifies the fine-grained evaluation scope.Source: §3.5 · p. 7, Table 8 · p. 7

3.6. Case-Study in Fine-Grained Visual Perception

• Case studies show Unified-GRPO recognizing a subtle difference in the number of people across images and following a skier-grounding instruction by identifying the correct region.The skier example requires matching the specified outfit colors to the labeled bounding box.Source: Figure 7 · p. 6, §3.6 · p. 8

Model architectures

• Autoregressive UMMs process image and text tokens sequentially, while hybrid architectures pair autoregressive modeling with diffusion or flow-matching image synthesis.Examples include Chameleon and EMU [8, 28, 47, 55, 63, 65], and Transfusion [11] [53] [71] [82].Source: §4 · p. 8

Alternative architectures

• Other approaches route representations from a frozen pretrained MLLM to an external image generator, or combine autoregressive language processing with masked-autoregressive visual reconstruction.The passage names learnable query modules or MLPs [7] [32] [43] and MAR [29], including its adoption in Harmon [12] [59] [66].Source: §4 · p. 8

Post-training approaches

• Related post-training work uses dense image features as rich-text conditions for diffusion training, and other frameworks apply reinforcement learning to multimodal learning.Source: §4 · p. 8

• The paper concludes that Unified-GRPO's joint optimization creates a synergistic feedback loop associated with stronger fine-grained perception, richer semantic encoding, and improved complex instruction following.It frames these findings as support for treating multimodal tasks as mutually reinforcing components of a unified system.Source: §5 · p. 8

• The supplement lists additional related work, dataset details, training settings, qualitative examples, and experimental results.Source: Supplementary Overview · p. 13

Reinforcement Learning in Generative Models

• RLHF-inspired text-to-image methods train reward models from human judgments before using reinforcement learning, but reward models can be brittle and vulnerable to reward hacking [73] [72] [5] [62].GRPO and extensions including FlowGRPO [34] and DanceGRPO [74] are described as alternatives for tuning diffusion and flow-matching models [52].Source: §1 · p. 13

Benchmarking Multimodal Understanding, Generation, and Unification

• Existing UMM evaluation aggregates specialized benchmarks, while UniBench assesses comprehension directly on generated images without real-image annotations [36] [79] [22].Generative benchmarks assess attributes such as object presence, spatial relations, counting, color, and position; integrated UMM evaluation remains a gap [14, 20, 21, 4, 9, 26].Source: §1 · p. 13

RL stage data (1K)

• The RL phase uses 1,000 curated real-world photographs spanning portraits, architecture, nature, and street photography, alongside photorealistic Echo-4o image-text pairs.The combined data support reconstruction-driven optimization, with generated images evaluated against originals for fidelity, detail preservation, and semantic alignment.Source: §2 · p. 13

Data for evaluation in Unified-Bench

• Unified-Bench uses 100 randomly sampled LAION-5B images as a test split, selected without content or aesthetic filtering.The passage describes the selection as intended to provide a representative, unbiased distribution across categories, styles, and complexity levels.Source: §2 · p. 14

Training details of Unified-GRPO

• Training updates only the LLM; the visual encoder and decoder remain frozen, and a diffusion decoder plus pretrained CLIP form the inference-only reward module.The authors report that updating the visual encoder can destabilize training and degrade image quality, including through artifacts, structural collapse, or semantic inconsistency.Source: §3 · p. 14
• Training uses a 1 × 10−6 learning rate, batch size 1, four sampled images per prompt, and a 1×10−6 KL regularization coefficient.The temperature is 1.0; the authors cite diffusion-based RL's high computational cost as the reason for batch size 1.Source: §3 · p. 14
• After RL, the LLM produces longer, richer captions despite no explicit descriptive-caption requirement or caption-content loss during training.The authors attribute this emergent behavior to reconstruction-quality reward implicitly guiding more detailed, image-friendly descriptions.Source: §3 · p. 14

Qualitative Examples

• Unfreezing ViT during joint training collapses generated output, omitting semantically important details such as two pumpkins and one candle.This degradation motivates keeping ViT frozen during finetuning across experiments.Source: Figure 9 · p. 14

Enhancing model’s comprehensive perception by the generation model

• UAE's caption specifies the dog's species, clothing, eyewear, pose, occlusions, background, and lighting, and its reconstruction preserves identity, attire, and overall aesthetic.Baseline captions exhibit category drift, omitted or mismatched attributes, or underspecified scenes; the passage relates denser, better-bound descriptions to better generation.Source: §4 · p. 14

Text-to-image results

• On DPG-Bench, UAE scores 91.43 on Entity, 91.49 on Attribute, and 92.07 on Relation, and ranks second overall with 84.74 behind Bagel’s 85.07.The paper attributes UAE’s strong sub-scores to faithful entity grounding and relation handling under long prompts.Source: §5 · p. 14, Table 9 · p. 17

Scaled Unified-Bench evaluation

• The authors expanded Unified-Bench to 2,000 source images and report that the results in Table 10 validate the method’s effectiveness.The supplied passages do not provide Table 10’s result values.Source: Table 10 · p. 21, §5 · p. 21

OCR and Document Understanding

• OCR and Document Understanding accuracy drops by approximately 10%; replacing SD3 with stronger-text-rendering Qwen-Image makes the performance drop disappear.The authors attribute the drop to SD3’s text-rendering bottleneck, which provides noisy supervision to the understanding model.Source: Table 11 · p. 21, §5 · p. 21

Reward-model choice

• The paper uses CLIP’s semantic similarity as the reconstruction reward because pixel-level MSE penalizes lighting, texture, and style variations that do not affect semantics.The I2T2I pipeline preserves semantic content through text as an intermediate representation.Source: §5 · p. 21

Training conditions

• Effective training requires a T2I model with basic long-instruction following; short token limits can bottleneck reconstruction quality and RL optimization.The paper notes that modern T2I foundations such as Qwen and Longcat-Image can handle long, structured prompts.Source: §5 · p. 21

Limitations and future work

• The authors identify improving text reconstruction fidelity as future work and propose extending the text-mediated reconstructive framework to audio and video.They link the current framework’s constrained benefits to generation-module weaknesses, including difficulty faithfully rendering text.Source: §5 · p. 21

Caption-quality evaluation

• The judge scores each caption from 1 to 10 and compares them head-to-head, quoting phrases from both captions to justify differences.The prompt requests a markdown scoring table and criterion-specific analysis.Source: §5 · p. 25
• The rubric asks the judge to assess descriptive accuracy and richness, linguistic fluency and naturalness, and semantic or compositional insight.Examples include details such as colors, textures, context, mood, and foreground-versus-background composition.Source: §5 · p. 25
• After the comparative analysis, the judge must declare the winner using only the letter A or B.The final-answer format is specified as a single letter.Source: §5 · p. 25