PaperlayerView exact revision on arXiv

Source-linked AI summary

EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture

Xin He, Longhui Wei, Jianbo Ouyang, Minghui Liao, Lingxi Xie, Qi Tian

arXiv:2512.04810v5cs.CV

TL;DR

Unified multimodal approaches have not converged on a design that combines understanding, generation, and editing. EMMA combines compressed visual encoding, channel-wise fusion, and a shared-and-decoupled network; it reports stronger results than unified-model baselines across tasks, with fewer visual context tokens.

  • Problem

    Unified multimodal approaches have distinct advantages and limitations and have not converged, motivating a unified architecture for understanding, generation, and editing.

    Source: §1 · p. 2

  • Method

    EMMA combines 32x visual compression, channel-wise concatenation, a shared-and-decoupled network, and a visual-understanding mixture of experts.

    Source: §1 · p. 2, §2.1 · pp. 4–5

  • Results

    73.0 on MMVet and 0.93 on GenEval for EMMA-4B compare with BAGEL-7B’s 67.2 on MMVet and 0.88 on GenEval, while EMMA uses 20% of BAGEL’s visual context tokens.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EMMA reduces visual tokens by 5x in image editing versus recent unified models such as BAGEL while reporting superior performance across understanding, generation, and editing benchmarks.

    Source: §1 · p. 4, §2.1 · p. 4

    EMMA excludes GPT-Image-Edit-1.5M because it disrupts character consistency, and the authors call for more accurate image-editing evaluation criteria.

    Source: §3 · p. 7

Abstract

from arXiv · show

We propose EMMA, an efficient and unified architecture for multimodal understanding, generation and editing. Specifically, EMMA primarily consists of 1) An efficient autoencoder with a 32x compression ratio, which significantly reduces the number of tokens required for generation. This also ensures the training balance between understanding and generation tasks by applying the same compression ratio to images. 2) Channel-wise concatenation instead of token-wise concatenation among visual understanding and generation tokens, which further reduces the visual tokens in unified architectures. 3) A shared-and-decoupled network that enables mutual improvements across tasks while meeting the task-specific modeling requirements. 4) A mixture-of-experts mechanism adopted for visual understanding encoder, which substantially improves perceptual capabilities with a few parameters increase. Extensive experiments have shown that EMMA-4B can significantly outperform state-of-the-art unified multimodal approaches (e.g., BAGEL-7B) in both efficiency and performance, while also achieving competitive results compared to recent multimodal understanding and generation experts (e.g., Qwen3-VL and Qwen-Image). We believe that EMMA lays a solid foundation for the future development of unified multimodal architectures.

1. Introduction

Unified multimodal approaches span architecture, task-format, and learning-paradigm designs, but their directions have not converged. EMMA targets architecture unification with shared compression and task-specific network design, and reports strong benchmark performance with fewer visual context tokens.

1. Introduction

  • Unified multimodal approaches include architecture-format, task-format, and learning-paradigm unification, and these directions have not converged.The approaches are described as having distinct advantages and limitations.Source: §1 · pp. 1–2
  • EMMA uses a 32x-compression autoencoder, channel-wise visual-token concatenation, shallow-layer sharing with deeper-layer decoupling, and a visual-understanding MoE.The design aligns image compression across branches, reduces visual context tokens, supports cross-task parameter sharing, and targets varied image perception.Source: §1 · p. 2
  • 73.0 on MMVet and 0.93 on GenEval for EMMA-4B compare with BAGEL-7B’s 67.2 on MMVet and 0.88 on GenEval; EMMA uses 20% of BAGEL’s visual context tokens.The comparison is reported for cross-model interaction generation tasks such as image editing; Table 1 covers understanding, text-to-image generation, and editing benchmarks.Source: §1 · p. 2, Table 1 · p. 2
  • EMMA’s contributions are framed as efficiency through fewer visual tokens, performance against unified models, and competitive results against understanding and generation specialists.The reported image-editing reduction is 5x versus recent unified models such as BAGEL.Source: §1 · p. 4

2. Method

EMMA fuses semantic and detailed visual information while limiting token growth, then uses a shared-and-decoupled network to support the different modeling needs of understanding and generation. Its visual encoder combines 32x compression with a routed STEM expert.

2.1. Visual Encoder

  • EMMA’s visual encoder uses SigLIP2 with pixel shuffling and a 32x-compression DCAE, reducing generation visual tokens to one quarter of the count required by 8x-compression autoencoders with 2x2 merging.A 1024×1024 image becomes 1024 visual tokens in the understanding branch; experiments are reported to retain competitive generation quality.Source: §2.1 · p. 4
  • Matching the branches’ 32x compression lets EMMA concatenate corresponding visual tokens by channel, fusing semantic and detailed information without increasing token count.The passage reports a 5x visual-token reduction in image editing versus BAGEL while achieving superior performance across understanding, generation, and editing benchmarks.Source: §2.1 · p. 4
  • A router sends STEM images to a STEM expert initialized from the versatile expert; the expert is trained during the final tuning phase.Adding approximately 50M parameters and a few fine-tuning steps improves understanding benchmarks while preserving generation and editing capabilities.Source: §2.1 · p. 5

2.2. Network Architecture Design

  • EMMA shares shallow network layers across understanding and generation while keeping deeper layers independent, reflecting their common and task-specific modeling needs.Some shallow-layer parameters, including value projection layers, remain task-specific; self-attention provides cross-modal interaction, and both branches’ layer weights are initialized from Qwen3-4B [62].Source: §2.2 · pp. 5–6
  • For understanding, EMMA uses next-token prediction; for generation, it uses flow matching with velocity prediction, after fusing visual semantic and detail features by channel concatenation.It applies 2D positional encoding to visual tokens before the LLM, then processes text and visual tokens with 1D RoPE.Source: §2.2 · p. 6
  • Understanding tasks use causal masking for text and visual tokens, while generation text attends only to previous tokens and image tokens also attend within the same image.The hybrid attention strategy follows Transfusion [68].Source: §2.2 · p. 6

3. Data Construction

EMMA’s end-to-end training uses multimodal understanding, text-to-image generation, and instruction-editing data. The data mix combines sourced and constructed datasets, with synthesized editing pairs filtered for instruction adherence and portrait similarity.

3. Data Construction

  • EMMA’s end-to-end training combines multimodal understanding, text-to-image generation, and instruction-editing data.Table 2 presents data and hyperparameter statistics for I2T, T2I, and IT2I.Source: §3 · p. 6, Table 2 · p. 7
  • Understanding data include alignment, pre-training, supervised-finetuning, quality-tuning, STEM-expert, and router-tuning sets assembled from open-source and internal sources.The sets include image-text pairs, image-question-answer triplets spanning tasks such as document parsing and chart recognition, 15M STEM examples, and 3M router examples.Source: §3 · p. 6
  • Text-to-image data comprise pre-training, supervised-finetuning, and quality-tuning sets, with supervised examples filtered for 1K-plus resolution and aesthetic score.The supervised set balances general and portrait images, and synthesized text-rendering images address scarce text-rendering data.Source: §3 · p. 6
  • Editing data combine open-source sets with synthesized pairs covering varied edits, and the construction pipeline filters pairs for instruction adherence and portrait face similarity.The pipeline generates instructions and edited images, reverses instructions to form paired examples, and uses a VLM for adherence filtering; GPT-Image-Edit-1.5M is excluded because it disrupts character consistency.Source: §3 · pp. 6–7

4. Training Details

Training proceeds through alignment, pre-training, supervised fine-tuning, quality tuning, and separate STEM-expert and router tuning. The stages move from adapter alignment to joint task training and then targeted component tuning.

4. Training Details

  • During alignment, the visual encoders and unified model are frozen while only the understanding adapter is tuned on 512×512 images.The generation adapter is randomly initialized and trained during pre-training.Source: §4 · p. 7
  • Pre-training tunes all EMMA parameters except DCAE on 512×512 images, with understanding and generation samples at a 1:1 ratio.The learning rate is 1e−4.Source: §4 · p. 7
  • Supervised fine-tuning supports native-resolution understanding images and bucketed 1K-resolution generation images, then balances sampling across tasks and categories.The supplied training description includes approximately 50M T2I examples balanced between portrait and general images, followed by an I2T and T2I and IT2I mixture at 1:1:1.Source: §4 · pp. 7–8
  • The editing-data pipeline generates instruction-edit pairs, reverses the instructions for paired examples, and filters for adherence, with face-similarity checks for portraits.A VLM assesses whether edited images follow instructions.Source: Figure 3 · p. 8
  • Quality tuning jointly trains T2I, I2T, and IT2I at a 1:1:1 batch ratio, followed by expert-only STEM and router tuning with other parameters frozen.The STEM expert uses 12M STEM examples and the router uses 3M selected examples; their learning rates are 4e−6 and 1e−4, respectively.Source: §4 · p. 8

5. Experiments

EMMA reports stronger understanding, text-to-image generation, and editing results than unified-model baselines, alongside reduced visual-token use. Its evaluation also reports emergent Chinese-instruction and complex-editing capabilities and identifies a limitation in GEdit's subject-consistency assessment.

5.1. Comparisons on Understanding Tasks; 5.2. Comparisons on Text-to-Image Generation Tasks; 5.3. Comparisons on Editing Tasks

  • EMMA-4B improves MMVet by 5.8% over BAGEL-7B, scores 0.91 on GenEval without prompt rewriting or reinforcement learning, and has a slight GEdit advantage.On GenEval, EMMA's score exceeds BAGEL-7B (0.82) and Qwen-Image (0.87).Source: §5.1 · p. 8, §5.2 · p. 9, §5.3 · p. 9

5.1. Comparisons on Understanding Tasks

  • The vision-encoder MoE adds an average 0.4% across 11 understanding datasets; EMMA exceeds Qwen3-VL by 0.4% and InternVL3.5 by 2.6%.EMMA-4B is reported to surpass BAGEL-7B by 5.8% on MMVet.Source: §5.1 · pp. 8–9

5.2. Comparisons on Text-to-Image Generation Tasks

  • EMMA scores 0.91 on GenEval without prompt rewriting or reinforcement learning, exceeding BAGEL-7B (0.82) and Qwen-Image (0.87).The paper describes this as the first 0.91 GenEval score without either technique.Source: §5.2 · p. 9

5.3. Comparisons on Editing Tasks

  • On GEdit, EMMA has a slight advantage over unified models while using one-fifth of BAGEL-7B's visual tokens for reference images.The authors cite relatively limited image-text interaction data (12M versus BAGEL's 65M) and editing complexity as possible reasons for the marginal improvement.Source: §5.3 · p. 9
  • GEdit's vision-language-model evaluator cannot assess subject consistency, and the authors exclude GPT-Image-Edit-1.5M because they observed it disrupts that consistency.The paper calls for more accurate image-editing evaluation criteria.Source: §5.3 · p. 11

5.4. Emergence of Capabilities

  • Without Chinese text-to-image or editing training data, EMMA supports those tasks from Chinese instructions; it also handles complex editing instructions despite single-instruction editing training.The paper attributes these observations respectively to Chinese data in multimodal-understanding datasets and multimodal chain-of-thought data.Source: §5.4 · p. 11

6. Conclusion

EMMA unifies multimodal understanding, generation, and editing, combining compressed visual tokens with channel-wise concatenation, a mixture-of-experts understanding encoder, and a shared-and-decoupled architecture. It outperforms current unified architectures across multiple benchmarks, and its editing examples include Chinese and complex instructions despite training data consisting almost entirely of single English instructions.

  • EMMA combines an efficiently compressed autoencoder and channel-wise concatenation with a mixture-of-experts visual understanding encoder and a shared-and-decoupled architecture.The architecture is designed to mutually benefit multimodal tasks while meeting their distinct modeling requirements.Source: §6 · pp. 11, 13
  • EMMA’s editing examples show character consistency and handling of Chinese and complex instructions, although almost all editing training data used single English instructions.The authors describe these results as evidence of strong generalization to complex multimodal tasks.Source: Figure 5 · p. 12
  • The GEdit-Bench-EN comparison table excludes methods trained with GPT-Image-Edit-1.5M.Source: Table 6 · p. 13
  • EMMA outperforms current unified architectures across multiple benchmarks in multimodal understanding, generation, and editing.Source: §6 · p. 13
Loading 2512.04810v5…