Source-linked AI summary
AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Model
Dian Zheng, Manyuan Zhang, Hongyu Li, Kai Zou, Hongbo Liu, Ziyu Guo, Kaituo Feng, Yexin Liu, Ying Luo, Hongsheng Li
TL;DR
Unified multimodal models face conflicting generation and understanding targets, and excessive decoupling risks losing interleaved generation. The paper analyzes cross-modal attention and proposes AIA loss to align task-specific interaction patterns without architecture decoupling; it reports improved generation and understanding performance on Emu3 and Janus-Pro.
Problem
Unified models must reconcile conflicting generation and understanding targets, while architecture decoupling can risk losing interleaved generation and cross-task reasoning.
Source: §1 · p. 1Method
AIA loss aligns layer-wise cross-modal interaction intensity to task-specific patterns during training without architecture decoupling.
Source: §1 · pp. 2–3, §3.2 · p. 5Results
AIA improves generation and understanding performance on Emu3 and Janus-Pro, narrowing the gap with more strongly decoupled methods.
Source: §1 · pp. 2–3, §5 · p. 11Takeaways & Limitations
The results support using attention-pattern alignment to improve both tasks without resorting to architecture decoupling.
Source: §1 · pp. 2–3In unified Emu3, the generation attention pattern captures the correct directional trend across layers, but its actual values remain incorrect and are harder to change.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
Unified multimodal models for image generation and understanding represent a significant step toward AGI and have attracted widespread attention from researchers. The main challenge of this task lies in the difficulty in establishing an optimal training paradigm due to inherent conflicting targets in understanding and generation tasks. To alleviate these conflicts and pursue higher performance, many researchers adopt varying degrees of architecture decoupling (e.g., Double image encoders, MOE/MOT architecture, or frozen MLLM). However, excessive model decoupling can lead to the loss of interleave generation ability, undermining the original intent of unified models. In this work, we aim to explore how to mitigate task conflicts without resorting to model decoupling. Firstly, we analyze why decoupling boosts performance by studying the cross-modal attention behavior of models. We observe that architecture decoupling does not solve task conflicts, but essentially drives models toward cross-modal interaction patterns of task-specific models, as seen in Qwen3-VL and HunyuanImage-3.0, and that the more thorough the decoupling, the more consistent the behavior becomes. Motivated by this observation, we propose Attention Interaction Alignment (AIA) loss, which explicitly learns task-specific multimodal interaction patterns during training. To demonstrate the generalizability of our AIA loss, we apply it to Emu3 and Janus-Pro during SFT and post-training stage respectively. Without bells and whistles, AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
1 Introduction
Unified multimodal models face conflicting generation and understanding targets, while extensive architecture decoupling risks sacrificing interleaved reasoning. The paper analyzes cross-modal interaction patterns and introduces AIA loss to improve both tasks without architecture decoupling.
1 Introduction
- Unified models aim to combine image generation and understanding in one network, but their conflicting feature requirements have motivated decoupling despite risks to cross-modal reasoning and interleaved generation.The paper notes that double image encoders can also require additional decode–encode steps.Source: §1 · p. 1
- As architecture decoupling increases, task interaction patterns increasingly resemble task-specific models, but the negative correlation between generation and understanding persists.The authors interpret this as decoupling shifting each task toward its single-task behavior rather than eliminating the underlying conflict.Source: §1 · p. 2, Figure 1 · p. 2
- AIA loss explicitly aligns layer-wise cross-modal interaction intensity during training without requiring architecture decoupling.The targets are derived from Qwen3-VL-8B for understanding and HunyuanImage-3.0 for generation, using average attention patterns from 100 samples per task and a Huber loss.Source: §1 · pp. 2–3
- AIA improves generation and understanding performance on Emu3 and Janus-Pro while narrowing the gap with more strongly decoupled methods.The authors attribute the performance gains to aligning attention patterns rather than merely distilling knowledge.Source: §1 · pp. 2–3
2 Related Works
Related work traces a shift from unified multimodal models toward architectures that decouple components to address task conflicts, alongside highly task-specific systems. It also distinguishes AIA from conventional knowledge distillation by its use of interaction-intensity statistics and relaxed constraints.
2.1 Unified Multimodal Models (UMMs)
- Unified multimodal models have evolved from shared components toward tokenizer and architecture designs that address task conflicts, but stronger decoupling can weaken unified latent reasoning and require decode–encode steps.The related work also describes partially decoupled MOE/MOT systems and approaches that fix the MLLM while optimizing a diffusion head.Source: §2.1 · p. 3
2.2 Ultra Task-Specific Models
- The paper uses Qwen3-VL and HunyuanImage-3.0 as task-specific interaction references, against a background of distinct architectures for understanding and generation.Qwen3-VL represents understanding, while HunyuanImage-3.0 is trained in a unified manner and is selected for its generation capability.Source: §2.2 · p. 3
2.3 Knowledge Distillation
- Knowledge distillation transfers teacher knowledge to a student [46], and FitNets incorporate intermediate features [47]; AIA instead uses layer-wise interaction intensity without a concurrent teacher model.The paper says rigid distillation constraints can degrade performance because they inherit the generation–understanding conflict in unified models.Source: §2.3 · pp. 3–4
3 Attention Interaction Alignment
The section analyzes how cross-modal attention patterns relate to architecture decoupling, then proposes AIA loss to constrain those patterns during training. It also examines how AIA behaves in Emu3 SFT and Janus-Pro post-training.
3.1 Analysis of Cross-Modal Attention Interaction Pattern in Various Model Architectures
- Directional interaction intensity sums text-token attention per row and averages across layers, image tokens, and heads; 100-sample standard deviations test stability across input lengths and types.Figure 2 illustrates the text-to-image calculation.Source: §3.1 · p. 4, Figure 2 · p. 4, Table 1 · p. 4
- Across architectures, task attention curves are negatively correlated, and greater decoupling makes them more task-specific; Emu3 deviates from single-task patterns in both tasks.Qwen3-VL shows low attention to image tokens, while HunyuanImage-3.0 allocates around 40% of attention to text in its first 80 layers before a sharp decline.Source: §3.1 · pp. 4–5
3.2 Attention Interaction Alignment Loss
- AIA loss matches layer-wise intensities to task-specific targets, grouping magnitudes into sub-stages and using Huber loss to relax exact-value constraints.Layer-specific target boundaries and Huber thresholds are configured, with thresholds adjusted according to layer-wise similarity.Source: §3.2 · p. 5
- The AIA term is combined with next-token-prediction loss; the default coefficient λ is 40.The section sets the number of layers L to 30 for Emu3 and 29 for Janus-Pro.Source: §3.2 · p. 5
3.3 Training Details Under Different Architectures
- With AIA applied during SFT, Emu3 shows nearly identical NTP-loss convergence across loss weights, indicating little effect on pretrained knowledge when targets are appropriate.The plotted curves show NTP loss, excluding AIA loss.Source: §3.3 · p. 5, Figure 3 · p. 6
- Janus-Pro receives AIA during post-training and is highly sensitive to its coefficient, but an appropriately set weight can still achieve the desired effect.This setting starts from final SFT weights.Source: §3.3 · p. 5, Figure 3 · p. 6
4 Experiment
AIA improves understanding, generation, and interleave-reasoning results across the evaluated models. The experiments examine its effects, ablations, and broader implications for unified-model training.
4.1 Experimental Setup
- Training uses open-source image datasets, 1.5M samples per task, and 200K internal samples; Emu3 and Janus-Pro use balanced task data.Emu3 uses approximately 720×720 images and takes about 10 days to train; Janus-Pro uses approximately 384×384 images and takes about one day.Source: §4.1 · p. 6
4.2 Evaluation on the benchmarks
- AIA improves understanding and generation benchmarks for Emu3 and Janus-Pro, narrows gaps with more decoupled models, and improves Janus-Pro interleave reasoning.Understanding and generation results are reported in Table 3; interleave reasoning is evaluated on the “Understanding aids Generation” metric from RealUnify.Source: §4.2 · p. 6, Table 5 · p. 6, Table 3 · p. 7
4.3 Ablation Study
- Further Janus-Pro fine-tuning with the study’s data yields performance comparable to the original model, while attention-target choice matters and traditional distillation does not improve unified training.HunyuanImage-3.0 attention patterns perform best among the tested generation targets; SimpleAR and FLUX yield mediocre results.Source: §4.3 · pp. 7, 9
- AIA improves both tasks and shifts Emu3 and Janus-Pro attention patterns toward task-specific models, though changing Emu3’s generation pattern remains harder.For Emu3, the generation pattern captures the correct directional trend across layers, but its values remain incorrect.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
- The AIA loss coefficient requires balancing, while Huber loss and stage-level intensity provide flexible targets; removing either component drops performance below baseline.The authors attribute the degradation to overly rigid attention constraints impeding training.Source: §4.3 · pp. 8–9
- With AIA, a balanced 1:1 task-data ratio performs best, and AIA outperforms baseline across model sizes and data scales, including low-resource settings.The scale comparison covers 1B and 7B models trained with 600k and 3M data.Source: §4.3 · pp. 9–10, Table 6 · p. 10
4.4 Discussions
- Understanding and generation show mutually exclusive cross-modal interactions within layers across decoupling strategies, and task tokens can cue models to adjust interactions.The discussion frames task-exclusive interaction patterns as behavior to manage with explicit guidance, rather than a conflict necessarily to avoid.Source: §4.4 · p. 10
- A proposed future direction uses a diffusion head to restore fine detail from abstract conceptual images, aligning image and text output information density.The paper presents this as an alternative path toward unified models.Source: §4.4 · p. 10
5 Conclusion
The conclusion identifies task conflict as a central challenge for unified multimodal models and finds that decoupling shifts interaction toward task-specific patterns. AIA learns those patterns without architecture changes and improves generation and understanding across two models and training stages.
5 Conclusion
- Decoupling guides models toward task-specific multimodal interaction patterns, with stronger decoupling yielding stronger alignment to single-task behaviors.The authors frame this as shifting rather than resolving conflicts between generation and understanding.Source: §5 · p. 11
- AIA explicitly learns task-specific interaction patterns without modifying model architectures.The authors apply it to Emu3 during SFT and Janus-Pro during post-training.Source: §5 · p. 11
- Across Emu3 and Janus-Pro, AIA refines cross-modal attention patterns and consistently improves generation and understanding performance.The conclusion reports these outcomes across different training stages and architectures.Source: §5 · p. 11