Source-linked AI summary
Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
Zhongbin Guo, Jiahao Xie, Dongling Xiao, Qianle Wang, Ruiqi Lu, Xiaomin He, Wanxuan Sun, Cheng Yang
TL;DR
Visual generation supervision is difficult to transfer to pretrained multimodal understanding models because generative targets often differ from their continuous representations. GAS uses NEP with a decoupled MoT branch as auxiliary training supervision, improving aggregate understanding with zero inference overhead.
Problem
Existing visual-generation objectives use targets that differ from the continuous representations consumed by visual understanding models, leaving their integration unresolved.
Method
GAS uses continuous Next Embedding Prediction within a decoupled Mixture-of-Transformers architecture to provide generation supervision while isolating understanding layers from generation gradients.
Results
GAS delivers consistent aggregate gains across diverse multimodal understanding benchmarks, with stronger improvements from generation tasks more correlated with understanding.
Takeaways & Limitations
Generation can serve as removable auxiliary supervision for stronger multimodal understanding without inference-time overhead.
Takeaways & Limitations
Whether GAS generalizes to longer contexts, video generation, and modalities such as audio or 3D perception remains future work.
Abstract
from arXiv · showhide
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress, visual understanding and generation are typically treated as divergent objectives. Existing unified frameworks often rely on discrete visual tokenization or diffusion objectives whose generative targets differ from the continuous representations consumed by visual understanding models, making direct transfer to enhance existing pretrained MLLMs non-trivial. In this work, we present GAS, a generation-guided training framework that reinterprets visual generation as auxiliary supervision for representation learning. Concretely, GAS adapts Next Embedding Prediction (NEP) as a cross-modal generation paradigm within a decoupled Mixture-of-Transformers (MoT) architecture. By maintaining a shared lower trunk and parallel upper layers, GAS lets generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention while shielding the upper understanding layers from direct generation gradients. To maximize this synergy, we further construct highly correlated generation tasks that demand deep cognitive grounding rather than generic synthesis alone. Across model scales and training stages, GAS improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension. Crucially, because the auxiliary generation branch is discarded after training, these gains incur zero inference overhead. Extensive controlled comparisons and representation-level analyses further clarify when and why generation-guided training benefits understanding, and demonstrate the feasibility of generation-guided training as a practical route to stronger multimodal understanding.
1. Introduction
GAS treats visual generation as auxiliary training supervision for multimodal understanding, using a removable MoT generation branch with continuous NEP targets to enrich shared visual representations without inference overhead. It combines automatically synthesized, cognitively correlated generation tasks with systematic analyses, improving aggregate understanding across 2B- and 4B-parameter models while producing task-dependent benchmark gains.
- Motivation: Text-only autoregressive supervision provides only indirect visual supervision, limiting retention of fine-grained spatial, pixel-level, interaction, and appearance information.Natural language cannot precisely encode many aspects of visual structure, so representations are shaped by what language can articulate.
- Framework: GAS uses a dedicated MoT generation branch with independent transformer parameters so generation gradients refine shared visual representations while preserving the understanding branch.The framework treats generation as training-time auxiliary supervision rather than an end goal.
- Data and analysis: About 10M generation samples span five categories and 15 subtasks, generated through an automated synthesis pipeline without manual annotation.The tasks range from pixel-level perception to high-level reasoning.
- Framework: Continuous NEP supervision and a removable generation branch transfer generative learning to the deployed understanding model at zero inference overhead.The generation branch is entirely discarded at inference time.
- Findings: Tasks with stronger latent correlations to understanding deliver the largest gains, while combining tasks yields complementary improvements on perception- and reasoning-oriented benchmarks.GAS improves aggregate visual understanding on 2B- and 4B-parameter backbones, although transfer varies by benchmark.
- Data and analysis: Systematic analyses examine parameter isolation, layer-wise supervision, task contributions, representations, matched budgets, and repeated-run statistics to clarify when and how generation enhances understanding.These analyses address whether objectives conflict and which representations and generation tasks benefit comprehension.
2. Methodology
GAS uses Next Embedding Prediction to provide generation supervision in the same continuous representation space as understanding, while a decoupled Mixture-of-Transformers architecture shares lower layers but isolates upper-layer objectives. Its generation tasks are designed to provide complementary, comprehension-correlated supervision for fine-grained visual structure.
- Next Embedding Prediction: NEP formulates image generation as autoregressive prediction of continuous embeddings in the LLM’s understanding space, avoiding a separate visual-token or synthesis-latent interface.Targets are extracted with the vision encoder and projector, and the loss compares ℓ2-normalized predicted and target embeddings using cosine distance.
- Next Embedding Prediction: The conditional NEP setup predicts target-image embeddings from multimodal contexts containing source images and task instructions across diverse generation tasks.Unlike single-image self-supervision, the formulation supports instructions such as segmentation or grounding.
- Decoupled Architecture: A decoupled MoT architecture shares a lower trunk, assigns understanding and generation to parallel upper branches, and supervises them solely with their respective losses.The generation branch uses NEP loss, while the understanding branch uses text cross-entropy; generation-branch parameters are removed after training.
- Mechanism of Synergy: Asymmetric gradient flow enriches shared intermediate representations with fine-grained visual detail while shielding upper understanding layers from generation gradients.The upper layers nevertheless adapt under understanding loss to exploit the enriched features, with reported sharper attention to question-relevant regions.
- Generation Task Design: GAS emphasizes comprehension-correlated generation tasks because generic synthesis can offer little or negative understanding benefit, whereas diverse tasks provide complementary supervision.On Counting & Spatial benchmarks, combining all categories tends to outperform the best single-task gain.
3. Experiments
GAS is evaluated across four visual-understanding dimensions using paired baselines and GAS models, with gains concentrated in reasoning, counting, and spatial behavior. The generation branch is discarded after training, so these improvements add zero inference cost.
- Evaluation Dimensions: Evaluation spans general perception, visual reasoning, counting and spatial understanding, and video understanding across 14 multimodal benchmarks.The benchmarks cover multi-image comprehension, charts and diagrams, object counting, spatial relations, and temporal reasoning.
- Efficiency: All GAS improvements incur zero additional inference cost because the generation branch is discarded after training.The auxiliary generation components are removed at inference.
- Main Results: At 2B, DynaMath rises from 46.2 to 47.9 (+1.7pp), while MathVista rises from 54.4 to 56.4 (+2.0pp).These paired GAS/baseline gains provide direct evidence of stronger mathematical and visual reasoning.
- Main Results: At 2B, CountBenchQA increases from 87.7 to 90.1, CV-Bench-2D from 69.9 to 73.2, and VisuLogic from 26.8 to 28.6.The gains are aligned with fine-grained counting, spatial perception, and visual reasoning.
- Main Results: The sole noted exception is VisuLogic at 4B (23.6), a benchmark-specific regression that does not transfer to other reasoning suites.The result suggests idiosyncratic sensitivity of this puzzle-style benchmark to backbone scale.
- Qualitative Results: Qualitative cases show improved spatial localization, recognition of small objects and subtle differences, and visual reasoning chains that reference image evidence.Examples cover CV-Bench, DynaMath, VisuLogic, and RealWorldQA and align with the quantitative gains.
4. Analysis
GAS improves understanding across training stages when generation tasks are correlated with downstream capabilities and generation gradients are isolated from the understanding pathway. Its gains reflect improved visual retention, fine-grained spatial representations, and focused attention, with the strongest design balancing spatial and semantic features near the network midpoint.
- Training-stage analysis: GAS is stage-agnostic, yielding its largest gains from scratch while providing complementary improvements after pretraining and consistent benefits after SFT.From scratch, gains reach +1.00pp Overall, +1.97pp Perception, and +1.25pp Reasoning, while Count&Spatial decreases by −1.77pp; after pretraining, gains remain +0.48pp Overall and +0.90pp Reasoning.
- Objective and gradient routing: Autoregressive NEP with MoT produces the strongest aggregate transfer and best Perception and Video scores, whereas direct NEP on shared upper layers broadly degrades performance.Position-aligned no-shift prediction remains stronger on Count&Spatial, showing that objective choice and gradient routing affect capability-specific outcomes.
- Ablation analysis: Generation data alone, additional training, or an extra branch cannot explain GAS gains; transferable learning requires task-conditioned NEP with an isolated upper generation pathway.Understanding-only continuation improves Overall modestly, mixing generation samples without image-target supervision provides no benefit, and MoT without generation loss underperforms GAS.
- Task correlation: Task correlation determines transfer: T2I gives the highest single-task Overall gain (+0.49pp), while Segmentation and Grounding improve Count&Spatial by +2.00pp and +2.37pp respectively.The combined All configuration reaches 75.72 on Count&Spatial and 20.1 on MathVision, exceeding the strongest single-task settings; rewritten T2I prompts improve Overall by +0.71pp, unlike generic T2I, which gains +0.24pp and decreases Perception by 0.75pp.
- Architecture and optimization: Supervision branching near L/2 with progressive warmup provides the best balance, while unfreezing the ViT degrades performance and corrupts pretrained visual features.Layer 14 outperforms layers 8 and 20 on overall metrics and perception, layer 8 reaches 75.88 on Count&Spatial, and warmup adds +0.40pp Overall and +0.99pp Reasoning; unfreezing reduces Overall by 1.65pp and CountBenchQA from 86.4 to 79.7.
- Representation analysis: Representation analyses attribute GAS improvements to higher deep-layer visual information retention, stronger RefCOCO spatial representations, and sharper task-focused attention throughout reasoning.The baseline’s visual similarity attenuates beyond L/3 and its deeper attention becomes diffuse, whereas GAS retains visual features, gains increasingly on RefCOCO from layer 8 onward, and maintains focused attention.
5. Related Works
Related work spans conventional MLLMs, unified multimodal models, and generation-assisted understanding methods. These studies expose a recurring tension: generation can enrich visual representations, but conflicting gradients often degrade understanding when it is not the primary optimization target.
- MLLMs: MLLMs typically align pretrained visual encoders with language backbones, with Flamingo and InstructBLIP among early systems and LLaVA and Qwen-VL among later open-source families.
- Unified multimodal models: Unified multimodal models jointly train understanding and generation, including autoregressive approaches that represent both modalities through shared discrete visual-token spaces.
- Generation-assisted understanding: ROSS, ASVR, UniHetero, and UniMRG explore continuous-feature, discrete-token, embedding, and RGB-reconstruction objectives for connecting generation with visual understanding.
- Limitations: Existing unified methods often prioritize generation quality or scale-driven capabilities, while generation objectives can harm understanding through conflicting gradient signals.
6. Conclusion and Future Work
GAS uses visual generation as removable, training-time auxiliary supervision to improve multimodal understanding without inference overhead. Its analyses link gains to visual retention, focused attention, and task-dependent transfer, while future work targets broader contexts, modalities, and adaptive training.
- Core contribution: GAS improves aggregate multimodal understanding by using visual generation as training-time auxiliary supervision, with NEP aligned to the LLM input space and a removable MoT branch preserving zero inference overhead.The framework mitigates direct optimization conflict while retaining the generation branch only during training.
- Analysis: Representation analyses associate GAS gains with better visual retention, more focused attention, and task-dependent transfer shaped by generation-task design.Across diverse understanding benchmarks, the framework delivers consistent gains.
- Implications: The results position visual generation as a carefully structured supervisory signal for building stronger understanding models rather than an inference-time capability to preserve.The paper presents generation-guided training as a principled and practical paradigm for advancing multimodal understanding.
- Future work: Future work includes longer-context scenarios, video-level temporal generation, adaptive task weighting, and testing whether generation-understanding synergy generalizes to audio and 3D perception.Adaptive weighting would dynamically balance generation categories during training.
A. Additional Controlled Experiments · A.1. Repeated-Run Robustness Across Data Scales
Repeated GAS runs at 2.5B and 20B tokens show aggregate transfer remains reproducible, with mean Overall performance exceeding both single-run understanding-only controls at each scale. Capability-level variation is larger, especially for Reasoning at 2.5B and Count&Spatial at 20B.
- A.1. Repeated-Run Robustness Across Data Scales: GAS is repeated three times at 2.5B tokens and twice at 20B tokens.These repeated runs evaluate robustness across training-data scales.
- A.1. Repeated-Run Robustness Across Data Scales: At each scale, GAS’s mean Overall score remains above both single-run understanding-only controls.The comparison includes the original and longer-trained understanding-only controls.
- A.1. Repeated-Run Robustness Across Data Scales: The repeated-run results therefore support reproducible aggregate transfer from GAS training.This conclusion follows from the mean Overall comparison across both data scales.
- A.1. Repeated-Run Robustness Across Data Scales: Variation is larger for individual capabilities than for aggregate performance, with the most notable dispersion in Reasoning at 2.5B tokens.The passage identifies capability-level variability as a limitation of the otherwise consistent aggregate result.
- A.1. Repeated-Run Robustness Across Data Scales: Count&Spatial shows especially large variation among individual capabilities at 20B tokens.This capability is highlighted alongside Reasoning at 2.5B as having notable variability.
- A.1. Repeated-Run Robustness Across Data Scales: Table 9 reports GAS as mean ± sample standard deviation, while baseline controls are single runs.Bold values mark the best reported result within each regime and capability.
A.2. Aligned Comparison with Reconstructive Objectives
Using the same Qwen3-VL-2B backbone, data mixture, and optimization setting, the comparison finds that adapted reconstructive objectives do not automatically transfer to generation-augmented training. GAS’s task-conditioned output prediction achieves the strongest aggregate result.
- A.2. Aligned Comparison with Reconstructive Objectives: The study reproduces ROSS-style and ASVR-style objectives with the same Qwen3-VL-2B backbone, 2.5B understanding plus 2.5B generation data, and optimization setting as GAS.
- A.2. Aligned Comparison with Reconstructive Objectives: On generation samples, the adapted controls reconstruct assistant output images using continuous appearance features or discrete semantic tokens.
- A.2. Aligned Comparison with Reconstructive Objectives: Input-reconstruction objectives designed for understanding data do not automatically transfer to the generation-augmented setting, whereas GAS’s task-conditioned output prediction gives the strongest aggregate result.
B. More Qualitative Cases of GAS
The qualitative cases illustrate GAS solving visual reasoning tasks by identifying structural patterns, graph properties, and objects in images before producing answers D, 2, and C. These examples span visual logic, mathematical graph coloring, and real-world counting.
- Qualitative Case 4: VisuLogic: In VisuLogic, the model identifies alternating triangle and quadrilateral pairs with line counts of 6 and 8, selecting option D.It matches the overlapping two-shape composition and contour connectivity to the pattern.
- Qualitative Case 5: DynaMath: In DynaMath, the model classifies the graph as bipartite and concludes that its chromatic number is 2.The reasoning uses two disjoint vertex sets with edges connecting only across sets.
- Qualitative Case 6: RealWorldQA: In RealWorldQA, the model counts two visible wooden poles supporting a tree and gives final answer C.It also checks the background and reports no additional poles elsewhere in the image.