What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

1 INTRODUCTION

• Prior approaches use generative supervision or explicit visual intermediates, while direct use of native generation-side representations remains underexplored.Existing fine-grained methods also rely on language-aligned encoders that can discard visual evidence before language modeling.Source: §1 · pp. 1–2
• Generation-side representations respond more strongly to subtle semantic changes, capture changes missed by understanding-side representations, and rely on spatial organization to retain this advantage.Source: §1 · p. 2
• GAFU outperforms same-data SFT in all 15 model–benchmark combinations, with average gains of 3.56–5.98 percentage points across five benchmarks.The framework is evaluated across UMM architectures and general, spatial, and domain-specific visual understanding tasks.Source: §1 · p. 2
• GAFU compresses native generation-side representations into compact G tokens and selectively retrieves task-relevant evidence for the understanding pathway.It also uses pairwise preference learning to promote sample-relevant evidence while preserving original understanding capability.Source: §1 · p. 2

2.1 UNIFIED MULTIMODAL MODELS

• UMMs have evolved from shared dense backbones toward specialized visual pathways, reflecting different representation needs for understanding and generation.Understanding relies more on high-level semantics, while generation requires richer visual details; capability gaps and limited transfer remain.Source: §2.1 · pp. 2–3

2.2 UNDERSTANDING–GENERATION COLLABORATION IN UNIFIED MULTIMODAL MODELS

• Understanding-guided generation is a relatively mature paradigm, while generation-assisted understanding remains less systematically studied and relies mainly on generative supervision or generated intermediates.These approaches can constrain representation learning through indirect objectives or add a costly, lossy intermediate generation process.Source: §2.2 · p. 3

3.1 SYSTEMATIC ANALYSIS OF VISUAL REPRESENTATIONS

• The analysis compares six UMMs’ generation- and understanding-side representations across sensitivity, complementarity, and spatial dependence using 2,084 minimally changed image pairs.The pairs span object, attribute, count, and relation domains.Source: §3.1 · p. 3

3.1.1 FINE-GRAINED SEMANTIC SENSITIVITY

• Generation-side representations show higher control-corrected sensitivity in 24 model–domain combinations, with advantages varying by domain and remaining positive across tested input resolutions.The advantage is smallest for count and larger for attribute, object, and relation.Source: §3.1.1 · p. 4, Figure 1 · p. 4

3.1.2 COMPLEMENTARITY ANALYSIS

• Generation-side representations are uniquely strongly responsive more often than understanding-side representations in all 24 model–domain combinations, revealing complementary task-relevant evidence.This advantage extends beyond overall sensitivity to changes missed by the understanding-side representation.Source: §3.1.2 · p. 4, Figure 1 · p. 4

3.1.3 SPATIAL DEPENDENCE

• Spatial permutation reduces the sensitivity gap in all 24 model–domain combinations, with the strongest dependence where fine-grained evidence must remain locally grounded.The reduction is smallest for count and extends to attribute and object changes, not only relations.Source: §3.1.3 · pp. 4–5, Figure 1 · p. 4

3.2 GENERATION-ASSISTED UNDERSTANDING FRAMEWORK

• GAFU compresses and aligns generation-side information into compact G tokens, then enables task-conditioned retrieval by the understanding pathway.The framework comprises representation compression and alignment followed by task-conditioned injection.Source: §3.2 · p. 5, Figure 2 · p. 6

3.2.1 GENERATION REPRESENTATION COMPRESSION AND ALIGNMENT

• An encoder–decoder compresses generation representations into fixed-size G tokens, using spatial positional encoding to aggregate information while retaining spatial structure.Training freezes the native UMM pathways and optimizes reconstruction, spatial binding, and relation alignment losses.Source: §3.2.1 · p. 5

3.2.2 TASK-CONDITIONED INJECTION AND FINE-TUNING

• Task-conditioned cross-attention uses evolving text states to retrieve G-token evidence, while pairwise CE-space preference learning favors matched-sample G over shuffled G.A forward KL loss preserves the original model’s output distribution on the mismatched-G branch; training also uses gradient controls and decays KL weight.Source: §3.2.2 · pp. 6–7

4.1 EXPERIMENTAL SETUP

• The experiments use BAGEL, Janus-Pro, and Lance, with general and domain-specific datasets and Base and same-data SFT comparison models.Compression/alignment training uses 1M Cambrian images; downstream data include FineVision, VSR, and PathVQA.Source: §4.1 · p. 7

4.2 GENERAL FINE-GRAINED VISUAL UNDERSTANDING

• 5.23–7.85 points on VStar-Bench, 2.61–4.44 on OmniSpatial, and 3.67–5.67 on MMVP are GAFU’s gains over SFT across three UMMs.The gains span models from 3B Lance to 14B BAGEL.Source: §4.2 · p. 7
• Across all nine model–benchmark combinations, GAFU’s improvement over SFT exceeds SFT’s improvement over the Base model.On MMVP, SFT brings no improvement to Janus-Pro and slightly degrades Lance, while GAFU achieves further gains.Source: §4.2 · p. 7

4.3 DOMAIN-SPECIFIC VISUAL UNDERSTANDING

• PathVQA gains are 1.12–7.65 points and VSR-Bench gains are 1.09–6.15 points over SFT across all three models.The gains persist after domain-specific adaptation, including where SFT already improves PathVQA substantially and where its VSR-Bench gains over Base are marginal.Source: §4.3 · p. 8

4.4 ABLATION STUDY

• Removing spatial binding lowers OmniSpatial by 1.37 points, while removing relation alignment lowers VStar-Bench by 1.57 points; using both beats either alone on MMVP by 4.67–5.34 points.The ablations are conducted on BAGEL under the general setting.Source: §4.4 · p. 8, Table 2 · p. 8
• Text hidden-state queries outperform visual hidden-state queries by 1.05–4.67 points, while learnable queries lag by 4.71 and 4.00 points on VStar-Bench and MMVP.Learnable queries are only 0.33 points behind on OmniSpatial; the results distinguish stable spatial structure from task-specific evidence selection.Source: Table 2 · p. 8, §4.4 · p. 9
• 10-layer sparse injection stays within 0.00–1.67 points of full-layer performance, while the full CE + DPO + KL objective improves all three benchmarks by 3.14–12.27 points over CE + DPO.DPO without KL degrades OmniSpatial by 11.09 points; the full objective combines preference learning with KL regularization.Source: Table 2 · p. 8, §4.4 · p. 9

• GAFU preserves and organizes generation-side information through compression, spatial binding, and relation alignment, then retrieves it through task-conditioned injection.Pairwise preference learning promotes use of sample-relevant evidence.Source: §5 · p. 9
• Across UMM architectures, GAFU consistently outperforms same-data SFT on general, spatial, and domain-specific visual understanding tasks.Maximum relative gains are 20.00% on VStar-Bench, 17.94% on OmniSpatial, and 8.62% on VSR-Bench.Source: §5 · p. 9
• The paper concludes that GAFU converts complementary native generation-side information into usable visual evidence for the understanding pathway.Source: §5 · p. 9

• Generative AI tools assisted with translation and editing the paper to improve readability.Source: AI USE STATEMENT · p. 10
• The authors state that the tools were not used to generate datasets, develop methods, implement experiments, interpret results, or draft paper sections.Source: AI USE STATEMENT · p. 10
• The authors reviewed all AI-assisted text and take full responsibility for the final content.Source: AI USE STATEMENT · p. 10

• The study uses existing datasets and benchmark suites, including Cambrian, FineVision, VStar-Bench, OmniSpatial, MMVP, VSR-Bench, VisMin, and PathVQA.Source: ETHICS STATEMENT · p. 10
• PathVQA medical question-answering results are intended solely for research benchmarking and should not support clinical decisions.Source: ETHICS STATEMENT · p. 10
• The authors report no privacy-sensitive data, human subjects, new data collection, dataset release, potentially harmful insights, conflicts of interest, or external sponsorship.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The reproducibility statement directs readers to the main text and appendices for analysis, GAFU training and evaluation, and supporting implementation details.Appendix A covers representation extraction and analyses; Appendices B–D specify architecture, objectives, data, configurations, and evaluation.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 REPRESENTATION EXTRACTION AND PREPROCESSING

• Lance uses merged Qwen visual tokens for RU and frozen Wan2.2-VAE posterior means for RG.Their native grids are 16 × 16 and 32 × 32, respectively.Source: §A.1 · p. 15
• Representations preserve spatial arrangement and are aligned to a 16 × 16 grid; VAE latents are first grouped into non-overlapping 2×2 patch features.Adaptive average pooling standardizes the grid without requiring understanding- and generation-side features to share a dimension.Source: §A.1 · p. 15

A.2 CONTROL-CORRECTED CONSTRUCTION

• Control images match the semantic edit’s linear-RGB RMS within the edited region, remain in range, and are chosen for LPIPS proximity to the original.Candidate changes use structural-preserving natural degradations; pixels outside the edited-region mask remain unchanged.Source: §A.2 · pp. 15–16

A.3 COMPLEMENTARITY ANALYSIS

• 28.18% of samples have generation-side-only strong responses on average, versus 2.48% understanding-side-only, a net difference of 25.70 percentage points across 24 model–domain combinations.The generation-side-only response is highest for attribute and object changes, at 35.66% and 33.19%.Source: §A.3 · p. 16, Table 3 · p. 17

A.4 RESOLUTION STABILITY

• The control-corrected generation–understanding response gap remains positive for all six models across domains and tested input resolutions.The comparison repeats the procedures at resolutions of 128, 256, and 512.Source: §A.4 · p. 16, Table 4 · p. 18

A.5 SPATIAL DEPENDENCE

• Spatial permutation reduces the generation–understanding response gap in all 24 model–domain combinations.This supports the dependence of the generation-side advantage on spatial organization.Source: §A.5 · p. 17, Table 5 · p. 19

B COMPRESSION AND ALIGNMENT

• The Stage 1 encoder aggregates generation information with 256 learnable queries and spatial positional encoding to form fixed-length G tokens.The decoder reconstructs the native generation representation from G; spatial binding additionally trains tokens to predict features at corresponding locations.Source: B COMPRESSION AND ALIGNMENT · pp. 17–18
• Figure 4 compares original images with reconstructions from native generation representations and compressed G tokens.The native-representation reconstruction is identified as the VAE upper bound.Source: Figure 4 · p. 18, B COMPRESSION AND ALIGNMENT · p. 18

C TASK-CONDITIONED INJECTION AND FINE-TUNING

• G tokens are projected into the understanding pathway, where text states retrieve information through internal cross-attention and residual injection.Training combines matched-sample answer supervision, a pairwise CE objective comparing matched and mismatched memories, and a shuffle-G KL regularizer against the frozen native model.Source: C TASK-CONDITIONED INJECTION AND… · pp. 19, 21

D.1 DATA

• The training set includes 1,344,016 relevant FineVision samples, while VSR-Bench and PathVQA use their official training splits.The compression stage uses one million stratified Cambrian images; examples span multi-object reasoning, local spatial relations, and pathological visual question answering.Source: §D.1 · p. 21, Figure 5 · p. 22, Figure 6 · p. 22

D.2 TRAINING

• With CE + DPO + KL, DPO promotes lower answer-token CE for matched G tokens, while KL constrains output shifts on the mismatched-memory branch.Validation compares matched and shuffled G branches; CE-only shows limited sample-specific G use, and KL regularization protects the shuffled branch from uncontrolled deviation.Source: §D.2 · p. 23

D.3 EVALUATION

• Figure 8 compares answer-token CE and the shuffled-minus-real CE gap for CE only, CE + DPO, and CE + DPO + KL.A positive gap means lower answer-token cross-entropy for matched G tokens.Source: Figure 8 · p. 24
• VStar-Bench, OmniSpatial, and MMVP select candidate answers using average answer-token or token cross-entropy, while VStar-Bench reports choice accuracy.The protocols use 191 VStar-Bench samples, 1,533 OmniSpatial samples, and 300 MMVP questions.Source: §D.3 · p. 24
• VSR-Bench uses True/False greedy generation, while PathVQA uses greedy decoding and normalized exact-match accuracy.VSR-Bench evaluation uses 2,195 samples; PathVQA uses all 6,719 official test samples.Source: §D.3 · p. 24

E LIMITATIONS

• GAFU targets mainstream dual-pathway visual encoders, so other UMM architectures may require adaptation for incompatible pathway designs or representation interfaces.Its current scope does not cover all UMM architectures.Source: E LIMITATIONS · p. 24
• GAFU's benefits may vary across models and tasks because its effectiveness depends on the quality of native generation representations.Source: E LIMITATIONS · p. 24
• Extracting, projecting, and injecting generation tokens adds inference computation and memory overhead relative to using only the native understanding pathway.Source: E LIMITATIONS · p. 24