PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward

Yuwei Niu, Weiyang Jin, Jiaqi Liao, Chaoran Feng, Peng Jin, Bin Lin, Zongjian Li, Bin Zhu, Weihao Yu, Li Yuan

arXiv:2511.20561v3cs.CVcs.CL

TL;DR

Unified multimodal models raise the question of whether understanding can guide generation, but evaluations can blur distinct failure sources and be affected by data leakage. UniSandbox uses controlled synthetic data to investigate this gap, finding that CoT helps with reasoning generation and knowledge transfer, with self-training and query-based architectures showing further potential.

  • Problem

    It remains unclear whether unified models’ understanding capabilities genuinely inform generation, while prior evaluations can conflate knowledge, reasoning, and transfer failures and risk data leakage.

    Source: §1 · p. 1

  • Method

    UniSandbox uses decoupled evaluation and controlled synthetic data to analyze knowledge and reasoning separately and reduce data contamination.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    UniSandbox reveals an understanding-generation gap: CoT helps activate reasoning and transfer newly learned knowledge, self-training can internalize reasoning, and query-based architectures show latent CoT-like properties.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings provide preliminary insights for designing unified architectures and training strategies to address the understanding-generation gap.

    Source: §6 · p. 9

    Synthetic tasks isolate reasoning generation and knowledge transfer rather than the full diversity of real-world text-to-image prompts, and self-training scalability to open-ended generation remains unresolved.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent years have witnessed significant progress in Unified Multimodal Models, yet a fundamental question remains: Does understanding truly inform generation in Unified Multimodal Models? To investigate this, we introduce UniSandbox, a decoupled evaluation framework paired with controlled, synthetic datasets to avoid data leakage and enable detailed analysis. Our findings reveal a significant understanding-generation gap, which is mainly reflected in two key dimensions: reasoning generation and knowledge transfer. Specifically, for reasoning generation tasks, we observe that explicit Chain-of-Thought (CoT) in the understanding module effectively bridges the gap, and further demonstrate that a self-training approach can successfully internalize this ability, enabling implicit reasoning during generation. Additionally, for knowledge transfer tasks, we find that CoT assists the generative process by helping retrieve newly learned knowledge, and also discover that query-based architectures inherently exhibit latent CoT-like properties that affect this transfer. UniSandbox provides preliminary insights for designing future unified architectures and training strategies that truly bridge the understanding-generation gap.

1 Introduction

UniSandbox examines whether understanding informs generation, addressing prior evaluations that blur knowledge, reasoning, and transfer failures and may be affected by data leakage. It finds that Chain-of-Thought (CoT) helps bridge gaps in reasoning generation and knowledge transfer, with self-training and query-based architectures offering further insights.

1 Introduction

  • Prior work tests whether models can use world knowledge to guide image generation, such as deducing “carnation” from a Mother’s Day flower prompt.The cited example illustrates generation guided by a concept inferred from world knowledge.Source: §1 · p. 1
  • Earlier evaluations conflate knowledge deficits, reasoning deficits, and transfer failures, while possible data leakage makes it difficult to identify genuine understanding-generation synergy.These problems also make it hard to determine which architectures or training strategies foster synergy.Source: §1 · p. 1
  • UniSandbox uses decoupled evaluation and synthetic, low-leakage data to isolate knowledge and reasoning and attribute performance beyond memorization.The framework evaluates state-of-the-art unified models.Source: §1 · pp. 1–2
  • CoT bridges reasoning-generation failures and helps transfer newly learned knowledge; self-training internalizes reasoning, while query-based architectures exhibit latent CoT-like properties.Reasoning-driven tasks require image generation from mathematical calculation or logical deduction.Source: §1 · pp. 1–2

2 UniSandbox: Controlled Evaluation

UniSandbox defines understanding through knowledge retrieval and reasoning with procedural rules, then uses controlled synthetic data and representative unified-model architectures for fine-grained evaluation.

2 UniSandbox: Controlled Evaluation

  • Understanding is divided into knowledge, the memory and retrieval of factual information, and reasoning, the application of procedural rules.This decomposition helps attribute failures to invoking new knowledge or applying logical reasoning when guiding image generation.Source: §2 · p. 2
  • Controlled synthetic training and evaluation data are designed to reduce contamination from overlap with pre-training data.The setup provides a “sandbox environment” for analysis.Source: §2 · p. 2
  • The evaluated models span autoregressive, shallow-fusion autoregressive-plus-diffusion, and deep-fusion autoregressive-plus-diffusion paradigms.Examples include Janus-Pro-7B, Qwen-Image and Blip3o, and BAGEL, respectively.Source: §2 · p. 2

3 Reasoning Generation

This section tests whether internal reasoning can guide image generation through synthetic math and symbolic-mapping tasks, then examines explicit CoT and self-training as routes to non-CoT reasoning.

3.1.1 Mathematical Operations

  • Math tasks require sequential calculations before generating the corresponding number of objects, with difficulty increased from one operation to two or three.Final answers are constrained to integers below seven, and the tasks use 100 prompts at each difficulty level.Source: §3.1.1 · pp. 2–3

3.1.2 Symbolic Mapping

  • Symbolic-mapping tasks require following arbitrary rules through one- to three-step chains to generate the target object from two options.Paired prompts use the same rules but ask for different results to prevent success by random guessing.Source: §3.1.2 · p. 3

3.2 Evaluation Protocol

  • Generated images are first captioned by an MLLM, then judged against the ground-truth answer without access to the image; mapping succeeds only when both paired prompts are correct.This two-stage protocol is intended to reduce direct visual-judge bias and random-guess success.Source: §3.2 · pp. 3–4

3.4 Can Reasoning Capabilities Be Internalized?

  • STARS generates CoT-and-image examples, filters them for instruction-output consistency, then fine-tunes standard generation on the retained instruction-output pairs.The intermediate reasoning trace is discarded before fine-tuning, aiming to encode the logical steps implicitly.Source: §3.4 · pp. 4–5, Figure 2 · p. 5

3.3 From Reasoning Failure to CoT Activation

  • 0.5083 (BAGEL + CoT) versus 0.0283 (BAGEL) average score; explicit CoT guides visual generation with a reasoned conclusion.Without CoT, most models performed near zero on tasks requiring logical reasoning, with failures especially pronounced on abstract symbolic mapping.Source: §3.3 · p. 4, Table 1 · p. 5

3.4.1 STARS on Mathematical Operations

  • Substantial improvement in standard, non-CoT performance across training difficulties, from a level previously near zero.The reported math results include cross-difficulty generalization: training at one level improved harder tasks, and vice versa.Source: §3.4.1 · p. 5, Table 2 · p. 6

3.4.2 STARS on Symbolic Mapping

  • STARS failed to distill CoT into standard generation on higher-difficulty symbolic mapping tasks; success stayed near zero and extended single-difficulty training worsened performance.Training directly on second- or third-level data also degraded the model’s original CoT performance.Source: §3.4.2 · p. 6
  • 0.55 final average after three curriculum rounds, versus 0.30 for the baseline; Normal performance improved across all three difficulty levels while CoT was preserved or sometimes improved.The authors report curriculum learning was more effective than single-difficulty training in the controlled symbolic-mapping setting.Source: §3.4.2 · p. 6

4 Knowledge Transfer

Knowledge Transfer evaluation injects fictional character profiles into the understanding module, then tests whether generation can use them. The results show that transfer is weak overall, varies by architecture and task direction, and can be aided by CoT on Forward Retrieval.

4.1 Task Design: Controlled Knowledge Injection

  • The evaluation fine-tunes fictional character profiles into the understanding module, then tests image generation from names or from descriptions of profile attributes.Forward Retrieval maps a character name to a portrait or associated attribute; Inverse Search maps attributes back to the character.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Evaluation Protocol

  • Generated images are captioned and semantically compared with ground truth, with portrait correctness requiring all specified attributes and fruit or flower correctness requiring the matching type.The task-specific criteria include skin color, hair color, gender, and age for portraits.Source: §4.2 · p. 7

4.3 Knowledge Transfer Results

  • Standard-mode transfer ranks Blip3o highest, BAGEL and Qwen-Image intermediate, and Janus-Pro lowest, suggesting query bottlenecks may help more than dense autoregressive conditioning.The proposed architectural comparison is suggestive rather than a demonstrated causal mechanism.Source: §4.3 · p. 7, Table 4 · p. 8
  • All tested models score very low overall on knowledge transfer, and Table 4 reports the highest standard-mode performance for query-based Blip3o.The results indicate that current unified multimodal models largely fail to use knowledge injected into their understanding modules when generating images.Source: §4.3 · p. 7, Table 4 · p. 8
  • Inverse Search (Value → Key) lags markedly behind Forward Retrieval (Key → Value); applying CoT to BAGEL raises Forward Retrieval from 0.10 to 0.63 but yields only a marginal Inverse Search gain (0.15).Source: §4.3 · p. 8

4.4 Architecture Analysis

  • In Blip3o, intermediate queries make related attributes prominent, while target-knowledge terms emerge later and peak at the final query, consistent with an implicit CoT-like retrieval process.Figure 4 plots relevant-word probabilities across query positions and includes the last text token as a baseline.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Related Work

Related work evaluates text-to-image fidelity, semantic consistency, internal knowledge and reasoning, and the benefits of unification. UniSandbox instead uses synthetic, low-leakage tasks to attribute the understanding-generation gap.

Evaluation of text-to-image generation

  • Text-to-image evaluation has progressed from fidelity metrics such as FID to semantic-consistency benchmarks including GenEval, CLIPScore, and VQA-Score.These metrics are cited as examples of the shift in evaluation focus.Source: §5 · p. 8

Knowledge, reasoning, and unification

  • Recent studies probe internal knowledge and reasoning, while RealUnify and Uni-MMMU test whether unification benefits unified models on real-world prompts.Source: §5 · p. 8

UniSandbox’s distinction

  • Unlike broad capability scoring, UniSandbox uses synthetic, low-leakage tasks to attribute the understanding-generation gap.Source: §5 · p. 8

6 Conclusion

The conclusion reports a significant understanding-generation gap in reasoning generation and knowledge transfer. It presents explicit CoT, self-training, and query-based architectures as findings that inform future unified-model design.

Main conclusion

  • UniSandbox finds that models often fail to translate understanding into generation, especially for reasoning generation and knowledge transfer.Source: §6 · p. 9

Reasoning generation

  • For reasoning generation, explicit CoT activates reasoning during generation, and self-training can internalize that ability.Source: §6 · p. 9

Knowledge transfer and design

  • For knowledge transfer, CoT helps activate retrieval of newly learned knowledge, while query-based architectures show implicit CoT-like properties that inform future design.Source: §6 · p. 9

Limitations

The paper’s controlled synthetic tasks isolate reasoning generation and knowledge transfer but do not cover the full diversity of real-world text-to-image prompts.

Evaluation scope

  • The synthetic evaluation isolates reasoning generation and knowledge transfer rather than representing the full diversity of real-world text-to-image prompts.Source: Limitations · p. 9

Model coverage

  • Experiments cover a representative set of unified multimodal architectures, so broader model coverage could reveal additional architectural trade-offs.Source: Limitations · p. 9

Evaluation protocol

  • The caption-and-comparison evaluation can still inherit errors from its captioning and comparison models.Source: Limitations · p. 9

Self-training scope

  • The self-training experiments establish a practical path for internalizing explicit reasoning, but scalability to more open-ended generation remains future work.Source: Limitations · p. 9

Ethical Considerations

The paper frames misleading visual content as a downstream risk of stronger reasoning-aware generation and positions UniSandbox as a diagnostic tool, not a deployment method. Its synthetic prompts and virtual character profiles avoid direct reuse of private or personally identifying information; the experimental data were constructed using GPT-4o.

Ethical Considerations

  • The paper identifies misleading visual content from stronger reasoning-aware generation as its main ethical risk and frames UniSandbox as an analysis tool, not a deployment method.It reports no human-subject or real-person identity data.Source: Ethical Considerations · p. 9
  • Synthetic prompts and virtual character profiles are designed to avoid direct reuse of private or personally identifying information.Source: Ethical Considerations · p. 9

A Synthetic Data Generated Using GPT-4o

  • All experimental data were constructed using GPT-4o, with prompts for other data following the same controlled templates.The paper lists prompts for first-order difficulty Mathematical Operations and Symbolic Mapping.Source: A Synthetic Data Generated Using GPT-4o · p. 13

B.1 Ablation on Rejection Sampling

Rejection sampling and curriculum learning both improved reported performance in their respective comparisons. The appendix also describes caption-based MLLM evaluation for the paper’s simple, discernible image tasks.

B.1 Ablation on Rejection Sampling

  • 16% to 23% success rate on Math3: complete STARS with rejection sampling outperformed training on 5,000 randomly sampled, unfiltered CoT-generated instances per difficulty tier.The reported rejection-sampling improvement was consistent across difficulty levels and most pronounced on Math3.Source: Figure 5 · p. 13, §B.1 · p. 13

B.2 Comparison of Curriculum Learning and Mixed Training

  • Curriculum learning achieved better results than combining data from all difficulty levels in mixed training.The comparison is reported for BAGEL trained with STARS on symbolic mapping datasets of varying difficulty.Source: §B.2 · p. 13

C Evaluation Using an MLLM

  • Evaluation used an MLLM to caption each image, then compared the caption with the ground-truth answer; the authors say simple, discernible images make this approach suitable.They consistently used Qwen2.5-VL-7B-Instruct with vLLM.Source: C Evaluation Using an MLLM · p. 13

D Implementation Details of Knowledge Injection

Knowledge injection used character-specific or paired fine-tuning from the pre-trained weights, with text QA checks to verify injected attributes. The appendix also reports separate training settings for BAGEL and other models and summarizes three unified-model architecture families.

D Implementation Details of Knowledge Injection

E.1 Unified Multimodal Models

  • The appendix groups unified multimodal models into autoregressive, AR-plus-diffusion deep-fusion, and AR-plus-diffusion shallow-fusion architectures.The deep-fusion group uses diffusion to predict images; shallow fusion maps VLM hidden states into the image hidden dimension before DiT generation.Source: §E.1 · p. 14

E.2 Text-to-Image Evaluation

Text-to-image evaluation has shifted from fidelity toward semantic consistency and, more recently, knowledge- and reasoning-based generation. The appendix describes caption-grounded evaluation prompts and a rejection-sampling filter for generated images.

E.2 Text-to-Image Evaluation

  • Text-to-image evaluation has moved from fidelity metrics such as FID toward semantic consistency, then toward testing world knowledge and reasoning in generation.WISE (Niu et al., 2025) introduced 1,000 prompts across domains including cross-cultural common sense, spatio-temporal understanding, and natural sciences; later benchmarks include R2I-Bench (Chen et al., 2025c) and T2I-ReasonBench (Sun et al., 2025).Source: §E.2 · pp. 14–15

F Details of Rejection Sampling

  • Rejection sampling uses BAGEL’s understanding module to filter generated images; the filtering prompt checks object counts, types, and visual quality against the input prompt.Source: F Details of Rejection Sampling · p. 15, Table 14 · p. 21
  • The evaluation tables cover caption generation and comparison with ground truth for Mathematical Operations, Symbolic Mapping, and Knowledge Transfer.The Knowledge Transfer caption prompt categorizes persons, flowers, or fruits using specified formats.Source: Table 9 · p. 17, Table 10 · p. 18, Table 11 · p. 19
  • The captioning criteria require one identifiable object type and fully visible, clear, normally shaped objects; images with letters, symbols, or numbers are errors.Source: F Details of Rejection Sampling · p. 18

Related papers

Loading 2511.20561v3…