PaperlayerView exact revision on arXiv

Source-linked AI summary

Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

Guoxuan Chen, Chufeng Xiao, Haoran Yang, Siyue Xie, Binxiao Huang, Ming Zhang, Cheuk Him Chau, Xinyu Fu, Yingzhao Lian, Tom S. Y. Li, Jintao Lin, Bowen Dong, Zian Qian, Yuhao Liu, Yuxuan Hu, Weikang Shi, Bin Zou, Bowen Zheng, Haoxuan Che, Chang Chen, Yuyang He, Heyang Sun, Tianyu Huang, Chong Hou Choi, Cheng Gong, Han Shi, Haoli Bai, Xihui Liu, Hongsheng Li, Qifeng Chen, Chao Huang, Rui Liu, Chenyang Lei

arXiv:2607.13125v2cs.CVcs.AI

TL;DR

Open-source image generation must interpret complex intentions, implicit constraints, and cross-modal context, but evidence for unified understanding-and-generation systems under constrained budgets remains limited. Boogu-Image-0.1 addresses this with understanding-driven design, careful data construction, and agentic inference-time scaling, achieving top-tier open-source performance with 208.62 million images and roughly $400K in theoretical base-model training cost.

  • Problem

    Image generation increasingly requires interpreting complex intentions, implicit constraints, and cross-modal context beyond single descriptive prompts, while open-source systems still trail leading closed-source systems.

    Source: §1 · p. 6

  • Method

    Boogu-Image-0.1 jointly designs multimodal understanding and generation using stronger encoders, agentic prompt rewriting, curated data, and inference-time model selection.

    Source: §1 · p. 6, §4 · p. 45

  • Results

    Boogu-Image-0.1 achieves top-tier performance on Boogu Arena and state-of-the-art open-source results on Qwen-Image-Bench, using 208.62 million unique images.

    Source: §1 · p. 6, §3.2.2 · p. 33

  • Takeaways & Limitations

    The released unified system, weights, code, and practical lessons support further research on understanding-driven open-source image generation and editing.

    Source: §1 · p. 6, §4 · p. 45

    Boogu trails leading closed-source systems on knowledge-intensive tasks, currently optimizes text rendering only for Chinese and English, and can produce anatomical inconsistencies.

    Source: §4 · p. 45

Abstract

from arXiv · show

We introduce Boogu-Image-0.1, an open-source unified multimodal understanding and generation model family, comprising Base, Turbo, Edit, and Edit-Turbo variants. It delivers competitive performance in high-quality text-to-image generation, fast inference, instruction-based editing, and bilingual (Chinese-English) text rendering. Closed-source multimodal systems like Nano-Banana-Pro and GPT-Image-2 achieve strong performance through system-level integration rather than a single model, yet their internal practices remain largely undisclosed. In this work, we demonstrate that strengthening the understanding capability of the system, through a stronger multimodal encoder, agentic prompt rewriting, and related techniques, together with improvements in data quality, training pipelines, and agentic inference-time scaling, can substantially enhance generation and editing performance even under highly constrained compute budgets. Comprehensive evaluations show that Boogu-Image-0.1 consistently matches or surpasses other open-source models across standard benchmarks, and achieves results approaching leading closed-source systems. Notably, this is accomplished with only 208.62 million unique images. The base model's theoretical training cost is only approximately \$400K. We share practical discussions that we believe are valuable to the broader research community, and release weights, code, and recipes under Apache 2.0 to advance the open ecosystem for unified multimodal understanding and generation. Our code is available here: https://github.com/Boogu-Project/Boogu-Image.

Preamble · 1 Introduction

Boogu-Image-0.1 treats understanding as a first-class design target for Requirement-to-Image generation, coordinating stronger language perception, agentic inference, and efficient training. The resulting open-source system achieves top-tier Boogu Arena performance and state-of-the-art open-source results on Qwen-Image-Bench [73].

1 Introduction

  • Coordinated improvements in understanding, data quality, training recipes, and inference pipelines substantially improve instruction following and generation fidelity.This design target responds to rapid advances in closed-source systems and open-source efforts that primarily improve visual quality or add auxiliary prompt rewriting.Source: §1 · p. 6
  • The paper frames image generation as Requirement-to-Image Generation, requiring interpretation of complex intentions, implicit constraints, multi-level instructions, and cross-modal context.Understanding connects ambiguous human requirements to precise generation and to strategies suited to users’ cost and latency constraints.Source: §1 · p. 6
  • The system uses Qwen3-VL-8B [3] as its text encoder after analysis found that stronger LLMs provide stronger text-encoding capabilities.The selected encoder balances language-understanding capability against a parameter budget acceptable to the open-source community.Source: §1 · p. 6
  • Agentic Image Generation injects understanding at inference through prompt rewriting, model-variant selection, and reflection techniques [60] [75] [81] [142].An agent interprets requests, chooses Base or Turbo to balance speed and quality, and progressively improves performance as more understanding is added.Source: §1 · p. 6
  • The training pipeline reaches competitive performance using 208.62 million unique images at a total cost of roughly $400K.Under constrained compute, data curation prioritizes quality over quantity and uses training optimization strategies.Source: §1 · p. 6
  • The paper documents evaluation protocols [42] [57], data filtering [130], and caption design [138] as practically decisive details that are often hidden from public reports.Exposing these practices aims to reduce open-source engineering costs and provide reference points for future work.Source: §1 · p. 6
  • Boogu-Image-0.1 achieves top-tier performance on Boogu Arena and state-of-the-art results among open-source models on Qwen-Image-Bench [73].The system unifies understanding and generation to translate complex user intents into high-fidelity images.Source: §1 · p. 6

2 Experimental Results

Boogu models achieve strong text-to-image and image-editing results, but the section argues that benchmark scores must be interpreted cautiously because several established benchmarks diverge from human preference and real-world quality. The authors therefore emphasize human-preference evaluation, newer benchmarks, and qualitative comparisons alongside supplementary legacy results.

2.1 Rethinking Open-Source Research Evaluation

  • Public GenEval [42] and DPG-Bench [57] scores can invert human-preference rankings: GPT-Image-2 leads human preference but ranks mid-pack, so these benchmarks are not used for primary evaluation.The section attributes this mismatch to synthetic evaluation conditions, saturation, and data contamination or test-set leakage, while reporting legacy results only for reference.Source: §2.1 · p. 7, Figure 5 · p. 7, §2.2.4 · p. 12

2.2.1 Boogu Arena Benchmark

  • Boogu models lead the open-source tier across Boogu Arena categories, trailing only GPT-Image-2 and Nano-Banana-Pro, while Thinking most improves text rendering.The in-house arena uses bilingual prompts, blind pairwise votes, and Elo aggregation; more than 4,000 votes were collected across nine models, and its rankings strongly agree with LMArena (r = 0.986; ρ = 1.0).Source: §2.1 · p. 7, §2.2.1 · pp. 8–9

2.2.2 Qwen-Image Benchmark

  • 53.57 and 53.73, Boogu-Image-0.1-Base-Thinking achieves the best overall scores among evaluated open-source models on Chinese and English Qwen-Image-Bench prompts.Its largest reported gain is in Chinese Creativity, improving from 48.62 to 56.74 over the non-thinking counterpart, with stable trends across both languages.Source: §2.2.2 · p. 9, Table 1 · p. 10, Table 2 · p. 10

2.2.3 LongText Benchmark

  • 0.985, Boogu-Image-0.1-Turbo-Thinking ranks second overall on Chinese LongText-Bench, behind Seedream-4.5, but the benchmark mainly measures short-sequence OCR accuracy.Base produces more coherent typography than Turbo in dense layouts despite Turbo’s comparable benchmark score and visible dense-text artifacts.Source: §2.2.3 · p. 11, Table 3 · p. 11
  • LongText-Bench is treated as a text-correctness check rather than a complete quality measure because it overlooks glyph naturalness, layout integration, artifacts, and perceptual quality while clustering strong models near saturation.The authors recommend complementing it with human evaluation or perceptual metrics.Source: §2.2.3 · p. 11

2.2.5 Qualitative Comparison

  • Boogu-Image-0.1-Turbo delivers competitive qualitative text-to-image quality and narrows the gap to top closed-source systems, while Base is recommended for ultra-dense text beyond 100 characters or words at 2K resolution.The qualitative comparisons cover photorealistic, stylized, and text-rendering categories and show strengths in prompt alignment, visual quality, and text fidelity.Source: §2.2.5 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15

2.3.1 ImgEdit Benchmark

  • 4.64 overall, Boogu-Image-0.1-Edit-Thinking ranks first on ImgEdit-Bench, surpassing open- and closed-source editors while leading Remove, Hybrid, and Action.The Thinking variant improves over Base overall (4.64 vs. 4.51), especially on Extract (4.32 vs. 3.69), although the authors caution that benchmark rankings do not match human evaluations.Source: §2.3.1 · p. 19, Table 6 · p. 19

2.3.2 Qualitative Comparison

  • Qualitative image-editing comparisons show Boogu-Image-0.1-Edit is highly competitive in instruction alignment, photorealism, precise scene-text modification, and artistic stylization.The comparisons use public ImgEdit-Bench [136] and an in-house benchmark against GPT-Image-2, Nano-Banana-Pro, Seedream-5.0-Lite, and Qwen-Image-Edit-2511.Source: §2.3.2 · p. 20, Figure 13 · p. 21, Figure 14 · p. 22

3 Experimental Analysis and Discussion

This section examines how stronger understanding, carefully structured data, and targeted training and inference techniques shape generation quality, efficiency, controllability, and text rendering under limited compute. It also discusses evaluation practices and persistent challenges involving cultural coverage, memorization, and scenario relevance.

3.1 Boosting Generation via Better Understanding

  • Boogu decomposes understanding into user intent, training-image captions, and task complexity, improving downstream quality while controlling computation.The framework combines stronger text encoding and agentic rewriting, demand-aligned supervision, and complexity-aware model selection.Source: §3.1 · p. 24

3.1.1 Instruction Encoder: The Sensor of Text-to-Image Models

  • GenEval scores improve monotonically as the frozen instruction encoder scales from 1.7B to 14B, with no saturation observed.Training data, hyperparameters, and DiT architecture remain fixed; the reported models are intermediate rather than fully trained.Source: Table 7 · p. 25, §3.1.1 · p. 25

3.1.2 Prompt Rewriter: A Translator, Not an Enhancer

  • An agentic prompt rewriter resolves ambiguity, reasoning, counting, unsafe requests, and text-rendering specifications while preserving clear prompts through identity transformation.Its role is translation rather than unconditional enhancement, limiting hallucinated details and intent drift.Source: §3.1.2 · p. 26
  • Stronger vision-language backbones deliver larger prompt-rewriting gains, linking rewriting quality to world knowledge and compositional reasoning.Figure 18a reports this positive correlation across thinking models.Source: §3.1.2 · p. 26, Figure 18 · p. 27

3.1.3 Captions: Good Supervision Requires Understanding User Demand

  • Per-aspect captioning produces more accurate supervision than any single-VLM baseline, improving the corresponding generation capabilities.Boogu selects models and prompts separately for counting, spatial relations, attribute binding, style, and text rendering, supplementing weak dimensions with expert or human annotations.Source: §3.1.3 · p. 28

3.1.4 Model Router: The Right Model for the Right Task

  • Boogu routes simple requests to Turbo and complex requests to Base, because their inference costs can differ by 50 to 100 times despite often indistinguishable outputs.This routing trades quality against efficiency while adding system complexity.Source: §3.1.4 · p. 29

3.2.1 Discussions Beyond Leaderboards

  • Quality evaluations should fix inference budgets or report quality and latency jointly, because best-of-N, reflection, rewriting, and ensembling trade computation for performance.Figure 19 depicts performance rising from the Raw Model to the Agentic Model as inference cost increases.Source: §3.2.1 · p. 30, Figure 19 · p. 30
  • Open-source evaluation should compare against the community capability envelope and use scenario-driven, human-rated production prompts rather than a single baseline or static academic metrics.Cross-system latency comparisons also require caution because closed-source runtimes include safety and production constraints beyond core generation.Source: §3.2.1 · pp. 30–31
  • Evaluation remains unresolved because current text-to-image benchmarks lack leakage controls, world-knowledge coverage, ambiguity handling, deployment-aware latency tradeoffs, and dynamic human-aligned protocols.The section warns that benchmark optimization can trigger Goodhart’s Law, making public scores unreliable indicators of real-user performance.Source: §3.2.1 · p. 32

3.2.2 Data Ablation Evaluation: Just Enough Data Is Not Enough

  • Open-source datasets remain inadequate despite 187M images because semantic noise, factual errors, duplication, shallow captions, and missing domain-specific coverage limit high-fidelity generation.The section argues that data refinement, synthetic generation, and expert curation are needed beyond raw public-dataset scale.Source: §3.2.2 · p. 35
  • Large-scale Chinese SFT does not eliminate Western-centric outputs, indicating that cultural diversity must be addressed during pretraining rather than treated as downstream alignment.Semantically equivalent Chinese and English prompts still predominantly produce Western faces, architecture, street scenes, and clothing.Source: §3.2.2 · p. 36, Figure 24 · p. 36
  • Syllabus curation targets long-tail capabilities through systematic decomposition, comprehensive coverage, sufficient category-level exposure, and balanced sampling rather than relying on scale or parameter growth alone.Boogu retains some flawed images when detailed captions identify artifacts, converting defects into controllable generation signals; balancing is worthwhile mainly under limited compute.Source: §3.2.2 · pp. 33, 34, 37
  • Boogu’s 21.62M-sample syllabus, upsampled to 47.19M, significantly outperforms models trained on 187M open-source samples across instruction following, world knowledge, and photographic quality.Under identical settings, the syllabus improves all three benchmarks; its broader design emphasizes fine-grained coverage, human-prior structure, and targeted sampling.Source: §3.2.2 · pp. 35, 38, Table 8 · p. 35
  • At least 300 training exposures per Chinese glyph are required for accurate rendering, and targeted oversampling beyond this threshold reliably corrects previously unseen characters.When full vocabulary enumeration is infeasible, covering the 3,500 most frequently used modern Chinese characters with at least 300 appearances supports everyday Chinese text rendering.Source: §3.2.2 · p. 39, Figure 27 · p. 39, Figure 28 · p. 39

3.2.3 Ablation Experiments on Multimodal Learning and Understanding

  • The multimodal-learning ablation tests memorization of an unseen human identity using 170 photographs, four caption variants, and oversampling to 11,628 images within a 12.6M-image SFT dataset.The identity subset contributes approximately 0.13% of the total SFT data and is balanced across Chinese and English captions.Source: §3.2.3 · p. 40

Identity memorization

  • Full recognizability of an unseen identity emerged by 8k–9k steps, requiring about 4,500 language-matched exposures during training.The model began producing recognizable facial features around 5k steps when trained with Chinese prompts.Source: §3.2.3 · p. 40, Figure 29 · p. 40

Reinforcement-learning trade-offs

  • Heavy aesthetic RL narrows the reachable output distribution toward idealized mainstream appearances, whereas the model produces more natural subjects outside those preferences.For “A sixty-year-old Chinese woman,” heavy-RL models generated glamorized features while the model produced a more realistic result.Source: §3.2.3 · p. 41, Figure 30 · p. 41
  • The training strategy avoids heavy aesthetic RL, embedding preference in the understanding system while limiting RL to anatomy and text-rendering issues to preserve diversity and reduce cost.RL remains useful for targeted artifact reduction but requires reward-model inference, repeated sampling, and additional optimization steps.Source: §3.2.3 · p. 41

Rectified dynamic time shifting

  • At 2K resolution, standard dynamic timestep shifting over-squeezes the distribution, degrading signal-to-noise ratio and slowing convergence while causing early noisy generations.The shift magnitude grows with resolution-dependent token count, pushing timesteps toward smaller values.Source: §3.2.3 · p. 42, Figure 31 · p. 42
  • Clamping the effective token count at ncap = 4096 preserves 1K timestep percentiles beyond that regime, extending tuned behavior to 2K without exacerbating over-squeezing.Under the original shift, the median collapsed from 0.24 at 1K to 0.04 at 2K.Source: §3.2.3 · p. 43

2D structural awareness in DiT

  • Boosted Orthogonal Guidance preserves DiT’s 2D latent structure and produces stronger photographic realism and finer texture detail than standard CFG across four prompts.The method applies 2D normalization directly to matrix-structured predictions instead of flattening them into 1D updates.Source: Figure 33 · p. 44, §3.2.3 · p. 44

4 Conclusion, Limitations, and Future Directions · Appendix

Boogu-Image-0.1 unifies multimodal understanding, generation, and editing, achieving strong open-source performance near leading closed-source systems at approximately $400K training cost. Its limitations motivate more faithful evaluation, higher-quality data, and deeper agentic inference.

4 Conclusion, Limitations, and Future Directions

  • Boogu-Image-0.1 jointly designs understanding, generation, and editing, ranking among top open-source systems while approaching leading closed-source performance at approximately $400K cost.The full training pipeline uses 208.62M unique images, and agentic inference with Deepseek-V4-Flash rewrites prompts and selects models.Source: §4 · p. 45
  • Boogu still trails closed-source systems on world-knowledge tasks, while reported scores may underestimate this gap because reliable measurement remains difficult.The limitation includes artistic styles, landmarks, public figures, and commercial products.Source: §4 · p. 45
  • Current text rendering is optimized only for Chinese and English, limiting multilingual coverage.Source: §4 · p. 45
  • Multi-person interactions, heavy occlusions, atypical viewpoints, and FLUX.1 VAE reconstruction errors can impair anatomy and fine details.The affected details include small faces, limbs, and text, although native 2K generation partially alleviates these problems.Source: §4 · p. 45
  • The open-source release aims to advance image-generation systems that genuinely understand user intent through community exploration of these directions.Source: §4 · p. 45
  • Static benchmarks can mislead as they saturate, so evaluation should prioritize real-world utility through multi-turn editing, identity consistency, and long-context adherence.The proposed direction emphasizes faithfully translating complex, multifaceted intent rather than optimizing single-shot aesthetics.Source: §4 · p. 45
  • Open-source datasets impose a performance ceiling through semantic noise and limited cognitive depth, requiring high-fidelity synthesis and expert curation.Source: §4 · p. 45
  • Inference-time understanding is effective, but stronger agentic generation requires tighter reasoning-generation loops, self-verification, and tool use.These mechanisms could shift generation toward systems that actively plan, critique, and refine outputs.Source: §4 · p. 45

A Contributors · B Background and Technical Choices · B.1 Background

The project is led and supported by a multi-institutional team, while its background traces visual synthesis from diffusion-based generation and controllable editing to VLMs and unified multimodal systems.

A Contributors

  • The project lists Chenyang Lei as project leader, with Rui Liu, Chao Huang, Han Shi, Haoli Bai, Xihui Liu, Hongsheng Li, and Qifeng Chen as sponsors.Source: A Contributors · p. 55
  • Core and additional contributors include researchers from the Artificial Intelligence Laboratory at Huawei Technologies and collaborating Hong Kong and Nanjing universities.The primary affiliation is Artificial Intelligence Laboratory (Leibniz), Huawei Technologies, Hong Kong; collaborating institutions include six listed universities.Source: A Contributors · p. 55
  • Correspondence is handled by Chenyang Lei, Rui Liu, and Chao Huang.Source: A Contributors · p. 55

B.1 Background

  • Text-to-image diffusion progressed from LDM [107] and DALL-E 2 [104] toward improved prompt adherence in DALL-E 3, transformer scalability in DiT, and current closed-source systems including Nano Banana and ChatGPT Images.The passage also situates Sora and Stable Diffusion 3 [36] as demonstrations of transformer-driven video generation and rectified-flow image synthesis.Source: §B.1 · p. 56
  • ControlNet [141] added auxiliary spatial conditioning, while InstructPix2Pix [11] enabled instruction-driven image editing without complex manual masks.Source: §B.1 · p. 56
  • VLMs evolved from LLaVA [80] and ViT [34]-based visual instruction tuning toward high-resolution structural alignment and reasoning in Ovis [87] and Qwen-VL [3] [4].Source: §B.1 · p. 56
  • Unified multimodal generation combines VLM reasoning with diffusion generation, with OmniGen [132], OmniGen2 [131], Qwen-Image [130], and Z-Image [138] advancing integrated generation, editing, and efficient synthesis.The passage also describes Google’s Nano Banana as using reasoning-guided synthesis through interleaved prompts.Source: §B.1 · p. 56

B.2 Macro-Architecture Design · B.2.1 Instruction Reasoner · B.2.2 Instruction Encoder and Image Generation

Boogu-Image uses a compute-efficient decoupled architecture that separates complex instruction reasoning from lightweight conditioning and image synthesis. A unified pipeline supports text-to-image generation and editing by adapting prompts, visual inputs, and latent diffusion processing to the task.

B.2 Macro-Architecture Design

  • A moderately large VLM serves as an Instruction Reasoner while a lightweight VLM encodes its rewritten instructions for the DiT, reducing the costs of a monolithic 72B VLM design.The preliminary design paired a frozen 72B VLM with a trainable 10B DiT, but its training and inference overhead was prohibitive.Source: §B.2 · p. 56

B.2.1 Instruction Reasoner

  • The Instruction Reasoner determines text-to-image versus editing from the presence of a reference image, applies the corresponding system prompt, and produces a parsed, decomposed, rewritten instruction.Figure 34 shows that editing inputs append a reference image before reasoning, while text-only inputs use the T2I prompt.Source: §B.2.1 · p. 57, Figure 34 · p. 57

B.2.2 Instruction Encoder and Image Generation

  • The text tokenizer projects rewritten text instructions into embeddings, while a ViT encodes zero or more reference images; their embeddings are concatenated before being fed into the autoregressive transformer.Reference images are downsampled to 384 × 384; without references, the ViT is bypassed and the task is pure text-to-image generation.Source: §B.2.2 · p. 57
  • An optional three-layer Prompt Tuning Transformer prepends 32 trainable prompt embeddings to the frozen Instruction Encoder, globally modulating hidden states toward the DiT’s semantic space.It uses causal masking and is designed to improve instruction parsing and comprehension while keeping the encoder frozen.Source: §B.2.2 · p. 57
  • Reference images are VAE-encoded into latent space for diffusion, patchified and refined alongside noise, while the VAE decoder reconstructs the final image; text-only generation bypasses the encoder.This latent-space design balances performance and computational efficiency, and the default Refiner contains 2 layers.Source: §B.2.2 · p. 58
  • The DiT processes instruction, reference-image, and noise representations through dual-stream layers followed by single-stream layers, then decodes the result into an output image.Each modality has independent Embedders and Refiners; dual-stream outputs are merged before unified single-stream processing.Source: §B.2.2 · pp. 58–59
  • During inference, the DiT synthesizes images through multiple iterative time-step updates using standard Euler flow matching or fast solvers such as DPM-Solver++.The time step transitions between 0 and 1, or the reverse, with corresponding Time Embedding updates.Source: §B.2.2 · p. 59

B.3 Micro-Architecture Details

This section presents the model’s micro-architecture: a unified framework for text-to-image generation and image editing, adapted from recent foundational works and augmented for fine-grained image comprehension and multimodal alignment.

B.3 Micro-Architecture Details

  • The model unifies text-to-image generation and image editing through adapted designs with specialized modules for fine-grained image comprehension and comprehensive multimodal alignment [29] [36] [79] [127] [130] [131] [132] [147].The micro-architecture builds on the preceding macro-level overview and modifies or extends these foundational designs to suit the model’s objectives.Source: §B.3 · p. 59

B.3.1 Dual-Stream Layer · B.3.2 Single-Stream Layer · B.4 Summary of Architectural Design

The architecture combines modality-specialized dual-stream processing with unified single-stream fusion, using residual main paths and computational branches to support stable, fine-grained multimodal modeling. It contains 8 dual-stream and 32 single-stream layers, totaling approximately 10B trainable DiT parameters.

B.3.1 Dual-Stream Layer

  • The dual-stream layer separately processes instruction and image streams, while expanded independent intra-stream and inter-stream modules enable deeper feature interaction.The image stream contains the noisy latent and optional reference image; the design extends beyond conventional architectures that isolate only QKV projections or limited modules.Source: §B.3.1 · p. 59
  • An independent image-attention module deepens visual comprehension, while the complementary stream performs holistic instruction-image fusion with instruction residual connections.Three independent QKV projection sets serve text, holistic image enhancement, and image self-attention, respectively.Source: §B.3.1 · p. 59
  • Dual-stream processing combines instruction-focused and image-focused representations, enhancing fine-grained multimodal comprehension before convergence into subsequent single-stream layers.This design is motivated by the need for stronger instruction understanding in image generation and editing.Source: §B.3.1 · p. 59
  • Residual main paths stabilize layer outputs, while branch paths perform attention and modulation before merging back into the main path.The dual-stream layer uses sandwich normalization and modulation; multi-norm design improves pre-training stability with large learning rates, accelerating convergence and robustness.Source: §B.3.1 · p. 60, Figure 36 · p. 60

B.3.2 Single-Stream Layer

  • The single-stream layer merges deeply processed instruction and image states into one modality-agnostic representation, applying shared parameters and QKV projections for comprehensive multimodal fusion.Its branch paths merge into the residual main path, which alone would reduce the layer to an identity mapping.Source: §B.3.2 · p. 61

B.4 Summary of Architectural Design

  • The DiT uses 32 single-stream and 8 dual-stream layers with 3D RoPE; it has 10,292.56M trainable parameters, increasing to 11,022.55M with Prompt Tuning Transformer.The default condition is the final Instruction Encoder layer’s hidden states, and further hyperparameters are summarized in Table 10.Source: §B.4 · p. 61, Table 10 · p. 61

B.5 Training Process · B.6 Lightweight Module Details

The training pipeline uses progressive resolution and task curricula, transitioning from low-resolution T2I pretraining to higher-resolution training and TI2I editing fine-tuning. Training also uses dynamic spatial constraints, mixed-resolution and mixed-task data, and a lightweight transformer module for refiners and optional prompt tuning.

B.5 Training Process

  • The pipeline progressively trains on low-resolution images before higher resolutions, then fine-tunes the T2I model into a TI2I editing model.The complete training process is summarized in Table 11.Source: §B.5 · p. 62, Table 11 · p. 62
  • Max Output Pixels, Max Input Pixels, and Max Side Length constrain training image dimensions.Images exceeding these thresholds are dynamically downscaled, while smaller or equal-sized images remain unchanged, enabling mixed-resolution training.Source: §B.5 · p. 62
  • The TI2I phase uses approximately 22M samples combining 11.29M T2I samples from the 47.19M Boogu Syllabus dataset with 10.70M augmented editing samples.The editing samples are augmented from an initially constructed 2.50M TI2I dataset.Source: §B.5 · p. 62
  • This mixed TI2I dataset optimizes the image editing model through both mixed-resolution and mixed-task training.The dataset combines pure T2I and editing samples during the TI2I training phase.Source: §B.5 · p. 62

B.6 Lightweight Module Details

  • The lightweight single-stream transformer layer is designed for various refiners and optional prompt tuning components.Figure 37 presents the module architecture.Source: Figure 37 · p. 63
  • The lightweight module follows the classic transformer design with minimal modifications [24, 29, 36, 38, 79, 97, 120, 127, 130–132, 147].Its stated design is illustrated in Figure 37.Source: Figure 37 · p. 63

C Boosted Orthogonal Guidance · C.1 Matrix-Structured View of DiT Predictions · C.1.1 Matrix Normalization

The section introduces Boosted Orthogonal Guidance (BOG), a training-free DiT inference method that strengthens prompt-aligned detail while mitigating appearance degradation from aggressive guidance. It reformulates predictions as spatial matrices and applies matrix normalization to preserve informative variation through efficient orthogonalization.

C Boosted Orthogonal Guidance

  • BOG targets oversaturation, flattened tones, over-smoothing, and artifacts caused by aggressively amplified guidance while enhancing fine-grained details and cinematic atmosphere.The method requires no additional training or meaningful sampling overhead.Source: C Boosted Orthogonal Guidance · p. 64
  • BOG emphasizes orthogonal guidance information to avoid color blow-up and texture collapse, producing richer micro-structures, tonal depth, and coherent film-like lighting.Its matrix-structured normalization encourages higher effective rank than vector-norm rescaling, which changes only global scale.Source: C Boosted Orthogonal Guidance · p. 64

C.1 Matrix-Structured View of DiT Predictions

  • DiT predictions have shape [B, C, H, W], so their [H, W] structure forms a meaningful image-aligned grid rather than an arbitrary reshaping.Flattening predictions into one-dimensional updates discards spatial inductive bias.Source: §C.1 · p. 64
  • Conventional vector normalization changes update magnitude while leaving spatial composition essentially untouched, whereas BOG normalizes the matrix structure directly at each step.Mean-centering variants also overlook spatial organization that could preserve informative variation and avoid overly smooth updates.Source: §C.1 · p. 64

C.1.1 Matrix Normalization

  • Matrix normalization decomposes D ∈ R^m×n with singular values and applies a scalar transformation to each singular-value index and value.The transformation can use different norm functions and includes a hyperparameter λ.Source: §C.1.1 · pp. 64–65
  • Setting norm(σ_i, i) = 1 makes MNorm(D) the optimal orthogonal or semi-orthogonal approximation of D.This setting is theoretically characterized as an orthogonal approximation rather than ordinary vector rescaling.Source: §C.1.1 · p. 65
  • In practical image generation, MNorm(D) can be estimated without explicit SVD using the Newton–Schulz method.The detailed algorithmic procedure is provided in Appendix C.5.Source: §C.1.1 · p. 65

C.1.2 Matrix Orthogonal Decomposition

The section formulates diffusion sampling through a probability-flow ODE and explains classifier-free guidance as conditional–unconditional prediction extrapolation. It then decomposes guidance updates into parallel and orthogonal components across matrix columns and rows, scaling the parallel terms to mitigate over-saturation.

Preliminary - Ordinary Differential Equation

  • Diffusion generation integrates a probability-flow ODE from Gaussian noise p_0 = N(0, σ_max^2I) to the data distribution, using a DiT denoiser to approximate the intractable score.Conditional inputs such as text instructions or class labels are incorporated as D_θ(x_t,t,c).Source: §C.1.2 · p. 65

Preliminary - Classifier-Free Guidance

  • Classifier-free guidance extrapolates from unconditional toward conditional DiT predictions using D_CFG(x_t,t,c) = D_θ(x_t,t,c_∅) + ω(D_θ(x_t,t,c) − D_θ(x_t,t,c_∅)).The unconditional prediction is typically obtained by randomly dropping the condition during training; larger ω generally improves instruction alignment but can reduce diversity and cause over-saturation.Source: §C.1.2 · p. 66

Parallel-Orthogonal Decomposition

  • Matrix parallel–orthogonal decomposition separates the guidance update into column-wise and row-wise components relative to conditional DiT outputs.The construction applies vector-function decomposition analogously to Gram–Schmidt orthogonalization, producing full matrices for both orientations.Source: §C.1.2 · p. 66
  • Setting μ ≤ 1, including 0, 0.1, or 0.25, controls parallel-component magnitude and mitigates image over-saturation at relatively large guidance scales ω.The row and column components use λ_c = W/(H+W)+ζ and λ_r = H/(H+W)+ζ, with ζ a small constant.Source: §C.1.2 · p. 66

C.2 Rolling-Sum Momentum

The method introduces a rolling-sum momentum update during text-to-image inference, using prior CFG update directions to influence the generation trajectory. Unlike reverse momentum [109], which uses ρ<0, the authors observe that negative momentum causes directional jitter.

C.2 Rolling-Sum Momentum

  • The mechanism is motivated by the intuition that trajectory inertia within the flow-matching vector field influences final image generation.It applies during inference and is inspired by momentum strategies in optimization.Source: §C.2 · p. 66
  • The unconditional condition c∅ is used broadly to denote either the true unconditional state or a negative-prompt-derived condition cneg.Negative prompts may include terms such as “low quality, deformed, blurry, ...”.Source: §C.2 · p. 66
  • Rolling-sum momentum updates the inference trajectory as ∆D(i) = η∆D(i) + ρ∆D(i−1), with typical defaults η = 0.9 and ρ = 0.1.The iteration i maps to diffusion time t through the scheduler function t = scheduler(i).Source: §C.2 · p. 67
  • Unlike reverse momentum [109], which sets ρ<0 to move away from previous CFG directions, negative momentum empirically causes directional jitter across iterations.The comparison concerns update directions within the vector field and the balance between previous and current guidance information.Source: §C.2 · p. 67

C.3 Putting It All Together

The complete pipeline introduces Boosted Orthogonal Guidance (BOG), which refines diffusion guidance through momentum, matrix normalization, and orthogonal decomposition to improve visual quality without meaningful overhead. Because BOG can cause structural distortions and weaker text rendering, the method applies it selectively through a configurable inference interval.

C.3 Putting It All Together

  • The complete pipeline computes a BOG-modified prediction at each inference step from the latent state, condition, unconditional state, and guidance scale.Algorithm 1 summarizes the single-step computation and its final BOG tensor.Source: §C.3 · p. 67
  • BOG refines the guidance direction by smoothing with momentum, preserving spatial rank through matrix normalization, and isolating orthogonal components to reduce parallel projections.This rank-preserving process is designed to prevent oversaturation while improving micro-details, tonal depth, and cinematic lighting zero-shot.Source: §C.3 · p. 67
  • BOG can increase structural distortions and reduce text-rendering success, so it is treated as an optional inference module rather than always enabled.Its main benefit is enhanced cinematic atmosphere, but these artifacts limit universal application.Source: §C.3 · p. 67
  • With BOG Interval ∆BOG = 2, BOG runs every other diffusion step while standard CFG handles the remaining steps, significantly reducing artifacts with a slight texture trade-off.For 10 denoising steps, BOG runs on steps 2, 4, 6, 8, and 10; ∆BOG = 1 denotes full BOG inference, and ∆BOG = 2 is recommended by default.Source: §C.3 · p. 67

C.4 Optimal Semi-Orthogonal Approximation · C.5 Newton-Schulz Approximation

C.4 proves that the SVD-based semi-orthogonal factor is the optimal Frobenius approximation, while C.5 provides a batched Newton–Schulz alternative for computing MNorm(D) without explicitly performing SVD. The approximation supports varied tensor shapes through normalization, optional transposition, iterative matrix updates, and shape restoration.

C.4 Optimal Semi-Orthogonal Approximation

  • The bound is attained by choosing the SVD-aligned factor, equivalently O = UV⊤, which makes Q = In.Although U and O are rectangular, Q is square and orthogonal, so the equality construction is valid.Source: §C.4 · pp. 68–69
  • The optimization reduces minimizing the squared Frobenius distance to maximizing tr(O⊤D).This reduction is the first proof step, followed by expressing the trace in SVD coordinates using Q = U⊤OV.Source: §C.4 · p. 68
  • Because Q is square and orthogonal, its diagonal entries satisfy |qii| ≤ 1, yielding an upper bound on the trace objective.The argument uses the diagonal structure of Σ and the unit spectral norm of Q.Source: §C.4 · p. 68
  • The SVD-based construction achieves the optimal semi-orthogonal approximation, minimizing ∥D − O∥F over all feasible O.The proof establishes tightness by constructing the optimizer and recovering the same result through Von Neumann’s trace inequality [89].Source: §C.4 · p. 69

C.5 Newton-Schulz Approximation

  • When norm(σi, i) = 1, MNorm(D) can be estimated with Newton–Schulz instead of explicitly computing an SVD, following.The supplied implementation is a batched PyTorch Newton–Schulz iteration.Source: §C.5 · p. 70
  • The implementation normalizes each matrix by its Frobenius norm and transposes inputs with H > W so iteration proceeds along the shorter dimension first.It performs the normalization with an epsilon term and conditionally transposes the last two dimensions.Source: §C.5 · p. 70
  • Each Newton–Schulz step uses batched GEMMs to form A = X @ X.transpose(-2, -1), update Bm, and replace X with a linear-plus-matrix product update.After iteration, the code transposes back when needed and restores the original tensor shape.Source: §C.5 · p. 70

Related papers

Loading 2607.13125v2…