PaperlayerView exact revision on arXiv

Source-linked AI summary

SenseNova-U1.5: Towards Native Unified Visual Intelligence

Haiwen Diao, Jiahao Wang, Chenjing Ding, Hanming Deng, Jiangnan Chen, Ruixi Zhang, Ruohui Wang, Wenwen Tong, Xiangyu Fan, Yubo Wang, Yue Zhu, Yuwei Niu, Zhengqi Bai, Zhiqian Lin, Zhitao Yang, Zhongang Cai, Bo Yang, Chen Feng, Chengguang Lv, Guangjia Liu, Guanlin Wang, Hanyu Zhang, Haojia Yu, Hongcan Xiao, Hongli Wang, Huan Wu, Huaping Zhong, Jian Fang, Jianan Fan, Jiaqi Li, Jiefan Lu, Jing Zuo, Jingcheng Ni, Junxiang Xu, Linjun Dai, Mutian Xu, Peishen Yan, Penghao Wu, Ruijie Mao, Ruisi Wang, Shihao Bai, Shuang Yang, Shuya Yang, Shuyan Zheng, Silei Wu, Siying Li, Tao Chu, Tianbo Zhong, Tongxi Zhou, Weichao Luo, Weichen Fan, Wenhao Jia, Wenjie Gao, Xiangli Kong, Yan Li, Yang Yong, Zimo Wen, Zixuan Qian, Wenxiu Sun, Ruihao Gong, Quan Wang, Lewei Lu, Lei Yang, Ziwei Liu, Dahua Lin

arXiv:2609.11929v1cs.CV

TL;DR

Visual creation has broadened beyond conventional image synthesis, while many systems retain separate encoder-based perception and VAE-based generation pathways. SenseNova-U1.5 uses a native unified architecture, and evaluations show strong understanding alongside high-fidelity generation, editing, interleaved generation, and reasoning-intensive visual tasks.

  • Problem

    As visual creation expands beyond conventional image synthesis, most systems use vision encoders for perception and VAEs for generation, leaving the tasks in distinct representational spaces.

    Source: §1 · p. 5

  • Method

    SenseNova-U1.5 is an 8B native unified model that jointly reconstructs image patches spatially and consolidates specialized visual-task experts through on-policy distillation.

    Source: §1 · p. 5

  • Results

    SenseNova-U1.5 sustains strong visual understanding while supporting high-fidelity generation, editing, interleaved generation, and reasoning-intensive visual tasks, and generalizes to long, structured instructions.

    Source: §1 · p. 5, §6 · p. 26

  • Takeaways & Limitations

    These findings position native unified modeling as a promising direction for multimodal systems with a shared substrate for understanding, reasoning, and generation.

    Source: §6 · p. 26

Abstract

from arXiv · show

We launch SenseNova-U1.5, an 8B-MoT native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. We strengthen its visual interface through spatially coherent patch reconstruction and scale its training with carefully curated generation and editing data, improved task formulation, structural prompt enhancement, and native resolutions of up to 4K. For post-training, we optimize specialized experts for visual aesthetics, bilingual text rendering, infographic generation, and image editing, and consolidate their capabilities through multi-expert on-policy distillation. Across extensive evaluations, SenseNova-U1.5 largely advances image fidelity, text rendering, complex composition, multi-reference editing, and interleaved generation, while improving instruction following and preserving subject identity, geometry, and unmodified regions. Despite limited exposure to structured formats in its generation data, SenseNova-U1.5 generalizes effectively to long, complex, and structured visual instructions, further proving that multimodal understanding can transfer to visual planning and creation. Together, these findings position native unified modelling as a promising path towards systems that perceive, reason and create within a fully end-to-end framework. We will open-source training code, including supervised fine-tuning, reinforcement learning, and on-policy distillation.

1 Introduction

SenseNova-U1.5 is an 8B native unified multimodal model that improves spatially coherent visual generation through joint patch reconstruction and consolidates specialized generation capabilities through multi-expert distillation. Evaluations show that its compact shared visual representation supports understanding, reasoning, generation, editing, and long structured visual instructions.

Introduction

  • Most systems use pretrained vision encoders [97] [107] [152] for perception and VAEs [55] for generation, separating their representational spaces and limiting coordination.SenseNova-U1 [29] established encoder-free and VAE-free NEO-unify, but independent RGB-patch reconstruction impeded information exchange and caused high-resolution seams, texture discontinuities, and geometric inconsistencies.Source: §1 · p. 5
  • Spatially joint reconstruction lets neighbouring regions exchange information before pixel formation, improving spatial coherence and enabling native generation at resolutions up to 4K.Visual tokens are projected onto a 2D feature field and progressively reconstructed with spatial convolutions and Pixel Shuffle upsampling.Source: §1 · p. 5
  • Dedicated RL experts for aesthetics, bilingual text, infographics, and editing are unified through on-policy distillation along the student’s own generation trajectories.Each expert uses tailored data, rewards, sampling strategies, and regularization to address distinct optimization challenges.Source: §1 · p. 5
  • SenseNova-U1.5 uses one visual representation for understanding, reasoning, generation, and editing, with strong performance across fidelity, typography, composition, editing, instruction following, and visual preservation.Each 32 × 32-pixel region is compressed into one visual token, supporting efficient inference without parallel visual pathways or repeated conversion between encoder features and VAE latents.Source: §1 · p. 5
  • Despite limited reliance on fixed generation templates during training, SenseNova-U1.5 generalizes to long, compositional, and highly structured visual instructions.The paper suggests that structural knowledge and planning acquired through multimodal understanding and reasoning can transfer to visual creation.Source: §1 · p. 5

2 Related Works

Related work traces native unified modeling across visual understanding and generation, alongside reinforcement-learning and on-policy-distillation methods that inform SenseNova-U1.5’s training design.

2.1 Native Multimodal Unified Models

  • Native VLMs and pixel-modeling generators motivate unified models that combine understanding and generation through discrete autoregression or continuous end-to-end modeling.Prior native VLMs include Fuyu-8B, EVE [25] [27], Mono-InternVL [83] [84], NEO [26] [28], Gemma4-12B [113], and Inkling; SenseNova-U builds on NEO-unify.Source: §2.1 · pp. 5–6

2.2 Reinforcement Learning for Diffusion Models

  • Task-dependent RL combines CPS or Precise sampling with interleaved or task-specific rewards to address distinct generation and editing optimization needs.Prior methods extend RL to diffusion and flow-matching models, while stochastic rollouts and sampling refinements support exploration and preserve flow dynamics [73] [121] [163].Source: §2.2 · p. 6

2.3 On-Policy Distillation for Unified Models

  • Unlike shared distillation recipes, the approach hard-routes supervision from four task-specialized external experts into one native pixel-space model.This preserves each expert’s task-dependent conditioning, guidance, and resolution policies while consolidating capabilities [85].Source: §2.3 · p. 6

3 Methodology

SenseNova-U1.5 combines encoder-free visual processing, spatially coherent image decoding, and shared multimodal reasoning and generation in one model. Its staged training integrates general capabilities with specialized experts, on-policy distillation, and task-specific rewards.

3.1 Model architecture

  • Compact 32 × 32 visual tokens feed a unified backbone, while a spatial decoder and specialized Transformer streams support coherent image synthesis and multimodal understanding.The decoder restores spatial topology and uses Pixel Shuffle and 3 × 3 convolutions to couple neighboring patches; asymmetric attention lets generation use preceding clean context without exposing it to stochastic generation states.Source: §3.1 · pp. 6–8, Figure 3 · p. 7
  • Resolution-dependent noise conditioning extends the reference range to 4096 × 4096, enabling the denoiser to account for resolution and aspect-ratio variation.A dedicated sinusoidal MLP embeds the normalized resolution-dependent noise scale and combines it with the diffusion timestep representation.Source: §3.1 · p. 7, Figure 3 · p. 7
  • A unified objective combines autoregressive language modeling, pixel-space flow matching, and LPIPS perceptual supervision to connect semantic understanding with visual formation.The complementary signals supervise text prediction, noise-to-image trajectories in RGB space, and perceptual coherence of generated endpoints.Source: §3.1 · p. 8

3.2 Training Procedure

  • Training progresses from generation pre-training through unified mid-training and instruction fine-tuning, adding native 4K synthesis, editing, and interleaved generation along the way.The stages use mixed understanding and generation data in mid-training and retain the combined generation objective during subsequent unified training.Source: §3.2 · p. 9
  • Capability-specific reinforcement learning trains experts for aesthetics, bilingual text rendering, image editing, and infographic generation.The infographic expert receives infographic-oriented mid-training and task-specific learning to improve small-text rendering, layout handling, information organization, and visual design.Source: §3.2 · pp. 10–11
  • On-policy velocity-field distillation consolidates the four specialized experts into one model using each frozen expert routed to its capability-specific samples.The student and routed expert are evaluated at the same student-generated state, timestep, and condition; queried timesteps shift from high-noise global structure toward low-noise details during training.Source: §3.2 · p. 11

3.3 Reward Modeling

  • Aesthetic and OCR rewards provide separate perceptual-quality and text-fidelity supervision, while infographic training routes between those rewards rather than introducing an infographic-specific model.OCR uses normalized multiset intersection-over-union, matching English at word level and Chinese at character level; stage-conditioned routing preserves the distinct reward scales and semantics.Source: §3.3 · p. 12
  • Editing reward evaluates instruction fulfillment, edit execution, visual quality, applicable text accuracy, and preservation of unedited regions, aggregating applicable criteria by their minimum score.Text-editing quality is omitted when text changes are not requested, and edit-execution score is zero if no requested change is visibly realized.Source: §3.3 · p. 12

4 Data Construction

Data construction expands and rebalances generation, editing, and interleaved corpora while adding specialized RL datasets. Across these data, curation emphasizes visual quality, instruction fidelity, high resolution, and complex tasks.

4.1 Image Generation Data

  • The generation corpus adds approximately 59M text–image pairs from 78 sources, with high-resolution data forming a substantial share of training.Approximately 88.2% of effective training volume exceeds 1024^2 resolution and 64.4% exceeds 2048^2. Multi-granularity captions, stronger Chinese-English coverage, targeted synthesis, and joint filtering improve alignment and balance.Source: §4.1 · p. 13, §4 · p. 13

4.2 Image Editing Data

  • The approximately 38M-example editing corpus spans general, infographic, reference-conditioned, and spatially controlled edits, with diversified instructions for precise, preservation-aware transformations.The mixture comprises ∼43% general editing, ∼42% spatially controlled infographic editing, and ∼15% reference-conditioned editing, with up to ten reference images per example. Structured prompts and chain-of-thought examples specify targets, constraints, and content to preserve; curation assesses fulfillment, fidelity, and preservation.Source: §4.2 · pp. 13–14

4.3 Interleaved Data

  • Interleaved data converts sources into unified text–visual trajectories [21] [139], supporting grounding, long-range progression, temporal continuity, and reasoning-conditioned generation.The corpus draws ∼44% from lifestyle-oriented trajectories, ∼29% from infographic data, approximately ∼19% from video-derived sequences, and ∼8% from reasoning-intensive examples. Shared processing and validation filter for linguistic validity, visual quality, cross-modal consistency, and trajectory coherence.Source: §4.3 · p. 14

4.4 RL Training Data

  • RL training uses specialized aesthetic, bilingual OCR, infographic, preference, and image-editing samples curated for targeted rewards and quality.The data include approximately 280K aesthetic prompts, 60K OCR prompts, and 120K filtered preference pairs; the OCR set includes approximately 20K short prompts expanded into approximately 40K longer prompts. The editing expert uses approximately 120K examples, roughly 80% local and 20% global, filtered for grounded instructions, faithful edits, and image quality.Source: §4.4 · p. 14

5 Experiments

SenseNova-U1.5 shows strong visual and language understanding alongside advances in bilingual generation, complex editing, and interleaved reasoning. Across these tasks, results highlight improved instruction following, composition, and preservation of visual content.

5.1 General Understanding

  • U1.5 scores 86.67 on MMLU-Pro, 90.41 on C-Eval, 93.35 on IFEval, and 69.00 on IFBench, improving instruction following over SenseNova-U1 while retaining multimodal understanding.It preserves or improves performance on most multimodal benchmarks versus SenseNova-U1 and compares favorably with strong modular and encoder-free baselines across visual evaluations.Source: §5.1 · p. 15

5.2 Image Generation

  • With prompt enhancement, U1.5 scores 60.22 on the English and 60.13 on the Chinese Qwen-Image-Bench subsets, leading evaluated open-source models; its 8B model also balances bilingual text-centric generation.On OneIG-Bench, its strongest advantage is bilingual text-centric generation, while alignment, reasoning, and style remain competitive.Source: Table 4 · p. 16, Table 5 · p. 16, §5.2 · pp. 16, 18
  • U1.5 reaches 0.92 on GenEval, the best overall open-source result, and scores 88.11 on DPG-Bench; it also strengthens structured-prompt composition on GenEval2.GenEval gains are especially clear in counting, position, and attribute binding; GenEval2 gains stand out in attribute, counting, and verb-related generation. DPG-Bench results remain balanced across fine-grained prompt dimensions.Source: Table 6 · p. 17, §5.2 · pp. 17–18, Table 7 · p. 17
  • On CVTG-2K, U1.5 achieves a 0.948 average and word accuracy above 0.95 in four- and five-region settings, while leading open-source performance on both LongText-Bench language tracks.LongText-Bench accuracy remains robust as text length and layout complexity increase, including for dense, structured, and bilingual rendering.Source: Table 11 · p. 19, §5.2 · pp. 19–20, Table 12 · p. 20
  • U1.5 leads open-source IGenBench without prompt enhancement and improves further with it; prompt enhancement and CoT also yield leading open-source BizGenEval and WISE performance.BizGenEval gains are especially clear in knowledge-intensive cases, while WISE gains are notable in cultural knowledge, biology, and chemistry.Source: Table 13 · p. 20, §5.2 · pp. 20–21, Table 14 · p. 21

5.3 Image Editing

  • U1.5 leads on ImgEdit and GEdit-Bench, improving edit execution and semantic consistency, while WeEdit results show strong instruction adherence, text clarity, and background preservation.On WeEdit, it outperforms other open-source models and remains competitive with leading proprietary systems; translation-related and reasoning-intensive edits remain challenging.Source: Table 16 · p. 22, §5.3 · pp. 22–23, Table 17 · p. 22
  • U1.5 leads open-source performance on OmniRef-Bench, preserving style, backgrounds, subjects, and poses; on RISEBench, CoT further improves results, especially for causal, logical, and temporal edits.Spatial editing benefits less consistently from explicit reasoning than implicit multi-step transformations.Source: Table 19 · p. 23, §5.3 · pp. 23–24, Table 20 · p. 24

5.4 Interleaved Generation

  • With CoT, U1.5 leads on OpenING; it also achieves state-of-the-art VBVR-Pro-Bench results, generalizes out of domain, and leads RealUnify-GEU while remaining competitive on Uni-MMMU-GaU.These results support generation as an intermediate representation for multimodal reasoning and understanding, not only as an output modality.Source: Table 21 · p. 24, §5.4 · p. 25, Table 22 · p. 25

6 Conclusion

SenseNova-U1.5 advances native multimodal modeling by learning visual understanding and generation within a shared computational substrate. Its results support strong understanding alongside high-fidelity generation, editing, interleaved generation, and reasoning-intensive visual tasks, while suggesting that reasoning and generation can interact within unified visual intelligence.

6 Conclusion

  • SenseNova-U1.5 shows that a compact native representation can sustain strong visual understanding while supporting high-fidelity generation, editing, interleaved generation, and reasoning-intensive visual tasks.These results extend native multimodal modeling beyond basic image synthesis.Source: §6 · p. 26
  • The paper frames multimodal unification as capability interaction: reasoning can improve visual creation and transformation, while generation can help solve visual problems.Seeing, reasoning, and generating are presented as expressions of shared visual intelligence rather than loosely coupled functions.Source: §6 · p. 26
  • SenseNova-U1.5 is presented as a step toward models that interpret, imagine, and act upon visual worlds through a unified mechanism.The conclusion emphasizes unified visual intelligence, not merely architectural simplification or modality combination.Source: §6 · p. 26

Related papers

Loading 2609.11929v1…