PaperlayerView exact revision on arXiv

Source-linked AI summary

SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture

Haiwen Diao, Penghao Wu, Hanming Deng, Jiahao Wang, Shihao Bai, Silei Wu, Weichen Fan, Wenjie Ye, Wenwen Tong, Xiangyu Fan, Yan Li, Yubo Wang, Zhijie Cao, Zhiqian Lin, Zhitao Yang, Zhongang Cai, Yuwei Niu, Yue Zhu, Bo Liu, Chengguang Lv, Haojia Yu, Haozhe Xie, Hongli Wang, Jianan Fan, Jiaqi Li, Jiefan Lu, Jingcheng Ni, Junxiang Xu, Kaihuan Liang, Lianqiang Shi, Linjun Dai, Linyan Wang, Oscar Qian, Peng Gao, Pengfei Liu, Qingping Sun, Rui Shen, Ruisi Wang, Shengnan Ma, Shuang Yang, Siyi Xie, Siying Li, Tianbo Zhong, Xiangli Kong, Xuanke Shi, Yang Gao, Yongqiang Yao, Yves Wang, Zhengqi Bai, Zhengyu Lin, Zixin Yin, Wenxiu Sun, Ruihao Gong, Quan Wang, Lewei Lu, Lei Yang, Ziwei Liu, Dahua Lin

arXiv:2605.12500v1cs.CV

TL;DR

Multimodal understanding and generation have largely evolved in isolation, with divergent representations and pipelines. SenseNova-U1 unifies them in a native architecture, and its variants rival top-tier understanding-only VLMs across multiple task areas while achieving strong any-to-image generation.

  • Problem

    Multimodal perception and generation have largely relied on distinct systems and training pipelines, producing divergent feature representations.

    Source: §1 · p. 5

  • Method

    SenseNova-U1 uses a native Mixture-of-Transformers architecture that processes understanding and generation streams in one sequence under shared self-attention.

    Source: §1 · p. 5, §3.2 · p. 8

  • Results

    SenseNova-U1 rivals top-tier understanding-only VLMs across five task areas and achieves strong any-to-image generation across conventional, knowledge-intensive, and text-rich scenarios.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The results suggest that a single native architecture can support perception, reasoning, high-fidelity generation, and interleaved multimodal interaction.

    Source: §6 · p. 34

Abstract

from arXiv · show

Recent large vision-language models (VLMs) remain fundamentally constrained by a persistent dichotomy: understanding and generation are treated as distinct problems, leading to fragmented architectures, cascaded pipelines, and misaligned representation spaces. We argue that this divide is not merely an engineering artifact, but a structural limitation that hinders the emergence of native multimodal intelligence. Hence, we introduce SenseNova-U1, a native unified multimodal paradigm built upon NEO-unify, in which understanding and generation evolve as synergistic views of a single underlying process. We launch two native unified variants, SenseNova-U1-8B-MoT and SenseNova-U1-A3B-MoT, built on dense (8B) and mixture-of-experts (30B-A3B) understanding baselines, respectively. Designed from first principles, they rival top-tier understanding-only VLMs across text understanding, vision-language perception, knowledge reasoning, agentic decision-making, and spatial intelligence. Meanwhile, they deliver strong semantic consistency and visual fidelity, excelling in conventional or knowledge-intensive any-to-image (X2I) synthesis, complex text-rich infographic generation, and interleaved vision-language generation, with or without think patterns. Beyond performance, we show detailed model design, data preprocessing, pre-/post-training, and inference strategies to support community research. Last but not least, preliminary evidence demonstrates that our models extend beyond perception and generation, performing strongly in vision-language-action (VLA) and world model (WM) scenarios. This points toward a broader roadmap where models do not translate between modalities, but think and act across them in a native manner. Multimodal AI is no longer about connecting separate systems, but about building a unified one and trusting the necessary capabilities to emerge from within.

1 Introduction

SenseNova-U1 frames multimodal understanding and generation as jointly learned capabilities in a native architecture trained directly from pixels and words. The introduction presents it as a step toward unifying perception, reasoning, and generation rather than connecting separate modality-specific systems.

1 Introduction

  • Understanding and generation have often evolved in separate systems, with vision encoders and VAEs, distinct objectives, and training pipelines producing divergent representations.Earlier unified models connected perception and generation through separate tokenizers, latent spaces, or auxiliary modules rather than learning them jointly.Source: §1 · p. 5
  • Discrete-token approaches can compress non-linguistic signals, while continuous visual interfaces face trade-offs, leaving semantic abstraction and pixel-level fidelity unreconciled.The introduction poses native unification from pixels and words, without latent bottlenecks or intermediate representations, as an open challenge.Source: §1 · p. 5
  • SenseNova-U1 combines a near-lossless visual interface and raw-input end-to-end modeling with a native MoT architecture, jointly training language cross-entropy and vision pixel-space flow matching.It dispenses with pretrained vision encoders and deep decoder heads; the MoT design aims to limit interference between heterogeneous objectives while scaling efficiently.Source: §1 · p. 5
  • SenseNova-U1’s 8B and A3B MoT variants rival top-tier understanding-only VLMs across five areas and deliver strong X2I generation across conventional, knowledge-intensive, and text-rich scenarios.The variants use dense 8B and MoE 30B-A3B backbones and support X2I generation at a 32× compression ratio, as well as coherent interleaved multimodal generation.Source: §1 · p. 5
  • Preliminary experiments suggest SenseNova-U1 can perform vision-language-action and world-model tasks, reasoning and acting across modalities without external adapters or modular bridges.The paper presents this as evidence for a broader shift toward natively unified perception, reasoning, and generation.Source: §1 · p. 5

2 Related Works

Related work moves from modular vision–language models toward native pixel–word modeling, while unified systems still face trade-offs between architectural sharing, intermediate representations, and visual fidelity. SenseNova-U1 extends NEO-unify’s continuous native direction across data, model capacity, and application scenarios.

2.1 Native Multimodal Models

  • VLMs couple visual encoders with LLMs through staged pretraining or joint optimization, but inherit semantic biases and add complexity with capacity trade-offs [2] [52] [57] [115] [134].Native backbones remove visual encoders, and follow-on work mitigates vision–language conflicts through distillation [29] [70] [131], data mixing [63], shared modules [121] [151], and modality decomposition [31] [88] [89].Source: §2.1 · pp. 5–6
  • VE-free native backbones include Fuyu and EVE [29], while NEO [30] narrows the gap with modular VLMs across understanding tasks.NEO explores a native pixel–word primitive.Source: §2.1 · p. 6
  • VAE/VQ-VAE compression limits semantic expressivity [58] [128], while direct pixel-space modeling [17] [20] [69] [158] points toward end-to-end optimization from raw pixels.Pretrained representations or auxiliary objectives enrich latents but leave compression bottlenecks and fragmented pipelines.Source: §2.1 · p. 6

2.2 Native Multimodal Unified Models

  • Shared-backbone systems such as Show-o [145] [146], Janus [18] [92] [140], OmniGen [141] [144], and BAGEL [28] unite perception and synthesis but retain different tokenizers, heads, or pathways.Shared discrete tokenizers [76] [90] [105] [143] [167] and continuous autoencoders [13] [35] [84] [113] [127] [161] [168] partly reconcile the tasks but trade semantic structure against visual fidelity.Source: §2.2 · p. 6
  • Discrete unified models [23] [66] [73] [78] [122] [125] [135] gain autoregressive architectural unity but sacrifice fidelity and expressivity; continuous NEO-unify models near-lossless inputs, and Tuna-2 [85] matches latent-space methods.SenseNova-U1 builds on NEO-unify, scaling across data corpus, model capacity, and application scenarios.Source: §2.2 · p. 6

3 Methodology

SenseNova-U1 presents NEO-unify as a native end-to-end framework that processes pixels and words directly, unifying multimodal understanding and generation without pretrained encoder priors. Its method combines a shared multimodal backbone with pixel-space generation, staged joint training, and separately optimized inference pathways.

3 Methodology

  • SenseNova-U1 operates directly on pixels and words, avoiding pretrained encoder priors and fixed-representation scaling limits.The framework is presented in Figure 4.Source: §3 · p. 7

3.1 Near-Lossless Visual Interface

  • Visual patches and unchanged text tokens share an embedding space; generation predicts pixel patches directly, while resolution-adaptive noise scaling maintains consistent SNR across resolutions.Two convolutional layers with strides 16 and 2 yield 32 × 32 image patches; generation uses an MLP head, and the noise scale initializes both training noise and the inference flow ODE.Source: §3.1 · p. 7

3.2 Native Multimodal Unified Modeling

  • The MoT backbone jointly processes clean and noise-conditioned tokens with shared self-attention, stream-specific parameters, and native rotary encoding across temporal and spatial axes.The 8B variant uses symmetric dense streams, while A3B uses stream-wise experts with about 3B active parameters per inference token.Source: §3.2 · p. 8

3.3 Joint Training Objective

  • Generation uses pixel-space flow matching alongside next-token text prediction, with classifier-free guidance independently controlling text and image conditions.For X2I tasks, γ = 4 and γimg = 1 consistently yield the best performance; text conditions are dropped 10% of the time and both conditions an additional 10%.Source: §3.3 · pp. 8–9

3.4 Training Procedure

  • Training progresses from understanding warmup and frozen-branch generation pretraining to joint multimodal training and instruction fine-tuning across understanding and generation tasks.Joint training uses text-only and multimodal understanding, text-to-image, editing, and interleaved data at sampling ratios 0.33 : 0.37 : 0.24 : 0.06.Source: §3.4 · pp. 9–10
  • SenseNova-U1 post-training uses Flow-GRPO reinforcement learning [80] and interleaved text-rendering, style-following, and general-visual-quality rewards; DMD2 distillation [157] reduces image-synthesis function evaluations from 100 to 8.Source: §3.4 · pp. 10–12

3.5 Inference Infrastructure

  • Inference separates understanding and image generation across LIGHTLLM and LIGHTX2V, enabling workload-specific parallelism, resource allocation, and operational scaling.The engines exchange generation state through pinned shared memory while preserving a unified API abstraction.Source: §3.5 · p. 12, Figure 5 · p. 12
  • A hybrid attention kernel keeps text rows causal while letting image rows attend to the full preceding text prefix and image span, preserving the causal fast path for text-only blocks.The mechanism is implemented in both a Triton kernel and a modified FlashAttention3 backend.Source: §3.5 · p. 13, Figure 6 · p. 13

4 Data Construction

SenseNova-U1’s data construction organizes understanding, generation, and interleaved data into capability-focused corpora, using curation and validation pipelines to shape their quality, diversity, and training coverage.

4.1 Understanding Data Organization

  • The pretraining corpus comprises image-text pairs (32%), captions (17%), infographic understanding (14%), and pure text (37%), curated through deduplication, filtering, and balanced re-captioning.The pipeline also includes content and safety filtering and image-quality filtering.Source: §4.1 · p. 13
  • Midtraining combines four data categories with diversity sampling, prompt augmentation, and multi-criteria QA filtering.The mix is General (39.2%), Agent and Spatial (22.3%), Knowledge Reasoning (19.3%), and Pure Text (19.2%); filtering checks correctness, visual support, and instruction-following.Source: §4.1 · pp. 13–14, Figure 7 · p. 14
  • The SFT corpus spans fine-grained capability categories, led by spatial intelligence (∼15%), general multimodal understanding (∼13%), and reasoning (∼12%).Remaining categories include general NLP, OCR and document analysis, agentic function calling, long-context conversation, code, multi-turn dialogue, complex compositional understanding, and supplementary capabilities.Source: §4.1 · p. 14
  • SFT candidates are selected for quality and reconstructed for difficulty by combining longer contexts, intermediate-difficulty reasoning examples, and more explicit query constraints.Quality scoring covers visual fidelity, instruction clarity, response correctness, reasoning quality, and safety; reconstruction includes long-context, multi-image, and multi-turn examples.Source: §4.1 · p. 14

4.2 Generation Data Organization

  • The generation corpus spans text-to-image and editing data across four domains, with balanced coverage and a unified filtering, deduplication, captioning, and quality-control pipeline.Text-to-image data includes Nature (∼40.5%), People (∼26.7%), and Design (∼20.7%), alongside infographics and bilingual text rendering.Source: §4.2 · pp. 14–15, Figure 8 · p. 15
  • Editing data covers varied content and operations, and each pair is checked against dynamic change objectives and static physical consistency with the source image.Natural scenes (∼52.3%) and human subjects (∼14.7%) dominate real-world coverage.Source: §4.2 · p. 15
  • Interleaved data alternates text and images across lifestyle (∼44%), infographics (∼29%), video (∼19%), and reasoning (∼8%), with quality and trajectory checks.Reasoning examples include explicit chain-of-thought traces; the pipeline jointly verifies text quality, image quality, image–text consistency, and trajectory-level correctness.Source: §4.2 · p. 15

5 Experiments · 5.1 Main Results

This section reports SenseNova-U1’s results across image and text understanding, generation, editing, and interleaved tasks, including compositional image generation, text-rich synthesis, and multimodal reasoning. It also evaluates the reciprocal benefits between understanding and generation on RealUnify.

5.1.1 Image Understanding

  • SenseNova-U1 outperforms same-LLM Qwen3VL-8B on multimodal reasoning, gains on text-rich understanding, and matches or surpasses leading models on general vision-language and hallucination benchmarks.Spatial evaluations also report strength across high-level reasoning and low-level geometric representation; VSI-Bench uses 32 frames for all models except Qwen variants, which use 128.Source: §5.1.1 · pp. 16–17

5.1.2 Text Understanding

  • SenseNova-U1 shows strong text understanding, especially on IFEval and IFBench, and its A3B variant demonstrates multi-turn agent capabilities across τ 2-Bench and Claw-Eval.It surpasses Qwen3.5 on both instruction-following benchmarks, narrows the gap with Qwen3.5 on MMLU-Pro and SuperGPQA, and approaches substantially larger reasoning-oriented models on agent benchmarks despite fewer active parameters. The paper attributes a remaining gap to dense reasoning-focused baselines partly to training trade-offs favoring high-fidelity generation and interleaved vision-language capabilities.Source: §5.1.2 · p. 17

5.1.3 Image Generation

  • Both SenseNova-U1 variants score 0.91 overall on GenEval, exceeding Qwen-Image (0.87), Lumina-DiMOO (0.88), and BAGEL (0.82).The models show balanced compositional performance across object-level factors, although their Attribute Binding scores remain below OneCAT and Mogao.Source: §5.1.3 · p. 18, Table 5 · p. 18
  • SenseNova-U1 remains competitive on general generation benchmarks; A3B achieves the highest Global score on DPG-Bench, while the variants show strengths in OneIG alignment and text rendering.On OneIG, A3B leads English Alignment and 8B achieves the best Text performance in both English and Chinese; on TIIF-Bench, 8B has the best overall results across short and long instruction settings.Source: §5.1.3 · pp. 19–20, Table 6 · p. 19, Table 7 · p. 20
  • SenseNova-U1-8B-MoT achieves 0.940 average word accuracy on CVTG-2K and scores 0.979 on LongText-Bench-EN and 0.962 on LongText-Bench-ZH.A3B scores 0.950 and 0.955 on the respective LongText-Bench tracks despite not being fully converged; CVTG-2K results remain strong across settings with 2–5 text regions.Source: §5.1.3 · pp. 21–22, Table 11 · p. 22, Table 12 · p. 22
  • SenseNova-U1 achieves the strongest IGenBench performance among open-source models and the best hard-split BizGenEval average among open-source models; CoT makes WISE competitive with closed-source systems.On WISE, A3B-SFT substantially outperforms representative open-source models even without CoT, and the 8B variant shows a similar reasoning-enhanced trend.Source: Table 13 · p. 23, Table 14 · p. 23, §5.1.3 · p. 24

5.1.4 Image Editing

  • SenseNova-U1 outperforms open-source unified editing systems on ImgEdit and performs competitively on GEdit-Bench, while specialized systems retain advantages on complex edits.On RISEBench, A3B-SFT improves from 25.3 without CoT to 30.0 with CoT; 8B-SFT improves from 23.9 to 26.9, with A3B Logical rising from 7.1 to 20.0.Source: §5.1.4 · pp. 25–26, Table 16 · p. 25, Table 17 · p. 26

5.1.5 Interleaved Generation

  • On OpenING, A3B-SFT with CoT scores 9.16 and 8B-SFT with CoT scores 9.07; on Uni-MMMU GaU, 8B-SFT scores 35.0 and A3B-SFT scores 32.6.SenseNova-U1 also outperforms competitive in-domain baselines and several proprietary systems on the preview VBVR-Image benchmark, where reasoning is exercised through visual generation.Source: §5.1.5 · pp. 27–28, Table 19 · p. 28, Table 20 · p. 28
  • SenseNova-U1-8B-MoT-SFT achieves 52.4 overall, with 55.7 on Avg-UEG and 47.5 on Avg-GEU; SenseNova-U1-A3B-MoT-SFT achieves 50.5 overall.The reported results use step-wise inference; GEU uses a single interleaved process, while UEG uses text-to-image generation inference.Source: Table 22 · p. 29, §5.1.5 · p. 29

5.2 Ablation Studies

The ablations examine whether encoder-free design retains semantic and pixel detail, whether understanding and generation work together in the MoT backbone, and whether performance scales with data. Results support reconstruction fidelity, editing capability, low-conflict co-evolution, and improving quality with more training data.

5.2.1 Native Encoder-Free Design Preserves Both Semantic and Pixel Representations

  • On MS-COCO 2017, NEO-unify (2B) reached 31.56 PSNR and 0.85 SSIM after 90K pretraining steps, versus FLUX.1-dev VAE’s 31.56 PSNR and 0.93 SSIM.The result suggests the native encoder-free interface retains semantic information and fine-grained visual detail without pretrained vision encoders or latent autoencoders.Source: §5.2.1 · p. 29, Table 23 · p. 30
  • With its understanding branch frozen, NEO-unify (2B) routed conditional contexts through understanding and directly synthesized edits, scoring 3.32 on ImgEdit after 60K mixed-training steps using public text-to-image and editing datasets.The passage reports strong editing capability despite the frozen understanding branch.Source: §5.2.1 · pp. 29–30

5.2.2 Understanding and Generation Synergize with Native MoT Backbone

  • Even with low data ratios and small understanding-loss weights, understanding remained stable as generation converged rapidly, with the MoT capabilities co-evolving with minimal intrinsic conflict.All components were jointly optimized during mid-training, starting from pretrained dual branches.Source: §5.2.2 · p. 31, Figure 12 · p. 31

5.2.3 Native Multimodal Architecture Shows High Data-Scaling Efficiency

  • As training data scaled, generation quality and understanding–generation synergy improved steadily, indicating strong data-scaling efficiency.The training sequence comprised web-scale pretraining, followed by mid-training and supervised fine-tuning on diverse, high-quality understanding and generation corpora.Source: §5.2.3 · p. 32, Figure 13 · p. 32

5.3 Visualization Results

The section presents qualitative visualizations of SenseNova-U1 across multimodal tasks, including video-based action reasoning and action-conditioned world-model predictions.

Visualization Results

  • The showcases span text-to-image and infographic generation, image editing, interleaved image-text generation, visual understanding, and agentic multimodal interaction.They cover both general and reasoning-intensive settings.Source: §5.3 · p. 32

Vision-Language-Action

  • Four uniformly sampled video frames illustrate SenseNova-U1 reasoning about object states, manipulation trajectories, and task progression from sparse temporal observations.The examples show action-relevant visual dynamics and coherent understanding in embodied settings.Source: §5.3 · p. 32, Figure 14 · p. 33

World Modeling

  • Given an image and action-oriented instruction, SenseNova-U1 predicts plausible visual state transitions while maintaining scene consistency and object coherence.The figure displays simplified instructions, while inference uses the original prompts.Source: §5.3 · p. 33, Figure 15 · p. 33

6 Conclusion

The paper presents a unified multimodal foundation model in which understanding, generation, and reasoning emerge within one native architecture. Its results suggest shared representations can support perception, creative generation, and interleaved interaction, with early VLA and world-model advances pointing toward embodied, goal-directed intelligence.

6 Conclusion

  • Understanding, generation, and reasoning emerge within a single native architecture rather than through coordination among separate systems.The conclusion frames unified architectures as an alternative to increasingly complex collections of modular components.Source: §6 · p. 34
  • Across broad task ranges, the model shows strong vision-language perception, semantic reasoning, high-fidelity generation, and interleaved multimodal interaction.The paper argues that shared representations can support both analytical and creative intelligence.Source: §6 · p. 34
  • Early advances in vision-language-action models and world modeling point toward embodied, goal-directed intelligence beyond passive understanding.The conclusion describes unified models as beginning to internalize a coherent abstraction of the world, enabling perception, imagination, and decision-making within a shared latent space.Source: §6 · p. 34

Related papers

Loading 2605.12500v1…