PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCat-Next: Lexicalizing Modalities as Discrete Tokens

Meituan LongCat Team, Bin Xiao, Chao Wang, Chengjiang Li, Chi Zhang, Chong Peng, Hang Yu, Hao Yang, Haonan Yan, Haoze Sun, Haozhe Zhao, Hong Liu, Hui Su, Jiaqi Zhang, Jiawei Wang, Jing Li, Kefeng Zhang, Manyuan Zhang, Minhao Jing, Peng Pei, Quan Chen, Taofeng Xue, Tongxin Pan, Xiaotong Li, Xiaoyang Li, Xiaoyu Zhao, Xing Hu, Xinyang Lin, Xunliang Cai, Yan Bai, Yan Feng, Yanjie Li, Yao Qiu, Yerui Sun, Yifan Lu, Ying Luo, Yipeng Mei, Yitian Chen, Yuchen Xie, Yufang Liu, Yufei Chen, Yulei Qian, Yuqi Peng, Zhihang Yu, Zhixiong Han, Changran Wang, Chen Chen, Dian Zheng, Fengjiao Chen, Ge Yang, Haowei Guo, Haozhe Wang, Hongyu Li, Huicheng Jiang, Jiale Hong, Jialv Zou, Jiamu Li, Jianping Lin, Jiaxing Liu, Jie Yang, Jing Jin, Jun Kuang, Juncheng She, Kunming Luo, Kuofeng Gao, Lin Qiu, Linsen Guo, Mianqiu Huang, Qi Li, Qian Wang, Rumei Li, Siyu Ren, Wei Wang, Wenlong He, Xi Chen, Xiao Liu, Xiaoyu Li, Xu Huang, Xuanyu Zhu, Xuezhi Cao, Yaoming Zhu, Yifei Cao, Yimeng Jia, Yizhen Jiang, Yufei Gao, Zeyang Hu, Zhenlong Yuan, Zijian Zhang, Ziwen Wang

arXiv:2603.27538v1cs.CVcs.CL

TL;DR

Existing multimodal systems commonly keep non-linguistic modalities as loosely coupled additions to language, leaving open how diverse signals can be represented in one discrete autoregressive space. LongCat-Next introduces DiNA and dNaViT to unify text, vision, and audio through shared discrete tokens and paired tokenization processes. Experiments report strong multimodal performance alongside preserved language capabilities, while the authors identify tokenizer fidelity and limited study scope as remaining boundaries.

  • Problem

    Multimodal systems often treat non-linguistic modalities as subordinate attachments, and practical methods for representing vision and audio in a shared discrete token space remain limited.

  • Method

    DiNA unifies text, vision, and audio under one autoregressive objective, using dNaViT for hierarchical any-resolution visual tokenization and modality-specific tokenizer–detokenizer pairs.

  • Results

    LongCat-Next achieves strong performance across multimodal tasks, including visual understanding and image generation, without compromising its foundational language capabilities.

  • Takeaways & Limitations

    The results suggest that carefully designed discrete tokenizers and training strategies can extend language-style autoregressive modeling to continuous perceptual signals.

  • Takeaways & Limitations

    The current study is limited by computational resources and data availability, while the dNaViT detokenizer prioritizes semantic consistency over pixel fidelity.

Abstract

from arXiv · show

The prevailing Next-Token Prediction (NTP) paradigm has driven the success of large language models through discrete autoregressive modeling. However, contemporary multimodal systems remain language-centric, often treating non-linguistic modalities as external attachments, leading to fragmented architectures and suboptimal integration. To transcend this limitation, we introduce Discrete Native Autoregressive (DiNA), a unified framework that represents multimodal information within a shared discrete space, enabling a consistent and principled autoregressive modeling across modalities. A key innovation is the Discrete Native Any-resolution Visual Transformer (dNaViT), which performs tokenization and de-tokenization at arbitrary resolutions, transforming continuous visual signals into hierarchical discrete tokens. Building on this foundation, we develop LongCat-Next, a native multimodal model that processes text, vision, and audio under a single autoregressive objective with minimal modality-specific design. As an industrial-strength foundation model, it excels at seeing, painting, and talking within a single framework, achieving strong performance across a wide range of multimodal benchmarks. In particular, LongCat-Next addresses the long-standing performance ceiling of discrete vision modeling on understanding tasks and provides a unified approach to effectively reconcile the conflict between understanding and generation. As an attempt toward native multimodality, we open-source the LongCat-Next and its tokenizers, hoping to foster further research and development in the community. GitHub: https://github.com/meituan-longcat/LongCat-Next

1 Introduction

LongCat-Next proposes native multimodality by representing text, vision, and audio as discrete tokens under one autoregressive framework. Its tokenizer designs and unified model target both multimodal capability and competitive language performance.

  • Motivation: Prevailing multimodal systems often treat vision and audio as loosely coupled additions to language modeling.The paper frames native multimodal modeling as a way to move beyond the language-plus-auxiliary paradigm.
  • Visual tokenization: dNaViT tokenizes and detokenizes images at arbitrary resolutions, producing semantically complete hierarchical tokens with up to 28 × compression.Multi-level residual tokens support both visual understanding and image reconstruction or generation.
  • Audio modeling: The audio tokenizer uses RVQ and a Whisper encoder to compress waveforms into discrete tokens at 12.5 Hz.A paired decoder and flow-matching refinement network support high-fidelity reconstruction, while stochastic delays align text and audio segments for autoregressive modeling.
  • Unified model: LongCat-Next unifies language, vision, and audio in a single model designed to reconcile competing multimodal objectives.Its architecture treats vision and audio as intrinsic extensions of language rather than external attachments.
  • DiNA: DiNA represents all modalities in a shared discrete token space under a consistent autoregressive objective.This reduces the central design problem to modality-specific tokenizer–detokenizer pairs while retaining decoder-only architectural simplicity.
  • Results: LongCat-Next surpasses Qwen3-Omni, outperforms Qwen3VL-A3B on visual understanding, and competes favorably with Flux-dev for high-fidelity image generation.The model is instantiated with an A3B configuration totaling 68.5B parameters and trained on over 2T tokens.

2 Methodology

The methodology frames multimodality as shared discrete token sequences under one autoregressive objective, then addresses visual representation capacity and discretization loss with dNaViT. LongCat-Next combines modality-specific tokenizers and detokenizers with a modality-agnostic autoregressive backbone.

  • Unified multimodal modeling: A unified discrete framework represents multimodal signals as interoperable token sequences governed by one next-token prediction objective.This is presented as a way to combine understanding and generation within a shared modeling framework.
  • Unified multimodal modeling: The framework targets an industrial-strength training recipe that can scale unified systems without sacrificing performance relative to specialized models.The stated criteria include performance parity or better in comprehension and generation, alongside preserved language capabilities.
  • System architecture: LongCat-Next uses modality-specific tokenizer and detokenizer pairs while keeping the decoder-only backbone modality-agnostic.The backbone serves as a multi-task learner for language, visual understanding and generation, and audio comprehension and synthesis.
  • Vision tokenizer motivation: Visual discretization is difficult because images are dense and continuous, while finite codebooks can lose information and understanding-optimized representations can diverge from generation-optimized ones.These constraints create a challenge for a unified visual interface.
  • Vision tokenizer: dNaViT is introduced as an any-resolution visual tokenizer for both comprehension and generation, designed around visual representation capacity and discretization loss.Its goal is a discrete representation that preserves enough information for discriminative understanding and high-fidelity generation.
  • Semantic completeness: Semantic completeness defines a discrete visual representation as an approximately lossless proxy for the original image across image-centric downstream tasks.The formulation requires the response distribution conditioned on discrete representation z to approximate that conditioned on the original image I.
  • Semantic encoder and tokenization: The proposed visual tokenization pipeline uses language-aligned semantic encoders, hierarchical discretization, and downstream autoregressive modeling.Existing vision-language encoders can serve as SAE candidates, avoiding dedicated SAE training from scratch while providing semantic structure.

Dicussion about the Information Recovery

The paper argues that semantically complete visual representations must also recover original visual signals, and attributes this capability partly to residual architectures. dNaViT builds on this insight by preserving hierarchical visual information in discrete tokens for both understanding and generation.

  • Information Recovery: Semantic completeness requires visual representations to support comprehension and recover the original visual signal.The paper calls this generative sufficiency and links it to intrinsic information recovery.
  • Information Recovery: Frozen semantic encoders with residual pathways retain reconstruction capability even without explicit reconstruction supervision.Figure 5 compares visual reconstructions using different frozen encoders and a lightweight pixel decoder.
  • Information Recovery: Residual connections preserve fine-grained visual signals by carrying identity mappings alongside progressively integrated feature increments.Earlier-layer information is not overwritten by higher-level semantic abstractions.
  • dNaViT: dNaViT represents images as semantically meaningful discrete sequences compatible with autoregressive modeling.Shared token representations support both captioning and synthesis as inverse sequence-modeling tasks.
  • dNaViT: dNaViT operates at native image resolution, avoiding resizing, cropping, and padding distortions while preserving details across arbitrary scales.It constructs hierarchically organized tokens that encode both global structure and local details.

2.3 Audio Tokenizer

The audio tokenizer converts speech into discrete representations that retain semantic and acoustic information while supporting both language-model understanding and waveform reconstruction. Its pipeline combines Whisper features, RVQ quantization, and a refinement-based decoder.

  • Tokenizer Pipeline: Whisper features are downsampled by 4× and quantized with an 8-layer residual vector quantizer into discrete audio tokens.The tokenizer processes continuous speech before splitting the resulting tokens into language-model and reconstruction branches.
  • Tokenizer Pipeline: The discrete audio tokens encode semantic and acoustic information and align with the pretrained LLM’s textual embedding space.The representations are reported as transferable across LLM architectures and vocabularies.
  • Reconstruction: A symmetric decoder reconstructs coarse Mel spectrograms, while flow matching refines them before vocoding into waveforms.The refinement stage is used to improve reconstruction fidelity.

2.4 Language Model Backbone

LongCat-Next uses a decoder-only, modality-agnostic MoE language backbone to process textual, visual, and acoustic tokens through one pathway. The architecture is intended to support multimodal multi-task learning without modality-specific backbone design.

  • Backbone: A modality-agnostic MoE language backbone functions as a multi-task learner across modalities.The model dynamically allocates capacity to multiple objectives without requiring modality-specific design.
  • Backbone: LongCat-Flash-Lite A3B is trained from scratch with 68.5B total parameters and an average of 3B activated parameters.Activated parameters range from 2.9B to 4.5B depending on context.
  • Backbone: Textual, visual, and acoustic tokens are processed through a single modality-agnostic pathway.This shared pathway is presented as part of the DiNA framework.
  • Speech Generation: Figure 7 distinguishes pure audio user input from text-guided audio generated by LongCat-Next and uses AS, AE, and TE tokens for modality conversion and segment alignment.AS and AE mark audio boundaries, while TE marks the end of a text segment.

2.5 Multimodality Component

The multimodality component uses jointly learned token-level embeddings and modality-aware tokenizers to integrate language, vision, and audio. Multi-level supervision separates semantic prediction from reconstruction of modality-specific details, while text-guided audio supports parallel and serial generation.

  • Native Representation: All multimodal embeddings are randomly initialized and jointly trained so vision, audio, and language share a unified token-level representation.This differs from projecting continuous visual features into the language model’s embedding space.
  • Native Representation: Visual embeddings use eight levels of 16,384-codebook entries combined by multi-level summation, while audio uses decreasing codebook sizes.Embeddings are not shared across visual levels and are learned end-to-end.
  • Multimodality Head: The language model predicts tokens autoregressively while a task-aware DepthTransformer decodes multi-level tokens in parallel to reconstruct visual and audio details.This multi-level supervision scheme separates semantic-level compression from modality-specific reconstruction.
  • Internal Linguistic Guidance: Text-guided audio fuses aligned text and audio tokens through dedicated embeddings and element-wise summation.Pure audio denotes real-time input without textual guidance.
  • Internal Linguistic Guidance: Parallel generation produces text and audio simultaneously with delayed audio tokens, whereas serial generation produces guided text before its corresponding audio.Random delays during training are intended to support arbitrary delay steps.

3 Main Experiments

LongCat-Next is evaluated as a unified omni-modal system across visual understanding, visual generation, audio, and general multimodal tasks. It achieves competitive or leading results against unified, specialized, and audio-focused baselines.

  • Evaluation setup: LongCat-Next processes text, vision, and speech simultaneously, with Qwen3-Omni-A3B-Instruct as the primary omni-modal baseline and specialized models as complementary references.Visual understanding is additionally compared with Qwen3-VL-A3B, while audio evaluation includes MiMo-Audio, Kimi-Audio, and Step-Audio-2-mini.
  • Visual understanding: 83.1 on MathVista and 64.7 on MathVision are the highest scores among compared models, while LongCat-Next reaches 70.6 on MMMU and 60.3 on MMMU-Pro.The model surpasses specialist MLLMs on the mathematics benchmarks and Qwen3-Omni-A3B-Instruct on MMMU and MMMU-Pro.
  • Visual understanding: 29.4 on VisuLogic places LongCat-Next first, ahead of InternVL3.5-A3B-Flash at 28.4 and Gemini2.5-Flash-Lite at 26.1.It also records 14.4 on BabyVision while not being specifically optimized for these datasets.
  • Visual understanding: 865 on OCRBench and 0.152 on OmniDocBenchen plus 0.226 on OmniDocBenchzh demonstrate strong document understanding and fine-grained textual perception.The model also leads the listed baselines on the reported OmniDocBench variants and exceeds Qwen3-Omni-A3B-Instruct, GPT5-minimal, and Gemini2.5-Flash-Lite on OCRBench.
  • Structured and general vision: 60.1 on CharXivRQ and 88.0 on ChartQA are the best reported scores, while 89.9 on CharXivDQ ranks second only to Gemini2.5-Flash-Lite.On MMStar, LongCat-Next scores 69.3, ahead of Qwen3-Omni-A3B-Instruct at 68.5 and GPT5-minimal at 65.2.
  • Visual generation: LongCat-Next outperforms prior unified approaches across most generation benchmarks and remains competitive with specialized T2I systems.The largest gains are reported for long-text understanding and text rendering, where language-guided generation helps preserve textual intent in complex compositions.
  • Audio: LongCat-Next excels at audio recognition and synthesis, surpasses most compared models in TTS accuracy, and achieves state-of-the-art performance across many audio-comprehension tasks.It also performs strongly on audio-instructed question answering.

Experiment ID I II III IV V VI VII

The experiments examine discrete-versus-continuous visual modeling, reconstruction, understanding-generation interaction, audio generation, MoE specialization, and cross-modal representation structure. Results indicate that training scale narrows discrete-model gaps while native multimodal training yields structured integration.

  • Discrete versus continuous modeling: dNaViT initially has higher pre-alignment loss than the continuous baseline, but the gap narrows with training and data scale.Under the vanilla discrete formulation, a noticeable downstream discrepancy remains in the early experiments.
  • Discrete versus continuous modeling: Within approximately a 1% margin of the continuous baseline, dNaViT maintains comparable downstream performance after sufficient training and data scaling.Experiments VI and VII approach parity, and large-scale LongCat-Next results remain competitive with Qwen3-VL-A3B-Instruct.
  • Reconstruction analysis: Residual pathways preserve non-trivial reconstruction capability without explicit reconstruction supervision, whereas aggressive 14× →28× downsampling degrades QwenViT reconstruction.The comparison isolates encoder characteristics using a lightweight ViT-based decoder.
  • Reconstruction analysis: A randomly initialized vanilla ViT achieves the highest PSNR among compared visual encoders.The authors suggest that noise-like outputs may be easier for the decoder to denoise, while residual pathways can preserve some pixel-level information.
  • Understanding and generation interaction: 79.33 versus 81.67 on LlamaQuestions and 74.95 versus 80.30 on ReasoningQA show that parallel audio generation remains comparable to serial generation.The parallel approach retains strong semantic fidelity while offering improved efficiency despite a slight gap.
  • MoE dynamics: Multimodal training causes some modality-agnostic MoE experts to specialize in vision or audio and increases average routed tokens per expert from 507.1 to 584.6.Routers also develop more distinct and stable expert-selection patterns.
  • Cross-modal representation: LongCat-Next exhibits interwoven visual and textual embeddings, contrasting with largely separated modality clusters in Qwen2.5-VL.The frozen visual tokenizer shows negligible performance degradation after adaptation to the LongCat architecture.

4 Implementation Details

Implementation proceeds from modality-specific tokenizers and detokenizers to unified multimodal training. The visual, audio, and text pipelines use discrete representations, staged optimization, and extensive multimodal data construction.

  • Training framework: DiNA optimization has two phases: modality-specific tokenizer training followed by unified multimodal training.Native multimodality training begins with pre-alignment of codebook embeddings and DepthTransformer decoders.
  • Visual tokenizer: dNaViT maps visual features to discrete IDs and reconstructs images at arbitrary resolution with up to 28 × compression.RVQ reduces quantization error, while sequence-based processing supports variable-resolution images.
  • Visual tokenizer: The visual detokenizer uses a 400M-parameter ViT pixel decoder and a flow-matching image refiner conditioned on reconstruction and discrete code embeddings.The SAE encoder and codebook remain frozen during this stage.
  • Audio tokenizer: The audio tokenizer uses Whisper-large-v3 features and RVQ to produce discrete tokens at 12.5 Hz, with a paired decoder and flow-matching refinement network for reconstruction.The audio corpus contains approximately 2.5 million hours of speech, multilingual, dialectal, music, and sound-captioning data.
  • Audio tokenizer: Audio tokenizer training comprises decoder warm-up, semantic-acoustic joint training, and subsequent staged optimization.The first stage freezes the encoder and language model while training the decoder on Mel spectrogram reconstruction.
  • Multimodal training: LongCat-Next training uses warm-up, full-modality pre-training, mid-training, and SFT, totaling approximately 2 trillion tokens.Mid-training adds synthetic and curated data, long chain-of-thought reasoning, and arbitrary-resolution visual feature training.
  • Data construction: The visual-language corpus combines caption, interleaved image-text, OCR, grounding, STEM, and GUI data to support broad multimodal comprehension.OCR construction mixes 75% in-house synthetic data with 25% filtered real-world data from over 90 open-source datasets.
  • Data construction: Progressively moving from noisy pairs to rebalanced clusters and high-fidelity instruction tuning improves concept coverage and aesthetic quality within an efficient computational budget.The text-to-image data strategy addresses long-tail concentration in generic portraits and common objects.

5 Infrastructure

LongCat-Next’s infrastructure addresses heterogeneous and dynamically varying modality workloads in pipeline parallelism. A profile-guided V-shaped schedule balances computation and reduces communication and pipeline bubbles.

  • Motivation: Embedding and modality-specific loss modules have execution times that differ from LLM layers and vary with each sample’s modality-token distribution.Naive linear, uniformly partitioned pipeline parallelism therefore creates pronounced imbalance.
  • V-Half schedule: The proposed schedule folds the pipeline into a V-shape, co-locating embedding and modality-loss stages on one physical device.A shared buffer complements this placement strategy.
  • Load balancing: Profile-guided assignment bounds the combined latency of embedding and modality-loss chunks within the ideal uniform chunk latency.The method profiles embedding, modality loss, the LLM head, and a single transformer layer before assigning stages.
  • System benefits: The V-shaped configuration absorbs volatile embedding and modality-loss overhead, substantially mitigating pipeline bubbles and achieving near-perfect load balance.It also isolates heterogeneous workloads and reduces cross-stage communication.

6 Advantages and Future Work

The section describes how discrete multimodal modeling supports unified training, reinforcement learning, and image generation, while identifying important limitations and future directions. It emphasizes sequence-level filtering for RL stability and broader validation beyond current understanding and generation tasks.

  • 6 Advantages: DiNA unifies generation and understanding under discrete training and supports infrastructure-friendly deployment.Discrete visual representations can also be optimized with strategies analogous to language modeling.
  • 6.1 Reinforcement Learning: Discrete visual latents serve as an action space compatible with language-model reinforcement learning methods such as GRPO.This avoids sampling-process conversion while retaining a finite Markov decision process structure.
  • 6.1 Reinforcement Learning: A multi-dimensional reward framework evaluates capability, OCR, semantic alignment, and image quality to mitigate single-reward hacking.The framework combines object and attribute evaluation, OCR edit distance, VLM-based semantic alignment, and complementary quality metrics.
  • 6.1 Reinforcement Learning: Sequence-level entropy and training-inference filters remove noisy rollouts from gradient updates, addressing mismatch-driven instability.The approach targets the root cause at sequence level rather than relying only on token-level approximations.
  • 6.2 Discussion and Future Work: The work remains limited by computational resources, data availability, tokenizer optimization, and evaluation focused mainly on understanding and generation.Future work targets higher pixel fidelity, any-to-any generation, interleaved reasoning, broader data scaling, and more comprehensive validation.

7 Conclusion

The conclusion presents LongCat-Next as an exploration of extending language-style discrete autoregressive modeling to vision and audio through a shared token interface. It reports that carefully designed tokenizers and training strategies can represent continuous perceptual signals within this framework.

  • 7 Conclusion: LongCat-Next explores unified representation and modeling of text, vision, and audio through language-style discrete autoregressive modeling.The conclusion frames the work as an investigation of native multimodality within a shared token interface.

8 Contributions and Acknowledgments

The acknowledgments define contributors as people responsible for core activities across LongCat-Next’s development lifecycle. Other contributors are recognized for specific supporting tasks.

  • 8 Contributions and Acknowledgments: Contributors held primary responsibilities in data curation, model design, model training, or infrastructure support throughout development.The acknowledgments separately recognize task-specific contributions such as collection, annotation, evaluation, and technical discussion.

9 Appendix

The appendix analyzes reinforcement learning, discrete quantization, multimodal token flow, and visual de-tokenization through experiments and qualitative examples. It reports gains from RL and residual quantization while examining reconstruction behavior and reasoning outputs.

  • 9.2 Experimental Analysis for RL: A few anomalous tokens can trigger sequence collapse when sampling and training policies diverge sharply.One token may have sampling probability above 0.4 but training-policy probability near 10^-2, while other-token differences are around 10^-3.
  • 9.2 Experimental Analysis for RL: Importance-sampling corrections delay entropy explosion only briefly, motivating direct token-level probability-difference filtering.The proposed threshold targets |πsampler − πactor| to detect fine-grained inconsistencies.
  • 9.2 Experimental Analysis for RL: RL improves STEM, general VQA, OCR, and image-generation performance, including 7.50% higher counting accuracy on GenEval.Position accuracy increases by 6.75% and color attribute accuracy by 4.56%.
  • 9.3 Discrete Quantization: Eight-stage residual vector quantization substantially lowers feature reconstruction loss compared with vanilla VQ.Two-stage RVQ already slightly outperforms vanilla VQ, supporting residual compositionality for information retention.
  • 9.4 Unified Tokenizer and Detokenizer: DiNA combines hierarchical visual tokens during understanding and uses a DepthTransformer during generation to balance performance and efficiency.The appendix describes multi-level encoding and decoding as the mechanism for using information across quantization levels.
  • 9.5 Mathematical Reasoning: Qualitative examples show LongCat-Next solving arithmetic-pattern, spatial-constraint, and geometric-area problems through multi-step reasoning.The examples include a cross-shaped arithmetic pattern, a 3x3 digit-placement puzzle, and nested geometric relationships.
  • 9.6 The Analysis of Visual De-tokenizer: The visual de-tokenizer reconstructs semantic content with a ViT pixel decoder, while the refiner improves perceptual sharpness and high-frequency detail.The current design prioritizes semantic consistency over pixel fidelity.
Loading 2603.27538v1…