Source-linked AI summary
AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation
Dongjie Cheng, Ruifeng Yuan, Yongqi Li, Runyang You, Wenjie Wang, Liqiang Nie, Lei Zhang, Wenjie Li
TL;DR
Existing omni MLLMs often depend on expert components for multimodal generation, whereas AR-Omni investigates a unified autoregressive alternative. It uses one Transformer decoder for text, image, and streaming speech generation, achieving competitive tri-modal quality while remaining real-time, including a 0.88 speech real-time factor.
Problem
Most omni MLLMs rely on additional expert components for multimodal generation, limiting unified autoregressive training and inference.
Method
AR-Omni tokenizes text, images, and speech into one discrete vocabulary and uses a unified autoregressive Transformer with task-aware reweighting, perceptual alignment loss, and finite-state decoding.
Results
AR-Omni achieves competitive tri-modal capabilities while remaining real-time, including a 0.88 real-time factor for speech generation.
Takeaways & Limitations
The model provides a diffusion-free, single-model pipeline for unified multimodal input and generation, while preserving competitive captioning quality and largely preserving autoregressive image generation.
Abstract
from arXiv · showhide
Real-world perception and interaction are inherently multimodal, encompassing not only language but also vision and speech, which motivates the development of "Omni" MLLMs that support both multimodal inputs and multimodal outputs. While a sequence of omni MLLMs has emerged, most existing systems still rely on additional expert components to achieve multimodal generation, limiting the simplicity of unified training and inference. Autoregressive (AR) modeling, with a single token stream, a single next-token objective, and a single decoder, is an elegant and scalable foundation in the text domain. Motivated by this, we present AR-Omni, a unified any-to-any model in the autoregressive paradigm without any expert decoders. AR-Omni supports autoregressive text and image generation, as well as streaming speech generation, all under a single Transformer decoder. We further address three practical issues in unified AR modeling: modality imbalance via task-aware loss reweighting, visual fidelity via a lightweight token-level perceptual alignment loss for image tokens, and stability-creativity trade-offs via a finite-state decoding mechanism. Empirically, AR-Omni achieves strong quality across three modalities while remaining real-time, achieving a 0.88 real-time factor for speech generation.
1 Introduction
AR-Omni targets unified any-to-any multimodal interaction with a single autoregressive model rather than external expert decoders. It supports text, image, and speech generation while addressing modality imbalance, visual fidelity, and decoding trade-offs.
- Motivation: Real-world interaction spans language, vision, and speech, motivating MLLMs that support both multimodal inputs and outputs.
- Results: AR-Omni is the only compared model with unified input/output and real-time streaming without external diffusion models.
- Motivation: Existing omni MLLMs commonly use expert diffusion or non-autoregressive components, motivating an omni model with one autoregressive decoder.
- AR-Omni: AR-Omni tokenizes text, images, and speech into a joint vocabulary and generates them with one 7B-parameter Transformer backbone.
- AR-Omni: AR-Omni addresses modality imbalance with task-aware reweighting, visual fidelity with token-level perceptual alignment, and decoding trade-offs with a finite-state machine.
- Results: 0.88 real-time factor for speech generation accompanies 146 ms first-token latency, 6.5 zero-shot TTS WER on VCTK, and 9.4 ASR WER on LibriSpeech test-clean.
2 Related Work
Prior multimodal language models extend text-based systems with modality adapters, discrete tokenization, or expert decoders. Related work develops autoregressive approaches for multimodal understanding and generation across vision and speech.
- Multimodal Large Language Models: Early MLLMs attach modality adapters to LLMs to enable multimodal perception and text responses.
- Multimodal Large Language Models: SpeechGPT discretizes speech for direct speech understanding and generation, while NExT-GPT combines an LLM with modality adapters and diffusion decoders.
- Multimodal Large Language Models: AnyGPT and MIO pursue tri-modal any-to-any interaction through autoregressive modeling paired with expert diffusion decoders.
- Multimodal Tokenization: Autoregressive multimodal modeling converts continuous modalities into discrete token sequences using vector-quantized tokenizers.
- Multimodal Tokenization: Image tokenizers compress images into discrete codebook indices, while SEED produces one-dimensional visual codes that condition a diffusion UNet.
- Multimodal Tokenization: Speech tokenizers quantize waveforms or factor speech into semantic and acoustic tokens; WavTokenizer explores single-codebook tokenization for streaming generation.
3 AR-Omni
AR-Omni unifies text, image, and speech processing by mapping modalities into one discrete token stream for a single Transformer to model autoregressively. Its training and inference use modality-aware mechanisms to balance tasks, improve visual coherence, and support task-specific decoding.
- Unified Autoregressive Modeling: AR-Omni maps text, images, and speech into a joint vocabulary and generates multimodal data through next-token prediction.The joint vocabulary is V = Vtext ∪ Vspeech ∪ Vimage, with special markers managing modality transitions in one interleaved sequence.
- Unified Autoregressive Modeling: A purely acoustic, low-rate speech tokenizer removes semantic-to-acoustic modeling and enables streamed speech responses with low latency.Speech tokens are integrated into the shared autoregressive interface rather than decoded by a separate modality-specific generator.
- Inference: Task-aware decoding uses greedy generation for deterministic ASR and TTS tasks and sampling for open-ended text-to-image generation.This assigns different decoding behaviors to tasks with different determinism and creativity requirements.
- Optimization: Weighted Next-Token Prediction assigns larger response-text weights for ASR and image captioning to reduce domination by modalities with longer token sequences.The weighting targets response tails in X2T tasks and amplifies supervision on the specific text outputs.
- Optimization: A perceptual loss aligns final hidden states with frozen image-code embeddings, encouraging visually coherent structures despite imperfect exact-token matching.It supplies geometric awareness and smoother similarity among visual codes than one-hot supervision, and is combined with the weighted NTP objective.
- Data: The pre-training mixture combines text-only, image-text, and speech-text corpora, with additional interleaved instruction data and speech augmentation.The reported pre-training sampling ratio is 0.5 : 1 : 2 for text-only, text-image, and text-speech data, respectively.
4 Experimental Setup
AR-Omni is trained in two stages and evaluated across image understanding, image generation, ASR, and zero-shot TTS, with metrics tailored to each task.
- Training: Training uses pretraining followed by fine-tuning on omni-interleaved instruction data.Stage 1 optimizes weighted NTP with perceptual loss; Stage 2 applies loss exclusively to response tokens.
- Image Evaluation: Image captioning is evaluated zero-shot on the MS-COCO 2014 Karpathy test split using CIDEr.CIDEr measures cosine similarity between TF-IDF-weighted n-gram vectors for generated and reference captions.
- Image Evaluation: Text-to-image generation is evaluated on 30k randomly sampled MS-COCO validation captions using CLIPscore.CLIPscore averages cosine similarity between generated-image and caption embeddings from CLIP ViT-L.
- Speech Evaluation: ASR is evaluated on LibriSpeech test-clean using Word Error Rate computed from Levenshtein alignment.WER is the sum of substitutions, deletions, and insertions divided by reference-word count.
- Speech Evaluation: Zero-shot TTS is evaluated on VCTK using WER, first-token latency, and real-time factor.FTL measures latency to the first speech token immediately decodable into audio, while RTF divides synthesis time by generated-audio duration.
5 Main Results
AR-Omni maintains competitive performance across image and speech tasks while using a single diffusion-free autoregressive generation pipeline and supporting favorable speech streaming behavior.
- Image Results: AR-Omni outperforms its Anole initialization on zero-shot image-to-text captioning in the diffusion-free autoregressive setting.The result indicates that extending the AR backbone to any-to-any training does not compromise captioning quality.
- Image Results: AR-Omni incurs only a slight text-to-image performance drop relative to Anole, while diffusion-based systems score higher.The comparison exposes a trade-off between single-model diffusion-free generation and higher image quality from diffusion decoders.
- Speech Results: AR-Omni achieves comparable ASR accuracy to any-to-any baselines while using significantly fewer speech tokens.The result supports single-codebook, low-rate speech tokenization as an interface for unified AR modeling.
- Speech Results: AR-Omni matches the best token-based baseline in TTS intelligibility and offers more favorable streaming latency and throughput.Prior any-to-any systems often show higher latency and slower generation under heavier or more complex pipelines.
6 Further Analysis
Further analyses examine loss dynamics, component ablations, training stability, and representative multimodal interactions, showing benefits from the proposed objectives and decoding design.
- Component Effectiveness: Table 7 evaluates component ablations on four omni-pretraining tasks at 40k steps using CIDEr, CLIPscore, and WER.I2T uses CIDEr, T2I uses CLIPscore, and ASR and TTS use WER; arrows indicate metric directionality.
- Training Objectives: The weighted NTP loss decreases smoothly, while perceptual loss converges quickly and saturates at a smaller magnitude during stage-1 pretraining.The total loss is therefore governed mainly by weighted NTP, with residual fluctuations attributed more consistently to mini-batch variance than instability.
- Component Effectiveness: Removing perceptual loss slightly reduces I2T and T2I, degrades TTS, and improves ASR.These results indicate that perceptual loss mainly benefits vision generation and speech synthesis in the unified framework.
- Component Effectiveness: Removing swin-norm improves I2T and ASR but substantially worsens TTS and slightly reduces T2I.The ablation identifies swin-norm as important for maintaining speech quality.
- Training Stability: Simple NTP shows late-stage loss rebounds and abrupt spikes, whereas AR-Omni maintains a smooth, convergent trajectory.The analysis characterizes Simple NTP as tending toward model collapse and AR-Omni as maintaining training stability.
- Multimodal Capabilities: Representative cases cover multi-turn speech conversation, speech-plus-image understanding with speech response, and speech-to-image generation.The examples show follow-up speech interaction, grounded spoken image description, and image generation from a spoken prompt.
7 Conclusion and Future Work
AR-Omni unifies text, image, and speech generation through one autoregressive backbone and introduces training and decoding strategies for practical multimodal operation. Its main limitation is lower diffusion-free image-generation quality than diffusion-based systems.
- Conclusion: AR-Omni tokenizes text, images, and speech into one discrete stream handled by a single Transformer backbone.The model supports autoregressive text and image generation as well as streaming speech generation without expert decoders.
- Conclusion: Task-aware reweighting, perceptual loss, and finite-state decoding address modality imbalance, visual fidelity, and decoding behavior.These mechanisms are presented as making unified autoregressive modeling practical.
- Conclusion: AR-Omni demonstrates competitive tri-modal capabilities while remaining real-time for streaming speech.The conclusion frames this as the model’s practical outcome within the unified AR setting.
- Future Work: Diffusion-free autoregressive image generation still lags behind diffusion-based systems in image-generation quality.Future work targets higher diffusion-free image quality while preserving the unified AR design.
A Training Details
AR-Omni uses a two-stage pre-training and fine-tuning pipeline optimized with Adam and a warmed linear learning-rate schedule. Global gradient clipping stabilizes unified multimodal training, with key settings summarized in Table 8.
- AR-Omni is trained on 8 NVIDIA A100 GPUs using pre-training followed by fine-tuning.
- Both training stages use Adam with a linear learning-rate schedule and warmup.
- Global gradient clipping is applied to stabilize unified multimodal training.
- Table 8 summarizes the key hyperparameters for pre-training and fine-tuning, including global batch size.
B Dataset Details
AR-Omni’s pre-training data covers image–text, speech–text, and text-only corpora. The datasets report modality pairings, scale statistics, sources, and preprocessing details, with filtering and deduplication applied to reduce noise and unsafe content.
- The pre-training corpora span image–text, speech–text, and text-only data.
- Each dataset is described by its modality pairing, scale statistics, source, and preprocessing notes.
- Publicly available datasets undergo standard filtering and deduplication to reduce noise and unsafe content.
C Prompt Templates
AR-Omni represents any-to-any multimodal tasks with a unified dialogue-style prompt format. Explicit boundary and termination tokens preserve modality distinctions across single-turn and multi-turn interactions while reducing tasks to next-token prediction.
- Each user turn begins with <bos> and ends with <eoh> in the unified dialogue-style format.
- Audio and image content is explicitly delimited with <boa>/<eoa> and <boi>/<eoi> boundary tokens.
- Single-turn assistant responses terminate with <eos>, whereas multi-turn responses terminate with <eom>.
- Multi-turn dialogue history is formed by concatenating previous turns in the same format.
- The unified formatting casts diverse tasks into one next-token prediction interface while keeping input and output modality boundaries explicit.
D Case Study
The case study demonstrates AR-Omni’s any-to-any behavior through multi-turn multimodal conversations and text-to-image generation. Images are produced without external diffusion decoders by detokenizing autoregressively generated discrete image tokens.
- Multi-turn multimodal interaction: Figures 5 and 6 show multi-turn interactions with interleaved multimodal inputs and outputs.
- Text-to-image generation: Figures 7–10 present text-to-image samples across diverse prompts and styles.
- Text-to-image generation: The generated images use discrete image tokens decoded by a detokenizer, without external diffusion decoders.
- Multi-turn multimodal interaction: Speech-input conversations include image generation followed by speech descriptions of the generated scene.
- Multi-turn multimodal interaction: Another dialogue combines speech advice with a request for an image of a small waterfall in a forest.
- Text-to-image generation: The qualitative examples include roads, interiors, campsites, landscapes, objects, and other varied scenes.