Source-linked AI summary
Context Unrolling in Omni Models
Ceyuan Yang, Zhijie Lin, Yang Zhao, Fei Xiao, Hao He, Qi Zhao, Chaorui Deng, Kunchang Li, Zihan Ding, Yuwei Guo, Fuyun Wang, Fangqi Zhu, Xiaonan Nie, Shenhan Zhu, Shanchuan Lin, Hongsheng Li, Weilin Huang, Guang Shi, Haoqi Fan
TL;DR
The paper addresses how a unified multimodal model can use heterogeneous modalities as complementary evidence for reasoning rather than merely aggregating capabilities. Omni trains across text, images, videos, 3D geometry, and hidden representations to enable Context Unrolling, and reports strong performance across multimodal understanding, generation, editing, and 3D tasks. Its video output remains limited to 480 × 640 resolution and 12 seconds, below state-of-the-art video-generation specialists.
Problem
Different modalities provide partial and biased views of shared world knowledge, motivating a model that can combine their complementary evidence for more complete multimodal reasoning.
Method
Omni natively trains across text, images, videos, 3D geometry, hidden representations, and reasoning-oriented content so it can unroll and compose modality-specific contexts before prediction.
Results
Omni achieves competitive or superior performance across multimodal understanding, image and video generation and editing, camera estimation, and depth estimation benchmarks.
Takeaways & Limitations
Unified multimodal modeling functions as context-conditioned inference, allowing task-specific primitives such as text reasoning, visual tokens, camera prediction, and novel-view synthesis to support downstream decisions.
Takeaways & Limitations
Omni currently generates videos only at 480 × 640 resolution for 12 seconds, behind state-of-the-art video-generation specialists.
Abstract
from arXiv · showhide
We present Omni, a unified multimodal model natively trained on diverse modalities, including text, images, videos, 3D geometry, and hidden representations. We find that such training enables Context Unrolling, where the model explicitly reasons across multiple modal representations before producing predictions. This process enables the model to aggregate complementary information across heterogeneous modalities, facilitating a more faithful approximation of the shared multimodal knowledge manifold and improving downstream reasoning fidelity. As a result, Omni achieves strong performance on both multimodal generation and understanding benchmarks, while demonstrating advanced multimodal reasoning capabilities, including in-context generation of text, image, video, and 3D geometry.
1 Introduction
Omni is a unified multimodal model that uses Context Unrolling to reason across complementary text, image, video, 3D, and hidden representations before prediction. This unified design supports multimodal understanding and generation, with competitive or superior performance across diverse benchmarks.
- Motivation: Omni treats each modality as a complementary projection of shared world knowledge and integrates them to form more faithful, coherent, and semantically accurate predictions.Context Unrolling selects and composes heterogeneous modal contexts before producing outputs.
- Method: Omni expands native multimodal training from image–text pairs to text, images, videos, 3D geometry, hidden visual representations, and reasoning-oriented multimodal content.The design includes structured descriptions, spatial and geometric constraints, and a dedicated hidden reasoning space.
- Results: Omni achieves competitive or superior performance across multimodal understanding, image generation, image editing, video generation and editing, camera estimation, and depth estimation.Reported comparisons include Qwen3-VL and InternVL3.5 for understanding, several image generators and editors, and video and 3D specialists.
- Model: Omni provides a unified framework for text, image, video, and 3D geometry within one model architecture.It is built upon BAGEL and supports context unrolling across modalities.
- Results: Context Unrolling emerges during native multimodal pretraining and is supported by improvements across visual understanding, visual generation, depth estimation, and spatial reasoning.The model can reason across modality-specific contexts before producing predictions.
2 Context Unrolling
Context Unrolling treats multimodal capabilities as composable primitives that construct task-relevant context before prediction. Across understanding, generation, spatial reasoning, and depth estimation, richer and complementary contexts improve inference, while current self-rollouts remain limited by noise and hallucinated details.
- Visual Generation: Textual and visual contexts complement each other in image generation, with combined text-think and visual-token rollout producing the most consistent improvements.Richer textual context also improves aggregate performance and robustness under increasing prompt compositionality.
- Visual Generation: Oracle textual contexts substantially improve generation quality, while additional self-rollout visual tokens provide further gains, indicating non-redundant textual and visual constraints.The oracle study exposes a gap between current self-rollout contexts and higher-quality context, while preserving the value of visual structure.
- Spatial Understanding: Geometry-grounded textual context improves spatial understanding, and synthesized visual contexts provide complementary evidence for resolving geometric ambiguity.These context-producing primitives turn pose summaries and novel views into actionable inputs for spatial reasoning.
- 3D Geometry: Depth estimation benefits from geometry-focused depth captions and visual tokens, which add spatial cues, stabilize generation, and produce sharper, more globally consistent predictions.Generic detailed captions provide little to no benefit, whereas depth captions focus on relative spatial structure and visual tokens encode layout and coarse geometry.
- Context Unrolling: Unified models construct a growing shared context by invoking atomic primitives such as textual reasoning, pose prediction, visual-token rollout, novel-view synthesis, and depth estimation.Inference is modeled as iterative context construction followed by context-conditioned decoding.
- Discussion: The paper concludes that unified context unrolling, rather than capability aggregation alone, is the primary value of a unified model, while post-training policies for context construction remain future work.The experiments focus on supervised and standard evaluations, leaving adaptive decisions about when and how to construct context unresolved.
3 Evaluation
Omni is evaluated as a unified model across multimodal understanding, image and video generation/editing, and 3D vision tasks. Results are compared with specialist and unified models across these settings, with strong performance reported alongside a video-generation resolution and duration limitation.
- Evaluation scope: Omni is evaluated on multimodal understanding, image generation and editing, video generation and editing, and 3D vision tasks.The 3D evaluations cover camera pose estimation and monocular depth estimation.
- Multimodal understanding: Omni achieves comparable performance across general VQA, chart and graph understanding, alignment, video understanding, and spatial understanding.The comparison includes Qwen3-VL-30B-A3B-Instruct and InternVL3.5-30B-A3B at a similar scale.
- Image generation and editing: Omni achieves comparable performances with only 3B activations on text-to-image and image-editing benchmarks despite prior approaches using separate expertise models.The reported benchmarks include GenEval2, DPG, LongText-EN, Inhouse evaluation, and GEdit.
- Video generation and editing: Omni achieves comparable performances on VBench text-to-video generation and shows significant superiority over other approaches in instruction following for video editing.Video editing comparisons are conducted at a similar resolution and duration on FiVE.
- 3D geometry: On RealEstate10K, Omni surpasses all baselines across three camera-pose metrics, while on CO3Dv2 it achieves the best translation error but weaker results on other metrics.The authors attribute the CO3Dv2 discrepancy to insufficient coverage of object-centric scenes in data collection.
- 3D geometry: Omni demonstrates zero-shot monocular-depth performance across multiple benchmarks and can achieve or exceed specialized single-task models without task-specific fine-tuning.The evaluation covers five standard datasets.