Source-linked AI summary
Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device
Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad, Ritesh Thawkar, Omkar Thawakar, Senmao Li, Hisham Cholakkal, Ian Reid, Eric P. Xing, Salman Khan, Fahad Shahbaz Khan
TL;DR
Existing unified multimodal models are too computationally heavy and data-hungry for practical edge deployment. Mobile-O addresses this with a compact vision–language–diffusion architecture, efficient conditioning, and joint quadruplet-based post-training, achieving strong understanding and generation performance with mobile-device efficiency.
Problem
Existing unified multimodal models require heavy architectures and typically 50M–1B pre-training samples, limiting their suitability for real-time consumer-device deployment.
Method
Mobile-O combines a Mobile Conditioning Projector with a unified post-training scheme using quadruplets of generation prompt, image, question, and answer.
Results
Mobile-O achieves competitive or superior understanding and generation performance while improving latency and memory efficiency, including 74% on GenEval and practical iPhone execution.
Takeaways & Limitations
Mobile-O provides a framework for unified multimodal understanding and image generation directly on mobile and other resource-constrained edge devices.
Takeaways & Limitations
Mobile-O reuses the unified VLM’s lightweight LLM as its text encoder, reducing memory use but potentially limiting textual expressiveness and depth.
Abstract
from arXiv · showhide
Unified multimodal models can both understand and generate visual content within a single architecture. Existing models, however, remain data-hungry and too heavy for deployment on edge devices. We present Mobile-O, a compact vision-language-diffusion model that brings unified multimodal intelligence to a mobile device. Its core module, the Mobile Conditioning Projector (MCP), fuses vision-language features with a diffusion generator using depthwise-separable convolutions and layerwise alignment. This design enables efficient cross-modal conditioning with minimal computational cost. Trained on only a few million samples and post-trained in a novel quadruplet format (generation prompt, image, question, answer), Mobile-O jointly enhances both visual understanding and generation capabilities. Despite its efficiency, Mobile-O attains competitive or superior performance compared to other unified models, achieving 74% on GenEval and outperforming Show-O and JanusFlow by 5% and 11%, while running 6x and 11x faster, respectively. For visual understanding, Mobile-O surpasses them by 15.3% and 5.1% averaged across seven benchmarks. Running in only ~3s per 512x512 image on an iPhone, Mobile-O establishes the first practical framework for real-time unified multimodal understanding and generation on edge devices. We hope Mobile-O will ease future research in real-time unified multimodal intelligence running entirely on-device with no cloud dependency. Our code, models, datasets, and mobile application are publicly available at https://amshaker.github.io/Mobile-O/
1. Introduction
Mobile-O targets the deployment gap in unified multimodal models by combining understanding and generation in a compact, mobile-optimized framework. It uses efficient conditioning and joint post-training to reduce computational and data requirements while preserving strong performance.
- Existing unified models combine visual understanding and generation, supporting captioning, visual question answering, and visual content generation within one framework.
- Consumer-device deployment is constrained by heavy visual encoders and denoising modules, with BLIP-3o totaling 7.1B parameters.BLIP-3o uses a 2.6B-parameter UNet, a 3B VLM, and a 1.5B diffusion transformer.
- 50M–1B pre-training samples are typically required for effective cross-modal alignment, making unified-model pre-training expensive and time-consuming.
- Mobile-O introduces a mobile-optimized architecture centered on the Mobile Conditioning Projector, which fuses VLM features with diffusion conditioning using lightweight operations.The MCP uses depthwise-separable convolutions and layerwise alignment.
- With 1.6B parameters, Mobile-O reaches 74% on GenEval, outperforming Show-O and JanusFlow by 5% and 11%, respectively, while achieving faster inference.
- Unified post-training uses 105k quadruplets—generation prompt, image, question, answer—to jointly optimize multimodal understanding and image generation.
3. Method
Mobile-O combines mobile-optimized understanding and generation components through the MCP projector and a unified multi-stage training scheme. Its method reduces conditioning overhead while jointly optimizing image-to-text and text-to-image capabilities.
- Baseline Mobile Unified Framework: Mobile-O uses FastVLM for multimodal understanding and a DiT-style diffusion decoder for image generation within one mobile unified architecture.The same LLM handles generation prompts, avoiding a separate text encoder.
- Mobile Conditioning Projector: The MCP directly maps VLM hidden states to diffusion-compatible conditioning without intermediate query tokens.It fuses the final or last K VLM layers, then compresses and refines the sequence before output projection.
- Mobile Conditioning Projector: MCP uses learned layerwise fusion, depthwise-separable sequence convolutions, channel attention, and output projection to provide diffusion cross-attention features efficiently.Operating along sequence length avoids expensive 2D convolutions while retaining token-level alignment.
- Training Scheme: Mobile-O trains in three stages: cross-modal alignment, supervised fine-tuning, and unified multimodal post-training.The first two stages freeze the visual encoders and LLM backbone while training the DiT and MCP; alignment uses 4 million JourneyDB pairs and 5 million BLIP3o-Short-Caption pairs.
- Training Scheme: Unified post-training jointly optimizes understanding and generation with a quadruplet of generation prompt, image, question, and answer.The objective combines language-model cross-entropy for I2T with a flow-matching diffusion objective for T2I.
- Evaluation: Mobile-O-0.5B scores 61.9 averaged over seven understanding datasets and exceeds FastVLM by 1.6% in average performance.The comparison covers unified models with total size at or below 2B parameters.
4. Experiments
Experiments show that Mobile-O combines competitive multimodal understanding and image generation with efficient mobile-oriented architecture, scaling, post-training, and deployment performance.
- Implementation Details: Mobile-O-0.5B uses FastVLM-0.5B and SANA-600M-512 connected by a lightweight Mobile Conditioning Projector with depthwise-separable convolutions.The shared projector connects the understanding and generation branches for efficient cross-modal alignment.
- Quantitative Comparison: 0.74 overall GenEval score lets Mobile-O-0.5B outperform Show-o-Clip-ViT by 5.0% among similarly sized unified models.The comparison uses models with total parameter counts of 2B or fewer.
- Quantitative Comparison: 4.9% absolute average gain over JanusFlow is achieved across seven understanding benchmarks with fewer parameters: 1.6B versus 2.1B.Mobile-O-0.5B also gains 1.6% over FastVLM in the same seven-benchmark average.
- Text-and-Image-to-Image Generation: 46k ShareGPT4V editing samples suffice for fine-tuning image editing without architectural modifications.The same MCP, language model, and generation backbone are reused, conditioning generation on both source-image and textual representations.
- Qualitative Comparison: Mobile-O produces sharper details, more coherent layouts, and more consistent illumination than Janus, JanusFlow, and Show-O in qualitative comparisons.It also correctly handles examples from ChartQA and TextVQA and performs object addition, attribute modification, and style transfer.
- Generality and Ablation: Post-training improves both capabilities in the scaled Mobile-O-1.5B model, increasing understanding from 64.8% to 66.2% and generation from 75% to 78%.The results support transfer of the MCP, unified data format, and recipe to larger backbones.
- Generality and Ablation: Joint generation-and-understanding post-training raises seven-task understanding accuracy from 60.5% to 62.1% and GenEval by 1%.The reported comparison contrasts standard supervised fine-tuning with joint post-training variants.
5. Conclusion
The conclusion presents Mobile-O as a unified vision–language–diffusion model designed for efficient, high-quality understanding and generation directly on edge devices.
- Conclusion: Mobile-O combines quadruplet unified post-training with a Mobile Conditioning Projector for image understanding and text-to-image generation on edge devices.The framework targets both latency and memory efficiency while preserving visual fidelity and semantic accuracy.
- Conclusion: Below-2GB memory usage and approximately 3-second iPhone inference make Mobile-O-0.5B practical for real-time on-device deployment.
Supplementary Material
The supplementary ablation varies how many VLM layers the MCP aggregates when conditioning the diffusion process.
- Supplementary Material: MCP layer-count ablation systematically varies the VLM layers aggregated to condition diffusion, while retaining learnable fusion, compression, and channel attention.
7. Mobile Conditioning Projector Depth
MCP layer aggregation affects GenEval quality by controlling the semantic information supplied to the diffusion model.
- Mobile Conditioning Projector Depth: 68.7% GenEval accuracy from a single VLM layer suggests insufficient semantic diversity for complex compositional prompts.Aggregating additional layers with learnable fusion improves performance.
8. On-Device Mobile Deployment
Mobile-O runs natively on an iPhone, supporting both text-to-image and image-to-text generation in a chat-based interface. The deployment demonstrates practical, offline unified multimodal operation with low memory overhead.
- 3 seconds: Mobile-O generates a detailed Bengal tiger image from a complex compositional prompt on an iPhone 17 Pro.The demonstration runs natively on consumer hardware.
- Mobile-O also generates rich visual descriptions that analyze scene composition, subject positioning, depth perception, and atmosphere.
- The iPhone deployment supports both text-to-image and image-to-text generation within a chat-based interface.
- Mobile-O operates without cloud dependency and uses less than 2GB of memory, supporting offline functionality and user privacy.
9. More Implementation Details
Mobile-O uses a three-stage training setup that progresses from cross-modal alignment to targeted fine-tuning and unified multimodal post-training. The setup uses large-scale initial alignment data, a smaller quadruplet dataset, and distributed training infrastructure.
- Infrastructure and optimization: Training uses 8 NVIDIA A100 GPUs, with ZeRO-3 for 9M Stage 1 samples and ZeRO-1 for later smaller datasets.
- Infrastructure and optimization: Reduced-rank LoRA uses r=16, α=32, and dropout 0.1 to regularize unified training on the 105K quadruplet dataset.
- Training stages: Stage 1 establishes cross-modal alignment using large-scale image-text pairs.
- Training stages: Stage 2 performs targeted fine-tuning for weaknesses in complex gestures, common objects, and landmarks.
- Training stages: Stage 3 jointly trains understanding and generation with quadruplet samples {p, ximg, q, a}.
- Infrastructure and optimization: Cosine-annealed learning rates and warmup schedules differ across the three stages, including a reduced Stage 3 initial rate of 1e-4.
10. More Image-to-Text Qualitative Results
Mobile-O shows robust image-to-text understanding across dense OCR, scientific reasoning, perspective-challenged text, domain-specific recognition, and cultural artifact identification. Qualitative comparisons emphasize accurate extraction and reduced hallucination in difficult visual inputs.
- Dense text understanding: Mobile-O correctly extracts a book’s title, author, editors, story titles, and 15.00 USD price from dense back-cover text.
- Dense text understanding: Competing models hallucinate information or misidentify bibliographic details in the same dense-text extraction task.
- Broad visual understanding: The qualitative evaluation spans scientific reasoning, OCR under challenging perspectives and lighting, fine-grained object recognition, stylized text extraction, and cultural classification.
- Generation comparison: Mobile-O demonstrates competitive or superior text-to-image quality relative to SANA-0.6B across photorealistic, complex-lighting, and fine-detail prompts.
- Broad visual understanding: These tasks draw on MMMU, ChartQA, and TextVQA and assess visual question answering across diverse cognitive domains.
11. Comparison with Generation-Only Baseline
Against a generation-only baseline and other unified models, Mobile-O maintains competitive or superior image-generation quality across varied subjects and challenging prompts. It does so with fewer parameters while also supporting visual understanding.
- Comparison with SANA-0.6B: 38% reduction: Mobile-O uses 1.6B total parameters versus SANA-0.6B’s 2.6B while achieving competitive or superior generation quality.
- Comparison with SANA-0.6B: Mobile-O renders sharper feather details, more natural backgrounds, realistic geological textures, and natural color grading in comparative scenes.
- Comparison with SANA-0.6B: Mobile-O produces more natural skin tones, facial features, lighting, and depth of field in the portrait comparison.
- Unified capability: Mobile-O supports visual understanding tasks simultaneously with the generation capabilities shown in these comparisons.
- Comparison with unified models: Mobile-O remains competitive across fantastical and photorealistic prompts compared with Janus, JanusFlow, and Show-O.
- Generation diversity: Its generation demonstrations cover portraits, macro nature photography, food imagery, and creative scenes with complex lighting effects.
13. Limitations
Mobile-O reduces memory demands by reusing a lightweight language model, but this limits textual representation depth relative to larger dedicated text encoders. Larger standalone encoders are impractical on most mobile and resource-limited edge devices because their memory requirements exceed deployment constraints.
- Mobile-O reuses its lightweight unified-VLM LLM as the text encoder, reducing memory footprint but potentially limiting textual expressiveness and representation depth.Larger text-only models can provide a more powerful linguistic backbone and better alignment.
- Most mobile and resource-limited edge devices cannot accommodate the several additional GBs of total memory required by such larger encoders.This conflicts with the efficiency and low-latency objectives of on-device deployment.