Source-linked AI summary
SmolVLM: Redefining small and efficient multimodal models
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, Thomas Wolf
TL;DR
Large VLMs offer strong capabilities but demand substantial computation and memory, while smaller models often inherit inefficient design choices. SmolVLM systematically explores compact architectures, tokenization, and training choices; its smallest model uses less than 1GB GPU memory and surpasses much larger 80B-parameter models. The resulting family supports efficient multimodal inference across image and video tasks.
Problem
Large VLMs require substantial computational resources, while smaller VLMs often retain inefficient architectural choices and high memory demands.
Method
SmolVLM systematically explores architectural configurations, tokenization methods, and training data choices for compact multimodal inference.
Results
Less than 1GB GPU memory is required by SmolVLM-256M, which surpasses much larger 80B-parameter models on nearly all evaluated benchmarks.
Takeaways & Limitations
Compact architectural innovations, efficient tokenization, and training strategies support powerful multimodal capabilities at a fraction of the computational cost.
Takeaways & Limitations
Excessive chain-of-thought data harms compact model performance, and image resolution is constrained by model size.
Abstract
from arXiv · showhide
Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.
1 Introduction
SmolVLM targets the high computational and memory demands of modern VLMs with compact models designed for efficient deployment. The paper combines systematic architectural exploration with practical on-device capabilities, including video understanding and open resources.
- Smaller VLMs can retain significant computational overhead when they inherit architectural decisions from much larger models.
- Efficient processing is especially important for video understanding because memory management becomes essential for longer multimodal inputs.
- SmolVLM introduces compact multimodal models that reduce resource requirements through careful architectural design without sacrificing capability.
- Less than 1GB GPU RAM is required for inference by the smallest SmolVLM model, lowering the barrier to on-device deployment.
- SmolVLM models generalize to video tasks and are accompanied by released weights, datasets, code, and a smartphone inference application.
2 Smoller Model Architecture
SmolVLM’s compact architecture allocates capacity across vision and language components while extending context and compressing visual tokens. Image splitting and targeted token strategies further support efficient processing of high-resolution images and videos.
- 2.1 How to assign compute between vision and language towers?: Balanced encoder-LM allocation makes smaller vision encoders preferable for compact multimodal models.With a 135M language model, a larger encoder reduces performance; at 360M, it improves performance 11.6% while increasing parameters 66%.
- 2.2 How can we efficiently pass the images to the Language Model?: 16k-token contexts improve performance for the 2.2B model, while smaller 135M and 360M models are limited to 8k tokens.
- 2.2 How can we efficiently pass the images to the Language Model?: Pixel shuffle reduces visual tokens by r^2 while increasing representational density, and smaller VLMs benefit from the more aggressive r = 4 setting.Higher ratios can impair precise localization tasks such as OCR.
- 2.3 How can we efficiently encode images and videos?: Image splitting maintains high-resolution image quality without excessive computational overhead, whereas video frame averaging degrades performance.
- 2.3 How can we efficiently encode images and videos?: Learned positional tokens improve convergence, OCR accuracy, and cross-task generalization compared with naive string positions in smaller models.String-based positions produce the “OCR loss plague,” with training loss drops that do not improve OCR performance.
3 Smol Instruction Tuning
Smol instruction tuning combines learned positional tokens, structured prompts, selective prompt masking, and carefully balanced training data to improve compact VLM performance. Experiments show that sparse CoT data and moderate video durations are preferable to heavier reasoning data or longer sequences.
- 3 Smol Instruction Tuning: Overall, compact VLM instruction tuning requires coordinated vision and text tokenization, structured multimodal prompting, and balanced data composition.The section combines learned positional encoding, media segmentation, prompt masking, text-mixture control, sparse CoT inclusion, and moderate video lengths.
- 3.1 Learned Tokens vs. String: Learned positional tokens improve convergence, OCR accuracy, and generalization compared with raw string positions in compact VLMs.Raw string positions caused early training plateaus, while learned tokens consistently performed better across image and video evaluations.
- 3.2 Structured Text Prompts and Media Segmentation: Sequentially adding tokenization and prompt-engineering strategies produces consistent image gains and more pronounced video gains.Figure 6 presents the cumulative progression across successive strategy configurations.
- 3.2 Structured Text Prompts and Media Segmentation: System prompts clarify task objectives, while media intro/outro tokens substantially improve video performance and also benefit image tasks.The media markers distinguish visual segments from surrounding instructions, reducing confusion among multiple video frames.
- 3.2 Structured Text Prompts and Media Segmentation: Masking user prompts during supervised fine-tuning improves both image and video performance, especially on repetitive multimodal question-answering tasks.The approach trains only on completions rather than user queries, encouraging reliance on task-relevant content.
- 3.3 Impact of Text Data Reuse from LLM-SFT: Reusing LLM-SFT text data degrades smaller-model performance by up to 3.7% on video tasks and 6.5% on image tasks.The training mix therefore retains a strict 14% text proportion rather than directly adopting the reused data.
- 3.4 Optimizing Chain-of-Thought Integration for Compact Models: A minimal 0.02%–0.05% fraction of CoT data slightly improves performance, whereas heavier CoT usage markedly degrades results.The degradation is especially pronounced on image tasks, consistent with the limited capacity of compact multimodal models.
- 3.5 Impact of Video Sequence Length on Model Performance: Training with video durations approaching approximately 3.5 minutes improves both video and image benchmarks, while longer durations yield minimal additional gains.Beyond 3.5 minutes, performance gains diminish relative to the added computational expense.
4 Experimental Results
SmolVLM is evaluated across vision and video benchmarks in three size variants, emphasizing performance relative to RAM usage. Results show strong accuracy, scaling gains, low memory requirements, and competitive video comprehension.
- Model Variants: Three SmolVLM variants target different computational environments: 256M, 500M, and 2.2B parameters.The 256M model combines SigLIP-B/16 with SmolLM2-135M, while the 500M and 2.2B models use larger language or vision backbones.
- Evaluation Setup: RAM usage is treated as a better computational-cost proxy than parameter count because VLM architecture strongly affects runtime expense.The evaluation compares performance against the RAM required to run models.
- Memory Efficiency: 0.8GB of VRAM is required for single-image inference with SmolVLM-256M, compared with 27.7GB for MolmoE-A1B-7B.SmolVLM-500M requires 1.2GB and SmolVLM-2.2B requires 4.9GB in the same comparison.
- Vision Results: 59.8% overall score is achieved by SmolVLM-2.2B, followed by 51.0% for 500M and 44.0% for 256M.The smallest model surpasses Idefics 80B on nearly all benchmarks, with exceptions including MMMU and AI2D.
- Video Results: 52.1 on Video-MME and 36.2 on WorldSense are achieved by SmolVLM-2.2B, while SmolVLM-500M scores 49.0 on TempCompass and 30.6 on WorldSense.The results indicate competitive temporal reasoning and real-world video understanding at compact scales.
- Throughput: 16.3 examples per second is reached by SmolVLM-256M at batch size 64 on an A100, while the 2.2B variant reaches 1.7 examples per second.On an L4, the 256M variant peaks at 2.7 examples per second with batch size 8, reflecting hardware-dependent memory constraints.
5 Related Work
Related work has pursued multimodal capability through larger models, modular interfaces, token compression, and video-specific training. These approaches improve efficiency or capability but retain trade-offs in scale, detail preservation, and computational demand.
- Large-Scale VLMs: Flamingo and Idefics demonstrated large-scale multimodal training with models reaching 80B parameters, creating deployment challenges despite strong capabilities.Flamingo used gated cross-attention and a Perceiver Resampler, while Idefics adopted its architecture at 9B and 80B scales.
- Parameter-Efficient Interfaces: BLIP-2 reduced trainable multimodal interface size by freezing the vision encoder and language model and adding a lightweight Q-Former.The Q-Former translates visual features into language-compatible tokens.
- Instruction-Tuned VLMs: LLaVA connected a pretrained CLIP image encoder to a LLaMA/Vicuna backbone and fine-tuned the combined model on instruction-following data.The resulting multimodal chatbot used 13B parameters.
- Visual Tokenization: Perceiver Resamplers and Q-Former compress visual inputs into fewer latent tokens, but compression can limit fine-grained tasks such as OCR.Earlier patch-wise encoding produced 196 tokens for a 224×224 image at 16×16 resolution.
- Video Understanding: Video models face increased token counts and computational demands from temporal dimensions, motivating frame alignment, self-training, temporal grounding, and dynamic token compression.Apollo reported efficiency gains and state-of-the-art results on LongVideoBench, MLVU, and Video-MME with modest parameter sizes.
6 Conclusion
SmolVLM provides memory-efficient multimodal models from 256M to 2.2B parameters. Its compact architecture, tokenization methods, and training strategies support strong image and video capabilities at substantially lower computational cost.
- Conclusion: SmolVLM spans 256M to 2.2B parameters and its smallest variant uses less than 1GB of GPU memory while surpassing earlier 80B-parameter models.The comparison concerns state-of-the-art 80B-parameter models from 18 months earlier.
- Conclusion: Specialized architectures, careful tokenization, and efficient training strategies provide powerful multimodal capabilities at a fraction of the computational cost.The conclusion contrasts these choices with scaling down architectures designed for resource-rich settings.
- Conclusion: Publicly released weights, datasets, and training code support reproducibility, transparency, and continued innovation.The authors connect these releases to lightweight multimodal inference research.