MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices

• The work aims to provide open mobile-scale VLMs trained on public datasets for cross-modal perception and reasoning in resource-constrained settings.Source: §1 · pp. 1–2
• The work presents a full-stack, reproducible mobile-oriented VLM built with controlled, open-source datasets and foundation models trained from scratch.The authors describe it as, to their knowledge, the first detailed, reproducible, and strong vision-language model from scratch.Source: §1 · p. 2
• The authors conduct ablations on visual encoders and evaluate sensitivity to training paradigms, input resolutions, and model sizes.Source: §1 · p. 2
• An efficient projector aligns visual and text features while reducing the inference budget.Source: §1 · p. 2
• MobileVLM performs comparably across many VLM benchmarks, outperforms many cloud-only VLMs, and reaches 21.5 tokens/s on a Qualcomm mobile CPU and 65.3 tokens/s on a Jeston Orin GPU.Source: §1 · p. 2

2.1. Vision Transformer

• Vision Transformer variants strengthen representation or efficiency, while pre-training shifted from labeled classification and masked auto-encoding toward CLIP-style language supervision.VLMo extends CLIP with unified multimodality training.Source: §2.1 · p. 2

2.2. LLMs

• Large language models use extensive text corpora and have emergent capabilities, while a trend builds smaller models around 1B parameters or fewer.The authors seek reproducible, efficient models and therefore do not use non-public data.Source: §2.2 · p. 2

2.3. VLMs

• VLMs vary in language and vision backbones, cross-modality designs, and training corpora, with many using CLIP-style ViTs and millions of text-image pairs.Recent datasets include GPT-4V-generated image-text pairs, while Table 1 compares open-source VLM architectures and training corpora.Source: §2.3 · p. 2, Table 1 · p. 3

2.4. Model Compression for LLMs

• Model compression methods, including pruning, quantization, distillation, and low-rank decomposition, target lower model size and computational demands; deployment tools address varied environments.Source: §2.4 · p. 3

2.5. VLM Benchmarks

• VLM benchmarks test distinct abilities, including hallucination, real-world reasoning, OCR, scientific reasoning, perception and cognition, and fine-grained skills.The cited benchmarks include POPE, GQA, TextVQA, ScienceQA, MME, and MMBench.Source: §2.5 · p. 3

2.6. Embodied AI

• Large vision-language models support embodied-AI tasks such as embodied question answering and vision-language navigation in an end-to-end fashion.Source: §2.6 · p. 3

3.1. Overall Architectural Design

• The architecture maps image patches into compressed word-space visual tokens, pairs them with text tokens, and generates responses autoregressively from multimodal input.The projector reduces the image-token count to one quarter of the original.Source: §3.1 · p. 4

3.2. Visual Encoder

• CLIP ViT-L/14 at 336×336 resolution serves as the visual encoder, extracting image features from uniform-sized patches.The encoder is selected based on empirical analysis reported later in the paper.Source: §3.2 · p. 4

3.3. MobileLLaMA

• MobileLLaMA downsizes LLaMA for deployment across popular inference frameworks, with latency measurements guiding design and neural architecture search left for future work.The models use a 32,000-token LLaMA2 SentencePiece vocabulary and a 2k pretraining context.Source: §3.3 · p. 4

3.4. Efficient Projector

• LDP uses depth-wise convolution and stride-2 downsampling to retain spatial information while reducing visual tokens by 75%; downsampling alone severely harms downstream performance, including OCR.The projector uses a stronger network than a single PEG to mitigate that degradation.Source: §3.4 · p. 4

4.1. Training

• Training proceeds from RedPajama language-model pretraining to dialogue SFT and multimodal training; VLM pretraining freezes the vision encoder and LLM while training the projector.VLM instruction tuning then fine-tunes the projector and LLM.Source: §4.1 · pp. 5–6

4.2. Evaluation of MobileLLaMA

• MobileLLaMA is on par with mobile-scale peers and outperforms other models on nearly all listed language-understanding benchmarks.Both MobileLLaMA sizes achieve the highest Winogrande performance at their respective 1B and 3B levels.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Comparison with SOTA VLMs

• MobileVLM matches or exceeds most 7B-or-larger VLMs on GQA, POPE, and MMBench, but shows performance gaps on ScienceQA and MME.The evaluation covers six VLM benchmarks.Source: Table 4 · p. 7, §4.3 · pp. 7–8

4.4. MobileVLM with LoRA

• LoRA achieves performance comparable to full fine-tuning on six benchmarks while updating 8.87% and 7.41% of the full LLM parameters for the two model sizes.The updated fractions correspond to MobileLLaMA 1.4B and 2.7B, respectively.Source: §4.4 · p. 8

4.5. Latency Measurement on Mobile Devices

• For 336 × 336 images, MobileVLM with MobileLLaMa 2.7B and 1.4B consistently achieves lower total inference time than LLaVA-v1.5 on Snapdragon SoC and Jetson Orin, using 144 versus 576 visual tokens.In the latency comparison, LLaVA-v1.5-336-Vicuna generates 51 output tokens, while the other VLMs generate 256.Source: §4.5 · p. 9, Table 6 · p. 9

5.1. Ablation on Vision Backbones

• Scaling model size gradually improves performance across six benchmarks, while gains may saturate with limited training data; supervised image-text alignment performs best among the tested pre-training paradigms.Grounding-detection pre-training is comparable to CLIP on GQA, SQA, POPE, and MME; the authors suggest image-level alignment has greater potential, especially with more visual tokens or training data.Source: §5.1 · pp. 9–10, Table 8 · p. 10

5.2. Abaltion on VL Projectors

• At 144 visual tokens, feature-level interactions from front-end pointwise layers make up for token-reduction losses, while adding parameters alone yields no desired gains.The study also reports a positive effect from downsampling tokens at the end of the projector.Source: §5.2 · pp. 10–11, Table 9 · p. 11

5.3. Visual Resolution and Token Numbers

• The resolution-reduction and lightweight-projector designs are compared at 144 visual tokens, and Table 11 is reported to verify LDP’s effectiveness.The comparison uses 336 × 336 input for LDP and 168 × 168 for resolution reduction, with patch size 14 in both cases.Source: §5.3 · p. 11, Table 11 · p. 11

5.4. Quantitative Analysis on SFT

• Vicuna dialogue-mode fine-tuning with ShareGPT data achieves the best performance, significantly enhancing SQA, VQA, MME, and MMBench scores.The comparison includes Alpaca and Vicuna SFT paradigms for MobileLLaMA 1.4B.Source: §5.4 · p. 11, Table 10 · p. 11

• MobileVLM is presented as an efficient set of mobile-scale vision-language models tailored to mobile and IoT devices.Source: §6 · p. 11
• The system combines refurbished language models and vision projection modules, with capacity-enhancing training schemes including SFT, two-stage pretraining and instruction tuning, and LoRA fine-tuning.Source: §6 · p. 11
• The authors report evaluation on mainstream VLM benchmarks and unprecedented speed on typical mobile and IoT devices, and suggest applications in multimodal assistants, self-driving cars, and embodied AI robots.Source: §6 · p. 11

• Table 12 presents the data-sampling strategy for MobileLLaMA training.Source: §A.1 · p. 18, Table 12 · p. 18

• Table 14 presents the prompt template used for MobileLLaMA SFT.Source: B. SFT prompt and response example · p. 18

C. VLM demos

• Figures 6 and 7 show benchmark examples covering attribute understanding, spatial and relational reasoning, science, OCR, object recognition, and word knowledge.The examples are drawn from GQA, MMBench, ScienceQA, TextVQA, POPE, and MME.Source: C. VLM demos · p. 18, Figure 6 · p. 22, Figure 7 · p. 23
• The section provides an example response from the MobileLLaMA 1.4B Base and SFT models and an example SFT-data prompt with its annotation.Source: Table 13 · p. 19, Table 14 · p. 20
• The section includes examples of MobileVLM on VLM training datasets.Figure 5 is captioned as showing these examples.Source: Figure 5 · p. 21