BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

• Mobile phones offer offline, low-latency interaction and broad access, but their limited memory and compute constrain MLLM deployment.Mainstream dynamic-resolution methods also increase ViT inferences and image-token counts, hindering image speed and end-side latency.Source: §1 · pp. 1–2
• The paper frames mobile deployment as an accessible way to bring MLLMs into everyday tasks, including augmented reality and real-time translation.Source: §1 · pp. 1–2

1) Algorithm and System Initiative

• BlueLM-V-3B addresses these constraints through relaxed aspect-ratio matching and hardware-aware system optimization for mobile MLLM inference.The paper also identifies batched image encoding, pipeline parallelism, token downsampling, chunked computing, mixed precision, and an optimized inference framework.Source: §1 · p. 2

2) State-of-the-art MLLM Performance

• 66.1 on OpenCompass is reported for BlueLM-V-3B, which leads among similarly sized models and surpasses some larger MLLMs.The comparison spans multiple benchmarks, and BlueLM-V-3B leads in the majority of datasets.Source: Figure 1 · p. 1, §1 · p. 2

3) High Deployment Efficiency

• 24.4 token/s generation speed and 2.2GB memory are reported on a MediaTek Dimensity 9300, with 768×1536 images encoded in approximately 2.1 seconds.Source: §1 · p. 2

2.1. Multimodal Large Language Models

• Dynamic resolution supports high-resolution image understanding, but too many patches and long image-token sequences challenge mobile image encoders and language-model latency.Source: §2.1 · p. 2
• Visual knowledge is integrated into language models through linear projectors, Q-Formers, and perceiver resamplers.Source: §2.1 · p. 2

2.2. On-device Large Language Models

• Compact 2–3B-parameter language and multimodal models are presented as deployable on user devices, and BlueLM-V-3B adds algorithm and system co-design for mobile efficiency.Source: §2.2 · pp. 2–3
• BlueLM-V-3B follows the classical LLaVA architecture, adding dynamic-resolution processing and token downsampling for capability and deployment complexity, respectively.Source: Figure 2 · p. 3

3. BlueLM-V-3B

• The section presents algorithm and system co-design as the approach for optimizing efficiency during training and mobile deployment.Source: §3 · p. 3

3.1. Basic Network Components

• The model uses a SigLIP ViT with 400M parameters, a two-layer MLP projector, and an in-house 2.7B BlueLM language model.Source: §3.1 · p. 3

3.2. Dynamic Image Resolution

• BlueLM-V-3B’s relaxed aspect-ratio matching counters image enlargement; in the cited examples it selects 1:1, minimizing image tokens after ViT encoding.The method uses a threshold to prevent always selecting larger resolutions.Source: Figure 3 · p. 4, §3.2 · p. 4
• Batched image-patch encoding and CPU–NPU pipeline parallelism overlap image-encoding work; the paper reports a 10% training speedup from batching.For inference, the NPU processes a fixed batch of patches at a time; the pipeline hides Conv2D execution latency.Source: §3.2 · pp. 4–5

3.3. Token Downsampler

• For a 2:4 image, downsampling reduces 6,561 SigLIP image tokens to 1,764, while chunked computing processes 128 input tokens in parallel per iteration.Source: §3.3 · p. 5

3.4. Model Quantization and Overall Framework

• The deployment uses INT8 ViT and projector weights, INT4 LLM weights, INT16 LLM activations, FP16 ViT and projector activations, and an INT8 KV cache.Source: §3.4 · p. 5
• 2.2GB is the reported peak memory usage when image processing is decoupled from user instruction handling, reducing first-token waiting time and improving responsiveness.Source: Figure 6 · p. 6, §3.4 · p. 6

4.1. Training Process

• Training proceeds from projection-layer pre-training with the ViT and LLM frozen to full-model fine-tuning on image-text pairs.The first stage equips the model with basic multimodal capabilities.Source: §4.1 · p. 6

4.2. Training Data

• Pre-training uses 2.5 million image-caption pairs from LLaVA 558k, ShareGPT4V 1200k, and ALLaVA 708k.Source: §4.2 · p. 6
• The fine-tuning dataset contains 645 million image-text pairs spanning captioning, VQA, OCR, and pure-text data.The data combines open-source and in-house sources.Source: §4.2 · p. 6
• In-house data expands the fine-tuning set with manually created examples, rendered documents and multilingual OCR, and captions and QA pairs made or revised with GPT-4o and Gemini Pro.PDFs, formulas, and tables are converted into image-text representations using PyMuPDF, Matplotlib, and IMGKit, respectively.Source: §4.2 · p. 6

5.1. Relaxed Aspect Ratio Matching

• BlueLM-V-3B’s relaxed aspect-ratio method selects smaller ratios than LLaVA-NeXT in 29k cases and than InternVL 1.5 in 523k cases, while selecting larger ratios than InternVL 1.5 in 25k cases.The paper reports that these selections improve NPU inference efficiency.Source: §5.1 · p. 7
• Across BlueLM-3B and MiniCPM-2B experiments, relaxed aspect-ratio matching reduces image tokens and improves accuracy, and the comparison table reports the proposed method’s best performance.Both language models have similar parameter counts of 2.7B; the experiments use LLaVA pre-training and fine-tuning data.Source: §5.1 · p. 7, Table 2 · p. 7

5.2. Accuracies on Different Benchmarks

• BlueLM-V-3B ranks second on average and achieves state-of-the-art results on 4 of 8 OpenCompass tasks, outperforming several larger models including InternVL2-8B and MiniCPM-V-2.6 (8B).The comparison covers models up to 10B parameters.Source: Table 3 · p. 7, §5.2 · p. 8
• BlueLM-V-3B performs comparably to similarly sized state-of-the-art MLLMs on text-centric/OCR benchmarks while significantly enhancing multilingual capability.The evaluation includes TextVQA, DocVQA, and MTVQA.Source: Table 4 · p. 7, §5.2 · p. 8

5.3. Deployment Efficiency Evaluation

• For a 2:4 image with nine patches, processing four patches in parallel gives the fastest ViT encoding performance on the NPU.Increasing patch parallelism initially reduces latency, but latency rises after a certain point.Source: §5.3 · p. 8, Figure 7 · p. 8
• The Conv2D and vision-transformer pipeline encodes a 2:4 image in 2.06 s and hides 200 ms of latency.The reported encoding time is 1.9 + 0.16 s.Source: §5.3 · p. 8
• With input length and KV cache fixed at 2048, t128/t1 has the lowest prefilling latency and fastest generation speed among the tested token-processing settings.The paper notes slight output-speed variation that may reflect testing-environment fluctuations.Source: Figure 8 · p. 8, §5.3 · p. 9
• On the Dimensity 9300 comparison, BlueLM-V-3B achieves shorter latency and faster token throughput than MiniCPM-V, whose reported deployment statistics are for an 8B model on the CPU.BlueLM-V-3B is tested with a 768×1536 image, 2048 input tokens, and a KV cache length of 2048; the MiniCPM-V paper does not specify equivalent configuration details.Source: Table 5 · p. 8, §5.3 · p. 9

6. Conclusion and Future Discussions

• BlueLM-V-3B is designed through algorithm-system co-design and hardware-aware optimization for training and deployment on mobile phones.Source: §6 · p. 9
• Extensive experiments and statistical analyses demonstrate strong capability and high efficiency on mobile devices.Source: §6 · p. 9
• Future work will optimize scalability across a broader range of mobile devices and explore algorithms to improve performance and usability.Source: §6 · p. 9

7. Relaxed Aspect Ratio Matching

• The procedure evaluates candidate resolutions formed from possible aspect ratios at a 384-pixel base dimension and returns the selected ratio as the best fit.The candidate ratios are converted to resolutions of 384m×384n.Source: §7 · p. 16
• The update rule uses α to favor candidates with greater retained area while also considering resized area not covered by the original image.The described update can increase the likelihood of selecting a candidate with smaller retained area and smaller unused area.Source: §7 · p. 16
• For the 380×393 image, the method selects 384×384 rather than LLaVA-NeXT’s 384×768; for the 500×102 image, it selects 768×384 rather than InternVL 1.5’s 1920×384.These examples illustrate the method’s resolution choices relative to the two approaches.Source: §7 · p. 16, Figure 9 · p. 16

8. Open-source Training Dataset

• Other listed datasets include ALLaVA [14], ScienceQA [79], Orca-Math [90], OpenOrca, MetaMathQA [137], WizardLM [130], and MathInstruct [117].The list also includes VizWiz [36], Laion [99], COCO [20], LLaVA, SVIT [146], SA1B, VSR [66], Chart2Text [48], MultiMath [94], ArXivCap [59], and COYO.Source: §8 · p. 17
• The listed caption-text datasets are TextCaps [104] and Screen2Words.Source: §8 · p. 17
• The listed OCR datasets include Wukong [32], HierText [76], TextOCR [108], WildReceipt [111], and DocILE.The list also includes SVRD, DocLayNet, XFUND, COCO-Text [121], SROIE, FUNSD [44], CORD, Paper2Fig100k [98], Docmatix, LAION-2B-OCR, SynthDoG [50], WebSight, DeepForm, Kleister [110], and TabFact [19].Source: §8 · p. 17
• Table 6 lists the open-source datasets used for fine-tuning by category and data volume.It corresponds to the categories and data volume in Table 1 of the main text.Source: Table 6 · p. 17
• Some datasets may appear in more than one category, and data may overlap across datasets.Source: §8 · p. 17

9. Hyper-parameters for Training

• Table 7 gives hyper-parameters for pre-training stage 1, while Table 8 gives them for fine-tuning stage 2.Source: §9 · p. 17, Table 7 · p. 17, Table 8 · p. 17
• The passage identifies Tables 7 and 8 as the sources for the two stages’ hyper-parameters.Table 7 covers pre-training and Table 8 covers fine-tuning.Source: Table 7 · p. 17, Table 8 · p. 17
• Upsampling smaller datasets can make training steps multiplied by global batch size exceed the total data volume.Source: §9 · p. 17

10. Visualization on Practical Cases

• The arithmetic demo simplifies 10x^8 / 5x^4 to 2x^4 and selects that matching option.Source: §10 · p. 18
• The chart response reports methane emissions of 30% for Natural Gas and Petroleum Systems and 14% for MSW Landfills, a 16% difference.Source: §10 · p. 18
• Image responses identify the 2024 Nobel Prize in Chemistry poster’s three scientists and name Claude Monet as the painter.The poster response names David Baker, Demis Hassabis, and John M. Jumper.Source: §10 · pp. 19–20
• For a multilingual sign, the response reads “Welcome” in English, Spanish, French, German, and Italian.Source: §10 · p. 19
• Other image responses describe a cartoon boy sculpture, judge a car’s aerodynamic shape, and advise against eating potentially poisonous mushrooms.Source: §10 · pp. 19–20
• Train-related responses provide the Praha hl.n.–Wien Hbf route, Brno- Židenice as the next stop, and a speed of 160 km/h.A separate response reads two alarm times: 8:30 tomorrow morning and 16:00 today.Source: §10 · pp. 19, 21