GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices

1. Introduction

• MLLMs generally lose more than 10% on MATH, AlignBench, and MT-Bench relative to corresponding language models, especially on subjective language tasks.The evaluation uses objective mathematical reasoning and subjective alignment and conversational benchmarks.Source: Table 1 · p. 1, §1 · p. 2
• Smartphone NPUs from MediaTek and Qualcomm do not support MoE architectures, whose initialization still requires loading the entire original model into memory.This creates a hardware and memory barrier for MoE-based approaches to preserving language capabilities.Source: §1 · pp. 1–2
• Adding 2M pure-language samples provides moderate benefits on objective NLP tasks but only minimal improvement on subjective tasks, so more data alone is insufficient.The limitation is attributed to the limited volume of high-quality data for human-preference alignment [8] [89].Source: §1 · p. 2
• GenieBlue freezes all original LLM parameters, fully trains every nth duplicated transformer block, and adds LoRA [31] modules to the remaining blocks.During inference, the original frozen model serves language tasks, while multimodal inference swaps in trained blocks and LoRA parameters.Source: §1 · p. 2
• GenieBlue achieves multimodal capabilities comparable to fully fine-tuned MLLMs without sacrificing pure-language abilities and runs efficiently on real smartphone NPUs.The design is presented as an efficient, hardware-friendly structure for edge computing applications.Source: §1 · p. 2

2.1. On-device LLMs and MLLMs

• Small language and multimodal models are increasingly designed for smartphone and edge deployment because they reduce memory and computational demands.Representative families include Qwen, InternLM, and MiniCPM, alongside work on more effective edge-device deployment.Source: §2.1 · pp. 2–3

2.2. Language Capability Maintenance of MLLMs

• Adding text-only data is a common strategy for preserving language capability, but experiments show limited restoration, particularly when high-quality subjective instruction data is scarce.The approach also increases training time and is difficult to scale with high-quality data.Source: §2.2 · p. 3, Table 2 · p. 3, Table 3 · p. 3
• Model-structure approaches commonly use MoE systems that separate text experts from other modality experts, but mainstream smartphone NPUs do not support MoE deployment.Preference-alignment methods such as DPO [60] also do not fully restore language capabilities according to the cited evaluations.Source: §2.2 · p. 3

2.3. Benchmarks for Evaluating LLMs

• Objective benchmarks measure knowledge, mathematics, science, or coding, whereas subjective benchmarks rely on human judgment, interpretation, creativity, and nuanced understanding.The distinction motivates evaluating both benchmark types when assessing language capability for on-device use.Source: §2.3 · p. 3
• For smartphone deployment, instruction following and subjective-task ability may matter more than mastery of complex knowledge.The paper highlights nuanced daily tasks such as text refinement and call summarization as relevant use cases.Source: §2.3 · p. 4

3.1. Training Data Perspective

• Adding 2M pure-text samples leaves multimodal performance virtually unchanged but only moderately improves objective NLP tasks and barely helps subjective tasks.The authors attribute the limited subjective-task improvement to insufficient high-quality human-alignment data.Source: §3.1 · p. 4

3. Text Capability Maintenance for MLLMs

• The approach targets smartphone constraints because mainstream mobile NPUs do not support MoE structures and have limited memory for loading separate model components.CogVLM can require simultaneous loading of the original LLM and visual experts, doubling model memory requirements.Source: §3.2 · p. 5

3.2. Model Structure Perspective

• CogVLM-style structural methods typically retain over 90% of full-fine-tuning multimodal performance while preserving pure-text capability through non-shared deployment.Full fine-tuning harms pure-text performance, whereas LoRA and CogVLM avoid that effect under the non-shared strategy.Source: §3.2 · p. 5

3.3. GenieBlue

• GenieBlue freezes the original LLM, fully trains replicated transformer blocks, adds LoRA to remaining blocks, and selects separate weights for text-only and multimodal inference.The non-shared base strategy avoids MoE deployment and decouples LLM and MLLM inference.Source: §3.3 · pp. 5–6, Figure 2 · p. 6
• GenieBlue-Skip achieves over 97% multimodal-performance retention versus full fine-tuning and outperforms the evaluated GenieBlue and CogVLM variants.The comparison uses last-quarter, first-quarter, and every-quarter replicated-block placements.Source: Table 5 · p. 6, §3.3 · p. 6
• Non-shared deployment maintains pure-text capabilities better than shared deployment, while shared-base outcomes vary with trainable parameters and model configuration.For Qwen2.5-3B, the LoRA-trained MLLM has the worst pure-text performance under the shared-base strategy.Source: §3.3 · pp. 6–7, Table 6 · p. 7

4.1. Training Recipe

• GenieBlue pre-trains the MLP projector with 2.5M samples while keeping the ViT and LLM frozen, then fine-tunes multimodal components with 645M samples while the original LLM remains frozen.The second stage trains the ViT, projector, replicated transformer blocks, and added LoRA parameters.Source: §4.1 · p. 7
• The training recipe uses SigLIP as the ViT, BlueLM-3B as the LLM, and LoRA modules with rank 8.Source: §4.1 · p. 7

4.2. Deployment Recipe

• GenieBlue is deployed on the iQOO 13 smartphone’s Snapdragon 8 Elite NPU using Qualcomm QNN SDK3 and quantization for the ViT, projector, LLM, and LoRA parameters.The ViT and projector use W8A16, the LLM uses W4A16, and LoRA parameters use W8A16 quantization.Source: §4.2 · p. 7
• The deployment currently supports single-patch ViT inference, while the Snapdragon 8 Elite NPU does not support MoE structures.Source: §4.2 · p. 7

5.1. MLLM Performance

• 97% performance retention of BlueLM-V-3B is reported for GenieBlue on multimodal benchmarks, while it slightly outperforms InternVL2-8B on average.The evaluation covers MMbench, MMStar, MMMU, MathVista, HallusionBench, AI2D, OCRBench, and MM-Vet; GenieBlue’s accuracy is slightly lower than Qwen2.5-VL-3B.Source: Table 7 · p. 8, §5.1 · p. 8

5.2. LLM Performance

• GenieBlue retains 100% of the original LLM’s performance, whereas Qwen2.5VL-3B exhibits degradation, especially on subjective language tasks.Source: Table 8 · p. 8, §5.2 · p. 8

5.3. Deployment Efficiency

• GenieBlue incurs longer model loading, slightly higher storage and memory use, and marginally slower token output than BlueLM-V-3B.Despite these costs, the reported token output speed is 30 token/s, described as sufficient for daily mobile use.Source: Table 9 · p. 8, §5.3 · p. 8

5.2. LLM Performance

• The non-shared base deployment strategy preserves GenieBlue’s pure-language performance by splitting LLM and MLLM inference processes.Source: §5.2 · p. 8

• GenieBlue freezes the original LLM during training and adds duplicated transformer blocks plus lightweight LoRA modules to acquire multimodal capabilities.Source: §6 · p. 8
• The resulting design maintains language performance while achieving competitive multimodal results on smartphone NPUs.Source: §6 · p. 8

• GenieBlue is described during MLLM inference using the BlueLM-V-3B architecture, which includes a redesigned dynamic-resolution processor and token downsampler for on-device deployment.Source: A. Structure Details of GenieBlue · p. 13
• The proposed structural focus is the design of transformer blocks within the language model.Source: A. Structure Details of GenieBlue · p. 13

B. Training Data Composition

• Cambrian-7M includes approximately 1.5M pure-text training samples, while GenieBlue’s data composition is described as 645M fine-tuning data, without specifying that figure as a sample count.Source: B. Training Data Composition · p. 13

C. More Discussions

• GenieBlue decouples multimodal training parameters from the original language model, aiming to preserve language performance while retaining good multimodal performance.The design also requires minimal hardware-side adaptation and reduces engineering difficulty for end-side deployment.Source: C. More Discussions · p. 13

D. Hyper Parameters

• The final GenieBlue model uses the same 2.5M pre-training data and 645M fine-tuning data as BlueLM-V-3B [53].These quantities are specified for the pre-training and fine-tuning stages.Source: D. Hyper Parameters · p. 14

D.1. Pre-training Stage

• The pre-training stage uses 2.5M training samples, with its hyper-parameters provided for stage 1 of GenieBlue.The supplied passage identifies the table as covering the pre-training stage.Source: Table 11 · p. 14

D.2. Fine-tuning Stage

• Training samples are concatenated to reach a sequence length of 4096, enhancing fine-tuning speed.The passage describes this concatenation as a speed-oriented fine-tuning choice.Source: §D.2 · p. 14
• The fine-tuning stage uses 645M training samples, with its stage-2 hyper-parameters provided in the corresponding table.The table describes the fine-tuning configuration for GenieBlue.Source: Table 12 · p. 14