AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model

• Across 32 open-source benchmarks, floating-point AndesVL models achieve first-tier results among similarly sized models across six domains.The domains include text-rich content, reasoning and math, general VQA, multi-image, multilingual understanding, and hallucination mitigation.Source: Figure 1 · p. 2, §1 · p. 3
• The Qwen3-based suite uses visual encoders and covers general VQA, text-rich content, reasoning, hallucination mitigation, multi-image and multilingual inputs, and UI understanding.The report describes its model architectures, training pipeline, and data preparation strategies.Source: §1 · pp. 2–3
• AndesVL trains Instruct and Thinking models separately to support instruct-following and reasoning, respectively.Source: §1 · p. 3
• The 1+N LoRA pipeline clusters similar downstream tasks for adaptation, while QALFT supports applying the architecture on mobile devices.The paper also identifies QAT and QALFT as necessary for compressing models for mobile use.Source: §1 · p. 3
• AndesVL-4B achieves a 6.7x peak decoding speedup, up to 30.9% memory reduction, and 1.8 bits-per-weight on MediaTek Dimensity 9500 chips.The acceleration and compression suite includes customized cache eviction, sparsification, and speculative decoding.Source: §1 · p. 3

2.1 Mobile-side MLLMs

• Scaling-driven MLLM research has produced models with hundreds of billions or trillions of parameters, while Qwen2-VL [227], Qwen2.5-VL [15], and InternVL [28] [229] offer mobile-suited sizes.The report identifies mobile-side MLLMs as relatively underexplored despite these efforts.Source: §2.1 · pp. 3–4
• MobileVLM [36] [37] advanced mobile MLLMs, Ferret UI [249] [123] targeted screen understanding, and Xiaomi’s MobileVLM [233] extended operation across multiple UIs.The report notes that Xiaomi’s 9.8B model was not successfully deployed on mobile devices.Source: §2.1 · p. 4
• BlueLM-V-3B [146] achieved 2.2G running memory and 24.4 tokens/s on MediaTek Dimensity 9300 NPUs, while AndesVL aims to document training, deployment, and benchmarks comprehensively.The report identifies a gap in documenting mobile-side MLLM training processes, deployment solutions, and benchmark results for general and mobile-specific tasks.Source: §2.1 · p. 4

2.2 Mobile-Side Deployment of MLLM

• Mobile deployment remains challenged by resource limits, cross-device compatibility, process standardization, and multimodal evaluation; AndesVL offers training, quantization, compilation, and hardware-specific deployment solutions.The report frames its suite as a foundation for mobile-side MLLM research and practical applications.Source: §2.2 · pp. 4–5
• CPU and GPU deployment work includes MNN’s runtime and kernel optimizations [86], llama.cpp quantization, MLC LLM’s cross-platform compilation, and Transformer-Lite optimizations [114].Transformer-Lite includes dynamic-shape inference, operator optimizations, FP4 quantization, M0E4, and a sub-tensor technique for avoiding KV-cache copying.Source: §2.2 · pp. 4–5
• Gemini Nano targets Android on-device inference, while Apple’s Core ML deployment enables Llama-3.1-8B-Instruct to decode at approximately 30 tokens per second on iPhone 15 Pro.The report describes Gemini Nano models of 1.8B and 3.25B parameters and Apple’s use of GQA, mixed 2-bit and 4-bit quantization, and memory management.Source: §2.2 · p. 5

3.1 Model Architecture

• AndesVL models range from 0.6B to 4B parameters and combine a visual encoder, MLP projector, and Qwen3 LLM with NaViT inputs and quarter-length pixel-shuffled sequences.The models support arbitrary image resolutions and preserve tied word embeddings.Source: §3.1 · p. 6

3.2 Training Pipeline

• AndesVL has instruction and thinking variants: the former supports non-thinking mode, while the latter supports both non-thinking and thinking modes, with distinct training data and methods.Source: §3.2 · p. 6

3.2.1 Pre-train

• Pre-training progresses through vision-language alignment, joint vision-language pre-training, and supervised multi-task training, with images moved before text at 50% probability during joint training.The final stage increases ViT patches to 16,384 and LLM context to 32,768, particularly to support high-resolution inputs and thinking-mode reasoning.Source: §3.2.1 · pp. 6–7

3.2.2 Post-train

• Both variants receive SFT; instruction models then use MPO, while thinking models use GRPO with 43.6K difficulty-selected examples and an easy-to-hard, two-stage curriculum.The selected GRPO dataset includes 15.3K text and 28.3K multimodal examples.Source: §3.2.2 · pp. 8–9

3.3.1 Pre-train Data

• Pre-training data covers image captioning, OCR, visual grounding, GUI tasks, VQA, and interleaved image-text, including an in-house GUI dataset collected across 90 apps.The GUI data includes annotated widget-level referring, grounding, descriptions, and question-answer pairs.Source: §3.3.1 · pp. 9–10

3.3.2 Post-train Data

• Approximately 16 million SFT entries comprise 10% unimodal text and multimodal data, after filtering for quality, difficulty, factual accuracy, image-text correspondence, and hallucination.Filtering used rule-based checks and model-based judgments of quality and difficulty.Source: §3.3.2 · p. 11
• About 80k valid MPO examples were retained after filtering preference pairs for sufficient contrast and rejected-response similarity to the ground truth.MPO data combines in-house examples with MMPR-v1.2.Source: §3.3.2 · p. 12
• Thinking-model GRPO data combines STEM tasks from public and in-house sources, including math, chart, and computational-statistics datasets, with difficulty grading and content categorization.Source: §3.3.2 · pp. 12–13
• Difficulty grading used n rollouts with the SFT-trained thinking model, assigning lower difficulty to data with more correct responses.Source: §3.3.2 · p. 13
• Content categorization identified deficient model capabilities, then used an LLM to select and group semantically similar data instances.Source: §3.3.2 · p. 13
• Approximately 43.6k samples, predominantly mathematical and STEM tasks, formed the reinforcement-learning dataset constructed through the refinement strategies.Source: §3.3.2 · p. 13

4 Mobile-side Application of AndesVL

• The on-device framework pairs an AndesVL foundational model with multiple scenario-specific LoRA adapters and supports quantization, acceleration, and released OPPO AI phone applications.The framework is built on AndesVL after pre-training and post-training.Source: §4 · p. 13

4.1 Multi-scenario LoRA Training

• Scenario-specific LoRAs keep the base model frozen and use customized SFT/RL data, losses, and rewards to adapt tasks while reducing training resource use and mitigating catastrophic forgetting.For captioning, the RL phase rewards greater entity density to produce more informative captions within the same token length.Source: §4.1 · pp. 13–15

4.2 Quantization and Deployment

• The end-to-end quantization pipeline combines base-model QAT with scenario-specific QALFT and fine-grained mixed-precision quantization to preserve performance while improving on-device inference efficiency.The pipeline leverages cloud-based computation and engineering investment.Source: §4.2 · p. 15

4.2.1 Quantization-Aware Training for AndesVL

• QAT supports 2-, 3-, 4-, or 8-bit weights and 8- or 16-bit activations, with mixed-precision assignment and deterministic mapping to hardware-compatible edge representations.The framework addresses the performance degradation and unpredictable accuracy loss associated with direct PTQ deployment.Source: §4.2.1 · p. 15

4.2.2 QALFT

• QALFT freezes the quantization encodings of a QAT-pretrained base model so scenario-specific LoRAs can be updated independently without requantizing the base; on-device performance degrades by only 3% relative to full precision.The approach was co-developed with MediaTek and supports a streamlined task-specific iteration cycle.Source: §4.2.2 · p. 15

4.3 Mobile-side Acceleration with Cache Eviction

• OKV outperforms streamingLLM [236], H2O, and snapKV [120] at the same compression rate and supports context lengths up to 128K.OKV is motivated by the observation that only a small number of tokens in long prompts contribute most of the value.Source: §4.3 · pp. 15–16

4.4 Mobile-Side Acceleration with Speculative Decoding

• Combined with hardware compression and LLM sparsification, customized speculative decoding achieves a 6.7x peak speedup over the baseline and block efficiency (BE) up to 7.9.The method adapts Eagle-like decoding to make use of mobile-device storage and computation resources.Source: §4.4 · p. 16

5 Experiments

• The experiments assess AndesVL’s broad multimodal capabilities, including text-rich images, reasoning and math, multi-image comprehension, general VQA, hallucination mitigation, multilingual tasks, and GUI tasks.Most benchmarks are tested using VLMEvalKit [54].Source: §5 · p. 16

5.1.2 Evaluation Results

• 70.9 overall score across 32 benchmarks puts AndesVL-4B-Thinking 3.2 points ahead of second-best InternVL3.5-4B [229], with 0.9–5.5-point margins across task categories.AndesVL-4B-instruct also performs strongly in multilingual and multi-image tasks.Source: Table 6 · p. 17, §5.1.2 · p. 17

5.1.1 Benchmarks

• The overall evaluation uses 32 benchmarks across six domains and compares AndesVL with advanced multimodal models grouped by parameter size.Comparators include Qwen2-VL [227], Qwen2.5-VL [16], InternVL3, InternVL3.5 [229], Gemma3, and other models.Source: §5.1.1 · p. 17

5.1.2 Evaluation Results

• 64.4 is the highest overall score at 2B for AndesVL-2B-Thinking, while the 1B and 0.6B Thinking and Instruct models lead their size groups.The 2B model excels in multi-image understanding and hallucination mitigation, surpassing some 4B models; the 0.6B series performs comparably to existing 1B models such as InternVL3.5-1B.Source: §5.1.2 · p. 18
• AndesVL’s range of model sizes and strong performance supports deployment across mobile scenarios, including those with highly limited computing resources.Source: §5.1.2 · p. 18

5.2.1 Benchmarks

• The reasoning and math suite covers expert-level college tasks, upgraded academic reasoning, visual-context math, competition problems, diagram-based math, and worst-case accuracy for reliability.These include MMMU [256], MMMU Pro [256], MathVista [139], MathVision [226], MathVerse, and DynaMath [276].Source: §5.2.1 · p. 18
• The math suite also includes WeMath’s hierarchical assessment of visual math reasoning and LogicVista’s five visual logical-reasoning tasks.WeMath covers 67 knowledge concepts across five granularity levels; LogicVista tests spatial, deductive, inductive, numeric, and mechanical reasoning.Source: §5.2.1 · p. 18

5.2.2 Evaluation Results

• 58.3 is AndesVL-4B-Thinking’s highest overall score across reasoning and math benchmarks; AndesVL-2B-Thinking scores 45.7, second to the literature-leading 49.9.AndesVL-4B is notably strong on MathVista, MathVerse, and WeMath, while 1B and 0.6B Thinking models lead their size groups overall and on most individual benchmarks. The authors attribute gains to refined long-CoT multimodal pretraining data and an intricately designed reinforcement-learning process in post-training.Source: §5.2.2 · p. 18, Table 7 · p. 19

5.3.1 Benchmarks

• Text-rich evaluation covers AI2D [97] science-diagram questions, OCRBench [134] OCR tasks, ChartQA [151] chart comprehension, and TextVQA [200] visual reasoning with image text.The reported measures include ChartQA relaxed accuracy and TextVQA validation accuracy.Source: §5.3.1 · p. 19
• The other text-rich benchmarks assess document question answering with DocVQA [153], infographic understanding with InfoVQA, and text-rich visual tasks with SEEDBench-2-Plus [109].DocVQA and InfoVQA use test-set ANLS scores; SEEDBench-2-Plus reports average accuracy.Source: §5.3.1 · p. 19

5.3.2 Evaluation Results

• AndesVL-4B-Thinking scores 86.0 overall on text-rich benchmarks, tops four of eight, and reaches 90.4 on ChartQA versus InternVL3.5-4B’s 86.0.The passage also reports a marked advantage on DocVQA; AndesVL-4B-Instruct scores 85.3 overall.Source: §5.3.2 · pp. 19–20, Table 8 · p. 20
• Smaller AndesVL models lead their respective text-rich model-size groups: 2B-Instruct scores 82.4, 1B-Thinking 77.4, and 0.6B-Instruct 73.5.The paper says these results demonstrate multimodal recognition and comprehension across varied text-rich tasks.Source: §5.3.2 · p. 20

5.4.2 Evaluation Results

• The multi-image assessments evaluate perception and understanding of relations among images across multiple benchmarks.The passage says AndesVL’s pre-training strategies and enhanced training data significantly improve its ability to capture and reason about inter-image relationships.Source: §5.4.1 · p. 20, §5.4.2 · p. 21

5.4.1 Benchmarks

• Multi-image evaluation includes BLINK [59] across computer-vision tasks, MMT-Bench [248] across multimodal tasks, MuirBench [224] across image relations, and Q-Bench [232] for low-level perception.The benchmarks report accuracy, using validation sets for BLINK, MMT-Bench, and Q-Bench.Source: §5.4.1 · pp. 20–21

5.4.2 Evaluation Results

• AndesVL-4B-Thinking scores 67.8 overall on multi-image benchmarks, ahead of InternVL3.5-4B at 62.3, and ranks first on three of four benchmarks.The passage attributes this advantage to pre-training strategies and enhanced training datasets that improve capture and reasoning about inter-image relationships.Source: Table 9 · p. 21, §5.4.2 · p. 21

5.5.1 Benchmarks

• General VQA evaluation includes MME for perception and cognition, MMBench v1.1 for multimodal understanding, and MMVet for six core competencies.MME reports an overall score across 14 sub-tasks; MMBench covers 20 dimensions in English and Chinese; MMVet spans 16 integrated tasks.Source: §5.5.1 · p. 22
• MMStar tests advanced perception, reasoning, math, and science and technology, while RealWorldQA assesses spatial understanding in real-world scenarios.Source: §5.5.1 · p. 22

5.5.2 Evaluation Results

• AndesVL-4B, 1B, and 0.6B achieve top general VQA performance in their respective groups, while AndesVL-2B is highly competitive.The evaluation covers a range of real-world and comprehensive benchmarks.Source: §5.5.2 · p. 22, Table 10 · p. 22

5.6.1 Benchmarks

• Hallusion evaluation covers language hallucination and visual illusions, object recognition and relation comprehension, and object-hallucination tendencies.Scores use the average of HallusionBench’s aAcc, fAcc, and qAcc; CRPE reports accuracy on its Relation subset, and POPE reports an overall average.Source: §5.6.1 · p. 23

5.6.2 Evaluation Results

• 74.8, 71.8, 67.4, and 65.9 are the AndesVL-4B, 2B, 1B, and 0.6B hallucination scores, leading comparable-size models by 1.5–11.5 points.The lead is more pronounced at smaller scales, including the 0.6B variant.Source: §5.6.2 · p. 23, Table 11 · p. 23

5.7.1 Benchmarks

• Multilingual evaluation uses MMMB, Multilingual MMBench, and MTVQA, covering six languages, six languages, and text-rich images in nine languages, respectively.MMMB comprises 15 categories and 12,000 questions; Multilingual MMBench includes English, Chinese, Portuguese, Arabic, Turkish, and Russian.Source: §5.7.1 · p. 24

5.7.2 Evaluation Results

• 64.9 is the leading multilingual score achieved by both AndesVL-4B Thinking and Instruct, exceeding Ovis2-4B [145] by 1.5 points.Smaller AndesVL variants also lead within their respective sub-groups.Source: §5.7.2 · p. 24, Table 12 · p. 24
• AndesVL’s professional-grade multilingual capability provides a foundation for transferring multimodal functions across languages and extending the models’ utility in mobile applications.Source: §5.7.2 · p. 24

5.8.1 Benchmarks

• The GUI evaluation covered ScreenSpot, its annotation-corrected V2 version, and ScreenSpot-Pro for high-resolution professional-app grounding.ScreenSpot spans mobile, desktop, and web environments; ScreenSpot-Pro contains 1,581 instructions from 23 applications across five industries.Source: §5.8.1 · p. 25
• AndesUI-Bench evaluates smartphone UI understanding using 9k referring, 7.6k grounding, 455 description, and 1.2k complex question-answer entries.It represents the test set of the AndesUI dataset.Source: §5.8.1 · p. 25

5.8.2 Evaluation Results

• AndesVL-4B reached 68.4 accuracy, ahead of comparable-size models but behind specialized UI-TARS-2B, and scored 83.5 on AndesUI-Bench.The paper also reports outstanding AndesVL-4B performance across all AndesUI-Bench evaluation metrics.Source: §5.8.2 · pp. 25–26, Table 14 · p. 26

5.9 Ablation Studies

• MPO improved AndesVL-2B-Instruct by over 1.0 pp on MMVet, 0.9 pp on MathVerse, 1.4 pp on HallusionBench, 0.9 pp on RealWorldQA, and 0.7 pp on OCRBench.The authors suggest MPO corrects some SFT errors on instances the SFT model could already solve.Source: Table 15 · p. 26, §5.9 · p. 26
• Both Instruct and Thinking models improved after SFT; MPO benefited Instruct reasoning, multimodal understanding, OCR, and hallucination resistance, while RL benefited Thinking reasoning and complex multimodal understanding.These are the overall ablation conclusions reported alongside the benchmark-specific gains.Source: §5.9 · p. 26
• RL raised AndesVL-2B-Thinking and AndesVL-4B-Thinking by about 2 pp on MathVista, MathVision, and WeMath, and improved MMMU_Pro by over 2 pp.The paper reports these gains even for the 2B model.Source: Table 16 · p. 26, §5.9 · p. 26

6.1 Results of Quantization-Aware Training

• 95% Top-1 overlap between quantized and floating-point models was achieved with QAT+PTQ, significantly improving over PTQ alone.The evaluation used OCR-related benchmarks including DocVQA, InfoVQA, TextVQA, and ChartQA.Source: §6.1 · p. 27

6.2 Results of QALFT

• 3%: QALFT’s performance was slightly below the floating-point model, while QAT and QALFT significantly improved on-device performance over PTQ alone.QALFT trains LoRA weights for specific situations using the pre-trained AndesVL-4B-Instruct-Base model.Source: §6.2 · p. 27

6.3 Results of Cache Eviction

• On the call summary task, OKV improved Rouge-1 relative to SnapKV at a 50% key-value cache eviction ratio and, in certain instances, outperformed the baseline with full KV caches.Experiments used the same AndesVL-4B-Instruct-Base model and were carried out on one device.Source: §6.3 · p. 28, Table 19 · p. 28

6.4 Results of Speculative Decoding

• 6.7x peak decoding speedup and 1.8 bits-per-weight were achieved with speculative decoding, sparsification, and hardware-aware compression; memory reduction reached 30.9% on MediaTek Dimensity 9500 chips.The comparison uses PTQ-only as its baseline.Source: Table 20 · p. 28, §6.4 · p. 28

7 Future Directions

• More optimal visual encoders could improve visual-processing efficiency and accuracy, helping mobile-side MLLMs understand complex content on resource-constrained devices.Proposed approaches include advanced network architectures and novel feature-extraction strategies.Source: §7 · p. 28
• Improved post-training schemes could strengthen multimodal task performance, reduce hallucinations, and make outputs more consistent and reliable.Possible avenues include new training data, adjusted objectives, and optimized algorithms for real-world adaptability.Source: §7 · p. 28
• Distilling knowledge from large cloud models into smaller mobile models could improve their performance-to-resource ratio while maintaining low computational and memory costs.Source: §7 · p. 29
• A unified text-image-speech model could support seamless multimodal interaction and more natural user experiences, contingent on effective fusion, cross-modal representation learning, and efficient mobile inference.The paper presents this three-mode integrated model as a frontier that could also expand mobile-side MLLM applications.Source: §7 · p. 29

8 Conclusion

• 1.8 bits-per-weight, 6.7x peak decoding speed ratio, and up to 30.9% memory reduction are achieved when deploying AndesVL-4B on MediaTek Dimenisity 9500 chips.These results use OKV, designed speculative decoding techniques, and compression strategies.Source: §8 · p. 29
• The 1+N LoRA architecture and QALFT framework enable efficient task adaptation and model compression, with only 3% degradation versus the original floating-point model after mobile deployment.The passage characterizes the resulting precision performance as high and the degradation as ignorable.Source: §8 · p. 29
• AndesVL spans 0.6B to 4B parameters and achieves first-tier performance on multiple open-source benchmarks and AndesUI, especially in mobile UI understanding.The suite integrates Qwen3’s LLM with various visual encoders.Source: §8 · p. 29

App Selection and Screenshot Data Collection

• The dataset spans 90 apps: 65 popular OPPO Software Store apps and 25 ColorOS pre-installed apps.Annotators captured diverse pages and collected 1–10 screenshots per heterogeneous page, with additional examples for homogeneous interfaces when content varied.Source: B AndesUI Dataset · p. 45

Screenshot Data Collection

• 10,747 third-party-app and 2,255 pre-installed-app screenshots formed the training set, alongside 455 heterogeneous test screenshots.Collection targeted atypical scenarios such as network interruptions and pop-ups to broaden interface coverage.Source: B AndesUI Dataset · p. 45

Annotation of Widgets

• Annotators refined Qwen2-VL-72B widget proposals into labels covering bounding boxes, types, text, and clickability, totaling 226,901 training and 9,068 test widgets.Interfaces averaged 18 widgets; VIA-2.0.12 was used for annotation, and Fig. 9 shows label examples.Source: B AndesUI Dataset · p. 45, Figure 9 · p. 46

Basic Data Generation

• Programmatic per-widget generation yielded 226,901 training referring entries and 185,968 grounding entries, versus 8,642 referring and 7,194 grounding test entries.Grounding counts were lower because multiple widgets on a page can share a name; questions came from a seed library initially expanded by GPT-4.Source: B AndesUI Dataset · p. 45

Generation of Advanced Data

• GPT-4 generated screenshot-level page descriptions and natural question-answer pairs, with annotator refinement producing 107,688 training pairs and 1,181 test pairs.The pairs cover descriptive, locating, interaction, and natural-scene questions at easy, medium, and hard difficulty levels.Source: B AndesUI Dataset · pp. 45–46

C Qualitative Examples

• Examples cover AndesVL-4B image understanding, dense captioning, knowledge question answering, multi-image perception, and visual reasoning.These are presented as examples rather than benchmark comparisons.Source: Table 23 · p. 47, Table 24 · p. 47, C Qualitative Examples · p. 47
• Document examples show text and visual-element extraction with query answering, as well as multilingual text extraction from a long receipt.The section groups these with AndesVL-4B’s document understanding and OCR examples.Source: C Qualitative Examples · p. 47, Table 28 · p. 50, Table 29 · p. 51
• AndesVL-4B-Thinking examples include solving a geometry problem and handling chart-related reasoning tasks.The section also provides examples of visual reasoning by AndesVL-4B.Source: C Qualitative Examples · p. 47, Table 30 · p. 52
• In an algebra example, the model uses e - f = 8 and g = 3 to obtain (e - f)/g = 8/3, approximately 2.666....The transcript shows the intermediate substitution and the resulting floating-point value.Source: C Qualitative Examples · p. 53, Table 31 · p. 53
• AndesVL-4B mobile UI examples demonstrate UI understanding and reasoning.The cases are shown in Fig. 33 and 34.Source: C Qualitative Examples · p. 47, Table 33 · p. 55, Table 34 · p. 55