PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices

HyperAI Team, Yuchen Liu, Kaiyang Han, Zhiqiang Xia, Yuhang Dong, Chen Song, Kangyu Tang, Jiaming Xu, Xiushi Feng, WenXuan Yu, Li Peng, Mingyang Wang, Kai Wang, Changpeng Yang, Yang Li, Haoyu Lu, Hao Wang, Bingna Xu, Guangyao Liu, Long Huang, Kaibin Guo, Jinyang Wu, Dan Wu, Hongzhen Wang, Peng Zhou, Shuai Nie, Shande Wang, Runyu Shi, Ying Huang

arXiv:2512.14052v1cs.CVcs.CL

TL;DR

On-device deployment of multimodal large language models is challenging because their complex architectures and large parameter scales are difficult to run efficiently. HyperVL combines image tiling, adaptive visual-resolution compression, and dual consistency learning; it achieves state-of-the-art performance across multimodal benchmarks while reducing latency and memory usage on mobile devices.

  • Problem

    Complex architectures and large parameter scales make multimodal large language models difficult to run efficiently on on-device platforms.

    Source: §1 · p. 1

  • Method

    HyperVL uses image tiling, a Visual Resolution Compressor to predict encoding resolutions, and Dual Consistency Learning to connect multi-scale ViT encoders to a shared LLM.

    Source: §1 · p. 2

  • Results

    HyperVL achieves state-of-the-art performance across multiple multimodal benchmarks while significantly reducing latency and memory usage on mobile devices.

    Source: §5 · p. 16

  • Takeaways & Limitations

    HyperVL demonstrates generalization across on-device tasks including UI understanding and parsing, intent recommendation, and image-text creation.

    Source: §5 · p. 16

Abstract

from arXiv · show

Current multimodal large lanauge models possess strong perceptual and reasoning capabilities, however high computational and memory requirements make them difficult to deploy directly on on-device environments. While small-parameter models are progressively endowed with strong general capabilities, standard Vision Transformer (ViT) encoders remain a critical bottleneck, suffering from excessive latency and memory consumption when processing high-resolution inputs.To address these challenges, we introduce HyperVL, an efficient multimodal large language model tailored for on-device inference. HyperVL adopts an image-tiling strategy to cap peak memory usage and incorporates two novel techniques: (1) a Visual Resolution Compressor (VRC) that adaptively predicts optimal encoding resolutions to eliminate redundant computation, and (2) Dual Consistency Learning (DCL), which aligns multi-scale ViT encoders within a unified framework, enabling dynamic switching between visual branches under a shared LLM. Extensive experiments demonstrate that HyperVL achieves state-of-the-art performance among models of comparable size across multiple benchmarks. Furthermore, it significantly significantly reduces latency and power consumption on real mobile devices, demonstrating its practicality for on-device multimodal inference.

1 Introduction

HyperVL targets efficient on-device multimodal inference, where constrained compute and memory make high-resolution visual encoding a key bottleneck. It combines image tiling, adaptive resolution selection, and dual-consistency training to reduce visual costs while supporting dynamic visual paths through a shared LLM.

1 Introduction

  • On-device MLLMs offer privacy protection and lower cloud inference costs, but cloud-oriented models are difficult to run within smartphone and tablet compute and memory budgets.Growing demand for on-device multimodal capabilities makes the performance–efficiency trade-off a central concern.Source: §1 · p. 1
  • Compact MLLMs still rely on quadratic-complexity ViT encoders that incur high memory use and latency on high-resolution inputs; aggressive token downsampling can substantially degrade performance.The introduction cites UI understanding and grounding as on-device scenarios requiring high-resolution inputs, and identifies Apple’s FastVLM [Vasu et al., 2025] as an example of convolutional downsampling.Source: §1 · p. 1
  • At comparable model sizes, HyperVL achieves state-of-the-art results across multiple multimodal benchmarks and outperforms general models on on-device scenarios; Figure 1 shows performance comparable to larger models.The paper also reports reduced latency and memory consumption on commercial mobile devices, without giving numerical values in these introduction passages.Source: Figure 1 · p. 2, §1 · p. 2
  • Image tiling splits high-resolution inputs into independently encoded patches to reduce peak memory, while VRC predicts an image’s optimal encoding resolution from its information density to avoid unnecessary computation.The paper describes VRC as a plug-and-play acceleration method that can be integrated into pretrained MLLMs.Source: §1 · p. 2
  • Dual Consistency Learning aligns ViT encoders of different capacities with a shared LLM, enabling dynamic switching between lightweight and high-accuracy visual paths.Path selection can depend on task type, latency budget, or device compute capability; the introduction also reports real-hardware validation of latency and memory reductions on commercial mobile devices.Source: §1 · p. 2

2 Related Work

Related work targets on-device multimodal efficiency through smaller models and algorithmic or system-level optimizations. Despite progress, visual encoders remain a key deployment bottleneck, especially for high-resolution inputs.

Small-scale Multimodal Models

  • Qwen-VL and InternVL offer 1B–4B variants with strong multimodal reasoning, yet ViT memory use and latency remain on-device bottlenecks.FastVLM compresses visual tokens with CNNs but introduces noticeable performance degradation [Vasu et al., 2025]; AndesVL enables stable execution of a 3B model within 2 GB of memory through system-level and ARM-kernel optimization [Jin et al., 2025].Source: §2 · p. 3

Efficiency Improvements for On-device MLLMs

  • Efficiency approaches compress visual tokens with MiniCPM-V’s Perceiver Resampler [Yu et al., 2025] or adapt token counts through visual consistency learning [Cui et al., 2025], while distillation transfers visual and cross-modal knowledge [Xu et al., 2024, Yang et al., 2025a, Wang et al., 2023].These methods combine with model quantization (QAT/PTQ) [Liu et al., Xiao et al., 2023, Liu et al., 2024a] and inference operator fusion as a technical stack for on-device inference.Source: §2 · p. 3

3 HyperVL

HyperVL combines adaptive image-resolution compression with two switchable visual branches that share an LLM, using dual-branch consistency training to align their outputs. A diverse multimodal corpus and three-stage training pipeline support the model.

3.1 Model Architecture

  • HyperVL combines two differently sized visual encoders, a visual-language projector, a shared Qwen3 1.7B LLM, and a compressor that scales images to 10%–100% of their original size.The SigLIP2-Large and SigLIP2-Base encoders use AnyRes processing and non-overlapping image tiles; a two-layer MLP projects their outputs, and pixel shuffle reduces visual-token length to one quarter. The branches share the LLM and can be switched across downstream tasks. SigLIP2, AnyRes, and Qwen3 [Yang et al., 2025b].Source: §3.1 · pp. 3–4

3.2 Dual Consistency Learning

  • Dual Consistency Learning alternates training between the two visual branches and distills the larger branch’s output distribution into the smaller branch to align their semantics.A temperature-smoothed KL-divergence constraint is applied only to text tokens, skipping image tokens; the shared LLM supports branch switching according to task or device capabilities.Source: §3.2 · pp. 4–5

3.3 Visual Resolution Compressor

  • The Visual Resolution Compressor predicts image-specific scaling to reduce visual tokens and computation while preserving task performance, with selectable ratios from 0.1–1.0.Its supervision selects the smallest compression ratio that stays within a predefined loss tolerance; at inference, it can retain high resolution for complex scenes and compress low-information images. Unlike ViCO, which supports 0.25 and 1, it offers finer-grained selection [Cui et al., 2025].Source: §3.3 · pp. 5–6

3.4.1 Overview

  • HyperVL’s training corpus covers captioning, VQA, OCR, document understanding, grounding and counting, GUI interaction, STEM, and text-only tasks.It combines open-source and synthetic data, dynamically adjusting modality and task proportions across training stages; captioning data is also re-annotated, using WIT’s original text as context. WIT [Srinivasan et al., 2021].Source: §3.4.1 · pp. 6–7

3.4.2 Data Pipeline

  • The data pipeline achieves categorization accuracy exceeding 96% on complex image–text samples through task decomposition–recombination.Its three stages cover preparation and categorization, cleaning and normalization, and quality filtering and mixed packaging; these include dual-modal deduplication and training–test decontamination, followed by quality assessment across six dimensions and hierarchical sampling.Source: §3.4.2 · pp. 8–9

3.5 Training Pipeline

  • HyperVL uses three training phases: vision-language alignment, knowledge enhancement, and multi-task training.The first trains an adapter with frozen ViT and LLM parameters; the second unfreezes them for joint pre-training with loss on text tokens only; the final phase uses approximately 23M high-quality examples, including short and long chain-of-thought data, for complex reasoning.Source: §3.5 · pp. 9–10

4 Evaluation · 4.1 Evaluation Setting

HyperVL is evaluated for multimodal understanding, reasoning, and generation on public and internal benchmarks, against comparable-scale baselines. Public benchmarks use VLMEvalKit, and GPT-4o judges answers and checks consistency.

4.1 Evaluation Setting

  • The evaluation covers multimodal understanding, reasoning, and generation across public and internal benchmarks.The authors describe it as a systematic evaluation.Source: §4.1 · p. 10
  • Comparisons use state-of-the-art multimodal baselines with comparable parameter scales to support fair comparisons.Source: §4.1 · p. 10
  • Public benchmarks are evaluated with VLMEvalKit, using a standardized and reproducible protocol.The passage identifies VLMEvalKit as an open-source evaluation framework.Source: §4.1 · p. 10
  • GPT-4o serves as the judge for automated answer evaluation and consistency checking.The passage says this is intended to ensure objectivity and reliability of reported results.Source: §4.1 · p. 10

4.2 Public Benchmarks

The public benchmark framework evaluates HyperVL across multimodal reasoning, OCR and document understanding, comprehensive multimodal tasks, and hallucination detection.

4.2.1 Multimodal Reasoning and Mathematics

  • Reasoning and mathematics evaluation uses MMMU_val [Yue et al., 2024], MMMU_Pro_V [Yue et al., 2025], and MathVista [Lu et al., 2024] to test expert knowledge, robust vision-only questions, and visual mathematics.The benchmarks cover cross-modal integration across more than 30 disciplines, strict filtering of text-only solvable items, and mathematical reasoning in text-image scenarios.Source: §4.2.1 · p. 10

4.2.2 OCR, Chart, and Document Understanding

  • OCR, chart, and document evaluation spans diagram reasoning (AI2D [Kembhavi et al., 2016]), chart parsing (ChartQA [Masry et al., 2022]), visual text QA (TextVQA [Singh et al., 2019]), and structured document extraction (DocVQA [Mathew et al., 2021]; InfoVQA [Mathew et al., 2022]).These tasks assess text recognition, numerical reasoning over charts, and information extraction from scanned documents, PDFs, and structured materials such as invoices and contracts.Source: §4.2.2 · p. 10

4.2.3 Comprehensive Multimodal tasks

  • The evaluation covers MME’s 14 perception and cognition subtasks, MMBench_CN [Liu et al., 2024d], MMBench v1.1 (EN/CN) image understanding and multilingual question answering, and MMVet factual accuracy [Yu et al., 2024].The MMMB_CN subset [Sun et al., 2025] specifically assesses Chinese language capability.Source: §4.2.3 · p. 11

4.2.4 Hallucination Evaluation

  • Hallucination evaluation uses HallusionBench [Guan et al., 2024], POPE [Li et al., 2023b], and CRPE [Wang et al., 2024c] to probe unsupported visual claims, object-existence errors, and fabricated reasoning.The tests examine factual hallucinations in visual scenes, visual hallucination rates through contrastive question-answering, and factual consistency in visual-text reasoning chains.Source: §4.2.4 · p. 11

4.3 Internal Benchmarks

HyperVL is evaluated on a specialized internal benchmark designed to capture real-world deployment scenarios across intent recognition and recommendation, image-text creation, UI understanding and structured parsing, and image relevance ranking.

4.3.1 Intent Recognition and Recommendation

  • For screenshot-based intent recognition, generated search queries must satisfy six criteria—relevance, accuracy, logic, fluency, recommendation value, and safety—or they are rejected.The task evaluates UI interpretation, intent extraction, and actionable query generation for downstream search recommendations.Source: §4.3.1 · p. 12

4.3.2 Image-Text Creation

  • Image-text creation evaluates generated text for relevance, multimodal alignment, content quality, and style consistency across social-media, posting, and product-review interfaces.The text must align with the application context and user-uploaded images while following user-defined stylistic constraints.Source: §4.3.2 · p. 12

4.3.3 UI Understanding and Structured Parsing

  • For order-detail screenshots, the model jointly analyzes text, layout, and semantic relationships to extract key fields without templates; Field Accuracy is correct fields divided by valid fields.The structured outputs are intended for downstream applications.Source: §4.3.3 · p. 12

4.3.4 Image Relevance Ranking

  • Image relevance ranking scores each query–candidate image pair through fine-grained cross-modal interaction, then reranks candidates to prioritize relevant, high-quality results.Evaluation uses Precision@10, Precision, and F1 Score.Source: §4.3.4 · pp. 12–13

4.4 Experimental Results

HyperVL combines competitive public-benchmark performance with strong results on internal tasks. Its ablations and on-device evaluations show benefits from DCL and VRC, efficient inference, and robustness to quantization.

4.4.1 Public Benchmarks

  • HyperVL scores 64.5 on OpenCompass, while HyperVL ViTL reaches 66.1 with gains across most benchmarks.HyperVL also scores 83.8 on ChartQA, 81.8 on AI2D, and 91.3 on DocVQA; the base model is competitive with several 2B–3B models.Source: §4.4.1 · p. 13
  • On public benchmarks, HyperVL scores 66.2 on MathVista and both variants show HallBench performance comparable to larger models.HyperVL maintains solid baseline performance on MMMU; the authors describe the variants’ hallucination-related performance as stable.Source: §4.4.1 · p. 13

4.4.2 Internal Benchmarks

  • Despite its 1.8B parameter size, HyperVL outperforms larger competitors in high-level semantic understanding and creative generation on internal benchmarks.It scores 94.0 in Intent Recognition and Recommendation, and ranks first in Image-Text Creation at 49.8 and in Image Relevance Ranking.Source: §4.4.2 · p. 14

4.4.3 Ablation Studies

  • DCL improves HyperVL over its baseline by 5.0 on AI2D, 2.6 on ChartQA, and 22.0 points on OCRBench.The passage attributes the gains to transfer of fine-grained visual knowledge that helps the lightweight model bridge the gap with larger vision encoders.Source: §4.4.3 · p. 14
  • VRC reduces HyperVL’s visual tokens by 20.2% while retaining 98.7% of baseline performance, with only 2 ms of overhead.Compression adapts to image content: it reduces tokens by 49.5%–63.4% on DocVQA, 2%–3% on ChartQA, and 30% on OCRBench.Source: §4.4.3 · p. 15

4.4.4 On-Device Evaluation

  • On-device evaluation reports an approximate 12.9× speedup and a 6.8× reduction in peak memory overhead for HyperVL.HyperVL maintains constant memory usage regardless of input complexity; the authors attribute its efficiency to serial processing of fixed-size chunks that fit in Qualcomm NPU VTCM.Source: Figure 5 · p. 15, §4.4.4 · pp. 15–16
  • Under W4A16 quantization, DocVQA changes from 91.3 to 91.2 and ChartQA from 83.8 to 83.2.OCRBench retains 98.2% of original capability, changing from 830 to 815.Source: Table 7 · p. 16, §4.4.4 · p. 16

5 Conclusion

HyperVL combines image tiling, a visual resolution compressor, and dual consistency learning to enable efficient on-device multimodal inference while achieving strong benchmark performance and generalization across mobile tasks.

5 Conclusion

  • HyperVL achieves state-of-the-art performance across multimodal benchmarks while reducing latency and memory usage on mobile devices, with generalization to several on-device tasks.The paper names UI understanding and parsing, intent recommendation, and image-text creation as examples.Source: §5 · p. 16
  • Future work will explore token sparsity and attention pruning, extend HyperVL to video and interactive scenarios, and add user-adaptive personalization for mobile applications.The stated aim is to improve efficiency, relevance, and resource-aware performance in real-world use.Source: §5 · p. 16

B Qualitative Examples

The section illustrates HyperVL’s range of multimodal capabilities through examples of visual reasoning, information extraction, interface planning, image description, common-sense inference, and knowledge-based understanding.

Related papers

Loading 2512.14052v1…