PaperlayerView exact revision on arXiv

Source-linked AI summary

UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Ioannis Maniadis Metaxas, Adrian Bulat, Alberto Baldrati, Anestis Zaganidis, Yassine Ouali, Hyeonuk Kim, Georgios Tzimiropoulos

arXiv:2607.23373v1cs.CV

TL;DR

LVLM deployment on resource-constrained edge devices is hindered by computationally heavy vision encoders that are not designed for on-device latency. UltraViT addresses this gap with a latency-informed heterogeneous pyramidal encoder and two-stage generative pre-training, achieving state-of-the-art efficient LVLM encoding while operating nearly 1.7× faster than prior encoder-centric baselines.

  • Problem

    LVLMs remain difficult to deploy on resource-constrained edge devices because existing approaches largely retain heavy vision encoders not designed for LVLMs or on-device latency.

  • Method

    UltraViT combines a latency-informed heterogeneous pyramidal architecture with dense feature distillation followed by direct generative supervision from a frozen capacity-mixed LLM.

  • Results

    Nearly 1.7× faster than prior encoder-centric computational baselines, UltraViT achieves state-of-the-art on-device performance for efficient LVLM encoding.

  • Takeaways & Limitations

    UltraViT establishes a state-of-the-art vision encoder for efficient LVLM encoding on mobile hardware across diverse multimodal benchmarks.

  • Takeaways & Limitations

    The work adopts the widely-used LLaVA-OV design.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

1 Introduction

LVLM deployment on mobile and edge devices is hindered by massive computational footprints, while prior efficiency efforts largely overlook vision encoder design. UltraViT addresses this gap through latency-informed heterogeneous architecture and two-stage generative pre-training, achieving nearly 1.7× faster on-device performance than prior encoder-centric baselines.

  • Motivation: LVLMs’ massive parameter counts and computational footprints fundamentally hinder deployment on resource-constrained mobile and edge devices.
  • Limitations of Prior Work: Prior efficient LVLMs primarily reduce visual-context length or pair lighter language models with efficient cross-modal interaction, largely overlooking the vision encoder architecture.
  • Limitations of Prior Work: Efficient ViTs are typically optimized for ImageNet classification and replicate one spatial mixer uniformly, limiting suitability for LVLM dense image understanding and reasoning.
  • UltraViT Architecture: UltraViT introduces a pyramidal vision encoder that uses latency measurements to select heterogeneous spatial mixers across stages and adapt operations for NPU/mobile execution.
  • UltraViT Pre-training: UltraViT’s two-stage generative pre-training combines dense spatial distillation with direct generative supervision from a frozen, capacity-mixed LLM.
  • Results: 1.7× faster: UltraViT achieves state-of-the-art on-device performance while outperforming prior encoder-centric computational baselines such as FastVLM.

2 Related Work

Related work on efficient vision encoders spans convolutional, transformer-based, and hybrid architectures, while efficient LVLM research mainly reduces visual tokens or pairs encoders with smaller language models.

  • Efficient Vision Encoders: Efficient vision architectures are broadly categorized as convolutional, transformer-based, or hybrid according to their operator primitives.This classification frames the design space for computationally efficient vision encoders.
  • Efficient Vision Encoders: CNN research introduced efficiency techniques including depth-wise convolutions, inverted residuals, channel shuffling, and cheap linear transformations.These techniques appeared in MobileNets, ShuffleNets, and GhostNet, respectively.
  • Efficient Vision-Language Models: Efficient LVLM research primarily reduces visual-token sequence length or couples vision encoders with smaller, highly optimized language models.Token reduction uses training-aware compression or post-hoc pruning and attention sparsification, and these methods provide speedups across accuracy-latency tradeoffs.

3 Architecture optimization

UltraViT optimizes the vision encoder for on-device latency through a heterogeneous four-stage pyramidal architecture that progressively aggregates spatial information and reduces token length. It preserves a standardized macro-block while selecting and adapting spatial mixers according to stage-specific accuracy–latency trade-offs.

  • Architecture optimization: UltraViT uses a four-stage pyramidal topology to progressively aggregate spatial information and reduce the visual token sequence length.The architecture targets efficient on-device deployment rather than relying on a rigid, homogeneous ViT backbone.
  • Architecture optimization: The network preserves one residual macro-block topology across all depths while modularly interchanging the internal token mixer.The standardized block retains the arrangement of residual connections, a token mixer, and a subsequent MLP channel mixer.
  • Spatial mixers: UltraViT evaluates convolutional, sparse, channel-partitioned, and vanilla attention mixers to balance latency and dense representational capacity across resolutions.Convolutions suit early high-resolution stages, sparse attention mediates between convolutional efficiency and full-attention range, and vanilla attention benefits later low-resolution stages.
  • Spatial mixers: The optimized sparse mixer replaces multi-head attention with single-head attention, fuses sampling into strided-convolution projections, uses nearest-neighbor upsampling, and compresses value projections by 4.These adaptations address latency penalties observed when deploying the original sparse-attention instantiation on mobile NPUs; Cqk is set to 16.
  • Block selection: Mixer selection is systematic and stage-specific because the accuracy–latency trade-off and computational behavior vary with feature resolution N and channel dimension C.The final architecture uses C1−4=[192, 512, 768, 1536] and N1−4=[4, 7, 10, 2].
  • Pyramidal structure and stem: A stem downsamples the high-resolution input by 4, followed by four stages that halve spatial dimensions while expanding channel capacity.Stage transitions use a downsampling block based on 1x1 convolution, 3x3 strided depthwise convolution, and the cited hierarchical design.

4 Efficient Vision Encoder pre-training for LVLMs

UltraViT uses a sample-efficient, two-stage pre-training strategy tailored for LVLM vision encoders: dense distillation transfers spatial and semantic knowledge, followed by generative pre-training with a frozen, vision-aligned LLM. The method preserves fine-grained spatial information while dedicating the encoder’s capacity to visual feature extraction.

  • Two-stage pre-training: The strategy combines dense distillation with generative pre-training, moving beyond standard CLIP or SigLIP contrastive training.Dense distillation transfers knowledge from a large teacher, while the second stage uses a pretrained, frozen LLM.
  • Dense distillation: Structural re-alignment preserves teacher spatial detail instead of naively downsampling high-resolution features to the student resolution.The method rearranges spatial blocks into channels and uses resolution-aligned downsampling when exact alignment is impossible.
  • Dense distillation: The dense distillation objective combines patch-level and globally pooled cosine alignment losses as Ldistill = 0.5Ldense + 0.5Lglobal.The dense loss matches localized representational distributions, while the pooled loss preserves global semantic understanding.
  • Generative pre-training: The generative stage uses a strong LLM that is pre-aligned to vision and then frozen as the decoder.Freezing the aligned LLM relieves the vision encoder of linguistic modeling and focuses its capacity on extracting rich visual features.

5 Experiments

Experiments show that UltraVLM substantially outperforms FastVLM and token-reduction baselines while using a faster encoder, producing 16.0× fewer visual tokens and requiring markedly less pre-training data. Evaluations use broad LVLM benchmarks and on-device measurements from an INT8-quantized implementation on a Samsung Galaxy S25 Ultra.

  • Evaluation protocol: On-device latency and throughput are measured on physical hardware after QNN compilation and INT8 quantization on a Samsung Galaxy S25 Ultra.The protocol applies to both block-level latency search and full-model evaluation.
  • Benchmark results: UltraVLM outperforms retrained FastVLM across most benchmarks, with gains of +6.0% on TextVQA, +5.1% on DocVQA, and +6.2% on ChartQA.FastVLM was retrained using the same LLaVA-OV training recipe for fair comparison.
  • Token reduction: 16.0× token reduction is achieved natively by UltraVLM’s pyramidal architecture, which outperforms existing token-reduction strategies across every benchmark.Unlike post-hoc pruning, the intrinsic reduction also speeds up the vision encoder; all methods use the same LLM and training data.
  • Inference efficiency: 19.1× faster inference is achieved by UltraViT than SigLIP or SigLIP-2 400M, while UltraViT operates nearly 1.7× faster than FastViT at 512 × 512px.UltraViT also outputs 16× fewer visual tokens, reducing the LLM input sequence length and computational burden.
  • Pre-training efficiency: 3× fewer total samples and 10× fewer unique images are used by UltraViT than FastVLM during pre-training, with 4.1B versus 13B samples and 235M versus 2B images.The two-stage framework combines dense distillation with generative supervision.

6 Ablation Studies

The ablations show that dense distillation consistently improves UltraViT and FastViT, especially on fine-grained tasks, while stronger teachers further enhance performance. Generative pre-training with a frozen LLM also yields notable gains, though smaller decoders provide more limited improvements.

  • Impact of the dense distillation: Dense distillation consistently outperforms standard SigLIP contrastive training, with larger gains on fine-grained DocVQA and InfoVQA tasks.The improvement is observed for UltraViT and, across all benchmarks, for FastViT.
  • Impact of the dense distillation: Stronger teachers and longer training further improve dense-distillation performance, indicating scalability with teacher capacity.Under the same training setup, UltraViT reaches comparable performance to FastViT while running significantly faster on-device.
  • Impact of generative pre-training: Fine-tuning UltraViT with a frozen LLM during generative pre-training produces notable accuracy gains across the evaluated benchmarks.The ablation also examines the choice of LLM decoder.
  • Impact of generative pre-training: A 0.5B LLM improves performance to a smaller degree because its limited linguistic capacity forces the vision encoder to compensate for language modeling.The supplied passage indicates that this compensation bottlenecks visual representation learning.

7 Conclusion

UltraViT is a latency-optimized vision encoder for deploying Large Vision-Language Models on edge devices, combining a heterogeneous architecture guided by real on-device measurements with a two-stage pre-training strategy.

  • 7 Conclusion: UltraViT is explicitly designed to deploy Large Vision-Language Models on edge devices.The paper presents it as a latency-optimized vision encoder.
  • 7 Conclusion: Its heterogeneous architecture is guided by real on-device latency measurements.This is the first stated contribution of the paper.
  • 7 Conclusion: Its two-stage pre-training strategy combines dense feature distillation with generative LLM supervision.This is the second stated contribution of the paper.

A Additional Implementation Details

UltraViT’s dense distillation primarily uses SigLIP2-B16 NaFlex5 as teacher, with a larger-teacher variant resuming from the dense-distilled checkpoint. The strategy aligns multi-scale student and teacher features through Pixel Shuffle, patch-wise loss, and global semantic consistency.

  • Teacher models: SigLIP2-B16 NaFlex5 serves as the teacher for most dense distillation experiments.The + Large Teacher setting instead resumes from the dense-distilled checkpoint and uses SigLIP2-400M.
  • Teacher models: The + Large Teacher setting resumes dense-distillation training with SigLIP2-400M replacing the original teacher.This setting is reported in Table 5(a).
  • Dense distillation pipeline: Pixel Shuffle re-aligns multi-scale student features to teacher resolution before dense patch-wise alignment and teacher-attention pooling enforce global semantic consistency.The losses are denoted L_dense and L_global, respectively.

B Additional Experiments · B.1 Extended OCR retrieval results · B.2 Effect of different teachers for dense pretraining

The additional experiments examine mixer configurations for OCR retrieval and assess how different dense-pretraining teachers affect UltraViT. The selected mixer remains competitive with better efficiency, while SigLIP2 and TULIP provide the strongest overall teacher performance and the approach remains robust to teacher choice.

  • B Additional Experiments: The block-selection study compares token mixer configurations in zero-shot image classification, OCR retrieval, and post-LLaVA-OV fine-tuning benchmarks.
  • B.1 Extended OCR retrieval results: The extended OCR evaluation reports recall@1 for five datasets individually and as their average: TextCaps plus four VisRAG-Ret datasets.The VisRAG-Ret datasets are SlideVQA, InfoVQA, ChartQA, and ArxivQA.
  • B.1 Extended OCR retrieval results: The (C, C+S, A, A) configuration achieves the highest average performance across the five OCR benchmarks.
  • B.1 Extended OCR retrieval results: The selected (C, C+S, CP, A) configuration remains competitive, with only a marginal accuracy drop while offering better efficiency.
  • B.2 Effect of different teachers for dense pretraining: The teacher study evaluates SigLIP2-NF (naflex), Perception Encoder, SigLIP2, and TULIP across eleven vision-language benchmarks.
  • B.2 Effect of different teachers for dense pretraining: SigLIP2 and TULIP consistently outperform the other teacher variants across most tasks and yield the best overall performance.Despite these differences, UltraViT is robust to the choice of teacher.

B.3 Efficient VLM comparisons · B.4 On-device measurements methodology · B.5 GPU and CPU performance

The supplementary experiments compare UltraViT with efficient VLMs using vision-only and end-to-end latency and memory measurements, while detailing on-device, block-level, GPU, and CPU evaluation protocols. These measurements support the paper’s claim that latency-informed vision-encoder design improves on-device efficiency.

  • B.3 Efficient VLM comparisons: UltraViT is compared with efficient VLMs using benchmark performance, fixed-resolution vision-encoder inference times, recipe-based TTFT, and peak on-device vision-encoder memory.The comparisons are reported in Table 8 and include both vision-only and end-to-end measurements.
  • B.3 Efficient VLM comparisons: UltraViT is significantly faster than other highly efficient VLMs, highlighting the benefit of designing its vision encoder specifically for on-device deployment.The supplied passage attributes this advantage to the architecture’s on-device design.
  • B.4 On-device measurements methodology: On-device measurements use a Samsung Galaxy S25 Ultra with models compiled using the Qualcomm Neural Processing SDK.The protocol runs 7 rounds of 100 inference passes per model, discards the fastest and slowest rounds, and averages the remaining 5 rounds.
  • B.4 On-device measurements methodology: Block-level measurements stack 10 blocks of one type and evaluate them across model stages using each stage’s UltraViT configuration.The measurements vary stage-specific block widths and corresponding processing conditions.
  • B.4 On-device measurements methodology: For 512×512 inputs, Stage 1 block measurements use width C = 192 and input shape (192, 64, 64), while Figure 3 balances parameters across block types.The number of blocks per type is varied so the compared block types have approximately the same parameter count.
  • B.5 GPU and CPU performance: UltraViT and FastViT are evaluated on 512x512 images using an Nvidia RTX4090 GPU and an Inter i7-14700K CPU.GPU tests use batch size 96, whereas CPU tests use single-image inference after ONNX conversion; both report inference time per sample.
  • B.5 GPU and CPU performance: Table 9 reports CPU and GPU inference times for UltraViT and FastViT.The reported metric is inference time per processed sample under the specified GPU and CPU protocols.
Loading 2607.23373v1…