PaperlayerView exact revision on arXiv

Source-linked AI summary

Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs

Luka Ribar, Jeevan Bhoot, Douglas Orr

arXiv:2608.21134v1cs.CVcs.LG

TL;DR

Mobile VLM deployment is constrained by substantial memory and compute requirements, while aggressive low-bit quantization can severely degrade multimodal performance without the original training data. Llama-Mobile combines model-generated-data QAT with S3D8, a 2.7-bit-per-parameter format designed for Arm CPUs, and compresses Llama 3.2 11B Vision Instruct to 3.7 GB with 8-bit activations while preserving good VQA performance.

  • Problem

    VLMs are difficult to deploy on low-resource Arm CPUs because they require substantial memory and computation, and aggressive quantization can severely degrade visual question-answering ability.

  • Method

    Llama-Mobile combines a model-generated synthetic multimodal data pipeline for QAT with S3D8, a 2.7-bit-per-parameter format that decodes weights to INT8 for Arm CPU inference.

  • Results

    The method compresses Llama 3.2 11B Vision Instruct to 3.7 GB with 8-bit activations while preserving good performance across VQA benchmarks.

  • Takeaways & Limitations

    The results suggest that sub-3-bit VLM compression is a viable route toward multimodal inference on resource-constrained devices.

  • Takeaways & Limitations

    Experiments focus on one model, CPU execution, and a small set of VQA-style benchmarks, while other architectures may require specialized kernels.

Abstract

from arXiv · show

Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.

1. Introduction

Llama-Mobile targets the memory and compute costs that make VLM deployment difficult on mobile devices. It combines model-generated-data QAT with the 2.7-bit S3D8 format and an Arm CPU inference implementation.

  • Motivation: VLMs offer multimodal understanding but remain computationally and memory-heavy for mobile deployment.
  • Motivation: Aggressive low-bit VLM quantization requires both a suitable numerical format and a quantization procedure, while QAT performance depends on training-data selection.
  • Approach: S3D8 packs three signed weights per byte through a shared centroid index and decodes them to INT8 for inference.
  • Approach: The QAT pipeline generates data from the model itself, avoiding access to original training data and downstream task fine-tuning.
  • Evaluation: 3.7 GB weights with 8-bit activations preserved limited degradation across standard visual-question answering tasks for Llama 3.2 11B Vision Instruct.
  • Implementation: The project provides a full VLM inference implementation for Arm CPUs on Linux and Android.

2. Related Work

Prior work explores quantization procedures, numerical representations, and hardware-aware kernels for efficient inference. Llama-Mobile co-designs its numerical format around the SIMD decode path of Arm CPUs.

  • Quantization methods: PTQ is less resource-intensive, whereas QAT is generally more effective at low bit widths where quantization error is significant.
  • Numerical formats: Quantized values can use uniform integers, floating-point values, lookup tables, scaling schemes, or vector-quantization codebooks.
  • Hardware co-design: Hardware-aware designs consider weight layouts, codebooks, SIMD kernels, and hardware-aligned blocks for efficient inference.
  • Hardware co-design: S3D8 differs by co-designing its numerical format around the SIMD decode path on Arm CPUs.

3. Method

The method combines synthetic multimodal data generation, knowledge-distillation-based QAT, and S3D8 weight compression for efficient Arm CPU inference without original training data.

  • Overview: The framework generates synthetic multimodal training samples with the model itself, then uses them for QAT.ImageNet images are paired with sampled prompts and teacher-generated responses.
  • Quantization-Aware Training: QAT trains a quantized student to match the full-precision teacher’s next-token distributions on generated response tokens.The student uses quantized parameters during the forward pass, with gradients propagated through quantization using a straight-through estimator.
  • Synthetic Data Generation: The synthetic dataset targets diverse response behavior because short image captions provide limited supervision for long-form, question-answering-oriented generation.The procedure varies prompt templates, question types, instructions, answer lengths, output formats, and styles without using downstream benchmark data.
  • The S3D8 Format: S3D8 stores three signed weights in 8 bits using a shared 5-bit centroid index and three sign indicator bits.A table lookup provides absolute values for a three-weight vector, which are combined with signs and per-channel scales for reconstruction.
  • The S3D8 Format: S3D8 is designed for fast dequantization to INT8 on Arm CPUs using logical operations and small lookup tables.With bfloat16 channel scales, the average representation size is approximately (8/3 + 16/k) bits/element for sufficiently large dimensions.

4. Results

Across visual question-answering tasks, S3D8 provides favorable accuracy–compression results at approximately 2.7 bits per parameter, while its Arm CPU runtime benefits depend on inference shape. Prompt sampling also improves quantization-aware training performance.

  • 4.1. Downstream Tasks: 0.083 average task degradation: S3D8 at 2.68 bits per parameter outperforms similarly sized INT, student-t, and lloyd-max formats.The comparison uses QAT variants at similar model sizes.
  • 4.1. Downstream Tasks: 0.661 average task performance: QAT improves S3D8 from 0.340 with GPTQ at approximately 2.7 bits per parameter.With GPTQ, rate-matched INT reaches 0.018, isolating a substantial weight-format difference relative to S3D8.
  • 4.2. Sampled vs. fixed prompts: Sampled prompts substantially improve downstream performance over a single fixed prompt in fixed block-scaled INT3 QAT experiments.The experiment varies the number of QAT steps and uses Figure 3b for comparison.
  • 4.3. Runtime Performance: 133μs (16.5 GB/s): S3D8 dequantization for the vision MLP up-projection on Pixel 8a using 5 cores.The benchmark reports read bandwidth under a shared-destination setup that simulates on-chip cache retention.
  • 4.3. Runtime Performance: S3D8 speeds up text generation relative to INT8 but slightly slows text prefill and vision encoding.Small-m matrix multiplies are memory-bandwidth-bound, whereas larger-m operations are compute-bound.
  • 4.3. Runtime Performance: 3.8 tokens/s: the custom S3D8 implementation achieves this median decoding rate on Pixel 8a for a 3.73 GB model file.The measurement generates 100 tokens from one image tile and a short prompt, excluding loading and prefill.

5. Limitations

The evaluation is limited to one VLM and CPU execution, with additional scope boundaries from the benchmark set and hardware-specific decoder design.

  • Scope: The experiments focus on Llama 3.2 11B Vision Instruct and CPU execution, limiting direct evidence across models and accelerators.S3D8’s Arm-optimized decoder may require specialized kernels on other architectures.

6. Conclusion

Llama-Mobile combines synthetic multimodal distillation data with S3D8 to compress Llama 3.2 11B Vision Instruct for low-resource inference. The method uses diverse generated prompts and responses, reaches 3.7 GB with 8-bit activations, and supports sub-3-bit Arm CPU inference.

  • 6. Conclusion: Llama-Mobile is a framework for aggressively quantizing vision-language models for low-resource inference.Its approach combines a synthetic multimodal data pipeline for quantization-aware distillation with the S3D8 weight format.
  • 6. Conclusion: 3.7 GB with 8-bit activations: the method compresses Llama 3.2 11B Vision Instruct while preserving good performance across VQA benchmarks.The conclusion also reports practical runtime efficiency for the method.
  • A.1. Data Generation: The synthetic QAT data uses ImageNet images paired with teacher-generated responses to sampled prompts, without downstream benchmark images or benchmark-specific prompts.Prompt construction includes instruction templates, generic image-comprehension questions, and optional instruction blocks.
  • A.1. Data Generation: Prompt templates are applied with probability 0.75, while additional instruction blocks are prepended with probability 0.7.The instruction block may include adherence, length, and format probes.

A.2. Training

The experiments train and compare multiple quantization procedures and scalar parameterizations, including QAT, GPTQ, and S3D8-based GPTQ. GPTQ scaling behavior is evaluated across codebook sizes and group sizes, with affine scaling stronger at lower rates and absmax competitive at higher rates.

  • Training setup: QAT examples use randomly sampled ImageNet images and random prompts, while training runs use batch size 128 and maximum sequence length 512.Most quantized models train for 2048 steps with AdamW and a cosine learning schedule.
  • Quantization procedures: The study compares direct casting, GPTQ, and QAT, including GPTQ applied to language and vision linear matrices rather than only the text decoder.GPTQ uses synthetic image-text calibration data and retains bfloat16 activations during evaluation.
  • GPTQ parameterizations: GPTQ evaluates uniform scalar INT grids using absmax and symmetric affine parameterizations, which reconstruct the same centered integer levels for even K.The parameterizations differ in scale normalization: s_absmax=a/(K/2−1), while s_affine=2a/(K−1).
  • GPTQ results: Affine scaling achieves better accuracy at lower rates, while absmax scaling matches or slightly outperforms it at higher rates.For K=16, the respective scales are a/7 and a/7.5; affine has finer resolution near zero, whereas absmax represents both ±a exactly.
  • S3D8 GPTQ: S3D8 GPTQ fits a 32×3 centroid table and per-output-channel bfloat16 scales, then assigns rescaled three-weight vectors to the closest centroid during sequential GPTQ processing.The scales and centroids remain fixed while GPTQ updates weights.

A.5. Evaluation

Evaluation uses fixed-size subsets of four visual question answering benchmarks with task-specific prompts and answer handling. All models are evaluated under the same procedure, while absolute bfloat16 scores are not intended to reproduce model-card results exactly.

  • Benchmarks: The evaluation covers VQAv2, DocVQA, ChartQA, and AI2D, using validation or test splits and a fixed sample size of 1024 per task.VQAv2, ChartQA, and AI2D use accuracy as the evaluation metric.
  • Benchmark controls: All models are evaluated using the same procedure as a controlled relative benchmark for quantization.The implementations use fixed subsets, validation splits for VQAv2 and DocVQA, and the authors’ own prompting and answer normalization.
  • Prompting: Prompt templates and maximum generated sequence lengths follow the model guidelines, with task-specific answer formats for visual questions, charts, documents, and scientific diagrams.AI2D additionally includes multiple-choice options because the original template lacked them.

B. Additional Results

After QAT, S3D8 maintains a favorable performance-compression trade-off across individual tasks relative to the compared numerical formats. The per-task curves follow a trend similar to the aggregate results.

  • Per-task trade-offs: All evaluated tasks show a similar performance-compression trend after QAT, with S3D8 offering a good trade-off compared with INT, lloyd-max, and student-t formats.Figure 5 presents the individual-task results corresponding to the broader trade-off analysis.

Training Loss vs. Task Performance

The additional results relate training loss to downstream performance and describe implementation choices that make S3D8 practical on Arm CPUs. They also examine compression rate, lookup design, bit packing, and output-channel organization.

  • Loss-performance relationship: Training KL divergence and average downstream task performance show a strong overall correlation after 2048 QAT steps, with Spearman’s ρ=−0.91.The training set was chosen independently of the task data.
  • Loss-performance relationship: The KL-performance relationship saturates at both low and high KL values, where downstream performance changes little across models.The loss still captures differences between models in these regimes.
  • Compression trade-offs: Per-task QAT results show a similar performance-compression trend across tasks and a good trade-off for S3D8 versus INT, lloyd-max, and student-t formats.The figure provides individual-task comparisons rather than only aggregate performance.
  • Bit packing: A straightforward S3D8 bit layout requires additional bit manipulation to extract and apply sign indicators after centroid lookup.The optimized design addresses this using Arm TBL and fused signed lookups.
  • Format design: S3D8 targets 2.7 bits per parameter because short vector quantization can provide a better rate-distortion trade-off than scalar quantization in the 2–4-bit regime.The selected rate is presented as an aggressive compression point with small degradation from the pretrained model.
  • Compute format: INT8 is selected for computation because Arm dotprod and i8mm extensions broadly support it and generally provide higher arithmetic throughput than bfloat16 or float16.The stated hardware rationale concerns modern mobile CPUs.
  • Arm implementation: S3D8 uses Arm TBL lookups with signed lookup tables, permuted bit layouts, and output-channel packing to reduce runtime bit manipulation and alignment overhead.Weights packed into each byte correspond to three output channels, avoiding complicated reduction-axis alignment and padding.

D. Runtime Performance Benchmarking

The benchmarking evaluates Arm CPU implementations across mobile and server configurations, with controlled sampling and attention to timing variability. It compares fused and unfused kernels and matrix-product workloads across generation and prefill settings.

  • Benchmark setup: The study benchmarks blocked and fused matrix products using template-generated C++ code and OpenMP outer-loop parallelism.These implementations target instruction-level parallelism and parallel execution across outer loops.
  • Measurement protocol: Each benchmark sample averages the final 15 of 20 iterations, with 50 samples collected per benchmark after discarding initial iterations.Distinct input copies reduce on-chip cache reuse, while asymmetric slow outliers are attributed to scheduling, frequency scaling, and memory management.
  • Benchmark setup: Pixel 8a benchmarking uses both a 5-core performance configuration and a 1-core Cortex-X3 configuration because broader OpenMP runs showed high timing variance.The 1-core setup provides more stable timings, while the 5-core setup retains the potential for higher performance.
  • Dequantization performance: S3D8 dequantization performance depends on cache and memory behavior, with read bandwidth changing as output size approaches and exceeds the L2 cache.The reported bandwidth uses source-tensor size for reads, while destination writes may remain cached; write bandwidth is 3× read bandwidth for S3D8 dequantization.
  • Kernel performance: The S3D8 fused kernel is fastest for generation, while INT8 is fastest for text and image prefill.For text and image prefill, the unfused S3D8 kernel runs only slightly slower than INT8.
  • Kernel implementation: The fused S3D8 matrix-vector product decodes packed values, accumulates dot products in int32, applies input and weight channel scales, and writes bfloat16 outputs.The hot loop uses five logical instructions for index generation and three tbl lookups to decode 48 elements.

E. Model Generation Examples

Examples compare original bfloat16 and S3D8 generations under the same prompts and decoding setup across four visual question answering tasks. The quantized model matches the shown answers on VQAv2, ChartQA, and AI2D, but misses the DocVQA reference answer.

  • Evaluation setup: Figures 11 to 14 provide one example from each downstream task using matched prompts and decoding settings.The examples cover VQAv2, ChartQA, DocVQA, and AI2D.
  • VQAv2: On VQAv2, both generations match the evaluation target because human answers include both “4” and “5”.The example illustrates evaluation agreement despite differing numeric generations.
  • ChartQA: On ChartQA, both models recover the correct final answer for the displayed example.The figure compares the original bfloat16 model with the S3D8 model under the same evaluation setup.
  • DocVQA: On DocVQA, the quantized model does not generate the reference answer, whereas the bfloat16 model does.The shown reference answer is “Preliminary.”
  • AI2D: On AI2D, both models select the correct multiple-choice answer, while the quantized model produces a longer rationale.The displayed question asks which insect part is used for locomotion or support; the correct option is 3) Leg.
Loading 2608.21134v1…