PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenPacker: Efficient Visual Projector for Multimodal LLM

Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, Lei Zhang

arXiv:2407.02392v4cs.CV

TL;DR

MLLM visual projectors must reduce redundant visual tokens without losing fine-grained information, because token count affects the LLM’s computational demands. TokenPacker uses coarse-to-fine region-to-point injection to enrich low-resolution queries with high-resolution features, and experiments report 75%–89% compression with comparable or better performance and higher efficiency. Performance declines when compression reaches 32 or fewer tokens.

  • Problem

    Visual projectors must preserve fine-grained visual information while reducing redundant tokens, whose quantity substantially affects MLLM efficiency.

  • Method

    TokenPacker interpolates low-resolution point queries and enriches them by injecting high-resolution, multi-level region cues within corresponding local contexts.

  • Results

    Across diverse benchmarks, TokenPacker reduces LLaVA-1.5 visual tokens by 75%–89% while achieving comparable or better performance with significantly higher efficiency.

  • Takeaways & Limitations

    TokenPacker provides compact visual token representations while retaining competitive multimodal and high-resolution comprehension performance.

  • Takeaways & Limitations

    When reduced to 32 (6%) or fewer tokens, TokenPacker shows a clear decline in performance.

Abstract

from arXiv · show

The visual projector serves as an essential bridge between the visual encoder and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs adopt a simple MLP to preserve all visual contexts via one-to-one transformation. However, the visual tokens are redundant and can be considerably increased when dealing with high-resolution images, impairing the efficiency of MLLMs significantly. Some recent works have introduced resampler or abstractor to reduce the number of resulting visual tokens. Unfortunately, they fail to capture finer details and undermine the visual reasoning capabilities of MLLMs. In this work, we propose a novel visual projector, which adopts a coarse-to-fine scheme to inject the enriched characteristics to generate the condensed visual tokens. In specific, we first interpolate the visual features as a low-resolution point query, providing the overall visual representation as the foundation. Then, we introduce a region-to-point injection module that utilizes high-resolution, multi-level region-based cues as fine-grained reference keys and values, allowing them to be fully absorbed within the corresponding local context region. This step effectively updates the coarse point query, transforming it into an enriched one for the subsequent LLM reasoning. Extensive experiments demonstrate that our approach compresses the visual tokens by 75%~89%, while achieves comparable or even better performance across diverse benchmarks with significantly higher efficiency. The source codes can be found at https://github.com/CircleRadon/TokenPacker.

1 Introduction

MLLM efficiency depends heavily on the visual tokens produced by the projector, creating a need for compact representations that retain visual quality. TokenPacker addresses this with coarse-to-fine feature injection and achieves substantial token compression with competitive performance.

  • Motivation: The visual projector bridges the visual encoder and LLM, while the LLM dominates computation and visual-token count strongly affects MLLM efficiency.The projector translates visual features into tokens in the LLM’s text embedding space, so token quality also affects overall efficacy.
  • Motivation: Current linear MLP projectors preserve visual contexts through one-to-one transformation but produce redundant tokens, especially for high-resolution images or videos.Resamplers and Q-Formers instead reduce token counts with learnable queries and cross-attention, but the passage introduces this as a separate research line without completing its limitations.
  • Approach: TokenPacker uses a coarse-to-fine projector that injects high-resolution, multi-level region details into low-resolution point queries within local context regions.Interpolated visual features provide coarse holistic queries, while region-to-point injection supplies fine-grained candidate keys and values that enrich them for the LLM.
  • Approach: TokenPacker also introduces dynamic image slicing for efficient high-resolution image understanding.This extension is presented as an additional component alongside the visual projector.
  • Results: 75% (576 vs. 144)∼89% (576 vs. 64) visual-token reduction is achieved in LLaVA-1.5 while maintaining comparable or better performance with higher efficiency.The reported result summarizes experiments across diverse multimodal benchmarks and includes competitive high-resolution comprehension.

2 Related Work

Related work centers on aligning visual encoders with LLMs through projectors, while addressing the tension between preserving detail and reducing visual-token counts. High-resolution encoder strategies target tasks requiring fine-grained visual information.

  • MLLM Foundations: MLLMs integrate LLMs with visual encoders to improve multimodal content comprehension and understanding.Earlier efforts such as Flamingo and BLIP-2 used image-text datasets for cross-modal alignment.
  • Visual Projectors: Visual projectors align visual signals with the LLM space and generally follow linear projection or token-reduction approaches.MLP projectors preserve visual contexts but generate redundant tokens, whereas resamplers and Q-Formers use learnable queries and cross-attention to control token counts.
  • Visual Projectors: Query Proposal Networks extend resampling with initial-query generation and multi-level cross-attention, while Abstractor and LDP use convolutional layers for local visual interaction.The passage lists these as developments building on or complementing projector-based token reduction.
  • High-Resolution Understanding: CLIP-ViT’s low-resolution input constrains performance on fine-detail tasks such as dense OCR, crowd counting, and small-object visual grounding.Other methods address this limitation by directly employing higher-capacity visual encoders such as SAM or ConvNeXt.

3 Method

TokenPacker is an efficient visual projector that bridges the vision encoder and LLM using a coarse-to-fine design. It combines dynamic aspect-ratio-preserving image slicing with region-to-point injection to produce compact visual tokens for high-resolution understanding.

  • Standard MLLM framework: The standard MLLM pipeline converts visual embeddings into visual tokens before autoregressive response generation, with the LLM receiving both visual and textual tokens.The visual encoder, projector, and LLM form the three principal components of the framework.
  • Standard MLLM framework: Reducing visual tokens is pivotal because LLM computation and memory demands dominate, while computational expense generally grows quadratically with input-token quantity.The projector maps N visual embeddings to M visual tokens, with the efficiency objective M < N.
  • TokenPacker overview: TokenPacker connects the vision encoder and language model while targeting a small number of visual tokens through a coarse-to-fine framework.It is designed to generate condensed visual token representations before LLM processing.
  • Region-to-point injection: TokenPacker downsamples visual features by bilinear interpolation into M = N/s^2 low-resolution embeddings, allowing the visual-token count to be controlled by scaling factor s.Each low-resolution embedding corresponds to an s×s high-resolution sub-region.
  • Region-to-point injection: Point-to-region cross-attention uses low-resolution embeddings as queries and high-resolution, multi-level regions as keys and values to update compact visual tokens with fine-grained information.The injection module absorbs local region details into coarse representations.
  • High-resolution image understanding: Dynamic image slicing selects grid configurations that preserve aspect ratio, minimize padding, and align grid resolution with the input image before TokenPacker processes and merges patch tokens.The method uses padding and overlap scores to identify suitable grids, resizes and zero-pads patches, and preserves an aspect-ratio-respecting overview.

4 Experiments

Experiments evaluate TokenPacker on normal- and high-resolution benchmarks, comparing visual-token compression, accuracy, throughput, projectors, and image-slicing schemes. The method retains competitive or improved performance while substantially reducing tokens and improving efficiency.

  • Experimental Setup: The experiments use LLaVA-1.5 with CLIP-ViT-L/14-336px and Vicuna-7/13B, training TokenPacker in two stages with fixed vision encoders.Training uses CC-558K and the 665K mixture, with additional Mini-Gemini data for competitive performance.
  • Normal Resolution: 75% fewer visual tokens, from 576 to 144, improves MMBench, VizWiz, and POPE metrics over LLaVA-1.5.Reported gains are +0.8%/+0.3% on MMBench, +2.0% on VizWiz, and +1.1%/+1.5% on POPE for Vicuna-7B/13B.
  • Normal Resolution: Around 5 times TPS, 4.9 vs. 24.9, accompanies average gains of +0.8% and +0.1% over LLaVA-1.5 with Vicuna-7B and Vicuna-13B.The paper notes marginal declines on VQAv2 and GQA despite the average gains.
  • Normal Resolution: At 64 tokens, TokenPacker achieves +3.9% on MMBench and +3.5% on VQAv2 against LLaVA-PruMerge with Vicuna-13B.The method is also evaluated at 36 tokens and surpasses prior fewer-token methods across three benchmarks.
  • High Resolution: At 1344×1344 with approximately 1393 visual tokens, TokenPacker reaches 70.6% on VQAT and 521 on OCRBench.At approximately 619 visual tokens, it obtains 38.2% MMMU, 1577/353 MME, and 61.0% VizWiz, while comprehensive benchmarks favor 1088×1088.
  • Ablation Results: TokenPacker outperforms MLP-based projection by +1.9% on MM-Vet, +1.1% on POPE, and +2.0% on VizWiz.With 64 tokens, it reaches 61.9% average performance, comparable to the MLP-based method at 62.0% and +1.3% over LDP-v2.

5 Conclusion and Limitation

TokenPacker condenses enriched high-resolution image features into compact visual tokens and maintains comparable or better performance with higher efficiency. However, performance clearly declines when compression reduces the representation to 32 tokens or fewer.

  • 5 Conclusion and Limitation: TokenPacker condenses enriched high-resolution image features into compact visual tokens through a coarse-to-fine design.The method is presented as a visual projector for MLLMs.
  • 5 Conclusion and Limitation: 75%∼89% visual-token reduction maintains comparable or even better performance with significantly higher efficiency in LLaVA-1.5.
  • 5 Conclusion and Limitation: At 32 tokens (6%) or fewer, a clear performance decline is evident.The authors identify this as a limitation and motivate more sophisticated projectors for very small token budgets.

Supplemental Material

The supplemental material provides additional experimental results, broader impacts, and asset licensing and consent information.

  • Supplemental Material: Additional experimental results are provided in supplemental section §A.
  • Supplemental Material: Broader impacts are covered in supplemental section §B.
  • Supplemental Material: Asset license and consent information is provided in supplemental section §C.

A.1 More Ablation study

The ablation study evaluates how single-level and multi-level visual features affect TokenPacker's injection module. Single-level features from the 23rd layer achieve the strongest reported average performance among the compared single-level methods.

  • A.1 More Ablation study: The experiments compare various single-level and multi-level feature combinations used in TokenPacker.
  • A.1 More Ablation study: TokenPacker's injection module uses multi-level visual features as high-resolution reference keys and values to enhance low-resolution queries.
  • A.1 More Ablation study: Single-level features from the 23rd layer yield superior average performance compared with other single-level methods.

A.2 Comparisons on Training Times

The training-time analysis compares TokenPacker with original LLaVA-1.5 under a shared evaluation environment. TokenPacker consistently requires shorter training times when using fewer visual tokens.

  • A.2 Comparisons on Training Times: The analysis uses eight NVIDIA A100 GPUs in the same environment, with accuracy averaged across six benchmarks.
  • A.2 Comparisons on Training Times: Training-time evaluation compares TokenPacker with original LLaVA-1.5 using the Vicuna-7B model.
  • A.2 Comparisons on Training Times: TokenPacker consistently requires shorter training times than LLaVA-1.5 with fewer visual tokens.

A.3 More Visual Results

TokenPacker is evaluated on diverse real-world visual understanding and reasoning scenarios, including specialized high-resolution tasks. The qualitative results cover document VQA, Math&Counting, OCR recognition, and related applications.

  • TokenPacker is tested across diverse visual understanding and reasoning tasks in practical real-world scenarios.The evaluation uses TokenPacker with a dynamic image slicing scheme for high-resolution images.
  • Figure A1 presents qualitative results across various visual understanding scenarios with the proposed approach.
  • The qualitative results include document VQA, Math&Counting, and OCR recognition.

B Broader Impacts

The paper positions TokenPacker as a potentially efficient component for real-world image or video understanding. It reports reduced training and inference costs with competitive performance, while warning that erroneous responses may create misinformation or safety risks.

  • TokenPacker has the potential to benefit efficient MLLMs for real-world image or video understanding.
  • The approach can clearly reduce training and inference costs while maintaining competitive performance.
  • Erroneous responses may raise misinformation or human-safety issues, motivating stringent security protocols for real-world multimodal applications.
Loading 2407.02392v4…