PaperlayerView exact revision on arXiv

Source-linked AI summary

Matryoshka Multimodal Models

Mu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae Lee

arXiv:2405.17430v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Fixed visual-token counts make LMMs inefficient on high-resolution images and long videos, while existing reduction methods lack flexible length control. M3 learns nested coarse-to-fine token sets and trains the model across scales, enabling per-instance granularity selection. Its oracle uses 8.9 tokens on average while outperforming LLaVA-NeXT by 8% points on six common benchmarks, and the authors identify a remaining gap to practical scale prediction.

  • Problem

    Fixed large visual-token sequences are inefficient for dense images and videos, while pruning and merging methods produce only one output length and limit information–efficiency trade-offs.

  • Method

    M3 learns nested coarse-to-fine visual-token sets and trains the CLIP encoder and ensuing LLM with prediction losses averaged across scales.

  • Results

    8.9 visual tokens on average achieved performance 8% points better than LLaVA-NeXT using 576 tokens per image grid under six common LMM benchmarks.

  • Takeaways & Limitations

    M3 enables explicit per-instance visual-granularity control and supports analyzing how many tokens different benchmarks require.

  • Takeaways & Limitations

    The approach lacks an effective visual-token predictor that bridges the oracle and the model’s actual performance at a selected scale.

Abstract

from arXiv · show

Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a Large Language Model (LLM). However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning/merging methods do exist, they produce a single length output for each image and do not afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M3 provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around ~9 visual tokens to obtain accuracy similar to that of using all 576 tokens; (3) Our approach provides a foundation to explore the best trade-off between performance and visual token length at sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.

1 Introduction

M3 addresses the inefficiency of fixed-length visual-token representations by learning nested token sets that support controllable visual granularity. It enables per-instance efficiency trade-offs and reveals that many benchmarks need far fewer tokens than standard representations.

  • Motivation and approach: Fixed visual-token counts become prohibitively expensive for high-resolution images and long videos, while excessive tokens can distract the model from relevant information.Existing pruning and merging methods reduce sequence length but produce only a single output length, limiting efficiency–information trade-offs.
  • Motivation and approach: The model’s nested representation changes granularity from coarse to fine by deriving coarser visual-token sets from finer ones.Figure 1 illustrates this controllable progression using an MSCOCO validation image.
  • Motivation and approach: M3 learns nested visual-token sets that let users control image or video granularity during inference within one model.The approach supports using more tokens for dense content and fewer tokens for simpler content or long visual sequences.
  • Key findings: ∼9 tokens per image suffice for most natural-scene benchmarks such as COCO, whereas document understanding and OCR require 144–576 tokens.M3 therefore provides a framework for measuring the visual granularity required by different datasets.
  • Key findings: 8.9 visual tokens on average yielded performance 8% points better than LLaVA-NeXT using 576 tokens per image grid under six common LMM benchmarks.This oracle result exposes a substantial gap between sample-level token selection and current fixed-scale representations.

2 Related Work

Related work establishes LMMs, flexible Matryoshka representations, and token-reduction methods as foundations for M3. Unlike prior reduction approaches that return one sequence length, M3 targets controllable multi-scale visual representations.

  • Large Multimodal Models: LMMs extend language-model reasoning with visual inputs, while LLaVA-style systems feed fixed visual-token sequences into an LLM.This design is also used in video LMMs, where each frame contributes a fixed number of tokens.
  • Matryoshka Representation Learning: Matryoshka Representation Learning encodes multiple granularities within one feature vector to support tasks with different computational-resource requirements.M3 transfers the nested-representation idea to visual-token sequences rather than feature dimensions.
  • Token Reduction: Token reduction methods reduce Transformer costs through sparse attention, token merging, or other sequence-shortening strategies.Their central limitation for M3 is that prior approaches produce a single-length output per input.
  • Large Multimodal Models: M3’s architecture groups CLIP visual features into coarse-to-fine token sets whose granularity users can explicitly select at test time.This design connects flexible representation learning with the operational needs of LMM inference.
  • Token Reduction: A concurrent method also represents images with varying token counts under one set of weights, but M3 preserves spatial structure through average pooling.The comparison identifies a representational distinction rather than a difference in the overall multi-scale goal.

3 M3: Matryoshka Multimodal Models

M3 trains an LMM to represent each image with nested coarse-to-fine visual-token sets and optimizes prediction across all scales. The resulting model can answer questions at user-selected granularities while progressively adding visual detail.

  • Nested visual representation: M3 enforces nested token sets X_Si ⊂ X_Si+1 so visual content is represented from coarse to fine without adding learnable LMM parameters.The CLIP visual encoder learns the nested representation, and the ensuing LLM adapts to its token distribution.
  • Nested visual representation: The CLIP-ViT-L-336 encoder provides 576 tokens in a 24×24 grid, which are pooled into scales including 144, 36, 9, and 1 token.Coarser levels are derived through sequential 2×2 pooling followed by 3×3 pooling.
  • Training objective: Training averages the autoregressive next-token prediction loss across the M visual-token scales for each image.At each scale, the model maximizes the likelihood of the ground-truth answer conditioned on the visual representation and question.
  • Training objective: The trainable parameters include both the CLIP visual encoder and the ensuing LLM, with answer-token prediction conditioned on the question and prior answer tokens.The objective therefore adapts both visual representations and language-model behavior to the nested token structure.
  • Inference flexibility: Increasing scale adds visual detail, enabling visual question answering at any trained granularity and allowing resource-constrained inference to adjust token use.Smaller sets capture the scene broadly, while larger sets can preserve details such as a Pepsi cup.

4 Experiments

Across image and video benchmarks, M3 preserves or improves performance while enabling substantially fewer visual tokens and revealing task- and sample-level granularity differences.

  • 4.2 Image Understanding: M3 matches full-token LLaVA-1.5 performance across diverse benchmarks, while 9 tokens surpass Qwen-VL-Chat with 256 tokens on MMBench.With 9 tokens, M3 also surpasses InstructBLIP-7B and InstructBLIP-13B across all evaluated benchmarks.
  • 4.2 Image Understanding: On LLaVA-NeXT, M3 is at least as good as the specific-scale baseline and performs better on TextVQA, ChartQA, and MMBench.The oracle denotes selecting the best token-performance trade-off separately for each test instance.
  • 4.4 In-depth Analysis: Token-scale sensitivity varies by task: ScienceQA is stable, AI2D and MMBench change little, while TextVQA and DocVQA drop substantially with fewer tokens.This establishes M3 as a framework for analyzing the granularity required by different benchmarks.
  • 4.3 Video Understanding: For most video tasks, 45 tokens total yields accuracy within 1% of 2,880 full tokens, while full tokens are worse than 720 or 180 tokens on 4 of 6 benchmarks.The results suggest that many video benchmarks require sparse visual information, whereas very long contexts may distract predictions.
  • 4.4 In-depth Analysis: M3 is more resilient than average pooling, spatial sampling, and sequential sampling as the visual token count decreases.The comparison is conducted at inference time on MMBench with the LLaVA-NeXT architecture.
  • 4.4 In-depth Analysis: Across TextVQA samples, correctness changes with visual-token scale, allowing M3 to act as a sample-level image-complexity metric.The OCR behavior aligns with image complexity across scales.
  • 4.4 In-depth Analysis: At least 10% better performance than full visual tokens is achieved by the oracle upper bound using only 6–64 tokens.The oracle selects the fewest token scale that answers each test instance correctly, exposing a large gap from fixed full-token representations.
  • 4.5 Ablations: Training the whole LLM performs better than finetuning without it, indicating adaptation to Matryoshka-distributed visual-token patterns.This ablation is reported in Table 7.

5 Conclusion and Future Work

M3 represents visual content as nested token sets across coarse-to-fine granularities, enabling per-instance control of visual granularity. The paper also identifies a large gap between oracle performance and current model performance, while noting the absence of an effective visual token predictor.

  • M3 represents visual content as nested sets of visual tokens across multiple coarse-to-fine granularities.
  • M3 enables explicit control of visual granularity for each test instance during inference.
  • A large gap remains between the oracle upper bound and the model’s actual performance.
  • Future Work: The current approach lacks an effective visual token predictor to bridge the oracle–model gap at a specific scale.

A Broader Impact

M3 inherits the broader deployment concerns of the LLaVA models whose architecture and data it uses. In particular, bias from GPT-4- and GPT-4V-generated instruction-tuning data remains relevant.

  • M3 is trained with the same architecture and data as LLaVA-1.5 and LLaVA-NeXT, so their broader-impact concerns also apply.
  • Bias from GPT-4- and GPT-4V-generated data remains a concern for M3’s instruction tuning.

B Efficiency Analysis

M3 reduces visual token counts in LMM processing, lowering theoretical prefill computation and memory requirements. Its token reduction can also complement other acceleration methods such as quantization and factorization.

  • M3’s nested visual tokens substantially reduce token counts, lowering LMM prefill FLOPs and computational memory requirements.The analysis considers a 336 × 336 image encoded into 576 tokens with a 30-token text prompt.
  • M3’s token reduction can complement other LMM acceleration methods, including quantization and factorization.

C More Visualizations on Nested Visual Representation

Visualizations show that increasing the number of visual tokens lets LMMs discover more image details and produce higher-quality descriptions. The examples include MSCOCO validation images and OCR-related descriptions.

  • More visual tokens let LMMs discover additional details, including furniture and human attributes.
  • More visual tokens support higher-quality descriptions, including improved OCR-related descriptions.
  • Figure 6 provides visualization examples from the MSCOCO validation set.
Loading 2405.17430v2…