PaperlayerView exact revision on arXiv

Source-linked AI summary

Honeybee: Locality-enhanced Projector for Multimodal LLM

Junbum Cha, Wooyoung Kang, Jonghwan Mun, Byungseok Roh

arXiv:2312.06742v2cs.CVcs.AIcs.CLcs.LG

TL;DR

MLLM projectors must balance flexible visual-token counts, which affect efficiency, with local-context preservation, which supports spatial understanding. Honeybee introduces locality-enhanced abstractors and analyzes multifaceted instruction-data strategies; the resulting model reports state-of-the-art performance across multiple MLLM benchmarks.

  • Problem

    Projector design and multifaceted instruction-data handling remain underexplored despite their importance for MLLM performance, efficiency, and spatial understanding.

  • Method

    Honeybee uses locality-enhanced C-Abstractor and D-Abstractor projectors and studies dataset utilization, balancing, instructization, and template choices.

  • Results

    Honeybee achieves state-of-the-art performance across MME, MMBench, SEED-Bench, and LLaVA-Bench.

  • Takeaways & Limitations

    The paper identifies locality preservation and visual-token flexibility as complementary projector properties for balancing MLLM performance and efficiency.

Abstract

from arXiv · show

In Multimodal Large Language Models (MLLMs), a visual projector plays a crucial role in bridging pre-trained vision encoders with LLMs, enabling profound visual understanding while harnessing the LLMs' robust capabilities. Despite the importance of the visual projector, it has been relatively less explored. In this study, we first identify two essential projector properties: (i) flexibility in managing the number of visual tokens, crucial for MLLMs' overall efficiency, and (ii) preservation of local context from visual features, vital for spatial understanding. Based on these findings, we propose a novel projector design that is both flexible and locality-enhanced, effectively satisfying the two desirable properties. Additionally, we present comprehensive strategies to effectively utilize multiple and multifaceted instruction datasets. Through extensive experiments, we examine the impact of individual design choices. Finally, our proposed MLLM, Honeybee, remarkably outperforms previous state-of-the-art methods across various benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench, achieving significantly higher efficiency. Code and models are available at https://github.com/kakaobrain/honeybee.

1. Introduction

Honeybee addresses two underexplored projector requirements in MLLMs: flexible visual-token management for efficiency and local-context preservation for spatial understanding. It combines locality-enhanced projection with strategies for using multifaceted instruction data, achieving state-of-the-art results across several benchmarks.

  • Motivation: Projectors critically affect MLLM performance by conveying visual tokens and efficiency by determining how many tokens reach the language model.Most computational burden lies with the language model, so visual-token count strongly influences overall efficiency.
  • Motivation: Abstractors offer flexible visual-token counts but struggle with spatial understanding because abstraction can discard fine local details.Their locality-unaware attention tends to focus on a few regions.
  • Projector design: Honeybee introduces locality-enhanced projectors using convolution and deformable attention to balance local-context preservation with abstraction efficiency.The design targets both improved handling of intricate visual information and fewer tokens during LLM response generation.
  • Training recipe: The paper also studies how to utilize multifaceted instruction datasets and how to perform instructization effectively through extensive experiments.Instructization converts raw data into instruction-following format using predefined templates.
  • Results: Honeybee achieves state-of-the-art performance across MME, MMBench, SEED-Bench, and LLaVA-Bench.The paper reports significant gaps over previous state-of-the-art MLLMs.

2. Related Work

Prior MLLM progress has relied on visual instruction tuning and increasingly diverse multimodal capabilities, but projector trade-offs remain central. Linear projectors preserve local context, whereas abstractors provide token-count flexibility at the cost of locality.

  • MLLM development: Visual instruction tuning extends LLMs to visual tasks and supports broader multimodal inputs such as video, audio, 3D worlds, and point clouds.Recent work uses vision-language datasets and enhanced visual instruction-following data.
  • Benchmarks: MLLM benchmarks include objective evaluation with yes/no or multiple-choice questions and subjective evaluation for open-ended free-form responses.Objective benchmarks cover perceptual analysis and visual reasoning, while subjective benchmarks address more complex visual-language tasks.
  • Projector design: Linear projectors preserve local visual contexts through one-to-one transformation but are limited in flexibility, while abstractors reduce visual-token counts but may lose locality.This conceptual trade-off motivates locality-enhanced abstractors.
  • Research focus: The paper analyzes projector design and multifaceted-data handling to provide insights into training robust, high-performing MLLMs.These areas are characterized as underexplored or unclear.

3. Honeybee: Locality-enhanced MLLM

Honeybee uses a projector between the vision encoder and LLM to convert visual features into tokens while targeting both efficiency and spatial understanding. Its C-Abstractor and D-Abstractor preserve locality during flexible token abstraction, and training proceeds in two stages with multimodal instruction data.

  • Architecture: An MLLM combines a vision encoder, projector, and LLM, with the projector transferring region-level visual features into tokens for autoregressive response generation.The LLM processes fused visual and instruction tokens.
  • Efficiency: With the same LLM, computation, memory consumption, and throughput are mainly affected by the number of visual tokens fed into it.This makes token-count flexibility important for efficiency, especially with multiple or large images.
  • Motivation: Resamplers can abstract features flexibly but may overlook local regions, whereas linear projectors preserve locality but lack scalable token-count flexibility.The paper links preservation of all local contexts during abstraction to spatial-understanding performance.
  • C-Abstractor: C-Abstractor combines ResNet blocks with adaptive average pooling to abstract features into any squared number of visual tokens, including more than the original feature count.ResNet blocks outperform ConvNext and standard convolution blocks in the reported comparison.
  • D-Abstractor: D-Abstractor uses deformable attention, whose queries sample near uniformly distributed reference points using coordinates and offsets to preserve local and fine-grained information.This enhances resampler locality awareness while retaining flexibility.
  • Training: Honeybee training has two stages: projector-only training with the vision encoder and LLM frozen, followed by joint projector-and-LLM training.The first stage learns vision-language alignment, while the second enhances instruction following and visual understanding.

4. Hidden Recipe for Visual Instruction Tuning

The paper analyzes how multifaceted instruction data and instructization choices affect MLLM training, focusing on dataset selection, balancing, template design, and multi-turn construction.

  • Dataset combination: The study asks how individual datasets contribute to task performance and how diverse datasets should be balanced and combined.It evaluates dataset selection, utilization, and combination through systematic experiments.
  • Dataset balancing: Dataset balancing is crucial because available datasets differ substantially in size while training iterations remain limited.The study compares per-dataset, per-task, and sample-based balancing strategies.
  • Template granularity: The experiments compare fine-grained templates applied per dataset with coarse-grained templates shared within task categories.The goal is to determine the appropriate granularity for instruction formatting.
  • Template diversity: The study examines whether multiple templates and input inversion remain useful for GPT-assisted instruction data.It compares single-template, multi-template, and template-inversion settings.
  • Multi-turn template: Multi-turn construction concatenates multiple input-target pairs for one image, but semantic overlap can encourage shortcut learning.The paper therefore considers de-duplication when forming conversation-like examples.

5. Experiments

Experiments evaluate projector efficiency, spatial understanding, instruction-data choices, and comparisons with existing MLLMs across four benchmarks. Honeybee generally improves performance while using flexible visual-token budgets, with higher token counts helping detailed visual tasks.

  • Benchmarks: Four benchmarks evaluate perception, reasoning, and in-the-wild instruction following: MME, MMBench, SEED-Bench, and LLaVA-Bench.The reported splits include MMEP, MMBench-dev, and SEED-Bench Image-only.
  • Projector analysis: 56.3 versus 52.6 on the same M=256 computational budget shows C-Abstractor outperforming the linear projector on six spatial tasks.The comparison covers MME, MMBench, and SEED-Bench spatial understanding tasks.
  • Projector analysis: +0.9 point with M=144 shows C-Abstractor improving performance over a linear counterpart while using fewer visual tokens.The projector comparison also reports execution time in seconds per pre-training step.
  • Data mixtures: Dataset diversity within tasks and diversity across tasks both improve relevant benchmark performance, while removing existing vision-language data lowers MME, MMB, and SEEDI.The ablations connect VQA (Open) to MME, VQA (MC) to MMB and SEEDI, and captioning or instruction-following data to LLaVAW.
  • Instructization: Fine-grained templates consistently outperform coarse-grained templates across benchmarks, whereas greater template diversity does not guarantee improvement.Removing semantically overlapping pairs is reported as effective for mitigating shortcut training.
  • Comparison with existing MLLMs: Honeybee surpasses all previous methods at 13B and outperforms comparable 7B MLLMs on every benchmark except SEEDI.With M=256, Honeybee reaches 65.5 on SEEDI among 7B-scale MLLMs; the default 7B configuration uses M=144.

6. Conclusion

The paper identifies locality preservation and visual-token flexibility as overlooked projector properties, then combines locality-enhanced projection with instruction-data recipes in Honeybee. Honeybee outperforms prior state-of-the-art MLLMs across various benchmarks.

  • Conclusion: The proposed locality-enhanced projector balances performance through locality preservation with efficiency through abstraction capability.The paper also studies multifaceted instruction data and instructization choices through extensive experiments.
  • Conclusion: Honeybee achieves state-of-the-art performance across various MLLM benchmarks using the proposed projector and training recipes.The conclusion summarizes the combined contribution of projector design and instruction-tuning analysis.

A. Efficiency of MLLMs

This section explains that MLLM efficiency is driven mainly by the number of visual tokens and describes C-Abstractor and D-Abstractor as locality-preserving alternatives to inflexible projectors.

  • Efficiency: The number of visual tokens, rather than vision-model or projector parameter efficiency, predominantly affects MLLM efficiency.A resampler with 105M parameters and M=144 can be more efficient than a 4M-parameter linear projector with M=256.
  • Design principles: The design targets flexible visual-token counts and local-context preservation because flexibility affects efficiency and locality is critical for spatial understanding.A positional-embedding augmentation of the resampler did not yield notable improvements.
  • C-Abstractor: C-Abstractor combines convolution layers with adaptive average pooling, and a ResNet bottleneck block outperforms ConvNext and standard convolution blocks.These components provide the convolution-based locality-enhanced projector architecture.
  • D-Abstractor: D-Abstractor aggregates features at learned reference points using sampling offsets and attention weights over the visual feature map.Reference points, offsets, and weights are obtained through linear projections over learnable queries.
  • D-Abstractor: Adaptive-average-pooled query initialization and uniformly distributed reference points help cover the whole image region with fine-grained local information.The two techniques are called v-pooled Q and M-RP.

C. Implementation Details

Implementation uses manually balanced dataset sampling, dataset-specific templates, and multi-turn construction with de-duplication. Training uses six blocks across the two abstractors and substantial computational resources.

  • Hyperparameters: C-Abstractor uses L = 3 and D-Abstractor uses L = 6, totaling six blocks across the two projector variants.
  • Sampling ratio for datasets: COYO100M and BlipCapFilt are sampled at a 1:1 ratio during pre-training, while instruction-tuning ratios are manually specified per dataset.The instruction-tuning ratios are determined through short-schedule ablations.
  • Templates: Dataset templates adapt outputs to captioning, VQA, REC, VSR, and instruction tasks, with normalized coordinates used for bounding boxes.

D. Additional Recipes

Additional recipe studies evaluate training and representation choices, including indicator tokens, vision-feature layers, language-model backbones, LoRA, schedules, and dataset-specific formatting. The recipes also include multi-turn de-duplication and task-adapted templates.

  • Design choices: Omitting image indicator tokens improves performance, while second-last-layer CLIP features outperform last-layer features.
  • Design choices: Vicuna-v1.5 marginally outperforms LLaMA-2-chat, and full LLM tuning is superior to applying LoRA to every attention query and value layer.
  • Training schedule: Extending projector-only pre-training is beneficial with more computational resources, but the improvement is marginal.
  • Templates: Captioning templates request direct captions, while VQA and REC use fine-grained templates and VSR converts declarative captions into questions.
  • Evaluation: MME scores are reported for perception, cognition, and total categories, while MMBench reports accuracies across reasoning and perception subcategories.
  • Multi-turn with de-duplication: Multi-turn examples concatenate multiple input-target pairs for one image and remove duplicates sharing the same target.

F. Benchmark Characteristics

The benchmarks stress different capabilities: SEED-Bench emphasizes fine-grained visual comprehension, whereas MME-cognition is predominantly text-oriented and LLM-dependent. Honeybee is reported as efficient and effective across benchmark comparisons, including ScienceQA.

  • SEED-Bench: SEED-Bench frequently requires fine-grained understanding of details in small image regions, making larger images or more visual tokens important for performance.
  • SEED-Bench: Honeybee remains competitive on SEED-Bench even with smaller images or fewer visual tokens than comparative models.
  • MME-cognition: Three of four MME-cognition tasks are text-oriented reasoning tasks, so performance depends predominantly on the chosen LLM rather than visual comprehension.
  • Cross-benchmark comparison: Honeybee variants are compared with Qwen-VL-Chat and LLaVA-1.5 using AvgN, the normalized average of MMB, MMEP, and SEEDI.
  • ScienceQA: Honeybee with C-Abstractor and Vicuna-13B achieves new state-of-the-art ScienceQA scores without GPT-4 assistance or task-specific fine-tuning.
  • Locality visualization: The D-Abstractor attention map preserves local contexts more effectively, while the resampler concentrates visual information in fewer regions and loses details.

G.4. Additional Benchmark Results

Additional evaluation finds strong comprehensive visual understanding on MM-Vet and MMMU, while hallucination performance depends on model scale and image resolution.

  • Additional benchmarks: Honeybee shows superior comprehensive visual understanding on MM-Vet and MMMU, but 7B-scale Honeybee slightly trails LLaVA-1.5 on POPE.
  • Additional benchmarks: With 336-resolution images and a 13B LLM, Honeybee achieves competitive POPE performance.

H. Qualitative Analysis

Qualitative analyses show that locality-enhanced abstraction provides fine-grained image information, while instruction-following data improves responses to user requests.

  • Attention Comparison between Resampler and D-Abstractor: D-Abstractor locally abstracts visual features across the whole feature map, providing fine-grained and comprehensive image information.
  • Attention Comparison between Resampler and D-Abstractor: This locality-aware behavior benefits spatial understanding tasks that require capturing diverse relationships and objects in an image.
  • Qualitative Examples: Honeybee without instruction-following data captures image objects but tends to produce short answers that do not follow instructions.
  • Qualitative Examples: Honeybee with instruction-following data accurately responds to user instructions and generates detailed answers preferred by humans.
  • Qualitative Examples: Figure 10 presents generation results for varied user instructions using samples from OwlEval.
Loading 2312.06742v2…