PaperlayerView exact revision on arXiv

Source-linked AI summary

Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models

Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Dacheng Tao

arXiv:2408.15556v1cs.CV

TL;DR

MLLMs remain insufficiently evaluated and challenged on high-resolution images, where fixed-resolution processing can lose visual information. The paper introduces HR-Bench and the training-free DC2 framework, finding consistent improvements across HR and general multimodal benchmarks.

  • Problem

    MLLMs often process images at fixed resolution, causing distortion and information loss, while their ability to perceive 4K and 8K images lacks rigorous benchmark evaluation.

  • Method

    The paper introduces HR-Bench for systematic 4K and 8K evaluation and DC2, which divides images into patches, generates patch descriptions, and combines them to enhance perception.

  • Results

    DC2 produces consistent improvements on HR-Bench and general multimodal benchmarks, including up to 12% accuracy improvement on the 2K-resolution V* benchmark.

  • Takeaways & Limitations

    Textual information can compensate for visual information lost when high-resolution images are resized, and DC2 improves MLLM perception without training.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have experienced significant advancements recently, but still struggle to recognize and interpret intricate details in high-resolution (HR) images effectively. While state-of-the-art (SOTA) MLLMs claim to process images at 4K resolution, existing MLLM benchmarks only support up to 2K, leaving the capabilities of SOTA models on true HR images largely untested. Furthermore, existing methods for enhancing HR image perception in MLLMs rely on computationally expensive visual instruction tuning. To address these limitations, we introduce HR-Bench, the first deliberately designed benchmark to rigorously evaluate MLLM performance on 4K&8K images. Through extensive experiments, we demonstrate that while downsampling HR images leads to vision information loss, leveraging complementary modalities, e.g., text, can effectively compensate for this loss. Building upon this insight, we propose Divide, Conquer and Combine (DC$^2$), a novel training-free framework for enhancing MLLM perception of HR images. DC$^2$ follows a three-staged approach: 1) Divide: recursively partitioning the HR image into patches and merging similar patches to minimize computational overhead, 2) Conquer: leveraging the MLLM to generate accurate textual descriptions for each image patch, and 3) Combine: utilizing the generated text descriptions to enhance the MLLM's understanding of the overall HR image. Extensive experiments show that: 1) the SOTA MLLM achieves 63% accuracy, which is markedly lower than the 87% accuracy achieved by humans on HR-Bench; 2) our DC$^2$ brings consistent and significant improvements (a relative increase of +6% on HR-Bench and +8% on general multimodal benchmarks). The benchmark and code will be released to facilitate the multimodal R&D community.

Introduction

MLLMs face a gap between their ability to process high-resolution images and the resolutions used by existing benchmarks. The paper introduces HR-Bench and DC2 to evaluate and improve high-resolution perception without training.

  • Motivation: Most MLLMs perceive images at a fixed resolution, which can distort or blur high-resolution content.This limitation challenges perception across real-world images with varying resolutions.
  • Motivation: Existing benchmarks reach only 2K, although advanced MLLMs can handle 4K images, leaving high-resolution perception insufficiently validated.The paper identifies a need for benchmarks at higher resolutions.
  • Contributions: HR-Bench evaluates MLLM perception using manually annotated HR-Bench 8K images and object-centered HR-Bench 4K crops.The benchmark is designed to support systematic evaluation of high-resolution image perception.
  • Findings: Experiments show that downsampling loses visual information, while relevant text can mitigate the resulting degradation.The study evaluates models across multiple image resolutions and finds that complementary textual information helps recover lost information.
  • Contributions: DC2 is a training-free framework that recursively divides images, generates patch descriptions, and combines them to improve MLLM perception.Similar patches are merged to reduce computation, and the framework uses textual descriptions to support overall image understanding.

Preliminaries and Related Work

MLLMs connect visual encoders and language models, while prior high-resolution strategies rely on cropping, specialized visual encoders, or visual search. Existing evaluation remains limited because benchmark images are generally no larger than 2K.

  • MLLM Architecture: MLLMs typically use a visual encoder, a multimodal connector, and an LLM that autoregressively decodes text.The visual encoder and LLM are usually initialized from pretrained models.
  • Related Work: Cropping-based methods partition images into patches, encode them separately, and concatenate the resulting representations for LLM processing.LLaVA-NeXT and InternVL-v1.5 are cited as representative examples.
  • Related Work: HR visual encoder methods use architectures such as SAM or ConvNeXt to extract visual features for high-resolution understanding.These methods aim to improve high-resolution perception without substantially increasing visual-token counts.
  • Related Work: Visual search methods actively seek necessary visual information, especially for vision-intensive tasks involving high-resolution images.SEAL is presented as a meta-architecture inspired by human visual search.
  • Evaluation Gap: Current multimodal benchmarks remain capped at 2K despite newer MLLMs handling 4K images.This mismatch motivates more rigorous high-resolution evaluation.

How does Image Resolution Affect MLLMs? HR-Bench

HR-Bench tests fine-grained object attributes and spatial relations at 4K and 8K resolutions. Experiments show that downsampling increases visual information loss, uncertainty, and errors, while relevant text can compensate for the missing visual information.

  • HR-Bench Curation: HR-Bench contains Fine-grained Single-instance Perception and Fine-grained Cross-instance Perception tasks with 100 samples each.The tasks assess object attributes such as color and material, and relative positions between objects.
  • HR-Bench Curation: The benchmark uses human-selected images and questions, with 8K images cropped around queried objects to produce 4K versions.The 8K and 4K variants are called HR-Bench 8K and HR-Bench 4K.
  • Evaluation Protocol: Cyclic Permutation evaluates each multiple-choice question under every option ordering to reduce sensitivity to answer order.This protocol is used for quantitative comparison across MLLMs.
  • Resolution Effects: Downsampling higher-resolution images to a fixed resolution increases visual information loss, model uncertainty, and output errors.Across four models, accuracy decreases and uncertainty increases as image resolution grows before resizing; resizing to 336 causes blur and loss of detail.
  • Language Compensation: Relevant textual information can compensate for visual information lost through downsampling.Rich descriptions of object attributes and relative positions improve HR-Bench 8K performance and can approach the benefit of preserving key image regions.

Methodology

DC2 improves high-resolution image perception by recursively dividing images, generating patch-level descriptions, and combining retrieved textual information with the user prompt. Similar patches are merged for efficiency, while filtering and visual memory help preserve object integrity and limit unnecessary input text.

  • Divide: DC2 recursively divides an image into patches at the pretrained visual encoder’s resolution, merging similar patches to reduce computational overhead.The Divide stage recursively splits images into four parts until reaching resolutions such as 336 × 336, then uses hierarchical clustering to merge similar patches.
  • Evaluation setup: Figure 3 evaluates how rich text descriptions and key image regions affect model performance.Here, “T” denotes text descriptions and “P” denotes key image regions.
  • Conquer: The Conquer stage uses the MLLM to generate text descriptions and extract object information from image patches.Leaf nodes are described directly, while non-leaf nodes use descriptions from their child patches.
  • Combine: DC2 filters hallucinated objects by retaining objects found across successive recursive layers.The method treats the intersection of object sets from adjacent layers as the set of actually existing objects.
  • Combine: Visual memory stores object-linked patch coordinates so inference can retrieve only image patches relevant to the user prompt.Retrieved patch descriptions are concatenated with the prompt before the MLLM generates the final response.

Experiments

DC2 consistently improves performance on HR-Bench 8K and broader multimodal benchmarks, while ablations and recursion analyses identify efficiency and object-localization trade-offs.

  • Overall performance: 61.4% was the best open-source accuracy on HR-Bench 8K, while Gemini 1.5 Flash reached 62.8%.GPT4o and QWen-VL-max achieved 55.5% and 52.5%, respectively.
  • HR-Bench 8K: 5.7% and 3.0% were DC2’s maximum accuracy improvements on HR-Bench 8K for FSP and FCP, respectively.InternVL-v1.5 with DC2 reached 75.0% on FSP, surpassing Gemini 1.5 Flash.
  • General multimodal benchmarks: Up to 12% accuracy improvement was observed on the 2K V∗ benchmark, alongside significant gains on POPE and MME.These results extend DC2’s evaluation beyond HR images to general multimodal benchmarks.
  • Ablation study: 10.2% performance decline followed omission of recursive cropping, while removing the filter caused a 4.6% decrease.The merge module reduced performance by 0.5% but improved inference efficiency by reducing image patches; excluding visual memory caused a 2.6% drop.
  • Recursion analysis: Increasing recursion layers significantly improved accuracy, but gains gradually slowed as layers increased.More patches reduced visual information loss for FSP, whereas FCP slightly declined with deeper recursion.
  • Mechanism analysis: More accurate object localization corresponded to higher HR-Bench 4K accuracy, supporting DC2’s text-based compensation for missing visual information.Recall@2 and mIoU were used to evaluate object retrieval and predicted-box overlap.

Conclusion

The paper introduces HR-Bench and DC2, evaluates high-resolution perception with a semi-automatic benchmark pipeline, and concludes that text can compensate for visual information lost during resizing.

  • Contributions: HR-Bench and DC2 are proposed as an 8K image benchmark and a training-free framework for high-resolution perception.DC2 denotes Divide, Conquer and Combine.
  • Evaluation: 28 open-source and commercial models are systematically evaluated on HR-Bench.The benchmark is intended to assess MLLM perception of high-resolution images.
  • Conclusion: Current MLLMs lose visual information when HR images are resized to low resolution, but text information can compensate for that loss.The conclusion also reports that DC2 improves MLLMs’ high-resolution perception.
  • Benchmark construction: 200 image queries remain after manually annotating objects, generating query-answer pairs, and filtering incorrect or ambiguous examples with human experts.The curation procedure combines bounding-box annotation, GPT-4o generation, and expert review.

Benchmark Statistics

HR-Bench covers fine-grained single-instance and cross-instance perception, addressing the lack of existing benchmarks at 4K and above.

  • FSP: Fine-grained Single-instance Perception covers attribute recognition, OCR, and visual prompting.It is one of HR-Bench’s two sub-tasks.
  • FCP: Fine-grained Cross-instance Perception covers map analysis, chart analysis, and spatial relationship assessment.It is the second HR-Bench sub-task.
  • Resolution coverage: 2K was the highest resolution among 21 existing MLLM benchmarks, leaving 4K and higher resolutions underrepresented.The comparison also found few benchmarks involving visual prompting and map analysis.

Benchmark Evaluation

The benchmark evaluation uses cyclic permutation and multimodal gain and leakage metrics to make model comparisons more robust and assess whether performance reflects multimodal training.

  • Evaluation protocol: n computations are required per multiple-choice sample under cyclic permutation, reducing sensitivity to option order.The method circularly shifts the positions of n answer options.
  • Evaluation metrics: Multi-modal gain compares accuracy with visual inputs, Sv, against accuracy without visual inputs, Swv.The paper defines MG as MG = Sv − Swv.
  • Evaluation metrics: Multi-modal leakage uses the LLM-base accuracy, St, to quantify leakage from multimodal training.The paper defines ML as ML = max(0, Swv − St).
  • Results: 3.6 was the multi-modal leakage reported for HR-Bench 8K, providing a comparison arena for existing MLLMs.The paper attributes the relatively low multi-modal gain on HR-Bench 8K to low visual-input accuracy.

B. Implementation Details of DC2

DC2 builds a visual memory from recursively divided HR-image patches, generating text descriptions and retrieving relevant patch information to answer user questions.

  • Conquer: The conquer stage uses MLLMs to generate text descriptions and extract objects from image patches with leaf and non-leaf prompts.The implementation distinguishes Fleaf and Fnon−leaf prompts for generating descriptions and extracting objects.
  • Combine: DC2 stores detected objects and their coordinates in visual memory, then retrieves objects whose confidence exceeds α for a user question.Retrieved image patches are used to generate descriptions before the final response is produced.
  • Inference: The inference pipeline filters uncertain objects, retrieves relevant patches, and combines their generated descriptions with the user prompt for final answering.The final response is generated by the MLLM from the user question, retrieved patches, and their text descriptions.
  • Divide: DC2 recursively divides HR images into patches until reaching the visual encoder’s resolution, then merges similar patches to reduce computation.The pipeline uses hierarchical clustering to form groups of similar patches before further processing.

Experiment Settings

Experiments use a shared VLMEvalKit-based codebase, repeated trials, and fixed decoding and retrieval settings to improve robustness and comparability.

  • Hardware and software: Experiments run in a VLMEvalKit-based codebase on a PC with an Intel Xeon Platinum 8358P CPU and 8 NVIDIA Tesla A100 80GB GPUs.The hardware configuration includes an Intel(R) Xeon(R) Platinum 8358P at 2.60GHz.
  • Experimental protocol: Each experiment is conducted five times, using temperature 0.2 to reduce variance caused by differences in generated text descriptions.The default thresholds are θ = 0.1 and α = 0.3 unless otherwise stated.

Fully Experiment Results on HR-Bench

HR-Bench reveals substantial weaknesses in current MLLMs on high-resolution perception, while DC2 provides consistent improvements across benchmark settings and prompt configurations.

  • Overall HR-Bench results: 71.0% and 62.8% accuracy are achieved by SOTA MLLMs on HR-Bench 4K and 8K, versus human accuracy of 82.0% and 86.8%.DC2 consistently enhances performance on both HR-Bench 4K and HR-Bench 8K.
  • Fine-grained perception: 42.4% FSP accuracy and 38.8% FCP accuracy show weakness in fine-grained cross-instance perception, with more patches reducing FCP performance for cropping-based methods.The reported averages are computed across 28 MLLMs.
  • Chart analysis: 50% is the best accuracy on high-density chart tasks, achieved by GPT4o, while other MLLMs perform nearly at random-guessing levels.The chart tasks use varying densities and require calculations such as summation and averaging.
  • Retriever threshold: DC2 provides consistent improvements across different retriever thresholds α.The threshold study varies α during inference to assess retrieval sensitivity.
  • Model scaling: Performance increases as MLLM size increases when scaling DC2 across V∗Bench and HR-Bench 8K.Figure 7 reports scaling results for both benchmarks.
  • Prompt effects: Prompt performance varies minimally, and the shortest prompt substantially improves HR-Bench 4K while enabling faster inference.The concise prompt is selected as the default because its descriptions are significantly shorter.
  • Qualitative analysis: Qualitative examples show that SOTA MLLMs miss fine-grained HR-image objects, whereas DC2 supplies patch descriptions that help existing MLLMs perceive them.The examples cover FSP and FCP subtasks of HR-Bench 8K.
Loading 2408.15556v1…