PaperlayerView exact revision on arXiv

Source-linked AI summary

Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models

Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Dacheng Tao

arXiv:2408.15556v1cs.CV

TL;DR

MLLMs remain insufficiently evaluated and challenged on high-resolution images, where fixed-resolution processing can lose visual information. The paper introduces HR-Bench and the training-free DC2 framework, finding consistent improvements across HR and general multimodal benchmarks.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have experienced significant advancements recently, but still struggle to recognize and interpret intricate details in high-resolution (HR) images effectively. While state-of-the-art (SOTA) MLLMs claim to process images at 4K resolution, existing MLLM benchmarks only support up to 2K, leaving the capabilities of SOTA models on true HR images largely untested. Furthermore, existing methods for enhancing HR image perception in MLLMs rely on computationally expensive visual instruction tuning. To address these limitations, we introduce HR-Bench, the first deliberately designed benchmark to rigorously evaluate MLLM performance on 4K&8K images. Through extensive experiments, we demonstrate that while downsampling HR images leads to vision information loss, leveraging complementary modalities, e.g., text, can effectively compensate for this loss. Building upon this insight, we propose Divide, Conquer and Combine (DC$^2$), a novel training-free framework for enhancing MLLM perception of HR images. DC$^2$ follows a three-staged approach: 1) Divide: recursively partitioning the HR image into patches and merging similar patches to minimize computational overhead, 2) Conquer: leveraging the MLLM to generate accurate textual descriptions for each image patch, and 3) Combine: utilizing the generated text descriptions to enhance the MLLM's understanding of the overall HR image. Extensive experiments show that: 1) the SOTA MLLM achieves 63% accuracy, which is markedly lower than the 87% accuracy achieved by humans on HR-Bench; 2) our DC$^2$ brings consistent and significant improvements (a relative increase of +6% on HR-Bench and +8% on general multimodal benchmarks). The benchmark and code will be released to facilitate the multimodal R&D community.

Introduction

MLLMs face a gap between their ability to process high-resolution images and the resolutions used by existing benchmarks. The paper introduces HR-Bench and DC2 to evaluate and improve high-resolution perception without training.

Motivation

  • Most MLLMs perceive images at a fixed resolution, which can distort or blur high-resolution content.This limitation challenges perception across real-world images with varying resolutions.Source: Introduction · p. 1
  • Existing benchmarks reach only 2K, although advanced MLLMs can handle 4K images, leaving high-resolution perception insufficiently validated.The paper identifies a need for benchmarks at higher resolutions.Source: Introduction · pp. 1–2

Contributions

  • HR-Bench evaluates MLLM perception using manually annotated HR-Bench 8K images and object-centered HR-Bench 4K crops.The benchmark is designed to support systematic evaluation of high-resolution image perception.Source: Introduction · p. 2

Findings

  • Experiments show that downsampling loses visual information, while relevant text can mitigate the resulting degradation.The study evaluates models across multiple image resolutions and finds that complementary textual information helps recover lost information.Source: Introduction · p. 2

Contributions

  • DC2 is a training-free framework that recursively divides images, generates patch descriptions, and combines them to improve MLLM perception.Similar patches are merged to reduce computation, and the framework uses textual descriptions to support overall image understanding.Source: Introduction · p. 2

Preliminaries and Related Work

MLLMs connect visual encoders and language models, while prior high-resolution strategies rely on cropping, specialized visual encoders, or visual search. Existing evaluation remains limited because benchmark images are generally no larger than 2K.

MLLM Architecture

  • MLLMs typically use a visual encoder, a multimodal connector, and an LLM that autoregressively decodes text.The visual encoder and LLM are usually initialized from pretrained models.Source: Preliminaries and Related Work · p. 2

Related Work

  • Cropping-based methods partition images into patches, encode them separately, and concatenate the resulting representations for LLM processing.LLaVA-NeXT and InternVL-v1.5 are cited as representative examples.Source: Preliminaries and Related Work · p. 2
  • HR visual encoder methods use architectures such as SAM or ConvNeXt to extract visual features for high-resolution understanding.These methods aim to improve high-resolution perception without substantially increasing visual-token counts.Source: Preliminaries and Related Work · p. 2
  • Visual search methods actively seek necessary visual information, especially for vision-intensive tasks involving high-resolution images.SEAL is presented as a meta-architecture inspired by human visual search.Source: Preliminaries and Related Work · p. 2

Evaluation Gap

How does Image Resolution Affect MLLMs? HR-Bench

HR-Bench tests fine-grained object attributes and spatial relations at 4K and 8K resolutions. Experiments show that downsampling increases visual information loss, uncertainty, and errors, while relevant text can compensate for the missing visual information.

HR-Bench Curation

  • HR-Bench contains Fine-grained Single-instance Perception and Fine-grained Cross-instance Perception tasks with 100 samples each.The tasks assess object attributes such as color and material, and relative positions between objects.Source: HR-Bench · p. 3
  • The benchmark uses human-selected images and questions, with 8K images cropped around queried objects to produce 4K versions.The 8K and 4K variants are called HR-Bench 8K and HR-Bench 4K.Source: HR-Bench · p. 3

Evaluation Protocol

  • Cyclic Permutation evaluates each multiple-choice question under every option ordering to reduce sensitivity to answer order.This protocol is used for quantitative comparison across MLLMs.Source: HR-Bench · p. 3

Resolution Effects

  • Downsampling higher-resolution images to a fixed resolution increases visual information loss, model uncertainty, and output errors.Across four models, accuracy decreases and uncertainty increases as image resolution grows before resizing; resizing to 336 causes blur and loss of detail.Source: Pilot Experiments · p. 4

Language Compensation

  • Relevant textual information can compensate for visual information lost through downsampling.Rich descriptions of object attributes and relative positions improve HR-Bench 8K performance and can approach the benefit of preserving key image regions.Source: Pilot Experiments · p. 4

Methodology

DC2 improves high-resolution image perception by recursively dividing images, generating patch-level descriptions, and combining retrieved textual information with the user prompt. Similar patches are merged for efficiency, while filtering and visual memory help preserve object integrity and limit unnecessary input text.

Divide

  • DC2 recursively divides an image into patches at the pretrained visual encoder’s resolution, merging similar patches to reduce computational overhead.The Divide stage recursively splits images into four parts until reaching resolutions such as 336 × 336, then uses hierarchical clustering to merge similar patches.Source: Methodology · p. 4

Evaluation setup

  • Figure 3 evaluates how rich text descriptions and key image regions affect model performance.Here, “T” denotes text descriptions and “P” denotes key image regions.Source: Figure 3 · p. 4

Conquer

  • The Conquer stage uses the MLLM to generate text descriptions and extract object information from image patches.Leaf nodes are described directly, while non-leaf nodes use descriptions from their child patches.Source: Methodology · p. 5

Combine

  • DC2 filters hallucinated objects by retaining objects found across successive recursive layers.The method treats the intersection of object sets from adjacent layers as the set of actually existing objects.Source: Methodology · p. 5
  • Visual memory stores object-linked patch coordinates so inference can retrieve only image patches relevant to the user prompt.Retrieved patch descriptions are concatenated with the prompt before the MLLM generates the final response.Source: Methodology · pp. 4–5, Inference Details · p. 6

Experiments

DC2 consistently improves performance on HR-Bench 8K and broader multimodal benchmarks, while ablations and recursion analyses identify efficiency and object-localization trade-offs.

Overall performance

  • 61.4% was the best open-source accuracy on HR-Bench 8K, while Gemini 1.5 Flash reached 62.8%.GPT4o and QWen-VL-max achieved 55.5% and 52.5%, respectively.Source: Overall performance · p. 6

HR-Bench 8K

General multimodal benchmarks

Ablation study

  • 10.2% performance decline followed omission of recursive cropping, while removing the filter caused a 4.6% decrease.The merge module reduced performance by 0.5% but improved inference efficiency by reducing image patches; excluding visual memory caused a 2.6% drop.Source: Ablation Study · p. 6

Recursion analysis

Mechanism analysis

  • More accurate object localization corresponded to higher HR-Bench 4K accuracy, supporting DC2’s text-based compensation for missing visual information.Recall@2 and mIoU were used to evaluate object retrieval and predicted-box overlap.Source: When and Why Does Our Method Work? · p. 7

Conclusion

The paper introduces HR-Bench and DC2, evaluates high-resolution perception with a semi-automatic benchmark pipeline, and concludes that text can compensate for visual information lost during resizing.

Contributions

Evaluation

  • 28 open-source and commercial models are systematically evaluated on HR-Bench.The benchmark is intended to assess MLLM perception of high-resolution images.Source: Conclusion · p. 7

Conclusion

  • Current MLLMs lose visual information when HR images are resized to low resolution, but text information can compensate for that loss.The conclusion also reports that DC2 improves MLLMs’ high-resolution perception.Source: Conclusion · p. 7

Benchmark construction

  • 200 image queries remain after manually annotating objects, generating query-answer pairs, and filtering incorrect or ambiguous examples with human experts.The curation procedure combines bounding-box annotation, GPT-4o generation, and expert review.Source: Curation Procedure · p. 11

Benchmark Statistics

HR-Bench covers fine-grained single-instance and cross-instance perception, addressing the lack of existing benchmarks at 4K and above.

FSP

  • Fine-grained Single-instance Perception covers attribute recognition, OCR, and visual prompting.It is one of HR-Bench’s two sub-tasks.Source: Benchmark Statistics · p. 11

FCP

  • Fine-grained Cross-instance Perception covers map analysis, chart analysis, and spatial relationship assessment.It is the second HR-Bench sub-task.Source: Benchmark Statistics · p. 11

Resolution coverage

  • 2K was the highest resolution among 21 existing MLLM benchmarks, leaving 4K and higher resolutions underrepresented.The comparison also found few benchmarks involving visual prompting and map analysis.Source: Benchmark Statistics · p. 11

Benchmark Evaluation

The benchmark evaluation uses cyclic permutation and multimodal gain and leakage metrics to make model comparisons more robust and assess whether performance reflects multimodal training.

Evaluation protocol

  • n computations are required per multiple-choice sample under cyclic permutation, reducing sensitivity to option order.The method circularly shifts the positions of n answer options.Source: Benchmark Evaluation · p. 11

Evaluation metrics

Results

  • 3.6 was the multi-modal leakage reported for HR-Bench 8K, providing a comparison arena for existing MLLMs.The paper attributes the relatively low multi-modal gain on HR-Bench 8K to low visual-input accuracy.Source: Multi-modal Gain & Multi-modal Leakage · p. 11

B. Implementation Details of DC2

DC2 builds a visual memory from recursively divided HR-image patches, generating text descriptions and retrieving relevant patch information to answer user questions.

Conquer

Combine

Inference

Divide

  • DC2 recursively divides HR images into patches until reaching the visual encoder’s resolution, then merges similar patches to reduce computation.The pipeline uses hierarchical clustering to form groups of similar patches before further processing.Source: Inference pipeline of DC2 · pp. 11–12

Experiment Settings

Experiments use a shared VLMEvalKit-based codebase, repeated trials, and fixed decoding and retrieval settings to improve robustness and comparability.

Hardware and software

  • Experiments run in a VLMEvalKit-based codebase on a PC with an Intel Xeon Platinum 8358P CPU and 8 NVIDIA Tesla A100 80GB GPUs.The hardware configuration includes an Intel(R) Xeon(R) Platinum 8358P at 2.60GHz.Source: Experiment Settings · p. 12

Experimental protocol

  • Each experiment is conducted five times, using temperature 0.2 to reduce variance caused by differences in generated text descriptions.The default thresholds are θ = 0.1 and α = 0.3 unless otherwise stated.Source: Experiment Settings · p. 12

Fully Experiment Results on HR-Bench

HR-Bench reveals substantial weaknesses in current MLLMs on high-resolution perception, while DC2 provides consistent improvements across benchmark settings and prompt configurations.

Overall HR-Bench results

  • 71.0% and 62.8% accuracy are achieved by SOTA MLLMs on HR-Bench 4K and 8K, versus human accuracy of 82.0% and 86.8%.DC2 consistently enhances performance on both HR-Bench 4K and HR-Bench 8K.Source: Fully Experiment Results on HR-Bench · p. 12

Fine-grained perception

  • 42.4% FSP accuracy and 38.8% FCP accuracy show weakness in fine-grained cross-instance perception, with more patches reducing FCP performance for cropping-based methods.The reported averages are computed across 28 MLLMs.Source: Fully Experiment Results on HR-Bench · p. 12

Chart analysis

  • 50% is the best accuracy on high-density chart tasks, achieved by GPT4o, while other MLLMs perform nearly at random-guessing levels.The chart tasks use varying densities and require calculations such as summation and averaging.Source: Fully Experiment Results on HR-Bench · p. 12

Retriever threshold

  • DC2 provides consistent improvements across different retriever thresholds α.The threshold study varies α during inference to assess retrieval sensitivity.Source: Effect of α · p. 13, Figure 6 · p. 13

Model scaling

  • Performance increases as MLLM size increases when scaling DC2 across V∗Bench and HR-Bench 8K.Figure 7 reports scaling results for both benchmarks.Source: Scaling DC2 · p. 13, Figure 7 · p. 14

Prompt effects

Qualitative analysis

Related papers

Loading 2408.15556v1…