Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

1. Introduction

• LLaVA-HR outperforms existing MLLMs on 8 of 11 VL tasks, including +9.6% over LLaVA-1.5 on TextVQA.Pre-training and instruction tuning take 20.7 hours on 8 A800 GPUs, and inference is 3 times faster than LLaVA-1.5 at the same resolution.Source: Figure 1 · p. 1, §1 · p. 2
• Existing MLLMs fall short in granular visual recognition, including hallucinations when GPT4-V identifies small or occluded objects, limiting practical use.Scaling model size and training data is costly, often consuming thousands of GPU hours.Source: §1 · p. 1
• Increasing input resolution can improve recognition, but raises computation and can destabilize training when visual sequences exceed the pretrained context length.Increasing LLaVA-1.5 from 384 × 384 to 672 × 672 brings +4.6% on TextVQA; 448 × 448 raises LLaVA’s computation complexity about 1.4 times versus 336 × 336.Source: §1 · p. 2
• MRA processes high- and low-resolution images in dual visual pathways, embedding high-resolution information into low-resolution modeling through MR-Adapters.The design represents macro-to-micro views with fewer visual tokens and supports images up to 1,536 × 1,536 while maintaining high efficiency.Source: §1 · p. 2
• The paper applies MRA to LLaVA to create LLaVA-HR and evaluates it on 11 vision-language tasks.The evaluated tasks include common benchmarks such as VQA2.0 and emerging benchmarks such as POPE.Source: §1 · p. 2

2.1. Multimodal Large Language Models

• Most MLLMs project visual features into an LLM’s word-embedding space, using linear or MLP layers or sampler-based modules.Sampler-based modules reduce visual-token counts but often require large-scale pretraining for promising performance; most MLLMs still use low resolution such as 336 × 336.Source: §2.1 · p. 3

2.2. Visual Representations for MLLMs

• Prior visual-representation work includes object-level features, CLIP grid features, and scaling visual encoders; this paper instead studies image resolution.PaLI scales its visual encoder to 3 billion parameters and reports a significant MLLM performance boost.Source: §2.2 · p. 3

3. Preliminary

• An MLLM combines an image encoder, an intermediate network, and an LLM to process image and instruction inputs.The image encoder produces visual tokens, while the instruction is represented by corresponding word embeddings.Source: §3 · p. 3
• The LLM decodes target words step by step from visual and textual tokens, with p_t denoting predicted-word probabilities over a vocabulary of size m.Source: §3 · p. 3
• Directly projecting high-resolution visual tokens can make sequences computationally expensive; sampler-based alternatives often need large-scale pretraining for VL alignment.LLaVA-1.5 produces 5,329 tokens at 1,022 × 1,022 resolution.Source: §3 · p. 3
• The central challenge is encoding richer visual information with fewer tokens as high-resolution visual sequences grow rapidly.Source: §3 · p. 3

4.1. Overview

• MRA embeds high-resolution information into the low-resolution pathway, enabling richer visual information with fewer visual tokens.The high-resolution features are fused into the low-resolution pathway before visual features are processed by the LLM.Source: §4.1 · p. 3

4.2. Dual Visual Pathways

• The high-resolution pathway captures fine-grained semantics, while the low-resolution pathway encodes global information with a larger receptive field.The pathways use different downsampling rates so their output features can have the same spatial shape.Source: §4.2 · p. 4

4.3. Mixture-of-Resolution Adapter

• MR-Adapter maps features from both pathways and uses a dynamic score to control the weight of high-resolution information in the low-resolution features.High-resolution information can be fused into the ViT features at each block.Source: §4.3 · p. 4
• Table 1 compares LLaVA-HR and LLaVA-1.5 at different resolutions, reporting performance and training and inference costs measured on NVIDIA A800s.The caption notes that configurations are otherwise held constant and that “N/A” denotes GPU memory overflow.Source: Table 1 · p. 5

4.4. The Deployment on MLLM

• LLaVA-HR is trained first with low-resolution pre-training, then with high-resolution instruction tuning, when MR-Adapter is added and the full model is optimized.In stage one, the image encoder and LLM are frozen and the adapter is absent; stage two increases the high-resolution input size.Source: §4.4 · p. 5

5.1. Evaluations and Metrics

• The evaluation spans four emerging MLLM benchmarks and seven common VL datasets; ScienceQA, VizWiz, and TextVQA are zero-shot tasks.The listed emerging benchmarks are MME, POPE, SEED, and MM-VET.Source: §5.1 · p. 5

5.2. Implementation Details

• LLaVA-HR uses CLIP-ViT-L and CLIP-ConvNeXt-L for its low- and high-resolution paths, with low-resolution pre-training followed by high-resolution instruction tuning.During pre-training, the visual encoder and LLM are frozen and only the MLP projector is fine-tuned; instruction tuning updates the whole model.Source: §5.2 · pp. 5–6

5.3.1. QUANTITATIVE ANALYSIS

• Raising image resolution improves both models on four tasks, but LLaVA-1.5 drops at 1,024 × 1,024 while LLaVA-HR improves consistently, with total gains of +8.33% versus +4.82%.The paper attributes LLaVA-1.5's high-resolution drop to visual-token counts exceeding the LLM's pre-trained context length and destabilizing training.Source: §5.3.1 · p. 6
• At about 760 × 760, all four alternative backbone or token-reduction methods trail LLaVA-HR; ConvNeXt improves efficiency but scores -108.9 on MME relative to it.The comparisons include Conv+MLP, ViT+Resampler, and ViT+Pooling+MLP.Source: §5.3.1 · p. 6, Table 2 · p. 6
• At 1,024 × 1,024, training and inference costs rise 17.6% and 20.8% for LLaVA-HR, versus 344.8% and 325% for LLaVA-1.5; LLaVA-HR is three times faster.LLaVA-HR uses 52G GPU memory at 1,536 × 1,536, while the same LLaVA-1.5 setting causes memory overflow.Source: §5.3.1 · p. 6
• Removing either visual pathway lowers performance, while scaling the high-resolution encoder yields +2.1% on TextVQA versus +0.9% for scaling the low-resolution encoder.Changing fusion direction also degrades performance; the reported best adapter choices are convolution blocks for the low-resolution path, MLP blocks for the high-resolution path, and tanh gating.Source: §5.3.1 · pp. 6–7, Table 3 · p. 6
• LLaVA-HR shows comprehensive advantages on four MLLM benchmarks, leads on three in-domain tasks, and has stronger results on fine-grained zero-shot tasks, while gains are less apparent on low-resolution ScienceQA.The comparisons cover four MLLM benchmarks and seven common VL tasks.Source: Table 4 · p. 7, Table 5 · p. 7, §5.3.1 · pp. 7–8

5.3.2. QUALITATIVE EXPERIMENTS

• Qualitative examples show high resolution capturing a tiny boat and blurred text; LLaVA-HR-X extracts nearly all driver-license content in JSON and matches GPT4-V on the task.LLaVA-HR-X also identifies the person's hair color correctly, while LLaVA-1.5 misses most of the visual information.Source: Figure 5 · p. 8, §5.3.2 · p. 8

• MRA combines high- and low-resolution visual pathways, embedding high-resolution information into low-resolution modeling through MR-Adapters.Source: §6 · p. 8
• Applying MRA to LLaVA-1.5 produces a new high-resolution MLLM, LLaVA-HR.Source: §6 · p. 9
• Experiments validate LLaVA-HR's effectiveness in addressing visual shortcomings and its efficiency against existing MLLMs.Source: §6 · p. 9