PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, Gao Huang

arXiv:2403.11703v1cs.CVcs.AI

TL;DR

Conventional LMMs struggle with fixed-size, low-resolution visual encoding, which limits adaptive and fine-grained perception. LLaVA-UHD addresses this with native-resolution variable-sized slices, token compression, and spatial organization, and outperforms established models across nine benchmarks while supporting 672×1088 images at 94% inference computation. Its GPT-4V analysis also acknowledges that behavior reflects training dynamics and RLHF alongside visual encoding.

  • Problem

    Conventional LMMs use fixed aspect ratios and low resolutions, causing distortion and blur that harm fine-grained visual capabilities.

  • Method

    LLaVA-UHD uses variable-sized native-resolution slices, compressed visual tokens, and a spatial schema to organize slice tokens for LLM processing.

  • Results

    Across 9 benchmarks, LLaVA-UHD outperforms established LMMs trained with 2-3 orders of magnitude more data; its 672×1088 model uses 94% inference computation and improves TextVQA accuracy by 6.4.

  • Takeaways & Limitations

    LLaVA-UHD provides an efficient approach for perceiving native-resolution images across arbitrary aspect ratios and high resolutions.

  • Takeaways & Limitations

    GPT-4V behavior is influenced not only by visual encoding strategies but also by accumulated training dynamics and RLHF.

Abstract

from arXiv · show

Visual encoding constitutes the basis of large multimodal models (LMMs) in understanding the visual world. Conventional LMMs process images in fixed sizes and limited resolutions, while recent explorations in this direction are limited in adaptivity, efficiency, and even correctness. In this work, we first take GPT-4V and LLaVA-1.5 as representative examples and expose systematic flaws rooted in their visual encoding strategy. To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution. LLaVA-UHD includes three key components: (1) An image modularization strategy that divides native-resolution images into smaller variable-sized slices for efficient and extensible encoding, (2) a compression module that further condenses image tokens from visual encoders, and (3) a spatial schema to organize slice tokens for LLMs. Comprehensive experiments show that LLaVA-UHD outperforms established LMMs trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088) resolution images using only 94% inference computation, and achieves 6.4 accuracy improvement on TextVQA. Moreover, the model can be efficiently trained in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of LLaVA-1.5). We make the data and code publicly available at https://github.com/thunlp/LLaVA-UHD.

1 Introduction

Existing LMMs commonly encode images at fixed aspect ratios and low resolutions, causing distortion, blur, and challenges for fine-grained understanding. LLaVA-UHD addresses these issues with adaptive slicing, token compression, and spatial organization, improving performance across nine benchmarks.

  • Fixed 1:1, 224×224 image encoding can distort and blur content, harming small-object understanding, OCR, and factual grounding.
  • High-resolution, arbitrary-aspect-ratio perception presents adaptivity and efficiency challenges for pretrained visual encoders and downstream LLMs.Vision Transformers incur quadratic image-size computation, while high-resolution inputs can produce thousands of visual tokens.
  • GPT-4V and LLaVA-1.5 exhibit systematic visual-encoding issues, motivating mechanistic analysis and more cautious encoding design.
  • LLaVA-UHD modularizes native-resolution images into variable-sized slices, compresses visual tokens, and arranges slice tokens spatially for LLM processing.The design avoids padding and shape-distorting resizing while retaining closer alignment with visual-encoder pretraining settings.
  • Across 9 benchmarks, LLaVA-UHD outperforms established counterparts trained with 2-3 orders of magnitude more data.A LLaVA-1.5 336×336-based model supports 672×1088 images using 94% inference computation and improves TextVQA accuracy by 6.4.
  • The work contributes a mechanistic investigation of GPT-4V flaws, the LLaVA-UHD model, and comprehensive benchmark and analytical evaluations.

2 Pilot Experiments

Pilot experiments reveal position- and resolution-dependent GPT-4V errors and padding-related LLaVA-1.5 failures. These findings motivate LLaVA-UHD’s native-resolution, variable-slice encoding strategy.

  • GPT-4V Experiments: GPT-4V object-count responses vary systematically with object position, with central regions producing the highest counts and corners closest to ground truth.
  • GPT-4V Experiments: GPT-4V’s abnormal object counts include 8 and 16 for a ground truth of 4, potentially reflecting overlapping high-resolution slices.Correct answers account for 66.1%, while answers of 8 and 16 account for 5.2% and 1.9%, respectively.
  • GPT-4V Experiments: GPT-4V responses change in phases as image resolution increases: mostly correct without slices, then dominated by 12, followed by mixed answers of 9, 12, and 16.The authors note that several abnormal phenomena remain unexplained.
  • LLaVA-1.5 Experiments: LLaVA-1.5 padding wastes computation for non-square images and makes padding-like pixels difficult to distinguish from actual image content.A 1:4 image has only 25% effective computation after padding into a square.
  • LLaVA-1.5 Experiments: In padding-based tests, LLaVA-1.5 ignores grey input areas and answers the central rectangle’s color, exposing a systematic encoding behavior.
  • Conclusions on Pilot Experiments: The pilot findings motivate adaptive and efficient encoding, implemented in LLaVA-UHD through variable-sized native-resolution slices, compression, and spatial organization.

3 Method

LLaVA-UHD modularizes native-resolution images into adaptive slices, compresses their visual tokens, and preserves slice layout for LLM processing. Its partitioning and encoding choices target both out-of-distribution distortion and the computational cost of high-resolution perception.

  • Image modularization: LLaVA-UHD divides native-resolution images into variable-sized slices, avoiding padding and shape-distorting resizing while supporting arbitrary aspect ratios.The slices are selected to remain close to the ViT pretraining setting.
  • High-resolution image partition: N = ⌈WI×HI/(Wv×Hv)⌉ estimates the ideal number of slices from image and ViT pretraining areas.The slice count is factorized into m columns and n rows.
  • High-resolution image partition: The partition score favors factorization schemes whose slice shapes deviate least from the ViT pretraining setting.Candidate partitions include factorizations of N−1, N, and N+1 to avoid extreme layouts when N has few factors.
  • High-resolution image partition: For N ≤20 and aspect ratios from 1:6 to 6:1, each slice has aspect ratio 1:2 to 2:1 and area 0.33WIHI to 1.5WIHI.These bounds formalize the partition strategy’s control of slice distortion relative to the standard ViT setting.
  • Arbitrary aspect ratio slice encoding: Slices are proportionally resized, encoded with interpolated 2D ViT position embeddings, and accompanied by a low-resolution native-aspect-ratio overview image.The overview supplies coarse-grained information and global semantic connections, while ViT and position-embedding parameters can remain frozen during pretraining.
  • Token compression and spatial schema: A shared perceiver resampler compresses each slice’s visual tokens, while commas and newline tokens encode slice columns and rows for the LLM.The resampler reduces the token burden from high-resolution encoding, and the spatial schema communicates dynamic slice positions.

4 Experiments

LLaVA-UHD is evaluated against general and high-resolution LMM baselines on nine benchmarks, with accuracy, resolution support, computation, aspect-ratio coverage, and training-data volume reported. It improves over strong baselines and its LLaVA-1.5 backbone, while analyses examine component choices, extreme aspect ratios, visual-encoder tuning, and qualitative recognition.

  • Experimental setting: LLaVA-UHD is evaluated on 9 benchmarks spanning general VQA, OCR-based VQA, hallucination, and comprehensive multimodal assessment.The benchmarks include VQA-V2, GQA, ScienceQA, VizWiz, TextVQA, POPE, MME, MMBench, and MMBench-CN.
  • Experimental setting: The evaluation reports benchmark performance alongside maximum supported resolution, TFLOPs, supported aspect ratio, and accumulated multimodal training data.TFLOPs aggregate visual encoder, projector, and LLM computation for the maximum supported resolution.
  • Main results: LLaVA-UHD outperforms established general and high-resolution baselines, including models trained with 2-3 orders of magnitude more data or requiring substantially more computation.The comparisons include Qwen-VL, InstructBLIP, Fuyu-8B, OtterHD-8B, Monkey, and SPHINX-2k.
  • Main results: 6.4 accuracy improvement on TextVQA and 3.2 accuracy improvement on POPE are achieved over the LLaVA-1.5 backbone.The paper attributes these gains to native high-resolution perception of fine-grained objects and optical characters.
  • Ablation study: The adaptive partition strategy improves performance over padded encoding, the perceiver resampler uses 12.9% computation cost relative to the compared MLP setting, and spatial schema removal degrades performance.The ablations also report better performance than a naive fixed 2 × 2 partition strategy.
  • Extreme aspect ratios: LLaVA-UHD’s advantage increases on images with more extreme aspect ratios compared with LLaVA-1.5 and alternatives.The extended benchmarks double the longer side through padding to create more extreme aspect ratios.
  • Visual-encoder tuning: Updating ViT parameters during instruction tuning is sufficient for adaptation, while updating them during pretraining does not improve results.LLaVA-UHD can improve over LLaVA-1.5 even with ViT parameters frozen in both stages.
  • Case study: Qualitative cases show LLaVA-UHD recognizing dense timetable content, small-poster text, and phone icons and text that LLaVA-1.5 perceives only coarsely.These cases concern fine-grained recognition and reasoning in high-resolution images.

5 Related Work

Related work frames visual encoding as the interface between image features and LLMs, while existing high-resolution LMMs use fixed-resolution, resized, cropped, or otherwise limited strategies. LLaVA-UHD is positioned as an adaptive alternative supporting native aspect ratios and high resolutions with improved computation efficiency.

  • Visual encoding in LMMs: Visual encoding is central to LMMs because visual signals must be projected into LLMs for image understanding.CLIP aligns visual and language embeddings using image-text pairs.
  • Visual encoding in LMMs: Existing visual projection methods include perceiver resamplers, Q-Formers, and MLP connectors between visual encoders and LLMs.Flamingo, BLIP-2, and LLaVA exemplify these three categories.
  • Visual encoding in LMMs: Reliance on fixed-resolution CLIP-ViT inputs limits LMMs’ ability to handle high-resolution or arbitrary-aspect-ratio images and weakens OCR and small-object understanding.Fixed image encoding can also undermine fine-grained downstream tasks.
  • High-resolution LMMs: High-resolution LMM approaches include up-resizing, fixed-size cropping, and enumerated resolution types, each retaining limitations in visual representation or aspect-ratio adaptivity.Up-resizing can substantially alter CLIP-ViT position encoding, while fixed cropping uses predefined windows and slices.
  • LLaVA-UHD: LLaVA-UHD combines modularized adaptive encoding with a perceiver resampler to support any aspect ratio and high resolution with improved computation efficiency.Its native-resolution approach is contrasted with methods using fixed resolutions or several predefined shape types.

6 Conclusion

LLaVA-UHD efficiently perceives images with arbitrary aspect ratios and high resolutions, with effectiveness demonstrated across 9 benchmarks. The paper also identifies resolution limits and independent slice encoding as directions for future improvement.

  • LLaVA-UHD efficiently perceives images with any aspect ratio and high resolution.
  • 9 benchmarks demonstrate the model’s effectiveness, especially for fine-grained multimodal capabilities.
  • The model is currently limited to 672 × 1008 resolution and independently encodes image slices.Future work targets higher resolutions, small object detection and segmentation, and efficient interactions between slices.

A Detailed Illustration on GPT-4V Phases

The GPT-4V analysis finds resolution-dependent response phases that are consistent with changes in image slicing and padding. Some response patterns are explainable through slice structure, while others remain unresolved.

  • GPT-4V responses show a significant phase change with image resolution.The analysis interprets these phases from the perspective of visual encoding.
  • In phase 1, most answers are correct because the image has only one slice.For images under 512 resolution, suspected padding to 512 may explain behavior changes within this phase.
  • In phase 2, answer 12 dominates, possibly because each slice contains incomplete circles.
  • Phase 3 mixes answers 9, 12, and 16, with answer 16 explainable by the slice strategy.Many abnormal phenomena in this phase remain unexplained.

B Proofs

The proofs analyze slice area and aspect-ratio behavior under image-partition assumptions. They show that slices remain close to the standard ViT setting in expected aspect ratio and area concentration.

  • Range of Slice Aspect Ratios: The slice resolution exhibits modest changes relative to the original ViT resolution.
  • Range of Slice Aspect Ratios: The slice aspect ratio is determined from the original image dimensions and the best partition.The slice dimensions are denoted by Wv and Hv, while the original image dimensions are WI and HI.
  • Range of Slice Aspect Ratios: The slice aspect ratio is shown to remain within a bounded range through enumeration of possible partitions.
  • Expected Aspect Ratio: The expected slice aspect ratio is 1:1.258 with variance 0.048 under one image distribution, and 1:1.147 with variance 0.011 under another.Both expected ratios are described as close to the standard ViT setting.
  • Expected Slice Area: Slice partitions consider N−1, N, and N+1 slices, yielding bounds of 1.5 for Smax and 0.33 for Smin in limiting cases.
  • Expected Slice Area: The expected slice-area analysis assumes image sizes uniformly distributed within N ∈ [0, 20] and image aspect ratios within [1, 6].
  • Expected Slice Area: The calculated slice-area variance is 0.016, indicating relatively concentrated areas similar to the original ViT resolution.

C Discussions

The paper identifies resolution and slice-encoding boundaries, and warns that its analysis of model vulnerabilities could enable adversarial attacks. It outlines higher-resolution processing and improved inter-slice connections as future directions.

  • Limitations and Future Work: LLaVA-UHD is limited to a maximum resolution of 672 × 1008, leaving 4K and remote sensing images out of reach.The authors plan to explore higher-resolution images and tasks such as small object detection and segmentation.
  • Limitations and Future Work: Image slices are independently encoded, with interactions occurring only in the LLMs.Future work will establish efficient connections between slices for fine-grained global information interaction.
  • Potential Negative Impact: The GPT-4V and LLaVA-1.5 failure analysis could potentially be used for adversarial attacks on these models.The paper calls for further efforts to mitigate the revealed issues and improve LMM robustness and safety.
Loading 2403.11703v1…